JCWiki - User contributions [en]

Cabinetmap

2014-01-11T08:37:13Z

24.165.90.109:

<table cellpadding=5>
<tr>
<td valign="top"><table border=1>

<tr><th>u</th> <th style="background:#FFFF99">1 </th><th>sw port</th> <th>ats port</th></tr>
<tr><td>42</td><td style="background:#a5e9f3">[https://secure.johncompanies.com/mgmt/mrtg/switch.cgi?s=switch-p25&path= c2950 P25]
</td><td class=p>1</td> <td class=p>9</td></tr>
<tr><td>41</td><td>&nbsp </td><td class=p> </td> <td class=p></td></tr>
<tr><td>40</td><td>  </td><td class=p> </td> <td class=p></td></tr>
<tr><td>39</td><td>  </td><td class=p> </td> <td class=p></td></tr>
<tr><td>38</td><td>  </td><td class=p> </td> <td class=p></td></tr>
<tr><td>37</td><td>  </td><td class=p> </td> <td class=p></td></tr>
<tr><td>36</td><td>col02113 </td><td class=p>19 </td> <td class=p>1</td></tr>
<tr><td>35</td><td rowspan=2>col01402 RJclients </td><td class=p rowspan=2>2</td> <td class=p rowspan=2></td></tr>
<tr><td>34</td></tr>
<tr><td>33</td><td rowspan=2>col1500 </td><td class=p rowspan=2>3</td> <td class=p rowspan=2>15</td></tr>
<tr><td>32</td></tr>
<tr><td>31</td><td rowspan=2>col00138 data4 </td><td class=p rowspan=2>4</td> <td class=p rowspan=2>14</td> </tr>
<tr><td>30</td></tr>
<tr><td>29</td><td rowspan=2>col01899 </td><td class=p rowspan=2>5,21 (RMM)</td> <td class=p rowspan=2>13</td></tr>
<tr><td>28</td></tr>
<tr><td>27</td><td rowspan=2>col01741 </td><td class=p rowspan=2>6</td> <td class=p rowspan=2>12</td></tr>
<tr><td>26</td></tr>
<tr><td>25</td><td rowspan=2>col01369 logos JC-08017 </td><td class=p rowspan=2>7</td> <td class=p rowspan=2>11</td></tr>
<tr><td>24</td></tr>
<tr><td>23</td><td>col01114 </td><td class=p>8</td> <td class=p >10</td></tr>
<tr><td>22</td><td> </td><td class=p>9</td> <td class=p >2-1</td></tr>
<tr><td>21</td><td colspan=3>  </td></tr>
<tr><td>20</td><td style="text-align:center; background:#cecece">[https://ats1.johncompanies.com ATS-1]
</td><td class=p>24</td> <td class=p></td></tr>
<tr><td>19</td><td colspan=3>  </td></tr>
<tr><td>18</td><td rowspan=2>col00574 </td><td class=p rowspan=2>9</td> <td class=p rowspan=2>7</td></tr>
<tr><td>17</td></tr>
<tr><td>16</td><td rowspan=2>col02210 JC-13009 </td><td class=p rowspan=2>10,11(RMM)</td> <td class=p rowspan=2>6</td></tr>
<tr><td>15</td></tr>
<tr><td>14</td><td rowspan=2>col02078 raven.nostrum.com </td><td class=p rowspan=2>17</td> <td class=p rowspan=2>5</td></tr>
<tr><td>13</td></tr>
<tr><td>12</td><td rowspan=2>col02184 JC-08028 </td><td class=p rowspan=2>12 </td> <td class=p rowspan=2>4</td></tr>
<tr><td>11</td></tr>
<tr><td>10</td><td rowspan=2> </td><td class=p rowspan=2></td> <td class=p rowspan=2>3</td></tr>
<tr><td>9 </td></tr>
<tr><td>8 </td><td rowspan=2>col00726 </td><td class=p rowspan=2>15</td> <td class=p rowspan=2>2</td></tr>
<tr><td>7 </td></tr>
<tr><td>6 </td><td rowspan=2>col01257 JC-13004 </td><td class=p rowspan=2>14 16 (RMM)</td> <td class=p rowspan=2>8</td></tr>
<tr><td>5 </td><td>  </td><td class=p></td> <td class=p></td></tr>

</table></td>

<td valign="top"><table border=1>

<tr><th>u</th> <th style="background:#FFFF99">2 </th><th>sw port</th> <th>ats port</th></tr>
<tr><td>42</td><td style="background:#a5e9f3">[https://secure.johncompanies.com/mgmt/mrtg/switch.cgi?s=switch-p24&path= c2950 P24] </td><td class=p>1</td> <td class=p>9</td></tr>
<tr><td>41</td><td>&nbsp </td><td class=p> </td> <td class=p></td></tr>
<tr><td>40</td><td>  </td><td class=p> </td> <td class=p></td></tr>
<tr><td>39</td><td>  </td><td class=p> </td> <td class=p></td></tr>
<tr><td>38</td><td>  </td><td class=p> </td> <td class=p></td></tr>
<tr><td>37</td><td>  </td><td class=p> </td> <td class=p></td></tr>
<tr><td>36</td><td>  </td><td class=p> </td> <td class=p></td></tr>
<tr><td>35</td><td rowspan=2>col01378 </td><td class=p rowspan=2>2,15</td> <td class=p rowspan=2>15</td></tr>
<tr><td>34</td></tr>
<tr><td>33</td><td rowspan=2>col01629 newserver </td><td class=p rowspan=2>3</td> <td class=p rowspan=2>14</td></tr>
<tr><td>32</td></tr>
<tr><td>31</td><td rowspan=2>col00138 www </td><td class=p rowspan=2>4</td> <td class=p rowspan=2>13</td></tr>
<tr><td>30</td></tr>
<tr><td>29</td><td rowspan=2>col01631 </td><td class=p rowspan=2>5</td> <td class=p rowspan=2>12</td></tr>
<tr><td>28</td></tr>
<tr><td>27</td><td rowspan=2>col01621 agony JC-08013 </td><td class=p rowspan=2>6</td> <td class=p rowspan=2>11</td></tr>
<tr><td>26</td></tr>
<tr><td>25</td><td rowspan=2>col01567 KPSurplus </td><td class=p rowspan=2>7</td> <td class=p rowspan=2>10</td></tr>
<tr><td>24</td></tr>
<tr><td>23</td><td rowspan=2>col02216 JC-06044 webengineer </td><td class=p rowspan=2>18</td> <td class=p rowspan=2>8</td></tr>
<tr><td>22</td></tr>
<tr><td>21</td><td colspan=3>  </td></tr>
<tr><td>20</td><td style="text-align:center; background:#cecece">[https://ats2.johncompanies.com ATS-2]
</td><td class=p>24</td> <td class=p></td></tr>
<tr><td>19</td><td colspan=3>  </td></tr>
<tr><td>18</td><td rowspan=2>col01996 JC-08003 </td><td class=p rowspan=2>9</td> <td class=p rowspan=2>7</td></tr>
<tr><td>17</td></tr>
<tr><td>16</td><td rowspan=2>col01711 JC-07030 * </td><td class=p rowspan=2>10</td> <td class=p rowspan=2>6</td></tr>
<tr><td>15</td></tr>
<tr><td>14</td><td rowspan=2>col01788 JC-06021 </td><td class=p rowspan=2>11</td> <td class=p rowspan=2>5</td></tr>
<tr><td>13</td></tr>
<tr><td>12</td><td rowspan=2>col00226 JC-07037 </td><td class=p rowspan=2>12</td> <td class=p rowspan=2>4</td></tr>
<tr><td>11</td></tr>
<tr><td>10</td><td rowspan=2>col00633 JC-08044 </td><td class=p rowspan=2>13</td> <td class=p rowspan=2>3</td></tr>
<tr><td>9</td></tr>
<tr><td>8</td><td rowspan=2>col00433 djc-jc JC-07015 </td><td class=p rowspan=2>14</td> <td class=p rowspan=2>2</td></tr>
<tr><td>7</td></tr>
<tr><td>4</td><td rowspan=2>col00433 Dell 2950 </td><td class=p rowspan=2>16 DRAC 17</td> <td class=p rowspan=2>2</td></tr>
<tr><td>5</td></tr>

</table></td>

<td valign="top"><table border=1>

<tr><th>u</th> <th style="background:#FFFF99">3 </th><th>sw port</th> <th>ats port</th></tr>
<tr><td>42</td><td style="background:#a5e9f3">[https://secure.johncompanies.com/mgmt/mrtg/switch.cgi?s=switch-p23&path= c2950 P23] </td><td class=p>1</td> <td class=p>9</td></tr>
<tr><td>41</td><td>&nbsp </td><td class=p> </td> <td class=p></td></tr>
<tr><td>40</td><td>  </td><td class=p> </td> <td class=p></td></tr>
<tr><td>39</td><td>  </td><td class=p> </td> <td class=p></td></tr>
<tr><td>38</td><td>  </td><td class=p> </td> <td class=p></td></tr>
<tr><td>37</td><td rowspan=2>col01727 JC-10019 </td><td class=p rowspan=2>19, 23(RMM)</td> <td class=p rowspan=2>1</td></tr>
<tr><td>36</td><td>
<tr><td>35</td><td rowspan=2>col01408 issoasis.com </td><td class=p rowspan=2>2 16 (RMM)</td> <td class=p rowspan=2>15</td></tr>
<tr><td>34</td></tr>
<tr><td>33</td><td rowspan=2>col00789 on001 </td><td class=p rowspan=2>3</td> <td class=p rowspan=2>14</td></tr>
<tr><td>32</td></tr>
<tr><td>31</td><td rowspan=2>col00789 laertes </td><td class=p rowspan=2>4</td> <td class=p rowspan=2>13</td></tr>
<tr><td>30</td></tr>
<tr><td>29</td><td rowspan=2>col02152 NO LABEL </td><td class=p rowspan=2>5</td> <td class=p rowspan=2>12</td></tr>
<tr><td>28</td></tr>
<tr><td>27</td><td rowspan=2>col02219 JC-0811 </td><td class=p rowspan=2>6</td> <td class=p rowspan=2>11</td></tr>
<tr><td>26</td></tr>
<tr><td>25</td><td rowspan=2>col01333 AJA Video </td><td class=p rowspan=2>P24-22</td> <td class=p rowspan=2>10</td></tr>
<tr><td>24</td></tr>
<tr><td>23</td><td rowspan=2>col01741 cyrus.base </td><td class=p rowspan=2>15, 22 (RMM)</td> <td class=p rowspan=2>8</td></tr>
<tr><td>22</td></tr>
<tr><td>21</td><td colspan=3>  </td></tr>
<tr><td>20</td><td style="text-align:center; background:#cecece">[https://ats3.johncompanies.com ATS-3]
</td><td class=p>24</td> <td class=p></td></tr>
<tr><td>19</td><td colspan=3>  </td></tr>
<tr><td>18</td><td rowspan=2>col00789 bb001 </td><td class=p rowspan=2>9, 20 (RMM)</td> <td class=p rowspan=2>7</td></tr>
<tr><td>17</td></tr>
<tr><td>16</td><td rowspan=2>col01224 pariotrial.sea-incorporated </td><td class=p rowspan=2>10, 21 (RMM)</td> <td class=p rowspan=2>6</td></tr>
<tr><td>15</td></tr>
<tr><td>14</td><td rowspan=2>col01777 </td><td class=p rowspan=2>11</td> <td class=p rowspan=2>5</td></tr>
<tr><td>13</td></tr>
<tr><td>12</td><td rowspan=2>col01224 fuji </td><td class=p rowspan=2>12</td> <td class=p rowspan=2>4</td></tr>
<tr><td>11</td></tr>
<tr><td>10</td><td rowspan=2>col01817 </td><td class=p rowspan=2>13</td> <td class=p rowspan=2>3</td></tr>
<tr><td>9</td></tr>
<tr><td>8</td><td rowspan=2>col01332 vs4 </td><td class=p rowspan=2>14</td> <td class=p rowspan=2>2</td></tr>
<tr><td>7</td></tr>
<tr><td>6</td><td rowspan=2>col02202 JC-13006 </td><td class=p rowspan=2>8 17 (RMM)</td> <td class=p rowspan=2>16</td></tr>
<tr><td>5</td><td>  </td><td class=p></td> <td class=p></td></tr>

</table></td>

<td valign="top"><table border=1>

<tr><th>u</th> <th style="background:#FFFF99">4 </th><th>sw port</th> <th>ats port</th></tr>
<tr><td>42</td><td style="background:#a5e9f3">[https://secure.johncompanies.com/mgmt/mrtg/switch.cgi?s=switch-p22&path= c2950 P22] </td><td class=p>1</td> <td class=p>9</td></tr>
<tr><td>41</td><td>&nbsp </td><td class=p> </td> <td class=p></td></tr>
<tr><td>40</td><td>  </td><td class=p> </td> <td class=p></td></tr>
<tr><td>39</td><td>  </td><td class=p> </td> <td class=p></td></tr>
<tr><td>38</td><td>  </td><td class=p> </td> <td class=p></td></tr>
<tr><td>37</td><td>  </td><td class=p> </td> <td class=p></td></tr>
<tr><td>36</td><td>  </td><td class=p> </td> <td class=p></td></tr>
<tr><td>35</td><td rowspan=2>col01224 cisco (no rails) </td><td class=p rowspan=2>2</td> <td class=p rowspan=2>16</td></tr>
<tr><td>34</td></tr>
<tr><td>33</td><td rowspan=2>col01224 obc-c </td><td class=p rowspan=2> </td> <td class=p rowspan=2>15</td></tr>
<tr><td>32</td></tr>
<tr><td>31</td><td rowspan=2>col01224 obc-b </td><td class=p rowspan=2> </td> <td class=p rowspan=2>14</td></tr>
<tr><td>30</td></tr>
<tr><td>29</td><td rowspan=2>col01224 obc-a </td><td class=p rowspan=2> </td> <td class=p rowspan=2>13</td></tr>
<tr><td>28</td></tr>
<tr><td>27</td><td rowspan=2>col02220 </td><td class=p rowspan=2>13</td> <td class=p rowspan=2>12</td></tr>
<tr><td>26</td></tr>
<tr><td>25</td><td>col00534 </td><td class=p>4</td> <td class=p>11</td></tr>
<tr><td>24</td><td> </td><td class=p></td> <td class=p></td></tr>
<tr><td>23</td><td rowspan=2>col01224 hydro4gehosting </td><td class=p rowspan=2>6</td> <td class=p rowspan=2>8</td></tr>
<tr><td>22</td></tr>
<tr><td>21</td><td colspan=3>  </td></tr>
<tr><td>20</td><td style="text-align:center; background:#cecece">[https://ats4.johncompanies.com ATS-4]
</td><td class=p>24</td> <td class=p></td></tr>
<tr><td>19</td><td colspan=3>  </td></tr>
<tr><td>18</td><td rowspan=2>col01182 dev01 </td><td class=p rowspan=2>7</td> <td class=p rowspan=2>7</td></tr>
<tr><td>17</td></tr>
<tr><td>16</td><td rowspan=2>col01182 demo2 </td><td class=p rowspan=2>8</td> <td class=p rowspan=2>6</td></tr>
<tr><td>15</td></tr>
<tr><td>14</td><td rowspan=2>col00336 rb3 </td><td class=p rowspan=2>9</td> <td class=p rowspan=2>5</td></tr>
<tr><td>13</td></tr>
<tr><td>12</td><td rowspan=2>col02221 </td><td class=p rowspan=2>10</td> <td class=p rowspan=2>4</td></tr>
<tr><td>11</td></tr>
<tr><td>10</td><td rowspan=2>col01710 </td><td class=p rowspan=2>11</td> <td class=p rowspan=2>3</td></tr>
<tr><td>9</td></tr>
<tr><td>8</td><td rowspan=2>col00336 coach </td><td class=p rowspan=2>12</td> <td class=p rowspan=2>2</td></tr>
<tr><td>7</td></tr>
<tr><td>6</td><td rowspan=2>col00534 </td><td class=p rowspan=2>18 RMM 19</td> <td class=p rowspan=2>10</td></tr>
<tr><td>5</td><td></tr>

</table></td>

<td valign="top"><table border=1>

<tr><th>u</th> <th style="background:#FFFF99">5 </th><th>sw port</th> <th>ats port</th></tr>
<tr><td>42</td><td style="background:#a5e9f3">[https://secure.johncompanies.com/mgmt/mrtg/switch.cgi?s=switch-p21&path= c2950 P21]
</td><td class=p>g1</td> <td class=p>9</td></tr>
<tr><td>41</td><td>&nbsp </td><td class=p> </td> <td class=p></td></tr>
<tr><td>40</td><td>  </td><td class=p> </td> <td class=p></td></tr>
<tr><td>39</td><td>  </td><td class=p> </td> <td class=p></td></tr>
<tr><td>38</td><td>  </td><td class=p> </td> <td class=p></td></tr>
<tr><td>37</td><td rowspan=2>col01849 snacktime </td><td class=p rowspan=2>16 17 (RMM)</td> <td class=p rowspan=2>2</td></tr>
<tr><td>36</td><td>
<tr><td>35</td><td rowspan=2>col01080 lithium </td><td class=p rowspan=2>1</td> <td class=p rowspan=2>16</td></tr>
<tr><td>34</td></tr>
<tr><td>33</td><td rowspan=2>col01224 ewing </td><td class=p rowspan=2>2</td> <td class=p rowspan=2>15</td></tr>
<tr><td>32</td></tr>
<tr><td>31</td><td rowspan=2>col01224 luzerne </td><td class=p rowspan=2>11</td> <td class=p rowspan=2>14</td></tr>
<tr><td>30</td></tr>
<tr><td>29</td><td rowspan=2>col02215 JC-08019 </td><td class=p rowspan=2>20</td> <td class=p rowspan=2>13</td></tr>
<tr><td>28</td></tr>
<tr><td>27</td><td rowspan=2>col01556 </td><td class=p rowspan=2>5</td> <td class=p rowspan=2>12</td></tr>
<tr><td>26</td></tr>
<tr><td>25</td><td rowspan=2>col02138 webengineers </td><td class=p rowspan=2>7, 21 (RMM)</td> <td class=p rowspan=2>11</td></tr>
<tr><td>24</td></tr>
<tr><td>23</td><td rowspan=2>col01495 beany </td><td class=p rowspan=2>22</td> <td class=p rowspan=2>10</td></tr>
<tr><td>22</td></tr>
<tr><td>21</td><td colspan=3>  </td></tr>
<tr><td>20</td><td style="text-align:center; background:#cecece">[https://ats5.johncompanies.com ATS-5]
</td><td class=p>24</td> <td class=p></td></tr>
<tr><td>19</td><td colspan=3>  </td></tr>
<tr><td>18</td><td rowspan=2>col01841 paradigm </td><td class=p rowspan=2>12 13 (DRAC)</td> <td class=p rowspan=2>6</td></tr>
<tr><td>17</td></tr>
<tr><td>16</td><td rowspan=2>col01080 helium </td><td class=p rowspan=2>9</td> <td class=p rowspan=2>5</td></tr>
<tr><td>15</td></tr>
<tr><td>14</td><td rowspan=2>col01629 anitaborg mail </td><td class=p rowspan=2>10</td> <td class=p rowspan=2>7</td></tr>
<tr><td>13</td></tr>
<tr><td>12</td><td rowspan=2>col02205 </td><td class=p rowspan=2>5 3 (RMM)</td> <td class=p rowspan=2>3</td></tr>
<tr><td>11</td></tr>
<tr><td>10</td><td rowspan=2>col01841 smartypants * </td><td class=p rowspan=2>12,13 (DRAC)</td> <td class=p rowspan=2>8</td></tr>
<tr><td>9</td></tr>
<tr><td>8</td><td rowspan=2>col01408 pepclub </td><td class=p rowspan=2>6, 15 (RMM)</td> <td class=p rowspan=2>4</td></tr>
<tr><td>7</td></tr>

</table></td>
</tr>
</table>

<table cellpadding=5>
<tr>
<td valign="top"><table border=1>

<tr><th>u</th> <th style="background:#FFFF99">6 </th><th>sw port</th> <th>ats port</th></tr>
<tr><td>42</td><td class="jc">digi3 </td><td class=p></td><td class=p>10</td></tr>
<tr><td>41</td><td style="background:#a5e9f3">[https://secure.johncompanies.com/mgmt/mrtg/switch.cgi?s=switch-p20&path= c2950 P20]
</td><td class=p></td><td class=p>9</td></tr>
<tr><td>40</td><td style="background:#a5e9f3">[https://secure.johncompanies.com/mgmt/mrtg/switch.cgi?s=switch-p12&path= c2950 P12 private]
</td><td class=p></td><td class=p>2</td></tr>
<tr><td>39</td><td>  </td><td class=p></td><td class=p> </td></tr>
<tr><td>38</td><td>  </td><td class=p></td><td class=p> </td></tr>
<tr><td>37</td><td>isys </td><td class=p></td><td class=p>11</td></tr>
<tr><td>36</td><td rowspan=2>firewall2 </td><td class=p></td><td class=p rowspan=2>8 (A), 7 (B)</td></tr>
<tr><td>35</td></tr>
<tr><td>34</td><td rowspan=2>firewall3 </td><td class=p></td><td class=p rowspan=2>? (A), ? (B)</td></tr>
<tr><td>33</td></tr>
<tr><td>32</td><td rowspan=2>nat2 </td><td class=p></td><td class=p rowspan=2>6</td></tr>
<tr><td>31</td></tr>
<tr><td>30</td><td rowspan=2>bwdb2 </td><td class=p></td><td class=p rowspan=2>5</td></tr>
<tr><td>29</td></tr>
<tr><td>28</td><td rowspan=2>backup3 </td><td class=p></td><td class=p rowspan=2>4</td></tr>
<tr><td>27</td><td>  </td><td class=p></td><td class=p></td></tr>
<tr><td>26</td><td>jail3 (col01737) </td><td class=p></td><td class=p></td></tr>
<tr><td>25</td><td>  </td><td class=p></td><td class=p></td></tr>
<tr><td>24</td><td>  </td><td class=p></td><td class=p></td></tr>
<tr><td>23</td><td>  </td><td class=p></td><td class=p></td></tr>
<tr><td>22</td><td>  </td><td class=p></td><td class=p></td></tr>
<tr><td>21</td><td colspan=3>  </td><td class=p></td><td class=p></td></tr>
<tr><td>20</td><td style="text-align:center; background:#cecece">[https://ats6.johncompanies.com ATS-6]
</td><td class=p></td><td class=p></td></tr>
<tr><td>19</td><td colspan=3>  </td><td class=p></td></tr>
<tr><td>18</td><td rowspan=2> </td><td class=p></td><td class=p>16</td></tr>
<tr><td>17</td></tr>
<tr><td>16</td><td rowspan=2>magrathea </td><td class=p></td><td class=p rowspan=2>3</td></tr>
<tr><td>15</td></tr>
<tr><td>14</td><td> col02015 cisco </td><td class=p>p21 18</td><td class=p>13</td></tr>
<tr><td>13</td><td>  </td><td class=p></td></tr>
<tr><td>12</td><td rowspan=2> col02015 dell </td><td class=p>p21 15 drac</td><td class=p>12</td></tr>
<tr><td>11</td><td>  </td><td class=p></td></tr>
<tr><td>10</td><td>  </td><td class=p></td></tr>
<tr><td>9</td><td>  </td><td class=p></td></tr>
<tr><td>8</td><td>  </td><td class=p></td></tr>
<tr><td>7</td><td>  </td><td class=p></td></tr>

</table></td>

<td valign="top"><table border=1>

<tr><th>u</th> <th style="background:#FFFF99">7 </th><th>sw port</th> <th>ats port</th></tr>
<tr><td>42</td><td style="background:#a5e9f3">[https://secure.johncompanies.com/mgmt/mrtg/switch.cgi?s=switch-p26&path= c2950 P26]
</td><td class=p>1</td> <td class=p>9</td></tr>
<tr><td>41</td><td>&nbsp </td><td class=p> </td> <td class=p></td></tr>
<tr><td>40</td><td>  </td><td class=p> </td> <td class=p></td></tr>
<tr><td>39</td><td>  </td><td class=p> </td> <td class=p></td></tr>
<tr><td>38</td><td>  </td><td class=p> </td> <td class=p></td></tr>
<tr><td>37</td><td>  </td><td class=p> </td> <td class=p></td></tr>
<tr><td>36</td><td>  </td><td class=p> </td> <td class=p></td></tr>
<tr><td>35</td><td rowspan=2> </td><td class=p rowspan=2>?</td> <td class=p rowspan=2>15</td></tr>
<tr><td>34</td></tr>
<tr><td>33</td><td rowspan=2> </td><td class=p rowspan=2>3</td> <td class=p rowspan=2>14</td></tr>
<tr><td>32</td></tr>
<tr><td>31</td><td rowspan=2>col01224 brandgage </td><td class=p rowspan=2>4</td> <td class=p rowspan=2>13</td></tr>
<tr><td>30</td></tr>
<tr><td>29</td><td rowspan=2> </td><td class=p rowspan=2>5</td> <td class=p rowspan=2>12</td></tr>
<tr><td>28</td></tr>
<tr><td>27</td><td rowspan=2> </td><td class=p rowspan=2>6</td> <td class=p rowspan=2>11</td></tr>
<tr><td>26</td></tr>
<tr><td>25</td><td rowspan=2> </td><td class=p rowspan=2>7</td> <td class=p rowspan=2>10</td></tr>
<tr><td>24</td></tr>
<tr><td>23</td><td rowspan=2> </td><td class=p rowspan=2>8</td> <td class=p rowspan=2>8</td></tr>
<tr><td>22</td></tr>
<tr><td>21</td><td colspan=3>  </td></tr>
<tr><td>20</td><td style="text-align:center; background:#cecece">[https://ats7.johncompanies.com ATS-7]
</td><td class=p>24</td> <td class=p></td></tr>
<tr><td>19</td><td colspan=3> </td></tr>
<tr><td>18</td><td rowspan=2>col00875 www * </td><td class=p rowspan=2>9</td> <td class=p rowspan=2>7</td></tr>
<tr><td>17</td></tr>
<tr><td>16</td><td rowspan=2>col00875 backup </td><td class=p rowspan=2></td> <td class=p rowspan=2>6</td></tr>
<tr><td>15</td></tr>
<tr><td>14</td><td rowspan=2> </td><td class=p rowspan=2>10</td> <td class=p rowspan=2>5</td></tr>
<tr><td>13</td></tr>
<tr><td>12</td><td rowspan=2>col02171 </td><td class=p rowspan=2>11</td> <td class=p rowspan=2>4</td></tr>
<tr><td>11</td></tr>
<tr><td>10</td><td rowspan=2>jc11013 label </td><td class=p rowspan=2>12</td> <td class=p rowspan=2>3</td></tr>
<tr><td>9</td></tr>
<tr><td>8</td><td rowspan=2>col02108 </td><td class=p rowspan=2>13</td> <td class=p rowspan=2>2</td></tr>
<tr><td>7</td></tr>
<tr><td>6</td><td rowspan=4> On Floor -- col00336 fumble.fantasybowl.com (Atom) and col02108 JC-11009 (Atom)</td><td class=p>12</td> <td class=p>1</td></tr>
<tr><td>5</td><td>  </td><td class=p></td> <td class=p></td></tr>
<tr><td>4</td><td>  </td><td class=p></td> <td class=p></td></tr>
<tr><td>3</td><td>  </td><td class=p></td> <td class=p></td></tr>

</table></td>

<td valign="top"><table border=1>

<tr><th>u</th> <th style="background:#FFFF99">8 </th><th>sw port</th> <th>ats port</th></tr>
<tr><td>42</td><td style="background:#a5e9f3">[https://secure.johncompanies.com/mgmt/mrtg/switch.cgi?s=switch-p27&path= c2950 P27]
</td><td class=p>1</td> <td class=p>9</td></tr>
<tr><td>41</td><td>&nbsp </td><td class=p> </td> <td class=p></td></tr>
<tr><td>40</td><td>  </td><td class=p> </td> <td class=p></td></tr>
<tr><td>39</td><td>  </td><td class=p> </td> <td class=p></td></tr>
<tr><td>38</td><td>  </td><td class=p> </td> <td class=p></td></tr>
<tr><td>37</td><td rowspan=2>col01719 (1U) </td><td class=p rowspan=2>16</td> <td class=p rowspan=2>16</td></tr>
<tr><td>36</td></tr>
<tr><td>35</td><td rowspan=2> </td><td class=p rowspan=2>2</td> <td class=p rowspan=2>15</td></tr>
<tr><td>34</td></tr>
<tr><td>33</td><td rowspan=2>col01224 idgroup </td><td class=p rowspan=2>3</td> <td class=p rowspan=2>14</td></tr>
<tr><td>32</td></tr>
<tr><td>31</td><td rowspan=2> </td><td class=p rowspan=2>4</td> <td class=p rowspan=2>13</td></tr>
<tr><td>30</td></tr>
<tr><td>29</td><td rowspan=2>col02150 </td><td class=p rowspan=2>5, 21 (RMM)</td> <td class=p rowspan=2>12</td></tr>
<tr><td>28</td></tr>
<tr><td>27</td><td rowspan=2>col01860 </td><td class=p rowspan=2>6</td> <td class=p rowspan=2>11</td></tr>
<tr><td>26</td></tr>
<tr><td>25</td><td rowspan=2>col01025 feros </td><td class=p rowspan=2>7, 18 (RMM)</td> <td class=p rowspan=2>10</td></tr>
<tr><td>24</td></tr>
<tr><td>23</td><td rowspan=2> </td><td class=p rowspan=2>8</td> <td class=p rowspan=2>8</td></tr>
<tr><td>22</td></tr>
<tr><td>21</td><td colspan=3>  </td></tr>
<tr><td>20</td><td style="text-align:center; background:#cecece">[https://ats8.johncompanies.com ATS-8]
</td><td class=p>24</td> <td class=p></td></tr>
<tr><td>19</td><td colspan=3> </td></tr>
<tr><td>18</td><td rowspan=2>col01941 </td><td class=p rowspan=2>9</td> <td class=p rowspan=2>7</td></tr>
<tr><td>17</td></tr>
<tr><td>16</td><td rowspan=2>col01511 * </td><td class=p rowspan=2>10</td> <td class=p rowspan=2>6</td></tr>
<tr><td>15</td></tr>
<tr><td>14</td><td rowspan=2>col00138 data2 </td><td class=p rowspan=2>15</td> <td class=p rowspan=2>5</td></tr>
<tr><td>13</td></tr>
<tr><td>12</td><td rowspan=2>col01567 prod </td><td class=p rowspan=2>12</td> <td class=p rowspan=2>4</td></tr>
<tr><td>11</td></tr>
<tr><td>10</td><td rowspan=3> </td><td class=p rowspan=3></td> <td class=p rowspan=3>3</td></tr>
<tr><td>9</td></tr>
<tr><td>8</td></tr>
<tr><td>7</td><td rowspan=2>col02001 attic1jc </td><td class=p rowspan=2>11</td> <td class=p rowspan=2>2</td></tr>
<tr><td>6</td></tr>

</table></td>

<td valign="top"><table border=1>

<tr><th>u</th> <th style="background:#FFFF99">9 </th><th>sw port</th> <th>ats port</th></tr>
<tr><td>42</td><td style="background:#a5e9f3">[https://secure.johncompanies.com/mgmt/mrtg/switch.cgi?s=switch-p28&path= c2950 P28]
</td><td class=p>1</td> <td class=p>9</td></tr>
<tr><td>41</td><td>&nbsp </td><td class=p> </td> <td class=p></td></tr>
<tr><td>40</td><td>  </td><td class=p> </td> <td class=p></td></tr>
<tr><td>39</td><td>  </td><td class=p> </td> <td class=p></td></tr>
<tr><td>38</td><td>  </td><td class=p> </td> <td class=p></td></tr>
<tr><td>37</td><td>  </td><td class=p> </td> <td class=p></td></tr>
<tr><td>36</td><td>  </td><td class=p> </td> <td class=p></td></tr>
<tr><td>35</td><td rowspan=2> </td><td class=p rowspan=2>2</td> <td class=p rowspan=2>15</td></tr>
<tr><td>34</td></tr>
<tr><td>33</td><td rowspan=2> </td><td class=p rowspan=2>3 <td> <td class=p rowspan=2>14</td></tr>
<tr><td>32</td></tr>
<tr><td>31</td><td rowspan=2> </td><td class=p rowspan=2>4</td> <td class=p rowspan=2>13</td></tr>
<tr><td>30</td></tr>
<tr><td>29</td><td rowspan=2> </td><td class=p rowspan=2>5</td> <td class=p rowspan=2>12</td></tr>
<tr><td>28</td></tr>
<tr><td>27</td><td rowspan=2> </td><td class=p rowspan=2>6</td> <td class=p rowspan=2>11</td></tr>
<tr><td>26</td></tr>
<tr><td>25</td><td rowspan=2> </td><td class=p rowspan=2>7</td> <td class=p rowspan=2>10</td></tr>
<tr><td>24</td></tr>
<tr><td>23</td><td rowspan=2> </td><td class=p rowspan=2>8</td> <td class=p rowspan=2>8</td></tr>
<tr><td>22</td></tr>
<tr><td>21</td><td colspan=3>  </td></tr>
<tr><td>20</td><td style="text-align:center; background:#cecece">[https://ats9.johncompanies.com ATS-9]
</td><td class=p>24</td> <td class=p></td></tr>
<tr><td>19</td><td colspan=3> </td></tr>
<tr><td>18</td><td rowspan=2> </td><td class=p rowspan=2>9</td> <td class=p rowspan=2>7</td></tr>
<tr><td>17</td></tr>
<tr><td>16</td><td rowspan=2> </td><td class=p rowspan=2>10</td> <td class=p rowspan=2>6</td></tr>
<tr><td>15</td></tr>
<tr><td>14</td><td rowspan=2> </td><td class=p rowspan=2>11</td> <td class=p rowspan=2>5</td></tr>
<tr><td>13</td></tr>
<tr><td>12</td><td rowspan=2> </td><td class=p rowspan=2>12</td> <td class=p rowspan=2>4</td></tr>
<tr><td>11</td></tr>
<tr><td>10</td><td rowspan=2> </td><td class=p rowspan=2>13</td> <td class=p rowspan=2>3</td></tr>
<tr><td>9</td></tr>
<tr><td>8</td><td rowspan=2> * </td><td class=p rowspan=2>14</td> <td class=p rowspan=2>2</td></tr>
<tr><td>7</td></tr>

</table></td>

</tr>
</table>
<br>
<table border=0 cellpadding=5>
<tr>
<td valign="top">
<table border=1>

<tr><th>U</th> <th style="background:#FFFF99">3-5 </th> <th>port</th></tr>
<tr><td class=u>129-127</td> <td style="background:#a5e9f3">[https://secure.johncompanies.com/mgmt/mrtg/switch.cgi?s=switch-p3&path= c2924 P3] </td> <td class=p></td></tr>
<tr><td class=u>126-124</td> <td class=digi>digi2 </td> <td class=p></td></tr>
<tr><td class=u>123<br>118</td> <td>2U </td> <td class=p></td></tr>
<tr><td class=u>117<br>112</td> <td style="background:#f5ac81">virt14 * Dell 2950 </td> <td class=p>13</td></tr>
<tr><td class=u>111<br>106</td> <td style="background:#8df595">jail19 * </td> <td class=p></td></tr>
<tr><td class=u>105<br>100</td> <td style="background:#f5ac81">virt15 * </td> <td class=p></td></tr>
<tr><td class=u>99<br>94</td> <td>col00779 wendlink 233.77 </td> <td class=p></td></tr>
<tr><td class=u>93<br>88</td> <td style="background:#8df595">jail2 * </td> <td class=p></td></tr>
<tr><td class=u>87<br>82</td> <td style="background:#f5ac81">virt13 * Dell 2950 </td> <td class=p></td></tr>
<tr><td class=u>81<br>76</td> <td style="background:#8df595">old jail2 * </td> <td class=p></td></tr>
<tr><td class=u>75<br>70</td> <td style="background:#8df595">mx2 * </td> <td class=p></td></tr>
<tr><td class=u>69<br>64</td> <td>2U </td> <td class=p></td></tr>
<tr><td class=u>63<br>58</td> <td>col01372 cache2 </td> <td class=p></td></tr>
<tr><td class=u>57<br>52</td> <td>2U </td> <td class=p></td></tr>
<tr><td class=u>51<br>46</td> <td>col00499 </td> <td class=p></td></tr>
<tr><td class=u>45<br>40</td> <td>col02204 NIC P3-12 RMM P3-16 </td> <td class=p></td></tr>
<tr><td class=u>39<br>34</td> <td class=c>2U </td> <td class=p></td></tr>
<tr><td class=u>33<br>28</td> <td>col01727 - REMOVED 9/22/13 </td> <td class=p></td></tr>
</table>
</td>

<td valign="top">
<table border=1>

<tr><th>U</th> <th style="background:#FFFF99">3-6 </th> <th>port</th></tr>
<tr><td class=u>129-127</td> <td style="background:#a5e9f3">[https://secure.johncompanies.com/mgmt/mrtg/switch.cgi?s=switch-p4&path= c2924 P4] </td> <td class=p></td></tr>
<tr><td class=u>126-124</td> <td>1U </td> <td class=p></td></tr>
<tr><td class=u>123<br>118</td> <td style="background:#f5ac81">virt11 * </td> <td class=p>4</td></tr>
<tr><td class=u>117<br>112</td> <td style="background:#f5ac81">virt17 * </td> <td class=p>7</td></tr>
<tr><td class=u>111-109</td> <td>1U </td> <td class=p></td></tr>
<tr><td class=u>108-106</td> <td>2U </td> <td class=p></td></tr>
<tr><td class=u>105<br>100</td> <td style="background:#8df595">jail8 * </td> <td class=p>10</td></tr>
<tr><td class=u>99<br>94</td> <td style="background:#f5ac81">virt19 * </td> <td class=p>12</td></tr>
<tr><td class=u>93<br>88</td> <td style="background:#8df595">jail9 * </td> <td class=p></td></tr>
<tr><td class=u>87<br>82</td> <td>col01372 cache4 </td> <td class=p></td></tr>
<tr><td class=u>81<br>76</td> <td>col01372 dbmsg.bookoo </td> <td class=p></td></tr>
<tr><td class=u>75<br>70</td> <td>2U </td> <td class=p></td></tr>
<tr><td class=u>69<br>64</td> <td>2U </td> <td class=p></td></tr>
<tr><td class=u>63<br>58</td> <td>2U </td> <td class=p></td></tr>
<tr><td class=u>57<br>52</td> <td>2U </td> <td class=p></td></tr>
<tr><td class=u>51<br>46</td> <td>2U </td> <td class=p></td></tr>
<tr><td class=u>45<br>40</td> <td>col00985 </td> <td class=p></td></tr>
<tr><td class=u>39<br>34</td> <td>col01372 cache4 </td> <td class=p></td></tr>
<tr><td class=u>33<br>28</td> <td> </td> <td class=p></td></tr>

</table>
</td>

<td valign="top">
<table border=1>

<tr><th>U</th> <th style="background:#FFFF99">3-7 </th> <th>port</th></tr>
<tr><td class=u>129-127</td> <td style="background:#a5e9f3">[https://secure.johncompanies.com/mgmt/mrtg/switch.cgi?s=switch-p17&path= c2924 P17] </td> <td class=p></td></tr>
<tr><td class=u>126-124</td> <td class=c>1U </td> <td class=p></td></tr>
<tr><td class=u>123-121</td> <td class=digi>digi1 (no rails)</td> <td class=p></td></tr>
<tr><td class=u>120-118</td> <td style="background:#a5e9f3">switch-m2 </td> <td class=p></td></tr>
<tr><td class=u>117-115</td> <td style="text-align:center; background:#cecece">ATS </td> <td class=p></td></tr>
<tr><td class=u>114-112</td> <td style="text-align:center; background:#cecece">ATS </td> <td class=p></td></tr>
<tr><td class=u>111<br>106</td> <td style="background:#8df595">jail11 * </td> <td class=p></td></tr>
<tr><td class=u>105<br>100</td> <td class=c>col01372 solr7.bookoo </td> <td class=p></td></tr>
<tr><td class=u>99<br>94</td> <td>col01372 solr8.bookoo </td> <td class=p></td></tr>
<tr><td class=u>93<br>88</td> <td style="background:#f5ac81">virt12 * </td> <td class=p></td></tr>
<tr><td class=u>87<br>82</td> <td style="background:#8df595">jail7 * </td> <td class=p></td></tr>
<tr><td class=u>81<div class=s3></div>73</td> <td style="background:#f5ac81">virt14 </td> <td class=p></td></tr>
<tr><td class=u>72-70</td> <td style="background:#cecece"> bwdb1 (no rails) </td> <td class=p></td></tr>
<tr><td class=u>69<br>64</td> <td style="background:#f5ac81">virt16 * </td> <td class=p></td></tr>
<tr><td class=u>63-61</td> <td>col00991 </td> <td class=p></td></tr>
<tr><td class=u>60<div class=s4></div>49</td> <td style="background:#cecece">backup2 *</td> <td class=p></td></tr>
<tr><td class=u>48<br>43</td> <td style="background:#8df595">jail18 * </td> <td class=p></td></tr>
<tr><td class=u>42<br>37</td> <td style="background:#8df595">mx1 * </td> <td class=p></td></tr>
<tr><td class=u>36-34</td> <td class=c>1U </td> <td class=p></td></tr>
<tr><td class=u>33-31</td> <td>col00994 </td> <td class=p></td></tr>
<tr><td class=u>30<br>25</td> <td style="background:#f5ac81">virt9 * </td> <td class=p></td></tr>
<tr><td class=u>24<br>19</td> <td>2U </td> <td class=p></td></tr>
<tr><td class=u>18-16</td> <td>col0460 </td> <td class=p></td></tr>
<tr><td class=u>15-13</td> <td class=c>1U </td> <td class=p></td></tr>
<tr><td class=u>12-10</td> <td style="background:#cecece">nat </td> <td class=p></td></tr>

</table>
</td>

<td valign="top">
<table border=1>

<tr><th>U</th> <th style="background:#FFFF99">3-8 </th> <th>port</th></tr>
<tr><td class=u>129-127</td> <td class=c>1U </td> <td class=p></td></tr>
<tr><td class=u>126-124</td> <td style="background:#a5e9f3">c3750 </td> <td class=p></td></tr>
<tr><td class=u>122<br>118</td> <td style="background:#a5e9f3">c3550 </td> <td class=p></td></tr>
<tr><td class=u>117-115</td> <td style="background:#a5e9f3">[https://secure.johncompanies.com/mgmt/mrtg/switch.cgi?s=switch-p1&path= c2924 switch-p1] </td> <td class=p></td></tr>
<tr><td class=u>114-112</td> <td style="background:#a5e9f3">[https://secure.johncompanies.com/mgmt/mrtg/switch.cgi?s=switch-p1a&path= c2960 switch-p1a] </td> <td class=p></td></tr>
<tr><td class=u>111-109</td> <td style="background:#a5e9f3">[https://secure.johncompanies.com/mgmt/mrtg/switch.cgi?s=switch-p1b&path= c2960 switch-p1b] </td> <td class=p></td></tr>
<tr><td class=u>108<br>103</td> <td style="background:#cecece">firewall * </td> <td class=p></td></tr>
<tr><td class=u>102-100</td> <td style="background:#a5e9f3">ASA5500 </td> <td class=p></td></tr>
<tr><td class=u>99-97</td> <td style="background:#a5e9f3">M1 </td> <td class=p></td></tr>
<tr><td class=u>96<br>91</td> <td class=c>col02179 staging.issisystems.com </td> <td class=p></td></tr>
<tr><td class=u>90-88</td> <td style="background:#cecece">gate (old f/w) </td> <td class=p></td></tr>
<tr><td class=u>87<br>82</td> <td style="background:#f5ac81">quar1 * (no rails)</td> <td class=p></td></tr>
<tr><td class=u>81-79</td> <td>[shelf] </td> <td class=p></td></tr>
<tr><td class=u>78-76</td> <td class=c>1U </td> <td class=p></td></tr>
<tr><td class=u>75-73</td> <td style="text-align:center; background:#cecece">ATS </td> <td class=p></td></tr>
<tr><td class=u>72-70</td> <td style="text-align:center; background:#cecece">ATS </td> <td class=p></td></tr>
<tr><td class=u>69<br>64</td> <td style="background:#8df595">jail4</td> <td class=p></td></tr>
<tr><td class=u>63<br>58</td> <td style="background:#8df595">jail1 * (no rails)</td> <td class=p></td></tr>
<tr><td class=u>57-55</td> <td class=c>1U </td> <td class=p></td></tr>
<tr><td class=u>54<br>52</td> <td style="background:#cecece">console (no rails) </td> <td class=p></td></tr>
<tr><td class=u>51<br>46</td> <td style="background:#cecece">mail * (no rails) </td> <td class=p></td></tr>
<tr><td class=u>45<br>40</td> <td class=c>col01372 dbemail.bookoo </td> <td class=p></td></tr>
<tr><td class=u>39<br>34</td> <td class=>col01943 * </td> <td class=p></td></tr>
<tr><td class=u>33-31</td> <td>[shelf] </td> <td class=p></td></tr>
<tr><td class=u>30<br>25</td> <td class=c>2U </td> <td class=p></td></tr>
<tr><td class=u>24<br>19</td> <td style="background:#cecece">backup1 </td> <td class=p></td></tr>
<tr><td class=u>18<br>13</td> <td style="background:#8df595">jail17 * </td> <td class=p></td></tr>

</table>
</td>

</tr>
</table>
<br>
<table border=0 cellpadding=3>
<tr>

<td valign="top">
<table border=1>

<tr><th>U</th> <th style="background:#FFFF99">6-16 </th> <th>port</th></tr>
<tr><td class=u>129-127</td> <td style="background:#a5e9f3">[https://secure.johncompanies.com/mgmt/mrtg/switch.cgi?s=switch-p13&path= c2924 P13] </td> <td class=p></td></tr>
<tr><td class=u>126<br>121</td> <td class=c>2U </td> <td class=p></td></tr>
<tr><td class=u>120-118</td> <td>1372 gig sw </td> <td class=p></td></tr>
<tr><td class=u>117<br>112</td> <td class=c>col01372 cache5 bookoo Dell 1U </td> <td class=p></td></tr>
<tr><td class=u>111<br>106</td> <td>col01372 solr5.bookoo </td> <td class=p></td></tr>
<tr><td class=u>105<br>100</td> <td>col01372 monitor.bookoo </td> <td class=p></td></tr>
<tr><td class=u>99<br>94</td> <td>col01372 server3.kingwoodyardsales</td> <td class=p></td></tr>
<tr><td class=u>93<br>88</td> <td>col01372 web3.bookoo </td> <td class=p></td></tr>
<tr><td class=u>87<br>82</td> <td>col01372 web1.bookoounderground.com</td> <td class=p></td></tr>
<tr><td class=u>81<br>76</td> <td class=c>col01372 cache6 bookoo Dell 1U </td> <td class=p></td></tr>
<tr><td class=u>75-73</td> <td style="text-align:center; background:#cecece">PS </td> <td class=p></td></tr>
<tr><td class=u>72-70</td> <td style="text-align:center; background:#cecece">PS </td> <td class=p></td></tr>
<tr><td class=u>69<br>64</td> <td>col01372 dbstats.bookoo </td> <td class=p></td></tr>
<tr><td class=u>63<br>58</td> <td>col01372 db1.bookoounderground *</td> <td class=p></td></tr>
<tr><td class=u>57<br>52</td> <td>col01372 backup.clker </td> <td class=p></td></tr>
<tr><td class=u>51<br>46</td> <td>col01372 dbbackup.bookoo </td> <td class=p></td></tr>
<tr><td class=u>45<br>40</td> <td>col01372 dbpv.bookoo </td> <td class=p></td></tr>
<tr><td class=u>39<br>34</td> <td>col01372 web7.bookoo </td> <td class=p></td></tr>
<tr><td class=u>33<br>28</td> <td>col01372 dbm2.bookoo * </td> <td class=p></td></tr>
<tr><td class=u>27<br>22</td> <td>col01372 dbsense.bookoo </td> <td class=p></td></tr>

</table>
</td>

<td valign="top">
<table border=1>

<tr><th>U</th> <th style="background:#FFFF99">6-17 </th> <th>port</th></tr>
<tr><td class=u>129-127</td> <td>1U</td></tr>
<tr><td class=u>126-124</td> <td>col01372 gig sw </td> <td class=p></td></tr>
<tr><td class=u>123<br>118</td> <td class=c>2U </td> <td class=p></td></tr>
<tr><td class=u>117<br>112</td> <td>col01372 db1.clker </td> <td class=p>14</td></tr>
<tr><td class=u>111<br>106</td> <td>col01372 solr6.bookoo </td> <td class=p></td></tr>
<tr><td class=u>105<br>100</td> <td>col01372 solr4.bookoo </td> <td class=p>18</td></tr>
<tr><td class=u>99<br>94</td> <td>col01372 web5.bookoo </td> <td class=p>19</td></tr>
<tr><td class=u>93<br>88</td> <td>col01372 sso.bookoo </td> <td class=p>4</td></tr>
<tr><td class=u>87-85</td> <td>1U </td> <td class=p>9</td></tr>
<tr><td class=u>84-82</td> <td>col01372 balancer </td> <td class=p>3,4</td></tr>
<tr><td class=u>81-79</td> <td>1U </td> <td class=p></td></tr>
<tr><td class=u>78-76</td> <td style="text-align:center; background:#cecece">PS </td> <td class=p></td></tr>
<tr><td class=u>75-73</td> <td style="text-align:center; background:#cecece">PS </td> <td class=p></td></tr>
<tr><td class=u>72-70</td> <td style="text-align:center; background:#cecece">PS </td> <td class=p></td></tr>
<tr><td class=u>69<br>64</td> <td>col01372 solr2.bookoo </td> <td class=p></td></tr>
<tr><td class=u>63<br>58</td> <td>col01372 web6.bookoo </td> <td class=p></td></tr>
<tr><td class=u>57<br>52</td> <td>col01372 web8.bookoo </td> <td class=p></td></tr>
<tr><td class=u>51<br>46</td> <td>col01372 web2.bookoo </td> <td class=p></td></tr>
<tr><td class=u>45<br>40</td> <td>col01372 web1.bookoo </td> <td class=p></td></tr>
<tr><td class=u>39<br>34</td> <td>col01372 web4.bookoo </td> <td class=p></td></tr>
<tr><td class=u>33<br>28</td> <td>2U </td> <td class=p></td></tr>
<tr><td class=u>27<br>22</td> <td>col01372 dbm1.bookoo * </td> <td class=p></td></tr>
<tr><td class=u>21-19</td> <td>1U </td> <td class=p></td></tr>
<tr><td class=u>18<br>13</td> <td>col01372 web1.cliker * </td> <td class=p></td></tr>

</table>
</td>

</tr>
</table>

Cabinetmap

2014-01-03T07:14:13Z

24.165.90.109:

<table cellpadding=5>
<tr>
<td valign="top"><table border=1>

<tr><th>u</th> <th style="background:#FFFF99">1 </th><th>sw port</th> <th>ats port</th></tr>
<tr><td>42</td><td style="background:#a5e9f3">[https://secure.johncompanies.com/mgmt/mrtg/switch.cgi?s=switch-p25&path= c2950 P25]
</td><td class=p>1</td> <td class=p>9</td></tr>
<tr><td>41</td><td>&nbsp </td><td class=p> </td> <td class=p></td></tr>
<tr><td>40</td><td>  </td><td class=p> </td> <td class=p></td></tr>
<tr><td>39</td><td>  </td><td class=p> </td> <td class=p></td></tr>
<tr><td>38</td><td>  </td><td class=p> </td> <td class=p></td></tr>
<tr><td>37</td><td>  </td><td class=p> </td> <td class=p></td></tr>
<tr><td>36</td><td>col02113 </td><td class=p>19 </td> <td class=p>1</td></tr>
<tr><td>35</td><td rowspan=2>col01402 RJclients </td><td class=p rowspan=2>2</td> <td class=p rowspan=2></td></tr>
<tr><td>34</td></tr>
<tr><td>33</td><td rowspan=2>col1500 </td><td class=p rowspan=2>3</td> <td class=p rowspan=2>15</td></tr>
<tr><td>32</td></tr>
<tr><td>31</td><td rowspan=2>col00138 data4 </td><td class=p rowspan=2>4</td> <td class=p rowspan=2>14</td> </tr>
<tr><td>30</td></tr>
<tr><td>29</td><td rowspan=2>col01899 </td><td class=p rowspan=2>5,21 (RMM)</td> <td class=p rowspan=2>13</td></tr>
<tr><td>28</td></tr>
<tr><td>27</td><td rowspan=2>col01741 </td><td class=p rowspan=2>6</td> <td class=p rowspan=2>12</td></tr>
<tr><td>26</td></tr>
<tr><td>25</td><td rowspan=2>col01369 logos JC-08017 </td><td class=p rowspan=2>7</td> <td class=p rowspan=2>11</td></tr>
<tr><td>24</td></tr>
<tr><td>23</td><td>col01114 </td><td class=p>8</td> <td class=p >10</td></tr>
<tr><td>22</td><td> </td><td class=p>9</td> <td class=p >2-1</td></tr>
<tr><td>21</td><td colspan=3>  </td></tr>
<tr><td>20</td><td style="text-align:center; background:#cecece">[https://ats1.johncompanies.com ATS-1]
</td><td class=p>24</td> <td class=p></td></tr>
<tr><td>19</td><td colspan=3>  </td></tr>
<tr><td>18</td><td rowspan=2> </td><td class=p rowspan=2>10</td> <td class=p rowspan=2>7</td></tr>
<tr><td>17</td></tr>
<tr><td>16</td><td rowspan=2>col02210 JC-13009 </td><td class=p rowspan=2>10,11(RMM)</td> <td class=p rowspan=2>6</td></tr>
<tr><td>15</td></tr>
<tr><td>14</td><td rowspan=2>col02078 raven.nostrum.com </td><td class=p rowspan=2>17</td> <td class=p rowspan=2>5</td></tr>
<tr><td>13</td></tr>
<tr><td>12</td><td rowspan=2>col02184 JC-08028 </td><td class=p rowspan=2>12 </td> <td class=p rowspan=2>4</td></tr>
<tr><td>11</td></tr>
<tr><td>10</td><td rowspan=2> </td><td class=p rowspan=2></td> <td class=p rowspan=2>3</td></tr>
<tr><td>9 </td></tr>
<tr><td>8 </td><td rowspan=2>col00726 </td><td class=p rowspan=2>15</td> <td class=p rowspan=2>2</td></tr>
<tr><td>7 </td></tr>
<tr><td>6 </td><td rowspan=2>col01257 JC-13004 </td><td class=p rowspan=2>14 16 (RMM)</td> <td class=p rowspan=2>8</td></tr>
<tr><td>5 </td><td>  </td><td class=p></td> <td class=p></td></tr>

</table></td>

<td valign="top"><table border=1>

<tr><th>u</th> <th style="background:#FFFF99">2 </th><th>sw port</th> <th>ats port</th></tr>
<tr><td>42</td><td style="background:#a5e9f3">[https://secure.johncompanies.com/mgmt/mrtg/switch.cgi?s=switch-p24&path= c2950 P24] </td><td class=p>1</td> <td class=p>9</td></tr>
<tr><td>41</td><td>&nbsp </td><td class=p> </td> <td class=p></td></tr>
<tr><td>40</td><td>  </td><td class=p> </td> <td class=p></td></tr>
<tr><td>39</td><td>  </td><td class=p> </td> <td class=p></td></tr>
<tr><td>38</td><td>  </td><td class=p> </td> <td class=p></td></tr>
<tr><td>37</td><td>  </td><td class=p> </td> <td class=p></td></tr>
<tr><td>36</td><td>  </td><td class=p> </td> <td class=p></td></tr>
<tr><td>35</td><td rowspan=2>col01378 </td><td class=p rowspan=2>2,15</td> <td class=p rowspan=2>15</td></tr>
<tr><td>34</td></tr>
<tr><td>33</td><td rowspan=2>col01629 newserver </td><td class=p rowspan=2>3</td> <td class=p rowspan=2>14</td></tr>
<tr><td>32</td></tr>
<tr><td>31</td><td rowspan=2>col00138 www </td><td class=p rowspan=2>4</td> <td class=p rowspan=2>13</td></tr>
<tr><td>30</td></tr>
<tr><td>29</td><td rowspan=2>col01631 </td><td class=p rowspan=2>5</td> <td class=p rowspan=2>12</td></tr>
<tr><td>28</td></tr>
<tr><td>27</td><td rowspan=2>col01621 agony JC-08013 </td><td class=p rowspan=2>6</td> <td class=p rowspan=2>11</td></tr>
<tr><td>26</td></tr>
<tr><td>25</td><td rowspan=2>col01567 KPSurplus </td><td class=p rowspan=2>7</td> <td class=p rowspan=2>10</td></tr>
<tr><td>24</td></tr>
<tr><td>23</td><td rowspan=2>col02216 JC-06044 webengineer </td><td class=p rowspan=2>18</td> <td class=p rowspan=2>8</td></tr>
<tr><td>22</td></tr>
<tr><td>21</td><td colspan=3>  </td></tr>
<tr><td>20</td><td style="text-align:center; background:#cecece">[https://ats2.johncompanies.com ATS-2]
</td><td class=p>24</td> <td class=p></td></tr>
<tr><td>19</td><td colspan=3>  </td></tr>
<tr><td>18</td><td rowspan=2>col01996 JC-08003 </td><td class=p rowspan=2>9</td> <td class=p rowspan=2>7</td></tr>
<tr><td>17</td></tr>
<tr><td>16</td><td rowspan=2>col01711 JC-07030 * </td><td class=p rowspan=2>10</td> <td class=p rowspan=2>6</td></tr>
<tr><td>15</td></tr>
<tr><td>14</td><td rowspan=2>col01788 JC-06021 </td><td class=p rowspan=2>11</td> <td class=p rowspan=2>5</td></tr>
<tr><td>13</td></tr>
<tr><td>12</td><td rowspan=2>col00226 JC-07037 </td><td class=p rowspan=2>12</td> <td class=p rowspan=2>4</td></tr>
<tr><td>11</td></tr>
<tr><td>10</td><td rowspan=2>col00633 JC-08044 </td><td class=p rowspan=2>13</td> <td class=p rowspan=2>3</td></tr>
<tr><td>9</td></tr>
<tr><td>8</td><td rowspan=2>col00433 djc-jc JC-07015 </td><td class=p rowspan=2>14</td> <td class=p rowspan=2>2</td></tr>
<tr><td>7</td></tr>
<tr><td>4</td><td rowspan=2>col00433 Dell 2950 </td><td class=p rowspan=2>16 DRAC 17</td> <td class=p rowspan=2>2</td></tr>
<tr><td>5</td></tr>

</table></td>

<td valign="top"><table border=1>

<tr><th>u</th> <th style="background:#FFFF99">3 </th><th>sw port</th> <th>ats port</th></tr>
<tr><td>42</td><td style="background:#a5e9f3">[https://secure.johncompanies.com/mgmt/mrtg/switch.cgi?s=switch-p23&path= c2950 P23] </td><td class=p>1</td> <td class=p>9</td></tr>
<tr><td>41</td><td>&nbsp </td><td class=p> </td> <td class=p></td></tr>
<tr><td>40</td><td>  </td><td class=p> </td> <td class=p></td></tr>
<tr><td>39</td><td>  </td><td class=p> </td> <td class=p></td></tr>
<tr><td>38</td><td>  </td><td class=p> </td> <td class=p></td></tr>
<tr><td>37</td><td rowspan=2>col01727 JC-10019 </td><td class=p rowspan=2>19, 23(RMM)</td> <td class=p rowspan=2>1</td></tr>
<tr><td>36</td><td>
<tr><td>35</td><td rowspan=2>col01408 issoasis.com </td><td class=p rowspan=2>2 16 (RMM)</td> <td class=p rowspan=2>15</td></tr>
<tr><td>34</td></tr>
<tr><td>33</td><td rowspan=2>col00789 on001 </td><td class=p rowspan=2>3</td> <td class=p rowspan=2>14</td></tr>
<tr><td>32</td></tr>
<tr><td>31</td><td rowspan=2>col00789 laertes </td><td class=p rowspan=2>4</td> <td class=p rowspan=2>13</td></tr>
<tr><td>30</td></tr>
<tr><td>29</td><td rowspan=2>col02152 NO LABEL </td><td class=p rowspan=2>5</td> <td class=p rowspan=2>12</td></tr>
<tr><td>28</td></tr>
<tr><td>27</td><td rowspan=2>col02219 JC-0811 </td><td class=p rowspan=2>6</td> <td class=p rowspan=2>11</td></tr>
<tr><td>26</td></tr>
<tr><td>25</td><td rowspan=2>col01333 AJA Video </td><td class=p rowspan=2>P24-22</td> <td class=p rowspan=2>10</td></tr>
<tr><td>24</td></tr>
<tr><td>23</td><td rowspan=2>col01741 cyrus.base </td><td class=p rowspan=2>15, 22 (RMM)</td> <td class=p rowspan=2>8</td></tr>
<tr><td>22</td></tr>
<tr><td>21</td><td colspan=3>  </td></tr>
<tr><td>20</td><td style="text-align:center; background:#cecece">[https://ats3.johncompanies.com ATS-3]
</td><td class=p>24</td> <td class=p></td></tr>
<tr><td>19</td><td colspan=3>  </td></tr>
<tr><td>18</td><td rowspan=2>col00789 bb001 </td><td class=p rowspan=2>9, 20 (RMM)</td> <td class=p rowspan=2>7</td></tr>
<tr><td>17</td></tr>
<tr><td>16</td><td rowspan=2>col01224 pariotrial.sea-incorporated </td><td class=p rowspan=2>10, 21 (RMM)</td> <td class=p rowspan=2>6</td></tr>
<tr><td>15</td></tr>
<tr><td>14</td><td rowspan=2>col01777 </td><td class=p rowspan=2>11</td> <td class=p rowspan=2>5</td></tr>
<tr><td>13</td></tr>
<tr><td>12</td><td rowspan=2>col01224 fuji </td><td class=p rowspan=2>12</td> <td class=p rowspan=2>4</td></tr>
<tr><td>11</td></tr>
<tr><td>10</td><td rowspan=2>col01817 </td><td class=p rowspan=2>13</td> <td class=p rowspan=2>3</td></tr>
<tr><td>9</td></tr>
<tr><td>8</td><td rowspan=2>col01332 vs4 </td><td class=p rowspan=2>14</td> <td class=p rowspan=2>2</td></tr>
<tr><td>7</td></tr>
<tr><td>6</td><td rowspan=2>col02202 JC-13006 </td><td class=p rowspan=2>8 17 (RMM)</td> <td class=p rowspan=2>16</td></tr>
<tr><td>5</td><td>  </td><td class=p></td> <td class=p></td></tr>

</table></td>

<td valign="top"><table border=1>

<tr><th>u</th> <th style="background:#FFFF99">4 </th><th>sw port</th> <th>ats port</th></tr>
<tr><td>42</td><td style="background:#a5e9f3">[https://secure.johncompanies.com/mgmt/mrtg/switch.cgi?s=switch-p22&path= c2950 P22] </td><td class=p>1</td> <td class=p>9</td></tr>
<tr><td>41</td><td>&nbsp </td><td class=p> </td> <td class=p></td></tr>
<tr><td>40</td><td>  </td><td class=p> </td> <td class=p></td></tr>
<tr><td>39</td><td>  </td><td class=p> </td> <td class=p></td></tr>
<tr><td>38</td><td>  </td><td class=p> </td> <td class=p></td></tr>
<tr><td>37</td><td>  </td><td class=p> </td> <td class=p></td></tr>
<tr><td>36</td><td>  </td><td class=p> </td> <td class=p></td></tr>
<tr><td>35</td><td rowspan=2>col01224 cisco (no rails) </td><td class=p rowspan=2>2</td> <td class=p rowspan=2>16</td></tr>
<tr><td>34</td></tr>
<tr><td>33</td><td rowspan=2>col01224 obc-c </td><td class=p rowspan=2> </td> <td class=p rowspan=2>15</td></tr>
<tr><td>32</td></tr>
<tr><td>31</td><td rowspan=2>col01224 obc-b </td><td class=p rowspan=2> </td> <td class=p rowspan=2>14</td></tr>
<tr><td>30</td></tr>
<tr><td>29</td><td rowspan=2>col01224 obc-a </td><td class=p rowspan=2> </td> <td class=p rowspan=2>13</td></tr>
<tr><td>28</td></tr>
<tr><td>27</td><td rowspan=2>col02220 </td><td class=p rowspan=2>13</td> <td class=p rowspan=2>12</td></tr>
<tr><td>26</td></tr>
<tr><td>25</td><td>col00534 </td><td class=p>4</td> <td class=p>11</td></tr>
<tr><td>24</td><td> </td><td class=p></td> <td class=p></td></tr>
<tr><td>23</td><td rowspan=2>col01224 hydro4gehosting </td><td class=p rowspan=2>6</td> <td class=p rowspan=2>8</td></tr>
<tr><td>22</td></tr>
<tr><td>21</td><td colspan=3>  </td></tr>
<tr><td>20</td><td style="text-align:center; background:#cecece">[https://ats4.johncompanies.com ATS-4]
</td><td class=p>24</td> <td class=p></td></tr>
<tr><td>19</td><td colspan=3>  </td></tr>
<tr><td>18</td><td rowspan=2>col01182 dev01 </td><td class=p rowspan=2>7</td> <td class=p rowspan=2>7</td></tr>
<tr><td>17</td></tr>
<tr><td>16</td><td rowspan=2>col01182 demo2 </td><td class=p rowspan=2>8</td> <td class=p rowspan=2>6</td></tr>
<tr><td>15</td></tr>
<tr><td>14</td><td rowspan=2>col00336 rb3 </td><td class=p rowspan=2>9</td> <td class=p rowspan=2>5</td></tr>
<tr><td>13</td></tr>
<tr><td>12</td><td rowspan=2> </td><td class=p rowspan=2>10</td> <td class=p rowspan=2>4</td></tr>
<tr><td>11</td></tr>
<tr><td>10</td><td rowspan=2>col01710 </td><td class=p rowspan=2>11</td> <td class=p rowspan=2>3</td></tr>
<tr><td>9</td></tr>
<tr><td>8</td><td rowspan=2>col00336 coach </td><td class=p rowspan=2>12</td> <td class=p rowspan=2>2</td></tr>
<tr><td>7</td></tr>
<tr><td>6</td><td rowspan=2>col00534 </td><td class=p rowspan=2>18 RMM 19</td> <td class=p rowspan=2>10</td></tr>
<tr><td>5</td><td></tr>

</table></td>

<td valign="top"><table border=1>

<tr><th>u</th> <th style="background:#FFFF99">5 </th><th>sw port</th> <th>ats port</th></tr>
<tr><td>42</td><td style="background:#a5e9f3">[https://secure.johncompanies.com/mgmt/mrtg/switch.cgi?s=switch-p21&path= c2950 P21]
</td><td class=p>g1</td> <td class=p>9</td></tr>
<tr><td>41</td><td>&nbsp </td><td class=p> </td> <td class=p></td></tr>
<tr><td>40</td><td>  </td><td class=p> </td> <td class=p></td></tr>
<tr><td>39</td><td>  </td><td class=p> </td> <td class=p></td></tr>
<tr><td>38</td><td>  </td><td class=p> </td> <td class=p></td></tr>
<tr><td>37</td><td rowspan=2>col01849 snacktime </td><td class=p rowspan=2>16 17 (RMM)</td> <td class=p rowspan=2>2</td></tr>
<tr><td>36</td><td>
<tr><td>35</td><td rowspan=2>col01080 lithium </td><td class=p rowspan=2>1</td> <td class=p rowspan=2>16</td></tr>
<tr><td>34</td></tr>
<tr><td>33</td><td rowspan=2>col01224 ewing </td><td class=p rowspan=2>2</td> <td class=p rowspan=2>15</td></tr>
<tr><td>32</td></tr>
<tr><td>31</td><td rowspan=2>col01224 luzerne </td><td class=p rowspan=2>11</td> <td class=p rowspan=2>14</td></tr>
<tr><td>30</td></tr>
<tr><td>29</td><td rowspan=2>col02215 JC-08019 </td><td class=p rowspan=2>20</td> <td class=p rowspan=2>13</td></tr>
<tr><td>28</td></tr>
<tr><td>27</td><td rowspan=2>col01556 </td><td class=p rowspan=2>5</td> <td class=p rowspan=2>12</td></tr>
<tr><td>26</td></tr>
<tr><td>25</td><td rowspan=2>col02138 webengineers </td><td class=p rowspan=2>7, 21 (RMM)</td> <td class=p rowspan=2>11</td></tr>
<tr><td>24</td></tr>
<tr><td>23</td><td rowspan=2>col01495 beany </td><td class=p rowspan=2>22</td> <td class=p rowspan=2>10</td></tr>
<tr><td>22</td></tr>
<tr><td>21</td><td colspan=3>  </td></tr>
<tr><td>20</td><td style="text-align:center; background:#cecece">[https://ats5.johncompanies.com ATS-5]
</td><td class=p>24</td> <td class=p></td></tr>
<tr><td>19</td><td colspan=3>  </td></tr>
<tr><td>18</td><td rowspan=2>col01841 paradigm </td><td class=p rowspan=2>12 13 (DRAC)</td> <td class=p rowspan=2>6</td></tr>
<tr><td>17</td></tr>
<tr><td>16</td><td rowspan=2>col01080 helium </td><td class=p rowspan=2>9</td> <td class=p rowspan=2>5</td></tr>
<tr><td>15</td></tr>
<tr><td>14</td><td rowspan=2>col01629 anitaborg mail </td><td class=p rowspan=2>10</td> <td class=p rowspan=2>7</td></tr>
<tr><td>13</td></tr>
<tr><td>12</td><td rowspan=2>col02205 </td><td class=p rowspan=2>5 3 (RMM)</td> <td class=p rowspan=2>3</td></tr>
<tr><td>11</td></tr>
<tr><td>10</td><td rowspan=2>col01841 smartypants * </td><td class=p rowspan=2>12,13 (DRAC)</td> <td class=p rowspan=2>8</td></tr>
<tr><td>9</td></tr>
<tr><td>8</td><td rowspan=2>col01408 pepclub </td><td class=p rowspan=2>6, 15 (RMM)</td> <td class=p rowspan=2>4</td></tr>
<tr><td>7</td></tr>

</table></td>
</tr>
</table>

<table cellpadding=5>
<tr>
<td valign="top"><table border=1>

<tr><th>u</th> <th style="background:#FFFF99">6 </th><th>ats port</th></tr>
<tr><td>42</td><td class="jc">digi3 </td><td class=p>10</td></tr>
<tr><td>41</td><td style="background:#a5e9f3">[https://secure.johncompanies.com/mgmt/mrtg/switch.cgi?s=switch-p20&path= c2950 P20]
</td><td class=p>9</td></tr>
<tr><td>40</td><td style="background:#a5e9f3">[https://secure.johncompanies.com/mgmt/mrtg/switch.cgi?s=switch-p12&path= c2950 P12 private]
</td><td class=p>2</td></tr>
<tr><td>39</td><td>  </td><td class=p> </td></tr>
<tr><td>38</td><td>  </td><td class=p> </td></tr>
<tr><td>37</td><td>isys </td><td class=p>11</td></tr>
<tr><td>36</td><td rowspan=2>firewall2 </td><td class=p rowspan=2>8 (A), 7 (B)</td></tr>
<tr><td>35</td></tr>
<tr><td>34</td><td rowspan=2>firewall3 </td><td class=p rowspan=2>? (A), ? (B)</td></tr>
<tr><td>33</td></tr>
<tr><td>32</td><td rowspan=2>nat2 </td><td class=p rowspan=2>6</td></tr>
<tr><td>31</td></tr>
<tr><td>30</td><td rowspan=2>bwdb2 </td><td class=p rowspan=2>5</td></tr>
<tr><td>29</td></tr>
<tr><td>28</td><td rowspan=2>backup3 </td><td class=p rowspan=2>4</td></tr>
<tr><td>27</td><td>  </td><td class=p></td></tr>
<tr><td>26</td><td>jail3 (col01737) </td><td class=p></td></tr>
<tr><td>25</td><td>  </td><td class=p></td></tr>
<tr><td>24</td><td>  </td><td class=p></td></tr>
<tr><td>23</td><td>  </td><td class=p></td></tr>
<tr><td>22</td><td>  </td><td class=p></td></tr>
<tr><td>21</td><td colspan=3>  </td></tr>
<tr><td>20</td><td style="text-align:center; background:#cecece">[https://ats6.johncompanies.com ATS-6]
</td><td class=p></td></tr>
<tr><td>19</td><td colspan=3>  </td></tr>
<tr><td>18</td><td rowspan=2> </td><td class=p>16</td></tr>
<tr><td>17</td></tr>
<tr><td>16</td><td rowspan=2>magrathea </td><td class=p rowspan=2>3</td></tr>
<tr><td>15</td></tr>
<tr><td>14</td><td> col02045 cisco </td><td class=p>p21 18 ats 13</td></tr>
<tr><td>13</td><td>  </td><td class=p></td></tr>
<tr><td>12</td><td rowspan=2> col02045 dell </td><td class=p>p21 15 drac ats 12</td></tr>
<tr><td>11</td><td>  </td><td class=p></td></tr>
<tr><td>10</td><td>  </td><td class=p></td></tr>
<tr><td>9</td><td>  </td><td class=p></td></tr>
<tr><td>8</td><td>  </td><td class=p></td></tr>
<tr><td>7</td><td>  </td><td class=p></td></tr>

</table></td>

<td valign="top"><table border=1>

<tr><th>u</th> <th style="background:#FFFF99">7 </th><th>sw port</th> <th>ats port</th></tr>
<tr><td>42</td><td style="background:#a5e9f3">[https://secure.johncompanies.com/mgmt/mrtg/switch.cgi?s=switch-p26&path= c2950 P26]
</td><td class=p>1</td> <td class=p>9</td></tr>
<tr><td>41</td><td>&nbsp </td><td class=p> </td> <td class=p></td></tr>
<tr><td>40</td><td>  </td><td class=p> </td> <td class=p></td></tr>
<tr><td>39</td><td>  </td><td class=p> </td> <td class=p></td></tr>
<tr><td>38</td><td>  </td><td class=p> </td> <td class=p></td></tr>
<tr><td>37</td><td>  </td><td class=p> </td> <td class=p></td></tr>
<tr><td>36</td><td>  </td><td class=p> </td> <td class=p></td></tr>
<tr><td>35</td><td rowspan=2> </td><td class=p rowspan=2>?</td> <td class=p rowspan=2>15</td></tr>
<tr><td>34</td></tr>
<tr><td>33</td><td rowspan=2> </td><td class=p rowspan=2>3</td> <td class=p rowspan=2>14</td></tr>
<tr><td>32</td></tr>
<tr><td>31</td><td rowspan=2>col01224 brandgage </td><td class=p rowspan=2>4</td> <td class=p rowspan=2>13</td></tr>
<tr><td>30</td></tr>
<tr><td>29</td><td rowspan=2> </td><td class=p rowspan=2>5</td> <td class=p rowspan=2>12</td></tr>
<tr><td>28</td></tr>
<tr><td>27</td><td rowspan=2> </td><td class=p rowspan=2>6</td> <td class=p rowspan=2>11</td></tr>
<tr><td>26</td></tr>
<tr><td>25</td><td rowspan=2> </td><td class=p rowspan=2>7</td> <td class=p rowspan=2>10</td></tr>
<tr><td>24</td></tr>
<tr><td>23</td><td rowspan=2> </td><td class=p rowspan=2>8</td> <td class=p rowspan=2>8</td></tr>
<tr><td>22</td></tr>
<tr><td>21</td><td colspan=3>  </td></tr>
<tr><td>20</td><td style="text-align:center; background:#cecece">[https://ats7.johncompanies.com ATS-7]
</td><td class=p>24</td> <td class=p></td></tr>
<tr><td>19</td><td colspan=3> </td></tr>
<tr><td>18</td><td rowspan=2>col00875 www * </td><td class=p rowspan=2>9</td> <td class=p rowspan=2>7</td></tr>
<tr><td>17</td></tr>
<tr><td>16</td><td rowspan=2>col00875 backup </td><td class=p rowspan=2></td> <td class=p rowspan=2>6</td></tr>
<tr><td>15</td></tr>
<tr><td>14</td><td rowspan=2> </td><td class=p rowspan=2>10</td> <td class=p rowspan=2>5</td></tr>
<tr><td>13</td></tr>
<tr><td>12</td><td rowspan=2>col02171 </td><td class=p rowspan=2>11</td> <td class=p rowspan=2>4</td></tr>
<tr><td>11</td></tr>
<tr><td>10</td><td rowspan=2>jc11013 label </td><td class=p rowspan=2>12</td> <td class=p rowspan=2>3</td></tr>
<tr><td>9</td></tr>
<tr><td>8</td><td rowspan=2>col02108 </td><td class=p rowspan=2>13</td> <td class=p rowspan=2>2</td></tr>
<tr><td>7</td></tr>
<tr><td>6</td><td rowspan=4> On Floor -- col00336 fumble.fantasybowl.com (Atom) and col02108 JC-11009 (Atom)</td><td class=p>12</td> <td class=p>1</td></tr>
<tr><td>5</td><td>  </td><td class=p></td> <td class=p></td></tr>
<tr><td>4</td><td>  </td><td class=p></td> <td class=p></td></tr>
<tr><td>3</td><td>  </td><td class=p></td> <td class=p></td></tr>

</table></td>

<td valign="top"><table border=1>

<tr><th>u</th> <th style="background:#FFFF99">8 </th><th>sw port</th> <th>ats port</th></tr>
<tr><td>42</td><td style="background:#a5e9f3">[https://secure.johncompanies.com/mgmt/mrtg/switch.cgi?s=switch-p27&path= c2950 P27]
</td><td class=p>1</td> <td class=p>9</td></tr>
<tr><td>41</td><td>&nbsp </td><td class=p> </td> <td class=p></td></tr>
<tr><td>40</td><td>  </td><td class=p> </td> <td class=p></td></tr>
<tr><td>39</td><td>  </td><td class=p> </td> <td class=p></td></tr>
<tr><td>38</td><td>  </td><td class=p> </td> <td class=p></td></tr>
<tr><td>37</td><td rowspan=2>col01719 (1U) </td><td class=p rowspan=2>16</td> <td class=p rowspan=2>16</td></tr>
<tr><td>36</td></tr>
<tr><td>35</td><td rowspan=2> </td><td class=p rowspan=2>2</td> <td class=p rowspan=2>15</td></tr>
<tr><td>34</td></tr>
<tr><td>33</td><td rowspan=2>col01224 idgroup </td><td class=p rowspan=2>3</td> <td class=p rowspan=2>14</td></tr>
<tr><td>32</td></tr>
<tr><td>31</td><td rowspan=2> </td><td class=p rowspan=2>4</td> <td class=p rowspan=2>13</td></tr>
<tr><td>30</td></tr>
<tr><td>29</td><td rowspan=2>col02150 </td><td class=p rowspan=2>5, 21 (RMM)</td> <td class=p rowspan=2>12</td></tr>
<tr><td>28</td></tr>
<tr><td>27</td><td rowspan=2>col01860 </td><td class=p rowspan=2>6</td> <td class=p rowspan=2>11</td></tr>
<tr><td>26</td></tr>
<tr><td>25</td><td rowspan=2>col01025 feros </td><td class=p rowspan=2>7, 18 (RMM)</td> <td class=p rowspan=2>10</td></tr>
<tr><td>24</td></tr>
<tr><td>23</td><td rowspan=2> </td><td class=p rowspan=2>8</td> <td class=p rowspan=2>8</td></tr>
<tr><td>22</td></tr>
<tr><td>21</td><td colspan=3>  </td></tr>
<tr><td>20</td><td style="text-align:center; background:#cecece">[https://ats8.johncompanies.com ATS-8]
</td><td class=p>24</td> <td class=p></td></tr>
<tr><td>19</td><td colspan=3> </td></tr>
<tr><td>18</td><td rowspan=2>col01941 </td><td class=p rowspan=2>9</td> <td class=p rowspan=2>7</td></tr>
<tr><td>17</td></tr>
<tr><td>16</td><td rowspan=2>col01511 * </td><td class=p rowspan=2>10</td> <td class=p rowspan=2>6</td></tr>
<tr><td>15</td></tr>
<tr><td>14</td><td rowspan=2>col00138 data2 </td><td class=p rowspan=2>15</td> <td class=p rowspan=2>5</td></tr>
<tr><td>13</td></tr>
<tr><td>12</td><td rowspan=2>col01567 prod </td><td class=p rowspan=2>12</td> <td class=p rowspan=2>4</td></tr>
<tr><td>11</td></tr>
<tr><td>10</td><td rowspan=3> </td><td class=p rowspan=3></td> <td class=p rowspan=3>3</td></tr>
<tr><td>9</td></tr>
<tr><td>8</td></tr>
<tr><td>7</td><td rowspan=2>col02001 attic1jc </td><td class=p rowspan=2>11</td> <td class=p rowspan=2>2</td></tr>
<tr><td>6</td></tr>

</table></td>

<td valign="top"><table border=1>

<tr><th>u</th> <th style="background:#FFFF99">9 </th><th>sw port</th> <th>ats port</th></tr>
<tr><td>42</td><td style="background:#a5e9f3">[https://secure.johncompanies.com/mgmt/mrtg/switch.cgi?s=switch-p28&path= c2950 P28]
</td><td class=p>1</td> <td class=p>9</td></tr>
<tr><td>41</td><td>&nbsp </td><td class=p> </td> <td class=p></td></tr>
<tr><td>40</td><td>  </td><td class=p> </td> <td class=p></td></tr>
<tr><td>39</td><td>  </td><td class=p> </td> <td class=p></td></tr>
<tr><td>38</td><td>  </td><td class=p> </td> <td class=p></td></tr>
<tr><td>37</td><td>  </td><td class=p> </td> <td class=p></td></tr>
<tr><td>36</td><td>  </td><td class=p> </td> <td class=p></td></tr>
<tr><td>35</td><td rowspan=2> </td><td class=p rowspan=2>2</td> <td class=p rowspan=2>15</td></tr>
<tr><td>34</td></tr>
<tr><td>33</td><td rowspan=2> </td><td class=p rowspan=2>3 <td> <td class=p rowspan=2>14</td></tr>
<tr><td>32</td></tr>
<tr><td>31</td><td rowspan=2> </td><td class=p rowspan=2>4</td> <td class=p rowspan=2>13</td></tr>
<tr><td>30</td></tr>
<tr><td>29</td><td rowspan=2> </td><td class=p rowspan=2>5</td> <td class=p rowspan=2>12</td></tr>
<tr><td>28</td></tr>
<tr><td>27</td><td rowspan=2> </td><td class=p rowspan=2>6</td> <td class=p rowspan=2>11</td></tr>
<tr><td>26</td></tr>
<tr><td>25</td><td rowspan=2> </td><td class=p rowspan=2>7</td> <td class=p rowspan=2>10</td></tr>
<tr><td>24</td></tr>
<tr><td>23</td><td rowspan=2> </td><td class=p rowspan=2>8</td> <td class=p rowspan=2>8</td></tr>
<tr><td>22</td></tr>
<tr><td>21</td><td colspan=3>  </td></tr>
<tr><td>20</td><td style="text-align:center; background:#cecece">[https://ats9.johncompanies.com ATS-9]
</td><td class=p>24</td> <td class=p></td></tr>
<tr><td>19</td><td colspan=3> </td></tr>
<tr><td>18</td><td rowspan=2> </td><td class=p rowspan=2>9</td> <td class=p rowspan=2>7</td></tr>
<tr><td>17</td></tr>
<tr><td>16</td><td rowspan=2> </td><td class=p rowspan=2>10</td> <td class=p rowspan=2>6</td></tr>
<tr><td>15</td></tr>
<tr><td>14</td><td rowspan=2> </td><td class=p rowspan=2>11</td> <td class=p rowspan=2>5</td></tr>
<tr><td>13</td></tr>
<tr><td>12</td><td rowspan=2> </td><td class=p rowspan=2>12</td> <td class=p rowspan=2>4</td></tr>
<tr><td>11</td></tr>
<tr><td>10</td><td rowspan=2> </td><td class=p rowspan=2>13</td> <td class=p rowspan=2>3</td></tr>
<tr><td>9</td></tr>
<tr><td>8</td><td rowspan=2> * </td><td class=p rowspan=2>14</td> <td class=p rowspan=2>2</td></tr>
<tr><td>7</td></tr>

</table></td>

</tr>
</table>
<br>
<table border=0 cellpadding=5>
<tr>
<td valign="top">
<table border=1>

<tr><th>U</th> <th style="background:#FFFF99">3-5 </th> <th>port</th></tr>
<tr><td class=u>129-127</td> <td style="background:#a5e9f3">[https://secure.johncompanies.com/mgmt/mrtg/switch.cgi?s=switch-p3&path= c2924 P3] </td> <td class=p></td></tr>
<tr><td class=u>126-124</td> <td class=digi>digi2 </td> <td class=p></td></tr>
<tr><td class=u>123<br>118</td> <td>2U </td> <td class=p></td></tr>
<tr><td class=u>117<br>112</td> <td style="background:#f5ac81">virt14 * Dell 2950 </td> <td class=p>13</td></tr>
<tr><td class=u>111<br>106</td> <td style="background:#8df595">jail19 * </td> <td class=p></td></tr>
<tr><td class=u>105<br>100</td> <td style="background:#f5ac81">virt15 * </td> <td class=p></td></tr>
<tr><td class=u>99<br>94</td> <td>col00779 wendlink 233.77 </td> <td class=p></td></tr>
<tr><td class=u>93<br>88</td> <td style="background:#8df595">jail2 * </td> <td class=p></td></tr>
<tr><td class=u>87<br>82</td> <td style="background:#f5ac81">virt13 * Dell 2950 </td> <td class=p></td></tr>
<tr><td class=u>81<br>76</td> <td style="background:#8df595">old jail2 * </td> <td class=p></td></tr>
<tr><td class=u>75<br>70</td> <td style="background:#8df595">mx2 * </td> <td class=p></td></tr>
<tr><td class=u>69<br>64</td> <td>2U </td> <td class=p></td></tr>
<tr><td class=u>63<br>58</td> <td>col01372 cache2 </td> <td class=p></td></tr>
<tr><td class=u>57<br>52</td> <td>2U </td> <td class=p></td></tr>
<tr><td class=u>51<br>46</td> <td>col00499 </td> <td class=p></td></tr>
<tr><td class=u>45<br>40</td> <td>col02204 NIC P3-12 RMM P3-16 </td> <td class=p></td></tr>
<tr><td class=u>39<br>34</td> <td class=c>2U </td> <td class=p></td></tr>
<tr><td class=u>33<br>28</td> <td>col01727 - REMOVED 9/22/13 </td> <td class=p></td></tr>
</table>
</td>

<td valign="top">
<table border=1>

<tr><th>U</th> <th style="background:#FFFF99">3-6 </th> <th>port</th></tr>
<tr><td class=u>129-127</td> <td style="background:#a5e9f3">[https://secure.johncompanies.com/mgmt/mrtg/switch.cgi?s=switch-p4&path= c2924 P4] </td> <td class=p></td></tr>
<tr><td class=u>126-124</td> <td>1U </td> <td class=p></td></tr>
<tr><td class=u>123<br>118</td> <td style="background:#f5ac81">virt11 * </td> <td class=p>4</td></tr>
<tr><td class=u>117<br>112</td> <td style="background:#f5ac81">virt17 * </td> <td class=p>7</td></tr>
<tr><td class=u>111-109</td> <td>1U </td> <td class=p></td></tr>
<tr><td class=u>108-106</td> <td>2U </td> <td class=p></td></tr>
<tr><td class=u>105<br>100</td> <td style="background:#8df595">jail8 * </td> <td class=p>10</td></tr>
<tr><td class=u>99<br>94</td> <td style="background:#f5ac81">virt19 * </td> <td class=p>12</td></tr>
<tr><td class=u>93<br>88</td> <td style="background:#8df595">jail9 * </td> <td class=p></td></tr>
<tr><td class=u>87<br>82</td> <td>col01372 cache4 </td> <td class=p></td></tr>
<tr><td class=u>81<br>76</td> <td>col01372 dbmsg.bookoo </td> <td class=p></td></tr>
<tr><td class=u>75<br>70</td> <td>2U </td> <td class=p></td></tr>
<tr><td class=u>69<br>64</td> <td>2U </td> <td class=p></td></tr>
<tr><td class=u>63<br>58</td> <td>2U </td> <td class=p></td></tr>
<tr><td class=u>57<br>52</td> <td>2U </td> <td class=p></td></tr>
<tr><td class=u>51<br>46</td> <td>2U </td> <td class=p></td></tr>
<tr><td class=u>45<br>40</td> <td>col00985 </td> <td class=p></td></tr>
<tr><td class=u>39<br>34</td> <td>col01372 cache4 </td> <td class=p></td></tr>
<tr><td class=u>33<br>28</td> <td> </td> <td class=p></td></tr>

</table>
</td>

<td valign="top">
<table border=1>

<tr><th>U</th> <th style="background:#FFFF99">3-7 </th> <th>port</th></tr>
<tr><td class=u>129-127</td> <td style="background:#a5e9f3">[https://secure.johncompanies.com/mgmt/mrtg/switch.cgi?s=switch-p17&path= c2924 P17] </td> <td class=p></td></tr>
<tr><td class=u>126-124</td> <td class=c>1U </td> <td class=p></td></tr>
<tr><td class=u>123-121</td> <td class=digi>digi1 (no rails)</td> <td class=p></td></tr>
<tr><td class=u>120-118</td> <td style="background:#a5e9f3">switch-m2 </td> <td class=p></td></tr>
<tr><td class=u>117-115</td> <td style="text-align:center; background:#cecece">ATS </td> <td class=p></td></tr>
<tr><td class=u>114-112</td> <td style="text-align:center; background:#cecece">ATS </td> <td class=p></td></tr>
<tr><td class=u>111<br>106</td> <td style="background:#8df595">jail11 * </td> <td class=p></td></tr>
<tr><td class=u>105<br>100</td> <td class=c>col01372 solr7.bookoo </td> <td class=p></td></tr>
<tr><td class=u>99<br>94</td> <td>col01372 solr8.bookoo </td> <td class=p></td></tr>
<tr><td class=u>93<br>88</td> <td style="background:#f5ac81">virt12 * </td> <td class=p></td></tr>
<tr><td class=u>87<br>82</td> <td style="background:#8df595">jail7 * </td> <td class=p></td></tr>
<tr><td class=u>81<div class=s3></div>73</td> <td style="background:#f5ac81">virt14 </td> <td class=p></td></tr>
<tr><td class=u>72-70</td> <td style="background:#cecece"> bwdb1 (no rails) </td> <td class=p></td></tr>
<tr><td class=u>69<br>64</td> <td style="background:#f5ac81">virt16 * </td> <td class=p></td></tr>
<tr><td class=u>63-61</td> <td>col00991 </td> <td class=p></td></tr>
<tr><td class=u>60<div class=s4></div>49</td> <td style="background:#cecece">backup2 *</td> <td class=p></td></tr>
<tr><td class=u>48<br>43</td> <td style="background:#8df595">jail18 * </td> <td class=p></td></tr>
<tr><td class=u>42<br>37</td> <td style="background:#8df595">mx1 * </td> <td class=p></td></tr>
<tr><td class=u>36-34</td> <td class=c>1U </td> <td class=p></td></tr>
<tr><td class=u>33-31</td> <td>col00994 </td> <td class=p></td></tr>
<tr><td class=u>30<br>25</td> <td style="background:#f5ac81">virt9 * </td> <td class=p></td></tr>
<tr><td class=u>24<br>19</td> <td>2U </td> <td class=p></td></tr>
<tr><td class=u>18-16</td> <td>col0460 </td> <td class=p></td></tr>
<tr><td class=u>15-13</td> <td class=c>1U </td> <td class=p></td></tr>
<tr><td class=u>12-10</td> <td style="background:#cecece">nat </td> <td class=p></td></tr>

</table>
</td>

<td valign="top">
<table border=1>

<tr><th>U</th> <th style="background:#FFFF99">3-8 </th> <th>port</th></tr>
<tr><td class=u>129-127</td> <td class=c>1U </td> <td class=p></td></tr>
<tr><td class=u>126-124</td> <td style="background:#a5e9f3">c3750 </td> <td class=p></td></tr>
<tr><td class=u>122<br>118</td> <td style="background:#a5e9f3">c3550 </td> <td class=p></td></tr>
<tr><td class=u>117-115</td> <td style="background:#a5e9f3">[https://secure.johncompanies.com/mgmt/mrtg/switch.cgi?s=switch-p1&path= c2924 switch-p1] </td> <td class=p></td></tr>
<tr><td class=u>114-112</td> <td style="background:#a5e9f3">[https://secure.johncompanies.com/mgmt/mrtg/switch.cgi?s=switch-p1a&path= c2960 switch-p1a] </td> <td class=p></td></tr>
<tr><td class=u>111-109</td> <td style="background:#a5e9f3">[https://secure.johncompanies.com/mgmt/mrtg/switch.cgi?s=switch-p1b&path= c2960 switch-p1b] </td> <td class=p></td></tr>
<tr><td class=u>108<br>103</td> <td style="background:#cecece">firewall * </td> <td class=p></td></tr>
<tr><td class=u>102-100</td> <td style="background:#a5e9f3">ASA5500 </td> <td class=p></td></tr>
<tr><td class=u>99-97</td> <td style="background:#a5e9f3">M1 </td> <td class=p></td></tr>
<tr><td class=u>96<br>91</td> <td class=c>col02179 staging.issisystems.com </td> <td class=p></td></tr>
<tr><td class=u>90-88</td> <td style="background:#cecece">gate (old f/w) </td> <td class=p></td></tr>
<tr><td class=u>87<br>82</td> <td style="background:#f5ac81">quar1 * (no rails)</td> <td class=p></td></tr>
<tr><td class=u>81-79</td> <td>[shelf] </td> <td class=p></td></tr>
<tr><td class=u>78-76</td> <td class=c>1U </td> <td class=p></td></tr>
<tr><td class=u>75-73</td> <td style="text-align:center; background:#cecece">ATS </td> <td class=p></td></tr>
<tr><td class=u>72-70</td> <td style="text-align:center; background:#cecece">ATS </td> <td class=p></td></tr>
<tr><td class=u>69<br>64</td> <td style="background:#8df595">jail4</td> <td class=p></td></tr>
<tr><td class=u>63<br>58</td> <td style="background:#8df595">jail1 * (no rails)</td> <td class=p></td></tr>
<tr><td class=u>57-55</td> <td class=c>1U </td> <td class=p></td></tr>
<tr><td class=u>54<br>52</td> <td style="background:#cecece">console (no rails) </td> <td class=p></td></tr>
<tr><td class=u>51<br>46</td> <td style="background:#cecece">mail * (no rails) </td> <td class=p></td></tr>
<tr><td class=u>45<br>40</td> <td class=c>col01372 dbemail.bookoo </td> <td class=p></td></tr>
<tr><td class=u>39<br>34</td> <td class=>col01943 * </td> <td class=p></td></tr>
<tr><td class=u>33-31</td> <td>[shelf] </td> <td class=p></td></tr>
<tr><td class=u>30<br>25</td> <td class=c>2U </td> <td class=p></td></tr>
<tr><td class=u>24<br>19</td> <td style="background:#cecece">backup1 </td> <td class=p></td></tr>
<tr><td class=u>18<br>13</td> <td style="background:#8df595">jail17 * </td> <td class=p></td></tr>

</table>
</td>

</tr>
</table>
<br>
<table border=0 cellpadding=3>
<tr>

<td valign="top">
<table border=1>

<tr><th>U</th> <th style="background:#FFFF99">6-16 </th> <th>port</th></tr>
<tr><td class=u>129-127</td> <td style="background:#a5e9f3">[https://secure.johncompanies.com/mgmt/mrtg/switch.cgi?s=switch-p13&path= c2924 P13] </td> <td class=p></td></tr>
<tr><td class=u>126<br>121</td> <td class=c>2U </td> <td class=p></td></tr>
<tr><td class=u>120-118</td> <td>1372 gig sw </td> <td class=p></td></tr>
<tr><td class=u>117<br>112</td> <td class=c>col01372 cache5 bookoo Dell 1U </td> <td class=p></td></tr>
<tr><td class=u>111<br>106</td> <td>col01372 solr5.bookoo </td> <td class=p></td></tr>
<tr><td class=u>105<br>100</td> <td>col01372 monitor.bookoo </td> <td class=p></td></tr>
<tr><td class=u>99<br>94</td> <td>col01372 server3.kingwoodyardsales</td> <td class=p></td></tr>
<tr><td class=u>93<br>88</td> <td>col01372 web3.bookoo </td> <td class=p></td></tr>
<tr><td class=u>87<br>82</td> <td>col01372 web1.bookoounderground.com</td> <td class=p></td></tr>
<tr><td class=u>81<br>76</td> <td class=c>col01372 cache6 bookoo Dell 1U </td> <td class=p></td></tr>
<tr><td class=u>75-73</td> <td style="text-align:center; background:#cecece">PS </td> <td class=p></td></tr>
<tr><td class=u>72-70</td> <td style="text-align:center; background:#cecece">PS </td> <td class=p></td></tr>
<tr><td class=u>69<br>64</td> <td>col01372 dbstats.bookoo </td> <td class=p></td></tr>
<tr><td class=u>63<br>58</td> <td>col01372 db1.bookoounderground *</td> <td class=p></td></tr>
<tr><td class=u>57<br>52</td> <td>col01372 backup.clker </td> <td class=p></td></tr>
<tr><td class=u>51<br>46</td> <td>col01372 dbbackup.bookoo </td> <td class=p></td></tr>
<tr><td class=u>45<br>40</td> <td>col01372 dbpv.bookoo </td> <td class=p></td></tr>
<tr><td class=u>39<br>34</td> <td>col01372 web7.bookoo </td> <td class=p></td></tr>
<tr><td class=u>33<br>28</td> <td>col01372 dbm2.bookoo * </td> <td class=p></td></tr>
<tr><td class=u>27<br>22</td> <td>col01372 dbsense.bookoo </td> <td class=p></td></tr>

</table>
</td>

<td valign="top">
<table border=1>

<tr><th>U</th> <th style="background:#FFFF99">6-17 </th> <th>port</th></tr>
<tr><td class=u>129-127</td> <td>1U</td></tr>
<tr><td class=u>126-124</td> <td>col01372 gig sw </td> <td class=p></td></tr>
<tr><td class=u>123<br>118</td> <td class=c>2U </td> <td class=p></td></tr>
<tr><td class=u>117<br>112</td> <td>col01372 db1.clker </td> <td class=p>14</td></tr>
<tr><td class=u>111<br>106</td> <td>col01372 solr6.bookoo </td> <td class=p></td></tr>
<tr><td class=u>105<br>100</td> <td>col01372 solr4.bookoo </td> <td class=p>18</td></tr>
<tr><td class=u>99<br>94</td> <td>col01372 web5.bookoo </td> <td class=p>19</td></tr>
<tr><td class=u>93<br>88</td> <td>col01372 sso.bookoo </td> <td class=p>4</td></tr>
<tr><td class=u>87-85</td> <td>1U </td> <td class=p>9</td></tr>
<tr><td class=u>84-82</td> <td>col01372 balancer </td> <td class=p>3,4</td></tr>
<tr><td class=u>81-79</td> <td>1U </td> <td class=p></td></tr>
<tr><td class=u>78-76</td> <td style="text-align:center; background:#cecece">PS </td> <td class=p></td></tr>
<tr><td class=u>75-73</td> <td style="text-align:center; background:#cecece">PS </td> <td class=p></td></tr>
<tr><td class=u>72-70</td> <td style="text-align:center; background:#cecece">PS </td> <td class=p></td></tr>
<tr><td class=u>69<br>64</td> <td>col01372 solr2.bookoo </td> <td class=p></td></tr>
<tr><td class=u>63<br>58</td> <td>col01372 web6.bookoo </td> <td class=p></td></tr>
<tr><td class=u>57<br>52</td> <td>col01372 web8.bookoo </td> <td class=p></td></tr>
<tr><td class=u>51<br>46</td> <td>col01372 web2.bookoo </td> <td class=p></td></tr>
<tr><td class=u>45<br>40</td> <td>col01372 web1.bookoo </td> <td class=p></td></tr>
<tr><td class=u>39<br>34</td> <td>col01372 web4.bookoo </td> <td class=p></td></tr>
<tr><td class=u>33<br>28</td> <td>2U </td> <td class=p></td></tr>
<tr><td class=u>27<br>22</td> <td>col01372 dbm1.bookoo * </td> <td class=p></td></tr>
<tr><td class=u>21-19</td> <td>1U </td> <td class=p></td></tr>
<tr><td class=u>18<br>13</td> <td>col01372 web1.cliker * </td> <td class=p></td></tr>

</table>
</td>

</tr>
</table>

Routine Maintenance

2013-12-26T18:51:56Z

24.165.90.109: /* Update OS list */

= Daily Tasks =

== check load graphs ==

Click on the Load link in mgmt

This screen shows you load levels on our servers and network traffic for critical machines (firewalls, backup servers).

If you see load high or increasing

FreeBSD:
run [[VPS_Management#jtop|jtop]] (or [[VPS_Management#jt|jt]] > 7.x) and see if there are any runaway processes. Here are some examples of entries in top that are
definitely runaway processes:

<pre>79481 root 64 0 2256K 1056K CPU1 1 58:16 87.40% 87.40% nano
50650 1000 64 0 1852K 1112K RUN 0 207.9H 84.08% 84.08% screen
14829 www 2 0 39100K 31736K accept 0 104:24 46.54% 6.54% httpd
42065 root 61 0 1300K 844K RUN 1 47.8H 91.36% 91.36% ee
1328 www 56 0 18440K 10796K CPU1 0 64.4H 97.71% 97.71% httpd
26251 user 57 0 6124K 1160K CPU1 1 82.9H 98.44% 98.44% screen
89874 root 60 0 1352K 892K RUN 1 33.8H 65.82% 65.82% dialog
38656 1000 64 0 3088K 2136K CPU0 0 806:13 97.95% 97.95% StutBot
27630 root 64 0 1396K 972K RUN 1 76.8H 86.47% 86.47% ee</pre>

Linux:
run [[VPS_Management#vwe|vwe]] to see which VPS’s have high loads. From there run <tt>[[VPS_Management#vp|vp]] <veid></tt> and/or <tt>[[VPS_Management#vt|vt]] <veid></tt> to see what's going on in that system.
[[VPS_Management#vzstat|vzstat]] will also give you a nice picture of whats going on, systems with high numbers in the mlat column are likely culprits.

examples of out of control procs:

<pre>12183 nobody 16 0 4916 1348 1340 R 45.5 0.0 4249m httpd
29266 #502 16 0 1852 796 792 R 22.5 0.0 1104m vim
23860 #41 16 0 5472 5472 2076 R 98.9 0.2 31:41 python
19227 bin 19 0 1688 716 652 R 99.9 0.0 321:08 wtrs_ui
7762 apache 16 0 268 236 224 R 85.7 0.0 1010m ptrace
4624 #501 20 0 4304 2400 2044 R 53.6 0.1 284:32 YoSucker
20451 #506 20 0 1876 820 816 R 17.2 0.0 169:35 vim
8834 #514 20 0 900 724 672 R 77.6 0.0 382:30 neostats
31815 apache 14 0 3176 3176 1696 R 74.4 0.1 6:15 counter</pre>

Just kill -9 them and be done with it.

Also, anytime you see `kmod` or `ptrace` - kill those immediaely no matter how much they are using - they are attempts to exploit the linux ptrace bug. They won't work, but they suck a lot of CPU...

Also, any other processes that are at 90-100% cpu usage and have been running for any long period of time should be killed except for mysqld processes on FreeBSD. See above.

However, there is an exception:

if it is a mysqld, we don't want to kill their database. What you want to do is <tt>[[VPS_Management#jpid|jpid]] <pid></tt> to see who owns it, and then email them the paste containing the instructions for the nanny. Or you can simply do a <tt>kill -1 PID</tt> on the process to restart it.

=== Load averages jump at night ===

The load averages on the FreeBSD systems may jump up at night between 1 and 4 am - this is because the backups are running - if this is what is causing a jump in load, you will see processes like `rsync` in top eating a lot of CPU time.

== check backups ==

mgmt -> Motnroing -> Backups and make sure every machine was backed up the previous nite.
Also look at df on backup1 and backup2 to make sure no disk is approaching full, though bb should warn us in advance. Please note - errors encountered when a backup script on any of the particular systems run will gnerate an email to support@johncompanies.com so you can know immediately the day after if the directory to be backed up has been moved or no longer exists. A paste exists for this to notify the customer of a non-existant file/dir.

== check bb for warnings ==
mgmt -> BigBrother

Some events don't generate pages (on purpose). You will only see them by going to the bb main page.

= Monthly Tasks =
== rotate pine sent mail (1st of month) ==
On the 1st of the month, before any emails are sent out, quit out of pine, then log back in. Send mail from last month will be archived.
If you mess up and do it on the 3rd (for example), you can go into the previous month's saved email and save emails from the current month into the <tt>sent-mail</tt> (current month) mailbox.

== b/w caps ==
On the 1st: remove any bwcaps put into the firewall (only really applies if a bwcap was added cause someone went over on b/w):

ipfw list|grep pipe
ipfw del [each rule listed]

NOTE: this cronjob on newgateway will do some of that for you, provided you used one of the following pipe #s:
0 0 1 * * /sbin/ipfw del 3 4 5 17331

-------------------

We really don’t do this anymore since we have centralized traffic accounting with netflow, but for posterity:

Make sure all machines reset counters to 0 after midnight on the 1st
Make sure they dumped a counter

On each jail run:
trafficgather.pl

And on each virt:
linuxtrafficgather.pl

== Monthly RAID checks ==

Every month we check the health of and verfy the parity on all our RAID-based systems.
To facilitate this, we've created a simple script to start the process:

sh /root/verify.sh

=== Adaptec controllers ===
Here's some sample output:
<pre>
mail /usr/local/www/scripts# sh /root/verify.sh
---------------------------------------------------------------------------------------------

Adaptec SCSI RAID Controller Command Line Interface
Copyright 1998-2002 Adaptec, Inc. All rights reserved
---------------------------------------------------------------------------------------------

CLI > open aac0
Executing: open "aac0"

AAC0> container list /f
Executing: container list /full=TRUE
Num Total Oth Chunk Scsi Partition
Creation System
Label Type Size Ctr Size Usage B:ID:L Offset:Size State RO Lk Task Done% Ent
Date Time Files
----- ------ ------ --- ------ ------- ------ ------------- ------- -- -- ------- ------ ---
------ -------- ------
0 Mirror 33.9GB Open 0:01:0 64.0KB:33.9GB Normal 0
071002 05:39:32
/dev/aacd0 mirror0 0:00:0 64.0KB:33.9GB Normal 1
071002 05:39:32

1 Mirror 33.9GB Open 0:02:0 64.0KB:33.9GB Normal 0
071002 05:39:50
/dev/aacd1 mirror1 0:03:0 64.0KB:33.9GB Normal 1
071002 05:39:50

AAC0> disk list /f
Executing: disk list /full=TRUE

B:ID:L Device Type Removable media Vendor-ID Product-ID Rev Blocks Bytes/Bl
ock Usage Shared Rate
------ -------------- --------------- --------- ---------------- ----- --------- --------
--- ---------------- ------ ----
0:00:0 Disk N FUJITSU MAJ3364MC 3702 71390320 512
Initialized NO 160
0:01:0 Disk N FUJITSU MAJ3364MC 3702 71390320 512
Initialized NO 160
0:02:0 Disk N FUJITSU MAJ3364MC 3702 71390320 512
Initialized NO 160
0:03:0 Disk N FUJITSU MAJ3364MC 3702 71390320 512
Initialized NO 160

AAC0> disk show smart
Executing: disk show smart

Smart Method of Enable
Capable Informational Exception Performance Error
B:ID:L Device Exceptions(MRIE) Control Enabled Count
------ ------- ---------------- --------- ----------- ------
0:00:0 Y 6 Y N 0
0:01:0 Y 6 Y N 0
0:02:0 Y 6 Y N 0
0:03:0 Y 6 Y N 0
0:06:0 N

AAC0> task list
Executing: task list

Controller Tasks

TaskId Function Done% Container State Specific1 Specific2
------ -------- ------- --------- ----- --------- ---------

No tasks currently running on controller

AAC0> dia sh hi
Executing: diagnostic show history
No switches specified, defaulting to "/current".

*** HISTORY BUFFER FROM CURRENT CONTROLLER RUN ***

[00]: GetDiskLogEntry: container - 1, entry return 0
[01]: Container 1 started SCRUB task
[02]: Starting Mirror:1 scrub
[03]: Master disk: 2, start sector: 128, sector count = 71286784
[04]: Slave disk: 3, start sector: 128, sector count = 71286784
[05]: UpdateDiskLogIndex - Set - container 0, index 1
[06]: GetDiskLogEntry: container - 0, entry return 1
[07]: Container 0 started SCRUB task
[08]: Starting Mirror:0 scrub
[09]: Master disk: 1, start sector: 128, sector count = 71286784
[10]: Slave disk: 0, start sector: 128, sector count = 71286784
[11]: Mirror Scrub Container:1 ErrorsFound:0
[12]: Clear disk log: sector - 80, driveno 2
[13]: Clear disk log: sector - 80, driveno 3
[14]: Container 1 completed SCRUB task:
[15]: Mirror Scrub Container:0 ErrorsFound:0
[16]: Clear disk log: sector - 81, driveno 1
[17]: Clear disk log: sector - 81, driveno 0
[18]: Container 0 completed SCRUB task:
[19]: UpdateDiskLogIndex - Set - container 0, index 0
[20]: GetDiskLogEntry: container - 0, entry return 0
[21]: Container 0 started SCRUB task
[22]: Starting Mirror:0 scrub
[23]: Master disk: 1, start sector: 128, sector count = 71286784
[24]: Slave disk: 0, start sector: 128, sector count = 71286784
[25]: UpdateDiskLogIndex - Set - container 1, index 1
[26]: GetDiskLogEntry: container - 1, entry return 1
[27]: Container 1 started SCRUB task
[28]: Starting Mirror:1 scrub
[29]: Master disk: 2, start sector: 128, sector count = 71286784
[30]: Slave disk: 3, start sector: 128, sector count = 71286784
[31]: Mirror Scrub Container:1 ErrorsFound:0
[32]: Clear disk log: sector - 81, driveno 2
[33]: Clear disk log: sector - 81, driveno 3
[34]: Container 1 completed SCRUB task:
[35]: Mirror Scrub Container:0 ErrorsFound:0
[36]: Clear disk log: sector - 80, driveno 1
[37]: Clear disk log: sector - 80, driveno 0
[38]: Container 0 completed SCRUB task:
[39]: UpdateDiskLogIndex - Set - container 0, index 0
[40]: GetDiskLogEntry: container - 0, entry return 0
[41]: Container 0 started SCRUB task
[42]: Starting Mirror:0 scrub
[43]: Master disk: 1, start sector: 128, sector count = 71286784
[44]: Slave disk: 0, start sector: 128, sector count = 71286784
[45]: UpdateDiskLogIndex - Set - container 1, index 1
[46]: GetDiskLogEntry: container - 1, entry return 1
[47]: Container 1 started SCRUB task
[48]: Starting Mirror:1 scrub
[49]: Master disk: 2, start sector: 128, sector count = 71286784
[50]: Slave disk: 3, start sector: 128, sector count = 71286784
[51]: Mirror Scrub Container:1 ErrorsFound:0
[52]: Clear disk log: sector - 81, driveno 2
[53]: Clear disk log: sector - 81, driveno 3
[54]: Container 1 completed SCRUB task:
[55]: Mirror Scrub Container:0 ErrorsFound:0
[56]: Clear disk log: sector - 80, driveno 1
[57]: Clear disk log: sector - 80, driveno 0
[58]: Container 0 completed SCRUB task:
[59]: UpdateDiskLogIndex - Set - container 0, index 0
[60]: GetDiskLogEntry: container - 0, entry return 0
[61]: Container 0 started SCRUB task
[62]: Starting Mirror:0 scrub
[63]: Master disk: 1, start sector: 128, sector count = 71286784
[64]: Slave disk: 0, start sector: 128, sector count = 71286784
[65]: UpdateDiskLogIndex - Set - container 1, index 1
[66]: GetDiskLogEntry: container - 1, entry return 1
[67]: Container 1 started SCRUB task
[68]: Starting Mirror:1 scrub
[69]: Master disk: 2, start sector: 128, sector count = 71286784
[70]: Slave disk: 3, start sector: 128, sector count = 71286784
[71]: Mirror Scrub Container:1 ErrorsFound:0
[72]: Clear disk log: sector - 81, driveno 2
[73]: Clear disk log: sector - 81, driveno 3
[74]: Container 1 completed SCRUB task:
[75]: Mirror Scrub Container:0 ErrorsFound:0
[76]: Clear disk log: sector - 80, driveno 1
[77]: Clear disk log: sector - 80, driveno 0
[78]: Container 0 completed SCRUB task:
[79]: UpdateDiskLogIndex - Set - container 0, index 0
[80]: GetDiskLogEntry: container - 0, entry return 0
[81]: Container 0 started SCRUB task
[82]: Starting Mirror:0 scrub
[83]: Master disk: 1, start sector: 128, sector count = 71286784
[84]: Slave disk: 0, start sector: 128, sector count = 71286784
[85]: UpdateDiskLogIndex - Set - container 1, index 1
[86]: GetDiskLogEntry: container - 1, entry return 1
[87]: Container 1 started SCRUB task
[88]: Starting Mirror:1 scrub
[89]: Master disk: 2, start sector: 128, sector count = 71286784
[90]: Slave disk: 3, start sector: 128, sector count = 71286784
[91]: Mirror Scrub Container:1 ErrorsFound:0
[92]: Clear disk log: sector - 81, driveno 2
[93]: Clear disk log: sector - 81, driveno 3
[94]: Container 1 completed SCRUB task:
[95]: Mirror Scrub Container:0 ErrorsFound:0
[96]: Clear disk log: sector - 80, driveno 1
[97]: Clear disk log: sector - 80, driveno 0
[98]: Container 0 completed SCRUB task:
[99]:

========================
History Output Complete.

AAC0>
AAC0> exit
Executing: exit

press enter when ready to run verify <INS>
---------------------------------------------------------------------------------------------

Adaptec SCSI RAID Controller Command Line Interface
Copyright 1998-2002 Adaptec, Inc. All rights reserved
---------------------------------------------------------------------------------------------

CLI > open aac0
Executing: open "aac0"

AAC0> contai scr 0
Executing: container scrub 0

AAC0> contai scr 1
Executing: container scrub 1

AAC0> exit
Executing: exit

when done run:

aaccli
open aac0
dia sh hi
c

Nov 1 10:32:46 mail /kernel: aac0: **Monitor** Container 0 started SCRUB task
Nov 1 10:32:47 mail /kernel: aac0: **Monitor** Container 1 started SCRUB task

</pre>

Here's an analysis of what we're seeing and what we're looking for:

<pre>
AAC0> container list /f
Executing: container list /full=TRUE
Num Total Oth Chunk Scsi Partition
Creation System
Label Type Size Ctr Size Usage B:ID:L Offset:Size State RO Lk Task Done% Ent
Date Time Files
----- ------ ------ --- ------ ------- ------ ------------- ------- -- -- ------- ------ ---
------ -------- ------
0 Mirror 33.9GB Open 0:01:0 64.0KB:33.9GB Normal 0
071002 05:39:32
/dev/aacd0 mirror0 0:00:0 64.0KB:33.9GB Normal 1
071002 05:39:32

1 Mirror 33.9GB Open 0:02:0 64.0KB:33.9GB Normal 0
071002 05:39:50
/dev/aacd1 mirror1 0:03:0 64.0KB:33.9GB Normal 1
071002 05:39:50
</pre>

This is showing you the health of the arrays. You're looking for ''Normal'' under the State column, and the absence of a ! in the sector size - sometimes, you'll see this:
64.0KB!33.9GB

That indicates a problem.

<pre>
AAC0> disk show smart
Executing: disk show smart

Smart Method of Enable
Capable Informational Exception Performance Error
B:ID:L Device Exceptions(MRIE) Control Enabled Count
------ ------- ---------------- --------- ----------- ------
0:00:0 Y 6 Y N 0
0:01:0 Y 6 Y N 0
0:02:0 Y 6 Y N 0
0:03:0 Y 6 Y N 0
0:06:0 N
</pre>

This shows you a SMART report output. Looking for values in the Error Count column.
<pre>
AAC0> task list
Executing: task list

Controller Tasks

TaskId Function Done% Container State Specific1 Specific2
------ -------- ------- --------- ----- --------- ---------

No tasks currently running on controller
</pre>

Look for absence of tasks running- a bad thing would be to see a rebuild or verify running when you didn't initiate it.

With the history output, you're looking for any anomalies or events since the last time a verify was run. If you see a drive with lots of problems, you may want to take backups before allowing the verify to run since it could replicate errors onto the good drive.

After you see the history output, it will prompt you to press enter to run the verify. If you're happy with all the output you're seeing- mirror is healthy, history looks good, it's safe to proceed. Otherwise ^C to exit. After hitting enter it will start the verify and start to tail the messages log file (so you can easily see when the verify is complete). Here's what that'll look like:

<pre>
Nov 1 14:38:08 mail /kernel: aac0: **Monitor** Container 1 completed SCRUB task:
Nov 1 14:46:45 mail /kernel: aac0: **Monitor** Container 0 completed SCRUB task:
</pre>

So, putting it all together, after hitting enter to start the verify, you'll see:

<pre>
---------------------------------------------------------------------------------------------

Adaptec SCSI RAID Controller Command Line Interface
Copyright 1998-2002 Adaptec, Inc. All rights reserved
---------------------------------------------------------------------------------------------

CLI > open aac0
Executing: open "aac0"

AAC0> contai scr 0
Executing: container scrub 0

AAC0> contai scr 1
Executing: container scrub 1

AAC0> exit
Executing: exit

when done run:

aaccli
open aac0
dia sh hi
c

Nov 1 10:32:46 mail /kernel: aac0: **Monitor** Container 0 started SCRUB task
Nov 1 10:32:47 mail /kernel: aac0: **Monitor** Container 1 started SCRUB task

</pre>

When the scrub(s) (verify) are complete - if the server has multiple logical drives, it will run both in parallel - you should exit the tail of the log file (^C) and run:

<pre>
aaccli
open aac0
dia sh hi
c
</pre>

Which will show you the diagnostic history, you're looking for the results of the most recent scrub:

<pre>
[100]: Mirror Scrub Container:1 ErrorsFound:0
[101]: Clear disk log: sector - 81, driveno 2
[102]: Clear disk log: sector - 81, driveno 3
[103]: Container 1 completed SCRUB task:
[104]: Mirror Scrub Container:0 ErrorsFound:0
[105]: Clear disk log: sector - 80, driveno 1
[106]: Clear disk log: sector - 80, driveno 0
[107]: Container 0 completed SCRUB task:
</pre>

^C to exit the RAID CLI.

If you see:

[104]: Mirror Scrub Container:0 ErrorsFound:5

You'll want to rerun the verify on that drive till it shows 0, or perhaps replace the drive- you should be able to see from the output which drive had the problem.

Depending on the size and how busy the drive is, the verify can take anywhere from an hour to the better part of a day.

You will notice that the diagnostic history is not shown on our modern adaptec cards (i.e. any adaptec card not in a Dell 2450). The reason for this is the history is never cleared, so there's simply too much data to show and it just crashes the CLI. So, don't bother trying to see it...which does make it hard to see if there are problems going on, so you just need to watch the scrub to see it goes to 100%. You will also notice that on some servers there's no tail of messages. Again, this is cause no data is shown there about the completion of the scrub. The thing to do here is to go into the CLI and continue to <tt>show tasks</tt> to monitor scrub progress.

See [[RAIC_CLI#Adaptec|Adaptec RAID CLI Reference]] for more details on how to use the CLI

=== DELL (LSI-based) SAS controllers ===

Here's what the output looks like when running verify.sh on a LSI-based card:

<pre>
jail2 /mnt/data2# sh /root/verify.sh

Adapter #0

Enclosure Device ID: 32
Slot Number: 0
Device Id: 0
Sequence Number: 2
Media Error Count: 0
Other Error Count: 0
Predictive Failure Count: 0
Last Predictive Failure Event Seq Number: 0
PD Type: SAS
Raw Size: 140014MB [0x11177328 Sectors]
Non Coerced Size: 139502MB [0x11077328 Sectors]
Coerced Size: 139392MB [0x11040000 Sectors]
Firmware state: Online
SAS Address(0): 0x500000e018396142
SAS Address(1): 0x0
Connected Port Number: 0(path0)
Inquiry Data: FUJITSU MAX3147RC D207DQ03P7A0DESN
Foreign State: None
Media Type: Hard Disk Device
Device Speed: Unknown
Link Speed: Unknown

Enclosure Device ID: 32
Slot Number: 1
Device Id: 1
Sequence Number: 2
Media Error Count: 0
Other Error Count: 0
Predictive Failure Count: 0
Last Predictive Failure Event Seq Number: 0
PD Type: SAS
Raw Size: 140014MB [0x11177328 Sectors]
Non Coerced Size: 139502MB [0x11077328 Sectors]
Coerced Size: 139392MB [0x11040000 Sectors]
Firmware state: Online
SAS Address(0): 0x500000e018395db2
SAS Address(1): 0x0
Connected Port Number: 1(path0)
Inquiry Data: FUJITSU MAX3147RC D207DQ03P7A0DERV
Foreign State: None
Media Type: Hard Disk Device
Device Speed: Unknown
Link Speed: Unknown

Enclosure Device ID: 32
Slot Number: 2
Device Id: 2
Sequence Number: 2
Media Error Count: 0
Other Error Count: 0
Predictive Failure Count: 0
Last Predictive Failure Event Seq Number: 0
PD Type: SAS
Raw Size: 286102MB [0x22ecb25c Sectors]
Non Coerced Size: 285590MB [0x22dcb25c Sectors]
Coerced Size: 285568MB [0x22dc0000 Sectors]
Firmware state: Online
SAS Address(0): 0x5000c50006eece89
SAS Address(1): 0x0
Connected Port Number: 2(path0)
Inquiry Data: SEAGATE ST3300555SS T2113LM4BFBZ
Foreign State: None
Media Type: Hard Disk Device
Device Speed: Unknown
Link Speed: Unknown

Enclosure Device ID: 32
Slot Number: 3
Device Id: 3
Sequence Number: 2
Media Error Count: 0
Other Error Count: 0
Predictive Failure Count: 0
Last Predictive Failure Event Seq Number: 0
PD Type: SAS
Raw Size: 286102MB [0x22ecb25c Sectors]
Non Coerced Size: 285590MB [0x22dcb25c Sectors]
Coerced Size: 285568MB [0x22dc0000 Sectors]
Firmware state: Online
SAS Address(0): 0x5000c50006eee035
SAS Address(1): 0x0
Connected Port Number: 3(path0)
Inquiry Data: SEAGATE ST3300555SS T2113LM4BGF7
Foreign State: None
Media Type: Hard Disk Device
Device Speed: Unknown
Link Speed: Unknown

Enclosure Device ID: 32
Slot Number: 4
Device Id: 4
Sequence Number: 2
Media Error Count: 0
Other Error Count: 0
Predictive Failure Count: 0
Last Predictive Failure Event Seq Number: 0
PD Type: SAS
Raw Size: 286102MB [0x22ecb25c Sectors]
Non Coerced Size: 285590MB [0x22dcb25c Sectors]
Coerced Size: 285568MB [0x22dc0000 Sectors]
Firmware state: Online
SAS Address(0): 0x5000c50004bd7ea5
SAS Address(1): 0x0
Connected Port Number: 4(path0)
Inquiry Data: SEAGATE ST3300656SS HS093QP0G8SW
Foreign State: None
Media Type: Hard Disk Device
Device Speed: Unknown
Link Speed: Unknown

Enclosure Device ID: 32
Slot Number: 5
Device Id: 5
Sequence Number: 2
Media Error Count: 0
Other Error Count: 0
Predictive Failure Count: 0
Last Predictive Failure Event Seq Number: 0
PD Type: SAS
Raw Size: 286102MB [0x22ecb25c Sectors]
Non Coerced Size: 285590MB [0x22dcb25c Sectors]
Coerced Size: 285568MB [0x22dc0000 Sectors]
Firmware state: Online
SAS Address(0): 0x500000e01f1c4112
SAS Address(1): 0x0
Connected Port Number: 5(path0)
Inquiry Data: FUJITSU MBA3300RC D306BJ15P9201W06
Foreign State: None
Media Type: Hard Disk Device
Device Speed: Unknown
Link Speed: Unknown

Exit Code: 0x00

Adapter 0 -- Virtual Drive Information:
Virtual Disk: 0 (Target Id: 0)
Name:
RAID Level: Primary-1, Secondary-0, RAID Level Qualifier-0
Size:139392MB
State: Optimal
Stripe Size: 64kB
Number Of Drives:2
Span Depth:1
Default Cache Policy: WriteBack, ReadAdaptive, Direct, No Write Cache if Bad BBU
Current Cache Policy: WriteBack, ReadAdaptive, Direct, No Write Cache if Bad BBU
Access Policy: Read/Write
Disk Cache Policy: Disk's Default

Exit Code: 0x00

Adapter 0 -- Virtual Drive Information:
Virtual Disk: 1 (Target Id: 1)
Name:MIRROR1
RAID Level: Primary-1, Secondary-0, RAID Level Qualifier-0
Size:285568MB
State: Optimal
Stripe Size: 64kB
Number Of Drives:2
Span Depth:1
Default Cache Policy: WriteBack, ReadAdaptive, Direct, No Write Cache if Bad BBU
Current Cache Policy: WriteBack, ReadAdaptive, Direct, No Write Cache if Bad BBU
Access Policy: Read/Write
Disk Cache Policy: Disk's Default

Exit Code: 0x00

Adapter 0 -- Virtual Drive Information:
Virtual Disk: 2 (Target Id: 2)
Name:MIRROR2
RAID Level: Primary-1, Secondary-0, RAID Level Qualifier-0
Size:285568MB
State: Optimal
Stripe Size: 64kB
Number Of Drives:2
Span Depth:1
Default Cache Policy: WriteBack, ReadAdaptive, Direct, No Write Cache if Bad BBU
Current Cache Policy: WriteBack, ReadAdaptive, Direct, No Write Cache if Bad BBU
Access Policy: Read/Write
Disk Cache Policy: Disk's Default

Exit Code: 0x00
Battery FRU : N/A
Battery Warning : Enabled
Memory Correctable Errors : 0
Memory Uncorrectable Errors : 0
BBU : Present
BBU : Yes
Cache When BBU Bad : Disabled
press enter when ready to run verify

</pre>

Before pressing enter, here's what we're looking for:

<pre>
Enclosure Device ID: 32
Slot Number: 0
Device Id: 0
Sequence Number: 2
Media Error Count: 0
Other Error Count: 0
Predictive Failure Count: 0
Last Predictive Failure Event Seq Number: 0
PD Type: SAS
Raw Size: 140014MB [0x11177328 Sectors]
Non Coerced Size: 139502MB [0x11077328 Sectors]
Coerced Size: 139392MB [0x11040000 Sectors]
Firmware state: Online
SAS Address(0): 0x500000e018396142
SAS Address(1): 0x0
Connected Port Number: 0(path0)
Inquiry Data: FUJITSU MAX3147RC D207DQ03P7A0DESN
Foreign State: None
Media Type: Hard Disk Device
Device Speed: Unknown
Link Speed: Unknown
</pre>

This is the output shown for each physical drive in the system. We're looking to confirm it's Firmware state is ''Online'', and Media Error Count, Other Error Count, and Predictive Failure Count are all zero (or near zero).

<pre>
Adapter 0 -- Virtual Drive Information:
Virtual Disk: 1 (Target Id: 1)
Name:MIRROR1
RAID Level: Primary-1, Secondary-0, RAID Level Qualifier-0
Size:285568MB
State: Optimal
Stripe Size: 64kB
Number Of Drives:2
Span Depth:1
Default Cache Policy: WriteBack, ReadAdaptive, Direct, No Write Cache if Bad BBU
Current Cache Policy: WriteBack, ReadAdaptive, Direct, No Write Cache if Bad BBU
Access Policy: Read/Write
Disk Cache Policy: Disk's Default
</pre>

This is the output for each logical drive. We're looking for State ''Optimal''. Also confirm <tt>Default Cache Policy: WriteBack, ReadAdaptive, Direct, No Write Cache if Bad BBU</tt>

<pre>
Exit Code: 0x00
Battery FRU : N/A
Battery Warning : Enabled
Memory Correctable Errors : 0
Memory Uncorrectable Errors : 0
BBU : Present
BBU : Yes
Cache When BBU Bad : Disabled
</pre>

Confirm that the battery is present and error-free.

If all that checks out, you're ready to proceed with the verify. After pressing enter, the verify is started and here's what you see:
<pre>
Start Check Consistency on Virtual Drive 0 (target id: 0) Success.

Exit Code: 0x00

Start Check Consistency on Virtual Drive 1 (target id: 1) Success.

Exit Code: 0x00

Start Check Consistency on Virtual Drive 2 (target id: 2) Success.

Exit Code: 0x00

Check Consistency

Progress of Virtual Drives...

Virtual Drive # Percent Complete Time Elps
0 °°°°°°°°°°°°°°°°°°°°°°°00 %°°°°°°°°°°°°°°°°°°°°°°° 00:00:03
1 °°°°°°°°°°°°°°°°°°°°°°°00 %°°°°°°°°°°°°°°°°°°°°°°° 00:00:02
2 °°°°°°°°°°°°°°°°°°°°°°°00 %°°°°°°°°°°°°°°°°°°°°°°° 00:00:01

Press <ESC> key to quit...
</pre>

The progress for each drive is displayed until all drives have completed the verify. We just want to make sure that each drive goes to completion. No followup is needed...though there probably is a log or history where we can get more info.

You will notice that jail7 does not run a verify- that's on purpose. The last time we tried this it crashed the system. So, this must be run from the BIOS (take the system offline for a couple hours).

See [[RAIC_CLI#LSI|LSI RAID CLI Reference]] for more details on how to use the CLI

=== LSI-based controllers (megaraid) ===

There is a CLI for this however it's easier to do this with a curses GUI app: megaraid

Currently only on these servers: virt15, virt16, and firewall2

To run:

# cd /usr/local/sbin/; megamgr

Main menu:
<pre>²ÚÄÄManagement MenuÄÄ¿²
²³ Configure ³²
²³ Initialize ³²
²³ Objects ³²
²³ Rebuild ³²
²³ Check Consistency ³²
²³ Advanced Menu ³²
²ÀÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÙ²</pre>

Before you check consistency, make sure the arrays are healthy.

Objects -> Physical Drive

Then look to make sure they're all <tt>ONLIN</tt>
<pre>
²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²
²²²²²²²²ÚÄÄÄÄÄÄÄÄÄÄÄÄObjects - PHYSICAL DRIVE SELECTION MENUÄÄÄÄÄÄÄÄÄÄÄÄÄ¿²²²²²²
²²²²²²²²³ ³²²²²²²
²²²²²²²²³ Channel-1 ³²²²²²²
²ÚÄÄMana³ ID ÉÍÍÍÍÍÍÍÍÍÍÍÍÍÍ»x ³²²²²²²
²³ Confi³ 0º* ONLIN A01-01º ³²²²²²²
²³ Initi³ ÌÍÍÍÍÍÍÍÍÍÍÍÍÍÍ¹ ³²²²²²²
²³ Objec³ 1º* ONLIN A01-02º ³²²²²²²
²³ Rebui³ ÌÍÍÍÍÍÍÍÍÍÍÍÍÍÍ¹ ³²²²²²²
²³ Check³ 2º* ONLIN A02-01º ³²²²²²²
²³ Advan³ ÌÍÍÍÍÍÍÍÍÍÍÍÍÍÍ¹ ³²²²²²²
²ÀÄÄÄÄÄÄ³ 3º* ONLIN A02-02º ³²²²²²²
²²²²²²²²³ ÌÍÍÍÍÍÍÍÍÍÍÍÍÍÍ¹ ³²²²²²²
²²²²²²²²³ 4º* ONLIN A03-01º ³²²²²²²
²²²²²²²²³ ÌÍÍÍÍÍÍÍÍÍÍÍÍÍÍ¹ ³²²²²²²
²²²²²²²²³ 5º* ONLIN A03-02ºþ ³²²²²²²
²²²²²²²²³ ÌÍÍÍÍÍÍÍÍÍÍÍÍÍÍ¹ ³²²²²²²
²²²²²²²²³ 6º* º ³²²²²²²
²²²²²²²²³ ÈÍÍÍÍÍÍÍÍÍÍÍÍÍÍ¼x ³²²²²²²
²²²²²²²²ÀÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÙ²²²²²²
²²²²²²²²ÚÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄ¿²²²²²²²²
²²²²²²²²³Ch-1 ID-5 DISK 140013MB SEAGATE ST3146707LC 0003 ³²²²²²²²²
²²²²²²²²ÀÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÙ²²²²²²²²
</pre>

Once that's done, hit escape once then the back arror to move back to the Objects menu.
So you select Objects -> Logical Drive -> Logical Drive 1 -> Check Consistency -> YES

<pre>

²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²
²²²²²²²²²²²²²²²²²²²²²²²²²²²²ÚÄLogical Drives(02)Ä¿²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²
²²²²²²²²²²²²²²²²²²²²²²²²²²²²³ Logical Drive 1 ³²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²
²²²²²²²²²²²²²ÚÄÄÄÄObjectsÄÄÄ³ Logical Drive 2 ³²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²
²ÚÄÄManagemen³ Adapter ÀÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÙ²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²
²³ Configure ³ Logical Drive ³²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²
²³ Initialize³ Physical Drive ³²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²
²³ Objects ³ Channel ³²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²
²³ Rebuild ÀÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÙ²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²
²³ Check Consistency ³²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²
²³ Advanced Menu ³²²²²²²²ÚÄÄÄÄLogical Drive 1ÄÄÄÄÄ¿²²²²²²²²²²²²²²²²²²²²²²²²²
²ÀÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÙ²²²²²²²³ Initialize ÚÄCheck Consistency-1 ?Ä¿²²²²²²²²²
²²²²²²²²²²²²²²²²²²²²²²²²²²²²²³ Check Consiste³ YES ³²²²²²²²²²
²²²²²²²²²²²²²²²²²²²²²²²²²²²²²³ View/Update Pa³ NO ³²²²²²²²²²
²²²²²²²²²²²²²²²²²²²²²²²²²²²²²ÀÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÀÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÙ²²²²²²²²²
²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²
²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²
²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²
²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²
²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²
²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²ÚÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄ¿²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²
²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²³Select YES Or NO³²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²
²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²ÀÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÙ²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²
</pre>

Then watch the progress. When done, escape back to Logical Drive then repeat for Logical Drive 2. If you ^C or accidentally escape out, you can come back in running the same commands and watch the progress again (it won't restart).

You can exit megamgr by escaping out or ^C

=== 3ware ===

We are using 3ware controllers on backup1 & backup2. Running the verify script will give you different output based on the type of controller:

<pre>
backup2 /d2# sh /root/verify.sh
Controller: c0
-------------
Driver: 1.50.01.002
Model: 7500-8
FW: FE7X 1.05.00.068
BIOS: BE7X 1.08.00.048
Monitor: ME7X 1.01.00.040
Serial #: F11605A3180172
PCB: Rev3
PCHIP: 1.30-33
ACHIP: 3.20

# of units: 3
Unit 0: JBOD 186.31 GB ( 390721968 blocks): OK
Unit 1: RAID 5 465.77 GB ( 976790016 blocks): DEGRADED
Unit 5: RAID 5 698.65 GB ( 1465185024 blocks): DEGRADED

# of ports: 8
Port 0: WDC WD2000JB-00KFA0 WD-WCAMT1451690 186.31 GB (390721968 blocks): OK(unit 0)
Port 1: WDC WD2500JB-00GVC0 WD-WCAL78219488 232.88 GB (488397168 blocks): OK(unit 1)
Port 2: WDC WD2000 0.00 MB (0 blocks): OK(NO UNIT)
Port 3: WDC WD2500JB-00GVC0 WD-WMAL73882417 232.88 GB (488397168 blocks): OK(unit 1)
Port 4: WDC WD2000 0.00 MB (0 blocks): OK(NO UNIT)
Port 5: WDC WD2500JB-00GVA0 WD-WMAL71338097 232.88 GB (488397168 blocks): OK(unit 5)
Port 6: WDC WD2500JB-32EVA0 WD-WMAEH1301595 232.88 GB (488397168 blocks): OK(unit 5)
Port 7: WDC WD2500JB-00GVC0 WD-WCAL78165566 232.88 GB (488397168 blocks): OK(unit 5)
Controller: c1
-------------
Driver: 1.50.01.002
Model: 7500-8
FW: FE7X 1.05.00.068
BIOS: BE7X 1.08.00.048
Monitor: ME7X 1.01.00.040
Serial #: F11605A3180167
PCB: Rev3
PCHIP: 1.30-33
ACHIP: 3.20

# of units: 2
Unit 0: RAID 5 698.65 GB ( 1465185024 blocks): OK
Unit 4: RAID 5 698.65 GB ( 1465185024 blocks): OK

# of ports: 8
Port 0: WDC WD2500JB-00GVA0 WD-WMAL71301258 232.88 GB (488397168 blocks): OK(unit 0)
Port 1: WDC WD2500JB-00GVA0 WD-WMAL71322705 232.88 GB (488397168 blocks): OK(unit 0)
Port 2: WDC WD2500JB-00GVA0 WD-WMAL71945050 232.88 GB (488397168 blocks): OK(unit 0)
Port 3: WDC WD2500JB-00GVA0 WD-WMAL71316201 232.88 GB (488397168 blocks): OK(unit 0)
Port 4: WDC WD2500JB-00GVC0 WD-WCAL78323749 232.88 GB (488397168 blocks): OK(unit 4)
Port 5: WDC WD3200AAJB-00J3A0 WD-WCAV2V689068 298.09 GB (625142448 blocks): OK(unit 4)
Port 6: WDC WD2500JB-00GVC0 WD-WCAL78234420 232.88 GB (488397168 blocks): OK(unit 4)
Port 7: WDC WD2500JB-00GVC0 WD-WCAL78592213 232.88 GB (488397168 blocks): OK(unit 4)
backup2 /d2#
</pre>

On backup2 look for all ok, no verify.

<pre>
[root@backup3 ~]# sh /root/verify.sh
/c2 Driver Version = 1.26.02.002
/c2 Model = 8006-2LP
/c2 Available Memory = 512KB
/c2 Firmware Version = FE8S 1.05.00.068
/c2 Bios Version = BE7X 1.08.00.048
/c2 Boot Loader Version = ME7X 1.01.00.040
/c2 Serial Number = L018501C6481395
/c2 PCB Version = Rev5
/c2 PCHIP Version = 1.30-66
/c2 ACHIP Version = 3.20
/c2 Total Optimal Units = 1
/c2 Not Optimal Units = 0

Unit UnitType Status %RCmpl %V/I/M Stripe Size(GB) Cache AVrfy
------------------------------------------------------------------------------
u1 RAID-1 OK - - - 931.512 ON -

Port Status Unit Size Blocks Serial
---------------------------------------------------------------
p0 OK u1 931.51 GB 1953525168 WD-WMAW31148820
p1 OK u1 931.51 GB 1953525168 WD-WCATR0277515

Ctl Date Severity Alarm Message
------------------------------------------------------------------------------

Sending start verify message to /c2/u1 ... Done.

when done run:
tw_cli /c2 show alarms

[root@backup3 ~]#
</pre>

Automatically starts the verify, just run <tt>tw_cli /c2 show alarms</tt> as instructed to see the results of the verify.

<pre>
[root@backup1 /data/deprecated]# sh /root/verify.sh
/c0 Driver Version = 2.26.02.010
/c0 Model = 9650SE-8LPML
/c0 Available Memory = 224MB
/c0 Firmware Version = FE9X 4.06.00.004
/c0 Bios Version = BE9X 4.05.00.015
/c0 Boot Loader Version = BL9X 3.08.00.001
/c0 Serial Number = L326025A8270177
/c0 PCB Version = Rev 032
/c0 PCHIP Version = 2.00
/c0 ACHIP Version = 1.90
/c0 Number of Ports = 8
/c0 Number of Drives = 6
/c0 Number of Units = 1
/c0 Total Optimal Units = 1
/c0 Not Optimal Units = 0
/c0 JBOD Export Policy = off
/c0 Disk Spinup Policy = 1
/c0 Spinup Stagger Time Policy (sec) = 1
/c0 Auto-Carving Policy = off
/c0 Auto-Carving Size = 2048 GB
/c0 Auto-Rebuild Policy = on
/c0 Controller Bus Type = PCIe
/c0 Controller Bus Width = 1 lane
/c0 Controller Bus Speed = 2.5 Gbps/lane

Unit UnitType Status %RCmpl %V/I/M Stripe Size(GB) Cache AVrfy
------------------------------------------------------------------------------
u0 RAID-5 OK - - 64K 4656.56 ON ON

Port Status Unit Size Blocks Serial
---------------------------------------------------------------
p0 OK u0 931.51 GB 1953525168 9QJ1Y017
p1 DEVICE-ERROR u0 931.51 GB 1953525168 9QJ1ZN07
p2 OK u0 931.51 GB 1953525168 9QJ2XK1R
p3 OK u0 931.51 GB 1953525168 9QJ2010B
p4 OK u0 1.36 TB 2930277168 6XW0L36T
p5 OK u0 931.51 GB 1953525168 WD-WMATV2444836
p6 NOT-PRESENT - - - -
p7 NOT-PRESENT - - - -

Ctl Date Severity Alarm Message
------------------------------------------------------------------------------
c0 [Sat May 12 11:27:15 2012] WARNING Sector repair completed: port=0, LBA=0x6AE571C
c0 [Sat May 12 19:16:21 2012] WARNING Sector repair completed: port=1, LBA=0x40E62A23
c0 [Sat May 12 21:40:56 2012] INFO Verify completed: unit=0
c0 [Mon May 14 00:53:53 2012] WARNING Sector repair completed: port=1, LBA=0x7B8CFA7
c0 [Mon May 14 00:58:21 2012] WARNING Sector repair completed: port=1, LBA=0x7B8CFAA
c0 [Mon May 14 04:35:13 2012] WARNING Sector repair completed: port=0, LBA=0x8FEF2CF
c0 [Mon May 14 04:38:22 2012] WARNING Sector repair completed: port=0, LBA=0x8FEF2D1
c0 [Tue May 15 22:53:46 2012] WARNING Sector repair completed: port=0, LBA=0x13C2622
c0 [Wed May 16 00:39:31 2012] WARNING Sector repair completed: port=0, LBA=0x365A67F
c0 [Wed May 16 00:39:37 2012] WARNING Sector repair completed: port=0, LBA=0x365A685
c0 [Wed May 16 00:47:18 2012] WARNING Sector repair completed: port=0, LBA=0x365A687
c0 [Sat May 19 00:01:44 2012] INFO Verify started: unit=0
c0 [Sat May 19 04:46:20 2012] WARNING Sector repair completed: port=0, LBA=0x365A68E
c0 [Sat May 19 13:37:06 2012] WARNING Sector repair completed: port=1, LBA=0x7B8CFAC
c0 [Sat May 19 13:37:28 2012] WARNING Sector repair completed: port=1, LBA=0x7B8CFAE
c0 [Sat May 19 13:37:47 2012] WARNING Sector repair completed: port=1, LBA=0x7B8CFB1
c0 [Sat May 19 13:38:00 2012] WARNING Sector repair completed: port=1, LBA=0x7B8CFB3
c0 [Sat May 19 21:47:45 2012] INFO Verify completed: unit=0
c0 [Wed May 23 12:21:41 2012] INFO Cache synchronization completed: unit=0
c0 [Fri May 25 00:08:19 2012] WARNING Sector repair completed: port=0, LBA=0x12DA76C
c0 [Fri May 25 00:08:34 2012] WARNING Sector repair completed: port=0, LBA=0x12E4901
c0 [Fri May 25 00:09:33 2012] WARNING Sector repair completed: port=0, LBA=0x12DA773
c0 [Fri May 25 00:39:12 2012] WARNING Sector repair completed: port=0, LBA=0x42C597B
c0 [Sat May 26 00:01:45 2012] INFO Verify started: unit=0
c0 [Sat May 26 00:42:05 2012] WARNING Sector repair completed: port=1, LBA=0x323C1AC
c0 [Sat May 26 00:51:43 2012] WARNING Sector repair completed: port=1, LBA=0x323C1AE
c0 [Sat May 26 01:54:21 2012] WARNING Sector repair completed: port=1, LBA=0x2F0D302
c0 [Sat May 26 02:06:38 2012] WARNING Sector repair completed: port=0, LBA=0x12DA777
c0 [Sat May 26 02:07:21 2012] WARNING Sector repair completed: port=0, LBA=0x12E48FE
c0 [Sat May 26 04:20:00 2012] WARNING Sector repair completed: port=1, LBA=0x2F0D306
c0 [Sat May 26 04:32:58 2012] WARNING Sector repair completed: port=1, LBA=0x323C1B1
c0 [Sat May 26 04:33:21 2012] WARNING Sector repair completed: port=1, LBA=0x323C1B3
c0 [Sat May 26 04:33:44 2012] WARNING Sector repair completed: port=1, LBA=0x323C1BA
c0 [Sat May 26 05:24:07 2012] WARNING Sector repair completed: port=1, LBA=0x3F83862
c0 [Sat May 26 05:25:09 2012] WARNING Verify fixed data/parity mismatch: unit=0
c0 [Sat May 26 06:08:13 2012] WARNING Sector repair completed: port=0, LBA=0x4CDC6A2
c0 [Sat May 26 09:49:35 2012] WARNING Sector repair completed: port=1, LBA=0x6CACD4A
c0 [Sat May 26 18:10:44 2012] WARNING Sector repair completed: port=1, LBA=0x18F425EA
c0 [Sat May 26 19:45:40 2012] WARNING Verify fixed data/parity mismatch: unit=0
c0 [Sat May 26 20:22:52 2012] WARNING Verify fixed data/parity mismatch: unit=0
c0 [Sat May 26 20:23:15 2012] WARNING Verify fixed data/parity mismatch: unit=0
c0 [Sat May 26 20:23:22 2012] WARNING Verify fixed data/parity mismatch: unit=0
c0 [Sat May 26 20:23:35 2012] WARNING Verify fixed data/parity mismatch: unit=0
c0 [Sat May 26 20:23:41 2012] WARNING Verify fixed data/parity mismatch: unit=0
c0 [Sat May 26 20:23:49 2012] WARNING Verify fixed data/parity mismatch: unit=0
c0 [Sat May 26 20:23:57 2012] WARNING Verify fixed data/parity mismatch: unit=0
c0 [Sat May 26 20:24:02 2012] WARNING Verify fixed data/parity mismatch: unit=0
c0 [Sat May 26 20:54:41 2012] WARNING Verify fixed data/parity mismatch: unit=0
c0 [Sat May 26 22:00:30 2012] INFO Verify completed: unit=0
c0 [Sat Jun 2 00:01:43 2012] INFO Verify started: unit=0
c0 [Sat Jun 2 00:30:17 2012] WARNING Sector repair completed: port=0, LBA=0x2B911E4
c0 [Sat Jun 2 00:50:57 2012] WARNING Sector repair completed: port=0, LBA=0x5A807CA6
c0 [Sat Jun 2 04:13:13 2012] WARNING Sector repair completed: port=0, LBA=0x2D18291
c0 [Sat Jun 2 04:13:35 2012] WARNING Sector repair completed: port=0, LBA=0x2D1829F
c0 [Sat Jun 2 21:48:02 2012] INFO Verify completed: unit=0
c0 [Mon Jun 4 04:40:34 2012] WARNING Sector repair completed: port=1, LBA=0x4AF8098F
c0 [Tue Jun 5 00:28:19 2012] WARNING Sector repair completed: port=1, LBA=0x263C5CD
c0 [Tue Jun 5 00:33:06 2012] WARNING Sector repair completed: port=1, LBA=0x263C5CF
c0 [Thu Jun 7 00:34:27 2012] WARNING Sector repair completed: port=0, LBA=0x2A07B5F
c0 [Thu Jun 7 00:38:50 2012] WARNING Sector repair completed: port=0, LBA=0x2A07B61
c0 [Fri Jun 8 00:07:13 2012] WARNING Sector repair completed: port=0, LBA=0xC131F6B
c0 [Sat Jun 9 00:01:41 2012] INFO Verify started: unit=0
c0 [Sat Jun 9 00:29:11 2012] WARNING Sector repair completed: port=0, LBA=0x6C7614D
c0 [Sat Jun 9 00:38:25 2012] WARNING Sector repair completed: port=0, LBA=0x6C76152
c0 [Sat Jun 9 04:02:30 2012] WARNING Sector repair completed: port=1, LBA=0x263C5D1
c0 [Sat Jun 9 04:02:52 2012] WARNING Sector repair completed: port=1, LBA=0x263C5D3
c0 [Sat Jun 9 04:07:32 2012] WARNING Sector repair completed: port=0, LBA=0x27D3E12
c0 [Sat Jun 9 04:07:57 2012] WARNING Sector repair completed: port=0, LBA=0x27D3E15
c0 [Sat Jun 9 04:08:16 2012] WARNING Sector repair completed: port=0, LBA=0x27D3E17
c0 [Sat Jun 9 04:08:45 2012] WARNING Sector repair completed: port=0, LBA=0x27D3E19
c0 [Sat Jun 9 04:15:04 2012] WARNING Sector repair completed: port=0, LBA=0x2A07B64
c0 [Sat Jun 9 04:15:26 2012] WARNING Sector repair completed: port=0, LBA=0x2A07B66
c0 [Sat Jun 9 04:15:45 2012] WARNING Sector repair completed: port=0, LBA=0x2A07B68
c0 [Sat Jun 9 04:15:59 2012] WARNING Sector repair completed: port=0, LBA=0x2A07B6C
c0 [Sat Jun 9 04:16:13 2012] WARNING Sector repair completed: port=0, LBA=0x2A07B6E
c0 [Sat Jun 9 21:48:52 2012] INFO Verify completed: unit=0
c0 [Thu Jun 14 00:40:10 2012] WARNING Sector repair completed: port=0, LBA=0x334F14B
c0 [Sat Jun 16 00:01:38 2012] INFO Verify started: unit=0
c0 [Sat Jun 16 21:16:19 2012] INFO Verify completed: unit=0
c0 [Tue Jun 19 02:03:43 2012] WARNING Sector repair completed: port=1, LBA=0xFE41EAD
c0 [Wed Jun 20 02:30:02 2012] WARNING Sector repair completed: port=1, LBA=0xD99145C
c0 [Sat Jun 23 00:01:36 2012] INFO Verify started: unit=0
c0 [Sat Jun 23 04:27:04 2012] WARNING Sector repair completed: port=1, LBA=0x2FAD311
c0 [Sat Jun 23 06:52:38 2012] WARNING Sector repair completed: port=1, LBA=0x7C6AC8D
c0 [Sat Jun 23 06:53:03 2012] WARNING Sector repair completed: port=1, LBA=0x7C6AC91
c0 [Sat Jun 23 06:53:21 2012] WARNING Sector repair completed: port=1, LBA=0x7C6AC94
c0 [Sat Jun 23 17:00:22 2012] WARNING Sector repair completed: port=1, LBA=0xF9AC7C9
c0 [Sat Jun 23 21:15:19 2012] INFO Verify completed: unit=0
c0 [Sat Jun 30 00:01:34 2012] INFO Verify started: unit=0
c0 [Sat Jun 30 05:24:13 2012] WARNING Sector repair completed: port=0, LBA=0x3FAA9E7
c0 [Sat Jun 30 14:49:39 2012] WARNING Sector repair completed: port=1, LBA=0x869931C
c0 [Sat Jun 30 21:31:05 2012] INFO Verify completed: unit=0
c0 [Tue Jul 3 03:40:25 2012] WARNING Sector repair completed: port=1, LBA=0xD36C7F7
c0 [Fri Jul 6 02:50:18 2012] WARNING Sector repair completed: port=1, LBA=0x3562470
c0 [Fri Jul 6 22:18:26 2012] WARNING Sector repair completed: port=1, LBA=0x3563173
c0 [Sat Jul 7 00:01:31 2012] INFO Verify started: unit=0
c0 [Sat Jul 7 00:50:16 2012] WARNING Sector repair completed: port=0, LBA=0x76EE88
c0 [Sat Jul 7 00:50:39 2012] WARNING Sector repair completed: port=0, LBA=0x76EE8F
c0 [Sat Jul 7 21:39:36 2012] INFO Verify completed: unit=0
c0 [Sun Jul 8 02:51:05 2012] WARNING Sector repair completed: port=0, LBA=0x67759D
c0 [Sun Jul 8 02:53:55 2012] WARNING Sector repair completed: port=0, LBA=0x67759B
c0 [Tue Jul 10 16:17:21 2012] WARNING Sector repair completed: port=0, LBA=0x15C8C695
c0 [Wed Jul 11 22:51:22 2012] WARNING Sector repair completed: port=1, LBA=0x355BBD0
c0 [Sat Jul 14 00:01:28 2012] INFO Verify started: unit=0
c0 [Sat Jul 14 01:33:40 2012] WARNING Sector repair completed: port=1, LBA=0x1333BCF4
c0 [Sat Jul 14 03:36:23 2012] WARNING Sector repair completed: port=1, LBA=0x2174773
c0 [Sat Jul 14 11:26:44 2012] WARNING Sector repair completed: port=0, LBA=0x7429AB7
c0 [Sat Jul 14 16:53:50 2012] WARNING Sector repair completed: port=1, LBA=0xA17EB3F
c0 [Sat Jul 14 21:19:25 2012] INFO Verify completed: unit=0
c0 [Wed Jul 18 05:08:47 2012] WARNING Sector repair completed: port=1, LBA=0x17D62EDC
c0 [Wed Jul 18 05:14:15 2012] WARNING Sector repair completed: port=1, LBA=0x17D62EE1
c0 [Thu Jul 19 03:24:59 2012] WARNING Sector repair completed: port=0, LBA=0x7733C3D
c0 [Thu Jul 19 03:25:20 2012] WARNING Sector repair completed: port=0, LBA=0x773CEA5
c0 [Thu Jul 19 03:28:16 2012] WARNING Sector repair completed: port=0, LBA=0x7733C42
c0 [Thu Jul 19 03:28:41 2012] WARNING Sector repair completed: port=0, LBA=0x773CEAF
c0 [Sat Jul 21 00:01:26 2012] INFO Verify started: unit=0
c0 [Sat Jul 21 03:07:30 2012] WARNING Sector repair completed: port=1, LBA=0x1CC6936
c0 [Sat Jul 21 03:07:52 2012] WARNING Sector repair completed: port=1, LBA=0x1CC6938
c0 [Sat Jul 21 03:08:11 2012] WARNING Sector repair completed: port=1, LBA=0x1CC693A
c0 [Sat Jul 21 16:43:56 2012] WARNING Sector repair completed: port=0, LBA=0xD04C914
c0 [Sat Jul 21 16:45:31 2012] WARNING Sector repair completed: port=1, LBA=0xD456973
c0 [Sat Jul 21 21:14:29 2012] INFO Verify completed: unit=0
c0 [Wed Jul 25 03:37:25 2012] WARNING Sector repair completed: port=0, LBA=0x1F8E6C43
c0 [Sat Jul 28 00:01:24 2012] INFO Verify started: unit=0
c0 [Sat Jul 28 01:45:27 2012] WARNING Sector repair completed: port=0, LBA=0x11584AD
c0 [Sat Jul 28 18:54:25 2012] WARNING Sector repair completed: port=1, LBA=0x447C3E6C
c0 [Sat Jul 28 21:13:46 2012] INFO Verify completed: unit=0
c0 [Wed Aug 1 03:20:11 2012] WARNING Sector repair completed: port=0, LBA=0x805FEF
c0 [Fri Aug 3 00:50:03 2012] WARNING Sector repair completed: port=0, LBA=0xCED0ACA
c0 [Sat Aug 4 00:01:22 2012] INFO Verify started: unit=0
c0 [Sat Aug 4 00:52:51 2012] WARNING Sector repair completed: port=0, LBA=0x805FF3
c0 [Sat Aug 4 00:53:14 2012] WARNING Sector repair completed: port=0, LBA=0x805FF5
c0 [Sat Aug 4 00:53:33 2012] WARNING Sector repair completed: port=0, LBA=0x805FF7
c0 [Sat Aug 4 00:53:47 2012] WARNING Sector repair completed: port=0, LBA=0x805FF9
c0 [Sat Aug 4 00:54:00 2012] WARNING Sector repair completed: port=0, LBA=0x805FFB
c0 [Sat Aug 4 00:54:14 2012] WARNING Sector repair completed: port=0, LBA=0x805FFD
c0 [Sat Aug 4 00:54:27 2012] WARNING Sector repair completed: port=0, LBA=0x805FFF
c0 [Sat Aug 4 04:43:12 2012] WARNING Sector repair completed: port=1, LBA=0x16974289
c0 [Sat Aug 4 04:58:17 2012] WARNING Sector repair completed: port=1, LBA=0x1697428E
c0 [Sat Aug 4 20:54:53 2012] INFO Verify completed: unit=0
c0 [Wed Aug 8 03:21:55 2012] ERROR Drive timeout detected: port=1
c0 [Wed Aug 8 15:31:44 2012] WARNING Sector repair completed: port=0, LBA=0x1A366CD3
c0 [Sat Aug 11 00:01:21 2012] INFO Verify started: unit=0
c0 [Sat Aug 11 20:40:51 2012] INFO Verify completed: unit=0
c0 [Thu Aug 16 05:10:55 2012] WARNING Sector repair completed: port=0, LBA=0x1C22593
c0 [Sat Aug 18 00:01:18 2012] INFO Verify started: unit=0
c0 [Sat Aug 18 03:00:20 2012] WARNING Sector repair completed: port=0, LBA=0x1C225A5
c0 [Sat Aug 18 03:43:00 2012] WARNING Sector repair completed: port=1, LBA=0x23EE91E
c0 [Sat Aug 18 03:43:23 2012] WARNING Sector repair completed: port=1, LBA=0x23EE920
c0 [Sat Aug 18 17:00:06 2012] WARNING Sector repair completed: port=1, LBA=0x137D066A
c0 [Sat Aug 18 17:00:29 2012] WARNING Sector repair completed: port=1, LBA=0x137D066D
c0 [Sat Aug 18 21:13:01 2012] INFO Verify completed: unit=0
c0 [Wed Aug 22 01:36:08 2012] WARNING Sector repair completed: port=0, LBA=0x2560A0F
c0 [Wed Aug 22 01:37:42 2012] WARNING Sector repair completed: port=0, LBA=0x2560A13
c0 [Fri Aug 24 04:01:36 2012] WARNING Sector repair completed: port=1, LBA=0x55C1A5DF
c0 [Fri Aug 24 05:02:06 2012] WARNING Sector repair completed: port=1, LBA=0xCE3378A
c0 [Sat Aug 25 00:01:17 2012] INFO Verify started: unit=0
c0 [Sat Aug 25 00:31:06 2012] WARNING Sector repair completed: port=1, LBA=0x50F65D
c0 [Sat Aug 25 00:39:52 2012] WARNING Sector repair completed: port=0, LBA=0x678FF4
c0 [Sat Aug 25 03:43:15 2012] WARNING Sector repair completed: port=0, LBA=0x2560A15
c0 [Sat Aug 25 03:43:39 2012] WARNING Sector repair completed: port=0, LBA=0x2560A19
c0 [Sat Aug 25 03:43:58 2012] WARNING Sector repair completed: port=0, LBA=0x2560A1B
c0 [Sat Aug 25 03:44:30 2012] WARNING Sector repair completed: port=0, LBA=0x2560A21
c0 [Sat Aug 25 20:58:14 2012] INFO Verify completed: unit=0
c0 [Wed Aug 29 04:57:15 2012] WARNING Sector repair completed: port=1, LBA=0xF3957EB
c0 [Sat Sep 1 00:01:15 2012] INFO Verify started: unit=0
c0 [Sat Sep 1 03:21:52 2012] WARNING Sector repair completed: port=0, LBA=0x1DAFC86
c0 [Sat Sep 1 03:22:15 2012] WARNING Sector repair completed: port=0, LBA=0x1DAFC88
c0 [Sat Sep 1 03:22:34 2012] WARNING Sector repair completed: port=0, LBA=0x1DAFC8A
c0 [Sat Sep 1 03:22:47 2012] WARNING Sector repair completed: port=0, LBA=0x1DAFC8C
c0 [Sat Sep 1 17:17:22 2012] WARNING Sector repair completed: port=0, LBA=0xF917FD1
c0 [Sat Sep 1 17:17:45 2012] WARNING Sector repair completed: port=0, LBA=0xF917FD3
c0 [Sat Sep 1 17:18:04 2012] WARNING Sector repair completed: port=0, LBA=0xF917FD5
c0 [Sat Sep 1 21:36:56 2012] INFO Verify completed: unit=0
c0 [Thu Sep 6 00:07:30 2012] WARNING Sector repair completed: port=0, LBA=0xDA3C64B
c0 [Thu Sep 6 00:32:56 2012] WARNING Sector repair completed: port=1, LBA=0x6BBA816
c0 [Sat Sep 8 00:01:13 2012] INFO Verify started: unit=0
c0 [Sat Sep 8 00:09:56 2012] WARNING Sector repair completed: port=0, LBA=0xDEBC958
c0 [Sat Sep 8 04:38:45 2012] WARNING Sector repair completed: port=0, LBA=0x38D254F
c0 [Sat Sep 8 20:44:50 2012] INFO Verify completed: unit=0
c0 [Mon Sep 10 01:26:34 2012] WARNING Sector repair completed: port=1, LBA=0xFFD8D5E
c0 [Wed Sep 12 00:33:48 2012] WARNING Sector repair completed: port=1, LBA=0xE8DB928
c0 [Wed Sep 12 00:36:33 2012] WARNING Sector repair completed: port=1, LBA=0x6D49411
c0 [Fri Sep 14 01:59:39 2012] WARNING Sector repair completed: port=0, LBA=0x1467F1C
c0 [Fri Sep 14 02:08:27 2012] WARNING Sector repair completed: port=1, LBA=0x14C8ABD
c0 [Fri Sep 14 03:54:47 2012] WARNING Sector repair completed: port=0, LBA=0x1580C915
c0 [Sat Sep 15 00:01:11 2012] INFO Verify started: unit=0
c0 [Sat Sep 15 02:38:14 2012] WARNING Sector repair completed: port=0, LBA=0x1C178973
c0 [Sat Sep 15 02:59:02 2012] WARNING Sector repair completed: port=0, LBA=0x1C178975
c0 [Sat Sep 15 04:47:08 2012] WARNING Sector repair completed: port=0, LBA=0x3FA0356
c0 [Sat Sep 15 04:47:31 2012] WARNING Sector repair completed: port=0, LBA=0x3FA0359
c0 [Sat Sep 15 10:41:59 2012] WARNING Sector repair completed: port=0, LBA=0x6DFD1EC
c0 [Sat Sep 15 13:25:23 2012] WARNING Sector repair completed: port=0, LBA=0x7CBD100
c0 [Sat Sep 15 13:25:31 2012] WARNING Sector repair completed: port=0, LBA=0x7CBD104
c0 [Sat Sep 15 13:25:54 2012] WARNING Sector repair completed: port=0, LBA=0x7CBD106
c0 [Sat Sep 15 17:10:50 2012] WARNING Sector repair completed: port=0, LBA=0x1C178977
c0 [Sat Sep 15 20:59:57 2012] INFO Verify completed: unit=0
c0 [Tue Sep 18 01:17:18 2012] WARNING Sector repair completed: port=1, LBA=0x803B05B
c0 [Sat Sep 22 00:01:10 2012] INFO Verify started: unit=0
c0 [Sat Sep 22 20:54:31 2012] INFO Verify completed: unit=0
c0 [Tue Sep 25 01:56:47 2012] WARNING Sector repair completed: port=0, LBA=0x26E3909
c0 [Sat Sep 29 00:01:08 2012] INFO Verify started: unit=0
c0 [Sat Sep 29 02:04:14 2012] WARNING Sector repair completed: port=0, LBA=0x146AC03
c0 [Sat Sep 29 10:58:39 2012] WARNING Sector repair completed: port=0, LBA=0x6D4EB0E
c0 [Sat Sep 29 10:59:02 2012] WARNING Sector repair completed: port=0, LBA=0x6D4EB14
c0 [Sat Sep 29 11:22:44 2012] WARNING Sector repair completed: port=0, LBA=0x6F79623
c0 [Sat Sep 29 13:50:48 2012] WARNING Sector repair completed: port=1, LBA=0x7D1D65E
c0 [Sat Sep 29 13:51:11 2012] WARNING Sector repair completed: port=1, LBA=0x7D1D661
c0 [Sat Sep 29 13:51:30 2012] WARNING Sector repair completed: port=1, LBA=0x7D1D663
c0 [Sat Sep 29 20:57:34 2012] INFO Verify completed: unit=0
c0 [Mon Oct 1 04:47:24 2012] WARNING Sector repair completed: port=0, LBA=0xC5BC6F2
c0 [Tue Oct 2 02:00:27 2012] WARNING Sector repair completed: port=0, LBA=0x1547667
c0 [Tue Oct 2 02:01:56 2012] WARNING Sector repair completed: port=0, LBA=0x154766F
c0 [Tue Oct 2 05:02:31 2012] WARNING Sector repair completed: port=1, LBA=0xD67D054
c0 [Tue Oct 2 05:04:14 2012] WARNING Sector repair completed: port=1, LBA=0xD67D056
c0 [Wed Oct 3 01:22:12 2012] WARNING Sector repair completed: port=1, LBA=0x12AAF8CA
c0 [Thu Oct 4 04:29:22 2012] WARNING Sector repair completed: port=0, LBA=0x13E6F992
c0 [Thu Oct 4 05:10:51 2012] WARNING Sector repair completed: port=0, LBA=0x1C252A4
c0 [Sat Oct 6 00:01:07 2012] INFO Verify started: unit=0
c0 [Sat Oct 6 19:41:18 2012] WARNING Sector repair completed: port=1, LBA=0x5A5C3AE8
c0 [Sat Oct 6 21:01:05 2012] INFO Verify completed: unit=0
c0 [Mon Oct 8 00:32:06 2012] WARNING Sector repair completed: port=0, LBA=0x6C60D3E
c0 [Tue Oct 9 03:51:03 2012] WARNING Sector repair completed: port=1, LBA=0x89B5EC9
c0 [Thu Oct 11 04:21:17 2012] WARNING Sector repair completed: port=1, LBA=0x13F85833
c0 [Sat Oct 13 00:01:05 2012] INFO Verify started: unit=0
c0 [Sat Oct 13 05:12:40 2012] WARNING Sector repair completed: port=0, LBA=0x3FA5134
c0 [Sat Oct 13 21:08:35 2012] INFO Verify completed: unit=0
c0 [Tue Oct 16 03:53:50 2012] WARNING Sector repair completed: port=1, LBA=0x148AA1BD
c0 [Thu Oct 18 03:20:30 2012] WARNING Sector repair completed: port=0, LBA=0x1C8DABCB
c0 [Thu Oct 18 04:52:50 2012] WARNING Sector repair completed: port=0, LBA=0xE879057
c0 [Sat Oct 20 00:01:04 2012] INFO Verify started: unit=0
c0 [Sat Oct 20 02:19:25 2012] WARNING Sector repair completed: port=1, LBA=0x174B012
c0 [Sat Oct 20 03:41:38 2012] WARNING Sector repair completed: port=0, LBA=0x256D93B
c0 [Sat Oct 20 03:42:01 2012] WARNING Sector repair completed: port=0, LBA=0x256D93D
c0 [Sat Oct 20 03:42:40 2012] WARNING Sector repair completed: port=0, LBA=0x256D940
c0 [Sat Oct 20 03:42:59 2012] WARNING Sector repair completed: port=0, LBA=0x256D942
c0 [Sat Oct 20 03:43:12 2012] WARNING Sector repair completed: port=0, LBA=0x256D944
c0 [Sat Oct 20 03:43:26 2012] WARNING Sector repair completed: port=0, LBA=0x256D948
c0 [Sat Oct 20 16:37:52 2012] WARNING Sector repair completed: port=0, LBA=0xE879060
c0 [Sat Oct 20 16:38:15 2012] WARNING Sector repair completed: port=0, LBA=0xE879062
c0 [Sat Oct 20 21:00:18 2012] INFO Verify completed: unit=0
c0 [Sat Oct 20 23:49:01 2012] WARNING Sector repair completed: port=1, LBA=0x4473E908
c0 [Sun Oct 21 03:42:26 2012] WARNING Sector repair completed: port=1, LBA=0x175BADD5
c0 [Tue Oct 23 01:09:04 2012] WARNING Sector repair completed: port=1, LBA=0x6E524860
c0 [Fri Oct 26 03:21:25 2012] WARNING Sector repair completed: port=0, LBA=0x802C61
c0 [Fri Oct 26 04:22:21 2012] WARNING Sector repair completed: port=0, LBA=0x176353CD
c0 [Sat Oct 27 00:01:03 2012] INFO Verify started: unit=0
c0 [Sat Oct 27 00:49:35 2012] WARNING Sector repair completed: port=0, LBA=0x802C65
c0 [Sat Oct 27 17:02:24 2012] WARNING Sector repair completed: port=1, LBA=0xC1FF26D
c0 [Sat Oct 27 17:09:06 2012] WARNING Sector repair completed: port=0, LBA=0xDF621AD
c0 [Sat Oct 27 21:30:57 2012] INFO Verify completed: unit=0
c0 [Tue Oct 30 00:20:46 2012] WARNING Sector repair completed: port=0, LBA=0xE9FE2AB
c0 [Wed Oct 31 02:02:03 2012] WARNING Sector repair completed: port=0, LBA=0x1460C25
c0 [Wed Oct 31 02:04:05 2012] WARNING Sector repair completed: port=0, LBA=0x1460C28
c0 [Thu Nov 1 00:48:34 2012] WARNING Sector repair completed: port=1, LBA=0xA7C92BE
c0 [Thu Nov 1 05:04:45 2012] WARNING Sector repair completed: port=0, LBA=0x1C252C2

[root@backup1 /data/deprecated]#
</pre>

Look for failed drives and errors. Obviously from the above we need to probably replace drives 0 and 1 and drive 1 is even showing as having problems, yet the RAID array is healthy, amazingly. You also see the automatic verifies.

Note: when rebuilding a degraded mirror, you will see no progress as it rebuilds in the cli
<pre>Unit UnitType Status %RCmpl %V/I/M Stripe Size(GB) Cache AVrfy
------------------------------------------------------------------------------
u0 RAID-5 REBUILDING 0 - 64K 4656.56 OFF ON</pre>

=== areca ===

We are using an areca controller on backup3.

<pre>
[root@newbackup3 ~]# sh /root/verify.sh
# Name Raid Name Level Capacity Ch/Id/Lun State
===============================================================================
1 ARC-1160-VOL#00 Raid Set # 00 Raid5 5000.0GB 00/00/00 Checking(19.7%)
===============================================================================
GuiErrMsg<0x00>: Success.
# Name Disks TotalCap FreeCap DiskChannels State
===============================================================================
1 Raid Set # 00 6 6000.0GB 0.0GB 123456 Checking
===============================================================================
GuiErrMsg<0x00>: Success.
Date-Time Device Event Type Elapsed Time Errors
===============================================================================
2012-12-05 20:40:58 ARC-1160-VOL#00 Start Checking
2012-12-01 05:06:04 ARC-1160-VOL#00 Complete Init 027:30:45
2012-11-30 01:35:19 ARC-1160-VOL#00 Start Initialize
2026-08-06 01:34:52 H/W MONITOR Raid Powered On
2012-11-30 01:33:36 ARC-1160-VOL#00 Stop Initialization 000:31:48
2012-11-30 01:01:47 ARC-1160-VOL#00 Start Initialize
2026-08-06 00:58:13 H/W MONITOR Raid Powered On
2012-11-30 00:57:26 ARC-1160-VOL#00 Stop Initialization 000:57:07
2012-11-30 00:00:19 ARC-1160-VOL#00 Start Initialize
2026-08-05 23:56:48 H/W MONITOR Raid Powered On
2026-08-05 23:52:58 H/W MONITOR Raid Powered On
2026-08-05 23:50:14 H/W MONITOR Raid Powered On
2026-08-05 23:43:30 H/W MONITOR Raid Powered On
2012-11-29 23:10:07 ARC-1160-VOL#00 Stop Initialization 000:00:56
2012-11-29 23:09:11 ARC-1160-VOL#00 Start Initialize
2026-08-05 23:08:57 H/W MONITOR Raid Powered On
2012-11-29 23:08:10 ARC-1160-VOL#00 Stop Initialization 000:20:41
2012-11-29 22:47:29 ARC-1160-VOL#00 Start Initialize
2026-08-05 22:46:59 H/W MONITOR Raid Powered On
2026-08-05 22:45:55 H/W MONITOR Raid Powered On
2026-08-05 22:44:53 H/W MONITOR Raid Powered On
2026-08-05 22:42:06 H/W MONITOR Raid Powered On
2026-08-05 22:40:50 H/W MONITOR Raid Powered On
2012-11-29 22:40:04 ARC-1160-VOL#00 Stop Initialization 000:24:25
2012-11-29 22:15:38 ARC-1160-VOL#00 Start Initialize
2026-08-05 22:15:11 000:000001215B00 Restart Init LBA Point
2026-08-05 22:15:10 H/W MONITOR Raid Powered On
2012-11-29 21:56:38 ARC-1160-VOL#00 Start Initialize
2026-08-05 21:56:12 H/W MONITOR Raid Powered On
2026-08-05 21:56:04 IDE Channel #03 Device Inserted
2012-11-29 21:55:13 IDE Channel #04 Device Inserted
2012-11-29 21:55:03 IDE Channel #02 Device Inserted
2026-08-05 21:53:09 H/W MONITOR Raid Powered On
2026-08-05 20:51:46 H/W MONITOR Raid Powered On
2026-08-05 20:49:56 H/W MONITOR Raid Powered On
2026-08-05 20:48:29 H/W MONITOR Raid Powered On
2026-08-05 20:46:29 H/W MONITOR Raid Powered On
2026-08-05 20:44:49 H/W MONITOR Raid Powered On
2026-08-05 20:43:01 H/W MONITOR Raid Powered On
2026-08-05 20:36:25 H/W MONITOR Raid Powered On
2026-08-05 20:31:18 H/W MONITOR Raid Powered On
2026-08-05 20:30:08 H/W MONITOR Raid Powered On
2026-08-05 20:08:40 H/W MONITOR Raid Powered On
2026-08-05 20:06:11 H/W MONITOR Raid Powered On
2026-08-05 20:05:14 H/W MONITOR Raid Powered On
2026-08-05 20:03:58 H/W MONITOR Raid Powered On
2026-08-05 20:00:56 H/W MONITOR Raid Powered On
2026-08-05 19:57:57 H/W MONITOR Raid Powered On
2026-08-05 19:56:15 H/W MONITOR Raid Powered On
2026-08-05 19:55:05 H/W MONITOR Raid Powered On
2026-08-05 17:24:36 H/W MONITOR Raid Powered On
2026-08-05 17:22:43 H/W MONITOR Raid Powered On
2026-08-05 04:50:42 H/W MONITOR Raid Powered On
2026-08-05 04:47:33 H/W MONITOR Raid Powered On
2026-08-05 04:43:57 H/W MONITOR Raid Powered On
2026-08-05 04:18:52 H/W MONITOR Raid Powered On
2026-08-05 04:17:30 H/W MONITOR Raid Powered On
2026-08-05 04:13:30 H/W MONITOR Raid Powered On
2026-08-05 04:10:26 H/W MONITOR Raid Powered On
2026-08-05 04:09:23 H/W MONITOR Raid Powered On
2026-08-05 00:08:09 H/W MONITOR Raid Powered On
2026-08-05 00:07:12 H/W MONITOR Raid Powered On
2026-08-05 00:05:51 H/W MONITOR Raid Powered On
2026-08-05 00:04:27 H/W MONITOR Raid Powered On
===============================================================================
GuiErrMsg<0x00>: Success.
press enter when ready to run verify
</pre>

Look for failed drives and errors.

When it proceed's to verifying, you can confirm with:

<pre>
[root@newbackup3 ~]# cli64 vsf info
# Name Raid Name Level Capacity Ch/Id/Lun State
===============================================================================
1 ARC-1160-VOL#00 Raid Set # 00 Raid5 5000.0GB 00/00/00 Checking(22.5%)
===============================================================================
GuiErrMsg<0x00>: Success.
[root@newbackup3 ~]#
</pre>

== Update OS list ==

# check for any new VZ templates we want to offer: <tt>vzup2date -z</tt>
# see if there's any OS's we want to include in our colo install list. Update 2 places: <tt>signup/html/colo_quote.html</tt> & <tt>signup/html/step1.html</tt>
# update the mgmt database

= Infrequent tasks =

== Free up space on gateway ==

<pre>newgateway /var/spool# cd clientmqueue/
newgateway /var/spool/clientmqueue# sh
# for f in `ls`; do rm $f; done
exit</pre>

== Free up space on mail ==

You can clear out root mail:
<pre>
mail /var/log# ll -h /var/mail/root
-rw------- 1 root mail 543K Dec 19 13:05 /var/mail/root
mail /var/log# rm /var/mail/root
</pre>

Or you can archive mail logs:
<pre>
mail /var/log# ls -l htt*
-rw-r--r-- 1 root wheel 297436931 Dec 19 13:26 httpd-access.log
-rw-r--r-- 1 root wheel 9824324 Jul 4 11:34 httpd-access.log.old.0.gz
-rw-r--r-- 1 root wheel 6884137 Mar 17 2012 httpd-access.log.old.1.gz
-rw-r--r-- 1 root wheel 18557444 Dec 3 2009 httpd-access.log.old.10.gz
-rw-r--r-- 1 root wheel 14740263 Jan 9 2007 httpd-access.log.old.11.gz
-rw-r--r-- 1 root wheel 14209465 Nov 28 2007 httpd-access.log.old.12.gz
-rw-r--r-- 1 root wheel 16874396 Feb 19 2012 httpd-access.log.old.3.gz
-rw-r--r-- 1 root wheel 14554859 Jul 22 2011 httpd-access.log.old.4.gz
-rw-r--r-- 1 root wheel 10513227 Feb 18 2011 httpd-access.log.old.5.gz
-rw-r--r-- 1 root wheel 7201946 Oct 29 2010 httpd-access.log.old.6.gz
-rw-r--r-- 1 root wheel 10062537 May 6 2010 httpd-access.log.old.7.gz
-rw-r--r-- 1 root wheel 10157042 Aug 12 2010 httpd-access.log.old.8.gz
-rw-r--r-- 1 root wheel 11909534 Mar 4 2010 httpd-access.log.old.9.gz
-rw-r--r-- 1 root wheel 59030930 Dec 19 13:01 httpd-error.log
-rw-r--r-- 1 root wheel 3413134 Mar 4 2010 httpd-error.log.0.gz
-rw-r--r-- 1 root wheel 795515 May 1 2007 httpd-error.log.1.gz
-rw-r--r-- 1 root wheel 1142153 Nov 30 2007 httpd-error.log.2.gz
-rw-r--r-- 1 root wheel 2325801 Feb 18 2011 httpd-error.log.gz

mail /var/log# sh
# for f in 12 11 10 9 8 7 6 5 4 3 2 1 0; do g=`echo $f+1|bc`; mv httpd-access.log.old.$f.gz httpd-access.log.old.$g.gz; done
# mv httpd-access.log httpd-access.log.old.0
# touch httpd-access.log
# apachectl restart
# gzip httpd-access.log.old.0

# for f in 2 1 0; do g=`echo $f+1|bc`; mv httpd-error.log.$f.gz httpd-error.log.$g.gz; done
# mv httpd-error.log httpd-error.log.0
# touch httpd-error.log
# apachectl restart
# gzip httpd-error.log.0
# exit
</pre>

== Free up space on bwdb2 ==

You can either remove items from <tt>/usr/home/archive</tt> or you can scp them to backup3:/data/bwdb2/archive .

== Free up space on backup1 ==

backup1 is our primary customer backup system. As usage grows over time, it needs to be regularly purged of old files. The easiest way to do this is by removing deprecated files. These mostly consist of cancelled customers or temporary dump/storage files (created during dump/restores). Our standard policy is to hang onto cancelled customers for 6mos after which we remove their files (as far as customers know their data is purged immediately, but we hang onto it just in case.. and in some cases we cancel a server due to non payment so this makes it easy to restore their system). To find files to remove:

<pre>[root@backup1 ~]# cd /data/deprecated/
[root@backup1 /data/deprecated]# ls
2101-migrated-20120317.tgz old-683-cxld-20121021.tgz
69.55.230.2-wwwbackup old-744-cxld-20120708.tgz
991-DONTDELETE.tgz old-809-cxld-20120609.tgz
archive-col02050-mdfile-cxld-20120409.gz old-854-cxld-20120621.tgz
col01371.tgz old-931-cxld-20060513.tgz
deleteme_ubuntu-10.10-x86_20111205 old-col00123-mdfile-noarchive-20120417.gz
jail10_old old-col00147-vnfile-cxld-20120828.gz
jail14_rsync_old old-col00419-dump-cxld-20120224.gz
jail15_old old-col01098-vnfile-cxld-20120827.gz
jail3_old old-col01278-dump-cxld-20120822
jail4_old old-col01517-dump-cxld-20120828
jail5_old old-col01669-dump-cxld-20120203.gz
old-1009-cxld-20120608.tgz old-col01687-dump-cxld-20120909
old-1012-cxld-20120411.tgz old-col01790-dump-cxld-20120828
old-1052-cxld-20120721.tgz old-col01812-dump-cxld-20120820
old-10631-cxld-20120622.tgz old-col01938-mdfile-cxld-20120619.gz
old-10632-cxld-20120622.tgz old-col02095-mdfile-noarchive-20120523.gz
old-10633-cxld-20120622.tgz olddebian-3.0-v15-20110610.tgz
old-1236-cxld-20120621.tgz oldmod_frontpage-deb30-v15-20110610.tgz
old-1381-cxld-20120404.tgz oldmod_perl-deb30-v15-20110610.tgz
old-1422-cxld-20120721.tgz oldmod_ssl-deb30-v15-20110610.tgz
old-14681-cxld-20120619.tgz oldmysql-deb30-v15-20110610.tgz
old-1544-cxld-20120626.tgz oldproftpd-deb30-v15-20110610.tgz
old-18351-cxld-20120605.tgz old_virt14
old-1853-cxld-20120910.tgz old_virt18
old-1963-cxld-20120206.tgz oldwebmin-deb30-v15-20110610.tgz
old-1967-cxld-20120605.tgz suse.virt11.20120421.tgz
old-1981-noarchive-20120729.tgz virt11
old-2030-migrated-noarchive-20120727.tgz virt12_old
old-2037-cxld-20120716.tgz virt13_old
old-2065-cxld-20120727.tgz virt16_old
old-2068-cxld-20120424.tgz virt4_old
old-2085-cxld-20120531.tgz virt5_old
old-364-cxld-20120904.tgz virt6_old
old-446-cxld-20120512.tgz virt7_old
old-613-cxld-20120601.tgz virt8_old
[root@backup1 /data/deprecated]#
</pre>

virtX_old and jailX_old are permanently archived, so ignore those as well as anything else marked not to delete or otherwise suspicious. Likewise, probably a good idea to try to hang onto oldTEMPLATE.gz as long as we can as well.
Most of the stuff we want to delete is dated when it was deprecated, making this easy. So to remove files from 6 mos ago (running this in Oct):

<pre>[root@backup1 /data/deprecated]# ls old*201204*
old-1012-cxld-20120411.tgz old-2068-cxld-20120424.tgz
old-1381-cxld-20120404.tgz old-col00123-mdfile-noarchive-20120417.gz
[root@backup1 /data/deprecated]# rm old*201204*
</pre>

Every few months you will also want to remove some of the snapshot archives for mail. We typically save the 1st, 10th, and 20th of each month. To do this you set aside the dates you want to save then remove months at a time, followed by restoring the set aside dates. Here's how that works:
<pre>[root@backup1 /data/www/daily]# ls
05 08-10-11 10-04-10 11-10-10 12-07-29 12-09-21 12-11-14
06 08-10-21 10-04-20 11-10-20 12-07-30 12-09-22 12-11-15
06-06-01-usr-home.tgz 08-11-01 10-05-01 11-11-01 12-07-31 12-09-23 12-11-16
06-07-01-usr-home.tgz 08-11-10 10-05-11 11-11-10 12-08-01 12-09-24 12-11-17
06-08-01-usr-home.tgz 08-11-20 10-05-20 11-11-20 12-08-02 12-09-25 12-11-18
06-09-01-usr-home.tgz 08-12-01 10-06-01 11-12-01 12-08-03 12-09-26 12-11-19
06-11-10 08-12-10 10-06-10 11-12-10 12-08-04 12-09-27 12-11-20
06-12-21 08-12-20 10-06-20 11-12-20 12-08-05 12-09-28 12-11-21
07-01-10 09-01-01 10-07-01 12-01-01 12-08-06 12-09-29 12-11-22
07-01-20 09-01-10 10-07-10 12-01-10 12-08-07 12-09-30 12-11-23
07-02-10 09-01-20 10-07-20 12-01-20 12-08-08 12-10-01 12-11-24
07-02-20 09-02-01 10-08-01 12-02-01 12-08-09 12-10-02 12-11-25
07-03-01 09-02-10 10-08-10 12-02-10 12-08-10 12-10-03 12-11-26
07-03-20 09-02-20 10-08-20 12-02-20 12-08-11 12-10-04 12-11-27
07-04-01 09-03-01 10-09-01 12-03-01 12-08-12 12-10-05 12-11-28
07-04-10 09-03-10 10-09-10 12-03-10 12-08-13 12-10-06 12-11-29
07-04-20 09-03-20 10-09-20 12-03-20 12-08-14 12-10-07 12-11-30
07-05-01 09-04-01 10-10-01 12-04-01 12-08-15 12-10-08 12-12-01
07-05-10 09-04-10 10-10-10 12-04-10 12-08-16 12-10-09 12-12-02
07-05-20 09-04-20 10-10-20 12-04-20 12-08-17 12-10-10 12-12-03
07-06-01 09-05-01 10-11-01 12-05-01 12-08-18 12-10-11 12-12-04
07-06-10 09-05-10 10-11-10 12-05-10 12-08-19 12-10-12 12-12-05
07-06-20 09-05-20 10-11-20 12-05-20 12-08-20 12-10-13 12-12-06
07-07-20 09-06-01 10-12-01 12-06-01 12-08-21 12-10-14 12-12-07
07-08-10 09-06-10 10-12-10 12-06-10 12-08-22 12-10-15 12-12-08
07-08-20 09-06-20 10-12-20 12-06-20 12-08-23 12-10-16 12-12-09
07-09-01 09-07-01 11-01-01 12-07-01 12-08-24 12-10-17 12-12-10
07-10-01 09-07-10 11-01-10 12-07-02 12-08-25 12-10-18 12-12-11
07-10-10 09-07-20 11-01-21 12-07-03 12-08-26 12-10-19 12-12-12
07-10-20 09-08-01 11-02-01 12-07-04 12-08-27 12-10-20 12-12-13
07-12-01 09-08-10 11-02-10 12-07-05 12-08-28 12-10-21 12-12-14
07-12-10 09-08-20 11-02-20 12-07-06 12-08-29 12-10-22 12-12-15
08-01-01 09-09-01 11-03-01 12-07-07 12-08-30 12-10-23 12-12-16
08-01-20 09-09-10 11-03-10 12-07-08 12-08-31 12-10-24 12-12-17
08-02-20 09-09-20 11-03-20 12-07-09 12-09-01 12-10-25 12-12-18
08-03-01 09-10-01 11-04-01 12-07-10 12-09-02 12-10-26 12-12-19
08-03-10 09-10-10 11-04-10 12-07-11 12-09-03 12-10-27 12-12-20
08-03-20 09-10-20 11-04-20 12-07-12 12-09-04 12-10-28 12-12-21
08-04-01 09-11-01 11-05-01 12-07-13 12-09-05 12-10-29 12-12-22
08-04-20 09-11-10 11-05-10 12-07-14 12-09-06 12-10-30 12-12-23
08-05-01 09-11-20 11-05-20 12-07-15 12-09-07 12-10-31 12-12-24
08-05-10 09-12-01 11-06-01 12-07-16 12-09-08 12-11-01 12-12-25
08-06-10 09-12-10 11-06-10 12-07-17 12-09-09 12-11-02 12-12-26
08-06-20 09-12-20 11-06-20 12-07-18 12-09-10 12-11-03 12-12-27
08-07-02 10-01-01 11-07-01 12-07-19 12-09-11 12-11-04 12-12-28
08-07-10 10-01-10 11-07-10 12-07-20 12-09-12 12-11-05 2008-10-23
08-07-20 10-01-20 11-07-20 12-07-21 12-09-13 12-11-06 bb.tgz
08-08-01 10-02-01 11-08-01 12-07-22 12-09-14 12-11-07 boot
08-08-10 10-02-10 11-08-10 12-07-23 12-09-15 12-11-08 current
08-08-21 10-02-20 11-08-20 12-07-24 12-09-16 12-11-09 hold
08-09-01 10-03-01 11-09-01 12-07-25 12-09-17 12-11-10
08-09-10 10-03-10 11-09-10 12-07-26 12-09-18 12-11-11
08-09-21 10-03-20 11-09-20 12-07-27 12-09-19 12-11-12
08-10-01 10-04-01 11-10-01 12-07-28 12-09-20 12-11-13
[root@backup1 /data/www/daily]#
</pre>

So we see that everything up to July 2012 has been pruned. To prune July 2012 we do the following:

<pre>mv 12-07-01 hold
mv 12-07-10 hold
mv 12-07-20 hold
rm -fr 12-07*
mv hold/* .</pre>

Virtuozzo Server Install

2013-12-07T17:56:53Z

24.165.90.109: /* Configure system, install virtuozzo */

= 4.7 =

Server-specific changes in <span style="color:red">red</span>

http://www.parallels.com/download/pvc/lin

http://www.parallels.com/support/virtualization-suite/pvc/

== BIOS ==

* enable post info to console

2950:<br>
Serial comm.: on with cons. Redir via com1<br>
Failsafe: 115200<br>
Redir after boot: enabled

* Tweak DRAC settings (see [[DRAC/RMM#DRAC_setup|DRAC setup]])

== Install OS ==

CentOS6.x (get latest from http://mirrors.kernel.org/centos/6.4/isos/x86_64/CentOS-6.4-x86_64-netinstall.iso)

May have to disable DRAC virtual media to make sure we get sda as our first device

Set partitions:
<pre>Disk 1:
/ 10240M (ext4)
9216M swap
/vz (the rest) (ext4)
Disk 2:
9216M Swap
/vz1 (the rest) (ext4)</pre>

Basic server

Use grub, install to MBR

== Configure system, install virtuozzo ==
* add ssh

yum install openssh-clients

* create ssh key, add swsoft key to authorized_keys

(enter to everything)

<pre>cd /root/
ssh-keygen -t dsa

Add vz support key:
echo "ssh-dss AAAAB3NzaC1kc3MAAACBAIB7XHmpcsa1D/vsKS+mzEvBVo7udew/aRahbjnjywL4YXT7ruWzpGo7R3UA8GKIgVHoQGUbuXrJz5ICtBEXOlbgXVEz7Bu5nXOG2C9i/8pmFtUw6GwACII8uZE7t/Zq5CRj+5zN1Rv7n/4X4RAjzk5BHXoZ62a32FTbS4tPvyPrAAAAFQDiUi7YEiffNzGvB544lQpRdn29TwAAAIBkUchLTBE1+sX11iC80yhNh9EGeCMLdkT8UqHsTSaD0myQPxGALuNY4rO31q+25uKqSRDTXd6ZWXp4bplA6Ls6jl7r7XVR9loJ7WvRMqc4Dv68TZ7h2UBqAhu4det2UF4LHim3DNf1gKn9DzWNkmI/N2FvVVAL4aM3sPunVIvUYgAAAIAJX38INABVgnCEBKqpv5FMLm3k9MgMn54oQFmylBbgM0w4SUeuDj3jLrkJf8dT9w8ByEDb8xviexpWrPsbz2DeThle4xgzcyKrUkt2Wugtmm4DwS0S/68BAA9wXoWD/vZ67GBoCKH7uq8VAjUVcCbJ/7IydljnFAZoJpktdwN7iQ== russup@support.sw-soft.com" >> /root/.ssh/authorized_keys</pre>

* setup access to backup2:

<pre>echo "10.1.4.3 backup2" >> /etc/hosts
echo "10.1.4.8 backup1" >> /etc/hosts</pre>

# Copy id_dsa.pub to backup2 and add it to the end of ~root/.ssh/authorized_keys:

<pre>cat .ssh/id_dsa.pub | ssh backup2 'cat - >> /root/.ssh/authorized_keys'
cat .ssh/id_dsa.pub | ssh backup1 'cat - >> /root/.ssh/authorized_keys'</pre>

<pre>ssh backup2 hostname
ssh backup1 hostname</pre>
(confirm you can connect w/o entering a password)

* add bash backward history search
<pre>echo "\"\M-z\": history-search-backward
\"\M-x\": history-search-backward " > /root/.inputrc
echo "export GREP_OPTIONS='--color=auto'" >> ~/.bashrc
echo "\"\e[5~\": history-search-backward" >> ~/.inputrc
echo "\"\e[6~\": history-search-forward" >> ~/.inputrc</pre>

* add to /root/.bashrc, create log dir

<pre>vi /root/.bashrc
alias v='vzctl'
alias mbm='mb mount'
alias mbu='mb umount'
export PS1="[\u@\h \w]# "
alias cv='cd /vz/private'
alias cv1='cd /vz1/private'
alias cv2='cd /vz2/private'
alias pg='ps auxw|grep $1'</pre>

source /root/.bashrc

mkdir /root/logs

* make sure serial console and kernel messages are working

add these 2 lines to the top:

<pre>vi /boot/grub/menu.lst
serial --unit=0 --speed= 115200
terminal --timeout=10 serial console</pre>

and the kernel line should look like:

kernel /boot/vmlinuz-2.4.20-020stab009.24.777-enterprise ro root=/dev/sda1 console=tty0 console=ttyS0,115200

Remove or comment out these lines:
<pre>hiddensplashmenu
splashimage=(hd0,0)/boot/grub/splash.xpm.gz</pre>

add to /etc/inittab and /etc/securetty

<pre>echo "s0:2345:respawn:/sbin/agetty -L ttyS0 115200 linux" >> /etc/inittab
echo "ttyS0" >> /etc/securetty</pre>

kill -1 1

* make sure the hostname on the system is properly set - in the form virt4.johncompanies.com - and make sure it is set properly:
cat /etc/sysconfig/network

* Check/update fstab

vi /etc/fstab

/etc/fstab entries should look something like (note noatime option):

<pre>/dev/sda3 /vz ext4 defaults,noatime 1 2
/dev/sdb2 /vz1 ext4 defaults,noatime 1 2</pre>

make sure swap is configured in fstab

Use real devices instead of LABEL

* Install Virtuozzo 4.7

Have a license on hand - request a day ahead of the install. Go to:
http://www.parallels.com/download/pvc47/

<pre>yum install compat-libstdc++-33
yum install wget
yum install ftp
yum install pciutils
yum install ed bc

cd /vz
wget http://download.parallels.com/pvc/47/lin/vzinstall-linux-x86_64.bin
chmod 0700 vzinstall-linux-x86_64.bin
./vzinstall-linux-x86_64.bin</pre>

download and install

change wd's to /vz

Ignore OS version error

Custom install (don’t install templates)

Install PVA Agent
Install PVA Management Node

reboot

Get on KVM to watch PVA install (won't happen on console)

vzup2date

(run again. till no more updates offered)

reboot

<DEPRECATED>
Run vzup2date (make sure to choose enterprise kernel) and/or install latest kernel:

NOTE: skip over the manual kernel load below if you can get the 16G recognized before running vzup2date- then it will dl the ent kernel

NOTE: go to /vz/vzup2date/virtuozzo/linux/i386/4.0.0/CU-2.6.18-028stab068.3 and install kern there (downloaded by vzup2date)

vi /boot/grub/menu.lst
(set default kernel to new kernel)
</DEPRECATED>

* Install ntp

yum install ntp
(if not already installed)

rm -f /etc/ntp.conf

Create a new file with one line:

echo "server 10.1.4.5" > /etc/ntp.conf

Then set ntpd so it will start at boot:

chkconfig --level 3 ntpd on
service ntpd start

Make sure it’s talking to the ntp server:

ntpq -p

Correct output should have something unfer the refid col and a '+' next to the remote server:

<pre>remote refid st t when poll reach delay offset jitter
==============================================================================
10.1.4.105+ clock1.redhat.c 2 u - 64 1 0.370 -706677 0.008</pre>

* copy over binaries/scripts from backup2

scp backup2:/mnt/data4/bin/linux/* /usr/local/sbin/
scp <span style="color:red">10.1.4.62</span>:"/vzconf/jctmpl*" /etc/sysconfig/vz-scripts/
scp <span style="color:red">10.1.4.62</span>:/vzconf/ve-slm* /etc/sysconfig/vz-scripts/

* setup root crontab (alter backup time):
<pre>crontab -e
1 1 * * * /usr/local/sbin/backup
*/15 * * * * /usr/local/sbin/vim.watchdog.30 > /dev/null
#56 4,16 * * * root /usr/sbin/vzkeyupdate -c
*/5 * * * * /usr/local/sbin/perc5iraidchk64</pre>

* setup backups

<pre>
echo '#!/bin/sh
<tt>backupdir=/data/<span style="color:red">virt13</span> </tt>

server=backup1

## ENTRY /etc
## ENTRY /usr/local/sbin
## ENTRY /root/logs
' > /usr/local/sbin/backup.config
</pre>

* setup bigbrother

edit passwd and add the bb user:
echo "bb:x:1984:1984:Big Brother:/home/bb:/bin/bash" >> /etc/passwd

edit group and add the bb group:
echo "bb:x:1984:" >> /etc/group

rebuild the password database:
pwconv

Create bb’s home dir and set ownership:
mkdir /home/bb
chown bb.bb /home/bb

Go to bb’s home and fetch the client tarball:
<pre>cd ~bb
scp backup2:/mnt/data4/build/bb/GOOD-bb-linux-bbc1.9i-btf.tgz .

tar xvf GOOD-bb-linux-bbc1.9i-btf.tgz

yum install glibc-2.12-1.47.el6_2.5.i686

vi etc/bbsys.sh
SORT="/bin/sort"

vi bin/bb-disk.sh
eval "$DFCMD | $EGREP \"$DFUSE\" | $EGREP -v \"$DFEXCLUDE\" | $SORT \"-k${DFSORT}\" > $BBTMP/DFOUT.$$"
wclen1=`eval "$DFCMD | $EGREP \"$DFUSE\" | $EGREP -v \"$DFEXCLUDE\" | $SORT \"-k${DFSORT}\" | $WC"`

cd bbc1.9i-btf/etc/

echo "10.1.4.5 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh" > /home/bb/bbc1.9i-btf/etc/bb-hosts
echo "10.1.4.63 virt13.johncompanies.com # ssh" >> /home/bb/bbc1.9i-btf/etc/bb-hosts
(^^^change to correct host)

chmod +r /var/log/messages

cd ..
su bb
./runbb.sh start
exit</pre>

Add this line to /etc/rc.d/rc.local so bb will start at boot:

echo "su - bb -c \"cd /home/bb/bbc1.9i-btf; ./runbb.sh start\"" >> /etc/rc.d/rc.local

* Uncomment/add lines with the appropriate IP addresses in /etc/ssh/sshd_config:

vi /etc/ssh/sshd_config
<tt>ListenAddress <span style="color:red">69.55.226.2</span>
ListenAddress <span style="color:red">10.1.4.63</span></tt>

<DEPRECATED>
And make sure to uncomment
Protocol 2,1
</DEPRECATED>

* setup backup1/backup2 nfs mounts

<DEPRECATED>
setup portmap to start with server, and start it:

chkconfig --level 3 portmap on
service portmap start
</DEPRECATED>

mb mount

run df to confirm that nfs mounts are mounted

* make alias for /vzconf
ln -s /etc/sysconfig/vz-scripts /vzconf

* edit /etc/sysconfig/vz

vi /etc/sysconfig/vz

calc cpu to reserver for HN. take 5% of power:
vzcpucheck
Current CPU utilization: 27000
Power of the node: 1018320

so in this case:
VE0CPUUNITS=50916

enable fastboot:
VZFASTBOOT=yes

Disable auto mount
#VZMOUNTS="/vz"

Enable slm:
SLM=yes

<DEPRECATED>
Enable IPV6
IPV6=yes
</DEPRECATED>

add/edit to traffic parameters section:
TRAFFIC_SHAPING=yes
BANDWIDTH="eth0:102400"
TOTALRATE="eth0:1:102400"
RATE="eth0:1:50000"

for 2 nics, uncomment this line:
VE_ROUTE_SRC_DEV="eth0"

Make sure the IPTABLES section looks like:
<pre>IPTABLES="ip_tables iptable_filter iptable_mangle ipt_limit ipt_multiport ipt_tos ipt_TOS ipt_REJECT ipt_tcpmss ipt_ttl ipt_LOG ipt_length ip_conntrack ip_conntrack_ftp ip_conntrack_irc ipt_conntrack ipt_state ipt_helper iptable_nat ip_nat_ftp ip_nat_irc ipt_REDIRECT ipt_multiport ipt_TCPMSS"</pre>

* disable periodic fsck's, lower the root padding on the /vz and /vz1 filesystems to 100MB
<pre>
tune2fs -i 0 /dev/sda3
tune2fs -c -1 /dev/sda3
tune2fs -r 100000 /dev/sda3

tune2fs -i 0 /dev/sdb2
tune2fs -c -1 /dev/sdb2
tune2fs -r 100000 /dev/sdb2</pre>

* setup vzagent with pub IP (to allow migrations from 3.x):
vzctl set 50 --ipdel all --ipadd <span style="color:red">69.55.226.3</span> --userpasswd vzagent0:k0zub1k --save
vzagent_ctl restart

* edit ve config files:

From split of 80 take
<tt>CPUUNITS</tt>

And put into files in /vzconf/base_ct_conf
along with:
<pre>VERSION="2"

ONBOOT="yes"

KMEMSIZE="2147483647:2147483647"
LOCKEDPAGES="2147483647:2147483647"
PRIVVMPAGES="2147483647:2147483647"
SHMPAGES="2147483647:2147483647"
NUMPROC="32567:32567"
VMGUARPAGES="2147483647:2147483647"
OOMGUARPAGES="2147483647:2147483647"
NUMTCPSOCK="2147483647:2147483647"
NUMFLOCK="2147483647:2147483647"
NUMPTY="255:255"
NUMSIGINFO="1024:1024"
TCPSNDBUF="2147483647:2147483647"
TCPRCVBUF="2147483647:2147483647"
OTHERSOCKBUF="2147483647:2147483647"
DGRAMRCVBUF="2147483647:2147483647"
NUMOTHERSOCK="2147483647:2147483647"
DCACHESIZE="2147483647:2147483647"
NUMFILE="2147483647:2147483647"
NUMIPTENT="2147483647:2147483647"

DISKSPACE=""
DISKINODES=""
QUOTATIME="600"
IOPRIO="4"
CPULIMIT="75"
OFFLINE_MANAGEMENT="yes"
ARCH="x86"
PLATFORM="linux"

IP_ADDRESS="0.0.0.0"
NAMESERVER="69.55.225.225 69.55.230.3"
CPUUNITS="12313"
IOLIMIT="10485760"
IOPSLIMIT="100"
</pre>

* customize the control panel:
rm -fr /vz/backups

vi /var/opt/pva/pp/xsl/lib/vt/backups-forms.xsl

<pre>
<span style="font-size:13px; font-weight:bold">NOTE: You cannot create or view backups via this control panel- IT WILL NOT WORK. Please contact linux@johncompanies.com to setup or inquire about backups for your server.</span>

/var/opt/pva/pp/xsl/lib/vt/backups-list.xsl
</pre>

cd /var/opt/pva/pp/static/vz/skins
for f in `ls`; do scp backup2:/d4/build/vzcp/var/vzcp/static/vz/skins/winxp.new/images/logo_main_4.7.gif $f/images/logo_main.gif; done

* template install

To install new templates:

vzup2date -z

Copy over templates from old server. Ex: ubu 8.04 from virt19:

rsync -a -e ssh <span style="color:red">10.1.4.69</span>:/vz/template/ubuntu/8.04 /vz/template/ubuntu/8.04

install ubuntu 8.04 templates (get latest from vz)

<pre>cd /backup4/vzrpms/distro/ubuntu/8.04
rpm -ivh *
vzpkg update cache ubuntu-8.04-x86</pre>

If you install a tempalte from vzup2date it will download an old cache as well. Better to download the OS rpm from VZ site then create cache then use vzup2date to install the application templates.

* add traffic shaping support

Re-define class 0:

<pre>vi /etc/sysconfig/vz-scripts/networks_classes
0 69.55.224.0/20
0 10.1.4.0/24</pre>

make sure 1 is defined (0.0.0.0) and no other classes are needed

Then start it

/etc/init.d/vz shaperrestart

* udp dos fix

echo "/usr/local/sbin/iptables.sh" >> /etc/rc.d/rc.local

* make rc.local start before vz

mv /etc/rc.d/rc3.d/S99local /etc/rc.d/rc3.d/S95local

* disable unwanted services/daemons

Before:
<pre>abrt-ccpp 0:off 1:off 2:off 3:on 4:off 5:on 6:off
abrt-oops 0:off 1:off 2:off 3:on 4:off 5:on 6:off
abrtd 0:off 1:off 2:off 3:on 4:off 5:on 6:off
acpid 0:off 1:off 2:on 3:on 4:on 5:on 6:off
atd 0:off 1:off 2:off 3:on 4:on 5:on 6:off
auditd 0:off 1:off 2:on 3:on 4:on 5:on 6:off
autofs 0:off 1:off 2:off 3:on 4:on 5:on 6:off
cpuspeed 0:off 1:on 2:on 3:on 4:on 5:on 6:off
crond 0:off 1:off 2:on 3:on 4:on 5:on 6:off
cups 0:off 1:off 2:on 3:on 4:on 5:on 6:off
haldaemon 0:off 1:off 2:off 3:on 4:on 5:on 6:off
ip6tables 0:off 1:off 2:on 3:on 4:on 5:on 6:off
iptables 0:off 1:off 2:on 3:on 4:on 5:on 6:off
irqbalance 0:off 1:off 2:off 3:on 4:on 5:on 6:off
kdump 0:off 1:off 2:off 3:on 4:on 5:on 6:off
lvm2-monitor 0:off 1:on 2:on 3:on 4:on 5:on 6:off
mcelogd 0:off 1:off 2:off 3:on 4:off 5:on 6:off
mdmonitor 0:off 1:off 2:on 3:on 4:on 5:on 6:off
messagebus 0:off 1:off 2:on 3:on 4:on 5:on 6:off
netfs 0:off 1:off 2:off 3:on 4:on 5:on 6:off
network 0:off 1:off 2:on 3:on 4:on 5:on 6:off
nfslock 0:off 1:off 2:off 3:on 4:on 5:on 6:off
ntpd 0:off 1:off 2:off 3:on 4:off 5:off 6:off
portreserve 0:off 1:off 2:on 3:on 4:on 5:on 6:off
postfix 0:off 1:off 2:on 3:on 4:on 5:on 6:off
pvaagentd 0:off 1:off 2:on 3:on 4:on 5:on 6:off
pvapp 0:off 1:off 2:on 3:on 4:on 5:on 6:off
qpidd 0:off 1:off 2:on 3:on 4:on 5:on 6:off
rpcbind 0:off 1:off 2:on 3:on 4:on 5:on 6:off
rpcgssd 0:off 1:off 2:off 3:on 4:on 5:on 6:off
rpcidmapd 0:off 1:off 2:off 3:on 4:on 5:on 6:off
rsyslog 0:off 1:off 2:on 3:on 4:on 5:on 6:off
sshd 0:off 1:off 2:on 3:on 4:on 5:on 6:off
sysstat 0:off 1:on 2:on 3:on 4:on 5:on 6:off
udev-post 0:off 1:on 2:on 3:on 4:on 5:on 6:off
vz 0:off 1:off 2:on 3:on 4:on 5:on 6:off
vzlmond 0:off 1:off 2:on 3:on 4:on 5:on 6:off
vznetcfg 0:off 1:off 2:on 3:on 4:on 5:on 6:off
xinetd 0:off 1:off 2:off 3:on 4:on 5:on 6:off</pre>

<pre>chkconfig --level 3 cpuspeed off
chkconfig --level 3 cups off
chkconfig --level 3 haldaemon off
chkconfig --level 3 ip6tables off
chkconfig --level 3 iptables off
chkconfig --level 3 irqbalance off
chkconfig --level 3 lvm2-monitor off
chkconfig --level 3 mdmonitor off
chkconfig --level 3 messagebus off
chkconfig --level 3 netfs off
chkconfig --level 3 nfslock off
chkconfig --level 3 qpidd off
chkconfig --level 3 rpcgssd off
chkconfig --level 3 rpcidmapd off</pre>

* load mrtg graphs

yum install xinetd

create /etc/xinetd.d/load with the following contents:

<pre>echo "service load
{
disable = no
flags = REUSE
socket_type = stream
wait = no
user = root
server = /usr/local/sbin/load.pl
}" > /etc/xinetd.d/load</pre>

Add this line to /etc/services:
echo "load 12384/tcp" >> /etc/services

* Tasks to do on backup1:
setup backup dirs:

mkdir -p /data/<span style="color:red">virt13</span>/0

for f in 0 1 2 3 4 5 6; do mkdir -p /data/<span style="color:red">virt13</span>/$f/vz/private; mkdir -p /data/<span style="color:red">virt13</span>/$f/vz1/private; mkdir -p /data/<span style="color:red">virt13</span>/$f/vz2/private; done

add the system to
vi /usr/local/sbin/snapshot_rotate

* Tasks to do on mail:

Update backup graph:
vi /usr/local/www/mgmt/cgi/backupgraph.pl
(add hostname)

add to bbhosts:
<pre>vi /usr/home/bb/bbsrc/bb1.9i-btf/etc/bb-hosts
su bb
cd
bbsrc/bb/runbb.sh restart ; exit</pre>

add to load graph config:
vi /usr/local/www/mgmt/mrtg/mrtg1.cfg
(add new section for new machine)

* add to management db (on mail and devweb) jc.ref_machines and jc.ref_templates

insert new server:
insert into ref_machines values (null,'virt19','virt19.johncompanies.com',0,'l');

get new machine_id back out:
select * from ref_machines;

insert as a system to jc customer col00001:
insert into systems values ('', 'col00001',30,null,null,100,'virt11.johncompanies.com',null,'2010-03-11',null,null,1,0,null,null,null);

get sysid:
select * from systems where hostname='virt19.johncompanies.com';

update ipmap (main IP and IP used for VZCP):
update ipmap set cid='col00001', sysid=3772, assign_date='2008-01-07', status=1 where ip='69.55.236.2';
update ipmap set cid='col00001', sysid=3772, assign_date='2008-01-07', status=1 where ip='69.55.230.109';

add templates (or do via mgmt):
<pre>
insert into ref_templates values ('','fedora-core-1/20050725',33,'Fedora Core 1',1);
insert into ref_templates values ('','fedora-core-2/20050725',33,'Fedora Core 2',1);
insert into ref_templates values ('','debian-3.1/20050927',33,'Debian 3.1',1);</pre>

* Make sure that ns1c has proper forward and reverse entries for the new system (both private and public addresses), so that new system emailing will work.
<pre>vr johncompanies.com
vi internal.johncompanies.com
rndc reload johncompanies.com IN private</pre>
(edit the PTR too)

* add an outside blocking rule to the firewall, so this machine can only be reached from inside the firewall.

Follow example already in firewall virt19 is:

<pre>ipfw add 00143 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 or 62.213.72.101 or 64.131.90.27 or 82.200.65.190 or 91.204.24.0/22 or 194.44.160.178 or 195.214.232.10 or 195.214.233.10 or 208.251.212.1 } to 69.55.226.2
ipfw add 00143 deny ip from any to 69.55.226.2</pre>

(virt12 would be 00142)

* add to [[Cabinetmap|Cabinetmap]]

* make /vz1/private
<pre>mkdir /vz1/private ; chmod 0755 /vz1/private
mkdir /vz1/root ; chmod 0755 /vz1/root</pre>

* put config files in a place they’ll get backed up
<pre>mkdir /vzconf/confs
crontab -e
0 0 * * * rm -f /vzconf/confs/*.conf; cp /vzconf/*.conf /vzconf/confs</pre>

* fix pass/vzpp and localse issue with ubu 8.10

vi /vz/template/ubuntu/8.10/x86/config/os/default/post-install

sed '/^password.*pam_unix.so.*sha512/ s~sha512~md5~' -i etc/pam.d/common-password

localedef -i en_US -c -f UTF-8 en_US.UTF-8

* add tun device support

echo "modprobe tun" >> /etc/rc.local

* reinstall VZA/CT #1/CT #50

After the install, the PP was broken. So needed to reinstall everything. Did the pva_setup --uninstall and removed CT 50 and then setup the pva per http://kb.parallels.com/113071 then setup the CT 50 per the other article- centos6 had issues - and then authenticated with virt13 to get that into the management list.

http://kb.parallels.com/113072

http://kb.parallels.com/113071

* config logrotate (otherwise it creates weekly empty logs)

vi /etc/logrotate.conf

(add to end)

<pre>/var/log/messages {
rotate 12
monthly
size 10M
missingok
compress
delaycompress
notifempty
sharedscripts
postrotate
echo `date` > /var/log/messages
endscript
}</pre>

* install megacli64 (can we just move the bin and not install the lib?)

<pre>scp backup2: /d4/build/3ware/MegaCli-8.00.40-1.i386.rpm /root/MegaCli-8.00.40-1.i386.rpm
scp backup2: /d4/build/3ware/Lib_Utils-1.00-08.noarch.rpm /root/ Lib_Utils-1.00-08.noarch.rpm

rpm -ivh Lib_Utils-1.00-08.noarch.rpm
rpm -ivh MegaCli-8.00.40-1.i386.rpm

cp opt/MegaRAID/MegaCli/MegaCli64 /usr/local/sbin/megacli64</pre>

= Older versions =

== 4.0 ==

Last updated 2010-03-11

<pre>1. enable post info to console

2950:
Serial comm.: on with cons. Redir via com1
Failsafe: 115200
Redir after boot: enabled

Server
Custom- desel all but base
/root/install.log

2. Install OS

CentOS5.x (get latest from http://mirrors.kernel.org/centos/5.4/isos/i386/)

Set partitions:
Disk 1:
/ 3072M
6144M swap
/vz (the rest)
Disk 2:
6144M Swap

Use grub, install to MBR
No firewall
Disable SElinux
Install only base

3. create ssh key, add swsoft key to authorized_keys

(enter to everything)

cd /root/
ssh-keygen -t dsa

echo "ssh-dss AAAAB3NzaC1kc3MAAACBAIB7XHmpcsa1D/vsKS+mzEvBVo7udew/aRahbjnjywL4YXT7ruWzpGo7R3UA8GKIgVHoQGUbuXrJz5ICtBEXOlbgXVEz7Bu5nXOG2C9i/8pmFtUw6GwACII8uZE7t/Zq5CRj+5zN1Rv7n/4X4RAjzk5BHXoZ62a32FTbS4tPvyPrAAAAFQDiUi7YEiffNzGvB544lQpRdn29TwAAAIBkUchLTBE1+sX11iC80yhNh9EGeCMLdkT8UqHsTSaD0myQPxGALuNY4rO31q+25uKqSRDTXd6ZWXp4bplA6Ls6jl7r7XVR9loJ7WvRMqc4Dv68TZ7h2UBqAhu4det2UF4LHim3DNf1gKn9DzWNkmI/N2FvVVAL4aM3sPunVIvUYgAAAIAJX38INABVgnCEBKqpv5FMLm3k9MgMn54oQFmylBbgM0w4SUeuDj3jLrkJf8dT9w8ByEDb8xviexpWrPsbz2DeThle4xgzcyKrUkt2Wugtmm4DwS0S/68BAA9wXoWD/vZ67GBoCKH7uq8VAjUVcCbJ/7IydljnFAZoJpktdwN7iQ== russup@support.sw-soft.com" >> /root/.ssh/authorized_keys

4. setup access to backup2:

echo "10.1.4.3 backup2" >> /etc/hosts
echo "10.1.4.8 backup1" >> /etc/hosts

# Copy id_dsa.pub to backup2 and add it to the end of ~root/.ssh/authorized_keys:

cat .ssh/id_dsa.pub | ssh backup2 'cat - >> /root/.ssh/authorized_keys'
cat .ssh/id_dsa.pub | ssh backup1 'cat - >> /root/.ssh/authorized_keys'

ssh backup2 hostname
ssh backup1 hostname

(confirm you can connect w/o entering a password)

5. add bash backward history search
echo "\"\M-z\": history-search-backward
\"\M-x\": history-search-backward " > /root/.inputrc
echo "export GREP_OPTIONS='--color=auto'" >> ~/.bashrc
echo "\"\e[5~\": history-search-backward" >> ~/.inputrc
echo "\"\e[6~\": history-search-forward" >> ~/.inputrc

6. add to /root/.bashrc, create log dir

vi /root/.bashrc
alias v='vzctl'
alias mbm='mb mount'
alias mbu='mb umount'
export PS1="[\u@\h \w]# "
alias cv='cd /vz/private'
alias cv1='cd /vz1/private'
alias cv2='cd /vz2/private'

source /root/.bashrc

mkdir /root/logs

7. make sure serial console and kernel messages are working

add these 2 lines to the top:

vi /boot/grub/menu.lst
serial --unit=0 --speed= 115200
terminal --timeout=10 serial console

and the kernel line should look like:

kernel /boot/vmlinuz-2.4.20-020stab009.24.777-enterprise ro root=/dev/sda1 console=tty0 console=ttyS0,115200

Remove or comment out these lines:
hiddensplashmenu
splashimage=(hd0,0)/boot/grub/splash.xpm.gz

add to /etc/inittab and /etc/securetty

echo "s0:2345:respawn:/sbin/agetty -L ttyS0 115200 linux" >> /etc/inittab
echo "ttyS0" >> /etc/securetty

kill -1 1

DEPRECATED 8. configuring eth1 for management net:

echo "10.1.5/24 via 10.1.4.2
10.1.6/24 via 10.1.4.2" >> /etc/sysconfig/network-scripts/route-eth1

reboot the machine to make sure that the interface is configured properly.

also verify that the routes are in place like so:

netstat -rn | grep 10.1.4.2
10.1.5.0 10.1.4.2 255.255.255.0 UG 40 0 0 eth1
10.1.6.0 10.1.4.2 255.255.255.0 UG 40 0 0 eth1

9. make sure the hostname on the system is properly set - in the form virt4.johncompanies.com - and make sure it is set properly:
cat /etc/sysconfig/network

10. Check/update fstab

vi /etc/fstab

/etc/fstab entries should look something like (note noatime option):

/dev/sda3 /vz ext3 defaults,noatime 1 2
/dev/sdb2 /vz1 ext3 defaults,noatime 1 2

make sure swap is configured in fstab

Use real devices instead of LABEL

11. Install Virtuozzo 4.0

umount /vz
umount /vz1

mke2fs -i1024 -b1024 -j /dev/sda3 &
mke2fs -i1024 -b1024 -j /dev/sdb2 &

Experimentally ran this for sdc on virt12:
mke2fs -i2048 -b2048 -j /dev/sdb1

mount -a

scp backup2:/d4/vzrpms/iso/vzinstall-linux.bin /vz
cd /vz
chmod 0700 vzinstall-linux.bin
./vzinstall-linux.bin

change wd to /vz
Ignore OS version error
Custom install (don’t install templates)

reboot

vzup2date

(run again. till no more updates offered)

reboot

Run vzup2date (make sure to choose enterprise kernel) and/or install latest kernel:

NOTE: skip over the manual kernel load below if you can get the 16G recognized before running vzup2date- then it will dl the ent kernel

NOTE: go to /vz/vzup2date/virtuozzo/linux/i386/4.0.0/CU-2.6.18-028stab068.3 and install kern there (downloaded by vzup2date)

<DEPREACATED>
ftp downloads.swsoft.com
User:
gdawson
Pass:
qjdK1R

cd /updates/Rel400/20080307
bin
get vzkernel-ent-2.6.18-028stab053.10.i686.rpm
get vzmodules-ent-2.6.18-028stab053.10.i686.rpm
bye

rpm -ivh vzkernel-ent-2.6.18-028stab053.10.i686.rpm \
vzmodules-ent-2.6.18-028stab053.10.i686.rpm
</DEPREACATED>

vi /boot/grub/menu.lst
(set default kernel to new kernel)

12. Install ntp

yum install ntp

rm -f /etc/ntp.conf

# Create a new file with one line:

echo "server 10.1.4.5" > /etc/ntp.conf

# Then set ntpd so it will start at boot:

chkconfig --level 3 ntpd on
service ntpd start

# Make sure it’s talking to the ntp server:

ntpq -p

Correct output should have * or + to the left of the IP:

remote refid st t when poll reach delay offset jitter
==============================================================================
10.1.4.105 clock1.redhat.c 2 u - 64 1 0.370 -706677 0.008

13. copy over binaries/scripts from backup2

scp backup2:/mnt/data4/bin/linux/* /usr/local/sbin/
scp 10.1.4.62:/vzconf/jctmpl* /etc/sysconfig/vz-scripts/
scp 10.1.4.62:/vzconf/ve-slm* /etc/sysconfig/vz-scripts/

14. setup root crontab (alter backup time):
crontab -e
1 1 * * * /usr/local/sbin/backup
*/15 * * * * /usr/local/sbin/vim.watchdog.30 > /dev/null
#56 4,16 * * * root /usr/sbin/vzkeyupdate -c
*/5 * * * * /usr/local/sbin/perc5iraidchk

15. setup backups

echo '#!/bin/sh
backupdir=/data/virt11
server=backup1

## ENTRY /etc
## ENTRY /usr/local/sbin
## ENTRY /root/logs
' > /usr/local/sbin/backup.config

16. setup bb

edit passwd and add the bb user:
echo "bb:x:1984:1984:Big Brother:/home/bb:/bin/bash" >> /etc/passwd

# edit group and add the bb group:
echo "bb:x:1984:" >> /etc/group

# rebuild the password database:

pwconv

# Create bb’s home dir and set ownership:

mkdir /home/bb
chown bb.bb /home/bb

# Go to bb’s home and fetch the client tarball:

cd ~bb
scp backup2:/mnt/data4/build/bb/bb-linux.tar .

tar xf bb-linux.tar

cd bbc1.9e-btf/etc/

echo "10.1.4.5 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh" > /home/bb/bbc1.9e-btf/etc/bb-hosts
echo "10.1.4.61 virt11.johncompanies.com # ssh" >> /home/bb/bbc1.9e-btf/etc/bb-hosts

chmod +r /var/log/messages

cd ..
su bb
./runbb.sh start
exit

# Add this line to /etc/rc.d/rc.local so bb will start at boot:

echo "su - bb -c \"cd /home/bb/bbc1.9e-btf; ./runbb.sh start\"" >> /etc/rc.d/rc.local

17. Uncomment/add lines with the appropriate IP addresses in /etc/ssh/sshd_config:

vi /etc/ssh/sshd_config
ListenAddress 69.55.238.3
ListenAddress 10.1.4.61

And make sure to uncomment
Protocol 2,1

18. setup backup2 nfs mounts

setup portmap to start with server, and start it:

chkconfig --level 3 portmap on
service portmap start

mb mount

run df to confirm that nfs mounts are mounted

19. make alias for /vzconf
ln -s /etc/sysconfig/vz-scripts /vzconf

DEPRECATED 20. copy over the /etc/sysconfig/vz-scripts/4unlimited template file from the last virt server and put it in /etc/sysconfig/vz-scripts

scp 10.1.4.68:/etc/sysconfig/vz-scripts/4unlimited /etc/sysconfig/vz-scripts/

NOTE: if this system is more powerful (RAM, CPU) then the previous, you will want to create a new one using vzsplit:

cd /etc/sysconfig/vz-scripts/
vzsplit -f 4unlimited

21. edit /etc/sysconfig/vz

vi /etc/sysconfig/vz

enable fastboot:
VZFASTBOOT=yes

Disable auto mount
#VZMOUNTS="/vz"

Enable IPV6
IPV6=yes

add/edit to traffic parameters section:
TRAFFIC_SHAPING=yes
BANDWIDTH="eth0:102400"
TOTALRATE="eth0:1:102400"
RATE="eth0:1:50000"

for 2 nics, uncomment this line:
VE_ROUTE_SRC_DEV="eth0"

Make sure the IPTABLES section looks like:
IPTABLES="ip_tables iptable_filter iptable_mangle ipt_limit ipt_multiport ipt_tos ipt_TOS ipt_REJECT ipt_tcpmss ipt_ttl ipt_LOG ipt_length ip_conntrack ip_conntrack_ftp ip_conntrack_irc ipt_conntrack ipt_state ipt_helper iptable_nat ip_nat_ftp ip_nat_irc ipt_REDIRECT ipt_multiport ipt_TCPMSS"

22. disable periodic fsck's, lower the root padding on the /vz and /vz1 filesystems to 100MB

tune2fs -i 0 /dev/sda3
tune2fs -i 0 /dev/sdb2
tune2fs -c -1 /dev/sda3
tune2fs -c -1 /dev/sdb2
tune2fs -r 100000 /dev/sda3
tune2fs -r 100000 /dev/sdb2

23. setup vzagent with pub IP (to allow migrations from 3.x):

vzctl set 1 --ipdel all --ipadd 69.55.230.57 --userpasswd vzagent0:k0zub1k --save
vzagent_ctl restart

24. edit ve config files:

From split of 80 take
CPUUNITS

And put into files in /usr/local/sbin/ve-slm.1024MB.conf et. al.

25. customize the control panel:

vzmlocal 1:1:/vz/private/1:/vz/root/1
cd /vz/private/1/fs/root/var/vzcp/static/vz/skins
for f in `ls`; do scp backup2:/d4/build/vzcp/var/vzcp/static/vz/skins/winxp.new/images/logo_main.gif $f/images/logo_pim.gif; done

1) comment 'Backup' tab in /etc/vzcp/pp/tabs.xml:

<tab-bar id="maintance" obj-type="env.core.vt">


2) change 'href' parameter if 'Maintentance' item in /etc/vzcp/pp/menu.xml:


<item id="management.maintenance" href="/vz/cp/vzdir/infrman/envs/repair?branch=env-home,vzpp" icon="backup" tip="maintenance_screen">

3) /etc/vzcp/pp/dashboard.xml

<item id="management.maintenanance" href="$baseurl/vzdir/infrman/envs/repair" title="Maintenance" icon="env_backup" tip="maintenance_screen" cred="vzcp.env.backups.list" security-obj-type="ve.vz.vt">

URL for 'href' parameter can be obtained from tabs.xml.

vi /var/vzcp/xsl/screens/vzdir/monitman/traffic.xsl
<sw:text>Traffic for selected period and class (NOTE: this information is not certified by JohnCompanies and is not used to calculate actual usage or charges)</sw:text>

<DEPREACATED>
/vz/private/1/root/var/vzcp/static/vz/skins/
/vz/private/1/root/etc/vzcp/pp/menu.xml
/vz/private/1/root/etc/vzcp/pp/dashboard.xml
pp/tabs.xml
/vz/private/1/root/etc/vzcp/vzcpcon.conf

eg (2.6.1 or 3.0):
rsync -va -e ssh backup2:/d4/build/vzcp/var/vzcp/static/vz/skins/ /vz/private/1/root/var/vzcp/static/vz/skins/
rsync -va -e ssh backup2:/d4/build/vzcp/etc/vzcp/pp/menu.xml /vz/private/1/root/etc/vzcp/pp/menu.xml
rsync -va -e ssh backup2:/d4/build/vzcp/etc/vzcp/pp/dashboard.xml /vz/private/1/root/etc/vzcp/pp/dashboard.xml
rsync -va -e ssh backup2:/d4/build/vzcp/etc/vzcp/vzcpcon.conf /vz/private/1/root/etc/vzcp/vzcpcon.conf
</DEPREACATED>

24. template install

To install new templates:

vzup2date –z

Copy over ubu 8.04 from virt19

rsync -a -e ssh 10.1.4.69:/vz/template/ubuntu/8.04 /vz/template/ubuntu/8.04

install ubuntu 8.04 templates (get latest from vz)

cd /backup4/vzrpms/distro/ubuntu/8.04
rpm -ivh *
vzpkg update cache ubuntu-8.04-x86

26. add traffic shaping support

Re-define class 0:

vi /etc/sysconfig/vz-scripts/networks_classes
0 69.55.224.0/20
0 10.1.4.0/24

make sure 1 is defined (0.0.0.0) and no other classes are needed

Then start it

/etc/init.d/vz shaperrestart

27. udp dos fix

echo "/usr/local/sbin/iptables.sh" >> /etc/rc.d/rc.local

28. make rc.local start before vz

mv /etc/rc.d/rc3.d/S99local /etc/rc.d/rc3.d/S95local

29. disable unwanted services/daemons

chkconfig --level 3 avahi-daemon off
chkconfig --level 3 bluetooth off
chkconfig --level 3 cpuspeed off
chkconfig --level 3 cups off
chkconfig --level 3 firstboot off
chkconfig --level 3 gpm off
chkconfig --level 3 haldaemon off
chkconfig --level 3 hidd off
chkconfig --level 3 ip6tables off
chkconfig --level 3 iptables off
chkconfig --level 3 irqbalance off
chkconfig --level 3 isdn off
chkconfig --level 3 kudzu off
chkconfig --level 3 lvm2-monitor off
chkconfig --level 3 mcstrans off
chkconfig --level 3 mdmonitor off
chkconfig --level 3 messagebus off
chkconfig --level 3 netfs off
chkconfig --level 3 nfslock off
chkconfig --level 3 pcscd off
chkconfig --level 3 rawdevices off
chkconfig --level 3 readahead_early off
chkconfig --level 3 restorecond off
chkconfig --level 3 rpcgssd off
chkconfig --level 3 rpcidmapd off
chkconfig --level 3 smartd off
chkconfig --level 3 yum-updatesd off

30. load mrtg graphs

yum install xinetd

create /etc/xinetd.d/load with the following contents:

echo "service load
{
disable = no
flags = REUSE
socket_type = stream
wait = no
user = root
server = /usr/local/sbin/load.pl
}" > /etc/xinetd.d/load

# Add this line to /etc/services:
echo "load 12384/tcp" >> /etc/services

31. Tasks to do on backup1:

setup backup dirs:
mkdir -p /data/virt11/0

add the system to
vi /usr/local/sbin/snapshot_rotate

32. Tasks to do on mail:

Update backup graph:
vi /usr/local/www/mgmt/cgi/backupgraph.pl
(add hostname)

add to bbhosts:
vi /usr/home/bb/bbsrc/bb1.9e-btf/etc/bb-hosts
su bb
cd
bbsrc/bb/runbb.sh restart ; exit

add to load graph config:
vi /usr/local/www/mgmt/mrtg/mrtg1.cfg
(add new section for new machine)

33. add to management db (on mail and devweb) jc.ref_machines and jc.ref_templates

insert into ref_machines values (null,'virt19','virt19.johncompanies.com',0,'l');
select * from ref_machines;
insert into systems values ('', 'col00001',30,null,null,100,'virt11.johncompanies.com',null,'2010-03-11',null,null,1,0,null,null,null);
select * from systems where hostname='virt19.johncompanies.com';
update ipmap set cid='col00001', sysid=3772, assign_date='2008-01-07', status=1 where ip='69.55.236.2';
update ipmap set cid='col00001', sysid=3772, assign_date='2008-01-07', status=1 where ip='69.55.230.109';
insert into ref_templates values ('','fedora-core-1/20050725',33,'Fedora Core 1',1);
insert into ref_templates values ('','fedora-core-2/20050725',33,'Fedora Core 2',1);
insert into ref_templates values ('','debian-3.1/20050927',33,'Debian 3.1',1);

34. Make sure that ns1c has proper forward and reverse entries for the new system (both private and public addresses), so that new system emailing will work.
vr johncompanies.com
vi internal.johncompanies.com
rndc reload johncompanies.com IN private
(edit the PTR too)

35. add an outside blocking rule to the firewall, so this machine can only be reached from inside the firewall.

Follow example already in firewall virt19 is:

ipfw add 00149 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 or 62.213.72.101 or 64.131.90.27 or 82.200.65.190 or 91.204.24.0/22 or 194.44.160.178 or 195.214.232.10 or 195.214.233.10 or 208.251.212.1 } to 69.55.236.2
ipfw add 00149 deny ip from any to 69.55.236.2

virt12 would be 00142

36. add to server/cabinet map

37. make /vz1/private
mkdir /vz1/private ; chmod 0755 /vz1/private
mkdir /vz1/root ; chmod 0755 /vz1/root

38. put config files in a place they’ll get backed up
mkdir /vzconf/confs
crontab -e
0 0 * * * rm -f /vzconf/confs/*.conf; cp /vzconf/*.conf /vzconf/confs

39. fix pass/vzpp and localse issue with ubu 8.10

vi /vz/template/ubuntu/8.10/x86/config/os/default/post-install

sed '/^password.*pam_unix.so.*sha512/ s~sha512~md5~' -i etc/pam.d/common-password

localedef -i en_US -c -f UTF-8 en_US.UTF-8

40. add tun device support

echo "modprobe tun" >> /etc/rc.local</pre>

== 3.x ==

Last updated 2007-04-5

<pre>1. enable post info to console
Supermicro:
Console redirection:
Com port addr: on-board COM A
Baud: 38400
Console type: vt100
Flow control: none
Console connection: direct
Continue cr after post: off

2450:
Make sure running bios A09
Console Redirection: VT100/VT220

2950:
Serial comm.: on with cons. Redir via com1
Failsafe: 115200
Redir after boot: enabled

6144
Server
Custom- desel all but base
/root/install.log

2. Install OS
Set partitions:
/ 3072M
Disk 1: Swap 4096M
Disk 2: Swap 4096M
/vz (the rest)

No firewall
Disable SElinux

FC4
Choose a custom install, the “minimal” package

CentOS4 Server CD (only option for 2950)
Choose the “minimal” package

3. create ssh key, add swsoft key to authorized_keys

(enter to everything)

cd /root/
ssh-keygen -t dsa

echo "ssh-dss AAAAB3NzaC1kc3MAAACBAIB7XHmpcsa1D/vsKS+mzEvBVo7udew/aRahbjnjywL4YXT7ruWzpGo7R3UA8GKIgVHoQGUbuXrJz5ICtBEXOlbgXVEz7Bu5nXOG2C9i/8pmFtUw6GwACII8uZE7t/Zq5CRj+5zN1Rv7n/4X4RAjzk5BHXoZ62a32FTbS4tPvyPrAAAAFQDiUi7YEiffNzGvB544lQpRdn29TwAAAIBkUchLTBE1+sX11iC80yhNh9EGeCMLdkT8UqHsTSaD0myQPxGALuNY4rO31q+25uKqSRDTXd6ZWXp4bplA6Ls6jl7r7XVR9loJ7WvRMqc4Dv68TZ7h2UBqAhu4det2UF4LHim3DNf1gKn9DzWNkmI/N2FvVVAL4aM3sPunVIvUYgAAAIAJX38INABVgnCEBKqpv5FMLm3k9MgMn54oQFmylBbgM0w4SUeuDj3jLrkJf8dT9w8ByEDb8xviexpWrPsbz2DeThle4xgzcyKrUkt2Wugtmm4DwS0S/68BAA9wXoWD/vZ67GBoCKH7uq8VAjUVcCbJ/7IydljnFAZoJpktdwN7iQ== russup@support.sw-soft.com" >> /root/.ssh/authorized_keys

4. setup access to backup2:

echo "10.1.4.3 backup2" >> /etc/hosts

Copy id_dsa.pub to backup2 and add it to the end of ~root/.ssh/authorized_keys:

cat .ssh/id_dsa.pub | ssh backup2 'cat - >> /root/.ssh/authorized_keys'

ssh backup2

(confirm you can connect w/o entering a password)

5. add bash backward history search
echo "\"\M-z\": history-search-backward
\"\M-x\": history-search-backward " > /root/.inputrc
echo "export GREP_OPTIONS='--color=auto'" >> ~/.bashrc
echo "\"\e[5~\": history-search-backward" >> ~/.inputrc
echo "\"\e[6~\": history-search-forward" >> ~/.inputrc

6. add to /root/.bashrc, create log dir

vi /root/.bashrc
alias v='vzctl'
alias mbm='mb mount'
alias mbu='mb umount'
export PS1="[\u@\h \w]# "
alias cv='cd /vz/private'
alias cv1='cd /vz1/private'
alias cv2='cd /vz2/private'

mkdir /root/logs

7. make sure serial console and kernel messages are working

grub

add these 2 lines to the top:

vi /boot/grub/menu.lst
serial --unit=0 --speed= 115200
terminal --timeout=10 serial console

and the kernel line should look like:

kernel /boot/vmlinuz-2.4.20-020stab009.24.777-enterprise ro root=/dev/sda1 console=tty0 console=ttyS0,115200

Remove the line:
hiddensplashmenu

lilo

Make sure the append line in /etc/lilo.conf looks like this:

append="console=tty console=ttyS0,38400"

If you edit /etc/lilo.conf don’t forget to run lilo

# lilo
Added linux
Added linux-up
Added linux+virtuozzo *

(note: the last console listed is the one that gets output from the rc scripts.)

add to /etc/inittab and /etc/securetty

echo "s0:2345:respawn:/sbin/agetty -L ttyS0 115200 linux" >> /etc/inittab
echo "ttyS0" >> /etc/securetty

kill -1 1

8. configuring eth1 for management net:

echo "10.1.5/24 via 10.1.4.2
10.1.6/24 via 10.1.4.2" >> /etc/sysconfig/network-scripts/route-eth1

reboot the machine to make sure that the interface is configured properly.

also verify that the routes are in place like so:

netstat -rn | grep 10.1.4.2
10.1.5.0 10.1.4.2 255.255.255.0 UG 40 0 0 eth1
10.1.6.0 10.1.4.2 255.255.255.0 UG 40 0 0 eth1

9. make sure the hostname on the system is properly set - in the form virt4.johncompanies.com - and make sure it is set properly:
cat /etc/sysconfig/network

10. Check/update fstab

vi /etc/fstab

/etc/fstab entries should look something like (note noatime option):

/dev/sda3 /vz ext3 defaults,noatime 1 2
/dev/sdb1 /vz1 ext3 defaults,noatime 1 2

make sure swap is configured in fstab

Use real devices instead of LABEL

11. Install Virtuozzo 3.0

If it wasn’t done at install time:

setenforce 0

umount /vz
umount /vz1

mke2fs -i1024 -b1024 -j /dev/sda3 &
mke2fs -i1024 -b1024 -j /dev/sdb2 &

On CentOS 4:
yum install rpm-build; yum update iproute

mount -a

scp backup2:/d4/vzrpms/iso/Rel300.tar.gz /vz/.

cd /vz

tar xzf Rel300.tar.gz

cd Rel300/HW

./install

Choose Enterprise kernel
Vzagent password: k0zub1k
DON’T reboot

vzup2date
User:
gdawson
Pass:
qjdK1R
(run again. till no more updates offered)

reboot

Run vzup2date (make sure to choose enterprise kernel) and/or install latest kernel:

ftp downloads.swsoft.com
cd /updates/Rel300/20061206
bin
get vzkernel-enterprise-2.6.9-023stab033.9.i686.rpm
get vzmodules-enterprise-2.6.9-023stab033.9.i686.rpm
bye

rpm -ivh vzkernel-enterprise-2.6.9-023stab033.9.i686.rpm \
vzmodules-enterprise-2.6.9-023stab033.9.i686.rpm

vi /boot/grub/menu.lst
(set default kernel to new kernel)

12. Install various RPMs

scp backup2:/mnt/data4/linuxrpms/* /tmp

install them in this order:

12a. http fetcher

rpm -ivh /tmp/http_fetcher-1.1.0-1.i386.rpm
rpm -ivh /tmp/fetch-1.0-1.i386.rpm

(ignore error on CentOS4)

Now the fetch command, which vemake calls to email the new system owner, will work (it downloads the welcome email from www.johncompanies.com

12b. ntp client

sometimes ntp isn’t included in which case:

FCx:

rpm -ivh /tmp/libcap-1.10-22.i386.rpm
rpm -ivh /tmp/compat-readline43-4.3-2.i386.rpm
rpm -ivh /tmp/ntp-4.1.2-5.i386.rpm

CentOS4:

rpm -ivh /tmp/ntp-4.1.2-5.i386.rpm

rm -f /etc/ntp.conf

Create a new file with one line:

echo "server 10.1.4.5" > /etc/ntp.conf

Then set ntpd so it will start at boot:

chkconfig --level 3 ntpd on
service ntpd start

Make sure it’s talking to the ntp server:

ntpq -p

Correct output should have something unfer the refid col:

remote refid st t when poll reach delay offset jitter
==============================================================================
10.1.4.105 clock1.redhat.c 2 u - 64 1 0.370 -706677 0.008

12c. adaptec machines only: install aaccli for raid controller (LSI card util is with bin files):

rpm -ivh /tmp/aacapps-4.1-0.i386.rpm

May have to run:

cd /dev
./MAKEDEV.aac aac0

13. copy over binaries/scripts from backup2

scp backup2:/mnt/data4/bin/linux/* /usr/local/sbin/

14. setup root crontab (alter backup time):
crontab -e
1 1 * * * /usr/local/sbin/backup
*/15 * * * * /usr/local/sbin/vim.watchdog.30 > /dev/null
#56 4,16 * * * root /usr/sbin/vzkeyupdate -c

On Perc5/i machines:

*/5 * * * * /usr/local/sbin/perc5iraidchk

15. setup backups

echo '#!/bin/sh
backupdir=/mnt/data4/virt19

## ENTRY /etc
## ENTRY /usr/local/sbin
## ENTRY /root/logs
' > /usr/local/sbin/backup.config

16. setup bb

edit passwd and add the bb user:
echo "bb:x:1984:1984:Big Brother:/home/bb:/bin/bash" >> /etc/passwd

edit group and add the bb group:
echo "bb:x:1984:" >> /etc/group

rebuild the password database:

pwconv

Create bb’s home dir and set ownership:

mkdir /home/bb
chown bb.bb /home/bb

Go to bb’s home and fetch the client tarball:

cd ~bb
scp backup2:/mnt/data4/build/bb/bb-linux.tar .

tar xf bb-linux.tar

cd bbc1.9e-btf/etc/

echo "10.1.4.5 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh" > /home/bb/bbc1.9e-btf/etc/bb-hosts
echo "10.1.4.69 virt19.johncompanies.com # ssh" >> /home/bb/bbc1.9e-btf/etc/bb-hosts

chmod +r /var/log/messages

cd ..
su bb
./runbb.sh start
exit

Add this line to /etc/rc.d/rc.local so bb will start at boot:

echo "su - bb -c \"cd /home/bb/bbc1.9e-btf; ./runbb.sh start\"" >> /etc/rc.d/rc.local

17. Uncomment/add lines with the appropriate IP addresses in /etc/ssh/sshd_config:

vi /etc/ssh/sshd_config
ListenAddress 69.55.232.162
ListenAddress 10.1.4.66

And make sure to uncomment
Protocol 2,1

18. setup backup2 nfs mounts

setup portmap to start with server, and start it:

chkconfig --level 3 portmap on
service portmap start

mb mount

run df to confirm that nfs mounts are mounted

19. m ake alias for /vzconf
ln -s /etc/sysconfig/vz-scripts /vzconf

20. copy over the /etc/sysconfig/vz-scripts/4unlimited template file from the last virt server and put it in /etc/sysconfig/vz-scripts

scp 10.1.4.68:/etc/sysconfig/vz-scripts/4unlimited /etc/sysconfig/vz-scripts/

NOTE: if this system is more powerful (RAM, CPU) then the previous, you will want to create a new one using vzsplit:

cd /etc/sysconfig/vz-scripts/
vzsplit -f 4unlimited

21. edit /etc/sysconfig/vz

vi /etc/sysconfig/vz

enable fastboot:
VZFASTBOOT=yes

add/edit to traffic parameters section:
TRAFFIC_SHAPING=yes
BANDWIDTH="eth0:102400"
TOTALRATE="eth0:1:102400"

for 2 nics, uncomment this line:
VE_ROUTE_SRC_DEV="eth0"

Make sure the IPTABLES section looks like:
IPTABLES="ip_tables iptable_filter iptable_mangle ipt_limit ipt_multiport ipt_tos ipt_TOS ipt_REJECT ipt_tcpmss ipt_ttl ipt_LOG ipt_length ip_conntrack ip_conntrack_ftp ip_conntrack_irc ipt_conntrack ipt_state ipt_helper iptable_nat ip_nat_ftp ip_nat_irc ipt_REDIRECT"

22. disable periodic fsck's, lower the root padding on the /vz and /vz1 filesystems to 100MB

tune2fs -i 0 /dev/sda3
tune2fs -i 0 /dev/sdb2
tune2fs -c -1 /dev/sda3
tune2fs -c -1 /dev/sdb2
tune2fs -r 100000 /dev/sda3
tune2fs -r 100000 /dev/sdb2

23. copy over our versions of the vzpp files:

/vz/private/1/root/var/vzcp/static/vz/skins/
/vz/private/1/root/etc/vzcp/pp/menu.xml
/vz/private/1/root/etc/vzcp/pp/dashboard.xml
/vz/private/1/root/etc/vzcp/vzcpcon.conf

eg (2.6.1 or 3.0):
rsync -va -e ssh backup2:/d4/build/vzcp/var/vzcp/static/vz/skins/ /vz/private/1/root/var/vzcp/static/vz/skins/
rsync -va -e ssh backup2:/d4/build/vzcp/etc/vzcp/pp/menu.xml /vz/private/1/root/etc/vzcp/pp/menu.xml
rsync -va -e ssh backup2:/d4/build/vzcp/etc/vzcp/pp/dashboard.xml /vz/private/1/root/etc/vzcp/pp/dashboard.xml
rsync -va -e ssh backup2:/d4/build/vzcp/etc/vzcp/vzcpcon.conf /vz/private/1/root/etc/vzcp/vzcpcon.conf

To disable backups from scratch:

In svcve edit /etc/vzcp/vzcpcon.conf

In section:

<auth>
<type>VEUser</type>
<user>root</user>

Add into/below:
<cred>
<id>maintenance</id>
</cred>

The following:
<cred>
<id>show</id>
</cred>
<cred>
<id>reinstall</id>
</cred>
<cred>
<id>repair_mode</id>
</cred>

Will finally look like:

<cred>
<id>maintenance</id>
<cred>
<id>show</id>
</cred>
<cred>
<id>reinstall</id>
</cred>
<cred>
<id>repair_mode</id>
</cred>
</cred>
</cred>

24. template install

mb mount

install debian 3.0 templates

cd /backup4/vzrpms/distro/debian/3.0/p1/
for f in `cat /backup4/vzrpms/distro/debian/3.0/install_list`; do rpm -ivh $f; done

install debian 3.1 templates

cd /backup4/vzrpms/distro/debian/3.1/p3/
for f in `ls`; do rpm -ivh $f; done

install redhat templates

cd /backup4/vzrpms/distro/rh9/current/
for f in `cat /backup4/vzrpms/distro/rh9/install_list`; do rpm -ivh $f; done

install fedora1 templates

cd /backup4/vzrpms/distro/fc1/current/
for f in `cat /backup4/vzrpms/distro/fc1/install_list`; do rpm -ivh $f; done

change 'krb5-devel-1.3.4-5' to 'krb5-devel-1.3.3-6' in (applies to pre-p5)
/vz/template/fedora-core-devel-1/conf/fedora-core-devel-1.conf.20050314

install fedora2 templates

cd /backup4/vzrpms/distro/fc2/current/
for f in `cat /backup4/vzrpms/distro/fc2/install_list`; do rpm -ivh $f; done

install suse9.2 templates

cd /backup4/vzrpms/distro/suse/current/
for f in `cat /backup4/vzrpms/distro/suse/install_list`; do rpm -ivh $f; done

install centos templates

cd /backup4/vzrpms/distro/centos/current/
for f in `cat /backup4/vzrpms/distro/centos/install_list`; do rpm -ivh $f; done

install fedora4 templates

cd /backup4/vzrpms/distro/fc4/current/
for f in `cat /backup4/vzrpms/distro/fc2/install_list`; do rpm -ivh $f; done

vzpkgcache

To pull templates from another machine (before hand stop ve 1 and move it’s template so you preserve the latest template files just setup by vzup2date):

rsync -a -e ssh 10.1.4.68:/vz/template/ /vz/template/

Make sure ubuntu source is defined:

UBU_SERVER=http://archive.ubuntu.com

And change FC source:
FC_SERVER=http://rhold.fedoraproject.org

In /etc/vztt/vztt.conf

rsync -a -e ssh 10.1.4.68:/etc/vztt/vztt.conf /etc/vztt/

To update the cache:

vzpkg update cache ubuntu-6.10-x86

25. add tun device support

Add to /etc/modules.conf:
echo "alias char-major-10-200 tun" >> /etc/modules.conf

depmod -a

echo "modprobe tun" >> /etc/rc.local

26. add traffic shaping support

Re-define class 0:

vi /etc/sysconfig/vz-scripts/networks_classes
0 69.55.224.0/20
0 10.1.4.0/24

make sure 1 is defined (0.0.0.0) and no other classes are needed

Then start it

/etc/init.d/vz shaperrestart

27. RAIDmon (only applies to LSI 320 PCI card)

cd /tmp
cp /backup4/build/megaraid/linuxmon30.tgz .
tar xzf linuxmon30.tgz
cd linuxmon30
./install
(say no)
mv –f raidmon.init /etc/init.d/raidmon
service raidmon start
tail -f /var/log/messages
(check for output)

28. udp dos fix

echo "/usr/local/sbin/iptables.sh" >> /etc/rc.d/rc.local

29. make rc.local start before vz

mv /etc/rc.d/rc3.d/S99local /etc/rc.d/rc3.d/S95local

30. disable unwanted services/daemons

On CentOS:
chkconfig --del kudzu
chkconfig --del cpuspeed

This should be taken care of by vz install so probably safe to skip

chkconfig --del kudzu
chkconfig --del cups
chkconfig --del gpm
chkconfig --del isdn
chkconfig --del pcmcia
chkconfig --del apmd
chkconfig --del rhnsd
chkconfig --del mdmonitor

31. load mrtg graphs

On FCx xinetd not installed so:

rpm -ivh /tmp/xinetd-2.3.11-1.7x.i386.rpm

create /etc/xinetd.d/load with the following contents:

echo "service load
{
disable = no
flags = REUSE
socket_type = stream
wait = no
user = root
server = /usr/local/sbin/load.pl
}" > /etc/xinetd.d/load

Add this line to /etc/services:
echo "load 12384/tcp" >> /etc/services

On FCx: if xinetd isn’t running, configure it for startup at boot time:

chkconfig --level 3 xinetd on
service xinetd start

32. Tasks to do on backup2:

setup backup dirs:
mkdir -p /mnt/data4/virt19/0

add the system to
vi /mnt/data4/bin/rsync_houseclean

33. Tasks to do on mail:

Update backup graph:
vi /usr/local/www/mgmt/cgi/backupgraph.pl
(add hostname)

add to bbhosts:
vi /usr/home/bb/bbsrc/bb1.9e-btf/etc/bb-hosts
su bb
cd
bbsrc/bb/runbb.sh restart ; exit

add to load graph config:
vi /usr/local/www/mgmt/mrtg/mrtg1.cfg
(add new section for new machine)

34. add to management db (on mail and devweb) jc.ref_machines and jc.ref_templates

insert into ref_machines values (null,'virt19','virt19.johncompanies.com',0,'l');
select * from ref_machines;
insert into systems values ('', 'col00001',43,null,null,100,'virt19.johncompanies.com',null,'2008-01-07',null,null,1,0,null,null,null);
select * from systems where hostname='virt19.johncompanies.com';
update ipmap set cid='col00001', sysid=3772, assign_date='2008-01-07', status=1 where ip='69.55.236.2';
update ipmap set cid='col00001', sysid=3772, assign_date='2008-01-07', status=1 where ip='69.55.230.109';
insert into ref_templates values ('','fedora-core-1/20050725',33,'Fedora Core 1',1);
insert into ref_templates values ('','fedora-core-2/20050725',33,'Fedora Core 2',1);
insert into ref_templates values ('','debian-3.1/20050927',33,'Debian 3.1',1);

35. Make sure that ns1c has proper forward and reverse entries for the new system (both private and public addresses), so that new system emailing will work.
vr johncompanies.com
vi internal.johncompanies.com
rndc reload johncompanies.com IN private
(edit the PTR too)

36. add an outside blocking rule to the firewall, so this machine can only be reached from inside the firewall.

Follow example already in firewall virt19 is:
ipfw add 00149 allow ip from { 69.55.225.225 or 69.55.230.2 or 69.55.230.10 or 69.55.230.3 or 62.213.72.101 or 64.131.90.27 or 80.89.128.178 or 82.200.65.190 or 84.237.120.254 or 89.251.97.2 or 195.214.232.10 or 195.214.233.10 or 208.251.212.1 or 213.130.28.26 } to 69.55.236.2
ipfw add 00149 deny ip from any to 69.55.236.2

virt12 would be 00142

37. add to server/cabinet map

39. make /vz1/private
mkdir /vz1/private ; chmod 0755 /vz1/private
mkdir /vz1/root ; chmod 0755 /vz1/root</pre>

Virtuozzo Server Install

2013-11-14T22:09:26Z

24.165.90.109: /* Configure system, install virtuozzo */

= 4.7 =

Server-specific changes in <span style="color:red">red</span>

http://www.parallels.com/download/pvc/lin

http://www.parallels.com/support/virtualization-suite/pvc/

== BIOS ==

* enable post info to console

2950:<br>
Serial comm.: on with cons. Redir via com1<br>
Failsafe: 115200<br>
Redir after boot: enabled

* Tweak DRAC settings (see [[DRAC/RMM#DRAC_setup|DRAC setup]])

== Install OS ==

CentOS6.x (get latest from http://mirrors.kernel.org/centos/6.4/isos/x86_64/CentOS-6.4-x86_64-netinstall.iso)

May have to disable DRAC virtual media to make sure we get sda as our first device

Set partitions:
<pre>Disk 1:
/ 10240M (ext4)
9216M swap
/vz (the rest) (ext4)
Disk 2:
9216M Swap
/vz1 (the rest) (ext4)</pre>

Basic server

Use grub, install to MBR

== Configure system, install virtuozzo ==
* add ssh

yum install openssh-clients

* create ssh key, add swsoft key to authorized_keys

(enter to everything)

<pre>cd /root/
ssh-keygen -t dsa

Add vz support key:
echo "ssh-dss AAAAB3NzaC1kc3MAAACBAIB7XHmpcsa1D/vsKS+mzEvBVo7udew/aRahbjnjywL4YXT7ruWzpGo7R3UA8GKIgVHoQGUbuXrJz5ICtBEXOlbgXVEz7Bu5nXOG2C9i/8pmFtUw6GwACII8uZE7t/Zq5CRj+5zN1Rv7n/4X4RAjzk5BHXoZ62a32FTbS4tPvyPrAAAAFQDiUi7YEiffNzGvB544lQpRdn29TwAAAIBkUchLTBE1+sX11iC80yhNh9EGeCMLdkT8UqHsTSaD0myQPxGALuNY4rO31q+25uKqSRDTXd6ZWXp4bplA6Ls6jl7r7XVR9loJ7WvRMqc4Dv68TZ7h2UBqAhu4det2UF4LHim3DNf1gKn9DzWNkmI/N2FvVVAL4aM3sPunVIvUYgAAAIAJX38INABVgnCEBKqpv5FMLm3k9MgMn54oQFmylBbgM0w4SUeuDj3jLrkJf8dT9w8ByEDb8xviexpWrPsbz2DeThle4xgzcyKrUkt2Wugtmm4DwS0S/68BAA9wXoWD/vZ67GBoCKH7uq8VAjUVcCbJ/7IydljnFAZoJpktdwN7iQ== russup@support.sw-soft.com" >> /root/.ssh/authorized_keys</pre>

* setup access to backup2:

<pre>echo "10.1.4.3 backup2" >> /etc/hosts
echo "10.1.4.8 backup1" >> /etc/hosts</pre>

# Copy id_dsa.pub to backup2 and add it to the end of ~root/.ssh/authorized_keys:

<pre>cat .ssh/id_dsa.pub | ssh backup2 'cat - >> /root/.ssh/authorized_keys'
cat .ssh/id_dsa.pub | ssh backup1 'cat - >> /root/.ssh/authorized_keys'</pre>

<pre>ssh backup2 hostname
ssh backup1 hostname</pre>
(confirm you can connect w/o entering a password)

* add bash backward history search
<pre>echo "\"\M-z\": history-search-backward
\"\M-x\": history-search-backward " > /root/.inputrc
echo "export GREP_OPTIONS='--color=auto'" >> ~/.bashrc
echo "\"\e[5~\": history-search-backward" >> ~/.inputrc
echo "\"\e[6~\": history-search-forward" >> ~/.inputrc</pre>

* add to /root/.bashrc, create log dir

<pre>vi /root/.bashrc
alias v='vzctl'
alias mbm='mb mount'
alias mbu='mb umount'
export PS1="[\u@\h \w]# "
alias cv='cd /vz/private'
alias cv1='cd /vz1/private'
alias cv2='cd /vz2/private'
alias pg='ps auxw|grep $1'</pre>

source /root/.bashrc

mkdir /root/logs

* make sure serial console and kernel messages are working

add these 2 lines to the top:

<pre>vi /boot/grub/menu.lst
serial --unit=0 --speed= 115200
terminal --timeout=10 serial console</pre>

and the kernel line should look like:

kernel /boot/vmlinuz-2.4.20-020stab009.24.777-enterprise ro root=/dev/sda1 console=tty0 console=ttyS0,115200

Remove or comment out these lines:
<pre>hiddensplashmenu
splashimage=(hd0,0)/boot/grub/splash.xpm.gz</pre>

add to /etc/inittab and /etc/securetty

<pre>echo "s0:2345:respawn:/sbin/agetty -L ttyS0 115200 linux" >> /etc/inittab
echo "ttyS0" >> /etc/securetty</pre>

kill -1 1

* make sure the hostname on the system is properly set - in the form virt4.johncompanies.com - and make sure it is set properly:
cat /etc/sysconfig/network

* Check/update fstab

vi /etc/fstab

/etc/fstab entries should look something like (note noatime option):

<pre>/dev/sda3 /vz ext4 defaults,noatime 1 2
/dev/sdb2 /vz1 ext4 defaults,noatime 1 2</pre>

make sure swap is configured in fstab

Use real devices instead of LABEL

* Install Virtuozzo 4.7

Have a license on hand - request a day ahead of the install. Go to:
http://www.parallels.com/download/pvc47/

<pre>yum install compat-libstdc++-33
yum install wget
yum install ftp
yum install pciutils
yum install ed bc

cd /vz
wget http://download.parallels.com/pvc/47/lin/vzinstall-linux-x86_64.bin
chmod 0700 vzinstall-linux-x86_64.bin
./vzinstall-linux-x86_64.bin</pre>

download and install

change wd's to /vz

Ignore OS version error

Custom install (don’t install templates)

Install PVA Agent
Install PVA Management Node

reboot

Get on KVM to watch PVA install (won't happen on console)

vzup2date

(run again. till no more updates offered)

reboot

<DEPRECATED>
Run vzup2date (make sure to choose enterprise kernel) and/or install latest kernel:

NOTE: skip over the manual kernel load below if you can get the 16G recognized before running vzup2date- then it will dl the ent kernel

NOTE: go to /vz/vzup2date/virtuozzo/linux/i386/4.0.0/CU-2.6.18-028stab068.3 and install kern there (downloaded by vzup2date)

vi /boot/grub/menu.lst
(set default kernel to new kernel)
</DEPRECATED>

* Install ntp

yum install ntp
(if not already installed)

rm -f /etc/ntp.conf

Create a new file with one line:

echo "server 10.1.4.5" > /etc/ntp.conf

Then set ntpd so it will start at boot:

chkconfig --level 3 ntpd on
service ntpd start

Make sure it’s talking to the ntp server:

ntpq -p

Correct output should have something unfer the refid col and a '+' next to the remote server:

<pre>remote refid st t when poll reach delay offset jitter
==============================================================================
10.1.4.105+ clock1.redhat.c 2 u - 64 1 0.370 -706677 0.008</pre>

* copy over binaries/scripts from backup2

scp backup2:/mnt/data4/bin/linux/* /usr/local/sbin/
scp <span style="color:red">10.1.4.62</span>:"/vzconf/jctmpl*" /etc/sysconfig/vz-scripts/
scp <span style="color:red">10.1.4.62</span>:/vzconf/ve-slm* /etc/sysconfig/vz-scripts/

* setup root crontab (alter backup time):
<pre>crontab -e
1 1 * * * /usr/local/sbin/backup
*/15 * * * * /usr/local/sbin/vim.watchdog.30 > /dev/null
#56 4,16 * * * root /usr/sbin/vzkeyupdate -c
*/5 * * * * /usr/local/sbin/perc5iraidchk64</pre>

* setup backups

<pre>
echo '#!/bin/sh
<tt>backupdir=/data/<span style="color:red">virt13</span> </tt>

server=backup1

## ENTRY /etc
## ENTRY /usr/local/sbin
## ENTRY /root/logs
' > /usr/local/sbin/backup.config
</pre>

* setup bigbrother

edit passwd and add the bb user:
echo "bb:x:1984:1984:Big Brother:/home/bb:/bin/bash" >> /etc/passwd

edit group and add the bb group:
echo "bb:x:1984:" >> /etc/group

rebuild the password database:
pwconv

Create bb’s home dir and set ownership:
mkdir /home/bb
chown bb.bb /home/bb

Go to bb’s home and fetch the client tarball:
<pre>cd ~bb
scp backup2:/mnt/data4/build/bb/GOOD-bb-linux-bbc1.9i-btf.tgz .

tar xvf GOOD-bb-linux-bbc1.9i-btf.tgz

yum install glibc-2.12-1.47.el6_2.5.i686

vi etc/bbsys.sh
SORT="/bin/sort"

vi bin/bb-disk.sh
eval "$DFCMD | $EGREP \"$DFUSE\" | $EGREP -v \"$DFEXCLUDE\" | $SORT \"-k${DFSORT}\" > $BBTMP/DFOUT.$$"
wclen1=`eval "$DFCMD | $EGREP \"$DFUSE\" | $EGREP -v \"$DFEXCLUDE\" | $SORT \"-k${DFSORT}\" | $WC"`

cd bbc1.9i-btf/etc/

echo "10.1.4.5 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh" > /home/bb/bbc1.9i-btf/etc/bb-hosts
echo "10.1.4.63 virt13.johncompanies.com # ssh" >> /home/bb/bbc1.9i-btf/etc/bb-hosts
(^^^change to correct host)

chmod +r /var/log/messages

cd ..
su bb
./runbb.sh start
exit</pre>

Add this line to /etc/rc.d/rc.local so bb will start at boot:

echo "su - bb -c \"cd /home/bb/bbc1.9i-btf; ./runbb.sh start\"" >> /etc/rc.d/rc.local

* Uncomment/add lines with the appropriate IP addresses in /etc/ssh/sshd_config:

vi /etc/ssh/sshd_config
<tt>ListenAddress <span style="color:red">69.55.226.2</span>
ListenAddress <span style="color:red">10.1.4.63</span></tt>

<DEPRECATED>
And make sure to uncomment
Protocol 2,1
</DEPRECATED>

* setup backup1/backup2 nfs mounts

<DEPRECATED>
setup portmap to start with server, and start it:

chkconfig --level 3 portmap on
service portmap start
</DEPRECATED>

mb mount

run df to confirm that nfs mounts are mounted

* make alias for /vzconf
ln -s /etc/sysconfig/vz-scripts /vzconf

* edit /etc/sysconfig/vz

vi /etc/sysconfig/vz

calc cpu to reserver for HN. take 5% of power:
vzcpucheck
Current CPU utilization: 27000
Power of the node: 1018320

so in this case:
VE0CPUUNITS=50916

enable fastboot:
VZFASTBOOT=yes

Disable auto mount
#VZMOUNTS="/vz"

Enable slm:
SLM=yes

<DEPRECATED>
Enable IPV6
IPV6=yes
</DEPRECATED>

add/edit to traffic parameters section:
TRAFFIC_SHAPING=yes
BANDWIDTH="eth0:102400"
TOTALRATE="eth0:1:102400"
RATE="eth0:1:50000"

for 2 nics, uncomment this line:
VE_ROUTE_SRC_DEV="eth0"

Make sure the IPTABLES section looks like:
<pre>IPTABLES="ip_tables iptable_filter iptable_mangle ipt_limit ipt_multiport ipt_tos ipt_TOS ipt_REJECT ipt_tcpmss ipt_ttl ipt_LOG ipt_length ip_conntrack ip_conntrack_ftp ip_conntrack_irc ipt_conntrack ipt_state ipt_helper iptable_nat ip_nat_ftp ip_nat_irc ipt_REDIRECT ipt_multiport ipt_TCPMSS"</pre>

* disable periodic fsck's, lower the root padding on the /vz and /vz1 filesystems to 100MB
<pre>tune2fs -i 0 /dev/sda3
tune2fs -i 0 /dev/sdb2
tune2fs -c -1 /dev/sda3
tune2fs -c -1 /dev/sdb2
tune2fs -r 100000 /dev/sda3
tune2fs -r 100000 /dev/sdb2</pre>

* setup vzagent with pub IP (to allow migrations from 3.x):
vzctl set 50 --ipdel all --ipadd <span style="color:red">69.55.226.3</span> --userpasswd vzagent0:k0zub1k --save
vzagent_ctl restart

* edit ve config files:

From split of 80 take
<tt>CPUUNITS</tt>

And put into files in /vzconf/base_ct_conf
along with:
<pre>VERSION="2"

ONBOOT="yes"

KMEMSIZE="2147483647:2147483647"
LOCKEDPAGES="2147483647:2147483647"
PRIVVMPAGES="2147483647:2147483647"
SHMPAGES="2147483647:2147483647"
NUMPROC="32567:32567"
VMGUARPAGES="2147483647:2147483647"
OOMGUARPAGES="2147483647:2147483647"
NUMTCPSOCK="2147483647:2147483647"
NUMFLOCK="2147483647:2147483647"
NUMPTY="255:255"
NUMSIGINFO="1024:1024"
TCPSNDBUF="2147483647:2147483647"
TCPRCVBUF="2147483647:2147483647"
OTHERSOCKBUF="2147483647:2147483647"
DGRAMRCVBUF="2147483647:2147483647"
NUMOTHERSOCK="2147483647:2147483647"
DCACHESIZE="2147483647:2147483647"
NUMFILE="2147483647:2147483647"
NUMIPTENT="2147483647:2147483647"

DISKSPACE=""
DISKINODES=""
QUOTATIME="600"
IOPRIO="4"
CPULIMIT="75"
OFFLINE_MANAGEMENT="yes"
ARCH="x86"
PLATFORM="linux"

IP_ADDRESS="0.0.0.0"
NAMESERVER="69.55.225.225 69.55.230.3"
CPUUNITS="12313"
IOLIMIT="10485760"
IOPSLIMIT="100"
</pre>

* customize the control panel:
rm -fr /vz/backups

vi /var/opt/pva/pp/xsl/lib/vt/backups-forms.xsl

<pre>
<span style="font-size:13px; font-weight:bold">NOTE: You cannot create or view backups via this control panel- IT WILL NOT WORK. Please contact linux@johncompanies.com to setup or inquire about backups for your server.</span>

/var/opt/pva/pp/xsl/lib/vt/backups-list.xsl
</pre>

cd /var/opt/pva/pp/static/vz/skins
for f in `ls`; do scp backup2:/d4/build/vzcp/var/vzcp/static/vz/skins/winxp.new/images/logo_main_4.7.gif $f/images/logo_main.gif; done

* template install

To install new templates:

vzup2date -z

Copy over templates from old server. Ex: ubu 8.04 from virt19:

rsync -a -e ssh <span style="color:red">10.1.4.69</span>:/vz/template/ubuntu/8.04 /vz/template/ubuntu/8.04

install ubuntu 8.04 templates (get latest from vz)

<pre>cd /backup4/vzrpms/distro/ubuntu/8.04
rpm -ivh *
vzpkg update cache ubuntu-8.04-x86</pre>

If you install a tempalte from vzup2date it will download an old cache as well. Better to download the OS rpm from VZ site then create cache then use vzup2date to install the application templates.

* add traffic shaping support

Re-define class 0:

<pre>vi /etc/sysconfig/vz-scripts/networks_classes
0 69.55.224.0/20
0 10.1.4.0/24</pre>

make sure 1 is defined (0.0.0.0) and no other classes are needed

Then start it

/etc/init.d/vz shaperrestart

* udp dos fix

echo "/usr/local/sbin/iptables.sh" >> /etc/rc.d/rc.local

* make rc.local start before vz

mv /etc/rc.d/rc3.d/S99local /etc/rc.d/rc3.d/S95local

* disable unwanted services/daemons

Before:
<pre>abrt-ccpp 0:off 1:off 2:off 3:on 4:off 5:on 6:off
abrt-oops 0:off 1:off 2:off 3:on 4:off 5:on 6:off
abrtd 0:off 1:off 2:off 3:on 4:off 5:on 6:off
acpid 0:off 1:off 2:on 3:on 4:on 5:on 6:off
atd 0:off 1:off 2:off 3:on 4:on 5:on 6:off
auditd 0:off 1:off 2:on 3:on 4:on 5:on 6:off
autofs 0:off 1:off 2:off 3:on 4:on 5:on 6:off
cpuspeed 0:off 1:on 2:on 3:on 4:on 5:on 6:off
crond 0:off 1:off 2:on 3:on 4:on 5:on 6:off
cups 0:off 1:off 2:on 3:on 4:on 5:on 6:off
haldaemon 0:off 1:off 2:off 3:on 4:on 5:on 6:off
ip6tables 0:off 1:off 2:on 3:on 4:on 5:on 6:off
iptables 0:off 1:off 2:on 3:on 4:on 5:on 6:off
irqbalance 0:off 1:off 2:off 3:on 4:on 5:on 6:off
kdump 0:off 1:off 2:off 3:on 4:on 5:on 6:off
lvm2-monitor 0:off 1:on 2:on 3:on 4:on 5:on 6:off
mcelogd 0:off 1:off 2:off 3:on 4:off 5:on 6:off
mdmonitor 0:off 1:off 2:on 3:on 4:on 5:on 6:off
messagebus 0:off 1:off 2:on 3:on 4:on 5:on 6:off
netfs 0:off 1:off 2:off 3:on 4:on 5:on 6:off
network 0:off 1:off 2:on 3:on 4:on 5:on 6:off
nfslock 0:off 1:off 2:off 3:on 4:on 5:on 6:off
ntpd 0:off 1:off 2:off 3:on 4:off 5:off 6:off
portreserve 0:off 1:off 2:on 3:on 4:on 5:on 6:off
postfix 0:off 1:off 2:on 3:on 4:on 5:on 6:off
pvaagentd 0:off 1:off 2:on 3:on 4:on 5:on 6:off
pvapp 0:off 1:off 2:on 3:on 4:on 5:on 6:off
qpidd 0:off 1:off 2:on 3:on 4:on 5:on 6:off
rpcbind 0:off 1:off 2:on 3:on 4:on 5:on 6:off
rpcgssd 0:off 1:off 2:off 3:on 4:on 5:on 6:off
rpcidmapd 0:off 1:off 2:off 3:on 4:on 5:on 6:off
rsyslog 0:off 1:off 2:on 3:on 4:on 5:on 6:off
sshd 0:off 1:off 2:on 3:on 4:on 5:on 6:off
sysstat 0:off 1:on 2:on 3:on 4:on 5:on 6:off
udev-post 0:off 1:on 2:on 3:on 4:on 5:on 6:off
vz 0:off 1:off 2:on 3:on 4:on 5:on 6:off
vzlmond 0:off 1:off 2:on 3:on 4:on 5:on 6:off
vznetcfg 0:off 1:off 2:on 3:on 4:on 5:on 6:off
xinetd 0:off 1:off 2:off 3:on 4:on 5:on 6:off</pre>

<pre>chkconfig --level 3 cpuspeed off
chkconfig --level 3 cups off
chkconfig --level 3 haldaemon off
chkconfig --level 3 ip6tables off
chkconfig --level 3 iptables off
chkconfig --level 3 irqbalance off
chkconfig --level 3 lvm2-monitor off
chkconfig --level 3 mdmonitor off
chkconfig --level 3 messagebus off
chkconfig --level 3 netfs off
chkconfig --level 3 nfslock off
chkconfig --level 3 qpidd off
chkconfig --level 3 rpcgssd off
chkconfig --level 3 rpcidmapd off</pre>

* load mrtg graphs

yum install xinetd

create /etc/xinetd.d/load with the following contents:

<pre>echo "service load
{
disable = no
flags = REUSE
socket_type = stream
wait = no
user = root
server = /usr/local/sbin/load.pl
}" > /etc/xinetd.d/load</pre>

Add this line to /etc/services:
echo "load 12384/tcp" >> /etc/services

* Tasks to do on backup1:
setup backup dirs:

mkdir -p /data/<span style="color:red">virt13</span>/0

for f in 0 1 2 3 4 5 6; do mkdir -p /data/<span style="color:red">virt13</span>/$f/vz/private; mkdir -p /data/<span style="color:red">virt13</span>/$f/vz1/private; mkdir -p /data/<span style="color:red">virt13</span>/$f/vz2/private; done

add the system to
vi /usr/local/sbin/snapshot_rotate

* Tasks to do on mail:

Update backup graph:
vi /usr/local/www/mgmt/cgi/backupgraph.pl
(add hostname)

add to bbhosts:
<pre>vi /usr/home/bb/bbsrc/bb1.9i-btf/etc/bb-hosts
su bb
cd
bbsrc/bb/runbb.sh restart ; exit</pre>

add to load graph config:
vi /usr/local/www/mgmt/mrtg/mrtg1.cfg
(add new section for new machine)

* add to management db (on mail and devweb) jc.ref_machines and jc.ref_templates

insert new server:
insert into ref_machines values (null,'virt19','virt19.johncompanies.com',0,'l');

get new machine_id back out:
select * from ref_machines;

insert as a system to jc customer col00001:
insert into systems values ('', 'col00001',30,null,null,100,'virt11.johncompanies.com',null,'2010-03-11',null,null,1,0,null,null,null);

get sysid:
select * from systems where hostname='virt19.johncompanies.com';

update ipmap (main IP and IP used for VZCP):
update ipmap set cid='col00001', sysid=3772, assign_date='2008-01-07', status=1 where ip='69.55.236.2';
update ipmap set cid='col00001', sysid=3772, assign_date='2008-01-07', status=1 where ip='69.55.230.109';

add templates (or do via mgmt):
<pre>
insert into ref_templates values ('','fedora-core-1/20050725',33,'Fedora Core 1',1);
insert into ref_templates values ('','fedora-core-2/20050725',33,'Fedora Core 2',1);
insert into ref_templates values ('','debian-3.1/20050927',33,'Debian 3.1',1);</pre>

* Make sure that ns1c has proper forward and reverse entries for the new system (both private and public addresses), so that new system emailing will work.
<pre>vr johncompanies.com
vi internal.johncompanies.com
rndc reload johncompanies.com IN private</pre>
(edit the PTR too)

* add an outside blocking rule to the firewall, so this machine can only be reached from inside the firewall.

Follow example already in firewall virt19 is:

<pre>ipfw add 00143 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 or 62.213.72.101 or 64.131.90.27 or 82.200.65.190 or 91.204.24.0/22 or 194.44.160.178 or 195.214.232.10 or 195.214.233.10 or 208.251.212.1 } to 69.55.226.2
ipfw add 00143 deny ip from any to 69.55.226.2</pre>

(virt12 would be 00142)

* add to [[Cabinetmap|Cabinetmap]]

* make /vz1/private
<pre>mkdir /vz1/private ; chmod 0755 /vz1/private
mkdir /vz1/root ; chmod 0755 /vz1/root</pre>

* put config files in a place they’ll get backed up
<pre>mkdir /vzconf/confs
crontab -e
0 0 * * * rm -f /vzconf/confs/*.conf; cp /vzconf/*.conf /vzconf/confs</pre>

* fix pass/vzpp and localse issue with ubu 8.10

vi /vz/template/ubuntu/8.10/x86/config/os/default/post-install

sed '/^password.*pam_unix.so.*sha512/ s~sha512~md5~' -i etc/pam.d/common-password

localedef -i en_US -c -f UTF-8 en_US.UTF-8

* add tun device support

echo "modprobe tun" >> /etc/rc.local

* reinstall VZA/CT #1/CT #50

After the install, the PP was broken. So needed to reinstall everything. Did the pva_setup --uninstall and removed CT 50 and then setup the pva per http://kb.parallels.com/113071 then setup the CT 50 per the other article- centos6 had issues - and then authenticated with virt13 to get that into the management list.

http://kb.parallels.com/113072

http://kb.parallels.com/113071

* config logrotate (otherwise it creates weekly empty logs)

vi /etc/logrotate.conf

(add to end)

<pre>/var/log/messages {
rotate 12
monthly
size 10M
missingok
compress
delaycompress
notifempty
sharedscripts
postrotate
echo `date` > /var/log/messages
endscript
}</pre>

* install megacli64 (can we just move the bin and not install the lib?)

<pre>scp backup2: /d4/build/3ware/MegaCli-8.00.40-1.i386.rpm /root/MegaCli-8.00.40-1.i386.rpm
scp backup2: /d4/build/3ware/Lib_Utils-1.00-08.noarch.rpm /root/ Lib_Utils-1.00-08.noarch.rpm

rpm -ivh Lib_Utils-1.00-08.noarch.rpm
rpm -ivh MegaCli-8.00.40-1.i386.rpm

cp opt/MegaRAID/MegaCli/MegaCli64 /usr/local/sbin/megacli64</pre>

= Older versions =

== 4.0 ==

Last updated 2010-03-11

<pre>1. enable post info to console

2950:
Serial comm.: on with cons. Redir via com1
Failsafe: 115200
Redir after boot: enabled

Server
Custom- desel all but base
/root/install.log

2. Install OS

CentOS5.x (get latest from http://mirrors.kernel.org/centos/5.4/isos/i386/)

Set partitions:
Disk 1:
/ 3072M
6144M swap
/vz (the rest)
Disk 2:
6144M Swap

Use grub, install to MBR
No firewall
Disable SElinux
Install only base

3. create ssh key, add swsoft key to authorized_keys

(enter to everything)

cd /root/
ssh-keygen -t dsa

echo "ssh-dss AAAAB3NzaC1kc3MAAACBAIB7XHmpcsa1D/vsKS+mzEvBVo7udew/aRahbjnjywL4YXT7ruWzpGo7R3UA8GKIgVHoQGUbuXrJz5ICtBEXOlbgXVEz7Bu5nXOG2C9i/8pmFtUw6GwACII8uZE7t/Zq5CRj+5zN1Rv7n/4X4RAjzk5BHXoZ62a32FTbS4tPvyPrAAAAFQDiUi7YEiffNzGvB544lQpRdn29TwAAAIBkUchLTBE1+sX11iC80yhNh9EGeCMLdkT8UqHsTSaD0myQPxGALuNY4rO31q+25uKqSRDTXd6ZWXp4bplA6Ls6jl7r7XVR9loJ7WvRMqc4Dv68TZ7h2UBqAhu4det2UF4LHim3DNf1gKn9DzWNkmI/N2FvVVAL4aM3sPunVIvUYgAAAIAJX38INABVgnCEBKqpv5FMLm3k9MgMn54oQFmylBbgM0w4SUeuDj3jLrkJf8dT9w8ByEDb8xviexpWrPsbz2DeThle4xgzcyKrUkt2Wugtmm4DwS0S/68BAA9wXoWD/vZ67GBoCKH7uq8VAjUVcCbJ/7IydljnFAZoJpktdwN7iQ== russup@support.sw-soft.com" >> /root/.ssh/authorized_keys

4. setup access to backup2:

echo "10.1.4.3 backup2" >> /etc/hosts
echo "10.1.4.8 backup1" >> /etc/hosts

# Copy id_dsa.pub to backup2 and add it to the end of ~root/.ssh/authorized_keys:

cat .ssh/id_dsa.pub | ssh backup2 'cat - >> /root/.ssh/authorized_keys'
cat .ssh/id_dsa.pub | ssh backup1 'cat - >> /root/.ssh/authorized_keys'

ssh backup2 hostname
ssh backup1 hostname

(confirm you can connect w/o entering a password)

5. add bash backward history search
echo "\"\M-z\": history-search-backward
\"\M-x\": history-search-backward " > /root/.inputrc
echo "export GREP_OPTIONS='--color=auto'" >> ~/.bashrc
echo "\"\e[5~\": history-search-backward" >> ~/.inputrc
echo "\"\e[6~\": history-search-forward" >> ~/.inputrc

6. add to /root/.bashrc, create log dir

vi /root/.bashrc
alias v='vzctl'
alias mbm='mb mount'
alias mbu='mb umount'
export PS1="[\u@\h \w]# "
alias cv='cd /vz/private'
alias cv1='cd /vz1/private'
alias cv2='cd /vz2/private'

source /root/.bashrc

mkdir /root/logs

7. make sure serial console and kernel messages are working

add these 2 lines to the top:

vi /boot/grub/menu.lst
serial --unit=0 --speed= 115200
terminal --timeout=10 serial console

and the kernel line should look like:

kernel /boot/vmlinuz-2.4.20-020stab009.24.777-enterprise ro root=/dev/sda1 console=tty0 console=ttyS0,115200

Remove or comment out these lines:
hiddensplashmenu
splashimage=(hd0,0)/boot/grub/splash.xpm.gz

add to /etc/inittab and /etc/securetty

echo "s0:2345:respawn:/sbin/agetty -L ttyS0 115200 linux" >> /etc/inittab
echo "ttyS0" >> /etc/securetty

kill -1 1

DEPRECATED 8. configuring eth1 for management net:

echo "10.1.5/24 via 10.1.4.2
10.1.6/24 via 10.1.4.2" >> /etc/sysconfig/network-scripts/route-eth1

reboot the machine to make sure that the interface is configured properly.

also verify that the routes are in place like so:

netstat -rn | grep 10.1.4.2
10.1.5.0 10.1.4.2 255.255.255.0 UG 40 0 0 eth1
10.1.6.0 10.1.4.2 255.255.255.0 UG 40 0 0 eth1

9. make sure the hostname on the system is properly set - in the form virt4.johncompanies.com - and make sure it is set properly:
cat /etc/sysconfig/network

10. Check/update fstab

vi /etc/fstab

/etc/fstab entries should look something like (note noatime option):

/dev/sda3 /vz ext3 defaults,noatime 1 2
/dev/sdb2 /vz1 ext3 defaults,noatime 1 2

make sure swap is configured in fstab

Use real devices instead of LABEL

11. Install Virtuozzo 4.0

umount /vz
umount /vz1

mke2fs -i1024 -b1024 -j /dev/sda3 &
mke2fs -i1024 -b1024 -j /dev/sdb2 &

Experimentally ran this for sdc on virt12:
mke2fs -i2048 -b2048 -j /dev/sdb1

mount -a

scp backup2:/d4/vzrpms/iso/vzinstall-linux.bin /vz
cd /vz
chmod 0700 vzinstall-linux.bin
./vzinstall-linux.bin

change wd to /vz
Ignore OS version error
Custom install (don’t install templates)

reboot

vzup2date

(run again. till no more updates offered)

reboot

Run vzup2date (make sure to choose enterprise kernel) and/or install latest kernel:

NOTE: skip over the manual kernel load below if you can get the 16G recognized before running vzup2date- then it will dl the ent kernel

NOTE: go to /vz/vzup2date/virtuozzo/linux/i386/4.0.0/CU-2.6.18-028stab068.3 and install kern there (downloaded by vzup2date)

<DEPREACATED>
ftp downloads.swsoft.com
User:
gdawson
Pass:
qjdK1R

cd /updates/Rel400/20080307
bin
get vzkernel-ent-2.6.18-028stab053.10.i686.rpm
get vzmodules-ent-2.6.18-028stab053.10.i686.rpm
bye

rpm -ivh vzkernel-ent-2.6.18-028stab053.10.i686.rpm \
vzmodules-ent-2.6.18-028stab053.10.i686.rpm
</DEPREACATED>

vi /boot/grub/menu.lst
(set default kernel to new kernel)

12. Install ntp

yum install ntp

rm -f /etc/ntp.conf

# Create a new file with one line:

echo "server 10.1.4.5" > /etc/ntp.conf

# Then set ntpd so it will start at boot:

chkconfig --level 3 ntpd on
service ntpd start

# Make sure it’s talking to the ntp server:

ntpq -p

Correct output should have * or + to the left of the IP:

remote refid st t when poll reach delay offset jitter
==============================================================================
10.1.4.105 clock1.redhat.c 2 u - 64 1 0.370 -706677 0.008

13. copy over binaries/scripts from backup2

scp backup2:/mnt/data4/bin/linux/* /usr/local/sbin/
scp 10.1.4.62:/vzconf/jctmpl* /etc/sysconfig/vz-scripts/
scp 10.1.4.62:/vzconf/ve-slm* /etc/sysconfig/vz-scripts/

14. setup root crontab (alter backup time):
crontab -e
1 1 * * * /usr/local/sbin/backup
*/15 * * * * /usr/local/sbin/vim.watchdog.30 > /dev/null
#56 4,16 * * * root /usr/sbin/vzkeyupdate -c
*/5 * * * * /usr/local/sbin/perc5iraidchk

15. setup backups

echo '#!/bin/sh
backupdir=/data/virt11
server=backup1

## ENTRY /etc
## ENTRY /usr/local/sbin
## ENTRY /root/logs
' > /usr/local/sbin/backup.config

16. setup bb

edit passwd and add the bb user:
echo "bb:x:1984:1984:Big Brother:/home/bb:/bin/bash" >> /etc/passwd

# edit group and add the bb group:
echo "bb:x:1984:" >> /etc/group

# rebuild the password database:

pwconv

# Create bb’s home dir and set ownership:

mkdir /home/bb
chown bb.bb /home/bb

# Go to bb’s home and fetch the client tarball:

cd ~bb
scp backup2:/mnt/data4/build/bb/bb-linux.tar .

tar xf bb-linux.tar

cd bbc1.9e-btf/etc/

echo "10.1.4.5 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh" > /home/bb/bbc1.9e-btf/etc/bb-hosts
echo "10.1.4.61 virt11.johncompanies.com # ssh" >> /home/bb/bbc1.9e-btf/etc/bb-hosts

chmod +r /var/log/messages

cd ..
su bb
./runbb.sh start
exit

# Add this line to /etc/rc.d/rc.local so bb will start at boot:

echo "su - bb -c \"cd /home/bb/bbc1.9e-btf; ./runbb.sh start\"" >> /etc/rc.d/rc.local

17. Uncomment/add lines with the appropriate IP addresses in /etc/ssh/sshd_config:

vi /etc/ssh/sshd_config
ListenAddress 69.55.238.3
ListenAddress 10.1.4.61

And make sure to uncomment
Protocol 2,1

18. setup backup2 nfs mounts

setup portmap to start with server, and start it:

chkconfig --level 3 portmap on
service portmap start

mb mount

run df to confirm that nfs mounts are mounted

19. make alias for /vzconf
ln -s /etc/sysconfig/vz-scripts /vzconf

DEPRECATED 20. copy over the /etc/sysconfig/vz-scripts/4unlimited template file from the last virt server and put it in /etc/sysconfig/vz-scripts

scp 10.1.4.68:/etc/sysconfig/vz-scripts/4unlimited /etc/sysconfig/vz-scripts/

NOTE: if this system is more powerful (RAM, CPU) then the previous, you will want to create a new one using vzsplit:

cd /etc/sysconfig/vz-scripts/
vzsplit -f 4unlimited

21. edit /etc/sysconfig/vz

vi /etc/sysconfig/vz

enable fastboot:
VZFASTBOOT=yes

Disable auto mount
#VZMOUNTS="/vz"

Enable IPV6
IPV6=yes

add/edit to traffic parameters section:
TRAFFIC_SHAPING=yes
BANDWIDTH="eth0:102400"
TOTALRATE="eth0:1:102400"
RATE="eth0:1:50000"

for 2 nics, uncomment this line:
VE_ROUTE_SRC_DEV="eth0"

Make sure the IPTABLES section looks like:
IPTABLES="ip_tables iptable_filter iptable_mangle ipt_limit ipt_multiport ipt_tos ipt_TOS ipt_REJECT ipt_tcpmss ipt_ttl ipt_LOG ipt_length ip_conntrack ip_conntrack_ftp ip_conntrack_irc ipt_conntrack ipt_state ipt_helper iptable_nat ip_nat_ftp ip_nat_irc ipt_REDIRECT ipt_multiport ipt_TCPMSS"

22. disable periodic fsck's, lower the root padding on the /vz and /vz1 filesystems to 100MB

tune2fs -i 0 /dev/sda3
tune2fs -i 0 /dev/sdb2
tune2fs -c -1 /dev/sda3
tune2fs -c -1 /dev/sdb2
tune2fs -r 100000 /dev/sda3
tune2fs -r 100000 /dev/sdb2

23. setup vzagent with pub IP (to allow migrations from 3.x):

vzctl set 1 --ipdel all --ipadd 69.55.230.57 --userpasswd vzagent0:k0zub1k --save
vzagent_ctl restart

24. edit ve config files:

From split of 80 take
CPUUNITS

And put into files in /usr/local/sbin/ve-slm.1024MB.conf et. al.

25. customize the control panel:

vzmlocal 1:1:/vz/private/1:/vz/root/1
cd /vz/private/1/fs/root/var/vzcp/static/vz/skins
for f in `ls`; do scp backup2:/d4/build/vzcp/var/vzcp/static/vz/skins/winxp.new/images/logo_main.gif $f/images/logo_pim.gif; done

1) comment 'Backup' tab in /etc/vzcp/pp/tabs.xml:

<tab-bar id="maintance" obj-type="env.core.vt">


2) change 'href' parameter if 'Maintentance' item in /etc/vzcp/pp/menu.xml:


<item id="management.maintenance" href="/vz/cp/vzdir/infrman/envs/repair?branch=env-home,vzpp" icon="backup" tip="maintenance_screen">

3) /etc/vzcp/pp/dashboard.xml

<item id="management.maintenanance" href="$baseurl/vzdir/infrman/envs/repair" title="Maintenance" icon="env_backup" tip="maintenance_screen" cred="vzcp.env.backups.list" security-obj-type="ve.vz.vt">

URL for 'href' parameter can be obtained from tabs.xml.

vi /var/vzcp/xsl/screens/vzdir/monitman/traffic.xsl
<sw:text>Traffic for selected period and class (NOTE: this information is not certified by JohnCompanies and is not used to calculate actual usage or charges)</sw:text>

<DEPREACATED>
/vz/private/1/root/var/vzcp/static/vz/skins/
/vz/private/1/root/etc/vzcp/pp/menu.xml
/vz/private/1/root/etc/vzcp/pp/dashboard.xml
pp/tabs.xml
/vz/private/1/root/etc/vzcp/vzcpcon.conf

eg (2.6.1 or 3.0):
rsync -va -e ssh backup2:/d4/build/vzcp/var/vzcp/static/vz/skins/ /vz/private/1/root/var/vzcp/static/vz/skins/
rsync -va -e ssh backup2:/d4/build/vzcp/etc/vzcp/pp/menu.xml /vz/private/1/root/etc/vzcp/pp/menu.xml
rsync -va -e ssh backup2:/d4/build/vzcp/etc/vzcp/pp/dashboard.xml /vz/private/1/root/etc/vzcp/pp/dashboard.xml
rsync -va -e ssh backup2:/d4/build/vzcp/etc/vzcp/vzcpcon.conf /vz/private/1/root/etc/vzcp/vzcpcon.conf
</DEPREACATED>

24. template install

To install new templates:

vzup2date –z

Copy over ubu 8.04 from virt19

rsync -a -e ssh 10.1.4.69:/vz/template/ubuntu/8.04 /vz/template/ubuntu/8.04

install ubuntu 8.04 templates (get latest from vz)

cd /backup4/vzrpms/distro/ubuntu/8.04
rpm -ivh *
vzpkg update cache ubuntu-8.04-x86

26. add traffic shaping support

Re-define class 0:

vi /etc/sysconfig/vz-scripts/networks_classes
0 69.55.224.0/20
0 10.1.4.0/24

make sure 1 is defined (0.0.0.0) and no other classes are needed

Then start it

/etc/init.d/vz shaperrestart

27. udp dos fix

echo "/usr/local/sbin/iptables.sh" >> /etc/rc.d/rc.local

28. make rc.local start before vz

mv /etc/rc.d/rc3.d/S99local /etc/rc.d/rc3.d/S95local

29. disable unwanted services/daemons

chkconfig --level 3 avahi-daemon off
chkconfig --level 3 bluetooth off
chkconfig --level 3 cpuspeed off
chkconfig --level 3 cups off
chkconfig --level 3 firstboot off
chkconfig --level 3 gpm off
chkconfig --level 3 haldaemon off
chkconfig --level 3 hidd off
chkconfig --level 3 ip6tables off
chkconfig --level 3 iptables off
chkconfig --level 3 irqbalance off
chkconfig --level 3 isdn off
chkconfig --level 3 kudzu off
chkconfig --level 3 lvm2-monitor off
chkconfig --level 3 mcstrans off
chkconfig --level 3 mdmonitor off
chkconfig --level 3 messagebus off
chkconfig --level 3 netfs off
chkconfig --level 3 nfslock off
chkconfig --level 3 pcscd off
chkconfig --level 3 rawdevices off
chkconfig --level 3 readahead_early off
chkconfig --level 3 restorecond off
chkconfig --level 3 rpcgssd off
chkconfig --level 3 rpcidmapd off
chkconfig --level 3 smartd off
chkconfig --level 3 yum-updatesd off

30. load mrtg graphs

yum install xinetd

create /etc/xinetd.d/load with the following contents:

echo "service load
{
disable = no
flags = REUSE
socket_type = stream
wait = no
user = root
server = /usr/local/sbin/load.pl
}" > /etc/xinetd.d/load

# Add this line to /etc/services:
echo "load 12384/tcp" >> /etc/services

31. Tasks to do on backup1:

setup backup dirs:
mkdir -p /data/virt11/0

add the system to
vi /usr/local/sbin/snapshot_rotate

32. Tasks to do on mail:

Update backup graph:
vi /usr/local/www/mgmt/cgi/backupgraph.pl
(add hostname)

add to bbhosts:
vi /usr/home/bb/bbsrc/bb1.9e-btf/etc/bb-hosts
su bb
cd
bbsrc/bb/runbb.sh restart ; exit

add to load graph config:
vi /usr/local/www/mgmt/mrtg/mrtg1.cfg
(add new section for new machine)

33. add to management db (on mail and devweb) jc.ref_machines and jc.ref_templates

insert into ref_machines values (null,'virt19','virt19.johncompanies.com',0,'l');
select * from ref_machines;
insert into systems values ('', 'col00001',30,null,null,100,'virt11.johncompanies.com',null,'2010-03-11',null,null,1,0,null,null,null);
select * from systems where hostname='virt19.johncompanies.com';
update ipmap set cid='col00001', sysid=3772, assign_date='2008-01-07', status=1 where ip='69.55.236.2';
update ipmap set cid='col00001', sysid=3772, assign_date='2008-01-07', status=1 where ip='69.55.230.109';
insert into ref_templates values ('','fedora-core-1/20050725',33,'Fedora Core 1',1);
insert into ref_templates values ('','fedora-core-2/20050725',33,'Fedora Core 2',1);
insert into ref_templates values ('','debian-3.1/20050927',33,'Debian 3.1',1);

34. Make sure that ns1c has proper forward and reverse entries for the new system (both private and public addresses), so that new system emailing will work.
vr johncompanies.com
vi internal.johncompanies.com
rndc reload johncompanies.com IN private
(edit the PTR too)

35. add an outside blocking rule to the firewall, so this machine can only be reached from inside the firewall.

Follow example already in firewall virt19 is:

ipfw add 00149 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 or 62.213.72.101 or 64.131.90.27 or 82.200.65.190 or 91.204.24.0/22 or 194.44.160.178 or 195.214.232.10 or 195.214.233.10 or 208.251.212.1 } to 69.55.236.2
ipfw add 00149 deny ip from any to 69.55.236.2

virt12 would be 00142

36. add to server/cabinet map

37. make /vz1/private
mkdir /vz1/private ; chmod 0755 /vz1/private
mkdir /vz1/root ; chmod 0755 /vz1/root

38. put config files in a place they’ll get backed up
mkdir /vzconf/confs
crontab -e
0 0 * * * rm -f /vzconf/confs/*.conf; cp /vzconf/*.conf /vzconf/confs

39. fix pass/vzpp and localse issue with ubu 8.10

vi /vz/template/ubuntu/8.10/x86/config/os/default/post-install

sed '/^password.*pam_unix.so.*sha512/ s~sha512~md5~' -i etc/pam.d/common-password

localedef -i en_US -c -f UTF-8 en_US.UTF-8

40. add tun device support

echo "modprobe tun" >> /etc/rc.local</pre>

== 3.x ==

Last updated 2007-04-5

<pre>1. enable post info to console
Supermicro:
Console redirection:
Com port addr: on-board COM A
Baud: 38400
Console type: vt100
Flow control: none
Console connection: direct
Continue cr after post: off

2450:
Make sure running bios A09
Console Redirection: VT100/VT220

2950:
Serial comm.: on with cons. Redir via com1
Failsafe: 115200
Redir after boot: enabled

6144
Server
Custom- desel all but base
/root/install.log

2. Install OS
Set partitions:
/ 3072M
Disk 1: Swap 4096M
Disk 2: Swap 4096M
/vz (the rest)

No firewall
Disable SElinux

FC4
Choose a custom install, the “minimal” package

CentOS4 Server CD (only option for 2950)
Choose the “minimal” package

3. create ssh key, add swsoft key to authorized_keys

(enter to everything)

cd /root/
ssh-keygen -t dsa

echo "ssh-dss AAAAB3NzaC1kc3MAAACBAIB7XHmpcsa1D/vsKS+mzEvBVo7udew/aRahbjnjywL4YXT7ruWzpGo7R3UA8GKIgVHoQGUbuXrJz5ICtBEXOlbgXVEz7Bu5nXOG2C9i/8pmFtUw6GwACII8uZE7t/Zq5CRj+5zN1Rv7n/4X4RAjzk5BHXoZ62a32FTbS4tPvyPrAAAAFQDiUi7YEiffNzGvB544lQpRdn29TwAAAIBkUchLTBE1+sX11iC80yhNh9EGeCMLdkT8UqHsTSaD0myQPxGALuNY4rO31q+25uKqSRDTXd6ZWXp4bplA6Ls6jl7r7XVR9loJ7WvRMqc4Dv68TZ7h2UBqAhu4det2UF4LHim3DNf1gKn9DzWNkmI/N2FvVVAL4aM3sPunVIvUYgAAAIAJX38INABVgnCEBKqpv5FMLm3k9MgMn54oQFmylBbgM0w4SUeuDj3jLrkJf8dT9w8ByEDb8xviexpWrPsbz2DeThle4xgzcyKrUkt2Wugtmm4DwS0S/68BAA9wXoWD/vZ67GBoCKH7uq8VAjUVcCbJ/7IydljnFAZoJpktdwN7iQ== russup@support.sw-soft.com" >> /root/.ssh/authorized_keys

4. setup access to backup2:

echo "10.1.4.3 backup2" >> /etc/hosts

Copy id_dsa.pub to backup2 and add it to the end of ~root/.ssh/authorized_keys:

cat .ssh/id_dsa.pub | ssh backup2 'cat - >> /root/.ssh/authorized_keys'

ssh backup2

(confirm you can connect w/o entering a password)

5. add bash backward history search
echo "\"\M-z\": history-search-backward
\"\M-x\": history-search-backward " > /root/.inputrc
echo "export GREP_OPTIONS='--color=auto'" >> ~/.bashrc
echo "\"\e[5~\": history-search-backward" >> ~/.inputrc
echo "\"\e[6~\": history-search-forward" >> ~/.inputrc

6. add to /root/.bashrc, create log dir

vi /root/.bashrc
alias v='vzctl'
alias mbm='mb mount'
alias mbu='mb umount'
export PS1="[\u@\h \w]# "
alias cv='cd /vz/private'
alias cv1='cd /vz1/private'
alias cv2='cd /vz2/private'

mkdir /root/logs

7. make sure serial console and kernel messages are working

grub

add these 2 lines to the top:

vi /boot/grub/menu.lst
serial --unit=0 --speed= 115200
terminal --timeout=10 serial console

and the kernel line should look like:

kernel /boot/vmlinuz-2.4.20-020stab009.24.777-enterprise ro root=/dev/sda1 console=tty0 console=ttyS0,115200

Remove the line:
hiddensplashmenu

lilo

Make sure the append line in /etc/lilo.conf looks like this:

append="console=tty console=ttyS0,38400"

If you edit /etc/lilo.conf don’t forget to run lilo

# lilo
Added linux
Added linux-up
Added linux+virtuozzo *

(note: the last console listed is the one that gets output from the rc scripts.)

add to /etc/inittab and /etc/securetty

echo "s0:2345:respawn:/sbin/agetty -L ttyS0 115200 linux" >> /etc/inittab
echo "ttyS0" >> /etc/securetty

kill -1 1

8. configuring eth1 for management net:

echo "10.1.5/24 via 10.1.4.2
10.1.6/24 via 10.1.4.2" >> /etc/sysconfig/network-scripts/route-eth1

reboot the machine to make sure that the interface is configured properly.

also verify that the routes are in place like so:

netstat -rn | grep 10.1.4.2
10.1.5.0 10.1.4.2 255.255.255.0 UG 40 0 0 eth1
10.1.6.0 10.1.4.2 255.255.255.0 UG 40 0 0 eth1

9. make sure the hostname on the system is properly set - in the form virt4.johncompanies.com - and make sure it is set properly:
cat /etc/sysconfig/network

10. Check/update fstab

vi /etc/fstab

/etc/fstab entries should look something like (note noatime option):

/dev/sda3 /vz ext3 defaults,noatime 1 2
/dev/sdb1 /vz1 ext3 defaults,noatime 1 2

make sure swap is configured in fstab

Use real devices instead of LABEL

11. Install Virtuozzo 3.0

If it wasn’t done at install time:

setenforce 0

umount /vz
umount /vz1

mke2fs -i1024 -b1024 -j /dev/sda3 &
mke2fs -i1024 -b1024 -j /dev/sdb2 &

On CentOS 4:
yum install rpm-build; yum update iproute

mount -a

scp backup2:/d4/vzrpms/iso/Rel300.tar.gz /vz/.

cd /vz

tar xzf Rel300.tar.gz

cd Rel300/HW

./install

Choose Enterprise kernel
Vzagent password: k0zub1k
DON’T reboot

vzup2date
User:
gdawson
Pass:
qjdK1R
(run again. till no more updates offered)

reboot

Run vzup2date (make sure to choose enterprise kernel) and/or install latest kernel:

ftp downloads.swsoft.com
cd /updates/Rel300/20061206
bin
get vzkernel-enterprise-2.6.9-023stab033.9.i686.rpm
get vzmodules-enterprise-2.6.9-023stab033.9.i686.rpm
bye

rpm -ivh vzkernel-enterprise-2.6.9-023stab033.9.i686.rpm \
vzmodules-enterprise-2.6.9-023stab033.9.i686.rpm

vi /boot/grub/menu.lst
(set default kernel to new kernel)

12. Install various RPMs

scp backup2:/mnt/data4/linuxrpms/* /tmp

install them in this order:

12a. http fetcher

rpm -ivh /tmp/http_fetcher-1.1.0-1.i386.rpm
rpm -ivh /tmp/fetch-1.0-1.i386.rpm

(ignore error on CentOS4)

Now the fetch command, which vemake calls to email the new system owner, will work (it downloads the welcome email from www.johncompanies.com

12b. ntp client

sometimes ntp isn’t included in which case:

FCx:

rpm -ivh /tmp/libcap-1.10-22.i386.rpm
rpm -ivh /tmp/compat-readline43-4.3-2.i386.rpm
rpm -ivh /tmp/ntp-4.1.2-5.i386.rpm

CentOS4:

rpm -ivh /tmp/ntp-4.1.2-5.i386.rpm

rm -f /etc/ntp.conf

Create a new file with one line:

echo "server 10.1.4.5" > /etc/ntp.conf

Then set ntpd so it will start at boot:

chkconfig --level 3 ntpd on
service ntpd start

Make sure it’s talking to the ntp server:

ntpq -p

Correct output should have something unfer the refid col:

remote refid st t when poll reach delay offset jitter
==============================================================================
10.1.4.105 clock1.redhat.c 2 u - 64 1 0.370 -706677 0.008

12c. adaptec machines only: install aaccli for raid controller (LSI card util is with bin files):

rpm -ivh /tmp/aacapps-4.1-0.i386.rpm

May have to run:

cd /dev
./MAKEDEV.aac aac0

13. copy over binaries/scripts from backup2

scp backup2:/mnt/data4/bin/linux/* /usr/local/sbin/

14. setup root crontab (alter backup time):
crontab -e
1 1 * * * /usr/local/sbin/backup
*/15 * * * * /usr/local/sbin/vim.watchdog.30 > /dev/null
#56 4,16 * * * root /usr/sbin/vzkeyupdate -c

On Perc5/i machines:

*/5 * * * * /usr/local/sbin/perc5iraidchk

15. setup backups

echo '#!/bin/sh
backupdir=/mnt/data4/virt19

## ENTRY /etc
## ENTRY /usr/local/sbin
## ENTRY /root/logs
' > /usr/local/sbin/backup.config

16. setup bb

edit passwd and add the bb user:
echo "bb:x:1984:1984:Big Brother:/home/bb:/bin/bash" >> /etc/passwd

edit group and add the bb group:
echo "bb:x:1984:" >> /etc/group

rebuild the password database:

pwconv

Create bb’s home dir and set ownership:

mkdir /home/bb
chown bb.bb /home/bb

Go to bb’s home and fetch the client tarball:

cd ~bb
scp backup2:/mnt/data4/build/bb/bb-linux.tar .

tar xf bb-linux.tar

cd bbc1.9e-btf/etc/

echo "10.1.4.5 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh" > /home/bb/bbc1.9e-btf/etc/bb-hosts
echo "10.1.4.69 virt19.johncompanies.com # ssh" >> /home/bb/bbc1.9e-btf/etc/bb-hosts

chmod +r /var/log/messages

cd ..
su bb
./runbb.sh start
exit

Add this line to /etc/rc.d/rc.local so bb will start at boot:

echo "su - bb -c \"cd /home/bb/bbc1.9e-btf; ./runbb.sh start\"" >> /etc/rc.d/rc.local

17. Uncomment/add lines with the appropriate IP addresses in /etc/ssh/sshd_config:

vi /etc/ssh/sshd_config
ListenAddress 69.55.232.162
ListenAddress 10.1.4.66

And make sure to uncomment
Protocol 2,1

18. setup backup2 nfs mounts

setup portmap to start with server, and start it:

chkconfig --level 3 portmap on
service portmap start

mb mount

run df to confirm that nfs mounts are mounted

19. m ake alias for /vzconf
ln -s /etc/sysconfig/vz-scripts /vzconf

20. copy over the /etc/sysconfig/vz-scripts/4unlimited template file from the last virt server and put it in /etc/sysconfig/vz-scripts

scp 10.1.4.68:/etc/sysconfig/vz-scripts/4unlimited /etc/sysconfig/vz-scripts/

NOTE: if this system is more powerful (RAM, CPU) then the previous, you will want to create a new one using vzsplit:

cd /etc/sysconfig/vz-scripts/
vzsplit -f 4unlimited

21. edit /etc/sysconfig/vz

vi /etc/sysconfig/vz

enable fastboot:
VZFASTBOOT=yes

add/edit to traffic parameters section:
TRAFFIC_SHAPING=yes
BANDWIDTH="eth0:102400"
TOTALRATE="eth0:1:102400"

for 2 nics, uncomment this line:
VE_ROUTE_SRC_DEV="eth0"

Make sure the IPTABLES section looks like:
IPTABLES="ip_tables iptable_filter iptable_mangle ipt_limit ipt_multiport ipt_tos ipt_TOS ipt_REJECT ipt_tcpmss ipt_ttl ipt_LOG ipt_length ip_conntrack ip_conntrack_ftp ip_conntrack_irc ipt_conntrack ipt_state ipt_helper iptable_nat ip_nat_ftp ip_nat_irc ipt_REDIRECT"

22. disable periodic fsck's, lower the root padding on the /vz and /vz1 filesystems to 100MB

tune2fs -i 0 /dev/sda3
tune2fs -i 0 /dev/sdb2
tune2fs -c -1 /dev/sda3
tune2fs -c -1 /dev/sdb2
tune2fs -r 100000 /dev/sda3
tune2fs -r 100000 /dev/sdb2

23. copy over our versions of the vzpp files:

/vz/private/1/root/var/vzcp/static/vz/skins/
/vz/private/1/root/etc/vzcp/pp/menu.xml
/vz/private/1/root/etc/vzcp/pp/dashboard.xml
/vz/private/1/root/etc/vzcp/vzcpcon.conf

eg (2.6.1 or 3.0):
rsync -va -e ssh backup2:/d4/build/vzcp/var/vzcp/static/vz/skins/ /vz/private/1/root/var/vzcp/static/vz/skins/
rsync -va -e ssh backup2:/d4/build/vzcp/etc/vzcp/pp/menu.xml /vz/private/1/root/etc/vzcp/pp/menu.xml
rsync -va -e ssh backup2:/d4/build/vzcp/etc/vzcp/pp/dashboard.xml /vz/private/1/root/etc/vzcp/pp/dashboard.xml
rsync -va -e ssh backup2:/d4/build/vzcp/etc/vzcp/vzcpcon.conf /vz/private/1/root/etc/vzcp/vzcpcon.conf

To disable backups from scratch:

In svcve edit /etc/vzcp/vzcpcon.conf

In section:

<auth>
<type>VEUser</type>
<user>root</user>

Add into/below:
<cred>
<id>maintenance</id>
</cred>

The following:
<cred>
<id>show</id>
</cred>
<cred>
<id>reinstall</id>
</cred>
<cred>
<id>repair_mode</id>
</cred>

Will finally look like:

<cred>
<id>maintenance</id>
<cred>
<id>show</id>
</cred>
<cred>
<id>reinstall</id>
</cred>
<cred>
<id>repair_mode</id>
</cred>
</cred>
</cred>

24. template install

mb mount

install debian 3.0 templates

cd /backup4/vzrpms/distro/debian/3.0/p1/
for f in `cat /backup4/vzrpms/distro/debian/3.0/install_list`; do rpm -ivh $f; done

install debian 3.1 templates

cd /backup4/vzrpms/distro/debian/3.1/p3/
for f in `ls`; do rpm -ivh $f; done

install redhat templates

cd /backup4/vzrpms/distro/rh9/current/
for f in `cat /backup4/vzrpms/distro/rh9/install_list`; do rpm -ivh $f; done

install fedora1 templates

cd /backup4/vzrpms/distro/fc1/current/
for f in `cat /backup4/vzrpms/distro/fc1/install_list`; do rpm -ivh $f; done

change 'krb5-devel-1.3.4-5' to 'krb5-devel-1.3.3-6' in (applies to pre-p5)
/vz/template/fedora-core-devel-1/conf/fedora-core-devel-1.conf.20050314

install fedora2 templates

cd /backup4/vzrpms/distro/fc2/current/
for f in `cat /backup4/vzrpms/distro/fc2/install_list`; do rpm -ivh $f; done

install suse9.2 templates

cd /backup4/vzrpms/distro/suse/current/
for f in `cat /backup4/vzrpms/distro/suse/install_list`; do rpm -ivh $f; done

install centos templates

cd /backup4/vzrpms/distro/centos/current/
for f in `cat /backup4/vzrpms/distro/centos/install_list`; do rpm -ivh $f; done

install fedora4 templates

cd /backup4/vzrpms/distro/fc4/current/
for f in `cat /backup4/vzrpms/distro/fc2/install_list`; do rpm -ivh $f; done

vzpkgcache

To pull templates from another machine (before hand stop ve 1 and move it’s template so you preserve the latest template files just setup by vzup2date):

rsync -a -e ssh 10.1.4.68:/vz/template/ /vz/template/

Make sure ubuntu source is defined:

UBU_SERVER=http://archive.ubuntu.com

And change FC source:
FC_SERVER=http://rhold.fedoraproject.org

In /etc/vztt/vztt.conf

rsync -a -e ssh 10.1.4.68:/etc/vztt/vztt.conf /etc/vztt/

To update the cache:

vzpkg update cache ubuntu-6.10-x86

25. add tun device support

Add to /etc/modules.conf:
echo "alias char-major-10-200 tun" >> /etc/modules.conf

depmod -a

echo "modprobe tun" >> /etc/rc.local

26. add traffic shaping support

Re-define class 0:

vi /etc/sysconfig/vz-scripts/networks_classes
0 69.55.224.0/20
0 10.1.4.0/24

make sure 1 is defined (0.0.0.0) and no other classes are needed

Then start it

/etc/init.d/vz shaperrestart

27. RAIDmon (only applies to LSI 320 PCI card)

cd /tmp
cp /backup4/build/megaraid/linuxmon30.tgz .
tar xzf linuxmon30.tgz
cd linuxmon30
./install
(say no)
mv –f raidmon.init /etc/init.d/raidmon
service raidmon start
tail -f /var/log/messages
(check for output)

28. udp dos fix

echo "/usr/local/sbin/iptables.sh" >> /etc/rc.d/rc.local

29. make rc.local start before vz

mv /etc/rc.d/rc3.d/S99local /etc/rc.d/rc3.d/S95local

30. disable unwanted services/daemons

On CentOS:
chkconfig --del kudzu
chkconfig --del cpuspeed

This should be taken care of by vz install so probably safe to skip

chkconfig --del kudzu
chkconfig --del cups
chkconfig --del gpm
chkconfig --del isdn
chkconfig --del pcmcia
chkconfig --del apmd
chkconfig --del rhnsd
chkconfig --del mdmonitor

31. load mrtg graphs

On FCx xinetd not installed so:

rpm -ivh /tmp/xinetd-2.3.11-1.7x.i386.rpm

create /etc/xinetd.d/load with the following contents:

echo "service load
{
disable = no
flags = REUSE
socket_type = stream
wait = no
user = root
server = /usr/local/sbin/load.pl
}" > /etc/xinetd.d/load

Add this line to /etc/services:
echo "load 12384/tcp" >> /etc/services

On FCx: if xinetd isn’t running, configure it for startup at boot time:

chkconfig --level 3 xinetd on
service xinetd start

32. Tasks to do on backup2:

setup backup dirs:
mkdir -p /mnt/data4/virt19/0

add the system to
vi /mnt/data4/bin/rsync_houseclean

33. Tasks to do on mail:

Update backup graph:
vi /usr/local/www/mgmt/cgi/backupgraph.pl
(add hostname)

add to bbhosts:
vi /usr/home/bb/bbsrc/bb1.9e-btf/etc/bb-hosts
su bb
cd
bbsrc/bb/runbb.sh restart ; exit

add to load graph config:
vi /usr/local/www/mgmt/mrtg/mrtg1.cfg
(add new section for new machine)

34. add to management db (on mail and devweb) jc.ref_machines and jc.ref_templates

insert into ref_machines values (null,'virt19','virt19.johncompanies.com',0,'l');
select * from ref_machines;
insert into systems values ('', 'col00001',43,null,null,100,'virt19.johncompanies.com',null,'2008-01-07',null,null,1,0,null,null,null);
select * from systems where hostname='virt19.johncompanies.com';
update ipmap set cid='col00001', sysid=3772, assign_date='2008-01-07', status=1 where ip='69.55.236.2';
update ipmap set cid='col00001', sysid=3772, assign_date='2008-01-07', status=1 where ip='69.55.230.109';
insert into ref_templates values ('','fedora-core-1/20050725',33,'Fedora Core 1',1);
insert into ref_templates values ('','fedora-core-2/20050725',33,'Fedora Core 2',1);
insert into ref_templates values ('','debian-3.1/20050927',33,'Debian 3.1',1);

35. Make sure that ns1c has proper forward and reverse entries for the new system (both private and public addresses), so that new system emailing will work.
vr johncompanies.com
vi internal.johncompanies.com
rndc reload johncompanies.com IN private
(edit the PTR too)

36. add an outside blocking rule to the firewall, so this machine can only be reached from inside the firewall.

Follow example already in firewall virt19 is:
ipfw add 00149 allow ip from { 69.55.225.225 or 69.55.230.2 or 69.55.230.10 or 69.55.230.3 or 62.213.72.101 or 64.131.90.27 or 80.89.128.178 or 82.200.65.190 or 84.237.120.254 or 89.251.97.2 or 195.214.232.10 or 195.214.233.10 or 208.251.212.1 or 213.130.28.26 } to 69.55.236.2
ipfw add 00149 deny ip from any to 69.55.236.2

virt12 would be 00142

37. add to server/cabinet map

39. make /vz1/private
mkdir /vz1/private ; chmod 0755 /vz1/private
mkdir /vz1/root ; chmod 0755 /vz1/root</pre>

Virtuozzo Server Install

2013-11-14T22:05:51Z

24.165.90.109: /* Configure system, install virtuozzo */

= 4.7 =

Server-specific changes in <span style="color:red">red</span>

http://www.parallels.com/download/pvc/lin

http://www.parallels.com/support/virtualization-suite/pvc/

== BIOS ==

* enable post info to console

2950:<br>
Serial comm.: on with cons. Redir via com1<br>
Failsafe: 115200<br>
Redir after boot: enabled

* Tweak DRAC settings (see [[DRAC/RMM#DRAC_setup|DRAC setup]])

== Install OS ==

CentOS6.x (get latest from http://mirrors.kernel.org/centos/6.4/isos/x86_64/CentOS-6.4-x86_64-netinstall.iso)

May have to disable DRAC virtual media to make sure we get sda as our first device

Set partitions:
<pre>Disk 1:
/ 10240M (ext4)
9216M swap
/vz (the rest) (ext4)
Disk 2:
9216M Swap
/vz1 (the rest) (ext4)</pre>

Basic server

Use grub, install to MBR

== Configure system, install virtuozzo ==
* add ssh

yum install openssh-clients

* create ssh key, add swsoft key to authorized_keys

(enter to everything)

<pre>cd /root/
ssh-keygen -t dsa

Add vz support key:
echo "ssh-dss AAAAB3NzaC1kc3MAAACBAIB7XHmpcsa1D/vsKS+mzEvBVo7udew/aRahbjnjywL4YXT7ruWzpGo7R3UA8GKIgVHoQGUbuXrJz5ICtBEXOlbgXVEz7Bu5nXOG2C9i/8pmFtUw6GwACII8uZE7t/Zq5CRj+5zN1Rv7n/4X4RAjzk5BHXoZ62a32FTbS4tPvyPrAAAAFQDiUi7YEiffNzGvB544lQpRdn29TwAAAIBkUchLTBE1+sX11iC80yhNh9EGeCMLdkT8UqHsTSaD0myQPxGALuNY4rO31q+25uKqSRDTXd6ZWXp4bplA6Ls6jl7r7XVR9loJ7WvRMqc4Dv68TZ7h2UBqAhu4det2UF4LHim3DNf1gKn9DzWNkmI/N2FvVVAL4aM3sPunVIvUYgAAAIAJX38INABVgnCEBKqpv5FMLm3k9MgMn54oQFmylBbgM0w4SUeuDj3jLrkJf8dT9w8ByEDb8xviexpWrPsbz2DeThle4xgzcyKrUkt2Wugtmm4DwS0S/68BAA9wXoWD/vZ67GBoCKH7uq8VAjUVcCbJ/7IydljnFAZoJpktdwN7iQ== russup@support.sw-soft.com" >> /root/.ssh/authorized_keys</pre>

* setup access to backup2:

<pre>echo "10.1.4.3 backup2" >> /etc/hosts
echo "10.1.4.8 backup1" >> /etc/hosts</pre>

# Copy id_dsa.pub to backup2 and add it to the end of ~root/.ssh/authorized_keys:

<pre>cat .ssh/id_dsa.pub | ssh backup2 'cat - >> /root/.ssh/authorized_keys'
cat .ssh/id_dsa.pub | ssh backup1 'cat - >> /root/.ssh/authorized_keys'</pre>

<pre>ssh backup2 hostname
ssh backup1 hostname</pre>
(confirm you can connect w/o entering a password)

* add bash backward history search
<pre>echo "\"\M-z\": history-search-backward
\"\M-x\": history-search-backward " > /root/.inputrc
echo "export GREP_OPTIONS='--color=auto'" >> ~/.bashrc
echo "\"\e[5~\": history-search-backward" >> ~/.inputrc
echo "\"\e[6~\": history-search-forward" >> ~/.inputrc</pre>

* add to /root/.bashrc, create log dir

<pre>vi /root/.bashrc
alias v='vzctl'
alias mbm='mb mount'
alias mbu='mb umount'
export PS1="[\u@\h \w]# "
alias cv='cd /vz/private'
alias cv1='cd /vz1/private'
alias cv2='cd /vz2/private'
alias pg='ps auxw|grep $1'</pre>

source /root/.bashrc

mkdir /root/logs

* make sure serial console and kernel messages are working

add these 2 lines to the top:

<pre>vi /boot/grub/menu.lst
serial --unit=0 --speed= 115200
terminal --timeout=10 serial console</pre>

and the kernel line should look like:

kernel /boot/vmlinuz-2.4.20-020stab009.24.777-enterprise ro root=/dev/sda1 console=tty0 console=ttyS0,115200

Remove or comment out these lines:
<pre>hiddensplashmenu
splashimage=(hd0,0)/boot/grub/splash.xpm.gz</pre>

add to /etc/inittab and /etc/securetty

<pre>echo "s0:2345:respawn:/sbin/agetty -L ttyS0 115200 linux" >> /etc/inittab
echo "ttyS0" >> /etc/securetty</pre>

kill -1 1

* make sure the hostname on the system is properly set - in the form virt4.johncompanies.com - and make sure it is set properly:
cat /etc/sysconfig/network

* Check/update fstab

vi /etc/fstab

/etc/fstab entries should look something like (note noatime option):

<pre>/dev/sda3 /vz ext4 defaults,noatime 1 2
/dev/sdb2 /vz1 ext4 defaults,noatime 1 2</pre>

make sure swap is configured in fstab

Use real devices instead of LABEL

* Install Virtuozzo 4.7

Have a license on hand - request a day ahead of the install. Go to:
http://www.parallels.com/download/pvc47/

<pre>yum install compat-libstdc++-33
yum install wget
yum install ftp
yum install pciutils
yum install ed bc

cd /vz
wget http://download.parallels.com/pvc/47/lin/vzinstall-linux-x86_64.bin
chmod 0700 vzinstall-linux-x86_64.bin
./vzinstall-linux-x86_64.bin</pre>

download and install

change wd's to /vz

Ignore OS version error

Custom install (don’t install templates)

Install PVA Agent
Install PVA Management Node

reboot

Get on KVM to watch PVA install (won't happen on console)

vzup2date

(run again. till no more updates offered)

reboot

<DEPRECATED>
Run vzup2date (make sure to choose enterprise kernel) and/or install latest kernel:

NOTE: skip over the manual kernel load below if you can get the 16G recognized before running vzup2date- then it will dl the ent kernel

NOTE: go to /vz/vzup2date/virtuozzo/linux/i386/4.0.0/CU-2.6.18-028stab068.3 and install kern there (downloaded by vzup2date)

vi /boot/grub/menu.lst
(set default kernel to new kernel)
</DEPRECATED>

* Install ntp

yum install ntp
(if not already installed)

rm -f /etc/ntp.conf

Create a new file with one line:

echo "server 10.1.4.5" > /etc/ntp.conf

Then set ntpd so it will start at boot:

chkconfig --level 3 ntpd on
service ntpd start

Make sure it’s talking to the ntp server:

ntpq -p

Correct output should have something unfer the refid col and a '+' next to the remote server:

<pre>remote refid st t when poll reach delay offset jitter
==============================================================================
10.1.4.105+ clock1.redhat.c 2 u - 64 1 0.370 -706677 0.008</pre>

* copy over binaries/scripts from backup2

scp backup2:/mnt/data4/bin/linux/* /usr/local/sbin/
scp <span style="color:red">10.1.4.62</span>:"/vzconf/jctmpl*" /etc/sysconfig/vz-scripts/
scp <span style="color:red">10.1.4.62</span>:/vzconf/ve-slm* /etc/sysconfig/vz-scripts/

* setup root crontab (alter backup time):
<pre>crontab -e
1 1 * * * /usr/local/sbin/backup
*/15 * * * * /usr/local/sbin/vim.watchdog.30 > /dev/null
#56 4,16 * * * root /usr/sbin/vzkeyupdate -c
*/5 * * * * /usr/local/sbin/perc5iraidchk64</pre>

* setup backups

<pre>
echo '#!/bin/sh
</pre>
<tt>backupdir=/data/<span style="color:red">virt13</span>
<pre>
server=backup1

## ENTRY /etc
## ENTRY /usr/local/sbin
## ENTRY /root/logs
</pre>
' > /usr/local/sbin/backup.config</tt>

* setup bigbrother

edit passwd and add the bb user:
echo "bb:x:1984:1984:Big Brother:/home/bb:/bin/bash" >> /etc/passwd

edit group and add the bb group:
echo "bb:x:1984:" >> /etc/group

rebuild the password database:
pwconv

Create bb’s home dir and set ownership:
mkdir /home/bb
chown bb.bb /home/bb

Go to bb’s home and fetch the client tarball:
<pre>cd ~bb
scp backup2:/mnt/data4/build/bb/GOOD-bb-linux-bbc1.9i-btf.tgz .

tar xvf GOOD-bb-linux-bbc1.9i-btf.tgz

yum install glibc-2.12-1.47.el6_2.5.i686

vi etc/bbsys.sh
SORT="/bin/sort"

vi bin/bb-disk.sh
eval "$DFCMD | $EGREP \"$DFUSE\" | $EGREP -v \"$DFEXCLUDE\" | $SORT \"-k${DFSORT}\" > $BBTMP/DFOUT.$$"
wclen1=`eval "$DFCMD | $EGREP \"$DFUSE\" | $EGREP -v \"$DFEXCLUDE\" | $SORT \"-k${DFSORT}\" | $WC"`

cd bbc1.9i-btf/etc/

echo "10.1.4.5 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh" > /home/bb/bbc1.9i-btf/etc/bb-hosts
echo "10.1.4.63 virt13.johncompanies.com # ssh" >> /home/bb/bbc1.9i-btf/etc/bb-hosts
(^^^change to correct host)

chmod +r /var/log/messages

cd ..
su bb
./runbb.sh start
exit</pre>

Add this line to /etc/rc.d/rc.local so bb will start at boot:

echo "su - bb -c \"cd /home/bb/bbc1.9i-btf; ./runbb.sh start\"" >> /etc/rc.d/rc.local

* Uncomment/add lines with the appropriate IP addresses in /etc/ssh/sshd_config:

vi /etc/ssh/sshd_config
<tt>ListenAddress <span style="color:red">69.55.226.2</span>
ListenAddress <span style="color:red">10.1.4.63</span></tt>

<DEPRECATED>
And make sure to uncomment
Protocol 2,1
</DEPRECATED>

* setup backup1/backup2 nfs mounts

<DEPRECATED>
setup portmap to start with server, and start it:

chkconfig --level 3 portmap on
service portmap start
</DEPRECATED>

mb mount

run df to confirm that nfs mounts are mounted

* make alias for /vzconf
ln -s /etc/sysconfig/vz-scripts /vzconf

* edit /etc/sysconfig/vz

vi /etc/sysconfig/vz

calc cpu to reserver for HN. take 5% of power:
vzcpucheck
Current CPU utilization: 27000
Power of the node: 1018320

so in this case:
VE0CPUUNITS=50916

enable fastboot:
VZFASTBOOT=yes

Disable auto mount
#VZMOUNTS="/vz"

Enable slm:
SLM=yes

<DEPRECATED>
Enable IPV6
IPV6=yes
</DEPRECATED>

add/edit to traffic parameters section:
TRAFFIC_SHAPING=yes
BANDWIDTH="eth0:102400"
TOTALRATE="eth0:1:102400"
RATE="eth0:1:50000"

for 2 nics, uncomment this line:
VE_ROUTE_SRC_DEV="eth0"

Make sure the IPTABLES section looks like:
<pre>IPTABLES="ip_tables iptable_filter iptable_mangle ipt_limit ipt_multiport ipt_tos ipt_TOS ipt_REJECT ipt_tcpmss ipt_ttl ipt_LOG ipt_length ip_conntrack ip_conntrack_ftp ip_conntrack_irc ipt_conntrack ipt_state ipt_helper iptable_nat ip_nat_ftp ip_nat_irc ipt_REDIRECT ipt_multiport ipt_TCPMSS"</pre>

* disable periodic fsck's, lower the root padding on the /vz and /vz1 filesystems to 100MB
<pre>tune2fs -i 0 /dev/sda3
tune2fs -i 0 /dev/sdb2
tune2fs -c -1 /dev/sda3
tune2fs -c -1 /dev/sdb2
tune2fs -r 100000 /dev/sda3
tune2fs -r 100000 /dev/sdb2</pre>

* setup vzagent with pub IP (to allow migrations from 3.x):
vzctl set 50 --ipdel all --ipadd <span style="color:red">69.55.226.3</span> --userpasswd vzagent0:k0zub1k --save
vzagent_ctl restart

* edit ve config files:

From split of 80 take
<tt>CPUUNITS</tt>

And put into files in /vzconf/base_ct_conf
along with:
<pre>VERSION="2"

ONBOOT="yes"

KMEMSIZE="2147483647:2147483647"
LOCKEDPAGES="2147483647:2147483647"
PRIVVMPAGES="2147483647:2147483647"
SHMPAGES="2147483647:2147483647"
NUMPROC="32567:32567"
VMGUARPAGES="2147483647:2147483647"
OOMGUARPAGES="2147483647:2147483647"
NUMTCPSOCK="2147483647:2147483647"
NUMFLOCK="2147483647:2147483647"
NUMPTY="255:255"
NUMSIGINFO="1024:1024"
TCPSNDBUF="2147483647:2147483647"
TCPRCVBUF="2147483647:2147483647"
OTHERSOCKBUF="2147483647:2147483647"
DGRAMRCVBUF="2147483647:2147483647"
NUMOTHERSOCK="2147483647:2147483647"
DCACHESIZE="2147483647:2147483647"
NUMFILE="2147483647:2147483647"
NUMIPTENT="2147483647:2147483647"

DISKSPACE=""
DISKINODES=""
QUOTATIME="600"
IOPRIO="4"
CPULIMIT="75"
OFFLINE_MANAGEMENT="yes"
ARCH="x86"
PLATFORM="linux"

IP_ADDRESS="0.0.0.0"
NAMESERVER="69.55.225.225 69.55.230.3"
CPUUNITS="12313"
IOLIMIT="10485760"
IOPSLIMIT="100"
</pre>

* customize the control panel:
rm -fr /vz/backups

vi /var/opt/pva/pp/xsl/lib/vt/backups-forms.xsl

<pre>
<span style="font-size:13px; font-weight:bold">NOTE: You cannot create or view backups via this control panel- IT WILL NOT WORK. Please contact linux@johncompanies.com to setup or inquire about backups for your server.</span>

/var/opt/pva/pp/xsl/lib/vt/backups-list.xsl
</pre>

cd /var/opt/pva/pp/static/vz/skins
for f in `ls`; do scp backup2:/d4/build/vzcp/var/vzcp/static/vz/skins/winxp.new/images/logo_main_4.7.gif $f/images/logo_main.gif; done

* template install

To install new templates:

vzup2date -z

Copy over templates from old server. Ex: ubu 8.04 from virt19:

rsync -a -e ssh <span style="color:red">10.1.4.69</span>:/vz/template/ubuntu/8.04 /vz/template/ubuntu/8.04

install ubuntu 8.04 templates (get latest from vz)

<pre>cd /backup4/vzrpms/distro/ubuntu/8.04
rpm -ivh *
vzpkg update cache ubuntu-8.04-x86</pre>

If you install a tempalte from vzup2date it will download an old cache as well. Better to download the OS rpm from VZ site then create cache then use vzup2date to install the application templates.

* add traffic shaping support

Re-define class 0:

<pre>vi /etc/sysconfig/vz-scripts/networks_classes
0 69.55.224.0/20
0 10.1.4.0/24</pre>

make sure 1 is defined (0.0.0.0) and no other classes are needed

Then start it

/etc/init.d/vz shaperrestart

* udp dos fix

echo "/usr/local/sbin/iptables.sh" >> /etc/rc.d/rc.local

* make rc.local start before vz

mv /etc/rc.d/rc3.d/S99local /etc/rc.d/rc3.d/S95local

* disable unwanted services/daemons

Before:
<pre>abrt-ccpp 0:off 1:off 2:off 3:on 4:off 5:on 6:off
abrt-oops 0:off 1:off 2:off 3:on 4:off 5:on 6:off
abrtd 0:off 1:off 2:off 3:on 4:off 5:on 6:off
acpid 0:off 1:off 2:on 3:on 4:on 5:on 6:off
atd 0:off 1:off 2:off 3:on 4:on 5:on 6:off
auditd 0:off 1:off 2:on 3:on 4:on 5:on 6:off
autofs 0:off 1:off 2:off 3:on 4:on 5:on 6:off
cpuspeed 0:off 1:on 2:on 3:on 4:on 5:on 6:off
crond 0:off 1:off 2:on 3:on 4:on 5:on 6:off
cups 0:off 1:off 2:on 3:on 4:on 5:on 6:off
haldaemon 0:off 1:off 2:off 3:on 4:on 5:on 6:off
ip6tables 0:off 1:off 2:on 3:on 4:on 5:on 6:off
iptables 0:off 1:off 2:on 3:on 4:on 5:on 6:off
irqbalance 0:off 1:off 2:off 3:on 4:on 5:on 6:off
kdump 0:off 1:off 2:off 3:on 4:on 5:on 6:off
lvm2-monitor 0:off 1:on 2:on 3:on 4:on 5:on 6:off
mcelogd 0:off 1:off 2:off 3:on 4:off 5:on 6:off
mdmonitor 0:off 1:off 2:on 3:on 4:on 5:on 6:off
messagebus 0:off 1:off 2:on 3:on 4:on 5:on 6:off
netfs 0:off 1:off 2:off 3:on 4:on 5:on 6:off
network 0:off 1:off 2:on 3:on 4:on 5:on 6:off
nfslock 0:off 1:off 2:off 3:on 4:on 5:on 6:off
ntpd 0:off 1:off 2:off 3:on 4:off 5:off 6:off
portreserve 0:off 1:off 2:on 3:on 4:on 5:on 6:off
postfix 0:off 1:off 2:on 3:on 4:on 5:on 6:off
pvaagentd 0:off 1:off 2:on 3:on 4:on 5:on 6:off
pvapp 0:off 1:off 2:on 3:on 4:on 5:on 6:off
qpidd 0:off 1:off 2:on 3:on 4:on 5:on 6:off
rpcbind 0:off 1:off 2:on 3:on 4:on 5:on 6:off
rpcgssd 0:off 1:off 2:off 3:on 4:on 5:on 6:off
rpcidmapd 0:off 1:off 2:off 3:on 4:on 5:on 6:off
rsyslog 0:off 1:off 2:on 3:on 4:on 5:on 6:off
sshd 0:off 1:off 2:on 3:on 4:on 5:on 6:off
sysstat 0:off 1:on 2:on 3:on 4:on 5:on 6:off
udev-post 0:off 1:on 2:on 3:on 4:on 5:on 6:off
vz 0:off 1:off 2:on 3:on 4:on 5:on 6:off
vzlmond 0:off 1:off 2:on 3:on 4:on 5:on 6:off
vznetcfg 0:off 1:off 2:on 3:on 4:on 5:on 6:off
xinetd 0:off 1:off 2:off 3:on 4:on 5:on 6:off</pre>

<pre>chkconfig --level 3 cpuspeed off
chkconfig --level 3 cups off
chkconfig --level 3 haldaemon off
chkconfig --level 3 ip6tables off
chkconfig --level 3 iptables off
chkconfig --level 3 irqbalance off
chkconfig --level 3 lvm2-monitor off
chkconfig --level 3 mdmonitor off
chkconfig --level 3 messagebus off
chkconfig --level 3 netfs off
chkconfig --level 3 nfslock off
chkconfig --level 3 qpidd off
chkconfig --level 3 rpcgssd off
chkconfig --level 3 rpcidmapd off</pre>

* load mrtg graphs

yum install xinetd

create /etc/xinetd.d/load with the following contents:

<pre>echo "service load
{
disable = no
flags = REUSE
socket_type = stream
wait = no
user = root
server = /usr/local/sbin/load.pl
}" > /etc/xinetd.d/load</pre>

Add this line to /etc/services:
echo "load 12384/tcp" >> /etc/services

* Tasks to do on backup1:
setup backup dirs:

mkdir -p /data/<span style="color:red">virt13</span>/0

for f in 0 1 2 3 4 5 6; do mkdir -p /data/<span style="color:red">virt13</span>/$f/vz/private; mkdir -p /data/<span style="color:red">virt13</span>/$f/vz1/private; mkdir -p /data/<span style="color:red">virt13</span>/$f/vz2/private; done

add the system to
vi /usr/local/sbin/snapshot_rotate

* Tasks to do on mail:

Update backup graph:
vi /usr/local/www/mgmt/cgi/backupgraph.pl
(add hostname)

add to bbhosts:
<pre>vi /usr/home/bb/bbsrc/bb1.9i-btf/etc/bb-hosts
su bb
cd
bbsrc/bb/runbb.sh restart ; exit</pre>

add to load graph config:
vi /usr/local/www/mgmt/mrtg/mrtg1.cfg
(add new section for new machine)

* add to management db (on mail and devweb) jc.ref_machines and jc.ref_templates

insert new server:
insert into ref_machines values (null,'virt19','virt19.johncompanies.com',0,'l');

get new machine_id back out:
select * from ref_machines;

insert as a system to jc customer col00001:
insert into systems values ('', 'col00001',30,null,null,100,'virt11.johncompanies.com',null,'2010-03-11',null,null,1,0,null,null,null);

get sysid:
select * from systems where hostname='virt19.johncompanies.com';

update ipmap (main IP and IP used for VZCP):
update ipmap set cid='col00001', sysid=3772, assign_date='2008-01-07', status=1 where ip='69.55.236.2';
update ipmap set cid='col00001', sysid=3772, assign_date='2008-01-07', status=1 where ip='69.55.230.109';

add templates (or do via mgmt):
<pre>
insert into ref_templates values ('','fedora-core-1/20050725',33,'Fedora Core 1',1);
insert into ref_templates values ('','fedora-core-2/20050725',33,'Fedora Core 2',1);
insert into ref_templates values ('','debian-3.1/20050927',33,'Debian 3.1',1);</pre>

* Make sure that ns1c has proper forward and reverse entries for the new system (both private and public addresses), so that new system emailing will work.
<pre>vr johncompanies.com
vi internal.johncompanies.com
rndc reload johncompanies.com IN private</pre>
(edit the PTR too)

* add an outside blocking rule to the firewall, so this machine can only be reached from inside the firewall.

Follow example already in firewall virt19 is:

<pre>ipfw add 00143 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 or 62.213.72.101 or 64.131.90.27 or 82.200.65.190 or 91.204.24.0/22 or 194.44.160.178 or 195.214.232.10 or 195.214.233.10 or 208.251.212.1 } to 69.55.226.2
ipfw add 00143 deny ip from any to 69.55.226.2</pre>

(virt12 would be 00142)

* add to [[Cabinetmap|Cabinetmap]]

* make /vz1/private
<pre>mkdir /vz1/private ; chmod 0755 /vz1/private
mkdir /vz1/root ; chmod 0755 /vz1/root</pre>

* put config files in a place they’ll get backed up
<pre>mkdir /vzconf/confs
crontab -e
0 0 * * * rm -f /vzconf/confs/*.conf; cp /vzconf/*.conf /vzconf/confs</pre>

* fix pass/vzpp and localse issue with ubu 8.10

vi /vz/template/ubuntu/8.10/x86/config/os/default/post-install

sed '/^password.*pam_unix.so.*sha512/ s~sha512~md5~' -i etc/pam.d/common-password

localedef -i en_US -c -f UTF-8 en_US.UTF-8

* add tun device support

echo "modprobe tun" >> /etc/rc.local

* reinstall VZA/CT #1/CT #50

After the install, the PP was broken. So needed to reinstall everything. Did the pva_setup --uninstall and removed CT 50 and then setup the pva per http://kb.parallels.com/113071 then setup the CT 50 per the other article- centos6 had issues - and then authenticated with virt13 to get that into the management list.

http://kb.parallels.com/113072

http://kb.parallels.com/113071

* config logrotate (otherwise it creates weekly empty logs)

vi /etc/logrotate.conf

(add to end)

<pre>/var/log/messages {
rotate 12
monthly
size 10M
missingok
compress
delaycompress
notifempty
sharedscripts
postrotate
echo `date` > /var/log/messages
endscript
}</pre>

* install megacli64 (can we just move the bin and not install the lib?)

<pre>scp backup2: /d4/build/3ware/MegaCli-8.00.40-1.i386.rpm /root/MegaCli-8.00.40-1.i386.rpm
scp backup2: /d4/build/3ware/Lib_Utils-1.00-08.noarch.rpm /root/ Lib_Utils-1.00-08.noarch.rpm

rpm -ivh Lib_Utils-1.00-08.noarch.rpm
rpm -ivh MegaCli-8.00.40-1.i386.rpm

cp opt/MegaRAID/MegaCli/MegaCli64 /usr/local/sbin/megacli64</pre>

= Older versions =

== 4.0 ==

Last updated 2010-03-11

<pre>1. enable post info to console

2950:
Serial comm.: on with cons. Redir via com1
Failsafe: 115200
Redir after boot: enabled

Server
Custom- desel all but base
/root/install.log

2. Install OS

CentOS5.x (get latest from http://mirrors.kernel.org/centos/5.4/isos/i386/)

Set partitions:
Disk 1:
/ 3072M
6144M swap
/vz (the rest)
Disk 2:
6144M Swap

Use grub, install to MBR
No firewall
Disable SElinux
Install only base

3. create ssh key, add swsoft key to authorized_keys

(enter to everything)

cd /root/
ssh-keygen -t dsa

echo "ssh-dss AAAAB3NzaC1kc3MAAACBAIB7XHmpcsa1D/vsKS+mzEvBVo7udew/aRahbjnjywL4YXT7ruWzpGo7R3UA8GKIgVHoQGUbuXrJz5ICtBEXOlbgXVEz7Bu5nXOG2C9i/8pmFtUw6GwACII8uZE7t/Zq5CRj+5zN1Rv7n/4X4RAjzk5BHXoZ62a32FTbS4tPvyPrAAAAFQDiUi7YEiffNzGvB544lQpRdn29TwAAAIBkUchLTBE1+sX11iC80yhNh9EGeCMLdkT8UqHsTSaD0myQPxGALuNY4rO31q+25uKqSRDTXd6ZWXp4bplA6Ls6jl7r7XVR9loJ7WvRMqc4Dv68TZ7h2UBqAhu4det2UF4LHim3DNf1gKn9DzWNkmI/N2FvVVAL4aM3sPunVIvUYgAAAIAJX38INABVgnCEBKqpv5FMLm3k9MgMn54oQFmylBbgM0w4SUeuDj3jLrkJf8dT9w8ByEDb8xviexpWrPsbz2DeThle4xgzcyKrUkt2Wugtmm4DwS0S/68BAA9wXoWD/vZ67GBoCKH7uq8VAjUVcCbJ/7IydljnFAZoJpktdwN7iQ== russup@support.sw-soft.com" >> /root/.ssh/authorized_keys

4. setup access to backup2:

echo "10.1.4.3 backup2" >> /etc/hosts
echo "10.1.4.8 backup1" >> /etc/hosts

# Copy id_dsa.pub to backup2 and add it to the end of ~root/.ssh/authorized_keys:

cat .ssh/id_dsa.pub | ssh backup2 'cat - >> /root/.ssh/authorized_keys'
cat .ssh/id_dsa.pub | ssh backup1 'cat - >> /root/.ssh/authorized_keys'

ssh backup2 hostname
ssh backup1 hostname

(confirm you can connect w/o entering a password)

5. add bash backward history search
echo "\"\M-z\": history-search-backward
\"\M-x\": history-search-backward " > /root/.inputrc
echo "export GREP_OPTIONS='--color=auto'" >> ~/.bashrc
echo "\"\e[5~\": history-search-backward" >> ~/.inputrc
echo "\"\e[6~\": history-search-forward" >> ~/.inputrc

6. add to /root/.bashrc, create log dir

vi /root/.bashrc
alias v='vzctl'
alias mbm='mb mount'
alias mbu='mb umount'
export PS1="[\u@\h \w]# "
alias cv='cd /vz/private'
alias cv1='cd /vz1/private'
alias cv2='cd /vz2/private'

source /root/.bashrc

mkdir /root/logs

7. make sure serial console and kernel messages are working

add these 2 lines to the top:

vi /boot/grub/menu.lst
serial --unit=0 --speed= 115200
terminal --timeout=10 serial console

and the kernel line should look like:

kernel /boot/vmlinuz-2.4.20-020stab009.24.777-enterprise ro root=/dev/sda1 console=tty0 console=ttyS0,115200

Remove or comment out these lines:
hiddensplashmenu
splashimage=(hd0,0)/boot/grub/splash.xpm.gz

add to /etc/inittab and /etc/securetty

echo "s0:2345:respawn:/sbin/agetty -L ttyS0 115200 linux" >> /etc/inittab
echo "ttyS0" >> /etc/securetty

kill -1 1

DEPRECATED 8. configuring eth1 for management net:

echo "10.1.5/24 via 10.1.4.2
10.1.6/24 via 10.1.4.2" >> /etc/sysconfig/network-scripts/route-eth1

reboot the machine to make sure that the interface is configured properly.

also verify that the routes are in place like so:

netstat -rn | grep 10.1.4.2
10.1.5.0 10.1.4.2 255.255.255.0 UG 40 0 0 eth1
10.1.6.0 10.1.4.2 255.255.255.0 UG 40 0 0 eth1

9. make sure the hostname on the system is properly set - in the form virt4.johncompanies.com - and make sure it is set properly:
cat /etc/sysconfig/network

10. Check/update fstab

vi /etc/fstab

/etc/fstab entries should look something like (note noatime option):

/dev/sda3 /vz ext3 defaults,noatime 1 2
/dev/sdb2 /vz1 ext3 defaults,noatime 1 2

make sure swap is configured in fstab

Use real devices instead of LABEL

11. Install Virtuozzo 4.0

umount /vz
umount /vz1

mke2fs -i1024 -b1024 -j /dev/sda3 &
mke2fs -i1024 -b1024 -j /dev/sdb2 &

Experimentally ran this for sdc on virt12:
mke2fs -i2048 -b2048 -j /dev/sdb1

mount -a

scp backup2:/d4/vzrpms/iso/vzinstall-linux.bin /vz
cd /vz
chmod 0700 vzinstall-linux.bin
./vzinstall-linux.bin

change wd to /vz
Ignore OS version error
Custom install (don’t install templates)

reboot

vzup2date

(run again. till no more updates offered)

reboot

Run vzup2date (make sure to choose enterprise kernel) and/or install latest kernel:

NOTE: skip over the manual kernel load below if you can get the 16G recognized before running vzup2date- then it will dl the ent kernel

NOTE: go to /vz/vzup2date/virtuozzo/linux/i386/4.0.0/CU-2.6.18-028stab068.3 and install kern there (downloaded by vzup2date)

<DEPREACATED>
ftp downloads.swsoft.com
User:
gdawson
Pass:
qjdK1R

cd /updates/Rel400/20080307
bin
get vzkernel-ent-2.6.18-028stab053.10.i686.rpm
get vzmodules-ent-2.6.18-028stab053.10.i686.rpm
bye

rpm -ivh vzkernel-ent-2.6.18-028stab053.10.i686.rpm \
vzmodules-ent-2.6.18-028stab053.10.i686.rpm
</DEPREACATED>

vi /boot/grub/menu.lst
(set default kernel to new kernel)

12. Install ntp

yum install ntp

rm -f /etc/ntp.conf

# Create a new file with one line:

echo "server 10.1.4.5" > /etc/ntp.conf

# Then set ntpd so it will start at boot:

chkconfig --level 3 ntpd on
service ntpd start

# Make sure it’s talking to the ntp server:

ntpq -p

Correct output should have * or + to the left of the IP:

remote refid st t when poll reach delay offset jitter
==============================================================================
10.1.4.105 clock1.redhat.c 2 u - 64 1 0.370 -706677 0.008

13. copy over binaries/scripts from backup2

scp backup2:/mnt/data4/bin/linux/* /usr/local/sbin/
scp 10.1.4.62:/vzconf/jctmpl* /etc/sysconfig/vz-scripts/
scp 10.1.4.62:/vzconf/ve-slm* /etc/sysconfig/vz-scripts/

14. setup root crontab (alter backup time):
crontab -e
1 1 * * * /usr/local/sbin/backup
*/15 * * * * /usr/local/sbin/vim.watchdog.30 > /dev/null
#56 4,16 * * * root /usr/sbin/vzkeyupdate -c
*/5 * * * * /usr/local/sbin/perc5iraidchk

15. setup backups

echo '#!/bin/sh
backupdir=/data/virt11
server=backup1

## ENTRY /etc
## ENTRY /usr/local/sbin
## ENTRY /root/logs
' > /usr/local/sbin/backup.config

16. setup bb

edit passwd and add the bb user:
echo "bb:x:1984:1984:Big Brother:/home/bb:/bin/bash" >> /etc/passwd

# edit group and add the bb group:
echo "bb:x:1984:" >> /etc/group

# rebuild the password database:

pwconv

# Create bb’s home dir and set ownership:

mkdir /home/bb
chown bb.bb /home/bb

# Go to bb’s home and fetch the client tarball:

cd ~bb
scp backup2:/mnt/data4/build/bb/bb-linux.tar .

tar xf bb-linux.tar

cd bbc1.9e-btf/etc/

echo "10.1.4.5 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh" > /home/bb/bbc1.9e-btf/etc/bb-hosts
echo "10.1.4.61 virt11.johncompanies.com # ssh" >> /home/bb/bbc1.9e-btf/etc/bb-hosts

chmod +r /var/log/messages

cd ..
su bb
./runbb.sh start
exit

# Add this line to /etc/rc.d/rc.local so bb will start at boot:

echo "su - bb -c \"cd /home/bb/bbc1.9e-btf; ./runbb.sh start\"" >> /etc/rc.d/rc.local

17. Uncomment/add lines with the appropriate IP addresses in /etc/ssh/sshd_config:

vi /etc/ssh/sshd_config
ListenAddress 69.55.238.3
ListenAddress 10.1.4.61

And make sure to uncomment
Protocol 2,1

18. setup backup2 nfs mounts

setup portmap to start with server, and start it:

chkconfig --level 3 portmap on
service portmap start

mb mount

run df to confirm that nfs mounts are mounted

19. make alias for /vzconf
ln -s /etc/sysconfig/vz-scripts /vzconf

DEPRECATED 20. copy over the /etc/sysconfig/vz-scripts/4unlimited template file from the last virt server and put it in /etc/sysconfig/vz-scripts

scp 10.1.4.68:/etc/sysconfig/vz-scripts/4unlimited /etc/sysconfig/vz-scripts/

NOTE: if this system is more powerful (RAM, CPU) then the previous, you will want to create a new one using vzsplit:

cd /etc/sysconfig/vz-scripts/
vzsplit -f 4unlimited

21. edit /etc/sysconfig/vz

vi /etc/sysconfig/vz

enable fastboot:
VZFASTBOOT=yes

Disable auto mount
#VZMOUNTS="/vz"

Enable IPV6
IPV6=yes

add/edit to traffic parameters section:
TRAFFIC_SHAPING=yes
BANDWIDTH="eth0:102400"
TOTALRATE="eth0:1:102400"
RATE="eth0:1:50000"

for 2 nics, uncomment this line:
VE_ROUTE_SRC_DEV="eth0"

Make sure the IPTABLES section looks like:
IPTABLES="ip_tables iptable_filter iptable_mangle ipt_limit ipt_multiport ipt_tos ipt_TOS ipt_REJECT ipt_tcpmss ipt_ttl ipt_LOG ipt_length ip_conntrack ip_conntrack_ftp ip_conntrack_irc ipt_conntrack ipt_state ipt_helper iptable_nat ip_nat_ftp ip_nat_irc ipt_REDIRECT ipt_multiport ipt_TCPMSS"

22. disable periodic fsck's, lower the root padding on the /vz and /vz1 filesystems to 100MB

tune2fs -i 0 /dev/sda3
tune2fs -i 0 /dev/sdb2
tune2fs -c -1 /dev/sda3
tune2fs -c -1 /dev/sdb2
tune2fs -r 100000 /dev/sda3
tune2fs -r 100000 /dev/sdb2

23. setup vzagent with pub IP (to allow migrations from 3.x):

vzctl set 1 --ipdel all --ipadd 69.55.230.57 --userpasswd vzagent0:k0zub1k --save
vzagent_ctl restart

24. edit ve config files:

From split of 80 take
CPUUNITS

And put into files in /usr/local/sbin/ve-slm.1024MB.conf et. al.

25. customize the control panel:

vzmlocal 1:1:/vz/private/1:/vz/root/1
cd /vz/private/1/fs/root/var/vzcp/static/vz/skins
for f in `ls`; do scp backup2:/d4/build/vzcp/var/vzcp/static/vz/skins/winxp.new/images/logo_main.gif $f/images/logo_pim.gif; done

1) comment 'Backup' tab in /etc/vzcp/pp/tabs.xml:

<tab-bar id="maintance" obj-type="env.core.vt">


2) change 'href' parameter if 'Maintentance' item in /etc/vzcp/pp/menu.xml:


<item id="management.maintenance" href="/vz/cp/vzdir/infrman/envs/repair?branch=env-home,vzpp" icon="backup" tip="maintenance_screen">

3) /etc/vzcp/pp/dashboard.xml

<item id="management.maintenanance" href="$baseurl/vzdir/infrman/envs/repair" title="Maintenance" icon="env_backup" tip="maintenance_screen" cred="vzcp.env.backups.list" security-obj-type="ve.vz.vt">

URL for 'href' parameter can be obtained from tabs.xml.

vi /var/vzcp/xsl/screens/vzdir/monitman/traffic.xsl
<sw:text>Traffic for selected period and class (NOTE: this information is not certified by JohnCompanies and is not used to calculate actual usage or charges)</sw:text>

<DEPREACATED>
/vz/private/1/root/var/vzcp/static/vz/skins/
/vz/private/1/root/etc/vzcp/pp/menu.xml
/vz/private/1/root/etc/vzcp/pp/dashboard.xml
pp/tabs.xml
/vz/private/1/root/etc/vzcp/vzcpcon.conf

eg (2.6.1 or 3.0):
rsync -va -e ssh backup2:/d4/build/vzcp/var/vzcp/static/vz/skins/ /vz/private/1/root/var/vzcp/static/vz/skins/
rsync -va -e ssh backup2:/d4/build/vzcp/etc/vzcp/pp/menu.xml /vz/private/1/root/etc/vzcp/pp/menu.xml
rsync -va -e ssh backup2:/d4/build/vzcp/etc/vzcp/pp/dashboard.xml /vz/private/1/root/etc/vzcp/pp/dashboard.xml
rsync -va -e ssh backup2:/d4/build/vzcp/etc/vzcp/vzcpcon.conf /vz/private/1/root/etc/vzcp/vzcpcon.conf
</DEPREACATED>

24. template install

To install new templates:

vzup2date –z

Copy over ubu 8.04 from virt19

rsync -a -e ssh 10.1.4.69:/vz/template/ubuntu/8.04 /vz/template/ubuntu/8.04

install ubuntu 8.04 templates (get latest from vz)

cd /backup4/vzrpms/distro/ubuntu/8.04
rpm -ivh *
vzpkg update cache ubuntu-8.04-x86

26. add traffic shaping support

Re-define class 0:

vi /etc/sysconfig/vz-scripts/networks_classes
0 69.55.224.0/20
0 10.1.4.0/24

make sure 1 is defined (0.0.0.0) and no other classes are needed

Then start it

/etc/init.d/vz shaperrestart

27. udp dos fix

echo "/usr/local/sbin/iptables.sh" >> /etc/rc.d/rc.local

28. make rc.local start before vz

mv /etc/rc.d/rc3.d/S99local /etc/rc.d/rc3.d/S95local

29. disable unwanted services/daemons

chkconfig --level 3 avahi-daemon off
chkconfig --level 3 bluetooth off
chkconfig --level 3 cpuspeed off
chkconfig --level 3 cups off
chkconfig --level 3 firstboot off
chkconfig --level 3 gpm off
chkconfig --level 3 haldaemon off
chkconfig --level 3 hidd off
chkconfig --level 3 ip6tables off
chkconfig --level 3 iptables off
chkconfig --level 3 irqbalance off
chkconfig --level 3 isdn off
chkconfig --level 3 kudzu off
chkconfig --level 3 lvm2-monitor off
chkconfig --level 3 mcstrans off
chkconfig --level 3 mdmonitor off
chkconfig --level 3 messagebus off
chkconfig --level 3 netfs off
chkconfig --level 3 nfslock off
chkconfig --level 3 pcscd off
chkconfig --level 3 rawdevices off
chkconfig --level 3 readahead_early off
chkconfig --level 3 restorecond off
chkconfig --level 3 rpcgssd off
chkconfig --level 3 rpcidmapd off
chkconfig --level 3 smartd off
chkconfig --level 3 yum-updatesd off

30. load mrtg graphs

yum install xinetd

create /etc/xinetd.d/load with the following contents:

echo "service load
{
disable = no
flags = REUSE
socket_type = stream
wait = no
user = root
server = /usr/local/sbin/load.pl
}" > /etc/xinetd.d/load

# Add this line to /etc/services:
echo "load 12384/tcp" >> /etc/services

31. Tasks to do on backup1:

setup backup dirs:
mkdir -p /data/virt11/0

add the system to
vi /usr/local/sbin/snapshot_rotate

32. Tasks to do on mail:

Update backup graph:
vi /usr/local/www/mgmt/cgi/backupgraph.pl
(add hostname)

add to bbhosts:
vi /usr/home/bb/bbsrc/bb1.9e-btf/etc/bb-hosts
su bb
cd
bbsrc/bb/runbb.sh restart ; exit

add to load graph config:
vi /usr/local/www/mgmt/mrtg/mrtg1.cfg
(add new section for new machine)

33. add to management db (on mail and devweb) jc.ref_machines and jc.ref_templates

insert into ref_machines values (null,'virt19','virt19.johncompanies.com',0,'l');
select * from ref_machines;
insert into systems values ('', 'col00001',30,null,null,100,'virt11.johncompanies.com',null,'2010-03-11',null,null,1,0,null,null,null);
select * from systems where hostname='virt19.johncompanies.com';
update ipmap set cid='col00001', sysid=3772, assign_date='2008-01-07', status=1 where ip='69.55.236.2';
update ipmap set cid='col00001', sysid=3772, assign_date='2008-01-07', status=1 where ip='69.55.230.109';
insert into ref_templates values ('','fedora-core-1/20050725',33,'Fedora Core 1',1);
insert into ref_templates values ('','fedora-core-2/20050725',33,'Fedora Core 2',1);
insert into ref_templates values ('','debian-3.1/20050927',33,'Debian 3.1',1);

34. Make sure that ns1c has proper forward and reverse entries for the new system (both private and public addresses), so that new system emailing will work.
vr johncompanies.com
vi internal.johncompanies.com
rndc reload johncompanies.com IN private
(edit the PTR too)

35. add an outside blocking rule to the firewall, so this machine can only be reached from inside the firewall.

Follow example already in firewall virt19 is:

ipfw add 00149 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 or 62.213.72.101 or 64.131.90.27 or 82.200.65.190 or 91.204.24.0/22 or 194.44.160.178 or 195.214.232.10 or 195.214.233.10 or 208.251.212.1 } to 69.55.236.2
ipfw add 00149 deny ip from any to 69.55.236.2

virt12 would be 00142

36. add to server/cabinet map

37. make /vz1/private
mkdir /vz1/private ; chmod 0755 /vz1/private
mkdir /vz1/root ; chmod 0755 /vz1/root

38. put config files in a place they’ll get backed up
mkdir /vzconf/confs
crontab -e
0 0 * * * rm -f /vzconf/confs/*.conf; cp /vzconf/*.conf /vzconf/confs

39. fix pass/vzpp and localse issue with ubu 8.10

vi /vz/template/ubuntu/8.10/x86/config/os/default/post-install

sed '/^password.*pam_unix.so.*sha512/ s~sha512~md5~' -i etc/pam.d/common-password

localedef -i en_US -c -f UTF-8 en_US.UTF-8

40. add tun device support

echo "modprobe tun" >> /etc/rc.local</pre>

== 3.x ==

Last updated 2007-04-5

<pre>1. enable post info to console
Supermicro:
Console redirection:
Com port addr: on-board COM A
Baud: 38400
Console type: vt100
Flow control: none
Console connection: direct
Continue cr after post: off

2450:
Make sure running bios A09
Console Redirection: VT100/VT220

2950:
Serial comm.: on with cons. Redir via com1
Failsafe: 115200
Redir after boot: enabled

6144
Server
Custom- desel all but base
/root/install.log

2. Install OS
Set partitions:
/ 3072M
Disk 1: Swap 4096M
Disk 2: Swap 4096M
/vz (the rest)

No firewall
Disable SElinux

FC4
Choose a custom install, the “minimal” package

CentOS4 Server CD (only option for 2950)
Choose the “minimal” package

3. create ssh key, add swsoft key to authorized_keys

(enter to everything)

cd /root/
ssh-keygen -t dsa

echo "ssh-dss AAAAB3NzaC1kc3MAAACBAIB7XHmpcsa1D/vsKS+mzEvBVo7udew/aRahbjnjywL4YXT7ruWzpGo7R3UA8GKIgVHoQGUbuXrJz5ICtBEXOlbgXVEz7Bu5nXOG2C9i/8pmFtUw6GwACII8uZE7t/Zq5CRj+5zN1Rv7n/4X4RAjzk5BHXoZ62a32FTbS4tPvyPrAAAAFQDiUi7YEiffNzGvB544lQpRdn29TwAAAIBkUchLTBE1+sX11iC80yhNh9EGeCMLdkT8UqHsTSaD0myQPxGALuNY4rO31q+25uKqSRDTXd6ZWXp4bplA6Ls6jl7r7XVR9loJ7WvRMqc4Dv68TZ7h2UBqAhu4det2UF4LHim3DNf1gKn9DzWNkmI/N2FvVVAL4aM3sPunVIvUYgAAAIAJX38INABVgnCEBKqpv5FMLm3k9MgMn54oQFmylBbgM0w4SUeuDj3jLrkJf8dT9w8ByEDb8xviexpWrPsbz2DeThle4xgzcyKrUkt2Wugtmm4DwS0S/68BAA9wXoWD/vZ67GBoCKH7uq8VAjUVcCbJ/7IydljnFAZoJpktdwN7iQ== russup@support.sw-soft.com" >> /root/.ssh/authorized_keys

4. setup access to backup2:

echo "10.1.4.3 backup2" >> /etc/hosts

Copy id_dsa.pub to backup2 and add it to the end of ~root/.ssh/authorized_keys:

cat .ssh/id_dsa.pub | ssh backup2 'cat - >> /root/.ssh/authorized_keys'

ssh backup2

(confirm you can connect w/o entering a password)

5. add bash backward history search
echo "\"\M-z\": history-search-backward
\"\M-x\": history-search-backward " > /root/.inputrc
echo "export GREP_OPTIONS='--color=auto'" >> ~/.bashrc
echo "\"\e[5~\": history-search-backward" >> ~/.inputrc
echo "\"\e[6~\": history-search-forward" >> ~/.inputrc

6. add to /root/.bashrc, create log dir

vi /root/.bashrc
alias v='vzctl'
alias mbm='mb mount'
alias mbu='mb umount'
export PS1="[\u@\h \w]# "
alias cv='cd /vz/private'
alias cv1='cd /vz1/private'
alias cv2='cd /vz2/private'

mkdir /root/logs

7. make sure serial console and kernel messages are working

grub

add these 2 lines to the top:

vi /boot/grub/menu.lst
serial --unit=0 --speed= 115200
terminal --timeout=10 serial console

and the kernel line should look like:

kernel /boot/vmlinuz-2.4.20-020stab009.24.777-enterprise ro root=/dev/sda1 console=tty0 console=ttyS0,115200

Remove the line:
hiddensplashmenu

lilo

Make sure the append line in /etc/lilo.conf looks like this:

append="console=tty console=ttyS0,38400"

If you edit /etc/lilo.conf don’t forget to run lilo

# lilo
Added linux
Added linux-up
Added linux+virtuozzo *

(note: the last console listed is the one that gets output from the rc scripts.)

add to /etc/inittab and /etc/securetty

echo "s0:2345:respawn:/sbin/agetty -L ttyS0 115200 linux" >> /etc/inittab
echo "ttyS0" >> /etc/securetty

kill -1 1

8. configuring eth1 for management net:

echo "10.1.5/24 via 10.1.4.2
10.1.6/24 via 10.1.4.2" >> /etc/sysconfig/network-scripts/route-eth1

reboot the machine to make sure that the interface is configured properly.

also verify that the routes are in place like so:

netstat -rn | grep 10.1.4.2
10.1.5.0 10.1.4.2 255.255.255.0 UG 40 0 0 eth1
10.1.6.0 10.1.4.2 255.255.255.0 UG 40 0 0 eth1

9. make sure the hostname on the system is properly set - in the form virt4.johncompanies.com - and make sure it is set properly:
cat /etc/sysconfig/network

10. Check/update fstab

vi /etc/fstab

/etc/fstab entries should look something like (note noatime option):

/dev/sda3 /vz ext3 defaults,noatime 1 2
/dev/sdb1 /vz1 ext3 defaults,noatime 1 2

make sure swap is configured in fstab

Use real devices instead of LABEL

11. Install Virtuozzo 3.0

If it wasn’t done at install time:

setenforce 0

umount /vz
umount /vz1

mke2fs -i1024 -b1024 -j /dev/sda3 &
mke2fs -i1024 -b1024 -j /dev/sdb2 &

On CentOS 4:
yum install rpm-build; yum update iproute

mount -a

scp backup2:/d4/vzrpms/iso/Rel300.tar.gz /vz/.

cd /vz

tar xzf Rel300.tar.gz

cd Rel300/HW

./install

Choose Enterprise kernel
Vzagent password: k0zub1k
DON’T reboot

vzup2date
User:
gdawson
Pass:
qjdK1R
(run again. till no more updates offered)

reboot

Run vzup2date (make sure to choose enterprise kernel) and/or install latest kernel:

ftp downloads.swsoft.com
cd /updates/Rel300/20061206
bin
get vzkernel-enterprise-2.6.9-023stab033.9.i686.rpm
get vzmodules-enterprise-2.6.9-023stab033.9.i686.rpm
bye

rpm -ivh vzkernel-enterprise-2.6.9-023stab033.9.i686.rpm \
vzmodules-enterprise-2.6.9-023stab033.9.i686.rpm

vi /boot/grub/menu.lst
(set default kernel to new kernel)

12. Install various RPMs

scp backup2:/mnt/data4/linuxrpms/* /tmp

install them in this order:

12a. http fetcher

rpm -ivh /tmp/http_fetcher-1.1.0-1.i386.rpm
rpm -ivh /tmp/fetch-1.0-1.i386.rpm

(ignore error on CentOS4)

Now the fetch command, which vemake calls to email the new system owner, will work (it downloads the welcome email from www.johncompanies.com

12b. ntp client

sometimes ntp isn’t included in which case:

FCx:

rpm -ivh /tmp/libcap-1.10-22.i386.rpm
rpm -ivh /tmp/compat-readline43-4.3-2.i386.rpm
rpm -ivh /tmp/ntp-4.1.2-5.i386.rpm

CentOS4:

rpm -ivh /tmp/ntp-4.1.2-5.i386.rpm

rm -f /etc/ntp.conf

Create a new file with one line:

echo "server 10.1.4.5" > /etc/ntp.conf

Then set ntpd so it will start at boot:

chkconfig --level 3 ntpd on
service ntpd start

Make sure it’s talking to the ntp server:

ntpq -p

Correct output should have something unfer the refid col:

remote refid st t when poll reach delay offset jitter
==============================================================================
10.1.4.105 clock1.redhat.c 2 u - 64 1 0.370 -706677 0.008

12c. adaptec machines only: install aaccli for raid controller (LSI card util is with bin files):

rpm -ivh /tmp/aacapps-4.1-0.i386.rpm

May have to run:

cd /dev
./MAKEDEV.aac aac0

13. copy over binaries/scripts from backup2

scp backup2:/mnt/data4/bin/linux/* /usr/local/sbin/

14. setup root crontab (alter backup time):
crontab -e
1 1 * * * /usr/local/sbin/backup
*/15 * * * * /usr/local/sbin/vim.watchdog.30 > /dev/null
#56 4,16 * * * root /usr/sbin/vzkeyupdate -c

On Perc5/i machines:

*/5 * * * * /usr/local/sbin/perc5iraidchk

15. setup backups

echo '#!/bin/sh
backupdir=/mnt/data4/virt19

## ENTRY /etc
## ENTRY /usr/local/sbin
## ENTRY /root/logs
' > /usr/local/sbin/backup.config

16. setup bb

edit passwd and add the bb user:
echo "bb:x:1984:1984:Big Brother:/home/bb:/bin/bash" >> /etc/passwd

edit group and add the bb group:
echo "bb:x:1984:" >> /etc/group

rebuild the password database:

pwconv

Create bb’s home dir and set ownership:

mkdir /home/bb
chown bb.bb /home/bb

Go to bb’s home and fetch the client tarball:

cd ~bb
scp backup2:/mnt/data4/build/bb/bb-linux.tar .

tar xf bb-linux.tar

cd bbc1.9e-btf/etc/

echo "10.1.4.5 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh" > /home/bb/bbc1.9e-btf/etc/bb-hosts
echo "10.1.4.69 virt19.johncompanies.com # ssh" >> /home/bb/bbc1.9e-btf/etc/bb-hosts

chmod +r /var/log/messages

cd ..
su bb
./runbb.sh start
exit

Add this line to /etc/rc.d/rc.local so bb will start at boot:

echo "su - bb -c \"cd /home/bb/bbc1.9e-btf; ./runbb.sh start\"" >> /etc/rc.d/rc.local

17. Uncomment/add lines with the appropriate IP addresses in /etc/ssh/sshd_config:

vi /etc/ssh/sshd_config
ListenAddress 69.55.232.162
ListenAddress 10.1.4.66

And make sure to uncomment
Protocol 2,1

18. setup backup2 nfs mounts

setup portmap to start with server, and start it:

chkconfig --level 3 portmap on
service portmap start

mb mount

run df to confirm that nfs mounts are mounted

19. m ake alias for /vzconf
ln -s /etc/sysconfig/vz-scripts /vzconf

20. copy over the /etc/sysconfig/vz-scripts/4unlimited template file from the last virt server and put it in /etc/sysconfig/vz-scripts

scp 10.1.4.68:/etc/sysconfig/vz-scripts/4unlimited /etc/sysconfig/vz-scripts/

NOTE: if this system is more powerful (RAM, CPU) then the previous, you will want to create a new one using vzsplit:

cd /etc/sysconfig/vz-scripts/
vzsplit -f 4unlimited

21. edit /etc/sysconfig/vz

vi /etc/sysconfig/vz

enable fastboot:
VZFASTBOOT=yes

add/edit to traffic parameters section:
TRAFFIC_SHAPING=yes
BANDWIDTH="eth0:102400"
TOTALRATE="eth0:1:102400"

for 2 nics, uncomment this line:
VE_ROUTE_SRC_DEV="eth0"

Make sure the IPTABLES section looks like:
IPTABLES="ip_tables iptable_filter iptable_mangle ipt_limit ipt_multiport ipt_tos ipt_TOS ipt_REJECT ipt_tcpmss ipt_ttl ipt_LOG ipt_length ip_conntrack ip_conntrack_ftp ip_conntrack_irc ipt_conntrack ipt_state ipt_helper iptable_nat ip_nat_ftp ip_nat_irc ipt_REDIRECT"

22. disable periodic fsck's, lower the root padding on the /vz and /vz1 filesystems to 100MB

tune2fs -i 0 /dev/sda3
tune2fs -i 0 /dev/sdb2
tune2fs -c -1 /dev/sda3
tune2fs -c -1 /dev/sdb2
tune2fs -r 100000 /dev/sda3
tune2fs -r 100000 /dev/sdb2

23. copy over our versions of the vzpp files:

/vz/private/1/root/var/vzcp/static/vz/skins/
/vz/private/1/root/etc/vzcp/pp/menu.xml
/vz/private/1/root/etc/vzcp/pp/dashboard.xml
/vz/private/1/root/etc/vzcp/vzcpcon.conf

eg (2.6.1 or 3.0):
rsync -va -e ssh backup2:/d4/build/vzcp/var/vzcp/static/vz/skins/ /vz/private/1/root/var/vzcp/static/vz/skins/
rsync -va -e ssh backup2:/d4/build/vzcp/etc/vzcp/pp/menu.xml /vz/private/1/root/etc/vzcp/pp/menu.xml
rsync -va -e ssh backup2:/d4/build/vzcp/etc/vzcp/pp/dashboard.xml /vz/private/1/root/etc/vzcp/pp/dashboard.xml
rsync -va -e ssh backup2:/d4/build/vzcp/etc/vzcp/vzcpcon.conf /vz/private/1/root/etc/vzcp/vzcpcon.conf

To disable backups from scratch:

In svcve edit /etc/vzcp/vzcpcon.conf

In section:

<auth>
<type>VEUser</type>
<user>root</user>

Add into/below:
<cred>
<id>maintenance</id>
</cred>

The following:
<cred>
<id>show</id>
</cred>
<cred>
<id>reinstall</id>
</cred>
<cred>
<id>repair_mode</id>
</cred>

Will finally look like:

<cred>
<id>maintenance</id>
<cred>
<id>show</id>
</cred>
<cred>
<id>reinstall</id>
</cred>
<cred>
<id>repair_mode</id>
</cred>
</cred>
</cred>

24. template install

mb mount

install debian 3.0 templates

cd /backup4/vzrpms/distro/debian/3.0/p1/
for f in `cat /backup4/vzrpms/distro/debian/3.0/install_list`; do rpm -ivh $f; done

install debian 3.1 templates

cd /backup4/vzrpms/distro/debian/3.1/p3/
for f in `ls`; do rpm -ivh $f; done

install redhat templates

cd /backup4/vzrpms/distro/rh9/current/
for f in `cat /backup4/vzrpms/distro/rh9/install_list`; do rpm -ivh $f; done

install fedora1 templates

cd /backup4/vzrpms/distro/fc1/current/
for f in `cat /backup4/vzrpms/distro/fc1/install_list`; do rpm -ivh $f; done

change 'krb5-devel-1.3.4-5' to 'krb5-devel-1.3.3-6' in (applies to pre-p5)
/vz/template/fedora-core-devel-1/conf/fedora-core-devel-1.conf.20050314

install fedora2 templates

cd /backup4/vzrpms/distro/fc2/current/
for f in `cat /backup4/vzrpms/distro/fc2/install_list`; do rpm -ivh $f; done

install suse9.2 templates

cd /backup4/vzrpms/distro/suse/current/
for f in `cat /backup4/vzrpms/distro/suse/install_list`; do rpm -ivh $f; done

install centos templates

cd /backup4/vzrpms/distro/centos/current/
for f in `cat /backup4/vzrpms/distro/centos/install_list`; do rpm -ivh $f; done

install fedora4 templates

cd /backup4/vzrpms/distro/fc4/current/
for f in `cat /backup4/vzrpms/distro/fc2/install_list`; do rpm -ivh $f; done

vzpkgcache

To pull templates from another machine (before hand stop ve 1 and move it’s template so you preserve the latest template files just setup by vzup2date):

rsync -a -e ssh 10.1.4.68:/vz/template/ /vz/template/

Make sure ubuntu source is defined:

UBU_SERVER=http://archive.ubuntu.com

And change FC source:
FC_SERVER=http://rhold.fedoraproject.org

In /etc/vztt/vztt.conf

rsync -a -e ssh 10.1.4.68:/etc/vztt/vztt.conf /etc/vztt/

To update the cache:

vzpkg update cache ubuntu-6.10-x86

25. add tun device support

Add to /etc/modules.conf:
echo "alias char-major-10-200 tun" >> /etc/modules.conf

depmod -a

echo "modprobe tun" >> /etc/rc.local

26. add traffic shaping support

Re-define class 0:

vi /etc/sysconfig/vz-scripts/networks_classes
0 69.55.224.0/20
0 10.1.4.0/24

make sure 1 is defined (0.0.0.0) and no other classes are needed

Then start it

/etc/init.d/vz shaperrestart

27. RAIDmon (only applies to LSI 320 PCI card)

cd /tmp
cp /backup4/build/megaraid/linuxmon30.tgz .
tar xzf linuxmon30.tgz
cd linuxmon30
./install
(say no)
mv –f raidmon.init /etc/init.d/raidmon
service raidmon start
tail -f /var/log/messages
(check for output)

28. udp dos fix

echo "/usr/local/sbin/iptables.sh" >> /etc/rc.d/rc.local

29. make rc.local start before vz

mv /etc/rc.d/rc3.d/S99local /etc/rc.d/rc3.d/S95local

30. disable unwanted services/daemons

On CentOS:
chkconfig --del kudzu
chkconfig --del cpuspeed

This should be taken care of by vz install so probably safe to skip

chkconfig --del kudzu
chkconfig --del cups
chkconfig --del gpm
chkconfig --del isdn
chkconfig --del pcmcia
chkconfig --del apmd
chkconfig --del rhnsd
chkconfig --del mdmonitor

31. load mrtg graphs

On FCx xinetd not installed so:

rpm -ivh /tmp/xinetd-2.3.11-1.7x.i386.rpm

create /etc/xinetd.d/load with the following contents:

echo "service load
{
disable = no
flags = REUSE
socket_type = stream
wait = no
user = root
server = /usr/local/sbin/load.pl
}" > /etc/xinetd.d/load

Add this line to /etc/services:
echo "load 12384/tcp" >> /etc/services

On FCx: if xinetd isn’t running, configure it for startup at boot time:

chkconfig --level 3 xinetd on
service xinetd start

32. Tasks to do on backup2:

setup backup dirs:
mkdir -p /mnt/data4/virt19/0

add the system to
vi /mnt/data4/bin/rsync_houseclean

33. Tasks to do on mail:

Update backup graph:
vi /usr/local/www/mgmt/cgi/backupgraph.pl
(add hostname)

add to bbhosts:
vi /usr/home/bb/bbsrc/bb1.9e-btf/etc/bb-hosts
su bb
cd
bbsrc/bb/runbb.sh restart ; exit

add to load graph config:
vi /usr/local/www/mgmt/mrtg/mrtg1.cfg
(add new section for new machine)

34. add to management db (on mail and devweb) jc.ref_machines and jc.ref_templates

insert into ref_machines values (null,'virt19','virt19.johncompanies.com',0,'l');
select * from ref_machines;
insert into systems values ('', 'col00001',43,null,null,100,'virt19.johncompanies.com',null,'2008-01-07',null,null,1,0,null,null,null);
select * from systems where hostname='virt19.johncompanies.com';
update ipmap set cid='col00001', sysid=3772, assign_date='2008-01-07', status=1 where ip='69.55.236.2';
update ipmap set cid='col00001', sysid=3772, assign_date='2008-01-07', status=1 where ip='69.55.230.109';
insert into ref_templates values ('','fedora-core-1/20050725',33,'Fedora Core 1',1);
insert into ref_templates values ('','fedora-core-2/20050725',33,'Fedora Core 2',1);
insert into ref_templates values ('','debian-3.1/20050927',33,'Debian 3.1',1);

35. Make sure that ns1c has proper forward and reverse entries for the new system (both private and public addresses), so that new system emailing will work.
vr johncompanies.com
vi internal.johncompanies.com
rndc reload johncompanies.com IN private
(edit the PTR too)

36. add an outside blocking rule to the firewall, so this machine can only be reached from inside the firewall.

Follow example already in firewall virt19 is:
ipfw add 00149 allow ip from { 69.55.225.225 or 69.55.230.2 or 69.55.230.10 or 69.55.230.3 or 62.213.72.101 or 64.131.90.27 or 80.89.128.178 or 82.200.65.190 or 84.237.120.254 or 89.251.97.2 or 195.214.232.10 or 195.214.233.10 or 208.251.212.1 or 213.130.28.26 } to 69.55.236.2
ipfw add 00149 deny ip from any to 69.55.236.2

virt12 would be 00142

37. add to server/cabinet map

39. make /vz1/private
mkdir /vz1/private ; chmod 0755 /vz1/private
mkdir /vz1/root ; chmod 0755 /vz1/root</pre>

Virtuozzo Server Install

2013-11-14T21:54:33Z

24.165.90.109: /* Configure system, install virtuozzo */

= 4.7 =

Server-specific changes in <span style="color:red">red</span>

http://www.parallels.com/download/pvc/lin

http://www.parallels.com/support/virtualization-suite/pvc/

== BIOS ==

* enable post info to console

2950:<br>
Serial comm.: on with cons. Redir via com1<br>
Failsafe: 115200<br>
Redir after boot: enabled

* Tweak DRAC settings (see [[DRAC/RMM#DRAC_setup|DRAC setup]])

== Install OS ==

CentOS6.x (get latest from http://mirrors.kernel.org/centos/6.4/isos/x86_64/CentOS-6.4-x86_64-netinstall.iso)

May have to disable DRAC virtual media to make sure we get sda as our first device

Set partitions:
<pre>Disk 1:
/ 10240M (ext4)
9216M swap
/vz (the rest) (ext4)
Disk 2:
9216M Swap
/vz1 (the rest) (ext4)</pre>

Basic server

Use grub, install to MBR

== Configure system, install virtuozzo ==
* add ssh

yum install openssh-clients

* create ssh key, add swsoft key to authorized_keys

(enter to everything)

<pre>cd /root/
ssh-keygen -t dsa

Add vz support key:
echo "ssh-dss AAAAB3NzaC1kc3MAAACBAIB7XHmpcsa1D/vsKS+mzEvBVo7udew/aRahbjnjywL4YXT7ruWzpGo7R3UA8GKIgVHoQGUbuXrJz5ICtBEXOlbgXVEz7Bu5nXOG2C9i/8pmFtUw6GwACII8uZE7t/Zq5CRj+5zN1Rv7n/4X4RAjzk5BHXoZ62a32FTbS4tPvyPrAAAAFQDiUi7YEiffNzGvB544lQpRdn29TwAAAIBkUchLTBE1+sX11iC80yhNh9EGeCMLdkT8UqHsTSaD0myQPxGALuNY4rO31q+25uKqSRDTXd6ZWXp4bplA6Ls6jl7r7XVR9loJ7WvRMqc4Dv68TZ7h2UBqAhu4det2UF4LHim3DNf1gKn9DzWNkmI/N2FvVVAL4aM3sPunVIvUYgAAAIAJX38INABVgnCEBKqpv5FMLm3k9MgMn54oQFmylBbgM0w4SUeuDj3jLrkJf8dT9w8ByEDb8xviexpWrPsbz2DeThle4xgzcyKrUkt2Wugtmm4DwS0S/68BAA9wXoWD/vZ67GBoCKH7uq8VAjUVcCbJ/7IydljnFAZoJpktdwN7iQ== russup@support.sw-soft.com" >> /root/.ssh/authorized_keys</pre>

* setup access to backup2:

<pre>echo "10.1.4.3 backup2" >> /etc/hosts
echo "10.1.4.8 backup1" >> /etc/hosts</pre>

# Copy id_dsa.pub to backup2 and add it to the end of ~root/.ssh/authorized_keys:

<pre>cat .ssh/id_dsa.pub | ssh backup2 'cat - >> /root/.ssh/authorized_keys'
cat .ssh/id_dsa.pub | ssh backup1 'cat - >> /root/.ssh/authorized_keys'</pre>

<pre>ssh backup2 hostname
ssh backup1 hostname</pre>
(confirm you can connect w/o entering a password)

* add bash backward history search
<pre>echo "\"\M-z\": history-search-backward
\"\M-x\": history-search-backward " > /root/.inputrc
echo "export GREP_OPTIONS='--color=auto'" >> ~/.bashrc
echo "\"\e[5~\": history-search-backward" >> ~/.inputrc
echo "\"\e[6~\": history-search-forward" >> ~/.inputrc</pre>

* add to /root/.bashrc, create log dir

<pre>vi /root/.bashrc
alias v='vzctl'
alias mbm='mb mount'
alias mbu='mb umount'
export PS1="[\u@\h \w]# "
alias cv='cd /vz/private'
alias cv1='cd /vz1/private'
alias cv2='cd /vz2/private'
alias pg='ps auxw|grep $1'</pre>

source /root/.bashrc

mkdir /root/logs

* make sure serial console and kernel messages are working

add these 2 lines to the top:

<pre>vi /boot/grub/menu.lst
serial --unit=0 --speed= 115200
terminal --timeout=10 serial console</pre>

and the kernel line should look like:

kernel /boot/vmlinuz-2.4.20-020stab009.24.777-enterprise ro root=/dev/sda1 console=tty0 console=ttyS0,115200

Remove or comment out these lines:
<pre>hiddensplashmenu
splashimage=(hd0,0)/boot/grub/splash.xpm.gz</pre>

add to /etc/inittab and /etc/securetty

<pre>echo "s0:2345:respawn:/sbin/agetty -L ttyS0 115200 linux" >> /etc/inittab
echo "ttyS0" >> /etc/securetty</pre>

kill -1 1

* make sure the hostname on the system is properly set - in the form virt4.johncompanies.com - and make sure it is set properly:
cat /etc/sysconfig/network

* Check/update fstab

vi /etc/fstab

/etc/fstab entries should look something like (note noatime option):

<pre>/dev/sda3 /vz ext4 defaults,noatime 1 2
/dev/sdb2 /vz1 ext4 defaults,noatime 1 2</pre>

make sure swap is configured in fstab

Use real devices instead of LABEL

* Install Virtuozzo 4.7

Have a license on hand - request a day ahead of the install. Go to:
http://www.parallels.com/download/pvc47/

<pre>yum install compat-libstdc++-33
yum install wget
yum install ftp
yum install pciutils
yum install ed bc

cd /vz
wget http://download.parallels.com/pvc/47/lin/vzinstall-linux-x86_64.bin
chmod 0700 vzinstall-linux-x86_64.bin
./vzinstall-linux-x86_64.bin</pre>

download and install

change wd's to /vz

Ignore OS version error

Custom install (don’t install templates)

Install PVA Agent
Install PVA Management Node

reboot

Get on KVM to watch PVA install (won't happen on console)

vzup2date

(run again. till no more updates offered)

reboot

<DEPRECATED>
Run vzup2date (make sure to choose enterprise kernel) and/or install latest kernel:

NOTE: skip over the manual kernel load below if you can get the 16G recognized before running vzup2date- then it will dl the ent kernel

NOTE: go to /vz/vzup2date/virtuozzo/linux/i386/4.0.0/CU-2.6.18-028stab068.3 and install kern there (downloaded by vzup2date)

vi /boot/grub/menu.lst
(set default kernel to new kernel)
</DEPRECATED>

* Install ntp

yum install ntp
(if not already installed)

rm -f /etc/ntp.conf

Create a new file with one line:

echo "server 10.1.4.5" > /etc/ntp.conf

Then set ntpd so it will start at boot:

chkconfig --level 3 ntpd on
service ntpd start

Make sure it’s talking to the ntp server:

ntpq -p

Correct output should have something unfer the refid col and a '+' next to the remote server:

<pre>remote refid st t when poll reach delay offset jitter
==============================================================================
10.1.4.105+ clock1.redhat.c 2 u - 64 1 0.370 -706677 0.008</pre>

* copy over binaries/scripts from backup2

scp backup2:/mnt/data4/bin/linux/* /usr/local/sbin/
scp <span style="color:red">10.1.4.62</span>:"/vzconf/jctmpl*" /etc/sysconfig/vz-scripts/
scp <span style="color:red">10.1.4.62</span>:/vzconf/ve-slm* /etc/sysconfig/vz-scripts/

* setup root crontab (alter backup time):
<pre>crontab -e
1 1 * * * /usr/local/sbin/backup
*/15 * * * * /usr/local/sbin/vim.watchdog.30 > /dev/null
#56 4,16 * * * root /usr/sbin/vzkeyupdate -c
*/5 * * * * /usr/local/sbin/perc5iraidchk64</pre>

* setup backups

echo '#!/bin/sh
<tt>backupdir=/data/<span style="color:red">virt13</span>
server=backup1
<pre>
## ENTRY /etc
## ENTRY /usr/local/sbin
## ENTRY /root/logs
</pre>
' > /usr/local/sbin/backup.config</tt>

* setup bigbrother

edit passwd and add the bb user:
echo "bb:x:1984:1984:Big Brother:/home/bb:/bin/bash" >> /etc/passwd

edit group and add the bb group:
echo "bb:x:1984:" >> /etc/group

rebuild the password database:
pwconv

Create bb’s home dir and set ownership:
mkdir /home/bb
chown bb.bb /home/bb

Go to bb’s home and fetch the client tarball:
<pre>cd ~bb
scp backup2:/mnt/data4/build/bb/GOOD-bb-linux-bbc1.9i-btf.tgz .

tar xvf GOOD-bb-linux-bbc1.9i-btf.tgz

yum install glibc-2.12-1.47.el6_2.5.i686

vi etc/bbsys.sh
SORT="/bin/sort"

vi bin/bb-disk.sh
eval "$DFCMD | $EGREP \"$DFUSE\" | $EGREP -v \"$DFEXCLUDE\" | $SORT \"-k${DFSORT}\" > $BBTMP/DFOUT.$$"
wclen1=`eval "$DFCMD | $EGREP \"$DFUSE\" | $EGREP -v \"$DFEXCLUDE\" | $SORT \"-k${DFSORT}\" | $WC"`

cd bbc1.9i-btf/etc/

echo "10.1.4.5 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh" > /home/bb/bbc1.9i-btf/etc/bb-hosts
echo "10.1.4.63 virt13.johncompanies.com # ssh" >> /home/bb/bbc1.9i-btf/etc/bb-hosts
(^^^change to correct host)

chmod +r /var/log/messages

cd ..
su bb
./runbb.sh start
exit</pre>

Add this line to /etc/rc.d/rc.local so bb will start at boot:

echo "su - bb -c \"cd /home/bb/bbc1.9i-btf; ./runbb.sh start\"" >> /etc/rc.d/rc.local

* Uncomment/add lines with the appropriate IP addresses in /etc/ssh/sshd_config:

vi /etc/ssh/sshd_config
<tt>ListenAddress <span style="color:red">69.55.226.2</span>
ListenAddress <span style="color:red">10.1.4.63</span></tt>

<DEPRECATED>
And make sure to uncomment
Protocol 2,1
</DEPRECATED>

* setup backup1/backup2 nfs mounts

<DEPRECATED>
setup portmap to start with server, and start it:

chkconfig --level 3 portmap on
service portmap start
</DEPRECATED>

mb mount

run df to confirm that nfs mounts are mounted

* make alias for /vzconf
ln -s /etc/sysconfig/vz-scripts /vzconf

* edit /etc/sysconfig/vz

vi /etc/sysconfig/vz

calc cpu to reserver for HN. take 5% of power:
vzcpucheck
Current CPU utilization: 27000
Power of the node: 1018320

so in this case:
VE0CPUUNITS=50916

enable fastboot:
VZFASTBOOT=yes

Disable auto mount
#VZMOUNTS="/vz"

Enable slm:
SLM=yes

<DEPRECATED>
Enable IPV6
IPV6=yes
</DEPRECATED>

add/edit to traffic parameters section:
TRAFFIC_SHAPING=yes
BANDWIDTH="eth0:102400"
TOTALRATE="eth0:1:102400"
RATE="eth0:1:50000"

for 2 nics, uncomment this line:
VE_ROUTE_SRC_DEV="eth0"

Make sure the IPTABLES section looks like:
<pre>IPTABLES="ip_tables iptable_filter iptable_mangle ipt_limit ipt_multiport ipt_tos ipt_TOS ipt_REJECT ipt_tcpmss ipt_ttl ipt_LOG ipt_length ip_conntrack ip_conntrack_ftp ip_conntrack_irc ipt_conntrack ipt_state ipt_helper iptable_nat ip_nat_ftp ip_nat_irc ipt_REDIRECT ipt_multiport ipt_TCPMSS"</pre>

* disable periodic fsck's, lower the root padding on the /vz and /vz1 filesystems to 100MB
<pre>tune2fs -i 0 /dev/sda3
tune2fs -i 0 /dev/sdb2
tune2fs -c -1 /dev/sda3
tune2fs -c -1 /dev/sdb2
tune2fs -r 100000 /dev/sda3
tune2fs -r 100000 /dev/sdb2</pre>

* setup vzagent with pub IP (to allow migrations from 3.x):
vzctl set 50 --ipdel all --ipadd <span style="color:red">69.55.226.3</span> --userpasswd vzagent0:k0zub1k --save
vzagent_ctl restart

* edit ve config files:

From split of 80 take
<tt>CPUUNITS</tt>

And put into files in /vzconf/base_ct_conf
along with:
<pre>VERSION="2"

ONBOOT="yes"

KMEMSIZE="2147483647:2147483647"
LOCKEDPAGES="2147483647:2147483647"
PRIVVMPAGES="2147483647:2147483647"
SHMPAGES="2147483647:2147483647"
NUMPROC="32567:32567"
VMGUARPAGES="2147483647:2147483647"
OOMGUARPAGES="2147483647:2147483647"
NUMTCPSOCK="2147483647:2147483647"
NUMFLOCK="2147483647:2147483647"
NUMPTY="255:255"
NUMSIGINFO="1024:1024"
TCPSNDBUF="2147483647:2147483647"
TCPRCVBUF="2147483647:2147483647"
OTHERSOCKBUF="2147483647:2147483647"
DGRAMRCVBUF="2147483647:2147483647"
NUMOTHERSOCK="2147483647:2147483647"
DCACHESIZE="2147483647:2147483647"
NUMFILE="2147483647:2147483647"
NUMIPTENT="2147483647:2147483647"

DISKSPACE=""
DISKINODES=""
QUOTATIME="600"
IOPRIO="4"
CPULIMIT="75"
OFFLINE_MANAGEMENT="yes"
ARCH="x86"
PLATFORM="linux"

IP_ADDRESS="0.0.0.0"
NAMESERVER="69.55.225.225 69.55.230.3"
CPUUNITS="12313"
IOLIMIT="10485760"
IOPSLIMIT="100"
</pre>

* customize the control panel:
rm -fr /vz/backups

vi /var/opt/pva/pp/xsl/lib/vt/backups-forms.xsl

<pre>
<span style="font-size:13px; font-weight:bold">NOTE: You cannot create or view backups via this control panel- IT WILL NOT WORK. Please contact linux@johncompanies.com to setup or inquire about backups for your server.</span>

/var/opt/pva/pp/xsl/lib/vt/backups-list.xsl
</pre>

cd /var/opt/pva/pp/static/vz/skins
for f in `ls`; do scp backup2:/d4/build/vzcp/var/vzcp/static/vz/skins/winxp.new/images/logo_main_4.7.gif $f/images/logo_main.gif; done

* template install

To install new templates:

vzup2date -z

Copy over templates from old server. Ex: ubu 8.04 from virt19:

rsync -a -e ssh <span style="color:red">10.1.4.69</span>:/vz/template/ubuntu/8.04 /vz/template/ubuntu/8.04

install ubuntu 8.04 templates (get latest from vz)

<pre>cd /backup4/vzrpms/distro/ubuntu/8.04
rpm -ivh *
vzpkg update cache ubuntu-8.04-x86</pre>

If you install a tempalte from vzup2date it will download an old cache as well. Better to download the OS rpm from VZ site then create cache then use vzup2date to install the application templates.

* add traffic shaping support

Re-define class 0:

<pre>vi /etc/sysconfig/vz-scripts/networks_classes
0 69.55.224.0/20
0 10.1.4.0/24</pre>

make sure 1 is defined (0.0.0.0) and no other classes are needed

Then start it

/etc/init.d/vz shaperrestart

* udp dos fix

echo "/usr/local/sbin/iptables.sh" >> /etc/rc.d/rc.local

* make rc.local start before vz

mv /etc/rc.d/rc3.d/S99local /etc/rc.d/rc3.d/S95local

* disable unwanted services/daemons

Before:
<pre>abrt-ccpp 0:off 1:off 2:off 3:on 4:off 5:on 6:off
abrt-oops 0:off 1:off 2:off 3:on 4:off 5:on 6:off
abrtd 0:off 1:off 2:off 3:on 4:off 5:on 6:off
acpid 0:off 1:off 2:on 3:on 4:on 5:on 6:off
atd 0:off 1:off 2:off 3:on 4:on 5:on 6:off
auditd 0:off 1:off 2:on 3:on 4:on 5:on 6:off
autofs 0:off 1:off 2:off 3:on 4:on 5:on 6:off
cpuspeed 0:off 1:on 2:on 3:on 4:on 5:on 6:off
crond 0:off 1:off 2:on 3:on 4:on 5:on 6:off
cups 0:off 1:off 2:on 3:on 4:on 5:on 6:off
haldaemon 0:off 1:off 2:off 3:on 4:on 5:on 6:off
ip6tables 0:off 1:off 2:on 3:on 4:on 5:on 6:off
iptables 0:off 1:off 2:on 3:on 4:on 5:on 6:off
irqbalance 0:off 1:off 2:off 3:on 4:on 5:on 6:off
kdump 0:off 1:off 2:off 3:on 4:on 5:on 6:off
lvm2-monitor 0:off 1:on 2:on 3:on 4:on 5:on 6:off
mcelogd 0:off 1:off 2:off 3:on 4:off 5:on 6:off
mdmonitor 0:off 1:off 2:on 3:on 4:on 5:on 6:off
messagebus 0:off 1:off 2:on 3:on 4:on 5:on 6:off
netfs 0:off 1:off 2:off 3:on 4:on 5:on 6:off
network 0:off 1:off 2:on 3:on 4:on 5:on 6:off
nfslock 0:off 1:off 2:off 3:on 4:on 5:on 6:off
ntpd 0:off 1:off 2:off 3:on 4:off 5:off 6:off
portreserve 0:off 1:off 2:on 3:on 4:on 5:on 6:off
postfix 0:off 1:off 2:on 3:on 4:on 5:on 6:off
pvaagentd 0:off 1:off 2:on 3:on 4:on 5:on 6:off
pvapp 0:off 1:off 2:on 3:on 4:on 5:on 6:off
qpidd 0:off 1:off 2:on 3:on 4:on 5:on 6:off
rpcbind 0:off 1:off 2:on 3:on 4:on 5:on 6:off
rpcgssd 0:off 1:off 2:off 3:on 4:on 5:on 6:off
rpcidmapd 0:off 1:off 2:off 3:on 4:on 5:on 6:off
rsyslog 0:off 1:off 2:on 3:on 4:on 5:on 6:off
sshd 0:off 1:off 2:on 3:on 4:on 5:on 6:off
sysstat 0:off 1:on 2:on 3:on 4:on 5:on 6:off
udev-post 0:off 1:on 2:on 3:on 4:on 5:on 6:off
vz 0:off 1:off 2:on 3:on 4:on 5:on 6:off
vzlmond 0:off 1:off 2:on 3:on 4:on 5:on 6:off
vznetcfg 0:off 1:off 2:on 3:on 4:on 5:on 6:off
xinetd 0:off 1:off 2:off 3:on 4:on 5:on 6:off</pre>

<pre>chkconfig --level 3 cpuspeed off
chkconfig --level 3 cups off
chkconfig --level 3 haldaemon off
chkconfig --level 3 ip6tables off
chkconfig --level 3 iptables off
chkconfig --level 3 irqbalance off
chkconfig --level 3 lvm2-monitor off
chkconfig --level 3 mdmonitor off
chkconfig --level 3 messagebus off
chkconfig --level 3 netfs off
chkconfig --level 3 nfslock off
chkconfig --level 3 qpidd off
chkconfig --level 3 rpcgssd off
chkconfig --level 3 rpcidmapd off</pre>

* load mrtg graphs

yum install xinetd

create /etc/xinetd.d/load with the following contents:

<pre>echo "service load
{
disable = no
flags = REUSE
socket_type = stream
wait = no
user = root
server = /usr/local/sbin/load.pl
}" > /etc/xinetd.d/load</pre>

Add this line to /etc/services:
echo "load 12384/tcp" >> /etc/services

* Tasks to do on backup1:
setup backup dirs:

mkdir -p /data/<span style="color:red">virt13</span>/0

for f in 0 1 2 3 4 5 6; do mkdir -p /data/<span style="color:red">virt13</span>/$f/vz/private; mkdir -p /data/<span style="color:red">virt13</span>/$f/vz1/private; mkdir -p /data/<span style="color:red">virt13</span>/$f/vz2/private; done

add the system to
vi /usr/local/sbin/snapshot_rotate

* Tasks to do on mail:

Update backup graph:
vi /usr/local/www/mgmt/cgi/backupgraph.pl
(add hostname)

add to bbhosts:
<pre>vi /usr/home/bb/bbsrc/bb1.9i-btf/etc/bb-hosts
su bb
cd
bbsrc/bb/runbb.sh restart ; exit</pre>

add to load graph config:
vi /usr/local/www/mgmt/mrtg/mrtg1.cfg
(add new section for new machine)

* add to management db (on mail and devweb) jc.ref_machines and jc.ref_templates

insert new server:
insert into ref_machines values (null,'virt19','virt19.johncompanies.com',0,'l');

get new machine_id back out:
select * from ref_machines;

insert as a system to jc customer col00001:
insert into systems values ('', 'col00001',30,null,null,100,'virt11.johncompanies.com',null,'2010-03-11',null,null,1,0,null,null,null);

get sysid:
select * from systems where hostname='virt19.johncompanies.com';

update ipmap (main IP and IP used for VZCP):
update ipmap set cid='col00001', sysid=3772, assign_date='2008-01-07', status=1 where ip='69.55.236.2';
update ipmap set cid='col00001', sysid=3772, assign_date='2008-01-07', status=1 where ip='69.55.230.109';

add templates (or do via mgmt):
<pre>
insert into ref_templates values ('','fedora-core-1/20050725',33,'Fedora Core 1',1);
insert into ref_templates values ('','fedora-core-2/20050725',33,'Fedora Core 2',1);
insert into ref_templates values ('','debian-3.1/20050927',33,'Debian 3.1',1);</pre>

* Make sure that ns1c has proper forward and reverse entries for the new system (both private and public addresses), so that new system emailing will work.
<pre>vr johncompanies.com
vi internal.johncompanies.com
rndc reload johncompanies.com IN private</pre>
(edit the PTR too)

* add an outside blocking rule to the firewall, so this machine can only be reached from inside the firewall.

Follow example already in firewall virt19 is:

<pre>ipfw add 00143 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 or 62.213.72.101 or 64.131.90.27 or 82.200.65.190 or 91.204.24.0/22 or 194.44.160.178 or 195.214.232.10 or 195.214.233.10 or 208.251.212.1 } to 69.55.226.2
ipfw add 00143 deny ip from any to 69.55.226.2</pre>

(virt12 would be 00142)

* add to [[Cabinetmap|Cabinetmap]]

* make /vz1/private
<pre>mkdir /vz1/private ; chmod 0755 /vz1/private
mkdir /vz1/root ; chmod 0755 /vz1/root</pre>

* put config files in a place they’ll get backed up
<pre>mkdir /vzconf/confs
crontab -e
0 0 * * * rm -f /vzconf/confs/*.conf; cp /vzconf/*.conf /vzconf/confs</pre>

* fix pass/vzpp and localse issue with ubu 8.10

vi /vz/template/ubuntu/8.10/x86/config/os/default/post-install

sed '/^password.*pam_unix.so.*sha512/ s~sha512~md5~' -i etc/pam.d/common-password

localedef -i en_US -c -f UTF-8 en_US.UTF-8

* add tun device support

echo "modprobe tun" >> /etc/rc.local

* reinstall VZA/CT #1/CT #50

After the install, the PP was broken. So needed to reinstall everything. Did the pva_setup --uninstall and removed CT 50 and then setup the pva per http://kb.parallels.com/113071 then setup the CT 50 per the other article- centos6 had issues - and then authenticated with virt13 to get that into the management list.

http://kb.parallels.com/113072

http://kb.parallels.com/113071

* config logrotate (otherwise it creates weekly empty logs)

vi /etc/logrotate.conf

(add to end)

<pre>/var/log/messages {
rotate 12
monthly
size 10M
missingok
compress
delaycompress
notifempty
sharedscripts
postrotate
echo `date` > /var/log/messages
endscript
}</pre>

* install megacli64 (can we just move the bin and not install the lib?)

<pre>scp backup2: /d4/build/3ware/MegaCli-8.00.40-1.i386.rpm /root/MegaCli-8.00.40-1.i386.rpm
scp backup2: /d4/build/3ware/Lib_Utils-1.00-08.noarch.rpm /root/ Lib_Utils-1.00-08.noarch.rpm

rpm -ivh Lib_Utils-1.00-08.noarch.rpm
rpm -ivh MegaCli-8.00.40-1.i386.rpm

cp opt/MegaRAID/MegaCli/MegaCli64 /usr/local/sbin/megacli64</pre>

= Older versions =

== 4.0 ==

Last updated 2010-03-11

<pre>1. enable post info to console

2950:
Serial comm.: on with cons. Redir via com1
Failsafe: 115200
Redir after boot: enabled

Server
Custom- desel all but base
/root/install.log

2. Install OS

CentOS5.x (get latest from http://mirrors.kernel.org/centos/5.4/isos/i386/)

Set partitions:
Disk 1:
/ 3072M
6144M swap
/vz (the rest)
Disk 2:
6144M Swap

Use grub, install to MBR
No firewall
Disable SElinux
Install only base

3. create ssh key, add swsoft key to authorized_keys

(enter to everything)

cd /root/
ssh-keygen -t dsa

echo "ssh-dss AAAAB3NzaC1kc3MAAACBAIB7XHmpcsa1D/vsKS+mzEvBVo7udew/aRahbjnjywL4YXT7ruWzpGo7R3UA8GKIgVHoQGUbuXrJz5ICtBEXOlbgXVEz7Bu5nXOG2C9i/8pmFtUw6GwACII8uZE7t/Zq5CRj+5zN1Rv7n/4X4RAjzk5BHXoZ62a32FTbS4tPvyPrAAAAFQDiUi7YEiffNzGvB544lQpRdn29TwAAAIBkUchLTBE1+sX11iC80yhNh9EGeCMLdkT8UqHsTSaD0myQPxGALuNY4rO31q+25uKqSRDTXd6ZWXp4bplA6Ls6jl7r7XVR9loJ7WvRMqc4Dv68TZ7h2UBqAhu4det2UF4LHim3DNf1gKn9DzWNkmI/N2FvVVAL4aM3sPunVIvUYgAAAIAJX38INABVgnCEBKqpv5FMLm3k9MgMn54oQFmylBbgM0w4SUeuDj3jLrkJf8dT9w8ByEDb8xviexpWrPsbz2DeThle4xgzcyKrUkt2Wugtmm4DwS0S/68BAA9wXoWD/vZ67GBoCKH7uq8VAjUVcCbJ/7IydljnFAZoJpktdwN7iQ== russup@support.sw-soft.com" >> /root/.ssh/authorized_keys

4. setup access to backup2:

echo "10.1.4.3 backup2" >> /etc/hosts
echo "10.1.4.8 backup1" >> /etc/hosts

# Copy id_dsa.pub to backup2 and add it to the end of ~root/.ssh/authorized_keys:

cat .ssh/id_dsa.pub | ssh backup2 'cat - >> /root/.ssh/authorized_keys'
cat .ssh/id_dsa.pub | ssh backup1 'cat - >> /root/.ssh/authorized_keys'

ssh backup2 hostname
ssh backup1 hostname

(confirm you can connect w/o entering a password)

5. add bash backward history search
echo "\"\M-z\": history-search-backward
\"\M-x\": history-search-backward " > /root/.inputrc
echo "export GREP_OPTIONS='--color=auto'" >> ~/.bashrc
echo "\"\e[5~\": history-search-backward" >> ~/.inputrc
echo "\"\e[6~\": history-search-forward" >> ~/.inputrc

6. add to /root/.bashrc, create log dir

vi /root/.bashrc
alias v='vzctl'
alias mbm='mb mount'
alias mbu='mb umount'
export PS1="[\u@\h \w]# "
alias cv='cd /vz/private'
alias cv1='cd /vz1/private'
alias cv2='cd /vz2/private'

source /root/.bashrc

mkdir /root/logs

7. make sure serial console and kernel messages are working

add these 2 lines to the top:

vi /boot/grub/menu.lst
serial --unit=0 --speed= 115200
terminal --timeout=10 serial console

and the kernel line should look like:

kernel /boot/vmlinuz-2.4.20-020stab009.24.777-enterprise ro root=/dev/sda1 console=tty0 console=ttyS0,115200

Remove or comment out these lines:
hiddensplashmenu
splashimage=(hd0,0)/boot/grub/splash.xpm.gz

add to /etc/inittab and /etc/securetty

echo "s0:2345:respawn:/sbin/agetty -L ttyS0 115200 linux" >> /etc/inittab
echo "ttyS0" >> /etc/securetty

kill -1 1

DEPRECATED 8. configuring eth1 for management net:

echo "10.1.5/24 via 10.1.4.2
10.1.6/24 via 10.1.4.2" >> /etc/sysconfig/network-scripts/route-eth1

reboot the machine to make sure that the interface is configured properly.

also verify that the routes are in place like so:

netstat -rn | grep 10.1.4.2
10.1.5.0 10.1.4.2 255.255.255.0 UG 40 0 0 eth1
10.1.6.0 10.1.4.2 255.255.255.0 UG 40 0 0 eth1

9. make sure the hostname on the system is properly set - in the form virt4.johncompanies.com - and make sure it is set properly:
cat /etc/sysconfig/network

10. Check/update fstab

vi /etc/fstab

/etc/fstab entries should look something like (note noatime option):

/dev/sda3 /vz ext3 defaults,noatime 1 2
/dev/sdb2 /vz1 ext3 defaults,noatime 1 2

make sure swap is configured in fstab

Use real devices instead of LABEL

11. Install Virtuozzo 4.0

umount /vz
umount /vz1

mke2fs -i1024 -b1024 -j /dev/sda3 &
mke2fs -i1024 -b1024 -j /dev/sdb2 &

Experimentally ran this for sdc on virt12:
mke2fs -i2048 -b2048 -j /dev/sdb1

mount -a

scp backup2:/d4/vzrpms/iso/vzinstall-linux.bin /vz
cd /vz
chmod 0700 vzinstall-linux.bin
./vzinstall-linux.bin

change wd to /vz
Ignore OS version error
Custom install (don’t install templates)

reboot

vzup2date

(run again. till no more updates offered)

reboot

Run vzup2date (make sure to choose enterprise kernel) and/or install latest kernel:

NOTE: skip over the manual kernel load below if you can get the 16G recognized before running vzup2date- then it will dl the ent kernel

NOTE: go to /vz/vzup2date/virtuozzo/linux/i386/4.0.0/CU-2.6.18-028stab068.3 and install kern there (downloaded by vzup2date)

<DEPREACATED>
ftp downloads.swsoft.com
User:
gdawson
Pass:
qjdK1R

cd /updates/Rel400/20080307
bin
get vzkernel-ent-2.6.18-028stab053.10.i686.rpm
get vzmodules-ent-2.6.18-028stab053.10.i686.rpm
bye

rpm -ivh vzkernel-ent-2.6.18-028stab053.10.i686.rpm \
vzmodules-ent-2.6.18-028stab053.10.i686.rpm
</DEPREACATED>

vi /boot/grub/menu.lst
(set default kernel to new kernel)

12. Install ntp

yum install ntp

rm -f /etc/ntp.conf

# Create a new file with one line:

echo "server 10.1.4.5" > /etc/ntp.conf

# Then set ntpd so it will start at boot:

chkconfig --level 3 ntpd on
service ntpd start

# Make sure it’s talking to the ntp server:

ntpq -p

Correct output should have * or + to the left of the IP:

remote refid st t when poll reach delay offset jitter
==============================================================================
10.1.4.105 clock1.redhat.c 2 u - 64 1 0.370 -706677 0.008

13. copy over binaries/scripts from backup2

scp backup2:/mnt/data4/bin/linux/* /usr/local/sbin/
scp 10.1.4.62:/vzconf/jctmpl* /etc/sysconfig/vz-scripts/
scp 10.1.4.62:/vzconf/ve-slm* /etc/sysconfig/vz-scripts/

14. setup root crontab (alter backup time):
crontab -e
1 1 * * * /usr/local/sbin/backup
*/15 * * * * /usr/local/sbin/vim.watchdog.30 > /dev/null
#56 4,16 * * * root /usr/sbin/vzkeyupdate -c
*/5 * * * * /usr/local/sbin/perc5iraidchk

15. setup backups

echo '#!/bin/sh
backupdir=/data/virt11
server=backup1

## ENTRY /etc
## ENTRY /usr/local/sbin
## ENTRY /root/logs
' > /usr/local/sbin/backup.config

16. setup bb

edit passwd and add the bb user:
echo "bb:x:1984:1984:Big Brother:/home/bb:/bin/bash" >> /etc/passwd

# edit group and add the bb group:
echo "bb:x:1984:" >> /etc/group

# rebuild the password database:

pwconv

# Create bb’s home dir and set ownership:

mkdir /home/bb
chown bb.bb /home/bb

# Go to bb’s home and fetch the client tarball:

cd ~bb
scp backup2:/mnt/data4/build/bb/bb-linux.tar .

tar xf bb-linux.tar

cd bbc1.9e-btf/etc/

echo "10.1.4.5 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh" > /home/bb/bbc1.9e-btf/etc/bb-hosts
echo "10.1.4.61 virt11.johncompanies.com # ssh" >> /home/bb/bbc1.9e-btf/etc/bb-hosts

chmod +r /var/log/messages

cd ..
su bb
./runbb.sh start
exit

# Add this line to /etc/rc.d/rc.local so bb will start at boot:

echo "su - bb -c \"cd /home/bb/bbc1.9e-btf; ./runbb.sh start\"" >> /etc/rc.d/rc.local

17. Uncomment/add lines with the appropriate IP addresses in /etc/ssh/sshd_config:

vi /etc/ssh/sshd_config
ListenAddress 69.55.238.3
ListenAddress 10.1.4.61

And make sure to uncomment
Protocol 2,1

18. setup backup2 nfs mounts

setup portmap to start with server, and start it:

chkconfig --level 3 portmap on
service portmap start

mb mount

run df to confirm that nfs mounts are mounted

19. make alias for /vzconf
ln -s /etc/sysconfig/vz-scripts /vzconf

DEPRECATED 20. copy over the /etc/sysconfig/vz-scripts/4unlimited template file from the last virt server and put it in /etc/sysconfig/vz-scripts

scp 10.1.4.68:/etc/sysconfig/vz-scripts/4unlimited /etc/sysconfig/vz-scripts/

NOTE: if this system is more powerful (RAM, CPU) then the previous, you will want to create a new one using vzsplit:

cd /etc/sysconfig/vz-scripts/
vzsplit -f 4unlimited

21. edit /etc/sysconfig/vz

vi /etc/sysconfig/vz

enable fastboot:
VZFASTBOOT=yes

Disable auto mount
#VZMOUNTS="/vz"

Enable IPV6
IPV6=yes

add/edit to traffic parameters section:
TRAFFIC_SHAPING=yes
BANDWIDTH="eth0:102400"
TOTALRATE="eth0:1:102400"
RATE="eth0:1:50000"

for 2 nics, uncomment this line:
VE_ROUTE_SRC_DEV="eth0"

Make sure the IPTABLES section looks like:
IPTABLES="ip_tables iptable_filter iptable_mangle ipt_limit ipt_multiport ipt_tos ipt_TOS ipt_REJECT ipt_tcpmss ipt_ttl ipt_LOG ipt_length ip_conntrack ip_conntrack_ftp ip_conntrack_irc ipt_conntrack ipt_state ipt_helper iptable_nat ip_nat_ftp ip_nat_irc ipt_REDIRECT ipt_multiport ipt_TCPMSS"

22. disable periodic fsck's, lower the root padding on the /vz and /vz1 filesystems to 100MB

tune2fs -i 0 /dev/sda3
tune2fs -i 0 /dev/sdb2
tune2fs -c -1 /dev/sda3
tune2fs -c -1 /dev/sdb2
tune2fs -r 100000 /dev/sda3
tune2fs -r 100000 /dev/sdb2

23. setup vzagent with pub IP (to allow migrations from 3.x):

vzctl set 1 --ipdel all --ipadd 69.55.230.57 --userpasswd vzagent0:k0zub1k --save
vzagent_ctl restart

24. edit ve config files:

From split of 80 take
CPUUNITS

And put into files in /usr/local/sbin/ve-slm.1024MB.conf et. al.

25. customize the control panel:

vzmlocal 1:1:/vz/private/1:/vz/root/1
cd /vz/private/1/fs/root/var/vzcp/static/vz/skins
for f in `ls`; do scp backup2:/d4/build/vzcp/var/vzcp/static/vz/skins/winxp.new/images/logo_main.gif $f/images/logo_pim.gif; done

1) comment 'Backup' tab in /etc/vzcp/pp/tabs.xml:

<tab-bar id="maintance" obj-type="env.core.vt">


2) change 'href' parameter if 'Maintentance' item in /etc/vzcp/pp/menu.xml:


<item id="management.maintenance" href="/vz/cp/vzdir/infrman/envs/repair?branch=env-home,vzpp" icon="backup" tip="maintenance_screen">

3) /etc/vzcp/pp/dashboard.xml

<item id="management.maintenanance" href="$baseurl/vzdir/infrman/envs/repair" title="Maintenance" icon="env_backup" tip="maintenance_screen" cred="vzcp.env.backups.list" security-obj-type="ve.vz.vt">

URL for 'href' parameter can be obtained from tabs.xml.

vi /var/vzcp/xsl/screens/vzdir/monitman/traffic.xsl
<sw:text>Traffic for selected period and class (NOTE: this information is not certified by JohnCompanies and is not used to calculate actual usage or charges)</sw:text>

<DEPREACATED>
/vz/private/1/root/var/vzcp/static/vz/skins/
/vz/private/1/root/etc/vzcp/pp/menu.xml
/vz/private/1/root/etc/vzcp/pp/dashboard.xml
pp/tabs.xml
/vz/private/1/root/etc/vzcp/vzcpcon.conf

eg (2.6.1 or 3.0):
rsync -va -e ssh backup2:/d4/build/vzcp/var/vzcp/static/vz/skins/ /vz/private/1/root/var/vzcp/static/vz/skins/
rsync -va -e ssh backup2:/d4/build/vzcp/etc/vzcp/pp/menu.xml /vz/private/1/root/etc/vzcp/pp/menu.xml
rsync -va -e ssh backup2:/d4/build/vzcp/etc/vzcp/pp/dashboard.xml /vz/private/1/root/etc/vzcp/pp/dashboard.xml
rsync -va -e ssh backup2:/d4/build/vzcp/etc/vzcp/vzcpcon.conf /vz/private/1/root/etc/vzcp/vzcpcon.conf
</DEPREACATED>

24. template install

To install new templates:

vzup2date –z

Copy over ubu 8.04 from virt19

rsync -a -e ssh 10.1.4.69:/vz/template/ubuntu/8.04 /vz/template/ubuntu/8.04

install ubuntu 8.04 templates (get latest from vz)

cd /backup4/vzrpms/distro/ubuntu/8.04
rpm -ivh *
vzpkg update cache ubuntu-8.04-x86

26. add traffic shaping support

Re-define class 0:

vi /etc/sysconfig/vz-scripts/networks_classes
0 69.55.224.0/20
0 10.1.4.0/24

make sure 1 is defined (0.0.0.0) and no other classes are needed

Then start it

/etc/init.d/vz shaperrestart

27. udp dos fix

echo "/usr/local/sbin/iptables.sh" >> /etc/rc.d/rc.local

28. make rc.local start before vz

mv /etc/rc.d/rc3.d/S99local /etc/rc.d/rc3.d/S95local

29. disable unwanted services/daemons

chkconfig --level 3 avahi-daemon off
chkconfig --level 3 bluetooth off
chkconfig --level 3 cpuspeed off
chkconfig --level 3 cups off
chkconfig --level 3 firstboot off
chkconfig --level 3 gpm off
chkconfig --level 3 haldaemon off
chkconfig --level 3 hidd off
chkconfig --level 3 ip6tables off
chkconfig --level 3 iptables off
chkconfig --level 3 irqbalance off
chkconfig --level 3 isdn off
chkconfig --level 3 kudzu off
chkconfig --level 3 lvm2-monitor off
chkconfig --level 3 mcstrans off
chkconfig --level 3 mdmonitor off
chkconfig --level 3 messagebus off
chkconfig --level 3 netfs off
chkconfig --level 3 nfslock off
chkconfig --level 3 pcscd off
chkconfig --level 3 rawdevices off
chkconfig --level 3 readahead_early off
chkconfig --level 3 restorecond off
chkconfig --level 3 rpcgssd off
chkconfig --level 3 rpcidmapd off
chkconfig --level 3 smartd off
chkconfig --level 3 yum-updatesd off

30. load mrtg graphs

yum install xinetd

create /etc/xinetd.d/load with the following contents:

echo "service load
{
disable = no
flags = REUSE
socket_type = stream
wait = no
user = root
server = /usr/local/sbin/load.pl
}" > /etc/xinetd.d/load

# Add this line to /etc/services:
echo "load 12384/tcp" >> /etc/services

31. Tasks to do on backup1:

setup backup dirs:
mkdir -p /data/virt11/0

add the system to
vi /usr/local/sbin/snapshot_rotate

32. Tasks to do on mail:

Update backup graph:
vi /usr/local/www/mgmt/cgi/backupgraph.pl
(add hostname)

add to bbhosts:
vi /usr/home/bb/bbsrc/bb1.9e-btf/etc/bb-hosts
su bb
cd
bbsrc/bb/runbb.sh restart ; exit

add to load graph config:
vi /usr/local/www/mgmt/mrtg/mrtg1.cfg
(add new section for new machine)

33. add to management db (on mail and devweb) jc.ref_machines and jc.ref_templates

insert into ref_machines values (null,'virt19','virt19.johncompanies.com',0,'l');
select * from ref_machines;
insert into systems values ('', 'col00001',30,null,null,100,'virt11.johncompanies.com',null,'2010-03-11',null,null,1,0,null,null,null);
select * from systems where hostname='virt19.johncompanies.com';
update ipmap set cid='col00001', sysid=3772, assign_date='2008-01-07', status=1 where ip='69.55.236.2';
update ipmap set cid='col00001', sysid=3772, assign_date='2008-01-07', status=1 where ip='69.55.230.109';
insert into ref_templates values ('','fedora-core-1/20050725',33,'Fedora Core 1',1);
insert into ref_templates values ('','fedora-core-2/20050725',33,'Fedora Core 2',1);
insert into ref_templates values ('','debian-3.1/20050927',33,'Debian 3.1',1);

34. Make sure that ns1c has proper forward and reverse entries for the new system (both private and public addresses), so that new system emailing will work.
vr johncompanies.com
vi internal.johncompanies.com
rndc reload johncompanies.com IN private
(edit the PTR too)

35. add an outside blocking rule to the firewall, so this machine can only be reached from inside the firewall.

Follow example already in firewall virt19 is:

ipfw add 00149 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 or 62.213.72.101 or 64.131.90.27 or 82.200.65.190 or 91.204.24.0/22 or 194.44.160.178 or 195.214.232.10 or 195.214.233.10 or 208.251.212.1 } to 69.55.236.2
ipfw add 00149 deny ip from any to 69.55.236.2

virt12 would be 00142

36. add to server/cabinet map

37. make /vz1/private
mkdir /vz1/private ; chmod 0755 /vz1/private
mkdir /vz1/root ; chmod 0755 /vz1/root

38. put config files in a place they’ll get backed up
mkdir /vzconf/confs
crontab -e
0 0 * * * rm -f /vzconf/confs/*.conf; cp /vzconf/*.conf /vzconf/confs

39. fix pass/vzpp and localse issue with ubu 8.10

vi /vz/template/ubuntu/8.10/x86/config/os/default/post-install

sed '/^password.*pam_unix.so.*sha512/ s~sha512~md5~' -i etc/pam.d/common-password

localedef -i en_US -c -f UTF-8 en_US.UTF-8

40. add tun device support

echo "modprobe tun" >> /etc/rc.local</pre>

== 3.x ==

Last updated 2007-04-5

<pre>1. enable post info to console
Supermicro:
Console redirection:
Com port addr: on-board COM A
Baud: 38400
Console type: vt100
Flow control: none
Console connection: direct
Continue cr after post: off

2450:
Make sure running bios A09
Console Redirection: VT100/VT220

2950:
Serial comm.: on with cons. Redir via com1
Failsafe: 115200
Redir after boot: enabled

6144
Server
Custom- desel all but base
/root/install.log

2. Install OS
Set partitions:
/ 3072M
Disk 1: Swap 4096M
Disk 2: Swap 4096M
/vz (the rest)

No firewall
Disable SElinux

FC4
Choose a custom install, the “minimal” package

CentOS4 Server CD (only option for 2950)
Choose the “minimal” package

3. create ssh key, add swsoft key to authorized_keys

(enter to everything)

cd /root/
ssh-keygen -t dsa

echo "ssh-dss AAAAB3NzaC1kc3MAAACBAIB7XHmpcsa1D/vsKS+mzEvBVo7udew/aRahbjnjywL4YXT7ruWzpGo7R3UA8GKIgVHoQGUbuXrJz5ICtBEXOlbgXVEz7Bu5nXOG2C9i/8pmFtUw6GwACII8uZE7t/Zq5CRj+5zN1Rv7n/4X4RAjzk5BHXoZ62a32FTbS4tPvyPrAAAAFQDiUi7YEiffNzGvB544lQpRdn29TwAAAIBkUchLTBE1+sX11iC80yhNh9EGeCMLdkT8UqHsTSaD0myQPxGALuNY4rO31q+25uKqSRDTXd6ZWXp4bplA6Ls6jl7r7XVR9loJ7WvRMqc4Dv68TZ7h2UBqAhu4det2UF4LHim3DNf1gKn9DzWNkmI/N2FvVVAL4aM3sPunVIvUYgAAAIAJX38INABVgnCEBKqpv5FMLm3k9MgMn54oQFmylBbgM0w4SUeuDj3jLrkJf8dT9w8ByEDb8xviexpWrPsbz2DeThle4xgzcyKrUkt2Wugtmm4DwS0S/68BAA9wXoWD/vZ67GBoCKH7uq8VAjUVcCbJ/7IydljnFAZoJpktdwN7iQ== russup@support.sw-soft.com" >> /root/.ssh/authorized_keys

4. setup access to backup2:

echo "10.1.4.3 backup2" >> /etc/hosts

Copy id_dsa.pub to backup2 and add it to the end of ~root/.ssh/authorized_keys:

cat .ssh/id_dsa.pub | ssh backup2 'cat - >> /root/.ssh/authorized_keys'

ssh backup2

(confirm you can connect w/o entering a password)

5. add bash backward history search
echo "\"\M-z\": history-search-backward
\"\M-x\": history-search-backward " > /root/.inputrc
echo "export GREP_OPTIONS='--color=auto'" >> ~/.bashrc
echo "\"\e[5~\": history-search-backward" >> ~/.inputrc
echo "\"\e[6~\": history-search-forward" >> ~/.inputrc

6. add to /root/.bashrc, create log dir

vi /root/.bashrc
alias v='vzctl'
alias mbm='mb mount'
alias mbu='mb umount'
export PS1="[\u@\h \w]# "
alias cv='cd /vz/private'
alias cv1='cd /vz1/private'
alias cv2='cd /vz2/private'

mkdir /root/logs

7. make sure serial console and kernel messages are working

grub

add these 2 lines to the top:

vi /boot/grub/menu.lst
serial --unit=0 --speed= 115200
terminal --timeout=10 serial console

and the kernel line should look like:

kernel /boot/vmlinuz-2.4.20-020stab009.24.777-enterprise ro root=/dev/sda1 console=tty0 console=ttyS0,115200

Remove the line:
hiddensplashmenu

lilo

Make sure the append line in /etc/lilo.conf looks like this:

append="console=tty console=ttyS0,38400"

If you edit /etc/lilo.conf don’t forget to run lilo

# lilo
Added linux
Added linux-up
Added linux+virtuozzo *

(note: the last console listed is the one that gets output from the rc scripts.)

add to /etc/inittab and /etc/securetty

echo "s0:2345:respawn:/sbin/agetty -L ttyS0 115200 linux" >> /etc/inittab
echo "ttyS0" >> /etc/securetty

kill -1 1

8. configuring eth1 for management net:

echo "10.1.5/24 via 10.1.4.2
10.1.6/24 via 10.1.4.2" >> /etc/sysconfig/network-scripts/route-eth1

reboot the machine to make sure that the interface is configured properly.

also verify that the routes are in place like so:

netstat -rn | grep 10.1.4.2
10.1.5.0 10.1.4.2 255.255.255.0 UG 40 0 0 eth1
10.1.6.0 10.1.4.2 255.255.255.0 UG 40 0 0 eth1

9. make sure the hostname on the system is properly set - in the form virt4.johncompanies.com - and make sure it is set properly:
cat /etc/sysconfig/network

10. Check/update fstab

vi /etc/fstab

/etc/fstab entries should look something like (note noatime option):

/dev/sda3 /vz ext3 defaults,noatime 1 2
/dev/sdb1 /vz1 ext3 defaults,noatime 1 2

make sure swap is configured in fstab

Use real devices instead of LABEL

11. Install Virtuozzo 3.0

If it wasn’t done at install time:

setenforce 0

umount /vz
umount /vz1

mke2fs -i1024 -b1024 -j /dev/sda3 &
mke2fs -i1024 -b1024 -j /dev/sdb2 &

On CentOS 4:
yum install rpm-build; yum update iproute

mount -a

scp backup2:/d4/vzrpms/iso/Rel300.tar.gz /vz/.

cd /vz

tar xzf Rel300.tar.gz

cd Rel300/HW

./install

Choose Enterprise kernel
Vzagent password: k0zub1k
DON’T reboot

vzup2date
User:
gdawson
Pass:
qjdK1R
(run again. till no more updates offered)

reboot

Run vzup2date (make sure to choose enterprise kernel) and/or install latest kernel:

ftp downloads.swsoft.com
cd /updates/Rel300/20061206
bin
get vzkernel-enterprise-2.6.9-023stab033.9.i686.rpm
get vzmodules-enterprise-2.6.9-023stab033.9.i686.rpm
bye

rpm -ivh vzkernel-enterprise-2.6.9-023stab033.9.i686.rpm \
vzmodules-enterprise-2.6.9-023stab033.9.i686.rpm

vi /boot/grub/menu.lst
(set default kernel to new kernel)

12. Install various RPMs

scp backup2:/mnt/data4/linuxrpms/* /tmp

install them in this order:

12a. http fetcher

rpm -ivh /tmp/http_fetcher-1.1.0-1.i386.rpm
rpm -ivh /tmp/fetch-1.0-1.i386.rpm

(ignore error on CentOS4)

Now the fetch command, which vemake calls to email the new system owner, will work (it downloads the welcome email from www.johncompanies.com

12b. ntp client

sometimes ntp isn’t included in which case:

FCx:

rpm -ivh /tmp/libcap-1.10-22.i386.rpm
rpm -ivh /tmp/compat-readline43-4.3-2.i386.rpm
rpm -ivh /tmp/ntp-4.1.2-5.i386.rpm

CentOS4:

rpm -ivh /tmp/ntp-4.1.2-5.i386.rpm

rm -f /etc/ntp.conf

Create a new file with one line:

echo "server 10.1.4.5" > /etc/ntp.conf

Then set ntpd so it will start at boot:

chkconfig --level 3 ntpd on
service ntpd start

Make sure it’s talking to the ntp server:

ntpq -p

Correct output should have something unfer the refid col:

remote refid st t when poll reach delay offset jitter
==============================================================================
10.1.4.105 clock1.redhat.c 2 u - 64 1 0.370 -706677 0.008

12c. adaptec machines only: install aaccli for raid controller (LSI card util is with bin files):

rpm -ivh /tmp/aacapps-4.1-0.i386.rpm

May have to run:

cd /dev
./MAKEDEV.aac aac0

13. copy over binaries/scripts from backup2

scp backup2:/mnt/data4/bin/linux/* /usr/local/sbin/

14. setup root crontab (alter backup time):
crontab -e
1 1 * * * /usr/local/sbin/backup
*/15 * * * * /usr/local/sbin/vim.watchdog.30 > /dev/null
#56 4,16 * * * root /usr/sbin/vzkeyupdate -c

On Perc5/i machines:

*/5 * * * * /usr/local/sbin/perc5iraidchk

15. setup backups

echo '#!/bin/sh
backupdir=/mnt/data4/virt19

## ENTRY /etc
## ENTRY /usr/local/sbin
## ENTRY /root/logs
' > /usr/local/sbin/backup.config

16. setup bb

edit passwd and add the bb user:
echo "bb:x:1984:1984:Big Brother:/home/bb:/bin/bash" >> /etc/passwd

edit group and add the bb group:
echo "bb:x:1984:" >> /etc/group

rebuild the password database:

pwconv

Create bb’s home dir and set ownership:

mkdir /home/bb
chown bb.bb /home/bb

Go to bb’s home and fetch the client tarball:

cd ~bb
scp backup2:/mnt/data4/build/bb/bb-linux.tar .

tar xf bb-linux.tar

cd bbc1.9e-btf/etc/

echo "10.1.4.5 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh" > /home/bb/bbc1.9e-btf/etc/bb-hosts
echo "10.1.4.69 virt19.johncompanies.com # ssh" >> /home/bb/bbc1.9e-btf/etc/bb-hosts

chmod +r /var/log/messages

cd ..
su bb
./runbb.sh start
exit

Add this line to /etc/rc.d/rc.local so bb will start at boot:

echo "su - bb -c \"cd /home/bb/bbc1.9e-btf; ./runbb.sh start\"" >> /etc/rc.d/rc.local

17. Uncomment/add lines with the appropriate IP addresses in /etc/ssh/sshd_config:

vi /etc/ssh/sshd_config
ListenAddress 69.55.232.162
ListenAddress 10.1.4.66

And make sure to uncomment
Protocol 2,1

18. setup backup2 nfs mounts

setup portmap to start with server, and start it:

chkconfig --level 3 portmap on
service portmap start

mb mount

run df to confirm that nfs mounts are mounted

19. m ake alias for /vzconf
ln -s /etc/sysconfig/vz-scripts /vzconf

20. copy over the /etc/sysconfig/vz-scripts/4unlimited template file from the last virt server and put it in /etc/sysconfig/vz-scripts

scp 10.1.4.68:/etc/sysconfig/vz-scripts/4unlimited /etc/sysconfig/vz-scripts/

NOTE: if this system is more powerful (RAM, CPU) then the previous, you will want to create a new one using vzsplit:

cd /etc/sysconfig/vz-scripts/
vzsplit -f 4unlimited

21. edit /etc/sysconfig/vz

vi /etc/sysconfig/vz

enable fastboot:
VZFASTBOOT=yes

add/edit to traffic parameters section:
TRAFFIC_SHAPING=yes
BANDWIDTH="eth0:102400"
TOTALRATE="eth0:1:102400"

for 2 nics, uncomment this line:
VE_ROUTE_SRC_DEV="eth0"

Make sure the IPTABLES section looks like:
IPTABLES="ip_tables iptable_filter iptable_mangle ipt_limit ipt_multiport ipt_tos ipt_TOS ipt_REJECT ipt_tcpmss ipt_ttl ipt_LOG ipt_length ip_conntrack ip_conntrack_ftp ip_conntrack_irc ipt_conntrack ipt_state ipt_helper iptable_nat ip_nat_ftp ip_nat_irc ipt_REDIRECT"

22. disable periodic fsck's, lower the root padding on the /vz and /vz1 filesystems to 100MB

tune2fs -i 0 /dev/sda3
tune2fs -i 0 /dev/sdb2
tune2fs -c -1 /dev/sda3
tune2fs -c -1 /dev/sdb2
tune2fs -r 100000 /dev/sda3
tune2fs -r 100000 /dev/sdb2

23. copy over our versions of the vzpp files:

/vz/private/1/root/var/vzcp/static/vz/skins/
/vz/private/1/root/etc/vzcp/pp/menu.xml
/vz/private/1/root/etc/vzcp/pp/dashboard.xml
/vz/private/1/root/etc/vzcp/vzcpcon.conf

eg (2.6.1 or 3.0):
rsync -va -e ssh backup2:/d4/build/vzcp/var/vzcp/static/vz/skins/ /vz/private/1/root/var/vzcp/static/vz/skins/
rsync -va -e ssh backup2:/d4/build/vzcp/etc/vzcp/pp/menu.xml /vz/private/1/root/etc/vzcp/pp/menu.xml
rsync -va -e ssh backup2:/d4/build/vzcp/etc/vzcp/pp/dashboard.xml /vz/private/1/root/etc/vzcp/pp/dashboard.xml
rsync -va -e ssh backup2:/d4/build/vzcp/etc/vzcp/vzcpcon.conf /vz/private/1/root/etc/vzcp/vzcpcon.conf

To disable backups from scratch:

In svcve edit /etc/vzcp/vzcpcon.conf

In section:

<auth>
<type>VEUser</type>
<user>root</user>

Add into/below:
<cred>
<id>maintenance</id>
</cred>

The following:
<cred>
<id>show</id>
</cred>
<cred>
<id>reinstall</id>
</cred>
<cred>
<id>repair_mode</id>
</cred>

Will finally look like:

<cred>
<id>maintenance</id>
<cred>
<id>show</id>
</cred>
<cred>
<id>reinstall</id>
</cred>
<cred>
<id>repair_mode</id>
</cred>
</cred>
</cred>

24. template install

mb mount

install debian 3.0 templates

cd /backup4/vzrpms/distro/debian/3.0/p1/
for f in `cat /backup4/vzrpms/distro/debian/3.0/install_list`; do rpm -ivh $f; done

install debian 3.1 templates

cd /backup4/vzrpms/distro/debian/3.1/p3/
for f in `ls`; do rpm -ivh $f; done

install redhat templates

cd /backup4/vzrpms/distro/rh9/current/
for f in `cat /backup4/vzrpms/distro/rh9/install_list`; do rpm -ivh $f; done

install fedora1 templates

cd /backup4/vzrpms/distro/fc1/current/
for f in `cat /backup4/vzrpms/distro/fc1/install_list`; do rpm -ivh $f; done

change 'krb5-devel-1.3.4-5' to 'krb5-devel-1.3.3-6' in (applies to pre-p5)
/vz/template/fedora-core-devel-1/conf/fedora-core-devel-1.conf.20050314

install fedora2 templates

cd /backup4/vzrpms/distro/fc2/current/
for f in `cat /backup4/vzrpms/distro/fc2/install_list`; do rpm -ivh $f; done

install suse9.2 templates

cd /backup4/vzrpms/distro/suse/current/
for f in `cat /backup4/vzrpms/distro/suse/install_list`; do rpm -ivh $f; done

install centos templates

cd /backup4/vzrpms/distro/centos/current/
for f in `cat /backup4/vzrpms/distro/centos/install_list`; do rpm -ivh $f; done

install fedora4 templates

cd /backup4/vzrpms/distro/fc4/current/
for f in `cat /backup4/vzrpms/distro/fc2/install_list`; do rpm -ivh $f; done

vzpkgcache

To pull templates from another machine (before hand stop ve 1 and move it’s template so you preserve the latest template files just setup by vzup2date):

rsync -a -e ssh 10.1.4.68:/vz/template/ /vz/template/

Make sure ubuntu source is defined:

UBU_SERVER=http://archive.ubuntu.com

And change FC source:
FC_SERVER=http://rhold.fedoraproject.org

In /etc/vztt/vztt.conf

rsync -a -e ssh 10.1.4.68:/etc/vztt/vztt.conf /etc/vztt/

To update the cache:

vzpkg update cache ubuntu-6.10-x86

25. add tun device support

Add to /etc/modules.conf:
echo "alias char-major-10-200 tun" >> /etc/modules.conf

depmod -a

echo "modprobe tun" >> /etc/rc.local

26. add traffic shaping support

Re-define class 0:

vi /etc/sysconfig/vz-scripts/networks_classes
0 69.55.224.0/20
0 10.1.4.0/24

make sure 1 is defined (0.0.0.0) and no other classes are needed

Then start it

/etc/init.d/vz shaperrestart

27. RAIDmon (only applies to LSI 320 PCI card)

cd /tmp
cp /backup4/build/megaraid/linuxmon30.tgz .
tar xzf linuxmon30.tgz
cd linuxmon30
./install
(say no)
mv –f raidmon.init /etc/init.d/raidmon
service raidmon start
tail -f /var/log/messages
(check for output)

28. udp dos fix

echo "/usr/local/sbin/iptables.sh" >> /etc/rc.d/rc.local

29. make rc.local start before vz

mv /etc/rc.d/rc3.d/S99local /etc/rc.d/rc3.d/S95local

30. disable unwanted services/daemons

On CentOS:
chkconfig --del kudzu
chkconfig --del cpuspeed

This should be taken care of by vz install so probably safe to skip

chkconfig --del kudzu
chkconfig --del cups
chkconfig --del gpm
chkconfig --del isdn
chkconfig --del pcmcia
chkconfig --del apmd
chkconfig --del rhnsd
chkconfig --del mdmonitor

31. load mrtg graphs

On FCx xinetd not installed so:

rpm -ivh /tmp/xinetd-2.3.11-1.7x.i386.rpm

create /etc/xinetd.d/load with the following contents:

echo "service load
{
disable = no
flags = REUSE
socket_type = stream
wait = no
user = root
server = /usr/local/sbin/load.pl
}" > /etc/xinetd.d/load

Add this line to /etc/services:
echo "load 12384/tcp" >> /etc/services

On FCx: if xinetd isn’t running, configure it for startup at boot time:

chkconfig --level 3 xinetd on
service xinetd start

32. Tasks to do on backup2:

setup backup dirs:
mkdir -p /mnt/data4/virt19/0

add the system to
vi /mnt/data4/bin/rsync_houseclean

33. Tasks to do on mail:

Update backup graph:
vi /usr/local/www/mgmt/cgi/backupgraph.pl
(add hostname)

add to bbhosts:
vi /usr/home/bb/bbsrc/bb1.9e-btf/etc/bb-hosts
su bb
cd
bbsrc/bb/runbb.sh restart ; exit

add to load graph config:
vi /usr/local/www/mgmt/mrtg/mrtg1.cfg
(add new section for new machine)

34. add to management db (on mail and devweb) jc.ref_machines and jc.ref_templates

insert into ref_machines values (null,'virt19','virt19.johncompanies.com',0,'l');
select * from ref_machines;
insert into systems values ('', 'col00001',43,null,null,100,'virt19.johncompanies.com',null,'2008-01-07',null,null,1,0,null,null,null);
select * from systems where hostname='virt19.johncompanies.com';
update ipmap set cid='col00001', sysid=3772, assign_date='2008-01-07', status=1 where ip='69.55.236.2';
update ipmap set cid='col00001', sysid=3772, assign_date='2008-01-07', status=1 where ip='69.55.230.109';
insert into ref_templates values ('','fedora-core-1/20050725',33,'Fedora Core 1',1);
insert into ref_templates values ('','fedora-core-2/20050725',33,'Fedora Core 2',1);
insert into ref_templates values ('','debian-3.1/20050927',33,'Debian 3.1',1);

35. Make sure that ns1c has proper forward and reverse entries for the new system (both private and public addresses), so that new system emailing will work.
vr johncompanies.com
vi internal.johncompanies.com
rndc reload johncompanies.com IN private
(edit the PTR too)

36. add an outside blocking rule to the firewall, so this machine can only be reached from inside the firewall.

Follow example already in firewall virt19 is:
ipfw add 00149 allow ip from { 69.55.225.225 or 69.55.230.2 or 69.55.230.10 or 69.55.230.3 or 62.213.72.101 or 64.131.90.27 or 80.89.128.178 or 82.200.65.190 or 84.237.120.254 or 89.251.97.2 or 195.214.232.10 or 195.214.233.10 or 208.251.212.1 or 213.130.28.26 } to 69.55.236.2
ipfw add 00149 deny ip from any to 69.55.236.2

virt12 would be 00142

37. add to server/cabinet map

39. make /vz1/private
mkdir /vz1/private ; chmod 0755 /vz1/private
mkdir /vz1/root ; chmod 0755 /vz1/root</pre>

Virtuozzo Server Install

2013-11-14T21:52:10Z

24.165.90.109: /* Configure system, install virtuozzo */

= 4.7 =

Server-specific changes in <span style="color:red">red</span>

http://www.parallels.com/download/pvc/lin

http://www.parallels.com/support/virtualization-suite/pvc/

== BIOS ==

* enable post info to console

2950:<br>
Serial comm.: on with cons. Redir via com1<br>
Failsafe: 115200<br>
Redir after boot: enabled

* Tweak DRAC settings (see [[DRAC/RMM#DRAC_setup|DRAC setup]])

== Install OS ==

CentOS6.x (get latest from http://mirrors.kernel.org/centos/6.4/isos/x86_64/CentOS-6.4-x86_64-netinstall.iso)

May have to disable DRAC virtual media to make sure we get sda as our first device

Set partitions:
<pre>Disk 1:
/ 10240M (ext4)
9216M swap
/vz (the rest) (ext4)
Disk 2:
9216M Swap
/vz1 (the rest) (ext4)</pre>

Basic server

Use grub, install to MBR

== Configure system, install virtuozzo ==
* add ssh

yum install openssh-clients

* create ssh key, add swsoft key to authorized_keys

(enter to everything)

<pre>cd /root/
ssh-keygen -t dsa

Add vz support key:
echo "ssh-dss AAAAB3NzaC1kc3MAAACBAIB7XHmpcsa1D/vsKS+mzEvBVo7udew/aRahbjnjywL4YXT7ruWzpGo7R3UA8GKIgVHoQGUbuXrJz5ICtBEXOlbgXVEz7Bu5nXOG2C9i/8pmFtUw6GwACII8uZE7t/Zq5CRj+5zN1Rv7n/4X4RAjzk5BHXoZ62a32FTbS4tPvyPrAAAAFQDiUi7YEiffNzGvB544lQpRdn29TwAAAIBkUchLTBE1+sX11iC80yhNh9EGeCMLdkT8UqHsTSaD0myQPxGALuNY4rO31q+25uKqSRDTXd6ZWXp4bplA6Ls6jl7r7XVR9loJ7WvRMqc4Dv68TZ7h2UBqAhu4det2UF4LHim3DNf1gKn9DzWNkmI/N2FvVVAL4aM3sPunVIvUYgAAAIAJX38INABVgnCEBKqpv5FMLm3k9MgMn54oQFmylBbgM0w4SUeuDj3jLrkJf8dT9w8ByEDb8xviexpWrPsbz2DeThle4xgzcyKrUkt2Wugtmm4DwS0S/68BAA9wXoWD/vZ67GBoCKH7uq8VAjUVcCbJ/7IydljnFAZoJpktdwN7iQ== russup@support.sw-soft.com" >> /root/.ssh/authorized_keys</pre>

* setup access to backup2:

<pre>echo "10.1.4.3 backup2" >> /etc/hosts
echo "10.1.4.8 backup1" >> /etc/hosts</pre>

# Copy id_dsa.pub to backup2 and add it to the end of ~root/.ssh/authorized_keys:

<pre>cat .ssh/id_dsa.pub | ssh backup2 'cat - >> /root/.ssh/authorized_keys'
cat .ssh/id_dsa.pub | ssh backup1 'cat - >> /root/.ssh/authorized_keys'</pre>

<pre>ssh backup2 hostname
ssh backup1 hostname</pre>
(confirm you can connect w/o entering a password)

* add bash backward history search
<pre>echo "\"\M-z\": history-search-backward
\"\M-x\": history-search-backward " > /root/.inputrc
echo "export GREP_OPTIONS='--color=auto'" >> ~/.bashrc
echo "\"\e[5~\": history-search-backward" >> ~/.inputrc
echo "\"\e[6~\": history-search-forward" >> ~/.inputrc</pre>

* add to /root/.bashrc, create log dir

<pre>vi /root/.bashrc
alias v='vzctl'
alias mbm='mb mount'
alias mbu='mb umount'
export PS1="[\u@\h \w]# "
alias cv='cd /vz/private'
alias cv1='cd /vz1/private'
alias cv2='cd /vz2/private'
alias pg='ps auxw|grep $1'</pre>

source /root/.bashrc

mkdir /root/logs

* make sure serial console and kernel messages are working

add these 2 lines to the top:

<pre>vi /boot/grub/menu.lst
serial --unit=0 --speed= 115200
terminal --timeout=10 serial console</pre>

and the kernel line should look like:

kernel /boot/vmlinuz-2.4.20-020stab009.24.777-enterprise ro root=/dev/sda1 console=tty0 console=ttyS0,115200

Remove or comment out these lines:
<pre>hiddensplashmenu
splashimage=(hd0,0)/boot/grub/splash.xpm.gz</pre>

add to /etc/inittab and /etc/securetty

<pre>echo "s0:2345:respawn:/sbin/agetty -L ttyS0 115200 linux" >> /etc/inittab
echo "ttyS0" >> /etc/securetty</pre>

kill -1 1

* make sure the hostname on the system is properly set - in the form virt4.johncompanies.com - and make sure it is set properly:
cat /etc/sysconfig/network

* Check/update fstab

vi /etc/fstab

/etc/fstab entries should look something like (note noatime option):

<pre>/dev/sda3 /vz ext4 defaults,noatime 1 2
/dev/sdb2 /vz1 ext4 defaults,noatime 1 2</pre>

make sure swap is configured in fstab

Use real devices instead of LABEL

* Install Virtuozzo 4.7

Have a license on hand - request a day ahead of the install. Go to:
http://www.parallels.com/download/pvc47/

<pre>yum install compat-libstdc++-33
yum install wget
yum install ftp
yum install pciutils
yum install ed bc

cd /vz
wget http://download.parallels.com/pvc/47/lin/vzinstall-linux-x86_64.bin
chmod 0700 vzinstall-linux-x86_64.bin
./vzinstall-linux-x86_64.bin</pre>

download and install

change wd's to /vz

Ignore OS version error

Custom install (don’t install templates)

Install PVA Agent
Install PVA Management Node

reboot

Get on KVM to watch PVA install (won't happen on console)

vzup2date

(run again. till no more updates offered)

reboot

<DEPRECATED>
Run vzup2date (make sure to choose enterprise kernel) and/or install latest kernel:

NOTE: skip over the manual kernel load below if you can get the 16G recognized before running vzup2date- then it will dl the ent kernel

NOTE: go to /vz/vzup2date/virtuozzo/linux/i386/4.0.0/CU-2.6.18-028stab068.3 and install kern there (downloaded by vzup2date)

vi /boot/grub/menu.lst
(set default kernel to new kernel)
</DEPRECATED>

* Install ntp

yum install ntp
(if not already installed)

rm -f /etc/ntp.conf

Create a new file with one line:

echo "server 10.1.4.5" > /etc/ntp.conf

Then set ntpd so it will start at boot:

chkconfig --level 3 ntpd on
service ntpd start

Make sure it’s talking to the ntp server:

ntpq -p

Correct output should have something unfer the refid col and a '+' next to the remote server:

<pre>remote refid st t when poll reach delay offset jitter
==============================================================================
10.1.4.105+ clock1.redhat.c 2 u - 64 1 0.370 -706677 0.008</pre>

* copy over binaries/scripts from backup2

scp backup2:/mnt/data4/bin/linux/* /usr/local/sbin/
scp <span style="color:red">10.1.4.62</span>:"/vzconf/jctmpl*" /etc/sysconfig/vz-scripts/
scp <span style="color:red">10.1.4.62</span>:/vzconf/ve-slm* /etc/sysconfig/vz-scripts/

* setup root crontab (alter backup time):
<pre>crontab -e
1 1 * * * /usr/local/sbin/backup
*/15 * * * * /usr/local/sbin/vim.watchdog.30 > /dev/null
#56 4,16 * * * root /usr/sbin/vzkeyupdate -c
*/5 * * * * /usr/local/sbin/perc5iraidchk64</pre>

* setup backups

<pre>
echo '#!/bin/sh
<tt>backupdir=/data/<span style="color:red">virt13</span>
server=backup1

## ENTRY /etc
## ENTRY /usr/local/sbin
## ENTRY /root/logs
' > /usr/local/sbin/backup.config</tt>
</pre>

* setup bigbrother

edit passwd and add the bb user:
echo "bb:x:1984:1984:Big Brother:/home/bb:/bin/bash" >> /etc/passwd

edit group and add the bb group:
echo "bb:x:1984:" >> /etc/group

rebuild the password database:
pwconv

Create bb’s home dir and set ownership:
mkdir /home/bb
chown bb.bb /home/bb

Go to bb’s home and fetch the client tarball:
<pre>cd ~bb
scp backup2:/mnt/data4/build/bb/GOOD-bb-linux-bbc1.9i-btf.tgz .

tar xvf GOOD-bb-linux-bbc1.9i-btf.tgz

yum install glibc-2.12-1.47.el6_2.5.i686

vi etc/bbsys.sh
SORT="/bin/sort"

vi bin/bb-disk.sh
eval "$DFCMD | $EGREP \"$DFUSE\" | $EGREP -v \"$DFEXCLUDE\" | $SORT \"-k${DFSORT}\" > $BBTMP/DFOUT.$$"
wclen1=`eval "$DFCMD | $EGREP \"$DFUSE\" | $EGREP -v \"$DFEXCLUDE\" | $SORT \"-k${DFSORT}\" | $WC"`

cd bbc1.9i-btf/etc/

echo "10.1.4.5 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh" > /home/bb/bbc1.9i-btf/etc/bb-hosts
echo "10.1.4.63 virt13.johncompanies.com # ssh" >> /home/bb/bbc1.9i-btf/etc/bb-hosts
(^^^change to correct host)

chmod +r /var/log/messages

cd ..
su bb
./runbb.sh start
exit</pre>

Add this line to /etc/rc.d/rc.local so bb will start at boot:

echo "su - bb -c \"cd /home/bb/bbc1.9i-btf; ./runbb.sh start\"" >> /etc/rc.d/rc.local

* Uncomment/add lines with the appropriate IP addresses in /etc/ssh/sshd_config:

vi /etc/ssh/sshd_config
<tt>ListenAddress <span style="color:red">69.55.226.2</span>
ListenAddress <span style="color:red">10.1.4.63</span></tt>

<DEPRECATED>
And make sure to uncomment
Protocol 2,1
</DEPRECATED>

* setup backup1/backup2 nfs mounts

<DEPRECATED>
setup portmap to start with server, and start it:

chkconfig --level 3 portmap on
service portmap start
</DEPRECATED>

mb mount

run df to confirm that nfs mounts are mounted

* make alias for /vzconf
ln -s /etc/sysconfig/vz-scripts /vzconf

* edit /etc/sysconfig/vz

vi /etc/sysconfig/vz

calc cpu to reserver for HN. take 5% of power:
vzcpucheck
Current CPU utilization: 27000
Power of the node: 1018320

so in this case:
VE0CPUUNITS=50916

enable fastboot:
VZFASTBOOT=yes

Disable auto mount
#VZMOUNTS="/vz"

Enable slm:
SLM=yes

<DEPRECATED>
Enable IPV6
IPV6=yes
</DEPRECATED>

add/edit to traffic parameters section:
TRAFFIC_SHAPING=yes
BANDWIDTH="eth0:102400"
TOTALRATE="eth0:1:102400"
RATE="eth0:1:50000"

for 2 nics, uncomment this line:
VE_ROUTE_SRC_DEV="eth0"

Make sure the IPTABLES section looks like:
<pre>IPTABLES="ip_tables iptable_filter iptable_mangle ipt_limit ipt_multiport ipt_tos ipt_TOS ipt_REJECT ipt_tcpmss ipt_ttl ipt_LOG ipt_length ip_conntrack ip_conntrack_ftp ip_conntrack_irc ipt_conntrack ipt_state ipt_helper iptable_nat ip_nat_ftp ip_nat_irc ipt_REDIRECT ipt_multiport ipt_TCPMSS"</pre>

* disable periodic fsck's, lower the root padding on the /vz and /vz1 filesystems to 100MB
<pre>tune2fs -i 0 /dev/sda3
tune2fs -i 0 /dev/sdb2
tune2fs -c -1 /dev/sda3
tune2fs -c -1 /dev/sdb2
tune2fs -r 100000 /dev/sda3
tune2fs -r 100000 /dev/sdb2</pre>

* setup vzagent with pub IP (to allow migrations from 3.x):
vzctl set 50 --ipdel all --ipadd <span style="color:red">69.55.226.3</span> --userpasswd vzagent0:k0zub1k --save
vzagent_ctl restart

* edit ve config files:

From split of 80 take
<tt>CPUUNITS</tt>

And put into files in /vzconf/base_ct_conf
along with:
<pre>VERSION="2"

ONBOOT="yes"

KMEMSIZE="2147483647:2147483647"
LOCKEDPAGES="2147483647:2147483647"
PRIVVMPAGES="2147483647:2147483647"
SHMPAGES="2147483647:2147483647"
NUMPROC="32567:32567"
VMGUARPAGES="2147483647:2147483647"
OOMGUARPAGES="2147483647:2147483647"
NUMTCPSOCK="2147483647:2147483647"
NUMFLOCK="2147483647:2147483647"
NUMPTY="255:255"
NUMSIGINFO="1024:1024"
TCPSNDBUF="2147483647:2147483647"
TCPRCVBUF="2147483647:2147483647"
OTHERSOCKBUF="2147483647:2147483647"
DGRAMRCVBUF="2147483647:2147483647"
NUMOTHERSOCK="2147483647:2147483647"
DCACHESIZE="2147483647:2147483647"
NUMFILE="2147483647:2147483647"
NUMIPTENT="2147483647:2147483647"

DISKSPACE=""
DISKINODES=""
QUOTATIME="600"
IOPRIO="4"
CPULIMIT="75"
OFFLINE_MANAGEMENT="yes"
ARCH="x86"
PLATFORM="linux"

IP_ADDRESS="0.0.0.0"
NAMESERVER="69.55.225.225 69.55.230.3"
CPUUNITS="12313"
IOLIMIT="10485760"
IOPSLIMIT="100"
</pre>

* customize the control panel:
rm -fr /vz/backups

vi /var/opt/pva/pp/xsl/lib/vt/backups-forms.xsl

<pre>
<span style="font-size:13px; font-weight:bold">NOTE: You cannot create or view backups via this control panel- IT WILL NOT WORK. Please contact linux@johncompanies.com to setup or inquire about backups for your server.</span>

/var/opt/pva/pp/xsl/lib/vt/backups-list.xsl
</pre>

cd /var/opt/pva/pp/static/vz/skins
for f in `ls`; do scp backup2:/d4/build/vzcp/var/vzcp/static/vz/skins/winxp.new/images/logo_main_4.7.gif $f/images/logo_main.gif; done

* template install

To install new templates:

vzup2date -z

Copy over templates from old server. Ex: ubu 8.04 from virt19:

rsync -a -e ssh <span style="color:red">10.1.4.69</span>:/vz/template/ubuntu/8.04 /vz/template/ubuntu/8.04

install ubuntu 8.04 templates (get latest from vz)

<pre>cd /backup4/vzrpms/distro/ubuntu/8.04
rpm -ivh *
vzpkg update cache ubuntu-8.04-x86</pre>

If you install a tempalte from vzup2date it will download an old cache as well. Better to download the OS rpm from VZ site then create cache then use vzup2date to install the application templates.

* add traffic shaping support

Re-define class 0:

<pre>vi /etc/sysconfig/vz-scripts/networks_classes
0 69.55.224.0/20
0 10.1.4.0/24</pre>

make sure 1 is defined (0.0.0.0) and no other classes are needed

Then start it

/etc/init.d/vz shaperrestart

* udp dos fix

echo "/usr/local/sbin/iptables.sh" >> /etc/rc.d/rc.local

* make rc.local start before vz

mv /etc/rc.d/rc3.d/S99local /etc/rc.d/rc3.d/S95local

* disable unwanted services/daemons

Before:
<pre>abrt-ccpp 0:off 1:off 2:off 3:on 4:off 5:on 6:off
abrt-oops 0:off 1:off 2:off 3:on 4:off 5:on 6:off
abrtd 0:off 1:off 2:off 3:on 4:off 5:on 6:off
acpid 0:off 1:off 2:on 3:on 4:on 5:on 6:off
atd 0:off 1:off 2:off 3:on 4:on 5:on 6:off
auditd 0:off 1:off 2:on 3:on 4:on 5:on 6:off
autofs 0:off 1:off 2:off 3:on 4:on 5:on 6:off
cpuspeed 0:off 1:on 2:on 3:on 4:on 5:on 6:off
crond 0:off 1:off 2:on 3:on 4:on 5:on 6:off
cups 0:off 1:off 2:on 3:on 4:on 5:on 6:off
haldaemon 0:off 1:off 2:off 3:on 4:on 5:on 6:off
ip6tables 0:off 1:off 2:on 3:on 4:on 5:on 6:off
iptables 0:off 1:off 2:on 3:on 4:on 5:on 6:off
irqbalance 0:off 1:off 2:off 3:on 4:on 5:on 6:off
kdump 0:off 1:off 2:off 3:on 4:on 5:on 6:off
lvm2-monitor 0:off 1:on 2:on 3:on 4:on 5:on 6:off
mcelogd 0:off 1:off 2:off 3:on 4:off 5:on 6:off
mdmonitor 0:off 1:off 2:on 3:on 4:on 5:on 6:off
messagebus 0:off 1:off 2:on 3:on 4:on 5:on 6:off
netfs 0:off 1:off 2:off 3:on 4:on 5:on 6:off
network 0:off 1:off 2:on 3:on 4:on 5:on 6:off
nfslock 0:off 1:off 2:off 3:on 4:on 5:on 6:off
ntpd 0:off 1:off 2:off 3:on 4:off 5:off 6:off
portreserve 0:off 1:off 2:on 3:on 4:on 5:on 6:off
postfix 0:off 1:off 2:on 3:on 4:on 5:on 6:off
pvaagentd 0:off 1:off 2:on 3:on 4:on 5:on 6:off
pvapp 0:off 1:off 2:on 3:on 4:on 5:on 6:off
qpidd 0:off 1:off 2:on 3:on 4:on 5:on 6:off
rpcbind 0:off 1:off 2:on 3:on 4:on 5:on 6:off
rpcgssd 0:off 1:off 2:off 3:on 4:on 5:on 6:off
rpcidmapd 0:off 1:off 2:off 3:on 4:on 5:on 6:off
rsyslog 0:off 1:off 2:on 3:on 4:on 5:on 6:off
sshd 0:off 1:off 2:on 3:on 4:on 5:on 6:off
sysstat 0:off 1:on 2:on 3:on 4:on 5:on 6:off
udev-post 0:off 1:on 2:on 3:on 4:on 5:on 6:off
vz 0:off 1:off 2:on 3:on 4:on 5:on 6:off
vzlmond 0:off 1:off 2:on 3:on 4:on 5:on 6:off
vznetcfg 0:off 1:off 2:on 3:on 4:on 5:on 6:off
xinetd 0:off 1:off 2:off 3:on 4:on 5:on 6:off</pre>

<pre>chkconfig --level 3 cpuspeed off
chkconfig --level 3 cups off
chkconfig --level 3 haldaemon off
chkconfig --level 3 ip6tables off
chkconfig --level 3 iptables off
chkconfig --level 3 irqbalance off
chkconfig --level 3 lvm2-monitor off
chkconfig --level 3 mdmonitor off
chkconfig --level 3 messagebus off
chkconfig --level 3 netfs off
chkconfig --level 3 nfslock off
chkconfig --level 3 qpidd off
chkconfig --level 3 rpcgssd off
chkconfig --level 3 rpcidmapd off</pre>

* load mrtg graphs

yum install xinetd

create /etc/xinetd.d/load with the following contents:

<pre>echo "service load
{
disable = no
flags = REUSE
socket_type = stream
wait = no
user = root
server = /usr/local/sbin/load.pl
}" > /etc/xinetd.d/load</pre>

Add this line to /etc/services:
echo "load 12384/tcp" >> /etc/services

* Tasks to do on backup1:
setup backup dirs:

mkdir -p /data/<span style="color:red">virt13</span>/0

for f in 0 1 2 3 4 5 6; do mkdir -p /data/<span style="color:red">virt13</span>/$f/vz/private; mkdir -p /data/<span style="color:red">virt13</span>/$f/vz1/private; mkdir -p /data/<span style="color:red">virt13</span>/$f/vz2/private; done

add the system to
vi /usr/local/sbin/snapshot_rotate

* Tasks to do on mail:

Update backup graph:
vi /usr/local/www/mgmt/cgi/backupgraph.pl
(add hostname)

add to bbhosts:
<pre>vi /usr/home/bb/bbsrc/bb1.9i-btf/etc/bb-hosts
su bb
cd
bbsrc/bb/runbb.sh restart ; exit</pre>

add to load graph config:
vi /usr/local/www/mgmt/mrtg/mrtg1.cfg
(add new section for new machine)

* add to management db (on mail and devweb) jc.ref_machines and jc.ref_templates

insert new server:
insert into ref_machines values (null,'virt19','virt19.johncompanies.com',0,'l');

get new machine_id back out:
select * from ref_machines;

insert as a system to jc customer col00001:
insert into systems values ('', 'col00001',30,null,null,100,'virt11.johncompanies.com',null,'2010-03-11',null,null,1,0,null,null,null);

get sysid:
select * from systems where hostname='virt19.johncompanies.com';

update ipmap (main IP and IP used for VZCP):
update ipmap set cid='col00001', sysid=3772, assign_date='2008-01-07', status=1 where ip='69.55.236.2';
update ipmap set cid='col00001', sysid=3772, assign_date='2008-01-07', status=1 where ip='69.55.230.109';

add templates (or do via mgmt):
<pre>
insert into ref_templates values ('','fedora-core-1/20050725',33,'Fedora Core 1',1);
insert into ref_templates values ('','fedora-core-2/20050725',33,'Fedora Core 2',1);
insert into ref_templates values ('','debian-3.1/20050927',33,'Debian 3.1',1);</pre>

* Make sure that ns1c has proper forward and reverse entries for the new system (both private and public addresses), so that new system emailing will work.
<pre>vr johncompanies.com
vi internal.johncompanies.com
rndc reload johncompanies.com IN private</pre>
(edit the PTR too)

* add an outside blocking rule to the firewall, so this machine can only be reached from inside the firewall.

Follow example already in firewall virt19 is:

<pre>ipfw add 00143 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 or 62.213.72.101 or 64.131.90.27 or 82.200.65.190 or 91.204.24.0/22 or 194.44.160.178 or 195.214.232.10 or 195.214.233.10 or 208.251.212.1 } to 69.55.226.2
ipfw add 00143 deny ip from any to 69.55.226.2</pre>

(virt12 would be 00142)

* add to [[Cabinetmap|Cabinetmap]]

* make /vz1/private
<pre>mkdir /vz1/private ; chmod 0755 /vz1/private
mkdir /vz1/root ; chmod 0755 /vz1/root</pre>

* put config files in a place they’ll get backed up
<pre>mkdir /vzconf/confs
crontab -e
0 0 * * * rm -f /vzconf/confs/*.conf; cp /vzconf/*.conf /vzconf/confs</pre>

* fix pass/vzpp and localse issue with ubu 8.10

vi /vz/template/ubuntu/8.10/x86/config/os/default/post-install

sed '/^password.*pam_unix.so.*sha512/ s~sha512~md5~' -i etc/pam.d/common-password

localedef -i en_US -c -f UTF-8 en_US.UTF-8

* add tun device support

echo "modprobe tun" >> /etc/rc.local

* reinstall VZA/CT #1/CT #50

After the install, the PP was broken. So needed to reinstall everything. Did the pva_setup --uninstall and removed CT 50 and then setup the pva per http://kb.parallels.com/113071 then setup the CT 50 per the other article- centos6 had issues - and then authenticated with virt13 to get that into the management list.

http://kb.parallels.com/113072

http://kb.parallels.com/113071

* config logrotate (otherwise it creates weekly empty logs)

vi /etc/logrotate.conf

(add to end)

<pre>/var/log/messages {
rotate 12
monthly
size 10M
missingok
compress
delaycompress
notifempty
sharedscripts
postrotate
echo `date` > /var/log/messages
endscript
}</pre>

* install megacli64 (can we just move the bin and not install the lib?)

<pre>scp backup2: /d4/build/3ware/MegaCli-8.00.40-1.i386.rpm /root/MegaCli-8.00.40-1.i386.rpm
scp backup2: /d4/build/3ware/Lib_Utils-1.00-08.noarch.rpm /root/ Lib_Utils-1.00-08.noarch.rpm

rpm -ivh Lib_Utils-1.00-08.noarch.rpm
rpm -ivh MegaCli-8.00.40-1.i386.rpm

cp opt/MegaRAID/MegaCli/MegaCli64 /usr/local/sbin/megacli64</pre>

= Older versions =

== 4.0 ==

Last updated 2010-03-11

<pre>1. enable post info to console

2950:
Serial comm.: on with cons. Redir via com1
Failsafe: 115200
Redir after boot: enabled

Server
Custom- desel all but base
/root/install.log

2. Install OS

CentOS5.x (get latest from http://mirrors.kernel.org/centos/5.4/isos/i386/)

Set partitions:
Disk 1:
/ 3072M
6144M swap
/vz (the rest)
Disk 2:
6144M Swap

Use grub, install to MBR
No firewall
Disable SElinux
Install only base

3. create ssh key, add swsoft key to authorized_keys

(enter to everything)

cd /root/
ssh-keygen -t dsa

echo "ssh-dss AAAAB3NzaC1kc3MAAACBAIB7XHmpcsa1D/vsKS+mzEvBVo7udew/aRahbjnjywL4YXT7ruWzpGo7R3UA8GKIgVHoQGUbuXrJz5ICtBEXOlbgXVEz7Bu5nXOG2C9i/8pmFtUw6GwACII8uZE7t/Zq5CRj+5zN1Rv7n/4X4RAjzk5BHXoZ62a32FTbS4tPvyPrAAAAFQDiUi7YEiffNzGvB544lQpRdn29TwAAAIBkUchLTBE1+sX11iC80yhNh9EGeCMLdkT8UqHsTSaD0myQPxGALuNY4rO31q+25uKqSRDTXd6ZWXp4bplA6Ls6jl7r7XVR9loJ7WvRMqc4Dv68TZ7h2UBqAhu4det2UF4LHim3DNf1gKn9DzWNkmI/N2FvVVAL4aM3sPunVIvUYgAAAIAJX38INABVgnCEBKqpv5FMLm3k9MgMn54oQFmylBbgM0w4SUeuDj3jLrkJf8dT9w8ByEDb8xviexpWrPsbz2DeThle4xgzcyKrUkt2Wugtmm4DwS0S/68BAA9wXoWD/vZ67GBoCKH7uq8VAjUVcCbJ/7IydljnFAZoJpktdwN7iQ== russup@support.sw-soft.com" >> /root/.ssh/authorized_keys

4. setup access to backup2:

echo "10.1.4.3 backup2" >> /etc/hosts
echo "10.1.4.8 backup1" >> /etc/hosts

# Copy id_dsa.pub to backup2 and add it to the end of ~root/.ssh/authorized_keys:

cat .ssh/id_dsa.pub | ssh backup2 'cat - >> /root/.ssh/authorized_keys'
cat .ssh/id_dsa.pub | ssh backup1 'cat - >> /root/.ssh/authorized_keys'

ssh backup2 hostname
ssh backup1 hostname

(confirm you can connect w/o entering a password)

5. add bash backward history search
echo "\"\M-z\": history-search-backward
\"\M-x\": history-search-backward " > /root/.inputrc
echo "export GREP_OPTIONS='--color=auto'" >> ~/.bashrc
echo "\"\e[5~\": history-search-backward" >> ~/.inputrc
echo "\"\e[6~\": history-search-forward" >> ~/.inputrc

6. add to /root/.bashrc, create log dir

vi /root/.bashrc
alias v='vzctl'
alias mbm='mb mount'
alias mbu='mb umount'
export PS1="[\u@\h \w]# "
alias cv='cd /vz/private'
alias cv1='cd /vz1/private'
alias cv2='cd /vz2/private'

source /root/.bashrc

mkdir /root/logs

7. make sure serial console and kernel messages are working

add these 2 lines to the top:

vi /boot/grub/menu.lst
serial --unit=0 --speed= 115200
terminal --timeout=10 serial console

and the kernel line should look like:

kernel /boot/vmlinuz-2.4.20-020stab009.24.777-enterprise ro root=/dev/sda1 console=tty0 console=ttyS0,115200

Remove or comment out these lines:
hiddensplashmenu
splashimage=(hd0,0)/boot/grub/splash.xpm.gz

add to /etc/inittab and /etc/securetty

echo "s0:2345:respawn:/sbin/agetty -L ttyS0 115200 linux" >> /etc/inittab
echo "ttyS0" >> /etc/securetty

kill -1 1

DEPRECATED 8. configuring eth1 for management net:

echo "10.1.5/24 via 10.1.4.2
10.1.6/24 via 10.1.4.2" >> /etc/sysconfig/network-scripts/route-eth1

reboot the machine to make sure that the interface is configured properly.

also verify that the routes are in place like so:

netstat -rn | grep 10.1.4.2
10.1.5.0 10.1.4.2 255.255.255.0 UG 40 0 0 eth1
10.1.6.0 10.1.4.2 255.255.255.0 UG 40 0 0 eth1

9. make sure the hostname on the system is properly set - in the form virt4.johncompanies.com - and make sure it is set properly:
cat /etc/sysconfig/network

10. Check/update fstab

vi /etc/fstab

/etc/fstab entries should look something like (note noatime option):

/dev/sda3 /vz ext3 defaults,noatime 1 2
/dev/sdb2 /vz1 ext3 defaults,noatime 1 2

make sure swap is configured in fstab

Use real devices instead of LABEL

11. Install Virtuozzo 4.0

umount /vz
umount /vz1

mke2fs -i1024 -b1024 -j /dev/sda3 &
mke2fs -i1024 -b1024 -j /dev/sdb2 &

Experimentally ran this for sdc on virt12:
mke2fs -i2048 -b2048 -j /dev/sdb1

mount -a

scp backup2:/d4/vzrpms/iso/vzinstall-linux.bin /vz
cd /vz
chmod 0700 vzinstall-linux.bin
./vzinstall-linux.bin

change wd to /vz
Ignore OS version error
Custom install (don’t install templates)

reboot

vzup2date

(run again. till no more updates offered)

reboot

Run vzup2date (make sure to choose enterprise kernel) and/or install latest kernel:

NOTE: skip over the manual kernel load below if you can get the 16G recognized before running vzup2date- then it will dl the ent kernel

NOTE: go to /vz/vzup2date/virtuozzo/linux/i386/4.0.0/CU-2.6.18-028stab068.3 and install kern there (downloaded by vzup2date)

<DEPREACATED>
ftp downloads.swsoft.com
User:
gdawson
Pass:
qjdK1R

cd /updates/Rel400/20080307
bin
get vzkernel-ent-2.6.18-028stab053.10.i686.rpm
get vzmodules-ent-2.6.18-028stab053.10.i686.rpm
bye

rpm -ivh vzkernel-ent-2.6.18-028stab053.10.i686.rpm \
vzmodules-ent-2.6.18-028stab053.10.i686.rpm
</DEPREACATED>

vi /boot/grub/menu.lst
(set default kernel to new kernel)

12. Install ntp

yum install ntp

rm -f /etc/ntp.conf

# Create a new file with one line:

echo "server 10.1.4.5" > /etc/ntp.conf

# Then set ntpd so it will start at boot:

chkconfig --level 3 ntpd on
service ntpd start

# Make sure it’s talking to the ntp server:

ntpq -p

Correct output should have * or + to the left of the IP:

remote refid st t when poll reach delay offset jitter
==============================================================================
10.1.4.105 clock1.redhat.c 2 u - 64 1 0.370 -706677 0.008

13. copy over binaries/scripts from backup2

scp backup2:/mnt/data4/bin/linux/* /usr/local/sbin/
scp 10.1.4.62:/vzconf/jctmpl* /etc/sysconfig/vz-scripts/
scp 10.1.4.62:/vzconf/ve-slm* /etc/sysconfig/vz-scripts/

14. setup root crontab (alter backup time):
crontab -e
1 1 * * * /usr/local/sbin/backup
*/15 * * * * /usr/local/sbin/vim.watchdog.30 > /dev/null
#56 4,16 * * * root /usr/sbin/vzkeyupdate -c
*/5 * * * * /usr/local/sbin/perc5iraidchk

15. setup backups

echo '#!/bin/sh
backupdir=/data/virt11
server=backup1

## ENTRY /etc
## ENTRY /usr/local/sbin
## ENTRY /root/logs
' > /usr/local/sbin/backup.config

16. setup bb

edit passwd and add the bb user:
echo "bb:x:1984:1984:Big Brother:/home/bb:/bin/bash" >> /etc/passwd

# edit group and add the bb group:
echo "bb:x:1984:" >> /etc/group

# rebuild the password database:

pwconv

# Create bb’s home dir and set ownership:

mkdir /home/bb
chown bb.bb /home/bb

# Go to bb’s home and fetch the client tarball:

cd ~bb
scp backup2:/mnt/data4/build/bb/bb-linux.tar .

tar xf bb-linux.tar

cd bbc1.9e-btf/etc/

echo "10.1.4.5 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh" > /home/bb/bbc1.9e-btf/etc/bb-hosts
echo "10.1.4.61 virt11.johncompanies.com # ssh" >> /home/bb/bbc1.9e-btf/etc/bb-hosts

chmod +r /var/log/messages

cd ..
su bb
./runbb.sh start
exit

# Add this line to /etc/rc.d/rc.local so bb will start at boot:

echo "su - bb -c \"cd /home/bb/bbc1.9e-btf; ./runbb.sh start\"" >> /etc/rc.d/rc.local

17. Uncomment/add lines with the appropriate IP addresses in /etc/ssh/sshd_config:

vi /etc/ssh/sshd_config
ListenAddress 69.55.238.3
ListenAddress 10.1.4.61

And make sure to uncomment
Protocol 2,1

18. setup backup2 nfs mounts

setup portmap to start with server, and start it:

chkconfig --level 3 portmap on
service portmap start

mb mount

run df to confirm that nfs mounts are mounted

19. make alias for /vzconf
ln -s /etc/sysconfig/vz-scripts /vzconf

DEPRECATED 20. copy over the /etc/sysconfig/vz-scripts/4unlimited template file from the last virt server and put it in /etc/sysconfig/vz-scripts

scp 10.1.4.68:/etc/sysconfig/vz-scripts/4unlimited /etc/sysconfig/vz-scripts/

NOTE: if this system is more powerful (RAM, CPU) then the previous, you will want to create a new one using vzsplit:

cd /etc/sysconfig/vz-scripts/
vzsplit -f 4unlimited

21. edit /etc/sysconfig/vz

vi /etc/sysconfig/vz

enable fastboot:
VZFASTBOOT=yes

Disable auto mount
#VZMOUNTS="/vz"

Enable IPV6
IPV6=yes

add/edit to traffic parameters section:
TRAFFIC_SHAPING=yes
BANDWIDTH="eth0:102400"
TOTALRATE="eth0:1:102400"
RATE="eth0:1:50000"

for 2 nics, uncomment this line:
VE_ROUTE_SRC_DEV="eth0"

Make sure the IPTABLES section looks like:
IPTABLES="ip_tables iptable_filter iptable_mangle ipt_limit ipt_multiport ipt_tos ipt_TOS ipt_REJECT ipt_tcpmss ipt_ttl ipt_LOG ipt_length ip_conntrack ip_conntrack_ftp ip_conntrack_irc ipt_conntrack ipt_state ipt_helper iptable_nat ip_nat_ftp ip_nat_irc ipt_REDIRECT ipt_multiport ipt_TCPMSS"

22. disable periodic fsck's, lower the root padding on the /vz and /vz1 filesystems to 100MB

tune2fs -i 0 /dev/sda3
tune2fs -i 0 /dev/sdb2
tune2fs -c -1 /dev/sda3
tune2fs -c -1 /dev/sdb2
tune2fs -r 100000 /dev/sda3
tune2fs -r 100000 /dev/sdb2

23. setup vzagent with pub IP (to allow migrations from 3.x):

vzctl set 1 --ipdel all --ipadd 69.55.230.57 --userpasswd vzagent0:k0zub1k --save
vzagent_ctl restart

24. edit ve config files:

From split of 80 take
CPUUNITS

And put into files in /usr/local/sbin/ve-slm.1024MB.conf et. al.

25. customize the control panel:

vzmlocal 1:1:/vz/private/1:/vz/root/1
cd /vz/private/1/fs/root/var/vzcp/static/vz/skins
for f in `ls`; do scp backup2:/d4/build/vzcp/var/vzcp/static/vz/skins/winxp.new/images/logo_main.gif $f/images/logo_pim.gif; done

1) comment 'Backup' tab in /etc/vzcp/pp/tabs.xml:

<tab-bar id="maintance" obj-type="env.core.vt">


2) change 'href' parameter if 'Maintentance' item in /etc/vzcp/pp/menu.xml:


<item id="management.maintenance" href="/vz/cp/vzdir/infrman/envs/repair?branch=env-home,vzpp" icon="backup" tip="maintenance_screen">

3) /etc/vzcp/pp/dashboard.xml

<item id="management.maintenanance" href="$baseurl/vzdir/infrman/envs/repair" title="Maintenance" icon="env_backup" tip="maintenance_screen" cred="vzcp.env.backups.list" security-obj-type="ve.vz.vt">

URL for 'href' parameter can be obtained from tabs.xml.

vi /var/vzcp/xsl/screens/vzdir/monitman/traffic.xsl
<sw:text>Traffic for selected period and class (NOTE: this information is not certified by JohnCompanies and is not used to calculate actual usage or charges)</sw:text>

<DEPREACATED>
/vz/private/1/root/var/vzcp/static/vz/skins/
/vz/private/1/root/etc/vzcp/pp/menu.xml
/vz/private/1/root/etc/vzcp/pp/dashboard.xml
pp/tabs.xml
/vz/private/1/root/etc/vzcp/vzcpcon.conf

eg (2.6.1 or 3.0):
rsync -va -e ssh backup2:/d4/build/vzcp/var/vzcp/static/vz/skins/ /vz/private/1/root/var/vzcp/static/vz/skins/
rsync -va -e ssh backup2:/d4/build/vzcp/etc/vzcp/pp/menu.xml /vz/private/1/root/etc/vzcp/pp/menu.xml
rsync -va -e ssh backup2:/d4/build/vzcp/etc/vzcp/pp/dashboard.xml /vz/private/1/root/etc/vzcp/pp/dashboard.xml
rsync -va -e ssh backup2:/d4/build/vzcp/etc/vzcp/vzcpcon.conf /vz/private/1/root/etc/vzcp/vzcpcon.conf
</DEPREACATED>

24. template install

To install new templates:

vzup2date –z

Copy over ubu 8.04 from virt19

rsync -a -e ssh 10.1.4.69:/vz/template/ubuntu/8.04 /vz/template/ubuntu/8.04

install ubuntu 8.04 templates (get latest from vz)

cd /backup4/vzrpms/distro/ubuntu/8.04
rpm -ivh *
vzpkg update cache ubuntu-8.04-x86

26. add traffic shaping support

Re-define class 0:

vi /etc/sysconfig/vz-scripts/networks_classes
0 69.55.224.0/20
0 10.1.4.0/24

make sure 1 is defined (0.0.0.0) and no other classes are needed

Then start it

/etc/init.d/vz shaperrestart

27. udp dos fix

echo "/usr/local/sbin/iptables.sh" >> /etc/rc.d/rc.local

28. make rc.local start before vz

mv /etc/rc.d/rc3.d/S99local /etc/rc.d/rc3.d/S95local

29. disable unwanted services/daemons

chkconfig --level 3 avahi-daemon off
chkconfig --level 3 bluetooth off
chkconfig --level 3 cpuspeed off
chkconfig --level 3 cups off
chkconfig --level 3 firstboot off
chkconfig --level 3 gpm off
chkconfig --level 3 haldaemon off
chkconfig --level 3 hidd off
chkconfig --level 3 ip6tables off
chkconfig --level 3 iptables off
chkconfig --level 3 irqbalance off
chkconfig --level 3 isdn off
chkconfig --level 3 kudzu off
chkconfig --level 3 lvm2-monitor off
chkconfig --level 3 mcstrans off
chkconfig --level 3 mdmonitor off
chkconfig --level 3 messagebus off
chkconfig --level 3 netfs off
chkconfig --level 3 nfslock off
chkconfig --level 3 pcscd off
chkconfig --level 3 rawdevices off
chkconfig --level 3 readahead_early off
chkconfig --level 3 restorecond off
chkconfig --level 3 rpcgssd off
chkconfig --level 3 rpcidmapd off
chkconfig --level 3 smartd off
chkconfig --level 3 yum-updatesd off

30. load mrtg graphs

yum install xinetd

create /etc/xinetd.d/load with the following contents:

echo "service load
{
disable = no
flags = REUSE
socket_type = stream
wait = no
user = root
server = /usr/local/sbin/load.pl
}" > /etc/xinetd.d/load

# Add this line to /etc/services:
echo "load 12384/tcp" >> /etc/services

31. Tasks to do on backup1:

setup backup dirs:
mkdir -p /data/virt11/0

add the system to
vi /usr/local/sbin/snapshot_rotate

32. Tasks to do on mail:

Update backup graph:
vi /usr/local/www/mgmt/cgi/backupgraph.pl
(add hostname)

add to bbhosts:
vi /usr/home/bb/bbsrc/bb1.9e-btf/etc/bb-hosts
su bb
cd
bbsrc/bb/runbb.sh restart ; exit

add to load graph config:
vi /usr/local/www/mgmt/mrtg/mrtg1.cfg
(add new section for new machine)

33. add to management db (on mail and devweb) jc.ref_machines and jc.ref_templates

insert into ref_machines values (null,'virt19','virt19.johncompanies.com',0,'l');
select * from ref_machines;
insert into systems values ('', 'col00001',30,null,null,100,'virt11.johncompanies.com',null,'2010-03-11',null,null,1,0,null,null,null);
select * from systems where hostname='virt19.johncompanies.com';
update ipmap set cid='col00001', sysid=3772, assign_date='2008-01-07', status=1 where ip='69.55.236.2';
update ipmap set cid='col00001', sysid=3772, assign_date='2008-01-07', status=1 where ip='69.55.230.109';
insert into ref_templates values ('','fedora-core-1/20050725',33,'Fedora Core 1',1);
insert into ref_templates values ('','fedora-core-2/20050725',33,'Fedora Core 2',1);
insert into ref_templates values ('','debian-3.1/20050927',33,'Debian 3.1',1);

34. Make sure that ns1c has proper forward and reverse entries for the new system (both private and public addresses), so that new system emailing will work.
vr johncompanies.com
vi internal.johncompanies.com
rndc reload johncompanies.com IN private
(edit the PTR too)

35. add an outside blocking rule to the firewall, so this machine can only be reached from inside the firewall.

Follow example already in firewall virt19 is:

ipfw add 00149 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 or 62.213.72.101 or 64.131.90.27 or 82.200.65.190 or 91.204.24.0/22 or 194.44.160.178 or 195.214.232.10 or 195.214.233.10 or 208.251.212.1 } to 69.55.236.2
ipfw add 00149 deny ip from any to 69.55.236.2

virt12 would be 00142

36. add to server/cabinet map

37. make /vz1/private
mkdir /vz1/private ; chmod 0755 /vz1/private
mkdir /vz1/root ; chmod 0755 /vz1/root

38. put config files in a place they’ll get backed up
mkdir /vzconf/confs
crontab -e
0 0 * * * rm -f /vzconf/confs/*.conf; cp /vzconf/*.conf /vzconf/confs

39. fix pass/vzpp and localse issue with ubu 8.10

vi /vz/template/ubuntu/8.10/x86/config/os/default/post-install

sed '/^password.*pam_unix.so.*sha512/ s~sha512~md5~' -i etc/pam.d/common-password

localedef -i en_US -c -f UTF-8 en_US.UTF-8

40. add tun device support

echo "modprobe tun" >> /etc/rc.local</pre>

== 3.x ==

Last updated 2007-04-5

<pre>1. enable post info to console
Supermicro:
Console redirection:
Com port addr: on-board COM A
Baud: 38400
Console type: vt100
Flow control: none
Console connection: direct
Continue cr after post: off

2450:
Make sure running bios A09
Console Redirection: VT100/VT220

2950:
Serial comm.: on with cons. Redir via com1
Failsafe: 115200
Redir after boot: enabled

6144
Server
Custom- desel all but base
/root/install.log

2. Install OS
Set partitions:
/ 3072M
Disk 1: Swap 4096M
Disk 2: Swap 4096M
/vz (the rest)

No firewall
Disable SElinux

FC4
Choose a custom install, the “minimal” package

CentOS4 Server CD (only option for 2950)
Choose the “minimal” package

3. create ssh key, add swsoft key to authorized_keys

(enter to everything)

cd /root/
ssh-keygen -t dsa

echo "ssh-dss AAAAB3NzaC1kc3MAAACBAIB7XHmpcsa1D/vsKS+mzEvBVo7udew/aRahbjnjywL4YXT7ruWzpGo7R3UA8GKIgVHoQGUbuXrJz5ICtBEXOlbgXVEz7Bu5nXOG2C9i/8pmFtUw6GwACII8uZE7t/Zq5CRj+5zN1Rv7n/4X4RAjzk5BHXoZ62a32FTbS4tPvyPrAAAAFQDiUi7YEiffNzGvB544lQpRdn29TwAAAIBkUchLTBE1+sX11iC80yhNh9EGeCMLdkT8UqHsTSaD0myQPxGALuNY4rO31q+25uKqSRDTXd6ZWXp4bplA6Ls6jl7r7XVR9loJ7WvRMqc4Dv68TZ7h2UBqAhu4det2UF4LHim3DNf1gKn9DzWNkmI/N2FvVVAL4aM3sPunVIvUYgAAAIAJX38INABVgnCEBKqpv5FMLm3k9MgMn54oQFmylBbgM0w4SUeuDj3jLrkJf8dT9w8ByEDb8xviexpWrPsbz2DeThle4xgzcyKrUkt2Wugtmm4DwS0S/68BAA9wXoWD/vZ67GBoCKH7uq8VAjUVcCbJ/7IydljnFAZoJpktdwN7iQ== russup@support.sw-soft.com" >> /root/.ssh/authorized_keys

4. setup access to backup2:

echo "10.1.4.3 backup2" >> /etc/hosts

Copy id_dsa.pub to backup2 and add it to the end of ~root/.ssh/authorized_keys:

cat .ssh/id_dsa.pub | ssh backup2 'cat - >> /root/.ssh/authorized_keys'

ssh backup2

(confirm you can connect w/o entering a password)

5. add bash backward history search
echo "\"\M-z\": history-search-backward
\"\M-x\": history-search-backward " > /root/.inputrc
echo "export GREP_OPTIONS='--color=auto'" >> ~/.bashrc
echo "\"\e[5~\": history-search-backward" >> ~/.inputrc
echo "\"\e[6~\": history-search-forward" >> ~/.inputrc

6. add to /root/.bashrc, create log dir

vi /root/.bashrc
alias v='vzctl'
alias mbm='mb mount'
alias mbu='mb umount'
export PS1="[\u@\h \w]# "
alias cv='cd /vz/private'
alias cv1='cd /vz1/private'
alias cv2='cd /vz2/private'

mkdir /root/logs

7. make sure serial console and kernel messages are working

grub

add these 2 lines to the top:

vi /boot/grub/menu.lst
serial --unit=0 --speed= 115200
terminal --timeout=10 serial console

and the kernel line should look like:

kernel /boot/vmlinuz-2.4.20-020stab009.24.777-enterprise ro root=/dev/sda1 console=tty0 console=ttyS0,115200

Remove the line:
hiddensplashmenu

lilo

Make sure the append line in /etc/lilo.conf looks like this:

append="console=tty console=ttyS0,38400"

If you edit /etc/lilo.conf don’t forget to run lilo

# lilo
Added linux
Added linux-up
Added linux+virtuozzo *

(note: the last console listed is the one that gets output from the rc scripts.)

add to /etc/inittab and /etc/securetty

echo "s0:2345:respawn:/sbin/agetty -L ttyS0 115200 linux" >> /etc/inittab
echo "ttyS0" >> /etc/securetty

kill -1 1

8. configuring eth1 for management net:

echo "10.1.5/24 via 10.1.4.2
10.1.6/24 via 10.1.4.2" >> /etc/sysconfig/network-scripts/route-eth1

reboot the machine to make sure that the interface is configured properly.

also verify that the routes are in place like so:

netstat -rn | grep 10.1.4.2
10.1.5.0 10.1.4.2 255.255.255.0 UG 40 0 0 eth1
10.1.6.0 10.1.4.2 255.255.255.0 UG 40 0 0 eth1

9. make sure the hostname on the system is properly set - in the form virt4.johncompanies.com - and make sure it is set properly:
cat /etc/sysconfig/network

10. Check/update fstab

vi /etc/fstab

/etc/fstab entries should look something like (note noatime option):

/dev/sda3 /vz ext3 defaults,noatime 1 2
/dev/sdb1 /vz1 ext3 defaults,noatime 1 2

make sure swap is configured in fstab

Use real devices instead of LABEL

11. Install Virtuozzo 3.0

If it wasn’t done at install time:

setenforce 0

umount /vz
umount /vz1

mke2fs -i1024 -b1024 -j /dev/sda3 &
mke2fs -i1024 -b1024 -j /dev/sdb2 &

On CentOS 4:
yum install rpm-build; yum update iproute

mount -a

scp backup2:/d4/vzrpms/iso/Rel300.tar.gz /vz/.

cd /vz

tar xzf Rel300.tar.gz

cd Rel300/HW

./install

Choose Enterprise kernel
Vzagent password: k0zub1k
DON’T reboot

vzup2date
User:
gdawson
Pass:
qjdK1R
(run again. till no more updates offered)

reboot

Run vzup2date (make sure to choose enterprise kernel) and/or install latest kernel:

ftp downloads.swsoft.com
cd /updates/Rel300/20061206
bin
get vzkernel-enterprise-2.6.9-023stab033.9.i686.rpm
get vzmodules-enterprise-2.6.9-023stab033.9.i686.rpm
bye

rpm -ivh vzkernel-enterprise-2.6.9-023stab033.9.i686.rpm \
vzmodules-enterprise-2.6.9-023stab033.9.i686.rpm

vi /boot/grub/menu.lst
(set default kernel to new kernel)

12. Install various RPMs

scp backup2:/mnt/data4/linuxrpms/* /tmp

install them in this order:

12a. http fetcher

rpm -ivh /tmp/http_fetcher-1.1.0-1.i386.rpm
rpm -ivh /tmp/fetch-1.0-1.i386.rpm

(ignore error on CentOS4)

Now the fetch command, which vemake calls to email the new system owner, will work (it downloads the welcome email from www.johncompanies.com

12b. ntp client

sometimes ntp isn’t included in which case:

FCx:

rpm -ivh /tmp/libcap-1.10-22.i386.rpm
rpm -ivh /tmp/compat-readline43-4.3-2.i386.rpm
rpm -ivh /tmp/ntp-4.1.2-5.i386.rpm

CentOS4:

rpm -ivh /tmp/ntp-4.1.2-5.i386.rpm

rm -f /etc/ntp.conf

Create a new file with one line:

echo "server 10.1.4.5" > /etc/ntp.conf

Then set ntpd so it will start at boot:

chkconfig --level 3 ntpd on
service ntpd start

Make sure it’s talking to the ntp server:

ntpq -p

Correct output should have something unfer the refid col:

remote refid st t when poll reach delay offset jitter
==============================================================================
10.1.4.105 clock1.redhat.c 2 u - 64 1 0.370 -706677 0.008

12c. adaptec machines only: install aaccli for raid controller (LSI card util is with bin files):

rpm -ivh /tmp/aacapps-4.1-0.i386.rpm

May have to run:

cd /dev
./MAKEDEV.aac aac0

13. copy over binaries/scripts from backup2

scp backup2:/mnt/data4/bin/linux/* /usr/local/sbin/

14. setup root crontab (alter backup time):
crontab -e
1 1 * * * /usr/local/sbin/backup
*/15 * * * * /usr/local/sbin/vim.watchdog.30 > /dev/null
#56 4,16 * * * root /usr/sbin/vzkeyupdate -c

On Perc5/i machines:

*/5 * * * * /usr/local/sbin/perc5iraidchk

15. setup backups

echo '#!/bin/sh
backupdir=/mnt/data4/virt19

## ENTRY /etc
## ENTRY /usr/local/sbin
## ENTRY /root/logs
' > /usr/local/sbin/backup.config

16. setup bb

edit passwd and add the bb user:
echo "bb:x:1984:1984:Big Brother:/home/bb:/bin/bash" >> /etc/passwd

edit group and add the bb group:
echo "bb:x:1984:" >> /etc/group

rebuild the password database:

pwconv

Create bb’s home dir and set ownership:

mkdir /home/bb
chown bb.bb /home/bb

Go to bb’s home and fetch the client tarball:

cd ~bb
scp backup2:/mnt/data4/build/bb/bb-linux.tar .

tar xf bb-linux.tar

cd bbc1.9e-btf/etc/

echo "10.1.4.5 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh" > /home/bb/bbc1.9e-btf/etc/bb-hosts
echo "10.1.4.69 virt19.johncompanies.com # ssh" >> /home/bb/bbc1.9e-btf/etc/bb-hosts

chmod +r /var/log/messages

cd ..
su bb
./runbb.sh start
exit

Add this line to /etc/rc.d/rc.local so bb will start at boot:

echo "su - bb -c \"cd /home/bb/bbc1.9e-btf; ./runbb.sh start\"" >> /etc/rc.d/rc.local

17. Uncomment/add lines with the appropriate IP addresses in /etc/ssh/sshd_config:

vi /etc/ssh/sshd_config
ListenAddress 69.55.232.162
ListenAddress 10.1.4.66

And make sure to uncomment
Protocol 2,1

18. setup backup2 nfs mounts

setup portmap to start with server, and start it:

chkconfig --level 3 portmap on
service portmap start

mb mount

run df to confirm that nfs mounts are mounted

19. m ake alias for /vzconf
ln -s /etc/sysconfig/vz-scripts /vzconf

20. copy over the /etc/sysconfig/vz-scripts/4unlimited template file from the last virt server and put it in /etc/sysconfig/vz-scripts

scp 10.1.4.68:/etc/sysconfig/vz-scripts/4unlimited /etc/sysconfig/vz-scripts/

NOTE: if this system is more powerful (RAM, CPU) then the previous, you will want to create a new one using vzsplit:

cd /etc/sysconfig/vz-scripts/
vzsplit -f 4unlimited

21. edit /etc/sysconfig/vz

vi /etc/sysconfig/vz

enable fastboot:
VZFASTBOOT=yes

add/edit to traffic parameters section:
TRAFFIC_SHAPING=yes
BANDWIDTH="eth0:102400"
TOTALRATE="eth0:1:102400"

for 2 nics, uncomment this line:
VE_ROUTE_SRC_DEV="eth0"

Make sure the IPTABLES section looks like:
IPTABLES="ip_tables iptable_filter iptable_mangle ipt_limit ipt_multiport ipt_tos ipt_TOS ipt_REJECT ipt_tcpmss ipt_ttl ipt_LOG ipt_length ip_conntrack ip_conntrack_ftp ip_conntrack_irc ipt_conntrack ipt_state ipt_helper iptable_nat ip_nat_ftp ip_nat_irc ipt_REDIRECT"

22. disable periodic fsck's, lower the root padding on the /vz and /vz1 filesystems to 100MB

tune2fs -i 0 /dev/sda3
tune2fs -i 0 /dev/sdb2
tune2fs -c -1 /dev/sda3
tune2fs -c -1 /dev/sdb2
tune2fs -r 100000 /dev/sda3
tune2fs -r 100000 /dev/sdb2

23. copy over our versions of the vzpp files:

/vz/private/1/root/var/vzcp/static/vz/skins/
/vz/private/1/root/etc/vzcp/pp/menu.xml
/vz/private/1/root/etc/vzcp/pp/dashboard.xml
/vz/private/1/root/etc/vzcp/vzcpcon.conf

eg (2.6.1 or 3.0):
rsync -va -e ssh backup2:/d4/build/vzcp/var/vzcp/static/vz/skins/ /vz/private/1/root/var/vzcp/static/vz/skins/
rsync -va -e ssh backup2:/d4/build/vzcp/etc/vzcp/pp/menu.xml /vz/private/1/root/etc/vzcp/pp/menu.xml
rsync -va -e ssh backup2:/d4/build/vzcp/etc/vzcp/pp/dashboard.xml /vz/private/1/root/etc/vzcp/pp/dashboard.xml
rsync -va -e ssh backup2:/d4/build/vzcp/etc/vzcp/vzcpcon.conf /vz/private/1/root/etc/vzcp/vzcpcon.conf

To disable backups from scratch:

In svcve edit /etc/vzcp/vzcpcon.conf

In section:

<auth>
<type>VEUser</type>
<user>root</user>

Add into/below:
<cred>
<id>maintenance</id>
</cred>

The following:
<cred>
<id>show</id>
</cred>
<cred>
<id>reinstall</id>
</cred>
<cred>
<id>repair_mode</id>
</cred>

Will finally look like:

<cred>
<id>maintenance</id>
<cred>
<id>show</id>
</cred>
<cred>
<id>reinstall</id>
</cred>
<cred>
<id>repair_mode</id>
</cred>
</cred>
</cred>

24. template install

mb mount

install debian 3.0 templates

cd /backup4/vzrpms/distro/debian/3.0/p1/
for f in `cat /backup4/vzrpms/distro/debian/3.0/install_list`; do rpm -ivh $f; done

install debian 3.1 templates

cd /backup4/vzrpms/distro/debian/3.1/p3/
for f in `ls`; do rpm -ivh $f; done

install redhat templates

cd /backup4/vzrpms/distro/rh9/current/
for f in `cat /backup4/vzrpms/distro/rh9/install_list`; do rpm -ivh $f; done

install fedora1 templates

cd /backup4/vzrpms/distro/fc1/current/
for f in `cat /backup4/vzrpms/distro/fc1/install_list`; do rpm -ivh $f; done

change 'krb5-devel-1.3.4-5' to 'krb5-devel-1.3.3-6' in (applies to pre-p5)
/vz/template/fedora-core-devel-1/conf/fedora-core-devel-1.conf.20050314

install fedora2 templates

cd /backup4/vzrpms/distro/fc2/current/
for f in `cat /backup4/vzrpms/distro/fc2/install_list`; do rpm -ivh $f; done

install suse9.2 templates

cd /backup4/vzrpms/distro/suse/current/
for f in `cat /backup4/vzrpms/distro/suse/install_list`; do rpm -ivh $f; done

install centos templates

cd /backup4/vzrpms/distro/centos/current/
for f in `cat /backup4/vzrpms/distro/centos/install_list`; do rpm -ivh $f; done

install fedora4 templates

cd /backup4/vzrpms/distro/fc4/current/
for f in `cat /backup4/vzrpms/distro/fc2/install_list`; do rpm -ivh $f; done

vzpkgcache

To pull templates from another machine (before hand stop ve 1 and move it’s template so you preserve the latest template files just setup by vzup2date):

rsync -a -e ssh 10.1.4.68:/vz/template/ /vz/template/

Make sure ubuntu source is defined:

UBU_SERVER=http://archive.ubuntu.com

And change FC source:
FC_SERVER=http://rhold.fedoraproject.org

In /etc/vztt/vztt.conf

rsync -a -e ssh 10.1.4.68:/etc/vztt/vztt.conf /etc/vztt/

To update the cache:

vzpkg update cache ubuntu-6.10-x86

25. add tun device support

Add to /etc/modules.conf:
echo "alias char-major-10-200 tun" >> /etc/modules.conf

depmod -a

echo "modprobe tun" >> /etc/rc.local

26. add traffic shaping support

Re-define class 0:

vi /etc/sysconfig/vz-scripts/networks_classes
0 69.55.224.0/20
0 10.1.4.0/24

make sure 1 is defined (0.0.0.0) and no other classes are needed

Then start it

/etc/init.d/vz shaperrestart

27. RAIDmon (only applies to LSI 320 PCI card)

cd /tmp
cp /backup4/build/megaraid/linuxmon30.tgz .
tar xzf linuxmon30.tgz
cd linuxmon30
./install
(say no)
mv –f raidmon.init /etc/init.d/raidmon
service raidmon start
tail -f /var/log/messages
(check for output)

28. udp dos fix

echo "/usr/local/sbin/iptables.sh" >> /etc/rc.d/rc.local

29. make rc.local start before vz

mv /etc/rc.d/rc3.d/S99local /etc/rc.d/rc3.d/S95local

30. disable unwanted services/daemons

On CentOS:
chkconfig --del kudzu
chkconfig --del cpuspeed

This should be taken care of by vz install so probably safe to skip

chkconfig --del kudzu
chkconfig --del cups
chkconfig --del gpm
chkconfig --del isdn
chkconfig --del pcmcia
chkconfig --del apmd
chkconfig --del rhnsd
chkconfig --del mdmonitor

31. load mrtg graphs

On FCx xinetd not installed so:

rpm -ivh /tmp/xinetd-2.3.11-1.7x.i386.rpm

create /etc/xinetd.d/load with the following contents:

echo "service load
{
disable = no
flags = REUSE
socket_type = stream
wait = no
user = root
server = /usr/local/sbin/load.pl
}" > /etc/xinetd.d/load

Add this line to /etc/services:
echo "load 12384/tcp" >> /etc/services

On FCx: if xinetd isn’t running, configure it for startup at boot time:

chkconfig --level 3 xinetd on
service xinetd start

32. Tasks to do on backup2:

setup backup dirs:
mkdir -p /mnt/data4/virt19/0

add the system to
vi /mnt/data4/bin/rsync_houseclean

33. Tasks to do on mail:

Update backup graph:
vi /usr/local/www/mgmt/cgi/backupgraph.pl
(add hostname)

add to bbhosts:
vi /usr/home/bb/bbsrc/bb1.9e-btf/etc/bb-hosts
su bb
cd
bbsrc/bb/runbb.sh restart ; exit

add to load graph config:
vi /usr/local/www/mgmt/mrtg/mrtg1.cfg
(add new section for new machine)

34. add to management db (on mail and devweb) jc.ref_machines and jc.ref_templates

insert into ref_machines values (null,'virt19','virt19.johncompanies.com',0,'l');
select * from ref_machines;
insert into systems values ('', 'col00001',43,null,null,100,'virt19.johncompanies.com',null,'2008-01-07',null,null,1,0,null,null,null);
select * from systems where hostname='virt19.johncompanies.com';
update ipmap set cid='col00001', sysid=3772, assign_date='2008-01-07', status=1 where ip='69.55.236.2';
update ipmap set cid='col00001', sysid=3772, assign_date='2008-01-07', status=1 where ip='69.55.230.109';
insert into ref_templates values ('','fedora-core-1/20050725',33,'Fedora Core 1',1);
insert into ref_templates values ('','fedora-core-2/20050725',33,'Fedora Core 2',1);
insert into ref_templates values ('','debian-3.1/20050927',33,'Debian 3.1',1);

35. Make sure that ns1c has proper forward and reverse entries for the new system (both private and public addresses), so that new system emailing will work.
vr johncompanies.com
vi internal.johncompanies.com
rndc reload johncompanies.com IN private
(edit the PTR too)

36. add an outside blocking rule to the firewall, so this machine can only be reached from inside the firewall.

Follow example already in firewall virt19 is:
ipfw add 00149 allow ip from { 69.55.225.225 or 69.55.230.2 or 69.55.230.10 or 69.55.230.3 or 62.213.72.101 or 64.131.90.27 or 80.89.128.178 or 82.200.65.190 or 84.237.120.254 or 89.251.97.2 or 195.214.232.10 or 195.214.233.10 or 208.251.212.1 or 213.130.28.26 } to 69.55.236.2
ipfw add 00149 deny ip from any to 69.55.236.2

virt12 would be 00142

37. add to server/cabinet map

39. make /vz1/private
mkdir /vz1/private ; chmod 0755 /vz1/private
mkdir /vz1/root ; chmod 0755 /vz1/root</pre>

E-Monitoring notes

2013-11-13T18:33:36Z

24.165.90.109: /* isys */

E-Monitoring notes

2013-11-13T18:30:12Z

24.165.90.109: /* polling / magrathea */

Cabinetmap

2013-11-07T21:00:28Z

24.165.90.109:

<table cellpadding=5>
<tr>
<td valign="top"><table border=1>

<tr><th>u</th> <th style="background:#FFFF99">1 </th><th>sw port</th> <th>ats port</th></tr>
<tr><td>42</td><td style="background:#a5e9f3">[https://secure.johncompanies.com/mgmt/mrtg/switch.cgi?s=switch-p25&path= c2950 P25]
</td><td class=p>1</td> <td class=p>9</td></tr>
<tr><td>41</td><td>&nbsp </td><td class=p> </td> <td class=p></td></tr>
<tr><td>40</td><td>  </td><td class=p> </td> <td class=p></td></tr>
<tr><td>39</td><td>  </td><td class=p> </td> <td class=p></td></tr>
<tr><td>38</td><td>  </td><td class=p> </td> <td class=p></td></tr>
<tr><td>37</td><td>  </td><td class=p> </td> <td class=p></td></tr>
<tr><td>36</td><td>col02113 </td><td class=p>19 </td> <td class=p>1</td></tr>
<tr><td>35</td><td rowspan=2>col01402 RJclients </td><td class=p rowspan=2>2</td> <td class=p rowspan=2></td></tr>
<tr><td>34</td></tr>
<tr><td>33</td><td rowspan=2>col1500 </td><td class=p rowspan=2>3</td> <td class=p rowspan=2>15</td></tr>
<tr><td>32</td></tr>
<tr><td>31</td><td rowspan=2>col00138 data4 </td><td class=p rowspan=2>4</td> <td class=p rowspan=2>14</td> </tr>
<tr><td>30</td></tr>
<tr><td>29</td><td rowspan=2>col01899 </td><td class=p rowspan=2>5,21 (RMM)</td> <td class=p rowspan=2>13</td></tr>
<tr><td>28</td></tr>
<tr><td>27</td><td rowspan=2>col01741 </td><td class=p rowspan=2>6</td> <td class=p rowspan=2>12</td></tr>
<tr><td>26</td></tr>
<tr><td>25</td><td rowspan=2>col01369 logos JC-08017 </td><td class=p rowspan=2>7</td> <td class=p rowspan=2>11</td></tr>
<tr><td>24</td></tr>
<tr><td>23</td><td>col01114 </td><td class=p>8</td> <td class=p >10</td></tr>
<tr><td>22</td><td> </td><td class=p>9</td> <td class=p >2-1</td></tr>
<tr><td>21</td><td colspan=3>  </td></tr>
<tr><td>20</td><td style="text-align:center; background:#cecece">[https://ats1.johncompanies.com ATS-1]
</td><td class=p>24</td> <td class=p></td></tr>
<tr><td>19</td><td colspan=3>  </td></tr>
<tr><td>18</td><td rowspan=2> </td><td class=p rowspan=2>10</td> <td class=p rowspan=2>7</td></tr>
<tr><td>17</td></tr>
<tr><td>16</td><td rowspan=2>col02210 JC-13009 </td><td class=p rowspan=2>10,11(RMM)</td> <td class=p rowspan=2>6</td></tr>
<tr><td>15</td></tr>
<tr><td>14</td><td rowspan=2>col02078 raven.nostrum.com </td><td class=p rowspan=2>17</td> <td class=p rowspan=2>5</td></tr>
<tr><td>13</td></tr>
<tr><td>12</td><td rowspan=2>col02184 JC-08028 </td><td class=p rowspan=2>12 </td> <td class=p rowspan=2>4</td></tr>
<tr><td>11</td></tr>
<tr><td>10</td><td rowspan=2> </td><td class=p rowspan=2></td> <td class=p rowspan=2>3</td></tr>
<tr><td>9 </td></tr>
<tr><td>8 </td><td rowspan=2>col00726 </td><td class=p rowspan=2>15</td> <td class=p rowspan=2>2</td></tr>
<tr><td>7 </td></tr>
<tr><td>6 </td><td rowspan=2>col01257 JC-13004 </td><td class=p rowspan=2>14 16 (RMM)</td> <td class=p rowspan=2>8</td></tr>
<tr><td>5 </td><td>  </td><td class=p></td> <td class=p></td></tr>

</table></td>

<td valign="top"><table border=1>

<tr><th>u</th> <th style="background:#FFFF99">2 </th><th>sw port</th> <th>ats port</th></tr>
<tr><td>42</td><td style="background:#a5e9f3">[https://secure.johncompanies.com/mgmt/mrtg/switch.cgi?s=switch-p24&path= c2950 P24] </td><td class=p>1</td> <td class=p>9</td></tr>
<tr><td>41</td><td>&nbsp </td><td class=p> </td> <td class=p></td></tr>
<tr><td>40</td><td>  </td><td class=p> </td> <td class=p></td></tr>
<tr><td>39</td><td>  </td><td class=p> </td> <td class=p></td></tr>
<tr><td>38</td><td>  </td><td class=p> </td> <td class=p></td></tr>
<tr><td>37</td><td>  </td><td class=p> </td> <td class=p></td></tr>
<tr><td>36</td><td>  </td><td class=p> </td> <td class=p></td></tr>
<tr><td>35</td><td rowspan=2>col01378 </td><td class=p rowspan=2>2,15</td> <td class=p rowspan=2>15</td></tr>
<tr><td>34</td></tr>
<tr><td>33</td><td rowspan=2>col01629 newserver </td><td class=p rowspan=2>3</td> <td class=p rowspan=2>14</td></tr>
<tr><td>32</td></tr>
<tr><td>31</td><td rowspan=2>col00138 www </td><td class=p rowspan=2>4</td> <td class=p rowspan=2>13</td></tr>
<tr><td>30</td></tr>
<tr><td>29</td><td rowspan=2>col01631 </td><td class=p rowspan=2>5</td> <td class=p rowspan=2>12</td></tr>
<tr><td>28</td></tr>
<tr><td>27</td><td rowspan=2>col01621 agony JC-08013 </td><td class=p rowspan=2>6</td> <td class=p rowspan=2>11</td></tr>
<tr><td>26</td></tr>
<tr><td>25</td><td rowspan=2>col01567 KPSurplus </td><td class=p rowspan=2>7</td> <td class=p rowspan=2>10</td></tr>
<tr><td>24</td></tr>
<tr><td>23</td><td rowspan=2> </td><td class=p rowspan=2>8</td> <td class=p rowspan=2>8</td></tr>
<tr><td>22</td></tr>
<tr><td>21</td><td colspan=3>  </td></tr>
<tr><td>20</td><td style="text-align:center; background:#cecece">[https://ats2.johncompanies.com ATS-2]
</td><td class=p>24</td> <td class=p></td></tr>
<tr><td>19</td><td colspan=3>  </td></tr>
<tr><td>18</td><td rowspan=2>col01996 JC-08003 </td><td class=p rowspan=2>9</td> <td class=p rowspan=2>7</td></tr>
<tr><td>17</td></tr>
<tr><td>16</td><td rowspan=2>col01711 JC-07030 * </td><td class=p rowspan=2>10</td> <td class=p rowspan=2>6</td></tr>
<tr><td>15</td></tr>
<tr><td>14</td><td rowspan=2>col01788 JC-06021 </td><td class=p rowspan=2>11</td> <td class=p rowspan=2>5</td></tr>
<tr><td>13</td></tr>
<tr><td>12</td><td rowspan=2>col00226 JC-07037 </td><td class=p rowspan=2>12</td> <td class=p rowspan=2>4</td></tr>
<tr><td>11</td></tr>
<tr><td>10</td><td rowspan=2>col00633 JC-08044 </td><td class=p rowspan=2>13</td> <td class=p rowspan=2>3</td></tr>
<tr><td>9</td></tr>
<tr><td>8</td><td rowspan=2>col00433 djc-jc JC-07015 </td><td class=p rowspan=2>14</td> <td class=p rowspan=2>2</td></tr>
<tr><td>7</td></tr>
<tr><td>4</td><td rowspan=2>col00433 Dell 2950 </td><td class=p rowspan=2>16 DRAC 17</td> <td class=p rowspan=2>2</td></tr>
<tr><td>5</td></tr>

</table></td>

<td valign="top"><table border=1>

<tr><th>u</th> <th style="background:#FFFF99">3 </th><th>sw port</th> <th>ats port</th></tr>
<tr><td>42</td><td style="background:#a5e9f3">[https://secure.johncompanies.com/mgmt/mrtg/switch.cgi?s=switch-p23&path= c2950 P23] </td><td class=p>1</td> <td class=p>9</td></tr>
<tr><td>41</td><td>&nbsp </td><td class=p> </td> <td class=p></td></tr>
<tr><td>40</td><td>  </td><td class=p> </td> <td class=p></td></tr>
<tr><td>39</td><td>  </td><td class=p> </td> <td class=p></td></tr>
<tr><td>38</td><td>  </td><td class=p> </td> <td class=p></td></tr>
<tr><td>37</td><td rowspan=2>col01727 JC-10019 </td><td class=p rowspan=2>19, 23(RMM)</td> <td class=p rowspan=2>1</td></tr>
<tr><td>36</td><td>
<tr><td>35</td><td rowspan=2>col01408 issoasis.com </td><td class=p rowspan=2>2 16 (RMM)</td> <td class=p rowspan=2>15</td></tr>
<tr><td>34</td></tr>
<tr><td>33</td><td rowspan=2>col00789 on001 </td><td class=p rowspan=2>3</td> <td class=p rowspan=2>14</td></tr>
<tr><td>32</td></tr>
<tr><td>31</td><td rowspan=2>col00789 laertes </td><td class=p rowspan=2>4</td> <td class=p rowspan=2>13</td></tr>
<tr><td>30</td></tr>
<tr><td>29</td><td rowspan=2>col02152 NO LABEL </td><td class=p rowspan=2>5</td> <td class=p rowspan=2>12</td></tr>
<tr><td>28</td></tr>
<tr><td>27</td><td rowspan=2> </td><td class=p rowspan=2>6</td> <td class=p rowspan=2>11</td></tr>
<tr><td>26</td></tr>
<tr><td>25</td><td rowspan=2>col01333 </td><td class=p rowspan=2>7</td> <td class=p rowspan=2>10</td></tr>
<tr><td>24</td></tr>
<tr><td>23</td><td rowspan=2>col01741 cyrus.base </td><td class=p rowspan=2>15, 22 (RMM)</td> <td class=p rowspan=2>8</td></tr>
<tr><td>22</td></tr>
<tr><td>21</td><td colspan=3>  </td></tr>
<tr><td>20</td><td style="text-align:center; background:#cecece">[https://ats3.johncompanies.com ATS-3]
</td><td class=p>24</td> <td class=p></td></tr>
<tr><td>19</td><td colspan=3>  </td></tr>
<tr><td>18</td><td rowspan=2>col00789 bb001 </td><td class=p rowspan=2>9, 20 (RMM)</td> <td class=p rowspan=2>7</td></tr>
<tr><td>17</td></tr>
<tr><td>16</td><td rowspan=2>col01224 pariotrial.sea-incorporated </td><td class=p rowspan=2>10, 21 (RMM)</td> <td class=p rowspan=2>6</td></tr>
<tr><td>15</td></tr>
<tr><td>14</td><td rowspan=2>col01777 </td><td class=p rowspan=2>11</td> <td class=p rowspan=2>5</td></tr>
<tr><td>13</td></tr>
<tr><td>12</td><td rowspan=2>col01224 fuji </td><td class=p rowspan=2>12</td> <td class=p rowspan=2>4</td></tr>
<tr><td>11</td></tr>
<tr><td>10</td><td rowspan=2>col01817 </td><td class=p rowspan=2>13</td> <td class=p rowspan=2>3</td></tr>
<tr><td>9</td></tr>
<tr><td>8</td><td rowspan=2>col01332 vs4 </td><td class=p rowspan=2>14</td> <td class=p rowspan=2>2</td></tr>
<tr><td>7</td></tr>
<tr><td>6</td><td rowspan=2>col02202 JC-13006 </td><td class=p rowspan=2>8 17 (RMM)</td> <td class=p rowspan=2>16</td></tr>
<tr><td>5</td><td>  </td><td class=p></td> <td class=p></td></tr>

</table></td>

<td valign="top"><table border=1>

<tr><th>u</th> <th style="background:#FFFF99">4 </th><th>sw port</th> <th>ats port</th></tr>
<tr><td>42</td><td style="background:#a5e9f3">[https://secure.johncompanies.com/mgmt/mrtg/switch.cgi?s=switch-p22&path= c2950 P22] </td><td class=p>1</td> <td class=p>9</td></tr>
<tr><td>41</td><td>&nbsp </td><td class=p> </td> <td class=p></td></tr>
<tr><td>40</td><td>  </td><td class=p> </td> <td class=p></td></tr>
<tr><td>39</td><td>  </td><td class=p> </td> <td class=p></td></tr>
<tr><td>38</td><td>  </td><td class=p> </td> <td class=p></td></tr>
<tr><td>37</td><td>  </td><td class=p> </td> <td class=p></td></tr>
<tr><td>36</td><td>  </td><td class=p> </td> <td class=p></td></tr>
<tr><td>35</td><td rowspan=2>col01224 cisco (no rails) </td><td class=p rowspan=2>2</td> <td class=p rowspan=2>16</td></tr>
<tr><td>34</td></tr>
<tr><td>33</td><td rowspan=2>col01224 obc-c </td><td class=p rowspan=2> </td> <td class=p rowspan=2>15</td></tr>
<tr><td>32</td></tr>
<tr><td>31</td><td rowspan=2>col01224 obc-b </td><td class=p rowspan=2> </td> <td class=p rowspan=2>14</td></tr>
<tr><td>30</td></tr>
<tr><td>29</td><td rowspan=2>col01224 obc-a </td><td class=p rowspan=2> </td> <td class=p rowspan=2>13</td></tr>
<tr><td>28</td></tr>
<tr><td>27</td><td rowspan=2>col01679 <b>Cancelled</b> </td><td class=p rowspan=2>3</td> <td class=p rowspan=2>12</td></tr>
<tr><td>26</td></tr>
<tr><td>25</td><td>col00534 </td><td class=p>4</td> <td class=p>11</td></tr>
<tr><td>24</td><td> </td><td class=p></td> <td class=p></td></tr>
<tr><td>23</td><td rowspan=2>col01224 hydro4gehosting </td><td class=p rowspan=2>6</td> <td class=p rowspan=2>8</td></tr>
<tr><td>22</td></tr>
<tr><td>21</td><td colspan=3>  </td></tr>
<tr><td>20</td><td style="text-align:center; background:#cecece">[https://ats4.johncompanies.com ATS-4]
</td><td class=p>24</td> <td class=p></td></tr>
<tr><td>19</td><td colspan=3>  </td></tr>
<tr><td>18</td><td rowspan=2>col01182 dev01 </td><td class=p rowspan=2>7</td> <td class=p rowspan=2>7</td></tr>
<tr><td>17</td></tr>
<tr><td>16</td><td rowspan=2>col01182 demo2 </td><td class=p rowspan=2>8</td> <td class=p rowspan=2>6</td></tr>
<tr><td>15</td></tr>
<tr><td>14</td><td rowspan=2>col00336 rb3 </td><td class=p rowspan=2>9</td> <td class=p rowspan=2>5</td></tr>
<tr><td>13</td></tr>
<tr><td>12</td><td rowspan=2>JC-07025 (Temp for col01741) </td><td class=p rowspan=2>10</td> <td class=p rowspan=2>4</td></tr>
<tr><td>11</td></tr>
<tr><td>10</td><td rowspan=2>col01710 </td><td class=p rowspan=2>11</td> <td class=p rowspan=2>3</td></tr>
<tr><td>9</td></tr>
<tr><td>8</td><td rowspan=2>col00336 coach </td><td class=p rowspan=2>12</td> <td class=p rowspan=2>2</td></tr>
<tr><td>7</td></tr>
<tr><td>6</td><td rowspan=2>col00534 </td><td class=p rowspan=2>18</td> <td class=p rowspan=2>10</td></tr>
<tr><td>5</td><td></tr>

</table></td>

<td valign="top"><table border=1>

<tr><th>u</th> <th style="background:#FFFF99">5 </th><th>sw port</th> <th>ats port</th></tr>
<tr><td>42</td><td style="background:#a5e9f3">[https://secure.johncompanies.com/mgmt/mrtg/switch.cgi?s=switch-p21&path= c2950 P21]
</td><td class=p>g1</td> <td class=p>9</td></tr>
<tr><td>41</td><td>&nbsp </td><td class=p> </td> <td class=p></td></tr>
<tr><td>40</td><td>  </td><td class=p> </td> <td class=p></td></tr>
<tr><td>39</td><td>  </td><td class=p> </td> <td class=p></td></tr>
<tr><td>38</td><td>  </td><td class=p> </td> <td class=p></td></tr>
<tr><td>37</td><td rowspan=2>col01849 snacktime </td><td class=p rowspan=2>16 17 (RMM)</td> <td class=p rowspan=2>2</td></tr>
<tr><td>36</td><td>
<tr><td>35</td><td rowspan=2>col01080 lithium </td><td class=p rowspan=2>1</td> <td class=p rowspan=2>16</td></tr>
<tr><td>34</td></tr>
<tr><td>33</td><td rowspan=2>col01224 ewing </td><td class=p rowspan=2>2</td> <td class=p rowspan=2>15</td></tr>
<tr><td>32</td></tr>
<tr><td>31</td><td rowspan=2>col01224 luzerne </td><td class=p rowspan=2>11</td> <td class=p rowspan=2>14</td></tr>
<tr><td>30</td></tr>
<tr><td>29</td><td rowspan=2>col01333 www2 </td><td class=p rowspan=2>3,20 (RMM)</td> <td class=p rowspan=2>13</td></tr>
<tr><td>28</td></tr>
<tr><td>27</td><td rowspan=2>col01556 </td><td class=p rowspan=2>5</td> <td class=p rowspan=2>12</td></tr>
<tr><td>26</td></tr>
<tr><td>25</td><td rowspan=2>col02138 webengineers </td><td class=p rowspan=2>7, 21 (RMM)</td> <td class=p rowspan=2>11</td></tr>
<tr><td>24</td></tr>
<tr><td>23</td><td rowspan=2>col01495 beany </td><td class=p rowspan=2>22</td> <td class=p rowspan=2>10</td></tr>
<tr><td>22</td></tr>
<tr><td>21</td><td colspan=3>  </td></tr>
<tr><td>20</td><td style="text-align:center; background:#cecece">[https://ats5.johncompanies.com ATS-5]
</td><td class=p>24</td> <td class=p></td></tr>
<tr><td>19</td><td colspan=3>  </td></tr>
<tr><td>18</td><td rowspan=2>col01841 paradigm </td><td class=p rowspan=2>12 13 (DRAC)</td> <td class=p rowspan=2>6</td></tr>
<tr><td>17</td></tr>
<tr><td>16</td><td rowspan=2>col01080 helium </td><td class=p rowspan=2>9</td> <td class=p rowspan=2>5</td></tr>
<tr><td>15</td></tr>
<tr><td>14</td><td rowspan=2>col01629 anitaborg mail </td><td class=p rowspan=2>10</td> <td class=p rowspan=2>7</td></tr>
<tr><td>13</td></tr>
<tr><td>12</td><td rowspan=2>col02205 </td><td class=p rowspan=2>5 3 (RMM)</td> <td class=p rowspan=2>3</td></tr>
<tr><td>11</td></tr>
<tr><td>10</td><td rowspan=2>col01841 smartypants * </td><td class=p rowspan=2>12,13 (DRAC)</td> <td class=p rowspan=2>8</td></tr>
<tr><td>9</td></tr>
<tr><td>8</td><td rowspan=2>col01408 pepclub </td><td class=p rowspan=2>6, 15 (RMM)</td> <td class=p rowspan=2>4</td></tr>
<tr><td>7</td></tr>

</table></td>
</tr>
</table>

<table cellpadding=5>
<tr>
<td valign="top"><table border=1>

<tr><th>u</th> <th style="background:#FFFF99">6 </th><th>ats port</th></tr>
<tr><td>42</td><td class="jc">digi3 </td><td class=p>10</td></tr>
<tr><td>41</td><td style="background:#a5e9f3">[https://secure.johncompanies.com/mgmt/mrtg/switch.cgi?s=switch-p20&path= c2950 P20]
</td><td class=p>9</td></tr>
<tr><td>40</td><td style="background:#a5e9f3">[https://secure.johncompanies.com/mgmt/mrtg/switch.cgi?s=switch-p12&path= c2950 P12 private]
</td><td class=p>2</td></tr>
<tr><td>39</td><td>  </td><td class=p> </td></tr>
<tr><td>38</td><td>  </td><td class=p> </td></tr>
<tr><td>37</td><td>isys </td><td class=p>11</td></tr>
<tr><td>36</td><td rowspan=2>firewall2 </td><td class=p rowspan=2>8 (A), 7 (B)</td></tr>
<tr><td>35</td></tr>
<tr><td>34</td><td rowspan=2>firewall3 </td><td class=p rowspan=2>? (A), ? (B)</td></tr>
<tr><td>33</td></tr>
<tr><td>32</td><td rowspan=2>nat2 </td><td class=p rowspan=2>6</td></tr>
<tr><td>31</td></tr>
<tr><td>30</td><td rowspan=2>bwdb2 </td><td class=p rowspan=2>5</td></tr>
<tr><td>29</td></tr>
<tr><td>28</td><td rowspan=2>backup3 </td><td class=p rowspan=2>4</td></tr>
<tr><td>27</td><td>  </td><td class=p></td></tr>
<tr><td>26</td><td>jail3 (col01737) </td><td class=p></td></tr>
<tr><td>25</td><td>  </td><td class=p></td></tr>
<tr><td>24</td><td>  </td><td class=p></td></tr>
<tr><td>23</td><td>  </td><td class=p></td></tr>
<tr><td>22</td><td>  </td><td class=p></td></tr>
<tr><td>21</td><td colspan=3>  </td></tr>
<tr><td>20</td><td style="text-align:center; background:#cecece">[https://ats6.johncompanies.com ATS-6]
</td><td class=p></td></tr>
<tr><td>19</td><td colspan=3>  </td></tr>
<tr><td>18</td><td rowspan=2> </td><td class=p>16</td></tr>
<tr><td>17</td></tr>
<tr><td>16</td><td rowspan=2>magrathea </td><td class=p rowspan=2>3</td></tr>
<tr><td>15</td></tr>
<tr><td>14</td><td> col02045 cisco </td><td class=p>p21 18 ats 13</td></tr>
<tr><td>13</td><td>  </td><td class=p></td></tr>
<tr><td>12</td><td rowspan=2> col02045 dell </td><td class=p>p21 15 drac ats 12</td></tr>
<tr><td>11</td><td>  </td><td class=p></td></tr>
<tr><td>10</td><td>  </td><td class=p></td></tr>
<tr><td>9</td><td>  </td><td class=p></td></tr>
<tr><td>8</td><td>  </td><td class=p></td></tr>
<tr><td>7</td><td>  </td><td class=p></td></tr>

</table></td>

<td valign="top"><table border=1>

<tr><th>u</th> <th style="background:#FFFF99">7 </th><th>sw port</th> <th>ats port</th></tr>
<tr><td>42</td><td style="background:#a5e9f3">[https://secure.johncompanies.com/mgmt/mrtg/switch.cgi?s=switch-p26&path= c2950 P26]
</td><td class=p>1</td> <td class=p>9</td></tr>
<tr><td>41</td><td>&nbsp </td><td class=p> </td> <td class=p></td></tr>
<tr><td>40</td><td>  </td><td class=p> </td> <td class=p></td></tr>
<tr><td>39</td><td>  </td><td class=p> </td> <td class=p></td></tr>
<tr><td>38</td><td>  </td><td class=p> </td> <td class=p></td></tr>
<tr><td>37</td><td>  </td><td class=p> </td> <td class=p></td></tr>
<tr><td>36</td><td>  </td><td class=p> </td> <td class=p></td></tr>
<tr><td>35</td><td rowspan=2> </td><td class=p rowspan=2>?</td> <td class=p rowspan=2>15</td></tr>
<tr><td>34</td></tr>
<tr><td>33</td><td rowspan=2> </td><td class=p rowspan=2>3</td> <td class=p rowspan=2>14</td></tr>
<tr><td>32</td></tr>
<tr><td>31</td><td rowspan=2>col01224 brandgage </td><td class=p rowspan=2>4</td> <td class=p rowspan=2>13</td></tr>
<tr><td>30</td></tr>
<tr><td>29</td><td rowspan=2> </td><td class=p rowspan=2>5</td> <td class=p rowspan=2>12</td></tr>
<tr><td>28</td></tr>
<tr><td>27</td><td rowspan=2> </td><td class=p rowspan=2>6</td> <td class=p rowspan=2>11</td></tr>
<tr><td>26</td></tr>
<tr><td>25</td><td rowspan=2> </td><td class=p rowspan=2>7</td> <td class=p rowspan=2>10</td></tr>
<tr><td>24</td></tr>
<tr><td>23</td><td rowspan=2> </td><td class=p rowspan=2>8</td> <td class=p rowspan=2>8</td></tr>
<tr><td>22</td></tr>
<tr><td>21</td><td colspan=3>  </td></tr>
<tr><td>20</td><td style="text-align:center; background:#cecece">[https://ats7.johncompanies.com ATS-7]
</td><td class=p>24</td> <td class=p></td></tr>
<tr><td>19</td><td colspan=3> </td></tr>
<tr><td>18</td><td rowspan=2>col00875 www * </td><td class=p rowspan=2>9</td> <td class=p rowspan=2>7</td></tr>
<tr><td>17</td></tr>
<tr><td>16</td><td rowspan=2>col00875 backup </td><td class=p rowspan=2></td> <td class=p rowspan=2>6</td></tr>
<tr><td>15</td></tr>
<tr><td>14</td><td rowspan=2> </td><td class=p rowspan=2>10</td> <td class=p rowspan=2>5</td></tr>
<tr><td>13</td></tr>
<tr><td>12</td><td rowspan=2>col02171 </td><td class=p rowspan=2>11</td> <td class=p rowspan=2>4</td></tr>
<tr><td>11</td></tr>
<tr><td>10</td><td rowspan=2>jc11013 label </td><td class=p rowspan=2>12</td> <td class=p rowspan=2>3</td></tr>
<tr><td>9</td></tr>
<tr><td>8</td><td rowspan=2>col02108 </td><td class=p rowspan=2>13</td> <td class=p rowspan=2>2</td></tr>
<tr><td>7</td></tr>
<tr><td>6</td><td rowspan=4> On Floor -- col00336 fumble.fantasybowl.com (Atom) and col02108 JC-11009 (Atom)</td><td class=p>12</td> <td class=p>1</td></tr>
<tr><td>5</td><td>  </td><td class=p></td> <td class=p></td></tr>
<tr><td>4</td><td>  </td><td class=p></td> <td class=p></td></tr>
<tr><td>3</td><td>  </td><td class=p></td> <td class=p></td></tr>

</table></td>

<td valign="top"><table border=1>

<tr><th>u</th> <th style="background:#FFFF99">8 </th><th>sw port</th> <th>ats port</th></tr>
<tr><td>42</td><td style="background:#a5e9f3">[https://secure.johncompanies.com/mgmt/mrtg/switch.cgi?s=switch-p27&path= c2950 P27]
</td><td class=p>1</td> <td class=p>9</td></tr>
<tr><td>41</td><td>&nbsp </td><td class=p> </td> <td class=p></td></tr>
<tr><td>40</td><td>  </td><td class=p> </td> <td class=p></td></tr>
<tr><td>39</td><td>  </td><td class=p> </td> <td class=p></td></tr>
<tr><td>38</td><td>  </td><td class=p> </td> <td class=p></td></tr>
<tr><td>37</td><td rowspan=2>col01719 (1U) </td><td class=p rowspan=2>16</td> <td class=p rowspan=2>16</td></tr>
<tr><td>36</td></tr>
<tr><td>35</td><td rowspan=2> </td><td class=p rowspan=2>2</td> <td class=p rowspan=2>15</td></tr>
<tr><td>34</td></tr>
<tr><td>33</td><td rowspan=2>col01224 idgroup </td><td class=p rowspan=2>3</td> <td class=p rowspan=2>14</td></tr>
<tr><td>32</td></tr>
<tr><td>31</td><td rowspan=2> </td><td class=p rowspan=2>4</td> <td class=p rowspan=2>13</td></tr>
<tr><td>30</td></tr>
<tr><td>29</td><td rowspan=2>col02150 </td><td class=p rowspan=2>5, 21 (RMM)</td> <td class=p rowspan=2>12</td></tr>
<tr><td>28</td></tr>
<tr><td>27</td><td rowspan=2>col01860 </td><td class=p rowspan=2>6</td> <td class=p rowspan=2>11</td></tr>
<tr><td>26</td></tr>
<tr><td>25</td><td rowspan=2>col01025 feros </td><td class=p rowspan=2>7, 18 (RMM)</td> <td class=p rowspan=2>10</td></tr>
<tr><td>24</td></tr>
<tr><td>23</td><td rowspan=2> </td><td class=p rowspan=2>8</td> <td class=p rowspan=2>8</td></tr>
<tr><td>22</td></tr>
<tr><td>21</td><td colspan=3>  </td></tr>
<tr><td>20</td><td style="text-align:center; background:#cecece">[https://ats8.johncompanies.com ATS-8]
</td><td class=p>24</td> <td class=p></td></tr>
<tr><td>19</td><td colspan=3> </td></tr>
<tr><td>18</td><td rowspan=2>col01941 </td><td class=p rowspan=2>9</td> <td class=p rowspan=2>7</td></tr>
<tr><td>17</td></tr>
<tr><td>16</td><td rowspan=2>col01511 * </td><td class=p rowspan=2>10</td> <td class=p rowspan=2>6</td></tr>
<tr><td>15</td></tr>
<tr><td>14</td><td rowspan=2>col00138 data2 </td><td class=p rowspan=2>15</td> <td class=p rowspan=2>5</td></tr>
<tr><td>13</td></tr>
<tr><td>12</td><td rowspan=2>col01567 prod </td><td class=p rowspan=2>12</td> <td class=p rowspan=2>4</td></tr>
<tr><td>11</td></tr>
<tr><td>10</td><td rowspan=3> </td><td class=p rowspan=3></td> <td class=p rowspan=3>3</td></tr>
<tr><td>9</td></tr>
<tr><td>8</td></tr>
<tr><td>7</td><td rowspan=2>col02001 attic1jc </td><td class=p rowspan=2>11</td> <td class=p rowspan=2>2</td></tr>
<tr><td>6</td></tr>

</table></td>

<td valign="top"><table border=1>

<tr><th>u</th> <th style="background:#FFFF99">9 </th><th>sw port</th> <th>ats port</th></tr>
<tr><td>42</td><td style="background:#a5e9f3">[https://secure.johncompanies.com/mgmt/mrtg/switch.cgi?s=switch-p28&path= c2950 P28]
</td><td class=p>1</td> <td class=p>9</td></tr>
<tr><td>41</td><td>&nbsp </td><td class=p> </td> <td class=p></td></tr>
<tr><td>40</td><td>  </td><td class=p> </td> <td class=p></td></tr>
<tr><td>39</td><td>  </td><td class=p> </td> <td class=p></td></tr>
<tr><td>38</td><td>  </td><td class=p> </td> <td class=p></td></tr>
<tr><td>37</td><td>  </td><td class=p> </td> <td class=p></td></tr>
<tr><td>36</td><td>  </td><td class=p> </td> <td class=p></td></tr>
<tr><td>35</td><td rowspan=2> </td><td class=p rowspan=2>2</td> <td class=p rowspan=2>15</td></tr>
<tr><td>34</td></tr>
<tr><td>33</td><td rowspan=2> </td><td class=p rowspan=2>3 <td> <td class=p rowspan=2>14</td></tr>
<tr><td>32</td></tr>
<tr><td>31</td><td rowspan=2> </td><td class=p rowspan=2>4</td> <td class=p rowspan=2>13</td></tr>
<tr><td>30</td></tr>
<tr><td>29</td><td rowspan=2> </td><td class=p rowspan=2>5</td> <td class=p rowspan=2>12</td></tr>
<tr><td>28</td></tr>
<tr><td>27</td><td rowspan=2> </td><td class=p rowspan=2>6</td> <td class=p rowspan=2>11</td></tr>
<tr><td>26</td></tr>
<tr><td>25</td><td rowspan=2> </td><td class=p rowspan=2>7</td> <td class=p rowspan=2>10</td></tr>
<tr><td>24</td></tr>
<tr><td>23</td><td rowspan=2> </td><td class=p rowspan=2>8</td> <td class=p rowspan=2>8</td></tr>
<tr><td>22</td></tr>
<tr><td>21</td><td colspan=3>  </td></tr>
<tr><td>20</td><td style="text-align:center; background:#cecece">[https://ats9.johncompanies.com ATS-9]
</td><td class=p>24</td> <td class=p></td></tr>
<tr><td>19</td><td colspan=3> </td></tr>
<tr><td>18</td><td rowspan=2> </td><td class=p rowspan=2>9</td> <td class=p rowspan=2>7</td></tr>
<tr><td>17</td></tr>
<tr><td>16</td><td rowspan=2> </td><td class=p rowspan=2>10</td> <td class=p rowspan=2>6</td></tr>
<tr><td>15</td></tr>
<tr><td>14</td><td rowspan=2> </td><td class=p rowspan=2>11</td> <td class=p rowspan=2>5</td></tr>
<tr><td>13</td></tr>
<tr><td>12</td><td rowspan=2> </td><td class=p rowspan=2>12</td> <td class=p rowspan=2>4</td></tr>
<tr><td>11</td></tr>
<tr><td>10</td><td rowspan=2> </td><td class=p rowspan=2>13</td> <td class=p rowspan=2>3</td></tr>
<tr><td>9</td></tr>
<tr><td>8</td><td rowspan=2> * </td><td class=p rowspan=2>14</td> <td class=p rowspan=2>2</td></tr>
<tr><td>7</td></tr>

</table></td>

</tr>
</table>
<br>
<table border=0 cellpadding=5>
<tr>
<td valign="top">
<table border=1>

<tr><th>U</th> <th style="background:#FFFF99">3-5 </th> <th>port</th></tr>
<tr><td class=u>129-127</td> <td style="background:#a5e9f3">[https://secure.johncompanies.com/mgmt/mrtg/switch.cgi?s=switch-p3&path= c2924 P3] </td> <td class=p></td></tr>
<tr><td class=u>126-124</td> <td class=digi>digi2 </td> <td class=p></td></tr>
<tr><td class=u>123<br>118</td> <td>2U </td> <td class=p></td></tr>
<tr><td class=u>117<br>112</td> <td>2U </td> <td class=p></td></tr>
<tr><td class=u>111<br>106</td> <td style="background:#8df595">jail19 * </td> <td class=p></td></tr>
<tr><td class=u>105<br>100</td> <td style="background:#f5ac81">virt15 * </td> <td class=p></td></tr>
<tr><td class=u>99<br>94</td> <td>col00779 wendlink 233.77 </td> <td class=p></td></tr>
<tr><td class=u>93<br>88</td> <td style="background:#8df595">jail2 * </td> <td class=p></td></tr>
<tr><td class=u>87<br>82</td> <td style="background:#f5ac81">virt13 * </td> <td class=p></td></tr>
<tr><td class=u>81<br>76</td> <td style="background:#8df595">old jail2 * </td> <td class=p></td></tr>
<tr><td class=u>75<br>70</td> <td style="background:#8df595">mx2 * </td> <td class=p></td></tr>
<tr><td class=u>69<br>64</td> <td>2U </td> <td class=p></td></tr>
<tr><td class=u>63<br>58</td> <td>col01372 cache2 </td> <td class=p></td></tr>
<tr><td class=u>57<br>52</td> <td>2U </td> <td class=p></td></tr>
<tr><td class=u>51<br>46</td> <td>col00499 </td> <td class=p></td></tr>
<tr><td class=u>45<br>40</td> <td>col02204 NIC P3-12 RMM P3-16 </td> <td class=p></td></tr>
<tr><td class=u>39<br>34</td> <td class=c>2U </td> <td class=p></td></tr>
<tr><td class=u>33<br>28</td> <td>col01727 - REMOVED 9/22/13 </td> <td class=p></td></tr>
</table>
</td>

<td valign="top">
<table border=1>

<tr><th>U</th> <th style="background:#FFFF99">3-6 </th> <th>port</th></tr>
<tr><td class=u>129-127</td> <td style="background:#a5e9f3">[https://secure.johncompanies.com/mgmt/mrtg/switch.cgi?s=switch-p4&path= c2924 P4] </td> <td class=p></td></tr>
<tr><td class=u>126-124</td> <td>1U </td> <td class=p></td></tr>
<tr><td class=u>123<br>118</td> <td style="background:#f5ac81">virt11 * </td> <td class=p>4</td></tr>
<tr><td class=u>117<br>112</td> <td style="background:#f5ac81">virt17 * </td> <td class=p>7</td></tr>
<tr><td class=u>111-109</td> <td>1U </td> <td class=p></td></tr>
<tr><td class=u>108-106</td> <td>2U </td> <td class=p></td></tr>
<tr><td class=u>105<br>100</td> <td style="background:#8df595">jail8 * </td> <td class=p>10</td></tr>
<tr><td class=u>99<br>94</td> <td style="background:#f5ac81">virt19 * </td> <td class=p>12</td></tr>
<tr><td class=u>93<br>88</td> <td style="background:#8df595">jail9 * </td> <td class=p></td></tr>
<tr><td class=u>87<br>82</td> <td>col01372 cache4 </td> <td class=p></td></tr>
<tr><td class=u>81<br>76</td> <td>col01372 dbmsg.bookoo </td> <td class=p></td></tr>
<tr><td class=u>75<br>70</td> <td>2U </td> <td class=p></td></tr>
<tr><td class=u>69<br>64</td> <td>2U </td> <td class=p></td></tr>
<tr><td class=u>63<br>58</td> <td>2U </td> <td class=p></td></tr>
<tr><td class=u>57<br>52</td> <td>2U </td> <td class=p></td></tr>
<tr><td class=u>51<br>46</td> <td>2U </td> <td class=p></td></tr>
<tr><td class=u>45<br>40</td> <td>col00985 </td> <td class=p></td></tr>
<tr><td class=u>39<br>34</td> <td>col01372 cache4 </td> <td class=p></td></tr>
<tr><td class=u>33<br>28</td> <td> </td> <td class=p></td></tr>

</table>
</td>

<td valign="top">
<table border=1>

<tr><th>U</th> <th style="background:#FFFF99">3-7 </th> <th>port</th></tr>
<tr><td class=u>129-127</td> <td style="background:#a5e9f3">[https://secure.johncompanies.com/mgmt/mrtg/switch.cgi?s=switch-p17&path= c2924 P17] </td> <td class=p></td></tr>
<tr><td class=u>126-124</td> <td class=c>1U </td> <td class=p></td></tr>
<tr><td class=u>123-121</td> <td class=digi>digi1 (no rails)</td> <td class=p></td></tr>
<tr><td class=u>120-118</td> <td style="background:#a5e9f3">switch-m2 </td> <td class=p></td></tr>
<tr><td class=u>117-115</td> <td style="text-align:center; background:#cecece">ATS </td> <td class=p></td></tr>
<tr><td class=u>114-112</td> <td style="text-align:center; background:#cecece">ATS </td> <td class=p></td></tr>
<tr><td class=u>111<br>106</td> <td style="background:#8df595">jail11 * </td> <td class=p></td></tr>
<tr><td class=u>105<br>100</td> <td class=c>col01372 solr7.bookoo </td> <td class=p></td></tr>
<tr><td class=u>99<br>94</td> <td>col01372 solr8.bookoo </td> <td class=p></td></tr>
<tr><td class=u>93<br>88</td> <td style="background:#f5ac81">virt12 * </td> <td class=p></td></tr>
<tr><td class=u>87<br>82</td> <td style="background:#8df595">jail7 * </td> <td class=p></td></tr>
<tr><td class=u>81<div class=s3></div>73</td> <td style="background:#f5ac81">virt14 </td> <td class=p></td></tr>
<tr><td class=u>72-70</td> <td style="background:#cecece"> bwdb1 (no rails) </td> <td class=p></td></tr>
<tr><td class=u>69<br>64</td> <td style="background:#f5ac81">virt16 * </td> <td class=p></td></tr>
<tr><td class=u>63-61</td> <td>col00991 </td> <td class=p></td></tr>
<tr><td class=u>60<div class=s4></div>49</td> <td style="background:#cecece">backup2 *</td> <td class=p></td></tr>
<tr><td class=u>48<br>43</td> <td style="background:#8df595">jail18 * </td> <td class=p></td></tr>
<tr><td class=u>42<br>37</td> <td style="background:#8df595">mx1 * </td> <td class=p></td></tr>
<tr><td class=u>36-34</td> <td class=c>1U </td> <td class=p></td></tr>
<tr><td class=u>33-31</td> <td>col00994 </td> <td class=p></td></tr>
<tr><td class=u>30<br>25</td> <td style="background:#f5ac81">virt9 * </td> <td class=p></td></tr>
<tr><td class=u>24<br>19</td> <td>2U </td> <td class=p></td></tr>
<tr><td class=u>18-16</td> <td>col0460 </td> <td class=p></td></tr>
<tr><td class=u>15-13</td> <td class=c>1U </td> <td class=p></td></tr>
<tr><td class=u>12-10</td> <td style="background:#cecece">nat </td> <td class=p></td></tr>

</table>
</td>

<td valign="top">
<table border=1>

<tr><th>U</th> <th style="background:#FFFF99">3-8 </th> <th>port</th></tr>
<tr><td class=u>129-127</td> <td class=c>1U </td> <td class=p></td></tr>
<tr><td class=u>126-124</td> <td style="background:#a5e9f3">c3750 </td> <td class=p></td></tr>
<tr><td class=u>122<br>118</td> <td style="background:#a5e9f3">c3550 </td> <td class=p></td></tr>
<tr><td class=u>117-115</td> <td style="background:#a5e9f3">[https://secure.johncompanies.com/mgmt/mrtg/switch.cgi?s=switch-p1&path= c2924 switch-p1] </td> <td class=p></td></tr>
<tr><td class=u>114-112</td> <td style="background:#a5e9f3">[https://secure.johncompanies.com/mgmt/mrtg/switch.cgi?s=switch-p1a&path= c2960 switch-p1a] </td> <td class=p></td></tr>
<tr><td class=u>111-109</td> <td style="background:#a5e9f3">[https://secure.johncompanies.com/mgmt/mrtg/switch.cgi?s=switch-p1b&path= c2960 switch-p1b] </td> <td class=p></td></tr>
<tr><td class=u>108<br>103</td> <td style="background:#cecece">firewall * </td> <td class=p></td></tr>
<tr><td class=u>102-100</td> <td style="background:#a5e9f3">ASA5500 </td> <td class=p></td></tr>
<tr><td class=u>99-97</td> <td style="background:#a5e9f3">M1 </td> <td class=p></td></tr>
<tr><td class=u>96<br>91</td> <td class=c>col02179 staging.issisystems.com </td> <td class=p></td></tr>
<tr><td class=u>90-88</td> <td style="background:#cecece">gate (old f/w) </td> <td class=p></td></tr>
<tr><td class=u>87<br>82</td> <td style="background:#f5ac81">quar1 * (no rails)</td> <td class=p></td></tr>
<tr><td class=u>81-79</td> <td>[shelf] </td> <td class=p></td></tr>
<tr><td class=u>78-76</td> <td class=c>1U </td> <td class=p></td></tr>
<tr><td class=u>75-73</td> <td style="text-align:center; background:#cecece">ATS </td> <td class=p></td></tr>
<tr><td class=u>72-70</td> <td style="text-align:center; background:#cecece">ATS </td> <td class=p></td></tr>
<tr><td class=u>69<br>64</td> <td style="background:#8df595">jail4</td> <td class=p></td></tr>
<tr><td class=u>63<br>58</td> <td style="background:#8df595">jail1 * (no rails)</td> <td class=p></td></tr>
<tr><td class=u>57-55</td> <td class=c>1U </td> <td class=p></td></tr>
<tr><td class=u>54<br>52</td> <td style="background:#cecece">console (no rails) </td> <td class=p></td></tr>
<tr><td class=u>51<br>46</td> <td style="background:#cecece">mail * (no rails) </td> <td class=p></td></tr>
<tr><td class=u>45<br>40</td> <td class=c>col01372 dbemail.bookoo </td> <td class=p></td></tr>
<tr><td class=u>39<br>34</td> <td class=>col01943 * </td> <td class=p></td></tr>
<tr><td class=u>33-31</td> <td>[shelf] </td> <td class=p></td></tr>
<tr><td class=u>30<br>25</td> <td class=c>2U </td> <td class=p></td></tr>
<tr><td class=u>24<br>19</td> <td style="background:#cecece">backup1 </td> <td class=p></td></tr>
<tr><td class=u>18<br>13</td> <td style="background:#8df595">jail17 * </td> <td class=p></td></tr>

</table>
</td>

</tr>
</table>
<br>
<table border=0 cellpadding=3>
<tr>

<td valign="top">
<table border=1>

<tr><th>U</th> <th style="background:#FFFF99">6-16 </th> <th>port</th></tr>
<tr><td class=u>129-127</td> <td style="background:#a5e9f3">[https://secure.johncompanies.com/mgmt/mrtg/switch.cgi?s=switch-p13&path= c2924 P13] </td> <td class=p></td></tr>
<tr><td class=u>126<br>121</td> <td class=c>2U </td> <td class=p></td></tr>
<tr><td class=u>120-118</td> <td>1372 gig sw </td> <td class=p></td></tr>
<tr><td class=u>117<br>112</td> <td class=c>2U </td> <td class=p></td></tr>
<tr><td class=u>111<br>106</td> <td>col01372 solr5.bookoo </td> <td class=p></td></tr>
<tr><td class=u>105<br>100</td> <td>col01372 monitor.bookoo </td> <td class=p></td></tr>
<tr><td class=u>99<br>94</td> <td>col01372 server3.kingwoodyardsales</td> <td class=p></td></tr>
<tr><td class=u>93<br>88</td> <td>col01372 web3.bookoo </td> <td class=p></td></tr>
<tr><td class=u>87<br>82</td> <td>col01372 web1.bookoounderground.com</td> <td class=p></td></tr>
<tr><td class=u>81<br>76</td> <td class=c>2U </td> <td class=p></td></tr>
<tr><td class=u>75-73</td> <td style="text-align:center; background:#cecece">PS </td> <td class=p></td></tr>
<tr><td class=u>72-70</td> <td style="text-align:center; background:#cecece">PS </td> <td class=p></td></tr>
<tr><td class=u>69<br>64</td> <td>col01372 dbstats.bookoo </td> <td class=p></td></tr>
<tr><td class=u>63<br>58</td> <td>col01372 db1.bookoounderground *</td> <td class=p></td></tr>
<tr><td class=u>57<br>52</td> <td>col01372 backup.clker </td> <td class=p></td></tr>
<tr><td class=u>51<br>46</td> <td>col01372 dbbackup.bookoo </td> <td class=p></td></tr>
<tr><td class=u>45<br>40</td> <td>col01372 dbpv.bookoo </td> <td class=p></td></tr>
<tr><td class=u>39<br>34</td> <td>col01372 web7.bookoo </td> <td class=p></td></tr>
<tr><td class=u>33<br>28</td> <td>col01372 dbm2.bookoo * </td> <td class=p></td></tr>
<tr><td class=u>27<br>22</td> <td>col01372 dbsense.bookoo </td> <td class=p></td></tr>

</table>
</td>

<td valign="top">
<table border=1>

<tr><th>U</th> <th style="background:#FFFF99">6-17 </th> <th>port</th></tr>
<tr><td class=u>129-127</td> <td>1U</td></tr>
<tr><td class=u>126-124</td> <td>col01372 gig sw </td> <td class=p></td></tr>
<tr><td class=u>123<br>118</td> <td class=c>2U </td> <td class=p></td></tr>
<tr><td class=u>117<br>112</td> <td>col01372 db1.clker </td> <td class=p>14</td></tr>
<tr><td class=u>111<br>106</td> <td>col01372 solr6.bookoo </td> <td class=p></td></tr>
<tr><td class=u>105<br>100</td> <td>col01372 solr4.bookoo </td> <td class=p>18</td></tr>
<tr><td class=u>99<br>94</td> <td>col01372 web5.bookoo </td> <td class=p>19</td></tr>
<tr><td class=u>93<br>88</td> <td>col01372 sso.bookoo </td> <td class=p>4</td></tr>
<tr><td class=u>87-85</td> <td>1U </td> <td class=p>9</td></tr>
<tr><td class=u>84-82</td> <td>col01372 balancer </td> <td class=p>3,4</td></tr>
<tr><td class=u>81-79</td> <td>1U </td> <td class=p></td></tr>
<tr><td class=u>78-76</td> <td style="text-align:center; background:#cecece">PS </td> <td class=p></td></tr>
<tr><td class=u>75-73</td> <td style="text-align:center; background:#cecece">PS </td> <td class=p></td></tr>
<tr><td class=u>72-70</td> <td style="text-align:center; background:#cecece">PS </td> <td class=p></td></tr>
<tr><td class=u>69<br>64</td> <td>col01372 solr2.bookoo </td> <td class=p></td></tr>
<tr><td class=u>63<br>58</td> <td>col01372 web6.bookoo </td> <td class=p></td></tr>
<tr><td class=u>57<br>52</td> <td>col01372 web8.bookoo </td> <td class=p></td></tr>
<tr><td class=u>51<br>46</td> <td>col01372 web2.bookoo </td> <td class=p></td></tr>
<tr><td class=u>45<br>40</td> <td>col01372 web1.bookoo </td> <td class=p></td></tr>
<tr><td class=u>39<br>34</td> <td>col01372 web4.bookoo </td> <td class=p></td></tr>
<tr><td class=u>33<br>28</td> <td>2U </td> <td class=p></td></tr>
<tr><td class=u>27<br>22</td> <td>col01372 dbm1.bookoo * </td> <td class=p></td></tr>
<tr><td class=u>21-19</td> <td>1U </td> <td class=p></td></tr>
<tr><td class=u>18<br>13</td> <td>col01372 web1.cliker * </td> <td class=p></td></tr>

</table>
</td>

</tr>
</table>

Cabinetmap

2013-11-07T20:55:15Z

24.165.90.109:

<table cellpadding=5>
<tr>
<td valign="top"><table border=1>

<tr><th>u</th> <th style="background:#FFFF99">1 </th><th>sw port</th> <th>ats port</th></tr>
<tr><td>42</td><td style="background:#a5e9f3">[https://secure.johncompanies.com/mgmt/mrtg/switch.cgi?s=switch-p25&path= c2950 P25]
</td><td class=p>1</td> <td class=p>9</td></tr>
<tr><td>41</td><td>&nbsp </td><td class=p> </td> <td class=p></td></tr>
<tr><td>40</td><td>  </td><td class=p> </td> <td class=p></td></tr>
<tr><td>39</td><td>  </td><td class=p> </td> <td class=p></td></tr>
<tr><td>38</td><td>  </td><td class=p> </td> <td class=p></td></tr>
<tr><td>37</td><td>  </td><td class=p> </td> <td class=p></td></tr>
<tr><td>36</td><td>col02113 </td><td class=p>19 </td> <td class=p>1</td></tr>
<tr><td>35</td><td rowspan=2>col01402 RJclients </td><td class=p rowspan=2>2</td> <td class=p rowspan=2></td></tr>
<tr><td>34</td></tr>
<tr><td>33</td><td rowspan=2>col1500 </td><td class=p rowspan=2>3</td> <td class=p rowspan=2>15</td></tr>
<tr><td>32</td></tr>
<tr><td>31</td><td rowspan=2>col00138 data4 </td><td class=p rowspan=2>4</td> <td class=p rowspan=2>14</td> </tr>
<tr><td>30</td></tr>
<tr><td>29</td><td rowspan=2>col01899 </td><td class=p rowspan=2>5,21 (RMM)</td> <td class=p rowspan=2>13</td></tr>
<tr><td>28</td></tr>
<tr><td>27</td><td rowspan=2>col01741 </td><td class=p rowspan=2>6</td> <td class=p rowspan=2>12</td></tr>
<tr><td>26</td></tr>
<tr><td>25</td><td rowspan=2>col01369 logos JC-08017 </td><td class=p rowspan=2>7</td> <td class=p rowspan=2>11</td></tr>
<tr><td>24</td></tr>
<tr><td>23</td><td>col01114 </td><td class=p>8</td> <td class=p >10</td></tr>
<tr><td>22</td><td> </td><td class=p>9</td> <td class=p >2-1</td></tr>
<tr><td>21</td><td colspan=3>  </td></tr>
<tr><td>20</td><td style="text-align:center; background:#cecece">[https://ats1.johncompanies.com ATS-1]
</td><td class=p>24</td> <td class=p></td></tr>
<tr><td>19</td><td colspan=3>  </td></tr>
<tr><td>18</td><td rowspan=2> </td><td class=p rowspan=2>10</td> <td class=p rowspan=2>7</td></tr>
<tr><td>17</td></tr>
<tr><td>16</td><td rowspan=2>col02210 JC-13009 </td><td class=p rowspan=2>10,11(RMM)</td> <td class=p rowspan=2>6</td></tr>
<tr><td>15</td></tr>
<tr><td>14</td><td rowspan=2>col02078 raven.nostrum.com </td><td class=p rowspan=2>17</td> <td class=p rowspan=2>5</td></tr>
<tr><td>13</td></tr>
<tr><td>12</td><td rowspan=2>col02184 JC-08028 </td><td class=p rowspan=2>12 </td> <td class=p rowspan=2>4</td></tr>
<tr><td>11</td></tr>
<tr><td>10</td><td rowspan=2> </td><td class=p rowspan=2></td> <td class=p rowspan=2>3</td></tr>
<tr><td>9 </td></tr>
<tr><td>8 </td><td rowspan=2>col00726 </td><td class=p rowspan=2>15</td> <td class=p rowspan=2>2</td></tr>
<tr><td>7 </td></tr>
<tr><td>6 </td><td rowspan=2>col01257 JC-13004 </td><td class=p rowspan=2>14 16 (RMM)</td> <td class=p rowspan=2>8</td></tr>
<tr><td>5 </td><td>  </td><td class=p></td> <td class=p></td></tr>

</table></td>

<td valign="top"><table border=1>

<tr><th>u</th> <th style="background:#FFFF99">2 </th><th>sw port</th> <th>ats port</th></tr>
<tr><td>42</td><td style="background:#a5e9f3">[https://secure.johncompanies.com/mgmt/mrtg/switch.cgi?s=switch-p24&path= c2950 P24] </td><td class=p>1</td> <td class=p>9</td></tr>
<tr><td>41</td><td>&nbsp </td><td class=p> </td> <td class=p></td></tr>
<tr><td>40</td><td>  </td><td class=p> </td> <td class=p></td></tr>
<tr><td>39</td><td>  </td><td class=p> </td> <td class=p></td></tr>
<tr><td>38</td><td>  </td><td class=p> </td> <td class=p></td></tr>
<tr><td>37</td><td>  </td><td class=p> </td> <td class=p></td></tr>
<tr><td>36</td><td>  </td><td class=p> </td> <td class=p></td></tr>
<tr><td>35</td><td rowspan=2>col01378 </td><td class=p rowspan=2>2,15</td> <td class=p rowspan=2>15</td></tr>
<tr><td>34</td></tr>
<tr><td>33</td><td rowspan=2>col01629 newserver </td><td class=p rowspan=2>3</td> <td class=p rowspan=2>14</td></tr>
<tr><td>32</td></tr>
<tr><td>31</td><td rowspan=2>col00138 www </td><td class=p rowspan=2>4</td> <td class=p rowspan=2>13</td></tr>
<tr><td>30</td></tr>
<tr><td>29</td><td rowspan=2>col01631 </td><td class=p rowspan=2>5</td> <td class=p rowspan=2>12</td></tr>
<tr><td>28</td></tr>
<tr><td>27</td><td rowspan=2>col01621 agony JC-08013 </td><td class=p rowspan=2>6</td> <td class=p rowspan=2>11</td></tr>
<tr><td>26</td></tr>
<tr><td>25</td><td rowspan=2>col01567 KPSurplus </td><td class=p rowspan=2>7</td> <td class=p rowspan=2>10</td></tr>
<tr><td>24</td></tr>
<tr><td>23</td><td rowspan=2> </td><td class=p rowspan=2>8</td> <td class=p rowspan=2>8</td></tr>
<tr><td>22</td></tr>
<tr><td>21</td><td colspan=3>  </td></tr>
<tr><td>20</td><td style="text-align:center; background:#cecece">[https://ats2.johncompanies.com ATS-2]
</td><td class=p>24</td> <td class=p></td></tr>
<tr><td>19</td><td colspan=3>  </td></tr>
<tr><td>18</td><td rowspan=2>col01996 JC-08003 </td><td class=p rowspan=2>9</td> <td class=p rowspan=2>7</td></tr>
<tr><td>17</td></tr>
<tr><td>16</td><td rowspan=2>col01711 JC-07030 * </td><td class=p rowspan=2>10</td> <td class=p rowspan=2>6</td></tr>
<tr><td>15</td></tr>
<tr><td>14</td><td rowspan=2>col01788 JC-06021 </td><td class=p rowspan=2>11</td> <td class=p rowspan=2>5</td></tr>
<tr><td>13</td></tr>
<tr><td>12</td><td rowspan=2>col00226 JC-07037 </td><td class=p rowspan=2>12</td> <td class=p rowspan=2>4</td></tr>
<tr><td>11</td></tr>
<tr><td>10</td><td rowspan=2>col00633 JC-08044 </td><td class=p rowspan=2>13</td> <td class=p rowspan=2>3</td></tr>
<tr><td>9</td></tr>
<tr><td>8</td><td rowspan=2>col00433 djc-jc JC-07015 </td><td class=p rowspan=2>14</td> <td class=p rowspan=2>2</td></tr>
<tr><td>7</td></tr>
<tr><td>4</td><td rowspan=2>col00433 Dell 2950 </td><td class=p rowspan=2>16 DRAC 17</td> <td class=p rowspan=2>2</td></tr>
<tr><td>5</td></tr>

</table></td>

<td valign="top"><table border=1>

<tr><th>u</th> <th style="background:#FFFF99">3 </th><th>sw port</th> <th>ats port</th></tr>
<tr><td>42</td><td style="background:#a5e9f3">[https://secure.johncompanies.com/mgmt/mrtg/switch.cgi?s=switch-p23&path= c2950 P23] </td><td class=p>1</td> <td class=p>9</td></tr>
<tr><td>41</td><td>&nbsp </td><td class=p> </td> <td class=p></td></tr>
<tr><td>40</td><td>  </td><td class=p> </td> <td class=p></td></tr>
<tr><td>39</td><td>  </td><td class=p> </td> <td class=p></td></tr>
<tr><td>38</td><td>  </td><td class=p> </td> <td class=p></td></tr>
<tr><td>37</td><td rowspan=2>col01727 JC-10019 </td><td class=p rowspan=2>19, 23(RMM)</td> <td class=p rowspan=2>1</td></tr>
<tr><td>36</td><td>
<tr><td>35</td><td rowspan=2>col01408 issoasis.com </td><td class=p rowspan=2>2 16 (RMM)</td> <td class=p rowspan=2>15</td></tr>
<tr><td>34</td></tr>
<tr><td>33</td><td rowspan=2>col00789 on001 </td><td class=p rowspan=2>3</td> <td class=p rowspan=2>14</td></tr>
<tr><td>32</td></tr>
<tr><td>31</td><td rowspan=2>col00789 laertes </td><td class=p rowspan=2>4</td> <td class=p rowspan=2>13</td></tr>
<tr><td>30</td></tr>
<tr><td>29</td><td rowspan=2>col02152 NO LABEL </td><td class=p rowspan=2>5</td> <td class=p rowspan=2>12</td></tr>
<tr><td>28</td></tr>
<tr><td>27</td><td rowspan=2> </td><td class=p rowspan=2>6</td> <td class=p rowspan=2>11</td></tr>
<tr><td>26</td></tr>
<tr><td>25</td><td rowspan=2>col01333 </td><td class=p rowspan=2>7</td> <td class=p rowspan=2>10</td></tr>
<tr><td>24</td></tr>
<tr><td>23</td><td rowspan=2>col01741 cyrus.base </td><td class=p rowspan=2>15, 22 (RMM)</td> <td class=p rowspan=2>8</td></tr>
<tr><td>22</td></tr>
<tr><td>21</td><td colspan=3>  </td></tr>
<tr><td>20</td><td style="text-align:center; background:#cecece">[https://ats3.johncompanies.com ATS-3]
</td><td class=p>24</td> <td class=p></td></tr>
<tr><td>19</td><td colspan=3>  </td></tr>
<tr><td>18</td><td rowspan=2>col00789 bb001 </td><td class=p rowspan=2>9, 20 (RMM)</td> <td class=p rowspan=2>7</td></tr>
<tr><td>17</td></tr>
<tr><td>16</td><td rowspan=2>col01224 pariotrial.sea-incorporated </td><td class=p rowspan=2>10, 21 (RMM)</td> <td class=p rowspan=2>6</td></tr>
<tr><td>15</td></tr>
<tr><td>14</td><td rowspan=2>col01777 </td><td class=p rowspan=2>11</td> <td class=p rowspan=2>5</td></tr>
<tr><td>13</td></tr>
<tr><td>12</td><td rowspan=2>col01224 fuji </td><td class=p rowspan=2>12</td> <td class=p rowspan=2>4</td></tr>
<tr><td>11</td></tr>
<tr><td>10</td><td rowspan=2>col01817 </td><td class=p rowspan=2>13</td> <td class=p rowspan=2>3</td></tr>
<tr><td>9</td></tr>
<tr><td>8</td><td rowspan=2>col01332 vs4 </td><td class=p rowspan=2>14</td> <td class=p rowspan=2>2</td></tr>
<tr><td>7</td></tr>
<tr><td>6</td><td rowspan=2>col02202 JC-13006 </td><td class=p rowspan=2>8 17 (RMM)</td> <td class=p rowspan=2>16</td></tr>
<tr><td>5</td><td>  </td><td class=p></td> <td class=p></td></tr>

</table></td>

<td valign="top"><table border=1>

<tr><th>u</th> <th style="background:#FFFF99">4 </th><th>sw port</th> <th>ats port</th></tr>
<tr><td>42</td><td style="background:#a5e9f3">[https://secure.johncompanies.com/mgmt/mrtg/switch.cgi?s=switch-p22&path= c2950 P22] </td><td class=p>1</td> <td class=p>9</td></tr>
<tr><td>41</td><td>&nbsp </td><td class=p> </td> <td class=p></td></tr>
<tr><td>40</td><td>  </td><td class=p> </td> <td class=p></td></tr>
<tr><td>39</td><td>  </td><td class=p> </td> <td class=p></td></tr>
<tr><td>38</td><td>  </td><td class=p> </td> <td class=p></td></tr>
<tr><td>37</td><td>  </td><td class=p> </td> <td class=p></td></tr>
<tr><td>36</td><td>  </td><td class=p> </td> <td class=p></td></tr>
<tr><td>35</td><td rowspan=2>col01224 cisco (no rails) </td><td class=p rowspan=2>2</td> <td class=p rowspan=2>16</td></tr>
<tr><td>34</td></tr>
<tr><td>33</td><td rowspan=2>col01224 obc-c </td><td class=p rowspan=2> </td> <td class=p rowspan=2>15</td></tr>
<tr><td>32</td></tr>
<tr><td>31</td><td rowspan=2>col01224 obc-b </td><td class=p rowspan=2> </td> <td class=p rowspan=2>14</td></tr>
<tr><td>30</td></tr>
<tr><td>29</td><td rowspan=2>col01224 obc-a </td><td class=p rowspan=2> </td> <td class=p rowspan=2>13</td></tr>
<tr><td>28</td></tr>
<tr><td>27</td><td rowspan=2>col01679 <b>Cancelled</b> </td><td class=p rowspan=2>3</td> <td class=p rowspan=2>12</td></tr>
<tr><td>26</td></tr>
<tr><td>25</td><td>col00534 </td><td class=p>4</td> <td class=p>11</td></tr>
<tr><td>24</td><td> </td><td class=p></td> <td class=p></td></tr>
<tr><td>23</td><td rowspan=2>col01224 hydro4gehosting </td><td class=p rowspan=2>6</td> <td class=p rowspan=2>8</td></tr>
<tr><td>22</td></tr>
<tr><td>21</td><td colspan=3>  </td></tr>
<tr><td>20</td><td style="text-align:center; background:#cecece">[https://ats4.johncompanies.com ATS-4]
</td><td class=p>24</td> <td class=p></td></tr>
<tr><td>19</td><td colspan=3>  </td></tr>
<tr><td>18</td><td rowspan=2>col01182 dev01 </td><td class=p rowspan=2>7</td> <td class=p rowspan=2>7</td></tr>
<tr><td>17</td></tr>
<tr><td>16</td><td rowspan=2>col01182 demo2 </td><td class=p rowspan=2>8</td> <td class=p rowspan=2>6</td></tr>
<tr><td>15</td></tr>
<tr><td>14</td><td rowspan=2>col00336 rb3 </td><td class=p rowspan=2>9</td> <td class=p rowspan=2>5</td></tr>
<tr><td>13</td></tr>
<tr><td>12</td><td rowspan=2>JC-07025 (Temp for col01741) </td><td class=p rowspan=2>10</td> <td class=p rowspan=2>4</td></tr>
<tr><td>11</td></tr>
<tr><td>10</td><td rowspan=2>col01710 </td><td class=p rowspan=2>11</td> <td class=p rowspan=2>3</td></tr>
<tr><td>9</td></tr>
<tr><td>8</td><td rowspan=2>col00336 coach </td><td class=p rowspan=2>12</td> <td class=p rowspan=2>2</td></tr>
<tr><td>7</td></tr>
<tr><td>6</td><td rowspan=2>col00534 </td><td class=p rowspan=2>18</td> <td class=p rowspan=2>10</td></tr>
<tr><td>5</td><td></tr>

</table></td>

<td valign="top"><table border=1>

<tr><th>u</th> <th style="background:#FFFF99">5 </th><th>sw port</th> <th>ats port</th></tr>
<tr><td>42</td><td style="background:#a5e9f3">[https://secure.johncompanies.com/mgmt/mrtg/switch.cgi?s=switch-p21&path= c2950 P21]
</td><td class=p>g1</td> <td class=p>9</td></tr>
<tr><td>41</td><td>&nbsp </td><td class=p> </td> <td class=p></td></tr>
<tr><td>40</td><td>  </td><td class=p> </td> <td class=p></td></tr>
<tr><td>39</td><td>  </td><td class=p> </td> <td class=p></td></tr>
<tr><td>38</td><td>  </td><td class=p> </td> <td class=p></td></tr>
<tr><td>37</td><td rowspan=2>col01849 snacktime </td><td class=p rowspan=2>16 17 (RMM)</td> <td class=p rowspan=2>2</td></tr>
<tr><td>36</td><td>
<tr><td>35</td><td rowspan=2>col01080 lithium </td><td class=p rowspan=2>1</td> <td class=p rowspan=2>16</td></tr>
<tr><td>34</td></tr>
<tr><td>33</td><td rowspan=2>col01224 ewing </td><td class=p rowspan=2>2</td> <td class=p rowspan=2>15</td></tr>
<tr><td>32</td></tr>
<tr><td>31</td><td rowspan=2>col01224 luzerne </td><td class=p rowspan=2>11</td> <td class=p rowspan=2>14</td></tr>
<tr><td>30</td></tr>
<tr><td>29</td><td rowspan=2>col01333 www2 </td><td class=p rowspan=2>3,20 (RMM)</td> <td class=p rowspan=2>13</td></tr>
<tr><td>28</td></tr>
<tr><td>27</td><td rowspan=2>col01556 </td><td class=p rowspan=2>5</td> <td class=p rowspan=2>12</td></tr>
<tr><td>26</td></tr>
<tr><td>25</td><td rowspan=2>col02138 webengineers </td><td class=p rowspan=2>7, 21 (RMM)</td> <td class=p rowspan=2>11</td></tr>
<tr><td>24</td></tr>
<tr><td>23</td><td rowspan=2>col01495 beany </td><td class=p rowspan=2>22</td> <td class=p rowspan=2>10</td></tr>
<tr><td>22</td></tr>
<tr><td>21</td><td colspan=3>  </td></tr>
<tr><td>20</td><td style="text-align:center; background:#cecece">[https://ats5.johncompanies.com ATS-5]
</td><td class=p>24</td> <td class=p></td></tr>
<tr><td>19</td><td colspan=3>  </td></tr>
<tr><td>18</td><td rowspan=2>col01841 paradigm </td><td class=p rowspan=2>12 13 (DRAC)</td> <td class=p rowspan=2>6</td></tr>
<tr><td>17</td></tr>
<tr><td>16</td><td rowspan=2>col01080 helium </td><td class=p rowspan=2>9</td> <td class=p rowspan=2>5</td></tr>
<tr><td>15</td></tr>
<tr><td>14</td><td rowspan=2>col01629 anitaborg mail </td><td class=p rowspan=2>10</td> <td class=p rowspan=2>7</td></tr>
<tr><td>13</td></tr>
<tr><td>12</td><td rowspan=2>col02205 </td><td class=p rowspan=2>5 3 (RMM)</td> <td class=p rowspan=2>3</td></tr>
<tr><td>11</td></tr>
<tr><td>10</td><td rowspan=2>col01841 smartypants * </td><td class=p rowspan=2>12,13 (DRAC)</td> <td class=p rowspan=2>8</td></tr>
<tr><td>9</td></tr>
<tr><td>8</td><td rowspan=2>col01408 pepclub </td><td class=p rowspan=2>6, 15 (RMM)</td> <td class=p rowspan=2>4</td></tr>
<tr><td>7</td></tr>

</table></td>
</tr>
</table>

<table cellpadding=5>
<tr>
<td valign="top"><table border=1>

<tr><th>u</th> <th style="background:#FFFF99">6 </th><th>ats port</th></tr>
<tr><td>42</td><td class="jc">digi3 </td><td class=p>10</td></tr>
<tr><td>41</td><td style="background:#a5e9f3">[https://secure.johncompanies.com/mgmt/mrtg/switch.cgi?s=switch-p20&path= c2950 P20]
</td><td class=p>9</td></tr>
<tr><td>40</td><td style="background:#a5e9f3">[https://secure.johncompanies.com/mgmt/mrtg/switch.cgi?s=switch-p12&path= c2950 P12 private]
</td><td class=p>2</td></tr>
<tr><td>39</td><td>  </td><td class=p> </td></tr>
<tr><td>38</td><td>  </td><td class=p> </td></tr>
<tr><td>37</td><td>isys </td><td class=p>11</td></tr>
<tr><td>36</td><td rowspan=2>firewall2 </td><td class=p rowspan=2>8 (A), 7 (B)</td></tr>
<tr><td>35</td></tr>
<tr><td>34</td><td rowspan=2>firewall3 </td><td class=p rowspan=2>? (A), ? (B)</td></tr>
<tr><td>33</td></tr>
<tr><td>32</td><td rowspan=2>nat2 </td><td class=p rowspan=2>6</td></tr>
<tr><td>31</td></tr>
<tr><td>30</td><td rowspan=2>bwdb2 </td><td class=p rowspan=2>5</td></tr>
<tr><td>29</td></tr>
<tr><td>28</td><td rowspan=2>backup3 </td><td class=p rowspan=2>4</td></tr>
<tr><td>27</td><td>  </td><td class=p></td></tr>
<tr><td>26</td><td>jail3 (col01737) </td><td class=p></td></tr>
<tr><td>25</td><td>  </td><td class=p></td></tr>
<tr><td>24</td><td>  </td><td class=p></td></tr>
<tr><td>23</td><td>  </td><td class=p></td></tr>
<tr><td>22</td><td>  </td><td class=p></td></tr>
<tr><td>21</td><td colspan=3>  </td></tr>
<tr><td>20</td><td style="text-align:center; background:#cecece">[https://ats6.johncompanies.com ATS-6]
</td><td class=p></td></tr>
<tr><td>19</td><td colspan=3>  </td></tr>
<tr><td>18</td><td rowspan=2> </td><td class=p>16</td></tr>
<tr><td>17</td></tr>
<tr><td>16</td><td rowspan=2>magrathea </td><td class=p rowspan=2>3</td></tr>
<tr><td>15</td></tr>
<tr><td>14</td><td> col02045 cisco </td><td class=p>p21 18 ats 13</td></tr>
<tr><td>13</td><td>  </td><td class=p></td></tr>
<tr><td>12</td><td rowspan=2> col02045 dell </td><td class=p>p21 15 drac ats 12</td></tr>
<tr><td>11</td><td>  </td><td class=p></td></tr>
<tr><td>10</td><td>  </td><td class=p></td></tr>
<tr><td>9</td><td>  </td><td class=p></td></tr>
<tr><td>8</td><td>  </td><td class=p></td></tr>
<tr><td>7</td><td>  </td><td class=p></td></tr>

</table></td>

<td valign="top"><table border=1>

<tr><th>u</th> <th style="background:#FFFF99">7 </th><th>sw port</th> <th>ats port</th></tr>
<tr><td>42</td><td style="background:#a5e9f3">[https://secure.johncompanies.com/mgmt/mrtg/switch.cgi?s=switch-p26&path= c2950 P26]
</td><td class=p>1</td> <td class=p>9</td></tr>
<tr><td>41</td><td>&nbsp </td><td class=p> </td> <td class=p></td></tr>
<tr><td>40</td><td>  </td><td class=p> </td> <td class=p></td></tr>
<tr><td>39</td><td>  </td><td class=p> </td> <td class=p></td></tr>
<tr><td>38</td><td>  </td><td class=p> </td> <td class=p></td></tr>
<tr><td>37</td><td>  </td><td class=p> </td> <td class=p></td></tr>
<tr><td>36</td><td>  </td><td class=p> </td> <td class=p></td></tr>
<tr><td>35</td><td rowspan=2> </td><td class=p rowspan=2>?</td> <td class=p rowspan=2>15</td></tr>
<tr><td>34</td></tr>
<tr><td>33</td><td rowspan=2> </td><td class=p rowspan=2>3</td> <td class=p rowspan=2>14</td></tr>
<tr><td>32</td></tr>
<tr><td>31</td><td rowspan=2>col01224 brandgage </td><td class=p rowspan=2>4</td> <td class=p rowspan=2>13</td></tr>
<tr><td>30</td></tr>
<tr><td>29</td><td rowspan=2> </td><td class=p rowspan=2>5</td> <td class=p rowspan=2>12</td></tr>
<tr><td>28</td></tr>
<tr><td>27</td><td rowspan=2> </td><td class=p rowspan=2>6</td> <td class=p rowspan=2>11</td></tr>
<tr><td>26</td></tr>
<tr><td>25</td><td rowspan=2> </td><td class=p rowspan=2>7</td> <td class=p rowspan=2>10</td></tr>
<tr><td>24</td></tr>
<tr><td>23</td><td rowspan=2> </td><td class=p rowspan=2>8</td> <td class=p rowspan=2>8</td></tr>
<tr><td>22</td></tr>
<tr><td>21</td><td colspan=3>  </td></tr>
<tr><td>20</td><td style="text-align:center; background:#cecece">[https://ats7.johncompanies.com ATS-7]
</td><td class=p>24</td> <td class=p></td></tr>
<tr><td>19</td><td colspan=3> </td></tr>
<tr><td>18</td><td rowspan=2>col00875 www * </td><td class=p rowspan=2>9</td> <td class=p rowspan=2>7</td></tr>
<tr><td>17</td></tr>
<tr><td>16</td><td rowspan=2>col00875 backup </td><td class=p rowspan=2></td> <td class=p rowspan=2>6</td></tr>
<tr><td>15</td></tr>
<tr><td>14</td><td rowspan=2> </td><td class=p rowspan=2>10</td> <td class=p rowspan=2>5</td></tr>
<tr><td>13</td></tr>
<tr><td>12</td><td rowspan=2>col02171 </td><td class=p rowspan=2>11</td> <td class=p rowspan=2>4</td></tr>
<tr><td>11</td></tr>
<tr><td>10</td><td rowspan=2>jc11013 label </td><td class=p rowspan=2>12</td> <td class=p rowspan=2>3</td></tr>
<tr><td>9</td></tr>
<tr><td>8</td><td rowspan=2>col02108 </td><td class=p rowspan=2>13</td> <td class=p rowspan=2>2</td></tr>
<tr><td>7</td></tr>
<tr><td>6</td><td rowspan=4> On Floor -- col00336 fumble.fantasybowl.com (Atom) and col02108 JC-11009 (Atom)</td><td class=p>12</td> <td class=p>1</td></tr>
<tr><td>5</td><td>  </td><td class=p></td> <td class=p></td></tr>
<tr><td>4</td><td>  </td><td class=p></td> <td class=p></td></tr>
<tr><td>3</td><td>  </td><td class=p></td> <td class=p></td></tr>

</table></td>

<td valign="top"><table border=1>

<tr><th>u</th> <th style="background:#FFFF99">8 </th><th>sw port</th> <th>ats port</th></tr>
<tr><td>42</td><td style="background:#a5e9f3">[https://secure.johncompanies.com/mgmt/mrtg/switch.cgi?s=switch-p27&path= c2950 P27]
</td><td class=p>1</td> <td class=p>9</td></tr>
<tr><td>41</td><td>&nbsp </td><td class=p> </td> <td class=p></td></tr>
<tr><td>40</td><td>  </td><td class=p> </td> <td class=p></td></tr>
<tr><td>39</td><td>  </td><td class=p> </td> <td class=p></td></tr>
<tr><td>38</td><td>  </td><td class=p> </td> <td class=p></td></tr>
<tr><td>37</td><td rowspan=2>col01719 (1U) </td><td class=p rowspan=2>16</td> <td class=p rowspan=2>16</td></tr>
<tr><td>36</td></tr>
<tr><td>35</td><td rowspan=2> </td><td class=p rowspan=2>2</td> <td class=p rowspan=2>15</td></tr>
<tr><td>34</td></tr>
<tr><td>33</td><td rowspan=2>col01224 idgroup </td><td class=p rowspan=2>3</td> <td class=p rowspan=2>14</td></tr>
<tr><td>32</td></tr>
<tr><td>31</td><td rowspan=2> </td><td class=p rowspan=2>4</td> <td class=p rowspan=2>13</td></tr>
<tr><td>30</td></tr>
<tr><td>29</td><td rowspan=2>col02150 </td><td class=p rowspan=2>5, 21 (RMM)</td> <td class=p rowspan=2>12</td></tr>
<tr><td>28</td></tr>
<tr><td>27</td><td rowspan=2>col01860 </td><td class=p rowspan=2>6</td> <td class=p rowspan=2>11</td></tr>
<tr><td>26</td></tr>
<tr><td>25</td><td rowspan=2>col01025 feros </td><td class=p rowspan=2>7, 18 (RMM)</td> <td class=p rowspan=2>10</td></tr>
<tr><td>24</td></tr>
<tr><td>23</td><td rowspan=2> </td><td class=p rowspan=2>8</td> <td class=p rowspan=2>8</td></tr>
<tr><td>22</td></tr>
<tr><td>21</td><td colspan=3>  </td></tr>
<tr><td>20</td><td style="text-align:center; background:#cecece">[https://ats8.johncompanies.com ATS-8]
</td><td class=p>24</td> <td class=p></td></tr>
<tr><td>19</td><td colspan=3> </td></tr>
<tr><td>18</td><td rowspan=2>col01941 </td><td class=p rowspan=2>9</td> <td class=p rowspan=2>7</td></tr>
<tr><td>17</td></tr>
<tr><td>16</td><td rowspan=2>col01511 * </td><td class=p rowspan=2>10</td> <td class=p rowspan=2>6</td></tr>
<tr><td>15</td></tr>
<tr><td>14</td><td rowspan=2>col00138 data2 </td><td class=p rowspan=2>15</td> <td class=p rowspan=2>5</td></tr>
<tr><td>13</td></tr>
<tr><td>12</td><td rowspan=2>col01567 prod </td><td class=p rowspan=2>12</td> <td class=p rowspan=2>4</td></tr>
<tr><td>11</td></tr>
<tr><td>10</td><td rowspan=3> </td><td class=p rowspan=3></td> <td class=p rowspan=3>3</td></tr>
<tr><td>9</td></tr>
<tr><td>8</td></tr>
<tr><td>7</td><td rowspan=2>col02001 attic1jc </td><td class=p rowspan=2>11</td> <td class=p rowspan=2>2</td></tr>
<tr><td>6</td></tr>

</table></td>

<td valign="top"><table border=1>

<tr><th>u</th> <th style="background:#FFFF99">9 </th><th>sw port</th> <th>ats port</th></tr>
<tr><td>42</td><td style="background:#a5e9f3">[https://secure.johncompanies.com/mgmt/mrtg/switch.cgi?s=switch-p28&path= c2950 P28]
</td><td class=p>1</td> <td class=p>9</td></tr>
<tr><td>41</td><td>&nbsp </td><td class=p> </td> <td class=p></td></tr>
<tr><td>40</td><td>  </td><td class=p> </td> <td class=p></td></tr>
<tr><td>39</td><td>  </td><td class=p> </td> <td class=p></td></tr>
<tr><td>38</td><td>  </td><td class=p> </td> <td class=p></td></tr>
<tr><td>37</td><td>  </td><td class=p> </td> <td class=p></td></tr>
<tr><td>36</td><td>  </td><td class=p> </td> <td class=p></td></tr>
<tr><td>35</td><td rowspan=2> </td><td class=p rowspan=2>2</td> <td class=p rowspan=2>15</td></tr>
<tr><td>34</td></tr>
<tr><td>33</td><td rowspan=2> </td><td class=p rowspan=2>3 <td> <td class=p rowspan=2>14</td></tr>
<tr><td>32</td></tr>
<tr><td>31</td><td rowspan=2> </td><td class=p rowspan=2>4</td> <td class=p rowspan=2>13</td></tr>
<tr><td>30</td></tr>
<tr><td>29</td><td rowspan=2> </td><td class=p rowspan=2>5</td> <td class=p rowspan=2>12</td></tr>
<tr><td>28</td></tr>
<tr><td>27</td><td rowspan=2> </td><td class=p rowspan=2>6</td> <td class=p rowspan=2>11</td></tr>
<tr><td>26</td></tr>
<tr><td>25</td><td rowspan=2> </td><td class=p rowspan=2>7</td> <td class=p rowspan=2>10</td></tr>
<tr><td>24</td></tr>
<tr><td>23</td><td rowspan=2> </td><td class=p rowspan=2>8</td> <td class=p rowspan=2>8</td></tr>
<tr><td>22</td></tr>
<tr><td>21</td><td colspan=3>  </td></tr>
<tr><td>20</td><td style="text-align:center; background:#cecece">[https://ats9.johncompanies.com ATS-9]
</td><td class=p>24</td> <td class=p></td></tr>
<tr><td>19</td><td colspan=3> </td></tr>
<tr><td>18</td><td rowspan=2> </td><td class=p rowspan=2>9</td> <td class=p rowspan=2>7</td></tr>
<tr><td>17</td></tr>
<tr><td>16</td><td rowspan=2> </td><td class=p rowspan=2>10</td> <td class=p rowspan=2>6</td></tr>
<tr><td>15</td></tr>
<tr><td>14</td><td rowspan=2> </td><td class=p rowspan=2>11</td> <td class=p rowspan=2>5</td></tr>
<tr><td>13</td></tr>
<tr><td>12</td><td rowspan=2> </td><td class=p rowspan=2>12</td> <td class=p rowspan=2>4</td></tr>
<tr><td>11</td></tr>
<tr><td>10</td><td rowspan=2> </td><td class=p rowspan=2>13</td> <td class=p rowspan=2>3</td></tr>
<tr><td>9</td></tr>
<tr><td>8</td><td rowspan=2> * </td><td class=p rowspan=2>14</td> <td class=p rowspan=2>2</td></tr>
<tr><td>7</td></tr>

</table></td>

</tr>
</table>
<br>
<table border=0 cellpadding=5>
<tr>
<td valign="top">
<table border=1>

<tr><th>U</th> <th style="background:#FFFF99">3-5 </th> <th>port</th></tr>
<tr><td class=u>129-127</td> <td style="background:#a5e9f3">[https://secure.johncompanies.com/mgmt/mrtg/switch.cgi?s=switch-p3&path= c2924 P3] </td> <td class=p></td></tr>
<tr><td class=u>126-124</td> <td class=digi>digi2 </td> <td class=p></td></tr>
<tr><td class=u>123<br>118</td> <td>2U </td> <td class=p></td></tr>
<tr><td class=u>117<br>112</td> <td>2U </td> <td class=p></td></tr>
<tr><td class=u>111<br>106</td> <td style="background:#8df595">jail19 * </td> <td class=p></td></tr>
<tr><td class=u>105<br>100</td> <td style="background:#f5ac81">virt15 * </td> <td class=p></td></tr>
<tr><td class=u>99<br>94</td> <td>col00779 wendlink 233.77 </td> <td class=p></td></tr>
<tr><td class=u>93<br>88</td> <td style="background:#8df595">jail2 * </td> <td class=p></td></tr>
<tr><td class=u>87<br>82</td> <td style="background:#f5ac81">virt13 * </td> <td class=p></td></tr>
<tr><td class=u>81<br>76</td> <td style="background:#8df595">old jail2 * </td> <td class=p></td></tr>
<tr><td class=u>75<br>70</td> <td style="background:#8df595">mx2 * </td> <td class=p></td></tr>
<tr><td class=u>69<br>64</td> <td>2U </td> <td class=p></td></tr>
<tr><td class=u>63<br>58</td> <td>col01372 cache2 </td> <td class=p></td></tr>
<tr><td class=u>57<br>52</td> <td>2U </td> <td class=p></td></tr>
<tr><td class=u>51<br>46</td> <td>col00499 </td> <td class=p></td></tr>
<tr><td class=u>45<br>40</td> <td>col02204 NIC P3-12 RMM P3-16 </td> <td class=p></td></tr>
<tr><td class=u>39<br>34</td> <td class=c>2U </td> <td class=p></td></tr>
<tr><td class=u>33<br>28</td> <td>col01727 - REMOVED 9/22/13 </td> <td class=p></td></tr>
</table>
</td>

<td valign="top">
<table border=1>

<tr><th>U</th> <th style="background:#FFFF99">3-6 </th> <th>port</th></tr>
<tr><td class=u>129-127</td> <td style="background:#a5e9f3">[https://secure.johncompanies.com/mgmt/mrtg/switch.cgi?s=switch-p4&path= c2924 P4] </td> <td class=p></td></tr>
<tr><td class=u>126-124</td> <td>1U </td> <td class=p></td></tr>
<tr><td class=u>123<br>118</td> <td style="background:#f5ac81">virt11 * </td> <td class=p>4</td></tr>
<tr><td class=u>117<br>112</td> <td style="background:#f5ac81">virt17 * </td> <td class=p>7</td></tr>
<tr><td class=u>111-109</td> <td>1U </td> <td class=p></td></tr>
<tr><td class=u>108-106</td> <td>2U </td> <td class=p></td></tr>
<tr><td class=u>105<br>100</td> <td style="background:#8df595">jail8 * </td> <td class=p></td></tr>
<tr><td class=u>99<br>94</td> <td style="background:#f5ac81">virt19 * </td> <td class=p>12</td></tr>
<tr><td class=u>93<br>88</td> <td style="background:#8df595">jail9 * </td> <td class=p></td></tr>
<tr><td class=u>87<br>82</td> <td>col01372 cache4 </td> <td class=p></td></tr>
<tr><td class=u>81<br>76</td> <td>col01372 dbmsg.bookoo </td> <td class=p></td></tr>
<tr><td class=u>75<br>70</td> <td>2U </td> <td class=p></td></tr>
<tr><td class=u>69<br>64</td> <td>2U </td> <td class=p></td></tr>
<tr><td class=u>63<br>58</td> <td>2U </td> <td class=p></td></tr>
<tr><td class=u>57<br>52</td> <td>2U </td> <td class=p></td></tr>
<tr><td class=u>51<br>46</td> <td>2U </td> <td class=p></td></tr>
<tr><td class=u>45<br>40</td> <td>col00985 </td> <td class=p></td></tr>
<tr><td class=u>39<br>34</td> <td>col01372 cache4 </td> <td class=p></td></tr>
<tr><td class=u>33<br>28</td> <td> </td> <td class=p></td></tr>

</table>
</td>

<td valign="top">
<table border=1>

<tr><th>U</th> <th style="background:#FFFF99">3-7 </th> <th>port</th></tr>
<tr><td class=u>129-127</td> <td style="background:#a5e9f3">[https://secure.johncompanies.com/mgmt/mrtg/switch.cgi?s=switch-p17&path= c2924 P17] </td> <td class=p></td></tr>
<tr><td class=u>126-124</td> <td class=c>1U </td> <td class=p></td></tr>
<tr><td class=u>123-121</td> <td class=digi>digi1 (no rails)</td> <td class=p></td></tr>
<tr><td class=u>120-118</td> <td style="background:#a5e9f3">switch-m2 </td> <td class=p></td></tr>
<tr><td class=u>117-115</td> <td style="text-align:center; background:#cecece">ATS </td> <td class=p></td></tr>
<tr><td class=u>114-112</td> <td style="text-align:center; background:#cecece">ATS </td> <td class=p></td></tr>
<tr><td class=u>111<br>106</td> <td style="background:#8df595">jail11 * </td> <td class=p></td></tr>
<tr><td class=u>105<br>100</td> <td class=c>col01372 solr7.bookoo </td> <td class=p></td></tr>
<tr><td class=u>99<br>94</td> <td>col01372 solr8.bookoo </td> <td class=p></td></tr>
<tr><td class=u>93<br>88</td> <td style="background:#f5ac81">virt12 * </td> <td class=p></td></tr>
<tr><td class=u>87<br>82</td> <td style="background:#8df595">jail7 * </td> <td class=p></td></tr>
<tr><td class=u>81<div class=s3></div>73</td> <td style="background:#f5ac81">virt14 </td> <td class=p></td></tr>
<tr><td class=u>72-70</td> <td style="background:#cecece"> bwdb1 (no rails) </td> <td class=p></td></tr>
<tr><td class=u>69<br>64</td> <td style="background:#f5ac81">virt16 * </td> <td class=p></td></tr>
<tr><td class=u>63-61</td> <td>col00991 </td> <td class=p></td></tr>
<tr><td class=u>60<div class=s4></div>49</td> <td style="background:#cecece">backup2 *</td> <td class=p></td></tr>
<tr><td class=u>48<br>43</td> <td style="background:#8df595">jail18 * </td> <td class=p></td></tr>
<tr><td class=u>42<br>37</td> <td style="background:#8df595">mx1 * </td> <td class=p></td></tr>
<tr><td class=u>36-34</td> <td class=c>1U </td> <td class=p></td></tr>
<tr><td class=u>33-31</td> <td>col00994 </td> <td class=p></td></tr>
<tr><td class=u>30<br>25</td> <td style="background:#f5ac81">virt9 * </td> <td class=p></td></tr>
<tr><td class=u>24<br>19</td> <td>2U </td> <td class=p></td></tr>
<tr><td class=u>18-16</td> <td>col0460 </td> <td class=p></td></tr>
<tr><td class=u>15-13</td> <td class=c>1U </td> <td class=p></td></tr>
<tr><td class=u>12-10</td> <td style="background:#cecece">nat </td> <td class=p></td></tr>

</table>
</td>

<td valign="top">
<table border=1>

<tr><th>U</th> <th style="background:#FFFF99">3-8 </th> <th>port</th></tr>
<tr><td class=u>129-127</td> <td class=c>1U </td> <td class=p></td></tr>
<tr><td class=u>126-124</td> <td style="background:#a5e9f3">c3750 </td> <td class=p></td></tr>
<tr><td class=u>122<br>118</td> <td style="background:#a5e9f3">c3550 </td> <td class=p></td></tr>
<tr><td class=u>117-115</td> <td style="background:#a5e9f3">[https://secure.johncompanies.com/mgmt/mrtg/switch.cgi?s=switch-p1&path= c2924 switch-p1] </td> <td class=p></td></tr>
<tr><td class=u>114-112</td> <td style="background:#a5e9f3">[https://secure.johncompanies.com/mgmt/mrtg/switch.cgi?s=switch-p1a&path= c2960 switch-p1a] </td> <td class=p></td></tr>
<tr><td class=u>111-109</td> <td style="background:#a5e9f3">[https://secure.johncompanies.com/mgmt/mrtg/switch.cgi?s=switch-p1b&path= c2960 switch-p1b] </td> <td class=p></td></tr>
<tr><td class=u>108<br>103</td> <td style="background:#cecece">firewall * </td> <td class=p></td></tr>
<tr><td class=u>102-100</td> <td style="background:#a5e9f3">ASA5500 </td> <td class=p></td></tr>
<tr><td class=u>99-97</td> <td style="background:#a5e9f3">M1 </td> <td class=p></td></tr>
<tr><td class=u>96<br>91</td> <td class=c>col02179 staging.issisystems.com </td> <td class=p></td></tr>
<tr><td class=u>90-88</td> <td style="background:#cecece">gate (old f/w) </td> <td class=p></td></tr>
<tr><td class=u>87<br>82</td> <td style="background:#f5ac81">quar1 * (no rails)</td> <td class=p></td></tr>
<tr><td class=u>81-79</td> <td>[shelf] </td> <td class=p></td></tr>
<tr><td class=u>78-76</td> <td class=c>1U </td> <td class=p></td></tr>
<tr><td class=u>75-73</td> <td style="text-align:center; background:#cecece">ATS </td> <td class=p></td></tr>
<tr><td class=u>72-70</td> <td style="text-align:center; background:#cecece">ATS </td> <td class=p></td></tr>
<tr><td class=u>69<br>64</td> <td style="background:#8df595">jail4</td> <td class=p></td></tr>
<tr><td class=u>63<br>58</td> <td style="background:#8df595">jail1 * (no rails)</td> <td class=p></td></tr>
<tr><td class=u>57-55</td> <td class=c>1U </td> <td class=p></td></tr>
<tr><td class=u>54<br>52</td> <td style="background:#cecece">console (no rails) </td> <td class=p></td></tr>
<tr><td class=u>51<br>46</td> <td style="background:#cecece">mail * (no rails) </td> <td class=p></td></tr>
<tr><td class=u>45<br>40</td> <td class=c>col01372 dbemail.bookoo </td> <td class=p></td></tr>
<tr><td class=u>39<br>34</td> <td class=>col01943 * </td> <td class=p></td></tr>
<tr><td class=u>33-31</td> <td>[shelf] </td> <td class=p></td></tr>
<tr><td class=u>30<br>25</td> <td class=c>2U </td> <td class=p></td></tr>
<tr><td class=u>24<br>19</td> <td style="background:#cecece">backup1 </td> <td class=p></td></tr>
<tr><td class=u>18<br>13</td> <td style="background:#8df595">jail17 * </td> <td class=p></td></tr>

</table>
</td>

</tr>
</table>
<br>
<table border=0 cellpadding=3>
<tr>

<td valign="top">
<table border=1>

<tr><th>U</th> <th style="background:#FFFF99">6-16 </th> <th>port</th></tr>
<tr><td class=u>129-127</td> <td style="background:#a5e9f3">[https://secure.johncompanies.com/mgmt/mrtg/switch.cgi?s=switch-p13&path= c2924 P13] </td> <td class=p></td></tr>
<tr><td class=u>126<br>121</td> <td class=c>2U </td> <td class=p></td></tr>
<tr><td class=u>120-118</td> <td>1372 gig sw </td> <td class=p></td></tr>
<tr><td class=u>117<br>112</td> <td class=c>2U </td> <td class=p></td></tr>
<tr><td class=u>111<br>106</td> <td>col01372 solr5.bookoo </td> <td class=p></td></tr>
<tr><td class=u>105<br>100</td> <td>col01372 monitor.bookoo </td> <td class=p></td></tr>
<tr><td class=u>99<br>94</td> <td>col01372 server3.kingwoodyardsales</td> <td class=p></td></tr>
<tr><td class=u>93<br>88</td> <td>col01372 web3.bookoo </td> <td class=p></td></tr>
<tr><td class=u>87<br>82</td> <td>col01372 web1.bookoounderground.com</td> <td class=p></td></tr>
<tr><td class=u>81<br>76</td> <td class=c>2U </td> <td class=p></td></tr>
<tr><td class=u>75-73</td> <td style="text-align:center; background:#cecece">PS </td> <td class=p></td></tr>
<tr><td class=u>72-70</td> <td style="text-align:center; background:#cecece">PS </td> <td class=p></td></tr>
<tr><td class=u>69<br>64</td> <td>col01372 dbstats.bookoo </td> <td class=p></td></tr>
<tr><td class=u>63<br>58</td> <td>col01372 db1.bookoounderground *</td> <td class=p></td></tr>
<tr><td class=u>57<br>52</td> <td>col01372 backup.clker </td> <td class=p></td></tr>
<tr><td class=u>51<br>46</td> <td>col01372 dbbackup.bookoo </td> <td class=p></td></tr>
<tr><td class=u>45<br>40</td> <td>col01372 dbpv.bookoo </td> <td class=p></td></tr>
<tr><td class=u>39<br>34</td> <td>col01372 web7.bookoo </td> <td class=p></td></tr>
<tr><td class=u>33<br>28</td> <td>col01372 dbm2.bookoo * </td> <td class=p></td></tr>
<tr><td class=u>27<br>22</td> <td>col01372 dbsense.bookoo </td> <td class=p></td></tr>

</table>
</td>

<td valign="top">
<table border=1>

<tr><th>U</th> <th style="background:#FFFF99">6-17 </th> <th>port</th></tr>
<tr><td class=u>129-127</td> <td>1U</td></tr>
<tr><td class=u>126-124</td> <td>col01372 gig sw </td> <td class=p></td></tr>
<tr><td class=u>123<br>118</td> <td class=c>2U </td> <td class=p></td></tr>
<tr><td class=u>117<br>112</td> <td>col01372 db1.clker </td> <td class=p>14</td></tr>
<tr><td class=u>111<br>106</td> <td>col01372 solr6.bookoo </td> <td class=p></td></tr>
<tr><td class=u>105<br>100</td> <td>col01372 solr4.bookoo </td> <td class=p>18</td></tr>
<tr><td class=u>99<br>94</td> <td>col01372 web5.bookoo </td> <td class=p>19</td></tr>
<tr><td class=u>93<br>88</td> <td>col01372 sso.bookoo </td> <td class=p>4</td></tr>
<tr><td class=u>87-85</td> <td>1U </td> <td class=p>9</td></tr>
<tr><td class=u>84-82</td> <td>col01372 balancer </td> <td class=p>3,4</td></tr>
<tr><td class=u>81-79</td> <td>1U </td> <td class=p></td></tr>
<tr><td class=u>78-76</td> <td style="text-align:center; background:#cecece">PS </td> <td class=p></td></tr>
<tr><td class=u>75-73</td> <td style="text-align:center; background:#cecece">PS </td> <td class=p></td></tr>
<tr><td class=u>72-70</td> <td style="text-align:center; background:#cecece">PS </td> <td class=p></td></tr>
<tr><td class=u>69<br>64</td> <td>col01372 solr2.bookoo </td> <td class=p></td></tr>
<tr><td class=u>63<br>58</td> <td>col01372 web6.bookoo </td> <td class=p></td></tr>
<tr><td class=u>57<br>52</td> <td>col01372 web8.bookoo </td> <td class=p></td></tr>
<tr><td class=u>51<br>46</td> <td>col01372 web2.bookoo </td> <td class=p></td></tr>
<tr><td class=u>45<br>40</td> <td>col01372 web1.bookoo </td> <td class=p></td></tr>
<tr><td class=u>39<br>34</td> <td>col01372 web4.bookoo </td> <td class=p></td></tr>
<tr><td class=u>33<br>28</td> <td>2U </td> <td class=p></td></tr>
<tr><td class=u>27<br>22</td> <td>col01372 dbm1.bookoo * </td> <td class=p></td></tr>
<tr><td class=u>21-19</td> <td>1U </td> <td class=p></td></tr>
<tr><td class=u>18<br>13</td> <td>col01372 web1.cliker * </td> <td class=p></td></tr>

</table>
</td>

</tr>
</table>

Infrastructure Machines

2013-10-17T17:20:09Z

24.165.90.109: /* Notes */

= jails =

== jail1 ==

* Location: castle, cab 3-8
* OS: FreeBSD 6.2 i386
* Networking: Priv IP: 10.1.4.101 (PCI nic), Pub IP: 69.55.230.107 (onboard)
* Hardware: Dell 2450. 4 SCSI SCA drive bays (2 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: two 74 GB (4 x 74GB) RAID1 arrays running on an Adaptec-based, Dell-branded (perc) RAID card.
* Remote management: none
* Disk accounting: gvinum

== jail2 ==

* Location: castle, cab 3-5
* OS: FreeBSD 7.2 amd64
* Networking: Priv IP: 10.1.4.102, Pub IP: 69.55.228.53 (2 onboard nics)
* Hardware: Dell 2950. 6 SATA/SAS drive bays (3 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: one 146 GB (2 x 146GB) RAID1 array, two 300 GB (4 x 300GB) RAID1 arrays running on an LSI-based, Dell-branded (PERC 6/i) RAID card.
* Remote management: [[DRAC/RMM|DRAC]] @ 10.1.4.232
* Disk accounting: md

== jail3 ==
* Location: i2b, cab 6
* OS: FreeBSD 8.3 amd64
* Networking: Priv IP: 10.1.2.103, Pub IP: 69.55.229.7 (2 onboard nics)
* Hardware: Supermicro (custom build). 6 SATA drive bays (2 columns of 3, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: two 300 GB (2 x 300GB) RAID1 array running on a 3ware 8006-2LP RAID card.
* Remote management: none
* Disk accounting: md

=== Notes ===
* '''We should not add users to this server since it is at I2B'''
* must be ssh'd to from nat2
* is a super jail for customer col01737

== jail4 ==

* Location: castle, cab 3-8
* OS: FreeBSD 9.1 x86_64
* Networking: Priv IP: 10.1.4.104, Pub IP: 69.55.228.104 (2 onboard nics)
* Hardware: Dell 2850. 6 SATA/SAS drive bays (3 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: one 1.4 TB RAID 5 array Dell-branded (PERC 4e)LSI megarc RAID card.
* Remote management: None
* Disk accounting: md

=== Notes ===
Only FreeBSD 9.1 jail

== jail7 ==

* Location: castle, cab 3-7
* OS: FreeBSD 6.3 i386
* Networking: Priv IP: 10.1.4.107, Pub IP: 69.55.230.108 (2 onboard nics)
* Hardware: Dell 2950. 6 SATA/SAS drive bays (3 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: two 146 GB (4 x 146GB) RAID1 arrays, one 74 GB (2 x 74GB) RAID1 array running on an LSI-based, Dell-branded (PERC 6/i) RAID card.
* Remote management: [[DRAC/RMM|DRAC]] @ 10.1.4.237
* Disk accounting: gvinum

=== Notes ===
Do not run a verify while OS/jails running, will crash.

== jail8 ==

* Location: castle, cab 3-6
* OS: FreeBSD 8.0 amd64
* Networking: Priv IP: 10.1.4.108, Pub IP: 69.55.234.2 (2 onboard nics)
* Hardware: Dell 2950. 6 SATA/SAS drive bays (3 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: one 146 GB (2 x 146GB) RAID1 array, one 300 GB (2 x 300GB) RAID1 array, one 400 GB (2 x 400GB) RAID1 array, running on an LSI-based, Dell-branded (PERC 6/i) RAID card.
* Remote management: [[DRAC/RMM|DRAC]] @ 10.1.4.238
* Disk accounting: md

== jail9 ==

* Location: castle, cab 3-6
* OS: FreeBSD 8.2 amd64
* Networking: Priv IP: 10.1.4.109, Pub IP: 69.55.232.36 (2 onboard nics)
* Hardware: Dell 2950. 6 SATA/SAS drive bays (3 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: one 146 GB (2 x 146GB) RAID1 array, one 400 GB (2 x 300GB) RAID1 array running on an LSI-based, Dell-branded (PERC 5/i) RAID card.
* Remote management: [[DRAC/RMM|DRAC]] @ 10.1.4.239
* Disk accounting: md

== jail11 ==

* Location: castle, cab 3-7
* OS: FreeBSD 4.7 i386
* Networking: Priv IP: 10.1.4.111 (PCI nic), Pub IP: 69.55.236.92 (onboard)
* Hardware: Dell 2450. 4 SCSI SCA drive bays (2 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: two 74 GB (4 x 74GB) RAID1 arrays running on an Adaptec-based, Dell-branded (perc) RAID card.
* Remote management: none
* Disk accounting: vinum

== mx1 ==

* Location: castle, cab 3-7
* OS: FreeBSD 4.11 i386
* Networking: Priv IP: 10.1.4.201 (PCI nic), Pub IP: 69.55.237.3 (onboard)
* Hardware: Dell 2450. 4 SCSI SCA drive bays (2 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: one 36 GB (2 x 36GB) RAID1 array, one 74 GB (2 x 74GB) RAID1 array running on an Adaptec-based, Dell-branded (perc) RAID card.
* Remote management: none
* Disk accounting: vinum

=== Notes ===
* is our (old) backup mail/dns vps service host

== mx2 ==

* Location: castle, cab 3-5
* OS: FreeBSD 7.1 i386
* Networking: Priv IP: 10.1.4.202 (PCI nic), Pub IP: 69.55.237.90 (onboard)
* Hardware: Dell 2450. 4 SCSI SCA drive bays (2 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: two 74 GB (4 x 74GB) RAID1 arrays running on an Adaptec-based, Dell-branded (perc) RAID card.
* Disk accounting: gvinum

=== Notes ===
* is our latest backup mail/dns vps service host

== jail17 ==
* Location: castle, cab 3-8
* OS: FreeBSD 4.10 i386
* Networking: Priv IP: 10.1.4.117 (PCI nic), Pub IP: 69.55.228.2 (onboard nics)
* Hardware: Supermicro (custom build). 6 SCA SCSI drive bays (2 columns of 3, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: two 74 GB (4 x 74GB) RAID1 arrays, one 146 GB (2 x 146GB) RAID1 array, running on an Adaptec 2120S RAID card.
* Remote management: none
* Disk accounting: vinum

== jail18 ==
* Location: castle, cab 3-7
* OS: FreeBSD 4.10 i386
* Networking: Priv IP: 10.1.4.117 (PCI nic), Pub IP: 69.55.228.2 (onboard nics)
* Hardware: Supermicro (custom build). 6 SCA SCSI drive bays (2 columns of 3, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: two 74 GB (4 x 74GB) RAID1 arrays, one 146 GB (2 x 146GB) RAID1 array, running on an Adaptec 2120S RAID card.
* Remote management: none
* Disk accounting: vinum

== jail19 ==
* Location: castle, cab 3-6
* OS: FreeBSD 6.1 i386
* Networking: Priv IP: 10.1.4.119 (PCI nic), Pub IP: 69.55.228.200 (onboard nics)
* Hardware: Supermicro (custom build). 6 SCA SCSI drive bays (2 columns of 3, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: one 74 GB (2 x 74GB) RAID1 array, one 146 GB (2 x 146GB) RAID1 array, running on an Adaptec 2120S RAID card.
* Remote management: none
* Disk accounting: gvinum

= virts =

== quar1 ==

* Location: castle, cab 3-8
* OS: RedHat 7.3 x86
* Networking: Priv IP: 10.1.4.151 (PCI nic), Pub IP: 69.55.227.2 (onboard nic)
* Hardware: Dell 2450. 4 SCSI SCA drive bays (2 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: one 36 GB (2 x 36GB) RAID1 array, one 74 GB (2 x 74GB) RAID1 array, running on an Adaptec-based, Dell-branded (perc) RAID card.
* Remote management: none
* Virtuozzo version: 2.6.1
* VZ license: hwid=23C0.C0E1.6FDD.08BA.8971.8E1C.EBD5.1EDC serial=0DE6.903E.E239.E23F.470C.4369.4104.A5A4

=== Notes ===
* used to be the home of customers who's VE's would just run out of control/badly
* has a max of 10 VE's allowed to run

== virt9 ==

* Location: castle, cab 3-7
* OS: RedHat 7.3 x86
* Networking: Priv IP: 10.1.4.59 (PCI nic), Pub IP: 69.55.226.161 (onboard nic)
* Hardware: Dell 2450. 4 SCSI SCA drive bays (2 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: two 74 GB (2 x 74GB) RAID1 arrays running on an Adaptec-based, Dell-branded (perc) RAID card.
* Remote management: none
* Virtuozzo version: 2.6.1
* VZ license: hwid=BC15.B4D6.0D25.A5FE.F3BA.D518.E351.AE3F serial=F6AD.B6B4.5650.8869.C97C.73EE.AF65.FA8B

== virt11 ==

* Location: castle, cab 3-6
* OS: CentOS 5.4 x86
* Networking: Priv IP: 10.1.4.61, Pub IP: 69.55.238.3, 2 onboard nics
* Hardware: Dell 2950. 6 SATA/SAS drive bays (3 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: one 146 GB (2 x 146GB) RAID1 array, one 400 GB (2 x 400GB) RAID1 array, one 146 GB (2 x 146GB) RAID1 array, running on an LSI-based, Dell-branded (perc 6/i) RAID card.
* Remote management: [[DRAC/RMM|DRAC]] @ 10.1.4.211
* Virtuozzo version: 4.0.0
* VZ license: hwid="029D.A187.78E1.480F.49E3.E20A.7389.7F79" serial="163C.F3E2.195F.96B5.2D38.8937.9600.4A05"

== virt12 ==

* Location: castle, cab 3-6
* OS: CentOS 5.2 x86
* Networking: Priv IP: 10.1.4.62, Pub IP: 69.55.227.70, 2 onboard nics
* Hardware: Dell 2950. 6 SATA/SAS drive bays (3 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: two 300 GB (2 x 300GB) RAID1 arrays one 400 GB (2 x 400GB) RAID1 array, running on an LSI-based, Dell-branded (perc 6/i) RAID card.
* Remote management: [[DRAC/RMM|DRAC]] @ 10.1.4.212
* Virtuozzo version: 4.0.0
* VZ license: hwid="0C53.A413.E095.B4F4.51BC.D740.6919.A77B" serial="84E5.9498.3759.E683.E24B.2514.CA72.DC31"

== virt13 ==

* Location: castle, cab 3-5
* OS: CentOS 6.2 x86_64
* Networking: Priv IP: 10.1.4.63, Pub IP: 69.55.226.2, 2 onboard nics
* Hardware: Dell 2950. 6 SATA/SAS drive bays (3 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: one 146 GB (2 x 146GB) RAID1 array, one 600 GB (2 x 600GB) RAID1 array, running on an LSI-based, Dell-branded (perc 6/i) RAID card.
* Remote management: [[DRAC/RMM|DRAC]] @ 10.1.4.213
* Virtuozzo version: 4.7.0
* VZ license: hwid="7D07.93BE.0B1F.7D2B.B039.4B5B.48B6.453B" serial="60A4.A94C.44BB.DCD6.8D03.1778.605B.10FE"

=== Notes ===
* home to our latest/current signups
* currently the only 64bit vz host

== virt14 ==

* Location: castle, cab 3-?
* OS: CentOS 6.4x86_64
* Networking: Priv IP: 10.1.4.64 Pub IP: 69.55.225.14 2 onboard nics
* Hardware: Dell 2950. 6 SATA/SAS drive bays (3 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: one 600 GB (2 x 600GB) RAID1 array, running on an LSI-based, Dell-branded (perc 6/i) RAID card.
* Remote management: [[DRAC/RMM|DRAC]] @ 10.1.4.214
* Virtuozzo version: 4.7.0
* VZ license: hwid="" serial=""

=== Notes ===
* new host not yet in production
* virt 13 and 14 currently the only 64bit vz hosts

== virt15 ==

* Location: castle, cab 3-5
* OS: RedHat 9 x86
* Networking: Priv IP: 10.1.4.65, Pub IP: 69.55.232.160 (2 onboard nics)
* Hardware: Supermicro (custom build). 6 SCSI SCA drive bays (2 columns of 3, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: one 74 GB (2 x 74GB) RAID1 array, two 146 GB (2 x 146GB) RAID1 arrays, running on an LSI MegaRAID SCSI 320-1 RAID card.
* Remote management: none
* Virtuozzo version: 2.6.2
* VZ license: hwid=A90F.6F48.E723.D8BA.3025.184A.5B73.D11E serial=E94B.5164.C1E6.A67F.67D1.7D96.0B6C.5524

== virt16 ==

* Location: castle, cab 3-7
* OS: Fedora Core 4 x86
* Networking: Priv IP: 10.1.4.66, Pub IP: 69.55.232.2 (2 onboard nics)
* Hardware: Supermicro (custom build). 6 SCSI SCA drive bays (2 columns of 3, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: one 74 GB (2 x 74GB) RAID1 array, one 146 GB (2 x 146GB) RAID1 array, running on an LSI MegaRAID SCSI 320-1 RAID card.
* Remote management: none
* Virtuozzo version: 3.0.0
* VZ license: hwid=DEFA.A325.7230.BBC8.9715.8B52.3FD7.27BE serial=66C0.41EA.3FBB.11D3.9CC6.55C7.09AE.14AB

== virt17 ==

* Location: castle, cab 3-6
* OS: CentOS 4.4 x86
* Networking: Priv IP: 10.1.4.67, Pub IP: 69.55.232.162, 2 onboard nics
* Hardware: Dell 2950. 6 SATA/SAS drive bays (3 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: two 146 GB (2 x 146GB) RAID1 arrays running on an LSI-based, Dell-branded (perc 5/i) RAID card.
* Remote management: [[DRAC/RMM|DRAC]] @ 10.1.4.217
* Virtuozzo version: 3.0.0
* VZ license: hwid=2E14.AED9.70B8.C26E.D99F.B0D3.BCD2.229C serial=2A11.DAD0.61DB.E889.8DF4.9AF7.CF82.3C37

== virt19 ==

* Location: castle, cab 3-6
* OS: CentOS 5.2 x86
* Networking: Priv IP: 10.1.4.69, Pub IP: 69.55.236.2, 2 onboard nics
* Hardware: Dell 2950. 6 SATA/SAS drive bays (3 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: one 146 GB (2 x 146GB) RAID1 array, one 300 GB (2 x 300GB) RAID1 array, running on an LSI-based, Dell-branded (perc 5/i) RAID card.
* Remote management: [[DRAC/RMM|DRAC]] @ 10.1.4.219
* Virtuozzo version: 3.0.0
* VZ license: hwid=3968.13F7.B2AC.8952.8E19.13A9.6EF5.5822 serial=061D.84CD.CCE5.B213.15B5.C061.D6A7.B034

= mail =
== Summary ==
This machine (mail) is the swiss army knife of the company, playing host to many services and functions.

* Location: castle, cab 3-8
* OS: FreeBSD 4.10 x86
* Networking: Priv IP: 10.1.4.5, Pub IPs: 69.55.230.2, 69.55.225.225 (ns1c jail), 69.55.230.9. 1 onboard and 1 PCI
* Hardware: Dell 2450. 4 SCSI SCA drive bays (2 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: two 36 GB (2 x 36GB) RAID1 arrays running on an Adaptec-based, Dell-branded (perc) RAID card.

== Services Provided ==
* mail
* web
* mysql
* bigbrother server/pager
* snmp
* named in a jail (ns1c)

== email ==
This server hosts mail for johncompanies.com (mail.johncompanies.com). Sendmail 8.13.6/8.13.6 is listening on 69.55.230.2 port 25 for incoming mail. Relaying is allowed per /etc/mail/relay-domains
Other addresses (aliases) are defined per /etc/mail/aliases

The following active users have mail hosted on this server:
* dave
* linux
* support
* payments
* sales
* tech1
* info

Traditionally, mail is checked via shell apps (pine). qpopper (pop3s) is running to allow mail downloading. Checking mail in this way causes an opened INBOX in pine to lock read-only. For this reason, we tee incoming mail to support and linux to tech1.

Procmail rules are setup to filter spam and send text messages. They are enabled for info, support, linux, tech1, dave and can be found in ~/Procmail/, for example:
<pre># more ~support/Procmail/rc.emergency
:0c # use c only if you want to forward a copy and file the original later
* ^Subject:.*\<emergency\>
* ! ^Subject:\<re\>
{
:0h
FROMANDSUBJECT=|formail -XFrom: -XSubject:

:0fwh
| /usr/local/bin/formail -I"Subject: " -I"To: pager@johncompanies.com" ; echo $FROMANDSUBJECT ; echo

:0
! -t
}

</pre>

control: <tt>cd /etc/mail; make stop</tt> (stop), <tt>cd /etc/mail; make start</tt> (start)

The following aliases are also in place:

<pre>debian: linux
jobs: info
careers: info
#reboot: 6128102202@txt.att.net
#reboot: 8582298897@vtext.com
reboot: pager
#pager: 8582298897@vtext.com
pager: 4158718324@txt.att.net
tech1on: "| /usr/local/sbin/tech1on.sh"
tech1off: "| /usr/local/sbin/tech1off.sh"</pre>

To change them, edit <tt>/etc/aliases</tt> and then run <tt>newaliases</tt>

Note on tech1: this address was setup as a read-only address to be mirrored on all email coming into support and linux. We set this up so we could easily check support mail via a pop client- popping email locks out the user in pine so checking support/linux directly via pop was not an option. When checking and responding to email that comes into tech1, care should be taken to make sure it is sent as/under an address other than tech1. This is cause tech1 is not monitored by support staff as closely as email to support/linux. Further, the tech on call may not be checking tech1. Lastly, because of the nature of the copying, you will sometimes notice certain automated email/notices are received 2x in support- this is because of/related to the tech1 mirror.

To enable it (on mail, run):

<tt>~support/tech1on.sh</tt>

To disable

<tt>~support/tech1off.sh</tt>

Or via email:

<tt>tech1on@johncompanies.com
tech1off@johncompanies.com</tt>

== web ==

See [[Management_System_/_Public_Website_/_Signup|Management System / Public Website / Signup]]

== mysql ==
mysql 4.1.22 is running on port 3306

* datadir: <tt>/mnt/data1/db/mysql/</tt>
* config: <tt>/etc/my.cnf</tt>
* database: <tt>jc</tt>
* control: <tt>/usr/local/etc/rc.d/mysql-server.sh stop</tt> (stop), <tt>/usr/local/etc/rc.d/mysql-server.sh start</tt> (start)

== bigbrother ==
There is a client running on mail (which monitors the services running on mail and mail itself), installed under <tt>/usr/home/bb/bbc1.9e-btf</tt><br>
And the big brother pager/server (which displays information gathered from all bb-monitored machines, including mail) is installed under <tt>/usr/home/bb/bbsrc/bb1.9i-btf</tt>

Both are running under the user <tt>bb</tt>

Refer to [[BigBrother]] for more about use.

== DNS (ns1c.johncompanies.com) ==
ns1c is a jail running on the mail server, who's IP is 69.55.225.225

It's running from <tt>/mnt/data1/ns1c-dir</tt>

See [[DNS]] for more details

== Usage and Notes ==
* always mounted to backup1 and backup2 via nfs:
<pre>backup2:/mnt/data1 on /backup (nfs)
backup2:/mnt/data2 on /backup2 (nfs)
backup2:/mnt/data3 on /backup3 (nfs)
backup2:/mnt/data4 on /backup4 (nfs)
backup1:/data on /backup1 (nfs)
</pre>

== Cronjobs ==
* * * * * /usr/local/www/mgmt/mrtg/mrtg.sh > /dev/null 2>&1
Gathers up data for our mrtg/load graphs

*/5 * * * * /usr/local/bin/rsync -a root@nat2:/mnt/data1/mrtg/data/ /usr/local/www/mgmt/mrtg/data/
Gathers up data from i2b servers for our mrtg/load graphs

40 0 * * * /usr/local/bin/rsync -a root@nat2:"/mnt/data1/mrtg/*.cfg" /usr/local/www/mgmt/mrtg
Gathers up mrtg configuration (port names) from i2b switches for our mrtg/load graphs

41 0 * * * for f in `grep -l "mnt\/data1" /usr/local/www/mgmt/mrtg/switch-p*.cfg`; do cat $f | sed s#\/mnt\/data1#\/usr\/local\/www\/mgmt# > $f.new; mv $f.new $f; done
Gathers up mrtg configuration (port names) from castle switches for our mrtg/load graphs

1 0 1 * * cp /usr/local/www/mgmt/html/top20ip /usr/local/www/mgmt/html/top20ip_last
1 0 1 * * cp /usr/local/www/mgmt/html/top20customers /usr/local/www/mgmt/html/top20customers_last
2 * * * * /usr/local/www/cronjobs/top20ip.pl > /dev/null 2>&1
15 * * * * /usr/local/www/cronjobs/top20customer.pl > /dev/null 2>&1
1 0 1 * * rm /usr/local/www/mgmt/html/bandtrack
Archiving and generation of bandwidth statistics presented in mgmt -> Reference -> Bandwidth

1 0 * * * /usr/local/etc/rsync.backup
Nightly backup script

0 1 * * * /usr/local/www/mgmt/awstats/wwwroot/cgi-bin/awstats.pl -config=jcpub -update
Public web traffic stats

15 0 * * * rm /usr/local/www/mgmt/bwgraphs/*.png
16 0 * * * rm /usr/local/www/am/bwgraphs/*
Cleanup for graph-related temp data generated by customers using the bandwidth reports via the AM

10 0 1 * * /usr/local/www/cronjobs/monthly_bandwidth_report.pl
Monthly bandwidth overage report

*/3 * * * * /usr/local/www/cronjobs/bbcheck.pl
Updates mgmt with bb monitoring issues

5 0 * * * /usr/local/www/cronjobs/shutdownreminder.pl
Emails customers reminding them of upcoming shutdown date

7 0 * * * /usr/local/www/cronjobs/invoice_email.pl
Emails customers who have invoices and are set to auto-email (currently no customer gets these)

8 */4 * * * /usr/local/www/cronjobs/mysqlrepchk.pl
Checking that we are properly replicating (mysql) traffic data from bwdb to backup1

16 0 1 * * /usr/local/www/cronjobs/purge_traffic.pl
Removed old traffic data from the traffic database (running on backup1)

*/5 * * * * chmod 0700 /usr/local/www/ccard_orders/* && mv /usr/local/www/ccard_orders/* /usr/local/www/ccard_orders/done
Secure credit card data: set root-read-only

25 0 * * * /usr/local/www/cronjobs/biller.pl
Enters service charges in customer billing ledgers

10 13 * * * /usr/local/www/cronjobs/pfp_batch_gather.pl
Looks for customers with balance due and active credit card on file, prepares a payflow batch

10 14 * * * /usr/local/www/cronjobs/pfp_batch_process.pl
Tries to collect ccard funds for items in payflow batch - communicates with payflow

15 13 * * * /usr/local/www/cronjobs/pb_batch_gather.pl
Looks for customers with balance due and active paypal billing agreement on file, prepares a paypal batch

15 14 * * * /usr/local/www/cronjobs/pb_batch_process.pl
Tries to collect paypal funds for items in paypal batch - communicates with paypal

0 7 * * 1 /usr/local/www/cronjobs/email_pmt_reminder.pl
Emails customers in arrears, reminding them to pay

0 0 1 * * /usr/bin/mail -s 'archive sent mail in pine' support@johncompanies.com < /dev/null
Reminds us to archive sent mail

0 3 * * * /usr/local/bin/rsync -a isys.e-monitoring.net:/var/mail /backup2/isys; /usr/local/bin/rsync -a isys.e-monitoring.net:/usr/home /backup2/isys
Backup data on isys

== Regular maintenance ==
*[[Routine_Maintenance#Adaptec_Controllers|Check RAID array]]

= nat =
== Summary ==
This is the main machine to which we ssh and runs all our screen sessions. Further, it's ip runs in a special block which is not routed through the firewall and this is somewhat immune to DoS attacks which hobble our firewall. Lastly, it acts as a nat server for certain/random devices on the private network.

* Location: castle, cab 3-7
* OS: FreeBSD 9.1 i386
* Networking: Priv IP: 10.1.4.1, Pub IPs: 69.55.233.195, 69.55.233.196, 69.55.233.197, 69.55.233.198, 69.55.233.199. 1 onboard and 1 PCI
* Hardware: Custom 1U. single power supply.
* Drives: one 8 GB IDE drive

== Services Provided ==
* nat

== nat control ==
All rules are contained in and look like:

<pre>cat /etc/ipnat.rules
# www (was 69.55.230.12)
# virt19
#bimap fxp0 10.1.4.209/32 -> 69.55.233.198/32
# virt18
#bimap fxp0 10.1.4.208/32 -> 69.55.233.196/32
# virt13
#bimap fxp0 10.1.4.213/32 -> 69.55.233.196/32
# virt12
#bimap fxp0 10.1.4.212/32 -> 69.55.233.196/32
# virt17
bimap fxp0 10.1.4.217/32 -> 69.55.233.196/32
# virt11
#bimap fxp0 10.1.4.211/32 -> 69.55.233.196/32
# ASA
#bimap fxp0 10.1.4.172/32 -> 69.55.233.196/32
# P1A
bimap fxp0 10.1.4.240/32 -> 69.55.233.197/32
#bimap fxp0 10.1.4.238/32 -> 69.55.233.197/32
# developer (was 69.55.230.17)
# jail2
#bimap fxp0 10.1.4.232/32 -> 69.55.233.198/32
# jail8
#bimap fxp0 10.1.4.238/32 -> 69.55.233.198/32
# jail9
#bimap fxp0 10.1.4.239/32 -> 69.55.233.198/32
# POLL
#BIMAP EM0 10.1.6.134/32 -> 69.55.230.20/32
# 1U SUN
#BIMAP EM0 10.1.4.4/32 -> 69.55.227.46/32
# ??
#BIMAP EM0 10.1.6.3/32 -> 69.55.230.100/32
# random machine
#bimap fxp0 10.1.6.13/32 -> 69.55.233.199/32
#bimap fxp0 10.1.4.232/32 -> 69.55.233.199/32
# OFFICE OUTBOUND TRAFFIC
#map fxp0 10.1.6.0/24 -> 0.0.0.0/32 proxy port ftp ftp/tcp
#map fxp0 10.1.6.0/24 -> 0.0.0.0/32</pre>

A simple entry looks like:
bimap fxp0 10.1.4.240/32 -> 69.55.233.197/32
Which essentially means make private IP 10.1.4.240 reachable on 69.55.233.197 and allow 10.1.4.240 to communicate with the public internet via 69.55.233.197

To reload new rule config:
ipnat -C -F -f /etc/ipnat.rules

You may want to setup natting, as above, when you need to reach a DRAC card's web interface, wherin the DRAC card only has a private IP.

= nat2 =
== Summary ==
This is the main machine to which we ssh and runs all our screen sessions at i2b, and runs ns3c (this is kind of the what mail is to castle). Further, it's ip runs in IP space provided by i2b: 66.181.18.1 - 66.181.18.30, which is not routed through the firewall and this is somewhat immune to DoS attacks which hobble our firewall. Lastly, it acts as a nat server for certain/random devices on the private network.

* Location: i2b, cab 6
* OS: FreeBSD 6.4 x86
* Networking: Priv IP: 10.1.2.1, Pub IPs: 69.55.229.2, 69.55.229.3, 66.181.18.4, 66.181.18.5, 66.181.18.6, 66.181.18.7, 66.181.18.8, 66.181.18.9, 66.181.18.10, 66.181.18.11, 66.181.18.12, 66.181.18.13, 66.181.18.14 1 onboard and 1 PCI
* Hardware: Custom 2U. 6 drive bays, non-hot-swappable. single power supply.
* Drives: one 150 GB (2 x 150GB) RAID1 array running on a 3ware 8006 RAID card.

== Services Provided ==
* nat
* bigbrother
* ns3c (jail)
* ntp

== nat config ==
Here's what's currently nat'd on nat2:
<pre>cat /etc/ipnat.rules
# sample entry
#ATS-9
bimap em0 10.1.2.79/32 -> 66.181.18.14/32
#ATS-8
bimap em0 10.1.2.78/32 -> 66.181.18.13/32
#ATS-7
bimap em0 10.1.2.77/32 -> 66.181.18.12/32
#ATS-6
bimap em0 10.1.2.76/32 -> 66.181.18.6/32
#ATS-5
bimap em0 10.1.2.75/32 -> 66.181.18.7/32
#ATS-4
bimap em0 10.1.2.74/32 -> 66.181.18.8/32
#ATS-3
bimap em0 10.1.2.73/32 -> 66.181.18.9/32
#ATS-2
bimap em0 10.1.2.72/32 -> 66.181.18.10/32
#ATS-1
bimap em0 10.1.2.71/32 -> 66.181.18.11/32
#bwdb2
bimap em0 10.1.2.4/32 -> 66.181.18.5/32

# spare

map em0 10.1.2.0/24 -> 0.0.0.0/32 proxy port ftp ftp/tcp

#bimap fxp0 10.1.6.49/32 -> 10.1.1.2/32
#bimap fxp0 10.1.4.208/32 -> 69.55.233.196/32
</pre>

== build ==

* partition map:
<pre>/ 512m
swap 1G
/var 256m
/tmp 256m
/usr 5g
/mnt/data1 ~</pre>

* edit /etc/make.conf
<pre>echo "WITHOUT_X11=yes \
KERNCONF=nat2 \
BOOT_COMCONSOLE_SPEED=115200" >> /etc/make.conf</pre>

* add settings to /boot/loader.conf and /boot.config

<pre>echo "-Dh" >> /boot.config

echo 'console="comconsole,vidconsole" \
boot_multicons="YES" \
boot_serial="YES" \
comconsole_speed="115200"' >> /boot/loader.conf</pre>

* turn off all ttyv's except 0 and 1 in /etc/ttys
also turn on ttyd0, change type to vt100:
<pre>vi /etc/ttys

ttyv2 "/usr/libexec/getty Pc" cons25 off secure
ttyv3 "/usr/libexec/getty Pc" cons25 off secure
ttyv4 "/usr/libexec/getty Pc" cons25 off secure
ttyv5 "/usr/libexec/getty Pc" cons25 off secure
ttyv6 "/usr/libexec/getty Pc" cons25 off secure
ttyv7 "/usr/libexec/getty Pc" cons25 off secure
# Serial terminals
# The 'dialup' keyword identifies dialin lines to login, fingerd etc.
ttyd0 "/usr/libexec/getty std.9600" vt100 on secure

kill -1 1</pre>

on console server:
vi /etc/remote
(rename port to jail8 depending on where and which digi plugged into)
test serial console

* populate hosts
<pre>echo "69.55.230.10 backup2" >> /etc/hosts
echo "69.55.230.11 backup1" >> /etc/hosts
echo "10.1.2.4 bwdb2" >> /etc/hosts
echo "10.1.2.3 backup3" >> /etc/hosts</pre>

* put key in authorized_keys on backup3
<pre>cd
ssh-keygen -t dsa -b 1024</pre>
(default location, leave password blank)

<pre>cat /root/.ssh/id_dsa.pub | ssh backup3 'cat - >> /root/.ssh/authorized_keys'
cat /root/.ssh/id_dsa.pub | ssh backup2 'cat - >> /root/.ssh/authorized_keys'
cat /root/.ssh/id_dsa.pub | ssh backup1 'cat - >> /root/.ssh/authorized_keys'</pre>

confirm that you can ssh to backup3 and backup 2 without getting a login prompt

ssh backup3 hostname

ssh backup2 hostname

ssh backup1 hostname

* edit root's path and login script:
vi /root/.cshrc

Change alias entries (add G):
<pre>alias la ls -aG
alias lf ls -FAG
alias ll ls -lAG
alias ls ls -AG
alias mbm mb mount
alias mbu mb umount</pre>

and alter the prompt, set the following:
set prompt = "`/bin/hostname -s` %/# "

* install cvsup
<pre>cd /usr/ports/net/cvsup-without-gui
make install clean; rehash; mail -s 'cvs installed' dave.boodman@vtext.com < /dev/null</pre>

* get latest sources for this release:
<pre>cd /usr/src
echo "*default host=cvsup4.freebsd.org\
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_6_4\
*default delete use-rel-suffix\
*default compress\
src-all" > sup

cvsup sup ; mail -s 'cvs sup done' dave.boodman@vtext.com < /dev/null</pre>

* configure new kernel.

cd /usr/src/sys/i386/conf
scp backup2:/mnt/data4/build/freebsd/nat2-6.4 ./nat2

* build, install kernel and world

<pre>cd /boot

mv kernel kernel.GENERIC
cd kernel.GENERIC
cd /usr/src
make buildkernel installkernel

make buildworld ; mail -s 'buildworld done' support@johncompanies.com < /dev/null
make installworld
mergemaster -i
</pre>

* populate /etc/rc.conf with IPs and NFS settings
<pre>vi /etc/rc.conf

hostname="nat2.johncompanies.com"
kern_securelevel_enable="NO"
portmap_enable="NO"
sendmail_enable="NO"
usbd_enable="YES"

xntpd_enable="YES"
xntpd_flags="-A -p /var/run/ntpd.pid"

nfs_client_enable="YES"
nfs_reserved_port_only="YES"
ifconfig_em0="inet 10.1.6.50 netmask 255.255.255.0"
#ifconfig_em0="inet 69.55.229.2 netmask 255.255.255.0"
#ifconfig_em0_alias0="inet 69.55.229.229 netmask 255.255.255.255"
ifconfig_fxp0="inet 69.55.229.2 netmask 255.255.255.0"
ifconfig_fxp0_alias0="inet 69.55.229.3 netmask 255.255.255.255"
ifconfig_fxp1="inet 10.1.2.1 netmask 255.255.255.0"
defaultrouter="10.1.6.1"
#defaultrouter=" 66.181.14.250"
snmpd_enable="YES"
ipnat_enable="YES"
ipnat_rules="/etc/ipnat.rules"
gateway_enable="YES"

inetd_enable="YES"
inetd_flags="-wW -a 10.1.2.1"
fsck_y_enable="YES"
background_fsck="NO"
sshd_enable="YES"</pre>

* reboot. Confirm new kernel is loaded

uname -a

* update ports:
<pre>cd /usr/ports
echo "*default host=cvsup4.FreeBSD.org\
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_6_4\
*default delete use-rel-suffix\
*default compress\
ports-all tag=." > sup

cvsup sup; mail -s 'cvs sup ports done' dave.boodman@vtext.com < /dev/null</pre>

* Install raid mgmt tool

<pre>cd /usr/local/sbin
fetch http://3ware.com/download/Escalade9690SA-Series/9.5.3/tw_cli-freebsd-x86-9.5.3.tgz
tar xzf tw_cli-freebsd-x86-9.5.3.tgz
rm tw_cli-freebsd-x86-9.5.3.tgz
chmod 0700 tw_cli</pre>

Test:
./tw_cli info c0

* install rsync from ports
<pre>cd /usr/ports/net/rsync
make install clean</pre>
choose default options

* install perl from ports
<pre>cd /usr/ports/lang/perl5.8
make install clean</pre>

* install screen from ports
<pre>cd /usr/ports/sysutils/screen
make install clean</pre>

* install bb client
<pre>adduser
Username: bb
Full name: bb
Uid (Leave empty for default): 1984
Login group [bb]:
Login group is bb. Invite bb into other groups? []:
Login class [default]:
Shell (sh csh tcsh nologin) [sh]:
Home directory [/home/bb]:
Use password-based authentication? [yes]:
Use an empty password? (yes/no) [no]:
Use a random password? (yes/no) [no]: yes
Lock out the account after creation? [no]:
Username : bb
Password : <random>
Full Name : bb
Uid : 1984
Class :
Groups : bb
Home : /home/bb
Shell : /bin/sh
Locked : no
OK? (yes/no): yes

cd /usr/home/bb
scp backup2:/mnt/data4/build/bb/bb-freebsd.tar .
tar xvf bb-freebsd.tar</pre>
edit /home/bb/bbc1.9e-btf/etc/bb-hosts with something like:
<pre>echo "69.55.230.2 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh \
10.1.2.1 nat2.johncompanies.com # ssh" > /home/bb/bbc1.9e-btf/etc/bb-hosts
vi /home/bb/bbc1.9e-btf/ext/openfiles
MACHINE="nat2,johncompanies,com" # HAS TO BE IN A,B,C FORM

cd /usr/home/bb/bbc1.9e-btf/etc
./bbchkcfg.sh
(y to questions)
./bbchkhosts.sh
(ignore ssh errors)
cd ../..
chown -R bb .
su bb
cd
cd bbc1.9e-btf/src
make; make install
cd ..

vi /usr/home/bb/bbc1.9e-btf/bin/bb-cpu.sh
$1 $TOPARGS > $BBTMP/TOP.$$
# /usr/local/jail/bin/jtop > $BBTMP/TOP.$$

./runbb.sh start
more BBOUT
(look for errors)
exit

echo 'su - bb -c "cd /home/bb/bbc1.9e-btf; ./runbb.sh start"' > /usr/local/etc/rc.d/bb.sh
chmod +x /usr/local/etc/rc.d/bb.sh
</pre>

Punch a hole in the firewall to allow it to communicate with bb monitor (probably already exists):

ipfw add 96 allow ip from 66.181.18.0/27 to 69.55.230.2

* configure bb on mail:
<pre>vi /usr/home/bb/bbsrc/bb1.9e-btf/etc/bb-hosts
64.163.14.48 nat2.johncompanies.com # ssh

su bb
cd
bbsrc/bb/runbb.sh restart ; exit</pre>

* configure ntp
<pre>echo "server 69.55.230.2
server 0.pool.ntp.org
server 1.pool.ntp.org
server 2.pool.ntp.org
server 3.pool.ntp.org
server 66.187.233.4
server 217.204.76.170
server 64.112.189.11
server 66.69.112.130
server 80.85.129.25
server 80.237.234.15
server 130.60.7.44
server 134.99.176.3
server 198.144.202.250
server 202.74.170.194
server 204.17.42.199
server 204.87.183.6
server 213.15.3.1
server 213.239.178.33
server 217.114.97.97
server 69.55.230.2" > /etc/ntp.conf</pre>

<pre>/usr/sbin/ntpd -A -p /var/run/ntpd.pid
sleep 2; ntpq -p</pre>
(confirm it’s able to reach our time server)

echo '/usr/sbin/ntpd -c /etc/ntp.conf -p /var/run/ntpd.pid -f /var/db/ntpd.drift' > /usr/local/etc/rc.d/ntp.sh
chmod 0700 /usr/local/etc/rc.d/ntp.sh

* fwd and reverse lookups on ns1c
vr johncompanies.com
(edit the PTR too)

* setup backups, nfs mount

<pre>mkdir /backup3
echo 'backup3:/data /backup3 nfs rw,bg 0 0' >> /etc/fstab

echo '#\!/bin/sh\
backupdir=/data/nat2/current\
\
## ENTRY /etc ' > /usr/local/etc/backup.config</pre>

on backup3:
setup backup dirs:
ssh backup3 mkdir -p /data/nat2/current

on backup3, add the system to
vi /usr/local/sbin/snapshot_archive

<pre>scp 69.55.230.2:/usr/local/etc/rsync.backup /usr/local/etc/rsync.backup
vi /usr/local/etc/rsync.backup
backup1 > backup3</pre>

crontab -e
1 0 * * * /usr/local/etc/rsync.backup

* edit sshd_config for security
<pre>vi /etc/ssh/sshd_config
ListenAddress 66.181.18.1
ListenAddress 69.55.229.2
ListenAddress 10.1.2.1

kill -1 `cat /var/run/sshd.pid`</pre>

* raid chk

<pre>cat > /usr/local/sbin/lsiraidchk
#!/usr/bin/perl

my @out = split "\n", `megarc -ldInfo -a0 -Lall|grep Status:`;

foreach (@out) {
if ($_ =~ /DEGRADED/) { $date = `date`; chomp $date; `echo "$date: RAID ARRAY DEGRADED" >> /var/log/messages`; }
#print $_;
}</pre>

* netflow stuff
add crontab entries
<pre>crontab -e
30 3 * * * /usr/local/etc/rsync.backup
0 0 1 * * /sbin/ipfw zero
0 0 1 * * /sbin/ipfw del 3 4 5
59 23 30 * * /sbin/ipfw show > /tmp/ipfw_count
3 0 30 * * /sbin/ipfw show > /tmp/ipfw_count
0 3 * * * /etc/makefwrules.pl; /etc/makepiperules.pl;
*/5 * * * * /usr/local/sbin/lsiraidchk

#10 0 * * * rm /var/spool/clientmqueue/*</pre>

<pre>scp /etc/makefwrules.pl user@64.163.14.48:~
scp /etc/makepiperules.pl user@64.163.14.48:~
mv /home/user/makefwrules.pl /etc
mv /home/user/makepiperules.pl /etc
touch /etc/firewall.sh
mkdir /etc/oldrules/</pre>

other binaries

<pre>scp /usr/local/bin/rulemaker user@64.163.14.48:~
mv ~user/rulemaker /usr/local/sbin
scp ~user/Sendmail.pm user@64.163.14.48:~
scp ~user/doswatch.pl user@64.163.14.48:~</pre>

* add nat rules
<pre>vi /etc/ipnat.rules
# sample entry
bimap fxp0 10.1.6.70/32 -> 10.1.6.59/32
#bimap fxp0 10.1.4.208/32 -> 69.55.233.196/32

ipnat -C -f /etc/ipnat.rules</pre>

* shell for user
<pre>cp /root/.cshrc ~user/
vi ~user/</pre>
change # to $

* mrtg

<pre>cd /usr/ports/net-mgmt/mrtg
make install clean</pre>
(no FONTCONFIG, v3)

this didn't work cause of libtool incompat

so manually moved files:

<pre>scp /usr/local/bin/cfgmaker user@nat2:/usr/local/bin/cfgmaker
scp /usr/local/lib/perl5/site_perl/5.6.1/MRTG_lib.pm root@nat2:/usr/local/lib/perl5/site_perl/5.8.9/
scp /usr/local/lib/perl5/site_perl/5.6.1/SNMP_util.pm root@nat2:/usr/local/lib/perl5/site_perl/5.8.9/
scp /usr/local/lib/perl5/site_perl/5.6.1/BER.pm root@nat2:/usr/local/lib/perl5/site_perl/5.8.9/
scp /usr/local/lib/perl5/site_perl/5.6.1/SNMP_Session.pm root@nat2:/usr/local/lib/perl5/site_perl/5.8.9/
scp /usr/local/bin/mrtg root@nat2:/usr/local/bin/mrtg
scp /usr/local/lib/perl5/site_perl/5.6.1/locales_mrtg.pm root@nat2:/usr/local/lib/perl5/site_perl/5.8.9/locales_mrtg.pm
scp /usr/local/bin/rrdtool root@nat2:/usr/local/bin/rrdtool
scp /usr/local/lib/perl5/site_perl/5.6.1/mach/RRDs.pm root@nat2:/usr/local/lib/perl5/site_perl/5.8.9/mach/RRDs.pm
rsync -av /usr/local/lib/perl5/site_perl/5.6.1/mach/auto/RRDs/ root@nat2:/usr/local/lib/perl5/site_perl/5.8.9/mach/auto/RRDs/
scp /usr/lib/libz.so.2 root@nat2:/usr/lib/libz.so.2
scp /usr/lib/libm.so.2 root@nat2:/usr/lib/libm.so.2
rsync -av /usr/local/lib/librrd* root@nat2:/usr/local/lib/
scp /usr/lib/libc.so.4 root@nat2:/usr/lib/libc.so.4

rsync -av /usr/ports/net/rrdtool root@nat2:/usr/ports/net
cd /usr/ports/net/rrdtool
make install

mkdir -p /mnt/data1/mrtg/data
scp /usr/local/www/mgmt/mrtg/template.pl root@nat2:/mnt/data1/mrtg/
scp /usr/local/www/mgmt/mrtg/host.pl root@nat2:/mnt/data1/mrtg/

cfgmaker --if-template=template.pl --show-op-down --global "options[_]: growright,bits" --global 'WorkDir: /mnt/data1/mrtg/data' --global 'Interval: 1' --global 'LogFormat: rrdtool' --global 'PathAdd: /usr/local/bin' --global 'LibAdd: /usr/local/lib' --host-template=host.pl jc292401@10.1.2.50 --output=switch-p20.cfg

cat > /mnt/data1/mrtg/mrtg.sh
#!/bin/sh
/usr/local/bin/mrtg /mnt/data1/mrtg/switch-p20.cfg

chmod 0700 /mnt/data1/mrtg/mrtg.sh

crontab -e
* * * * * /mnt/data1/mrtg/mrtg.sh 2>&1 > /dev/null</pre>

* snmp firewall block
<pre>cat > /usr/local/etc/rc.d/boot.sh
ipfw add 10 allow udp from 69.55.230.2 to any 161
ipfw add 10 allow udp from 10.1.2.1 to any 161
ipfw add 11 deny udp from any to any 161
chmod 0700 /usr/local/etc/rc.d/boot.sh</pre>

= bwdb =
== Summary ==
This machine tracks and stores network traffic (netflow) at castle. It is our means to monitor customer bandwidth usage.

* Location: castle, cab 3-7
* OS: FreeBSD 4.10 x86
* Networking: Priv IP: 10.1.4.203 There are 2 onboard nic's, one of which is the "listener"
* Hardware: Custom 1U. Single power supply.
* Drives: one 250 GB (2 x 250GB) RAID1 array running on a Promise IDE RAID card.

== Services Provided ==
* netflow
* mysql
* bigbrother
* snmp

== netflow ==

The main function of this server is to run netflow on an eth device in promiscuous mode so as to hear everything happening on the port (wherein all network traffic is mirrored to that port via the cisco swith). Every 15min, it creates a flow file under <tt>/usr/home/flows/</tt> (organized by date). The flow file contains all traffic data for a 15min increment of time.

A cronjob moves that flow file (or files if there are multiple due to some delay)
1,16,31,46 * * * * /usr/home/flowbin/queue.pl

into a processing queue:
<tt>/usr/home/working</tt>

Then a separate file processes whatever flow files it finds there, inserting the data into the local mysql database:
2,17,32,47 * * * * /usr/home/flowbin/processflows.pl

== mysql ==

The database storing all the traffic data is named <tt>traffic</tt>
Tables:
<pre>mysql> show tables;
+---------------------------+
| Tables_in_traffic |
+---------------------------+
| dailyIpTotals_69_55_224 |
| dailyIpTotals_69_55_225 |
| dailyIpTotals_69_55_226 |
| dailyIpTotals_69_55_227 |
| dailyIpTotals_69_55_228 |
| dailyIpTotals_69_55_229 |
| dailyIpTotals_69_55_230 |
| dailyIpTotals_69_55_231 |
| dailyIpTotals_69_55_232 |
| dailyIpTotals_69_55_233 |
| dailyIpTotals_69_55_234 |
| dailyIpTotals_69_55_235 |
| dailyIpTotals_69_55_236 |
| dailyIpTotals_69_55_237 |
| dailyIpTotals_69_55_238 |
| dailyIpTotals_69_55_239 |
| dailyPortTotals_69_55_224 |
| dailyPortTotals_69_55_225 |
| dailyPortTotals_69_55_226 |
| dailyPortTotals_69_55_227 |
| dailyPortTotals_69_55_228 |
| dailyPortTotals_69_55_229 |
| dailyPortTotals_69_55_230 |
| dailyPortTotals_69_55_231 |
| dailyPortTotals_69_55_232 |
| dailyPortTotals_69_55_233 |
| dailyPortTotals_69_55_234 |
| dailyPortTotals_69_55_235 |
| dailyPortTotals_69_55_236 |
| dailyPortTotals_69_55_237 |
| dailyPortTotals_69_55_238 |
| dailyPortTotals_69_55_239 |
| ipTotals_69_55_224 |
| ipTotals_69_55_225 |
| ipTotals_69_55_226 |
| ipTotals_69_55_227 |
| ipTotals_69_55_228 |
| ipTotals_69_55_229 |
| ipTotals_69_55_230 |
| ipTotals_69_55_231 |
| ipTotals_69_55_232 |
| ipTotals_69_55_233 |
| ipTotals_69_55_234 |
| ipTotals_69_55_235 |
| ipTotals_69_55_236 |
| ipTotals_69_55_237 |
| ipTotals_69_55_238 |
| ipTotals_69_55_239 |
| portTotals_69_55_224 |
| portTotals_69_55_225 |
| portTotals_69_55_226 |
| portTotals_69_55_227 |
| portTotals_69_55_228 |
| portTotals_69_55_229 |
| portTotals_69_55_230 |
| portTotals_69_55_231 |
| portTotals_69_55_232 |
| portTotals_69_55_233 |
| portTotals_69_55_234 |
| portTotals_69_55_235 |
| portTotals_69_55_236 |
| portTotals_69_55_237 |
| portTotals_69_55_238 |
| portTotals_69_55_239 |
+---------------------------+
</pre>

So as you see we store each class-C block in its own table, for efficiency. Further, we store and organize data in 4 ways: "daily" tables and 15-minute granularity tables, and for each of those we track simple IP traffic and port-specific traffic. The daily tables contains 2 entries (one for each direction) for each IP for each day. For the current day, the row data is incremented as the day goes on.

<pre>mysql> describe dailyIpTotals_69_55_224;
+-----------+-------------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+-----------+-------------+------+-----+---------+-------+
| id | varchar(23) | | PRI | | |
| date | date | YES | | NULL | |
| ip | varchar(15) | YES | MUL | NULL | |
| direction | tinyint(1) | YES | | NULL | |
| octets | bigint(12) | YES | | NULL | |
| packets | int(11) | YES | | NULL | |
+-----------+-------------+------+-----+---------+-------+

mysql> select * from dailyIpTotals_69_55_224 limit 1\G
*************************** 1. row ***************************
id: 6955224194-20100917-1
date: 2010-09-17
ip: 69.55.224.194
direction: 1
octets: 8821
packets: 91
</pre>

The <tt>id</tt> is a unique identifier (key), <tt>direction</tt> indicates incoming or outgoing traffic (outbound = 2, inbound = 1), <tt>octets</tt> are the amount of traffic in kilobytes, and <tt>packets</tt> is the total number of packets.

The 15-minute table has similar information, but it's organized in 15 minute increments:

<pre>mysql> describe ipTotals_69_55_224;
+-----------+------------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+-----------+------------+------+-----+---------+-------+
| date | datetime | YES | | NULL | |
| ip | char(15) | YES | MUL | NULL | |
| direction | tinyint(1) | YES | | NULL | |
| octets | bigint(20) | YES | | NULL | |
| packets | int(11) | YES | | NULL | |
+-----------+------------+------+-----+---------+-------+

mysql> select * from ipTotals_69_55_224 limit 2\G
*************************** 1. row ***************************
date: 2010-01-11 19:30:00
ip: 69.55.224.13
direction: 1
octets: 288
packets: 6
*************************** 2. row ***************************
date: 2010-01-11 19:30:00
ip: 69.55.224.12
direction: 1
octets: 216
packets: 4</pre>

So for a given IP, there will be 192 rows in a given day: 4 rows per hour, *2 for 2 directions, *24 for 24hours in a day. Obviously this table is large which is why we broke it down into a daily table for quick, easy, daily-summary access.

That covers the simple traffic tabulation tables. We also track traffic by port:

<pre>mysql> describe dailyPortTotals_69_55_224;
+-----------+-------------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+-----------+-------------+------+-----+---------+-------+
| id | varchar(28) | | PRI | | |
| date | date | YES | | NULL | |
| ip | varchar(15) | YES | MUL | NULL | |
| direction | tinyint(1) | YES | | NULL | |
| protocol | smallint(3) | YES | | NULL | |
| port | int(11) | YES | | NULL | |
| octets | bigint(11) | YES | | NULL | |
| packets | int(11) | YES | | NULL | |
+-----------+-------------+------+-----+---------+-------+
8 rows in set (0.00 sec)

mysql> select * from dailyPortTotals_69_55_224 limit 1\G
*************************** 1. row ***************************
id: 695522496-20091218-1-6-23
date: 2009-12-18
ip: 69.55.224.96
direction: 1
protocol: 6
port: 23
octets: 1796
packets: 30

mysql> select * from portTotals_69_55_224 limit 1\G
*************************** 1. row ***************************
date: 2010-09-07 18:45:00
ip: 69.55.224.254
direction: 1
protocol: 6
port: 99999
octets: 144
packets: 3

</pre>

This is largely the same with 2 more additions: <tt>protocol</tt> (1=ICMP, 6=TCP, 17=UDP), and <tt>port</tt> which we set to 99999 if the traffic is return traffic and the port is above 1024. Obviously the potential for number of rows grows quickly when you consider the addition of port and protocol tracking per IP.

== Regular maintenance ==
*[[Routine_Maintenance#Adaptec_Controllers|Check RAID array]]
* archive data from database
archive_daily.pl 2012 09
This will archive data for the given year and month from the daily summary tables. Generally we want to have a year of history in the database.

archive_15min.pl 2012 09
This will archive data for the given year and month from the 15min-increment tables. Generally, we want to have 6 months of history in the database.

* if space becomes tight, move flow files and exported data to a backup server, both located in <tt>/usr/home/flowbin/archive</tt> and <tt>/usr/home/exported</tt>, respectively

== Slaving ==

If we were going to setup traffic database slaving (we don't do this anymore), perhaps cause the bwdb machine gets busy and it cannot handle traffic requests and netflow, here's how it's done:

On the traffic master:

GRANT REPLICATION SLAVE ON *.* TO 'repl'@'10.1.4.8' IDENTIFIED BY 'qERUG8wf';

in my.cnf:
<pre>bin-log
server-id=1
max_binlog_size=500M
expire_logs_days = 3</pre>

on slave:
in my.cnf:
<pre>server-id = 2
master-host = 10.1.4.203
master-user = repl
master-password = qERUG8wf
master-connect-retry=60
replicate-wild-do-table=traffic.daily%
max_relay_log_size=500M
expire_logs_days = 3

replicate-wild-do-table=traffic.%</pre>

on master:
touch /usr/home/working/.lock
(make sure processflows not running)

<pre>FLUSH TABLES WITH READ LOCK;
cd /usr/home/database/traffic
tar -czf mysql-traffic-snapshot.tgz ./daily*
(~1G)
SHOW MASTER STATUS;
+-----------------+-----------+--------------+------------------+
| File | Position | Binlog_Do_DB | Binlog_Ignore_DB |
+-----------------+-----------+--------------+------------------+
| bwdb-bin.000039 | 154432615 | | |
+-----------------+-----------+--------------+------------------+

(write down info)
UNLOCK TABLES;
scp mysql-traffic-snapshot.tgz 10.1.4.5:/mnt/data1/db/mysql/traffic/</pre>

on slave:
<pre>mkdir /mnt/data1/db/mysql/traffic
cd /mnt/data1/db/mysql/traffic/
tar xzvf mysql-traffic-snapshot.tgz
(restart mysql)
CHANGE MASTER TO MASTER_HOST='10.1.4.203',MASTER_USER='repl',MASTER_PASSWORD='qERUG8wf',MASTER_LOG_FILE='bwdb-bin.000059',MASTER_LOG_POS=482502186;
START SLAVE;</pre>

<pre>cd /usr/home/database/traffic
scp *</pre>

<pre>optimize table dailyPortTotals_69_55_224;
optimize table dailyPortTotals_69_55_225;
optimize table dailyPortTotals_69_55_226;
optimize table dailyPortTotals_69_55_227;
optimize table dailyPortTotals_69_55_228;
optimize table dailyPortTotals_69_55_229;
optimize table dailyPortTotals_69_55_230;
optimize table dailyPortTotals_69_55_231;
optimize table dailyPortTotals_69_55_232;
optimize table dailyPortTotals_69_55_233;
optimize table dailyPortTotals_69_55_234;
optimize table dailyPortTotals_69_55_235;
optimize table dailyPortTotals_69_55_236;
optimize table dailyPortTotals_69_55_237;
optimize table dailyPortTotals_69_55_238;
optimize table dailyPortTotals_69_55_239;</pre>

== Build ==

=== BIOS Config ===
disable quiet boot

set to last state after power loss

set date/time to GMT

enable serial console output (baud rate 115200)

=== Install OS ===

Install FreeBSD 8.3 amd64

* partition map:
<pre>/ 500m
swap 4096m
/var 256m
/tmp 256m
/usr ~</pre>

* edit /etc/make.conf
Castle:
<pre>echo "WITHOUT_X11=yes \
KERNCONF=bwdb \
BOOT_COMCONSOLE_SPEED=115200" >> /etc/make.conf</pre>

i2b:
<pre>echo "WITHOUT_X11=yes \
KERNCONF=bwdb2 \
BOOT_COMCONSOLE_SPEED=115200" >> /etc/make.conf</pre>

* add settings to /boot/loader.conf and /boot.config

<pre>echo "-Dh" >> /boot.config

echo 'console="comconsole,vidconsole" \
boot_multicons="YES" \
boot_serial="YES" \
comconsole_speed="115200"' >> /boot/loader.conf</pre>

* turn off all ttyv's except 0 and 1 in /etc/ttys
also turn on ttyu0, change type to vt100:
<pre>vi /etc/ttys

ttyv2 "/usr/libexec/getty Pc" cons25 off secure
ttyv3 "/usr/libexec/getty Pc" cons25 off secure
ttyv4 "/usr/libexec/getty Pc" cons25 off secure
ttyv5 "/usr/libexec/getty Pc" cons25 off secure
ttyv6 "/usr/libexec/getty Pc" cons25 off secure
ttyv7 "/usr/libexec/getty Pc" cons25 off secure
# Serial terminals
# The 'dialup' keyword identifies dialin lines to login, fingerd etc.
ttyu0 "/usr/libexec/getty std.9600" vt100 on secure

kill -1 1</pre>

on console server:
vi /etc/remote
(rename port to jail8 depending on where and which digi plugged into)
test serial console

* populate hosts
i2b:
<pre>echo "69.55.230.10 backup2" >> /etc/hosts
echo "69.55.230.11 backup1" >> /etc/hosts
echo "10.1.2.3 backup3" >> /etc/hosts</pre>

castle:
<pre>echo "10.1.4.3 backup2 backup2.johncompanies.com" >> /etc/hosts
echo "10.1.4.8 backup1 backup1.johncompanies.com" >> /etc/hosts
echo "10.1.4.4 mail mail.johncompanies.com" >> /etc/hosts
</pre>

* put key in authorized_keys on backup1 and backup2
cd
ssh-keygen -t dsa -b 1024
(default location, leave password blank)

castle:
cat /root/.ssh/id_dsa.pub | ssh backup1 'cat - >> /root/.ssh/authorized_keys'
cat /root/.ssh/id_dsa.pub | ssh backup2 'cat - >> /root/.ssh/authorized_keys'

i2b:
cat /root/.ssh/id_dsa.pub | ssh backup1 'cat - >> /root/.ssh/authorized_keys'
cat /root/.ssh/id_dsa.pub | ssh backup2 'cat - >> /root/.ssh/authorized_keys'
cat /root/.ssh/id_dsa.pub | ssh backup3 'cat - >> /root/.ssh/authorized_keys'

confirm that you can ssh to backup2 and backup1 (and backup3 if at i2b) without getting a login prompt

ssh backup1 hostname
ssh backup2 hostname

* edit root's path and login script:
vi /root/.cshrc

Change alias entries (add G):
<pre>alias la ls -aG
alias lf ls -FAG
alias ll ls -lAG
alias ls ls -AG
</pre>

and alter the prompt, set the following:
set prompt = "`/bin/hostname -s` %/# "

* install cvsup
cd /usr/ports/net/cvsup-without-gui
make install clean; rehash; mail -s 'cvs installed' support@johncompanies.com < /dev/null

* get latest sources for this release:
<pre>cd /usr/src
echo "*default host=cvsup4.freebsd.org\
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_8_3\
*default delete use-rel-suffix\
*default compress\
src-all" > sup

cvsup sup ; mail -s 'cvs sup done' support@johncompanies.com < /dev/null</pre>

* configure new kernel

cd /usr/src/sys/amd64/conf
scp backup2:/mnt/data4/build/freebsd/kern_config-bwdb-8.3-amd64 ./bwdb

Edit config and change name:
vi bwdb
ident bwdb

* build, install kernel and world

<pre>cd /boot

mv kernel kernel.GENERIC
cd kernel.GENERIC
cd /usr/src
make buildkernel installkernel

make buildworld ; mail -s 'buildworld done' support@johncompanies.com < /dev/null
(2450: 1:56min, supermicro: 59mins, 2950: 38mins)
make installworld
(2450: 3min, supermicro: 1min, 2950: :34)
mergemaster -i</pre>

* populate /etc/rc.conf with IPs and NFS settings

castle:
<pre>vi /etc/rc.conf

hostname="bwdb.johncompanies.com"
kern_securelevel_enable="NO"
portmap_enable="NO"
sendmail_enable="NO"
usbd_enable="YES"

xntpd_enable="YES"
nfs_client_enable="YES"
nfs_reserved_port_only="YES"
ifconfig_fxp0="inet 10.1.4.203 netmask 255.255.255.0"
ifconfig_em0="up promisc"
defaultrouter="10.1.4.1"
snmpd_enable="YES"

inetd_enable="YES"
inetd_flags="-wW -a 10.1.4.203"
fsck_y_enable="YES"
background_fsck="NO"
sshd_enable="YES"
ipfw_load="YES"</pre>

i2b:
<pre>vi /etc/rc.conf

hostname="bwdb2.johncompanies.com"
kern_securelevel_enable="NO"
portmap_enable="NO"
sendmail_enable="NO"
usbd_enable="YES"

xntpd_enable="YES"
nfs_client_enable="YES"
nfs_reserved_port_only="YES"
ifconfig_fxp0="inet 10.1.2.4 netmask 255.255.255.0"
ifconfig_em0="up promisc"
defaultrouter="10.1.2.1"
snmpd_enable="YES"

inetd_enable="YES"
inetd_flags="-wW -a 10.1.2.4"
fsck_y_enable="YES"
background_fsck="NO"
sshd_enable="YES"
ipfw_load="YES"</pre>

* reboot. Confirm new kernel is loaded

uname -a

* update ports:
<pre>cd /usr/ports
echo "*default host=cvsup4.FreeBSD.org\
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_8_3\
*default delete use-rel-suffix\
*default compress\
ports-all tag=." > sup

cvsup sup; mail -s 'cvs sup ports done' support@johncompanies.com < /dev/null</pre>

* Install raid mgmt tool

<pre>cd /usr/local/sbin
scp backup2:/d4/build/3ware/tw_cli-freebsd-x86_64-9.5.0.1.tgz .
tar xzf tw_cli-freebsd-x86_64-9.5.0.1.tgz
rm tw_cli-freebsd-x86_64-9.5.0.1.tgz
chmod 0700 tw_cli</pre>

Test:
./tw_cli info c0

Grab raid check script:
scp backup1:/usr/local/sbin/3wraidchk /usr/local/etc

Setup cronjob:
<pre>crontab -e
*/5 * * * * /usr/local/etc/3wraidchk</pre>

* install rsync from ports
cd /usr/ports/net/rsync
make install clean

choose default options

* install perl from ports
cd /usr/ports/lang/perl5.8
make install clean

choose default options

* install bb client

Compiling from source on AMD64 will not work. So, we use a linux-compiled version and rely on linux compat. Linux compat won't install on 8.x - libtool 2.4 need. So, instead we copy(ed) over linux:
rsync -aSHv --exclude=proc --exclude=sys 10.1.4.108:/usr/compat/linux/ /usr/compat/linux/

adduser

Output/response:

<pre>Username: bb
Full name: bb
Uid (Leave empty for default): 1984
Login group [bb]:
Login group is bb. Invite bb into other groups? []:
Login class [default]:
Shell (sh csh tcsh nologin) [sh]:
Home directory [/home/bb]:
Use password-based authentication? [yes]:
Use an empty password? (yes/no) [no]:
Use a random password? (yes/no) [no]: yes
Lock out the account after creation? [no]:
Username : bb
Password : <random>
Full Name : bb
Uid : 1984
Class :
Groups : bb
Home : /home/bb
Shell : /bin/sh
Locked : no
OK? (yes/no): yes</pre>

cd /usr/home/bb
scp backup2:/mnt/data4/build/bb/bb-freebsd_linuxcompat.tgz .
tar xzf bb-freebsd_linuxcompat.tgz

edit /home/bb/bbc1.9e-btf/etc/bb-hosts with something like:

echo "10.1.4.5 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh \
10.1.4.203 bwdb.johncompanies.com # ssh" > /home/bb/bbc1.9e-btf/etc/bb-hosts

Edit for machine name and private IP.

if this machine is at i2b:
echo "69.55.230.2 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh \
10.1.2.4 bwdb2.johncompanies.com # ssh" > /home/bb/bbc1.9e-btf/etc/bb-hosts

<pre>vi /home/bb/bbc1.9e-btf/ext/openfiles

MACHINE="bwdb,johncompanies,com" # HAS TO BE IN A,B,C FORM</pre>
Edit for machine name.

Have bb watch for flow-capture, mysql
<pre>cat >> /home/bb/bbc1.9e-btf/etc/bb-proctab
localhost: flow-capture :
localhost: mysqld :</pre>

<pre>cd /usr/home/bb/bbc1.9e-btf/etc
./bbchkcfg.sh
(y to questions)
./bbchkhosts.sh
(ignore ssh errors)
cd ../..
chown -R bb .
su bb
cd
cd bbc1.9e-btf
./runbb.sh start
more BBOUT
(look for errors)
exit</pre>

Put in script to start bb @ boot:
echo 'su - bb -c "cd /home/bb/bbc1.9e-btf; ./runbb.sh start"' > /usr/local/etc/rc.d/bb.sh
chmod +x /usr/local/etc/rc.d/bb.sh

If this is at i2b, punch a hole in the firewall to allow it to communicate with bb monitor:

ipfw add 00096 allow tcp from 66.181.18.0/27 to 69.55.230.2

* configure bb on mail
<pre>vi /usr/home/bb/bbsrc/bb1.9e-btf/etc/bb-hosts
10.1.4.203 bwdb.johncompanies.com # ssh

su bb
cd
bbsrc/bb/runbb.sh restart ; exit</pre>

* configure ntp server
Castle:
echo "server 10.1.4.1" > /etc/ntp.conf

I2b:
echo "server 10.1.2.1" > /etc/ntp.conf

<pre>/usr/sbin/ntpd -p /var/run/ntpd.pid
sleep 2; ntpq -p</pre>
(confirm it’s able to reach our time server)

<pre>echo '/usr/sbin/ntpd -c /etc/ntp.conf -p /var/run/ntpd.pid -f /var/db/ntpd.drift' > /usr/local/etc/rc.d/ntp.sh
chmod 0700 /usr/local/etc/rc.d/ntp.sh</pre>

* fwd and reverse lookups on ns1c
vr johncompanies.com
(edit the PTR too)

* setup backups
<pre>echo '#\!/bin/sh\
backupdir=/data/bwdb/current\
server=backup1\
\
## ENTRY /etc\
## ENTRY /usr/home/flowbin\
## ENTRY /usr/home/database' > /usr/local/etc/backup.config</pre>

Castle:
setup backup dirs:
ssh backup1 mkdir -p /data/bwdb/current
on backup1, add the system to
vi /usr/local/sbin/snapshot_rotate

I2b:
setup backup dirs:
ssh backup3 mkdir -p /data/bwdb/current
on backup3, add the system to
vi /usr/local/sbin/snapshot_archive

Copy over the backup script:
scp backup2:/d4/bin/freebsd8.x/rsync.backup /usr/local/etc/

Edit rsync.backup and change <tt>config</tt> var to point to correct config file location: <tt>/usr/local/etc/backup.config</tt>

<pre>crontab -e
5 0 * * * /usr/local/etc/rsync.backup</pre>

* make /root/logs
mkdir /root/logs

* edit sshd_config for security

<pre>vi /etc/ssh/sshd_config
ListenAddress 10.1.4.203
PermitRootLogin yes

kill -1 `cat /var/run/sshd.pid`</pre>

Edit for private IP.

* snmp

(Before doing this you may need to take down the firewall and also add to resolv.conf 69.43.143.41)
<pre>
cd /usr/ports/net-mgmt/net-snmp
make install clean
(defaults)

cat >> /etc/rc.conf
snmpd_enable="YES"
snmpd_flags="-a"
snmpd_conffile="/usr/local/share/snmp/snmpd.conf"
snmptrapd_enable="YES"
snmptrapd_flags="-a -p /var/run/snmptrapd.pid"

cat > /usr/local/share/snmp/snmpd.conf
rocommunity jcread 10.1.4.5
rocommunity jcread 10.1.4.202
</pre>

=== netflow ===

Install flow tools:
<pre>cd /usr/ports/net-mgmt/flow-tools
make install clean</pre>
Defaults.

mkdir /usr/home/flows

Flow start script:
echo "/usr/local/bin/flow-capture -w /usr/home/flows -S5 -N -2 0/10.1.4.203/4444" > /usr/local/etc/rc.d/flow-capture.sh

chmod 0700 /usr/local/etc/rc.d/flow-capture.sh

Edit for private IP.

Netgraph start script:
<pre>
cat > /usr/local/etc/rc.d/netgraph.sh

/usr/sbin/ngctl -f- <<-SEQ
mkpeer em0: netflow lower iface0
name em0:lower netflow
connect em0: netflow: upper out0
mkpeer netflow: ksocket export inet/dgram/udp
msg netflow:export connect inet/10.1.4.203:4444
SEQ

#/usr/sbin/ngctl -f- <<-SEQ
#shutdown netflow:
#SEQ

chmod 0700 /usr/local/etc/rc.d/netgraph.sh</pre>
Edit for private IP.

Confirm netflow is running after running scripts:
<pre>newbwdb /usr/ports/net-mgmt/flow-tools# /usr/sbin/ngctl
Available commands:
config get or set configuration of node at <path>
connect Connects hook <peerhook> of the node at <relpath> to <hook>
debug Get/set debugging verbosity level
dot Produce a GraphViz (.dot) of the entire netgraph.
help Show command summary or get more help on a specific command
list Show information about all nodes
mkpeer Create and connect a new node to the node at "path"
msg Send a netgraph control message to the node at "path"
name Assign name <name> to the node at <path>
read Read and execute commands from a file
rmhook Disconnect hook "hook" of the node at "path"
show Show information about the node at <path>
shutdown Shutdown the node at <path>
status Get human readable status information from the node at <path>
types Show information about all installed node types
write Send a data packet down the hook named by "hook".
quit Exit program
+ show netflow:
Name: netflow Type: netflow ID: 00000004 Num hooks: 3
Local hook Peer name Peer type Peer ID Peer hook
---------- --------- --------- ------- ---------
export <unnamed> ksocket 00000005 inet/dgram/udp
out0 em0 ether 00000001 upper
iface0 em0 ether 00000001 lower
+
</pre>

We notice that sometimes flow-capture is failing due to swap exhaustion (even after adding more swap). So we crontab flow-capture to restart (it's ok to start if it's already running, it just quits):

<pre>
crontab -e
#restart flow-capture
*/15 * * * * /usr/local/etc/rc.d/flow-capture.sh
</pre>

==== process flow tools ====

<pre>mkdir /usr/home/flowbin
mkdir /usr/home/working</pre>

Install modules:
<pre>cd /usr/ports/devel/p5-Date-Calc
make install clean
cd /usr/ports/mail/p5-Mail-Sendmail
make install clean</pre>

Queue script:
<pre>
cat > /usr/home/flowbin/queue.pl
#!/usr/bin/perl

use strict;

BEGIN {
push @INC, "/usr/home/flowbin";
}

use date;

my $flowbase = "/usr/home/flows";
#my $flowqueue = "/usr/home/queue";
my $flowqueue = "/usr/home/working";

my ($date, $time) = date::CurrentDateTime();

my $flowdir = mkFlowDir($date);
`mv $flowdir/ft-* $flowqueue`;

if (date::DateWindow($date, $time, $date, "00:00:00", 600)) {
my $newdate = date::AddDays($date, -1);
my $flowdir = mkFlowDir($newdate);
`mv $flowdir/ft-* $flowqueue`;
}

sub mkFlowDir {
my $date = shift;
$date =~ /([0-9]{4}-[0-9]{2})/;
my $yearmonth = $1;
return "$flowbase/$yearmonth/$date";
}
</pre>

Date.pm module:
<pre>
cat > /usr/home/flowbin/date.pm
#!/usr/local/bin/perl
#
# $Header: /usr/cvs/newgw/lib/date.pm,v 1.2 2003/11/24 17:06:02 glenn Exp $
#
# Copyright (c) 2001, 2002, 2003
# e-Monitoring Networks, Inc. All rights reserved.
#
#
#
# date.pl - Higher level functions written on top of Date::Calc

package date;

use strict;
use Date::Calc qw(:all);

sub DayDiff { #calculate the difference in days from two dates
my $date1 = shift;
my $date2 = shift;
my ($year1, $month1, $day1) = &DateToymd($date1);
my ($year2, $month2, $day2) = &DateToymd($date2);
my $diff = &Delta_Days($year1, $month1, $day1, $year2, $month2, $day2);
return $diff;
}

sub AddDays { #adds specified number of days to the supplied date
my $date = shift;
my $days = shift;
my ($year, $month, $day) = &DateToymd($date);
my ($nyear, $nmonth, $nday) = &Add_Delta_Days($year, $month, $day, $days);
my $ndate = &ymdToDate($nyear, $nmonth, $nday);
return $ndate;
}

sub AddHours { #adds specified number of hours to the supplied date and time
my $date = shift;
my $time = shift;
my $addhours = shift;
my $adddays = 0;
if (abs($addhours / 24) >= 1) {
$adddays = int($addhours / 24);
$addhours -= $adddays * 24;
}
my ($year, $month, $day) = &DateToymd($date);
my ($hour, $minute, $second) = &TimeTohms($time);
my ($ny, $nm, $nd, $nh, $nmin, $ns) = &Add_Delta_DHMS($year, $month, $day,
$hour, $minute, $second,
$adddays, $addhours, 0, 0);
my $ndate = &ymdToDate($ny, $nm, $nd);
my $ntime = &hmsToTime($nh, $nmin, $ns);
return $ndate, $ntime;
}

sub AddMinutes {
my $date = shift;
my $time = shift;
my $minutes = shift;
my ($year, $month, $day) = &DateToymd($date);
my ($hour, $minute, $second) = &TimeTohms($time);
my ($ny, $nm, $nd, $nh, $nmin, $ns) = &Add_Delta_DHMS($year, $month, $day,
$hour, $minute, $second,
0, 0, $minutes, 0);
my $ndate = &ymdToDate($ny, $nm, $nd);
my $ntime = &hmsToTime($nh, $nmin, $ns);
return $ndate, $ntime;
}

sub CurrentDateTime { #return the current date and time
my ($y, $m, $d, $h, $min, $s, $z, $z, $z) = &System_Clock;
my $date = &ymdToDate($y, $m, $d);
my $time = &hmsToTime($h, $min, $s);
return $date, $time;
}

sub Currentymd { #return the current year, month and day as separate variables
my ($y, $m, $d, $h, $min, $s, $z, $z, $z) = &System_Clock;
return $y, $m, $d;
}

sub DateToymd { #takes a date and returns year, month, day as individual values
my $date = shift;
if ($date =~ /([0-9]{4})-([0-9]{2})-([0-9]{2})/) {
my $day = $3;
my $month = $2;
my $year = $1;
return $year, $month, $day;
}
return undef;
}

sub TimeTohms { #takes a time and return hours minutes and seconds as individual values
my $time = shift;
if ($time =~ /([0-9]{1,2}):([0-9]{1,2}):([0-9]{1,2})/) {
my $hour = $1;
my $minute = $2;
my $second = $3;
if ($hour !~ /[0-9]{2}/) { $hour = "0$hour"; }
if ($minute !~ /[0-9]{2}/) { $minute = "0$minute"; }
if ($second !~ /[0-9]{2}/) { $second = "0$second"; }
return $hour, $minute, $second;
}
return undef;
}

sub ymdToDate { #takes year, month, day and assembles them into our date format
my $year = shift;
my $month = shift;
my $day = shift;
if (defined($year) && defined($month) && defined ($day)) {
$month = sprintf("%02d", $month);
$day = sprintf("%02d", $day);
return "$year-$month-$day";
}
return undef;
}

sub hmsToTime { #takes hour minute and second and assembles them into our time format
my $hour = shift;
my $minute = shift;
my $second = shift;
if (defined($hour) && defined($minute) && defined ($second)) {
if ($hour !~ /[0-9]{2}/) { $hour = "0$hour"; }
if ($minute !~ /[0-9]{2}/) { $minute = "0$minute"; }
if ($second !~ /[0-9]{2}/) { $second = "0$second"; }
return sprintf ("%02d:%02d:%02d", $hour, $minute, $second);
}
return undef;
}

sub CompareDates { #compares two date and time pairs
my $date1 = shift;
my $time1 = shift;
my $date2 = shift;
my $time2 = shift;

my ($year1, $month1, $day1) = &DateToymd($date1);
my ($hour1, $minute1, $second1) = &TimeTohms($time1);
my ($year2, $month2, $day2) = &DateToymd($date2);
my ($hour2, $minute2, $second2) = &TimeTohms($time2);

# &debug("$year1, $month1, $day1, $year2, $month2, $day2");
my $days = &Delta_Days($year1, $month1, $day1, $year2, $month2, $day2);
if ($days > 0) { return 1;}
if ($days < 0) { return -1;}
if ($days == 0) { #same day, compare times
my $seconds1 = $second1 + (60 * $minute1) + (3600 * $hour1);
my $seconds2 = $second2 + (60 * $minute2) + (3600 * $hour2);
if ($seconds1 < $seconds2) { return 1;}
if ($seconds1 > $seconds2) { return -1;}
if ($seconds1 == $seconds2) { return 0;}
}
return undef;
}

sub DateWindow { #compares two date time pairs to see if they are < X seconds apart
my $date1 = shift;
my $time1 = shift;
my $date2 = shift;
my $time2 = shift;
my $window = shift;

my ($year1, $month1, $day1) = &DateToymd($date1);
my ($hour1, $minute1, $second1) = &TimeTohms($time1);
my ($year2, $month2, $day2) = &DateToymd($date2);
my ($hour2, $minute2, $second2) = &TimeTohms($time2);

my ($day, $hour, $minute, $second) =
&Delta_DHMS($year1, $month1, $day1, $hour1, $minute1, $second1,
$year2, $month2, $day2, $hour2, $minute2, $second2);
$minute *= 60;
$hour *= 3600;
$day *= 86400;
my $total = $second + $minute + $hour + $day;
if (abs($total) < $window) {
return 1;
}
return 0;
}

sub CheckDateOrder { #takes three dates/times, returns true if they are in chronological order
my $date1 = shift;
my $time1 = shift;
my $date2 = shift;
my $time2 = shift;
my $date3 = shift;
my $time3 = shift;
if (&CompareDates($date1, $time1, $date2, $time2) == -1) {
return 0;
}
if (&CompareDates($date2, $time2, $date3, $time3) == -1) {
return 0;
}
return 1;
}

sub EpochSeconds { #calculates number of seconds since the epoch for the given date/time
my $date = shift;
my $time = shift;
my ($year, $month, $day) = &DateToymd($date);
my ($hour, $minute, $second) = &TimeTohms($time);
my ($d, $h, $m, $s) = &Delta_DHMS(1970, 1, 1, 0, 0, 0,
$year, $month, $day, $hour, $minute, $second);
my $seconds = $s + (60 * $m) + (3600 * $h) + (86400 * $d);
return $seconds;
}

sub SecondsToDateTime { #converts seconds since epoch to date/time
my $seconds = shift;
my $days = int($seconds / 86400);
$seconds -= $days * 86400;
my $hours = int($seconds / 3600);
$seconds -= $hours * 3600;
my $minutes = int($seconds / 60);
$seconds -= $minutes * 60;
my ($year, $month, $day, $hour, $minute, $second) =
&Add_Delta_DHMS(1970, 1, 1, 0, 0, 0, $days, $hours, $minutes, $seconds);
$month = sprintf("%02d", $month);
$day = sprintf("%02d", $day);
$hour = sprintf("%02d", $hour);
$minute = sprintf("%02d", $minute);
$second = sprintf("%02d", $second);
return "$year-$month-$day", "$hour:$minute:$second";
}

sub DateToDayName {
my $date = shift;
my ($year, $month, $day) = &DateToymd($date);
my $name = &Day_of_Week_to_Text(&Day_of_Week($year, $month, $day));
$name =~ /^[A-Za-z]{3}/;
$name = $&;
return $name;
}

sub ValiDate {
return @_;

}

sub CheckBusinessDay { # checks to see if date is business day. 1=yes, 0=no
my $date = shift;
my ($year, $month, $day) = &DateToymd($date);
if (Day_of_Week($year,$month,$day) < 6) { return 1; }
else { return 0; }
}

1; #don't remove this line
</pre>

chmod 0700 /usr/home/flowbin/queue.pl

Setup cronjob:
<pre>crontab -e
#move flow data into the queue
1,16,31,46 * * * * /usr/home/flowbin/queue.pl</pre>

==== flow processing: i2b ====
<pre>cat > /usr/home/flowbin/processflows-sql.pl
#!/usr/bin/perl

#use strict;
#$debug=1;
#$dry=1;

my $log = '/usr/home/flowbin/discards.log';

use Data::Dumper;

BEGIN {
push @INC, "/usr/home/flowbin";
}

#my $queuedir = "/usr/home/queue";
my $queuedir = "/usr/home/working";
my $archivedir = "/usr/home/archive";
my $sqldir = "/usr/home/sql";
my $sqldirworking = "/usr/home/sql/tmp";

unless ($dry) {
if (-e "$queuedir/.lock") {
open(FILE, "$queuedir/.lock");
my $pid = <FILE>;
chomp($pid);
close(FILE);
if (kill(0, $pid)) {
#another process is using the queue, bail out
exit(0);
}
else {
#dead lock file, remove it
`rm $queuedir/.lock`;
}
}
open(FILE, "> $queuedir/.lock");
print FILE "$$\n";
close(FILE);
}

opendir(DIR, $queuedir);
my @files = readdir(DIR);
closedir(DIR);

foreach my $file (sort @files) {
unless($file =~ /^\./) {
$file =~ /([0-9]{4}-[0-9]{2}-[0-9]{2})\.([0-9]{2})([0-9]{2})([0-9]{2})/;
my $date = "$1 $2:$3:$4";
my $outfile = "$1-$2:$3.sql";
unless (open (SQL, "+> $sqldirworking/$outfile")) { die "cant open $sqldirworking/$outfile"; }
my $condensedDate = $1;
$condensedDate =~ s/-//g;
my $iptotal = {};
my $protototal = {};
my $porttotal = {};

&debug("started file $file at ");
&debug(`date`);
&debug("getting raw flow data (flow-print)");
`cat $queuedir/$file | /usr/local/bin/flow-print -f 5 > /usr/home/working/tmp-$file`;
&debug("aggregating data at ");
&debug(`date`);
unless (open(DATA, "/usr/home/working/tmp-$file")) { die "can't open: $!"; }
LOOP: while (my $line = readline DATA) {
my @d = split /[\s]+/, $line;
if ($d[0] ne '' && $d[0] ne 'Start') {
my $addr = 0;
my $port = 0;

#Start End Sif SrcIPaddress SrcP DIf DstIPaddress DstP P Fl Pkts Octets
#0 1 2 3 4 5 6 7 8 9 10 11
#|
# outbound = 2, inbound = 1

my (@src_ip) = split '\.', $d[3];
my (@dst_ip) = split '\.', $d[6];

if ($src_ip[0] == 69 && $src_ip[1] == 55 && ($src_ip[2] == 229 || $src_ip[2] == 231)) { # for i2b
$d[2] = 2;
# hack for outbound bulk traffic counted 2x
#if ($src_ip[2] == 231) { $d[11] /= 2; $d[10] /= 2; }
}
# note- this is where we filter out IPs only found at i2b
elsif ($dst_ip[0] == 69 && $dst_ip[1] == 55 && ($dst_ip[2] == 229 || $dst_ip[2] == 231)) { # for i2b
$d[2] = 1;
}
else { next LOOP; }

if ($d[2] == 2) {
$addr = $d[3];
# if the dst-port is low, store that
if ($d[7] <= 1024) { $port = $d[7]; }
# if the src-port is low, store that
elsif ($d[4] <= 1024) { $port = $d[4]; }
else { $port = 99999; }
}
elsif ($d[2] == 1) {
$addr = $d[6];
# if the dst-port is high, assume its return traffic, try to store src-port if low
if ($d[7] > 1024) {
if ($d[4] <= 1024) { $port = $d[4]; }
else { $port = 99999; }
} else {
$port = $d[7];
}
} else {
next LOOP;
}

my (@ip) = split '\.', $addr;
unless ($ip[0] == 69) { next LOOP; }
unless ($ip[1] == 55) { next LOOP; }
unless ($ip[2] == 229 || $ip[2] == 231) { next LOOP; }

my $classC = "$ip[0]_$ip[1]_$ip[2]";

# IP dir
# if ($d[10] < 2147483647) { $iptotal->{$classC}->{$addr}->{$d[2]}->{'pktTotal'} += $d[10]; }
# if ($d[11] < 2147483647) { $iptotal->{$classC}->{$addr}->{$d[2]}->{'octetTotal'} += $d[11]; }
#
# if ($d[10] < 2147483647) { $protototal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{'pktTotal'} += $d[10]; }
# if ($d[11] < 2147483647) { $protototal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{'octetTotal'} += $d[11]; }
#
# if ($d[10] < 2147483647) { $porttotal->{$classC}->{$addr}->{$d[2]}->{$port}->{'pktTotal'} += $d[10]; }
# if ($d[11] < 2147483647) { $porttotal->{$classC}->{$addr}->{$d[2]}->{$port}->{'octetTotal'} += $d[11]; }
$iptotal->{$classC}->{$addr}->{$d[2]}->{'pktTotal'} += $d[10];
$iptotal->{$classC}->{$addr}->{$d[2]}->{'octetTotal'} += $d[11];

$protototal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{'pktTotal'} += $d[10];
$protototal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{'octetTotal'} += $d[11];

$porttotal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{$port}->{'pktTotal'} += $d[10];
$porttotal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{$port}->{'octetTotal'} += $d[11];
}
}
close(DATA);
`rm /usr/home/working/tmp-$file`;
&debug("processing ip totals at ");
&debug(`date`);
foreach my $classC (keys(%{$iptotal})) {
my @values;
foreach my $ip (keys(%{$iptotal->{$classC}})) {
foreach my $dir (keys(%{$iptotal->{$classC}->{$ip}})) {
my $octets = $iptotal->{$classC}->{$ip}->{$dir}->{'octetTotal'};
my $packets = $iptotal->{$classC}->{$ip}->{$dir}->{'pktTotal'};
# $packets = $packets > 2147483647 ? 0 : $packets;
if ($octets > 2147483647) {
my $ddir = $dir==1 ? 'in' : 'out';
#print SQL "$date $ip $ddir $octets\n";
# $octets = 0;
}
# dailyIpTotals.id = ip(no .'s)-yyyymmdd-direction
my $id = "$ip-$condensedDate-$dir";
$id =~ s/\.//g;
push @values, "('$date', '$ip', $dir, $octets, $packets)";
my $sql = "insert into dailyIpTotals_$classC values ('$id', '$date', '$ip', $dir, $octets, $packets) ON DUPLICATE KEY UPDATE octets=octets+$octets, packets=packets+$packets";
print "$sql\n" if $dry;
print SQL "$sql;\n";
# $db->query("insert into ipTotals values ('$date', '$ip', $dir, $octets, $packets)");
}
}

# break inserts into 100 records at a time
&debug("inserting $#values +1 values");
while ($#values > 0) {
my $sql = "insert into ipTotals_$classC values ";
my $max_index = $#values > 100 ? 100 : $#values;
for (my $i=0; $i<=$max_index; $i++) {
$sql .= shift @values;
$sql .= ',';
}
chop $sql;
print "$sql\n" if $dry;
print SQL "$sql;\n";
}
}

# &debug("processing protocol totals at ");
# &debug(`date`);
# foreach my $classC (keys(%{$protototal})) {
# $db->query("lock tables dailyProtoTotals_$classC write") unless $dry;
# my @values;
# foreach my $ip (keys(%{$protototal->{$classC}})) {
# foreach my $dir (keys(%{$protototal->{$classC}->{$ip}})) {
# foreach my $proto (keys(%{$protototal->{$classC}->{$ip}->{$dir}})) {
# my $octets = $protototal->{$classC}->{$ip}->{$dir}->{$proto}->{'octetTotal'};
# my $packets = $protototal->{$classC}->{$ip}->{$dir}->{$proto}->{'pktTotal'};
# # $octets = $octets > 2147483647 ? 0 : $octets;
# # $packets = $packets > 2147483647 ? 0 : $packets;
# # dailyProtoTotals.id = ip(no .'s)-yyyymmdd-direction-proto
# my $id = "$ip-$condensedDate-$dir-$proto";
# $id =~ s/\.//g;
# push @values, "('$date', '$ip', $dir, $proto, $octets, $packets)";
# my $sql = "insert into dailyProtoTotals_$classC values ('$id', '$date', '$ip', $dir, $proto, $octets, $packets) ON DUPLICATE KEY UPDATE octets=octets+$octets, packets=packets+$packets";
# print "$sql\n" if $dry;
# $db->query($sql) unless $dry;
# # $db->query("insert into protoTotals values ('$date', '$ip', $dir, $proto, $octets, $packets)");
# }
# }
# }
# $db->query("unlock tables") unless $dry;
# my $sql = "insert into protoTotals_$classC values ";
# $sql .= join ',', @values;
# $db->query("lock tables protoTotals_$classC write") unless $dry;
# print "$sql\n" if $dry;
# $db->query($sql) unless $dry;
# $db->query("unlock tables") unless $dry;
# }

&debug("processing port totals at ");
&debug(`date`);
foreach my $classC (keys(%{$porttotal})) {
my @values;
foreach my $ip (keys(%{$porttotal->{$classC}})) {
foreach my $dir (keys(%{$porttotal->{$classC}->{$ip}})) {
foreach my $proto (keys(%{$porttotal->{$classC}->{$ip}->{$dir}})) {
foreach my $port (keys(%{$porttotal->{$classC}->{$ip}->{$dir}->{$proto}})) {
my $octets = $porttotal->{$classC}->{$ip}->{$dir}->{$proto}->{$port}->{'octetTotal'};
my $packets = $porttotal->{$classC}->{$ip}->{$dir}->{$proto}->{$port}->{'pktTotal'};
# $octets = $octets > 2147483647 ? 0 : $octets;
# $packets = $packets > 2147483647 ? 0 : $packets;

# dailyPortTotals.id = ip(no .'s)-yyyymmdd-direction-protocol-port
my $id = "$ip-$condensedDate-$dir-$proto-$port";
$id =~ s/\.//g;
push @values, "('$date', '$ip', $dir, $proto, $port, $octets, $packets)";
my $sql = "insert into dailyPortTotals_$classC values ('$id', '$date', '$ip', $dir, $proto, $port, $octets, $packets) ON DUPLICATE KEY UPDATE octets=octets+$octets, packets=packets+$packets";
print "$sql\n" if $dry;
print SQL "$sql;\n";

# $db->query("insert into portTotals values ('$date', '$ip', $dir, $port, $octets, $packets)");
}
}
}
}

# break inserts into 100 records at a time
&debug("inserting $#values +1 values");
while ($#values > 0) {
my $sql = "insert into portTotals_$classC values ";
my $max_index = $#values > 100 ? 100 : $#values;
for (my $i=0; $i<=$max_index; $i++) {
$sql .= shift @values;
$sql .= ',';
}
chop $sql;
print "$sql\n" if $dry;
print SQL "$sql;\n";
}
}

# 12 1 8 1 1= 23
# dailyIpTotals.id = ip(no .'s)-yyyymmdd-direction
# 12 1 8 1 1 3=26
# dailyProtoTotals.id = ip(no .'s)-yyyymmdd-direction-proto
# 12 1 8 1 1 5=28
# dailyPortTotals.id = ip(no .'s)-yyyymmdd-direction-port
#print "finished at ";
#print `date`;
`mv $queuedir/$file $archivedir` unless $dry;
close(SQL);
`bzip2 $sqldirworking/$outfile`;
`mv $sqldirworking/$outfile.bz2 $sqldir/`;
}
}
`rm $queuedir/.lock` unless $dry;

sub debug {
my $message = shift;
if ($debug) {
print "$message\n";
}
}

# var full during ft-v05.2005-03-28.084500-0800 and
# 2005-02-24 69.55.226

# all port/daily totals before 2005-04-07
</pre>

This script sends the sql files to the traffic server for processing:
<pre>cat > /usr/home/flowbin/sendsql.pl
#!/usr/bin/perl

#use strict;
#$debug=1;
#$dry=1;

my $remote = "69.55.233.199";
my $sqldir = "/usr/home/sql";
my $archive = "/usr/home/archive";
my $sqldirremote = "/data/bwdb2/pending/";
my @err;
unless ($dry) {
if (-e "$sqldir/.lock") {
open(FILE, "$sqldir/.lock");
my $pid = <FILE>;
chomp($pid);
close(FILE);
if (kill(0, $pid)) {
#another process is using the queue, bail out
exit(0);
}
else {
#dead lock file, remove it
`rm $sqldir/.lock`;
}
}
open(FILE, "> $sqldir/.lock");
print FILE "$$\n";
close(FILE);
}

opendir(DIR, $sqldir);
my @files = readdir(DIR);
closedir(DIR);

foreach my $file (sort @files) {
next unless $file =~ /bz2$/;

my $r = `scp -Cq $sqldir/$file $remote:$sqldirremote 2>&1`;
# print "scp $sqldir/$file $remote:$sqldirremote";
unless ($?==0) {
push @err, "scp -Cq $sqldir/$file $remote:$sqldirremote ($r)";
}
else {
`mv $sqldir/$file $archive`;
`ssh $remote mv $sqldirremote/$file $sqldirremote/${file}.done`;
}
}

`rm $sqldir/.lock` unless $dry;

if (@err) {
email_support('bwdb2: sendsql.pl error',join "\n", @err);
}

sub email_support {
my $subj=shift;
my $body=shift;
use Mail::Sendmail;

# prepare message
my %mail = (
To => 'support@johncompanies.com,dave@johncompanies.com',
From => 'support@johncompanies.com',
Subject => $subj,
Message => $body,
smtp => 'mail.johncompanies.com',
);
sendmail(%mail) || warn "Error: $Mail::Sendmail::error";
}

sub debug {
my $message = shift;
if ($debug) {
print "$message\n";
}
}

# var full during ft-v05.2005-03-28.084500-0800 and
# 2005-02-24 69.55.226

# all port/daily totals before 2005-04-07
</pre>

<pre>crontab -e
#process flows
2,17,32,47 * * * * /usr/home/flowbin/processflows-sql.pl
#move sql commands to traffic db
8,23,38,53 * * * * /usr/home/flowbin/sendsql.pl
</pre>

==== flow processing: castle ====
<pre>
cat > /usr/home/flowbin/processflows.pl

#!/usr/bin/perl

#use strict;
#$debug=1;
#$dry=1;

my $log = '/usr/home/flowbin/discards.log';

use Data::Dumper;

BEGIN {
push @INC, "/usr/home/flowbin";
}

use db;

#my $queuedir = "/usr/home/queue";
my $queuedir = "/usr/home/working";
my $archivedir = "/usr/home/archive";

unless ($dry) {
if (-e "$queuedir/.lock") {
open(FILE, "$queuedir/.lock");
my $pid = <FILE>;
chomp($pid);
close(FILE);
if (kill(0, $pid)) {
#another process is using the queue, bail out
exit(0);
}
else {
#dead lock file, remove it
`rm $queuedir/.lock`;
}
}
open(FILE, "> $queuedir/.lock");
print FILE "$$\n";
close(FILE);
}

my $db = db->new();
$db->connect('traffic', '', 'root', '5over3') || die $db->{'error'};

opendir(DIR, $queuedir);
my @files = readdir(DIR);
closedir(DIR);

foreach my $file (sort @files) {
unless($file =~ /^\./) {
$file =~ /([0-9]{4}-[0-9]{2}-[0-9]{2})\.([0-9]{2})([0-9]{2})([0-9]{2})/;
my $date = "$1 $2:$3:$4";
my $condensedDate = $1;
$condensedDate =~ s/-//g;
my $iptotal = {};
my $protototal = {};
my $porttotal = {};

&debug("started file $file at ");
&debug(`date`);
&debug("getting raw flow data (flow-print)");
`cat $queuedir/$file | /usr/local/bin/flow-print -f 5 > /usr/home/working/tmp-$file`;
&debug("aggregating data at ");
&debug(`date`);
unless (open(DATA, "/usr/home/working/tmp-$file")) { die "can't open: $!"; }
LOOP: while (my $line = readline DATA) {
my @d = split /[\s]+/, $line;
if ($d[0] ne '' && $d[0] ne 'Start') {
my $addr = 0;
my $port = 0;

#Start End Sif SrcIPaddress SrcP DIf DstIPaddress DstP P Fl Pkts Octets
#0 1 2 3 4 5 6 7 8 9 10 11
#|
# outbound = 2, inbound = 1

my (@src_ip) = split '\.', $d[3];
my (@dst_ip) = split '\.', $d[6];

if ($src_ip[0] == 69 && $src_ip[1] == 55 &&
$src_ip[2] >= 224 && $src_ip[2] <= 239 &&
$src_ip[2] != 229 && $src_ip[2] != 231) { # for castle
# if ($src_ip[0] == 69 && $src_ip[1] == 55 && $src_ip[2] == 229) { # for i2b
$d[2] = 2;
# hack for outbound bulk traffic counted 2x
if ($dst_ip[2] == 234) { $d[11] /= 2; $d[10] /= 2; }
}
elsif ($dst_ip[0] == 69 && $dst_ip[1] == 55 &&
$dst_ip[2] >= 224 && $dst_ip[2] <= 239 &&
$dst_ip[2] != 229 && $dst_ip[2] != 231) { # for castle
# elsif ($dst_ip[0] == 69 && $dst_ip[1] == 55 && $dst_ip[2] == 229) { # for i2b
$d[2] = 1;
}
else { next LOOP; }

if ($d[2] == 2) {
$addr = $d[3];
# if the dst-port is low, store that
if ($d[7] <= 1024) { $port = $d[7]; }
# if the src-port is low, store that
elsif ($d[4] <= 1024) { $port = $d[4]; }
else { $port = 99999; }
}
elsif ($d[2] == 1) {
$addr = $d[6];
# if the dst-port is high, assume its return traffic, try to store src-port if low
if ($d[7] > 1024) {
if ($d[4] <= 1024) { $port = $d[4]; }
else { $port = 99999; }
} else {
$port = $d[7];
}
} else {
next LOOP;
}

my (@ip) = split '\.', $addr;
unless ($ip[0] == 69) { next LOOP; }
unless ($ip[1] == 55) { next LOOP; }
unless ($ip[2] >= 224 && $ip[2] <= 239 && $ip[2] != 229 && $ip[2] != 231) { next LOOP; }
# unless ($ip[2] == 229) { next LOOP; }

my $classC = "$ip[0]_$ip[1]_$ip[2]";

# IP dir
# if ($d[10] < 2147483647) { $iptotal->{$classC}->{$addr}->{$d[2]}->{'pktTotal'} += $d[10]; }
# if ($d[11] < 2147483647) { $iptotal->{$classC}->{$addr}->{$d[2]}->{'octetTotal'} += $d[11]; }
#
# if ($d[10] < 2147483647) { $protototal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{'pktTotal'} += $d[10]; }
# if ($d[11] < 2147483647) { $protototal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{'octetTotal'} += $d[11]; }
#
# if ($d[10] < 2147483647) { $porttotal->{$classC}->{$addr}->{$d[2]}->{$port}->{'pktTotal'} += $d[10]; }
# if ($d[11] < 2147483647) { $porttotal->{$classC}->{$addr}->{$d[2]}->{$port}->{'octetTotal'} += $d[11]; }
$iptotal->{$classC}->{$addr}->{$d[2]}->{'pktTotal'} += $d[10];
$iptotal->{$classC}->{$addr}->{$d[2]}->{'octetTotal'} += $d[11];

$protototal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{'pktTotal'} += $d[10];
$protototal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{'octetTotal'} += $d[11];

$porttotal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{$port}->{'pktTotal'} += $d[10];
$porttotal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{$port}->{'octetTotal'} += $d[11];
}
}
close(DATA);
`rm /usr/home/working/tmp-$file`;
&debug("processing ip totals at ");
&debug(`date`);
foreach my $classC (keys(%{$iptotal})) {
$db->query("lock tables dailyIpTotals_$classC write") unless $dry;
my @values;
foreach my $ip (keys(%{$iptotal->{$classC}})) {
foreach my $dir (keys(%{$iptotal->{$classC}->{$ip}})) {
my $octets = $iptotal->{$classC}->{$ip}->{$dir}->{'octetTotal'};
my $packets = $iptotal->{$classC}->{$ip}->{$dir}->{'pktTotal'};
# $packets = $packets > 2147483647 ? 0 : $packets;
if ($octets > 2147483647) {
my $ddir = $dir==1 ? 'in' : 'out';
`echo "$date $ip $ddir $octets\n" >> $log`;
# $octets = 0;
}
# dailyIpTotals.id = ip(no .'s)-yyyymmdd-direction
my $id = "$ip-$condensedDate-$dir";
$id =~ s/\.//g;
push @values, "('$date', '$ip', $dir, $octets, $packets)";
my $sql = "insert into dailyIpTotals_$classC values ('$id', '$date', '$ip', $dir, $octets, $packets) ON DUPLICATE KEY UPDATE octets=octets+$octets, packets=packets+$packets";
print "$sql\n" if $dry;
$db->query($sql) unless $dry;
# $db->query("insert into ipTotals values ('$date', '$ip', $dir, $octets, $packets)");
}
}
$db->query("unlock tables") unless $dry;

$db->query("lock tables ipTotals_$classC write") unless $dry;
# break inserts into 100 records at a time
&debug("inserting $#values +1 values");
while ($#values > 0) {
my $sql = "insert into ipTotals_$classC values ";
my $max_index = $#values > 100 ? 100 : $#values;
for (my $i=0; $i<=$max_index; $i++) {
$sql .= shift @values;
$sql .= ',';
}
chop $sql;
print "$sql\n" if $dry;
$db->query($sql) unless $dry;
}
$db->query("unlock tables") unless $dry;
}

sleep 20;
# &debug("processing protocol totals at ");
# &debug(`date`);
# foreach my $classC (keys(%{$protototal})) {
# $db->query("lock tables dailyProtoTotals_$classC write") unless $dry;
# my @values;
# foreach my $ip (keys(%{$protototal->{$classC}})) {
# foreach my $dir (keys(%{$protototal->{$classC}->{$ip}})) {
# foreach my $proto (keys(%{$protototal->{$classC}->{$ip}->{$dir}})) {
# my $octets = $protototal->{$classC}->{$ip}->{$dir}->{$proto}->{'octetTotal'};
# my $packets = $protototal->{$classC}->{$ip}->{$dir}->{$proto}->{'pktTotal'};
# # $octets = $octets > 2147483647 ? 0 : $octets;
# # $packets = $packets > 2147483647 ? 0 : $packets;
# # dailyProtoTotals.id = ip(no .'s)-yyyymmdd-direction-proto
# my $id = "$ip-$condensedDate-$dir-$proto";
# $id =~ s/\.//g;
# push @values, "('$date', '$ip', $dir, $proto, $octets, $packets)";
# my $sql = "insert into dailyProtoTotals_$classC values ('$id', '$date', '$ip', $dir, $proto, $octets, $packets) ON DUPLICATE KEY UPDATE octets=octets+$octets, packets=packets+$packets";
# print "$sql\n" if $dry;
# $db->query($sql) unless $dry;
# # $db->query("insert into protoTotals values ('$date', '$ip', $dir, $proto, $octets, $packets)");
# }
# }
# }
# $db->query("unlock tables") unless $dry;
# my $sql = "insert into protoTotals_$classC values ";
# $sql .= join ',', @values;
# $db->query("lock tables protoTotals_$classC write") unless $dry;
# print "$sql\n" if $dry;
# $db->query($sql) unless $dry;
# $db->query("unlock tables") unless $dry;
# }

&debug("processing port totals at ");
&debug(`date`);
foreach my $classC (keys(%{$porttotal})) {
$db->query("lock tables dailyPortTotals_$classC write") unless $dry;
my @values;
foreach my $ip (keys(%{$porttotal->{$classC}})) {
foreach my $dir (keys(%{$porttotal->{$classC}->{$ip}})) {
foreach my $proto (keys(%{$porttotal->{$classC}->{$ip}->{$dir}})) {
foreach my $port (keys(%{$porttotal->{$classC}->{$ip}->{$dir}->{$proto}})) {
my $octets = $porttotal->{$classC}->{$ip}->{$dir}->{$proto}->{$port}->{'octetTotal'};
my $packets = $porttotal->{$classC}->{$ip}->{$dir}->{$proto}->{$port}->{'pktTotal'};
# $octets = $octets > 2147483647 ? 0 : $octets;
# $packets = $packets > 2147483647 ? 0 : $packets;

# dailyPortTotals.id = ip(no .'s)-yyyymmdd-direction-protocol-port
my $id = "$ip-$condensedDate-$dir-$proto-$port";
$id =~ s/\.//g;
push @values, "('$date', '$ip', $dir, $proto, $port, $octets, $packets)";
my $sql = "insert into dailyPortTotals_$classC values ('$id', '$date', '$ip', $dir, $proto, $port, $octets, $packets) ON DUPLICATE KEY UPDATE octets=octets+$octets, packets=packets+$packets";
print "$sql\n" if $dry;
$db->query($sql) unless $dry;
# $db->query("insert into portTotals values ('$date', '$ip', $dir, $port, $octets, $packets)");
}
}
}
}
$db->query("unlock tables") unless $dry;

$db->query("lock tables portTotals_$classC write") unless $dry;
# break inserts into 100 records at a time
&debug("inserting $#values +1 values");
while ($#values > 0) {
my $sql = "insert into portTotals_$classC values ";
my $max_index = $#values > 100 ? 100 : $#values;
for (my $i=0; $i<=$max_index; $i++) {
$sql .= shift @values;
$sql .= ',';
}
chop $sql;
print "$sql\n" if $dry;
$db->query($sql) unless $dry;
}
$db->query("unlock tables") unless $dry;
sleep 10;
}

# 12 1 8 1 1= 23
# dailyIpTotals.id = ip(no .'s)-yyyymmdd-direction
# 12 1 8 1 1 3=26
# dailyProtoTotals.id = ip(no .'s)-yyyymmdd-direction-proto
# 12 1 8 1 1 5=28
# dailyPortTotals.id = ip(no .'s)-yyyymmdd-direction-port
#print "finished at ";
#print `date`;
`mv $queuedir/$file $archivedir` unless $dry;
}
}
`rm $queuedir/.lock` unless $dry;

sub debug {
my $message = shift;
if ($debug) {
print "$message\n";
}
}

# var full during ft-v05.2005-03-28.084500-0800 and
# 2005-02-24 69.55.226

# all port/daily totals before 2005-04-07
</pre>

<pre>
cat > /usr/home/flowbin/db.pm

#!/usr/bin/perl
#
# $Header: /usr/cvs/newgw/lib/db.pm,v 1.4 2003/06/05 18:20:01 glenn Exp $
#
# Copyright (c) 2003
# e-Monitoring Networks, Inc. All rights reserved.
#
#
package db;

use strict;
use DBI;

sub new {
my $class = shift;
my $self = {};

$self->{'debug'} = 0;
bless $self, $class;

return $self;
}

sub connect {
my $self = shift;
my $dbname = shift;
my $dbhost = shift;
my $dbuser = shift;
my $dbpass = shift;

my $host = '';
if (defined($dbhost)) {
$host = ";host=$dbhost";
}

eval {
$self->debug("connecting to: DBI:mysql:database=$dbname;$host", 1);
$self->{'dbh'} = DBI->connect("DBI:mysql:database=$dbname;$host", $dbuser, $dbpass);
};
if ($self->{'dbh'}) {
return 1;
}
$self->{'error'} = "Error connecting to database $@";
$self->debug("Error connecting to database $@");
return 0;
}

sub query {
my $self = shift;
my $query = shift;

$self->debug($query, 1);
my $sth;
eval {
$sth = $self->{'dbh'}->prepare($query);
};
unless ($sth) {
$self->{'error'} = "error preparing query $@";
$self->debug("error preparing query $@");
return undef;
}
my $qty;
eval {
$qty = $sth->execute;
};
unless ($qty) {
$self->{'error'} = "error executing query $@";
warn "error executing query $@ $query";
return undef;
}
$self->debug("returning $qty, $sth from query", 6);
return ($qty, $sth);
}

sub disconnect {
my $self = shift;

$self->{'dbh'}->disconnect;
return 0;
}

sub debug {
my $self = shift;
my $msg = shift;
my $level = shift || 0;

if ($level < $self->{'debug'}) {
print "$msg\n";
}
return 0;
}
1;
</pre>

mkdir /usr/home/archive
mkdir -p /usr/home/sql/tmp

<pre>crontab -e
#process flows
2,17,32,47 * * * * /usr/home/flowbin/processflows.pl</pre>

==== setup traffic db ====
* Install mysql:
<pre>cd /usr/ports/databases/mysql50-server
make install clean</pre>

cat >> /etc/rc.conf
mysql_enable="YES"

Move db data dir:
/usr/local/etc/rc.d/mysql-server stop
mkdir /usr/home/database/
mv /var/db/mysql/* /usr/home/database/
chown -R mysql:mysql /usr/home/database

Edit database location in startup script:
vi /usr/local/etc/rc.d/mysql-server
# : ${mysql_dbdir="/var/db/mysql"}
: ${mysql_dbdir="/usr/home/database"}

/usr/local/etc/rc.d/mysql-server start

* Install mysql perl database modules:
<pre>
cd /usr/ports/databases/p5-DBI
make install clean
cd /usr/ports/databases/p5-DBD-mysql50
make install clean
(no to SSL support)
</pre>

* Setting up database
<pre>
rehash
/usr/local/etc/rc.d/mysql-server start
mysql -u root
create database traffic;
grant all on *.* to root@localhost identified by '5over3';
grant all on traffic.* to jc@10.1.4.5 identified by '2gMKY3Wt';

</pre>

If this was a new server we'd setup new tables. See [[#mysql_2|mysql]] for how those tables would be setup.

We are assuming here we are moving data from an existing db, here's how that's done (from the current traffic db):
rsync -av --progress /usr/home/database/traffic/ 10.1.4.203:/usr/home/database/traffic/

When you're ready to do the cutover, shut down mysql on both hosts and do one last sync.

==== process flows from bwdb2 ====
On traffic database server (bwdb):

<pre>crontab -e
#import sql from bwdb2
10,25,40,55 * * * * /usr/home/flowbin/processsql.pl</pre>

Add access to mysql:
<pre>mysql -u root -p
grant all on traffic.* to bwdb2@localhost identified by 's1lver4d';
</pre>

<pre>cat > /usr/home/flowbin/processsql.pl

#!/usr/bin/perl

#use strict;
#$debug=1;
#$dry=1;

my $sqldir = "/usr/home/bwdb2/pending";
my $mysql = '/usr/local/bin/mysql';
my @err;
unless ($dry) {
if (-e "$sqldir/.lock") {
open(FILE, "$sqldir/.lock");
my $pid = <FILE>;
chomp($pid);
close(FILE);
if (kill(0, $pid)) {
#another process is using the queue, bail out
exit(0);
}
else {
#dead lock file, remove it
`rm $sqldir/.lock`;
}
}
open(FILE, "> $sqldir/.lock");
print FILE "$$\n";
close(FILE);
}

opendir(DIR, $sqldir);
my @files = readdir(DIR);
closedir(DIR);

foreach my $file (sort @files) {
next unless $file =~ /done$/;
my $r = `bzcat $sqldir/$file | $mysql -u bwdb2 -ps1lver4d traffic`;
unless ($?==0) {
push @err, "bzcat $sqldir/$file | $mysql -u bwdb2 -pxxxxx traffic ($r)";
}
else {
`rm $sqldir/$file`;
}
}

`rm $sqldir/.lock` unless $dry;

if (@err) {
email_support('bwdb: processsql.pl error',join "\n", @err);
}

sub email_support {
my $subj=shift;
my $body=shift;
use Mail::Sendmail;

# prepare message
my %mail = (
To => 'dave@johncompanies.com',
From => 'support@johncompanies.com',
Subject => $subj,
Message => $body,
smtp => 'mail.johncompanies.com',
);
sendmail(%mail) || warn "Error: $Mail::Sendmail::error";
}

sub debug {
my $message = shift;
if ($debug) {
print "$message\n";
}
}
</pre>

chmod 0700 /usr/home/flowbin/processsql.pl

Make sure bwdb is reachable from the outside only to bwdb2:

On nat, add to <tt>/etc/ipnat.rules</tt>
<pre># bwdb
bimap fxp0 10.1.4.203/32 -> 69.55.233.199/32</pre>

Reload:
ipnat -C -F -f /etc/ipnat.rules

Setup firewall rule on firewall:
ipfw add 00094 allow ip from 66.181.18.5 to 69.55.233.199 22
ipfw add 00094 deny ip from any to 69.55.233.199

Setup firewall on bwdb to restrict access now that it's nat'd:
<pre>
cat >> /usr/local/etc/rc.d/boot.sh
ipfw add 1 allow tcp from any to any established
ipfw add 2 allow ip from 10.1.4.0/24,66.181.18.5,69.55.233.195 to me 22
ipfw add 3 allow ip from 10.1.4.5 to me 3306
ipfw add 4 allow ip from 69.55.225.225 53 to me
ipfw add 5 allow ip from 69.55.230.2 25 to me
ipfw add 6 allow ip from me to me 4444
ipfw add 7 allow icmp from any to me
ipfw add 8 allow udp from 10.1.4.203 to 10.1.4.203 dst-port 4444
ipfw add 9 allow udp from 10.1.4.5 to me 161
ipfw add 100 deny ip from any to me
</pre>

chmod 0700 /usr/local/etc/rc.d/boot.sh

From bwdb2, add ssh key:
cat /root/.ssh/id_dsa.pub | ssh 69.55.233.199 'cat - >> /root/.ssh/authorized_keys'

Confirm no password access:
ssh 69.55.233.199 hostname

= bwdb2 =
== Summary ==
This machine tracks and stores network traffic (netflow) at i2b. It is our means to monitor customer bandwidth usage.

* Location: i2b, cab6
* OS: FreeBSD 6.4 x86
* Networking: Priv IP: 10.1.2.4 There are 2 onboard nic's, one of which is the "listener"
* Hardware: Custom 2U. Single power supply.
* Drives: two 150 GB (2 x 150GB) RAID1 arrays running on a 3ware 7006 RAID card.

== Services Provided ==
* netflow
* bigbrother

== netflow ==

The main function of this server is to run netflow on an eth device in promiscuous mode so as to hear everything happening on the port (wherein all network traffic is mirrored to that port via the cisco swith). Every 15min, it creates a flow file under <tt>/usr/home/flows/</tt> (organized by date). The flow file contains all traffic data for a 15min increment of time.

A cronjob moves that flow file (or files if there are multiple due to some delay)
1,16,31,46 * * * * /usr/home/flowbin/queue.pl

into a processing queue:
<tt>/usr/home/working</tt>

Then a separate file processes whatever flow files it finds there, and builds sql files ready for insertion into the traffic database:
2,17,32,47 * * * * /usr/home/flowbin/processflows-sql.pl

Then yet another process copies the sql files to the traffic database server for processing and insertion into the mysql database:
8,23,38,53 * * * * /usr/home/flowbin/sendsql.pl

== Regular maintenance ==
*[[Routine_Maintenance#Adaptec_Controllers|Check RAID array]]

* if space becomes tight, move sql files and flow files to backup server, both located in <tt>/usr/home/flowbin/archive</tt>

= firewall (newgateway) =

== Summary ==

This machine is the primary (only) firewall for the entire network at castle.

* Location: castle, cab 3-8
* OS: FreeBSD 4.11 x86
* Networking: Priv IP: 10.1.4.223, Pub IPs: 69.55.233.164 (external), 69.55.233.156 (internal). It has 3 network connections (2 onboard, 1 PCI) connecting to the external, internal and private networks. If you're looking at the back of the server, the internal-network-facing nic is on the right (em1), and the external-facing-network (3750) is on the left (em0).
* Hardware: 6 SCSI SCA drive bays (2 columns of 3, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: 36 GB (2 x 36GB) RAID1 array running on an Adaptec 2120S PCI RAID card.

== Services Provided ==
* firewall (ipfw)
* snmp
* bigbrother

== Firewall Rule Configuration ==

See [[FreeBSD_Reference#Firewall_Rule_Configuration|Firewall Rule Configuration]] for more discussion on how to actually manipulate firewall rules.

== Disaster Recovery ==

If there is ever an outage with the firewall, the old firewall "gate" is located just below and is running with the proper network configuration, but with no firewall rules in place (to facilitate good throughput). Have castle move the cable on the left on the current firewall to the left port in the old firewall and the right cable to the right port.

Here's what you need to put in /etc/rc.conf to get a firewall going (as far as routes and IPs)

<pre>hostname="newgateway.johncompanies.com"
firewall_script="/etc/firewall.sh"
firewall_enable="NO"
sendmail_enable="NONE"
sshd_enable="YES"
inetd_enable="NO"
xntpd_enable="YES"
snmpd_enable="YES"
#snmpd_flags="-as -p /var/run/snmpd.pid"
#ipnat_enable="YES"
#ipnat_rules="/etc/ipnat.rules"
gateway_enable="YES"

defaultrouter="69.55.233.161"

ifconfig_xl0="inet 10.1.4.223 netmask 255.255.255.0"
ifconfig_em0="inet 69.55.233.164 netmask 255.255.255.248"

#
# Original JohnCompanies 69.55.224.0/20
#
ifconfig_em1="inet 69.55.233.156 netmask 255.255.255.248"

static_routes="route1 route2 route3 route4 route5 route6 route7 route8 route9 route10 route11 route1
2 route13 route14 route15 route16 route17 route18"

route_route1="-net 69.55.224.0 69.55.233.153"
route_route2="-net 69.55.225.0 69.55.233.153"
route_route3="-net 69.55.226.0 69.55.233.153"
route_route4="-net 69.55.227.0 69.55.233.153"
route_route5="-net 69.55.228.0 69.55.233.153"
route_route6="-net 69.55.229.0 69.55.233.153"
route_route7="-net 69.55.230.0 69.55.233.153"
route_route8="-net 69.55.231.0 69.55.233.153"
route_route9="-net 69.55.232.0 69.55.233.153"
route_route10="-net 69.55.233.0 69.55.233.153"
route_route11="-net 69.55.234.0 69.55.233.153"
route_route12="-net 69.55.235.0 69.55.233.153"
route_route13="-net 69.55.236.0 69.55.233.153"
route_route14="-net 69.55.237.0 69.55.233.153"
route_route15="-net 69.55.238.0 69.55.233.153"
route_route16="-net 69.55.239.0 69.55.233.153"
route_route17="-net 10.1.5.0 10.1.4.2"
route_route18="-net 10.1.6.0 10.1.4.2"

#In case of 3750 failure:
#defaultrouter="69.43.128.81"
#ifconfig_em0="inet 69.43.129.84 netmask 255.255.255.248"

#bind .1's here:
#ifconfig_em1="inet 69.55.224.1 netmask 255.255.255.0"
#ifconfig_em1_alias0="inet 69.55.225.1 netmask 255.255.255.0"
#ifconfig_em1_alias1="inet 69.55.226.1 netmask 255.255.255.0"
#ifconfig_em1_alias2="inet 69.55.227.1 netmask 255.255.255.0"
#ifconfig_em1_alias3="inet 69.55.228.1 netmask 255.255.255.0"
#ifconfig_em1_alias4="inet 69.55.229.1 netmask 255.255.255.0"
#ifconfig_em1_alias5="inet 69.55.230.1 netmask 255.255.255.0"
#ifconfig_em1_alias6="inet 69.55.231.1 netmask 255.255.255.0"
#ifconfig_em1_alias7="inet 69.55.232.1 netmask 255.255.255.0"
#ifconfig_em1_alias8="inet 69.55.233.1 netmask 255.255.255.0"
#ifconfig_em1_alias9="inet 69.55.234.1 netmask 255.255.255.0"
#ifconfig_em1_alias10="inet 69.55.235.1 netmask 255.255.255.0"
#ifconfig_em1_alias11="inet 69.55.236.1 netmask 255.255.255.0"
#ifconfig_em1_alias12="inet 69.55.237.1 netmask 255.255.255.0"
#ifconfig_em1_alias13="inet 69.55.238.1 netmask 255.255.255.0"
#ifconfig_em1_alias14="inet 69.55.239.1 netmask 255.255.255.0"

#bulk:
# reassign 69.55.231.1 to the int iface on the firewall
# set the DG on the firewall to 69.43.138.9
# set the ext firewall IP to 69.43.138.12, NM: 255.255.255.248</pre>

== Cronjobs ==
1 0 * * * /usr/local/etc/rsync.backup
Backup to backup1

0 0 1 * * /sbin/ipfw zero
0 0 1 * * /sbin/ipfw del 3 4 5 17331
Reset counters and remove pipe rules on the 1st of the month. Pay attention when setting up a rule as 3 4 5 (that's not a temporary traffic cap).

Inside <tt>/etc/daily.local</tt> you will see a call to <tt>/etc/makepiperules.pl</tt>
This script will create <tt>/etc/firewall.sh</tt> which contains all the firewall and pipe rules in place at the time the script was run.

== DOS attacks ==

See [[FreeBSD_Reference#Handling_a_DoS_attack|Handling a DoS attack]] regarding how to handle a DOS attack.

Theres a background process (running from user shell) that monitors the firewall for incoming UDP DoS attacks. When it notices packets above a certain level it will
# enter a rule that allows all UDP to go through
# send an emergency email to support and indicating an attack is in progress
# send an email to castle (nocstaff@castleaccess.com and jcsupport@castleaccess.com) telling them to investigate and put up a null if warranted
# wait for a couple minutes to see if the attack subsides- if so it will remove the pass-all UDP rule, if not it will repeat the process from #1
This file lives under /usr/home/user/doswatch.pl
To run:
cd /usr/home/user
./doswatch.pl &

To kill;
fg
^C

It writes its findings to /usr/home/user/doswatch.log

= backup1 =

== Summary ==

This machine acts as the primary backup location for all VPS-based customers. No customer directly accesses this server to perform their backups. We also store cancelled customers on this server.

* Location: castle, cab 3-8
* OS: Ubuntu 8.04.1 server x86
* Networking: Priv IP: 10.1.4.8, Pub IP: 69.55.230.11 (firewalled from all but JC infrastructure @ i2b)
* Hardware: 6 SATA drive bays (2 columns of 3, drive 0 top left, drive 1 just below) all hot-swap. Single power supply.
* Drives: 4.5 TB (6 x 1TB) RAID5 array running on a 3ware 9650SE-8LPML (8-port) card

== Services provided ==
* backup via rsync
* mysql - traffic data
* nfs server - for backups
* snmp client - for big brother
* bigbrother client

== Usage and Notes ==
* all data is stored under /data
* virtually all jc infrastructure, and all VPS machines are setup to mount to backup1 via nfs (mountpoint: <tt>/backup1</tt>), and they all have their ssh keys setup to allow passwordless rsync's
* each virt or jail backs up each evening to backup1. Each server has it's own directory (named for the server). Under those directories are 7 daily snapshots (0-6)
* at the time of writing, the mysql server running here is replicating from (slave to) the mysql instance on bwdb. Requests for bandwidth data usage for customers (coming from management, account manager, and accounting scripts running on mail) all direct towards the database "traffic" running on this server.
* cancelled customer systems are compressed and stored under <tt>/data/deprecated</tt>
* archived bwdb2 flow files are stored under <tt>/data/bwdb2</tt>
* critical files from backup2 are stored under <tt>/data/backup2</tt>

== Cronjobs ==
<pre>
00 5 * * * /usr/local/sbin/backupwatch.pl 2>&1 > /dev/null
35 5 * * * /usr/local/sbin/usage_check; /usr/local/sbin/snapshot_archive; /usr/local/sbin/snapshot_rotate /data/backuplog.log</pre>
this runs daily the scripts to report on how much disk space each customer system occupies and how long their backups took. Then it rotates backups for each system, removing the oldest backup. It will email support@johncompanies.com at it’s conclusion. This email can be deleted, however note when it begins to take significantly longer to complete, ie runs past 2200 pm – this usually indicates a problem on the backup server.

<pre>10,25,40,55 * * * * /usr/local/sbin/processsql.pl
</pre>
this processes prepared sql command files sent from/by bwdb2 (@ i2b) and imports them into the traffic database.
<pre>0 0 * * * /usr/local/sbin/3wraidchk
</pre>
checks the health of the RAID array

== Regular maintenance ==
*[[Routine_Maintenance#Free_up_space_on_backup1|Remove old backups]]
*[[Routine_Maintenance#3ware|Check on auto-verify]]

== build ==

<pre>Setup raid5 with a boot vol of 12G 5.45tb
12G boot
4664 GB

Install ubuntu 8.04

Swap 4G

Don’t format data drive

http://www.unixgods.org/~tilo/linux_larger_2TB.html

parted /dev/sdb
print
mklabel gpt
print

#Disk /dev/sdb: 4987GB
#Sector size (logical/physical): 512B/512B
#Partition Table: gpt

#Number Start End Size File system Name Flags

mkpart primary ext3 0 4987GB
print

#Disk /dev/sdb: 5987GB
#Sector size (logical/physical): 512B/512B
#Partition Table: gpt

#Number Start End Size File system Name Flags
# 1 17.4kB 4987GB 4987GB primary

quit

mkfs.ext3 /dev/sdb1
#mke2fs 1.40.8 (13-Mar-2008)
#Filesystem label=
#OS type: Linux
#Block size=4096 (log=2)
#Fragment size=4096 (log=2)
#304390144 inodes, 1217544183 blocks
#60877209 blocks (5.00%) reserved for the super user
#First data block=0
#Maximum filesystem blocks=0
#37157 block groups
#32768 blocks per group, 32768 fragments per group
#8192 inodes per group
#Superblock backups stored on blocks:
# 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
# 4096000, 7962624, 11239424, 20480000, 23887872, 71663616, 78675968,
# 102400000, 214990848, 512000000, 550731776, 644972544
#
#Writing inode tables: 967/37157

mkdir /data

#root@backup1:~# df -h
#Filesystem Size Used Avail Use% Mounted on
#/dev/sda2 8.3G 540M 7.3G 7% /
#varrun 1013M 40K 1013M 1% /var/run
#varlock 1013M 0 1013M 0% /var/lock
#udev 1013M 56K 1013M 1% /dev
#devshm 1013M 0 1013M 0% /dev/shm
#/dev/sdb1 4.5T 192M 4.3T 1% /data

apt-get update
apt-get upgrade
apt-get install snmp snmpd ntp nfs-kernel-server

echo "\"\e[5~\": history-search-backward" >> ~/.inputrc
echo "\"\e[6~\": history-search-forward" >> ~/.inputrc

vi /etc/ntp.conf
server 10.1.4.5

scp root@10.1.4.3:/root/.ssh/authorized_keys /root/.ssh/
cd /root/
ssh-keygen -t dsa
echo "10.1.4.3 backup2" >> /etc/hosts

cat .ssh/id_dsa.pub | ssh backup2 'cat - >> /root/.ssh/authorized_keys'

ssh backup2

vi /root/.bashrc
export PS1="[\u@\h \w]# "
alias h='history'
alias vi='vim'
alias j='jobs'
export PS1="[\u@\h \w]# "
alias dr='screen -dr'
export EDITOR=vim
export GREP_OPTIONS='--color=auto'
export HISTFILESIZE=1000

source /root/.bashrc

echo "# ttyS0 - getty
#
# This service maintains a getty on ttyS0 from the point the system is
# started until it is shut down again.

start on runlevel 2
start on runlevel 3
start on runlevel 4
start on runlevel 5

stop on runlevel 0
stop on runlevel 1
stop on runlevel 6

respawn
exec /sbin/getty 38400 ttyS0" > /etc/event.d/ttyS0

vi /boot/grub/menu.lst

serial --unit=0 --speed=38400 --word=8 --parity=no --stop=1
terminal --timeout=15 serial console

append to kernel lines:
console=tty0 console=ttyS0,38400n8

show menu:
#hiddenmenu

echo 'rocommunity jcread 10.1.4.5
rocommunity jcread 10.1.4.3
agentaddress 10.1.4.8:161' > /etc/snmp/snmpd.conf

# to see which iface it is, on backup2:

snmpwalk -v 1 -c jcread 10.1.4.8 interface

echo "bb:x:1984:1984:Big Brother:/home/bb:/bin/bash" >> /etc/passwd

echo "bb:x:1984:" >> /etc/group

pwconv

mkdir /home/bb
chown bb.bb /home/bb

cd ~bb
scp backup2:/mnt/data4/build/bb/bb-linux.tar .

tar xf bb-linux.tar

cd /home/bb/bbc1.9e-btf/etc

echo "10.1.4.5 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh" > /home/bb/bbc1.9e-btf/etc/bb-hosts
echo "10.1.4.8 backup1.johncompanies.com # ssh" >> /home/bb/bbc1.9e-btf/etc/bb-hosts

echo "/:90:95
/var:90:95
/data:85:99" > /home/bb/bbc1.9e-btf/etc/bb-dftab

vi /home/bb/bbc1.9e-btf/bin/bb-disk.sh
(remove all | SORT xxxx)

chmod +r /var/log/messages

./bbchkcfg.sh
#(y to questions)
./bbchkhosts.sh
#(ignore ssh errors)
cd ../..
chown -R bb .
su bb
cd
cd bbc1.9e-btf/src

#make; make install
cd ..
./runbb.sh start
more BBOUT
(look for errors)
exit

vi /etc/rc.local
su - bb -c "cd /home/bb/bbc1.9e-btf; ./runbb.sh start"

echo '/data 10.1.4.0/24(rw, no_root_squash,async,no_subtree_check)' >> /etc/exports

/etc/init.d/nfs-kernel-server restart

echo 'chmod o+r /var/log/messages' >> /etc/cron.weekly/sysklogd

echo '10.1.4.8 backup1' >> /etc/hosts
echo '/dev/sdb1 /data ext3 rw,noatime 0 0' >> /etc/fstab

to install digi drivers:

wget http://ftp1.digi.com/support/driver/40002086_n.tgz
apt-get install linux-image-2.6.24-19-server
apt-get install linux-source-2.6.24 (not needed?)
apt-get install linux-headers-2.6.24-19-server
apt-get install make
apt-get install gcc
apt-get install g++
apt-get install libncurses5-dev
apt-get install expect
apt-get install libdbi-perl libdate-calc-perl libdbd-mysql-perl

cd /usr/src; ln -s linux-headers-2.6.24-19-server linux
./configure
make all
make install
make postinstall

/usr/bin/dgrp_cfg_node -v -v init el 65.116.11.2 8

apt-get install mysql

mkdir /data/mysql
chown mysql:mysql /data/mysql
/etc/init.d/mysql stop
mv /var/lib/mysql/* /data/mysql/
mv /data/mysql/ib_* /var/lib/mysql/
vi /etc/mysql/my.cnf
(change datadir to /data/mysql)
vi /etc/apparmor.d/usr.sbin.mysqld
add:
/data/mysql/ r,
/data/mysql/** rwk,
Comment out:
# /var/lib/mysql/ r,
# /var/lib/mysql/** rwk,

/etc/init.d/apparmor restart
/etc/init.d/mysql start

tw_cli /c0/u0 set ignoreECC=on
tw_cli /c0/u0 set storsave=balance
tw_cli /c0/u0 set cache=on

0 0 * * * /usr/local/sbin/3wraidchk
</pre>

= backup2 =

== Summary ==

This machine is used for archiving data and is a backup server for colo customers. It was the former primary backup location for all VPS-based customers before backup1 was installed. Only dedicated customers directly accesses this server to perform their backups. NOTE: power button is broken, so the reset button (paper clip) was rewired to be the power button.

* Location: castle, cab 3-7
* OS: FreeBSD 6.1 x86
* Networking: Priv IP: 10.1.4.3, Pub IP: 69.55.230.10 (firewalled from all but JC infrastructure @ i2b)
* Hardware: 16 IDE drive bays (4 columns of 4, drive 0-0 top left, drive 0-1 just to the right TODO) all hot-swap. Triple power supply.
* Drives:
**3ware 7500-8:
***200 GB JBOD (1 x 200G) labeled 0-0
***500 GB RAID5 (3 x 250G) 0-1 thru 0-3
***700 GB RAID5 (4 x 250G) 0-4 thru 0-7
**3ware 7500-8:
***700 GB RAID5 (4 x 250G) 1-0 thru 1-3
***700 GB RAID5 (4 x 250G) 1-4 thru 1-7

All drives MUST be western digital IDE drives. Other brands will not fit.

In case of an outage, nfs will hang on all connected servers until the nfs service returns. If you can't get backup2 back online, you can get nfs running elsewhere and fake backup2's MAC's: priv: 00:0e:0c:59:c1:a6, pub: 00:07:e9:5b:c6:45

To configure:
ifconfig fxp0 link 00:90:27:f9:0a:d9

== Services provided ==
* backup via rsync and nfs
* samba
* nfs
* snmp
* bigbrother

== Usage ==
* all data is stored under 4 mount points, corresponding to the 4 large RAID5 arrays: <tt>/mnt/data1 /mnt/data2 /mnt/data3 /mnt/data4</tt>
* iso images provided for customers wanting to mount an ISO as a CDROM via the IPKVM are provided via samba on this server. Images live under <tt>/mnt/data2/iso</tt>
* this used to be our primary backup server so you will see old backups from virt and jails around- missing customer data though, just the machine's data
* this server serves as an archive for exported db data from bwdb and old flow files.
* isys backs up here
* customers are nfs-moutned under /mnt/data3/customers as file-backed md devices
* in <tt>/mnt/data4</tt> there are lots of useful things used for building our vps servers, customer servers, and management scripts:
** <tt>/bin</tt>: the master repository of scripts and custom binaries we use on jails and virts. Each night every virt and jail rsync's what's in here to update the local files. So any global updates to scripts would need to be made here (or will be overwritten with what's in here)
** <tt>/build</tt>: files we use for setting up big brother, 3ware cli and scripts for colo's, vzcp customized setup files and so on
** <tt>/vzrpms</tt>: contains the OS templates for many-to-most of the OS's we offer on vz systems

== Cronjobs ==
* backs itself up nightly to nfs-mounted backup1 (mountpoint: <tt>/backup2</tt>)

== Regular maintenance ==
*[[Routine_Maintenance#3ware|Check on health]]

= backup3 =
== Summary ==
This machine is used for archiving data, is a backup server for colo customers, runs a samba server to make available iso's to the IPKVMs, and allows us to connect to the digi serial multiplexer at i2b. Only dedicated customers directly accesses this server to perform their backups.

* Location: i2b, cab 6
* OS: Ubuntu 10.04.1 server amd64
* Networking: Priv IP: 10.1.2.3, Pub IPs: 69.55.229.4 AND 69.55.231.2
* Hardware: 16 drive SATA bays (4 columns of 4, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: 5 TB (6 x 1TB) RAID5 array running on an Areca Technology Corp. ARC-1160 16-Port

== Services provided ==
* backup via rsync and nfs
* samba
* nfs
* digi realport
* snmp
* bigbrother

== Usage ==
* all data is stored under /data
* iso images provided for customers wanting to mount an ISO as a CDROM via the IPKVM are provided via samba on this server. Images live under <tt>/data/iso</tt>
* this server serves as an archive for exported db data from bwdb and old flow files.
* inftrastructure machines at i2b back up here
* customers are nfs-moutned under /data/customers as file-backed loopback devices

== management scripts ==
* mkbackups

== Cronjobs ==
0 0 * * * /usr/local/sbin/arecaraidchk
RAID checks

35 4 * * * /usr/local/sbin/snapshot_archive
Rotate daily snapshots for infrastructure machine backups

== Regular maintenance ==
*[[Routine_Maintenance#Areca|Check on RAID health]]

== Build ==

=== BIOS Config ===
disable quiet boot

set to last state after power loss

set date/time to GMT

enable serial console output (baud rate 115200)

=== Install OS ===
<pre>Ubuntu 10.04.1 amd64 (couldn't get 12.04 to load cause the H/W was incompat)
10G / ext3
2G swap
~ /data ext4

Install packages:
openssh
samba</pre>

=== DNS and private IP ===

echo "nameserver 69.55.225.225" >> /etc/resolv.conf

Add a 2nd IP to eth0 and setup priv net
<pre>vi /etc/network/interfaces

auto eth0
iface eth0 inet static
address 69.55.229.4
netmask 255.255.255.0
network 69.55.229.0
broadcast 69.55.229.255
gateway 69.55.229.1
# dns-* options are implemented by the resolvconf package, if installed
dns-nameservers 69.55.229.3 66.181.0.2
dns-search johncompanies.com

auto eth0:1
iface eth0:1 inet static
address 69.55.231.2
netmask 255.255.255.0
network 69.55.231.0
broadcast 69.55.231.255

auto eth1
iface eth1 inet static
address 10.1.2.3
netmask 255.255.255.0
network 10.1.2.0
broadcast 10.1.2.255

</pre>

=== Install packages ===
<pre>apt-get update
apt-get upgrade
apt-get install gcc
apt-get install libssl-dev
apt-get install libncurses5-dev
apt-get install cu
apt-get install unzip
apt-get install snmp snmpd ntp nfs-kernel-server</pre>

=== tweak grub, enable serial ===

<pre>vi /etc/default/grub
#GRUB_HIDDEN_TIMEOUT=0
GRUB_CMDLINE_LINUX_DEFAULT="max_loop=64"
GRUB_CMDLINE_LINUX="console=tty0 console=ttyS0"
update-grub</pre>

<pre>echo "start on stopped rc RUNLEVEL=[2345]
stop on runlevel [!2345]
respawn
exec /sbin/getty -L ttyS0 38400 vt102" > /etc/init/ttyS0.conf</pre>

=== install realport (digi) driver ===

give the digi an ip with DgIpServ.exe

<pre>cd /usr/src/
wget ftp://ftp1.digi.com/support/beta/linux/dgrp/dgrp-1.9.tgz
tar xzf dgrp-1.9.tgz
cd dgrp-1.9/
./configure
make
make install
make postinstall
update-rc.d dgrp_daemon defaults</pre>

configure ports:
dgrp_cfg_node init el 10.1.2.10 16

try connecting with:
cu -l /dev/ttyel00 -s 38400

=== shell, ntp, ssh key, hosts ===

Shell autocompletion search:
<pre>echo "\"\e[5~\": history-search-backward" >> ~/.inputrc
echo "\"\e[6~\": history-search-forward" >> ~/.inputrc</pre>

Setup ntp:
<pre>vi /etc/ntp.conf
server 10.1.2.1
server ntp.ubuntu.com</pre>

Generate ssh keys:
<pre>cd /root/
ssh-keygen -t dsa</pre>
Defaults, no password

Setup hosts:
<pre>echo "69.55.230.10 backup2" >> /etc/hosts
echo "69.55.230.11 backup1" >> /etc/hosts
echo "10.1.2.4 bwdb2" >> /etc/hosts
echo "10.1.2.3 backup3" >> /etc/hosts</pre>

Copy keys to servers where we need passwordless login:
<pre>cat .ssh/id_dsa.pub | ssh backup2 'cat - >> /root/.ssh/authorized_keys'
cat .ssh/id_dsa.pub | ssh backup1 'cat - >> /root/.ssh/authorized_keys'</pre>

Setup shell:
<pre>vi /root/.bashrc
(add to bottom)
alias h='history'
alias vi='vim'
alias j='jobs'
export PS1="[\u@\h \w]# "
alias dr='screen -dr'
export EDITOR=vim
export GREP_OPTIONS='--color=auto'
export HISTFILESIZE=1000

alias tip-switch-p20='cu -l ttyel00 -s 9600'
alias tip-switch-p21='cu -l ttyel15 -s 9600'
alias tip-switch-p22='cu -l ttyel14 -s 9600'
alias tip-switch-p23='cu -l ttyel05 -s 9600'
alias tip-switch-p24='cu -l ttyel06 -s 9600'
alias tip-switch-p25='cu -l ttyel09 -s 9600'
alias tip-switch-p26='cu -l ttyel07 -s 9600'
alias tip-switch-p27='cu -l ttyel08 -s 9600'
alias tip-firewall2='cu -l ttyel01 -s 115200'
alias tip-nat2='cu -l /dev/ttyel02 -s 115200'
alias tip-backup3='cu -l ttyel04 -s 38400'
alias tip-bwdb2='cu -l ttyel03 -s 115200'
alias tip-backup4='cu -l ttyel13 -s 115200'
alias tip-jail3='cu -l ttyel11 -s 115200'

Load new shell:
source /root/.bashrc

Setup snmpd (this is only valid for a server at castle):
echo 'rocommunity jcread 10.1.4.5
rocommunity jcread 10.1.4.3
agentaddress 10.1.4.8:161' > /etc/snmp/snmpd.conf

to see which iface it is, on backup2:

snmpwalk -v 1 -c jcread 10.1.4.8 interface

=== nfs ===

Allow mounts from private net:
echo '/data 10.1.2.0/24(rw,no_root_squash,async,no_subtree_check)' >> /etc/exports

Restart nfsd:
/etc/init.d/nfs-kernel-server restart

=== bb ===

Add user, group:
echo "bb:x:1984:1984:Big Brother:/home/bb:/bin/bash" >> /etc/passwd
echo "bb:x:1984:" >> /etc/group
pwconv

Create home:
mkdir /home/bb
chown bb.bb /home/bb
cd ~bb

Copy over and install files:
<pre>scp backup2:/mnt/data4/build/bb/bb-linux.tar .
tar xf bb-linux.tar
cd /home/bb/bbc1.9e-btf/etc</pre>

Configure main bb server:
echo "69.55.230.2 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh" > /home/bb/bbc1.9e-btf/etc/bb-hosts
echo "10.1.2.3 backup3.johncompanies.com # ssh" >> /home/bb/bbc1.9e-btf/etc/bb-hosts

Configure low disk alerts:
<pre>echo "/:90:95
/var:90:95
/data:85:99" > /home/bb/bbc1.9e-btf/etc/bb-dftab</pre>

vi /home/bb/bbc1.9e-btf/bin/bb-disk.sh
(remove all | SORT xxxx since SORT is broken)

chmod +r /var/log/messages

./bbchkcfg.sh
(y to questions)
./bbchkhosts.sh
(ignore ssh errors)
<pre>cd ../..
chown -R bb .
su bb
cd
cd bbc1.9e-btf/src</pre>

<pre>make; make install
cd ..
./runbb.sh start
more BBOUT</pre>
(look for errors)
exit

<pre>vi /etc/rc.local
su - bb -c "cd /home/bb/bbc1.9e-btf; ./runbb.sh start"
</pre>
(before the exit 0)

echo 'chmod o+r /var/log/messages' >> /etc/cron.weekly/sysklogd

Add f/w rule:
ipfw add 00096 allow ip from { 69.55.229.4 or 69.55.229.3 } to 69.55.230.2 1984

<pre>vi ~bb/bbc1.9e-btf/etc/bbdef-client.sh
DFWARN=199
DFPANIC=199</pre>

=== raid check ===

==== 3ware ====
<pre>
scp backup1:/usr/local/sbin/tw_cli /usr/local/sbin/tw_cli
scp backup1:/usr/local/sbin/checkraid.sh /usr/local/sbin/checkraid.sh
scp backup1:/usr/local/sbin/3wraidchk /usr/local/sbin/3wraidchk
vi /usr/local/sbin/checkraid.sh
:%s/c0/c2/g

crontab -e
0 0 * * * /usr/local/sbin/3wraidchk</pre>

==== areca ====
<pre>
cd /tmp
wget http://www.areca.us/support/s_linux/cli/linuxcli_V1.10.0_120815.zip
unzip linuxcli_V1.10.0_120815.zip
cp linuxcli_V1.10.0_120815/x86_64/cli64 /usr/local/sbin/
chmod 0700 /usr/local/sbin/cli64
cli64 rsf info
</pre>

<pre>scp backup2:/data4/bin/arecaraidchk /usr/local/sbin
scp backup1:/usr/local/sbin/Sendmail.pm /usr/local/sbin

crontab -e
0 0 * * * /usr/local/sbin/arecaraidchk
</pre>

<pre>cat > /root/verify.sh
cli64 vsf info
cli64 rsf info
cli64 disk info
cli64 event info
echo press enter when ready to run verify ; read x

cli64 vsf check vol=1
</pre>

=== misc binaries ===

scp backup1:/usr/local/sbin/snapshot_archive /usr/local/sbin/snapshot_archive
vi /usr/local/sbin/snapshot_archive
(remove entries)

crontab -e
35 4 * * * /usr/local/sbin/snapshot_archive

scp backup1:/usr/local/sbin/pagedave /usr/local/sbin/pagedave
scp backup1:/usr/local/sbin/taskdone /usr/local/sbin/taskdone

Since installing /bin/mail requires all sorts of packages (lame) we write a simple one here...which can only email johncompanies.com addr's unless you add relaying for this host:

<pre>
cat > /bin/mail
#!/usr/bin/perl
use strict;
use warnings;

use lib '/usr/local/sbin';
use Sendmail qw(sendmail);

my $sub = $ARGV[1];
my $to = $ARGV[2];

my %mail = (
To => $to,
From => $to,
Subject => $sub,
Message => '',
smtp => 'mail.johncompanies.com'
);
sendmail(%mail) || print "Error: $Sendmail::error";

</pre>

chmod 0700 /bin/mail

=== mkbackup ===

mkdir /data/customers

<pre>cat > /usr/local/sbin/mkbackup
#!/bin/sh

if test $1; then
cid=$1
else
echo "ERROR: Usage: mkbackup cid GB ip Terminating."
exit
fi

if test $2; then
gb=$2
else
echo "ERROR: Usage: mkbackup cid GB ip Terminating."
exit
fi

if test $3; then
ip=$3
else
echo "ERROR: Usage: mkbackup cid GB ip Terminating."
exit
fi

if test -e /data/customers/${cid}-file; then
echo "ERROR: /data/customers/${cid}-file exists"
exit
else
echo "touch /data/customers/${cid}-file"
touch /data/customers/${cid}-file
count=`echo $gb|awk '{print $1*1000}'`
echo "dd if=/dev/zero of=/data/customers/${cid}-file bs=1024K count=$count"
dd if=/dev/zero of=/data/customers/${cid}-file bs=1024K count=$count
echo "/sbin/mkfs -t ext3 -F -j -q /data/customers/${cid}-file"
/sbin/mkfs -t ext3 -F -j -q /data/customers/${cid}-file
fi

if test -e /data/customers/$cid; then
echo "ERROR: /data/customers/$cid exists"
exit
else
echo "mkdir /data/customers/${cid}"
mkdir /data/customers/${cid}
echo "mount -o loop /data/customers/${cid}-file /data/customers/$cid"
mount -o loop /data/customers/${cid}-file /data/customers/$cid
df -h /data/customers/$cid

echo "fsck -y /data/customers/${cid}-file" >> /etc/nfs_backup_mounts.sh
echo "mount -o loop /data/customers/${cid}-file /data/customers/$cid" >> /etc/nfs_backup_mounts.sh
echo "" >> /etc/nfs_backup_mounts.sh

echo "/data/customers/$cid $ip/32(rw,no_root_squash,async,no_subtree_check)" >> /etc/exports
/etc/init.d/nfs-kernel-server restart
tail /var/log/messages
fi</pre>

chmod 0700 /usr/local/sbin/mkbackup

vi /etc/rc.local
add:
/etc/nfs_backup_mounts.sh

=== samba ===

apt-get install samba

vi /etc/samba/smb.conf

; comment out any mounts, add:

<pre>[data]
read only = yes
locking = no
path = /data/iso
guest ok = yes</pre>

/etc/init.d/smbd restart

mkdir /data/iso

Bring over some stuff from backup2

<pre>cd /data/iso
scp backup2:/d2/iso/3wfirmware.iso .
scp backup2:/d2/iso/MD5SUMS .
scp backup2:/d2/iso/bootimg.iso .
scp backup2:/d2/iso/systemrescuecd-x86-0.2.19.iso .
scp backup2:/d2/iso/win98bootcd.iso .
scp backup2:/d2/iso/acronis_bootdisk.iso .
scp backup2:/d2/iso/memtest86-3.2.iso .</pre>

=== Moving from one server to another ===

Here are the steps you would take to move settings and data from one server to a new backup server:

* rsync over all /data/customers (we do this cause if we didn't use *-file it would copy over the files AND the data in the mountpoint)
rsync -av --progress --ignore-times *-file root@10.1.2.33:/data/customers/
after umounting all the customers, copy over the (empty) directories separately:
for f in `find . -type d`; do rsync -av $f root@69.55.229.25:/data/customers; done

* copy mount script
[root@backup3 /data/customers]# scp /etc/nfs_backup_mounts.sh root@69.55.229.25:/etc/nfs_backup_mounts.sh

* copy rc.local
[root@backup3 /data/customers]# scp /etc/rc.local root@69.55.229.25:/etc/rc.local

* copy /etc/exports
[root@backup3 /data/customers]# scp /etc/exports root@69.55.229.25:/etc/exports

* edit /etc/hostname on both machines (set current to oldbackup3)

* edit /etc/network/interfaces (swap IPs).

* stop mounts from mounting on old and new servers so it doesnt start with reboot right away:
chmod 000 /etc/nfs_backup_mounts.sh

* reboot both servers @ same time

* check everything out

* run /etc/nfs_backup_mounts.sh on new server

* if switch port changed update mrtg to reflect correct port pub nic is on (on p20):
vi /usr/local/www/mgmt/mrtg/mrtg1.cfg

= console =

== Summary ==
This box's only purpose is to serve as a means to connect to the digi serial multiplexer boxes at castle. Connect to it using the blue (cisco) ribbon cable with the beige RJ-45 to serial connector, 9600 8N1.

* Location: castle, cab 3-8
* OS: SunOS 5.8 (solaris)
* Networking: Priv IP: 10.1.4.4
* Hardware: Sun Netra

To connect to consoles, ssh in as user 'console' and use the <tt>tip</tt> command to connect to devices listed in <tt>/etc/remote</tt>

i.e.
tip switch-p1
tip jail1

== Configuring digi/ports ==

=== /etc/remote ===
This is where the configuration/mapping for ports and custom names which we use along with the tip command to connect to various ports on the digi switches.

We have 2 digi's at castle we connect to:

#3-7 10.1.4.10
virt15:dv=/dev/dty/CO001s:br#38400:el=^C^S^Q^U^D:ie=%$:oe=^D:
virt13:dv=/dev/dty/CO002s:br#115200:el=^C^S^Q^U^D:ie=%$:oe=^D:hf:

and

#3-6 10.1.4.11
jail4:dv=/dev/dty/CP001s:br#9600:el=^C^S^Q^U^D:ie=%$:oe=^D:hf:
jail16:dv=/dev/dty/CP002s:br#9600:el=^C^S^Q^U^D:ie=%$:oe=^D:hf:

The only things you need to edit are the first part (i.e. <tt>jail4</tt>) and the speed (i.e. <tt>9600</tt>). You can decipher which port on the digi each line corresponds to by the <tt>CP001s or CO001s</tt> (port 1 on digi1 and digi2), <tt>CP002s or CO002s</tt> (port 2 on digi1 and digi2)

=== drpadmin ===
The tool you use to configure a device to a digi box is drpadmin:

<pre>bash-2.03$ su
Password:
# drpadmin

Please select an option (a)dd (d)elete (s)how (r)eset (q)uit : s
0 10.1.4.10 32 CO 771 never 1027
1 10.1.4.11 32 CP 771 never 1027
2 65.116.11.2 8 el 771 never 1027

Please select an option (a)dd (d)elete (s)how (r)eset (q)uit :</pre>

Use those commands above to modify the devices available.

== Switching IP/hostname ==

Edit:
/etc/defaultrouter
/etc/hosts
/etc/hostname.hme0
/etc/nodename
Maybe needed to run: # ifconfig hme0 10.1.4.4 up

= devweb =

We do web development on devweb.johncompanies.com

Currently this is a jail running on jail17 / 69.55.230.8

If the jail is restarted, you will need to manually restart the web service with:
httpsdctl restart

All website development work should be done here first. It works exactly like and is setup like our [[Management_System_/_Public_Website_/_Signup_/_Account_Manager|main site]].

= firewall2 =

== Summary ==

This machine is the primary firewall for the entire network at i2b. firewall3 is a hot standby replacement for
firewall2. Both firewall2 and firewall3 should not be connected at the same time since they use the same internal
and external IP addresses.

* Location: i2b, cab 6
* OS: FreeBSD 6.4 x86
* Networking: Priv IP: 10.1.2.2, Pub IPs: 66.181.18.3 (external), 69.55.229.1 & 69.55.231.1 (internal). It has 3 network connections (2 onboard, 1 PCI) connecting to the external, internal and private networks. TODO: describe NIC location/orientation

* Hardware: 2 SCSI SCA drive bays (2 columns of 3, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: 73 GB (2 x 73GB) RAID1 array running on an LSI MegaRAID SCSI 320 PCI RAID card.

== Services Provided ==
* firewall (ipfw)
* bigbrother for customer machines

== Firewall Rule Configuration ==

See [[FreeBSD_Reference#Firewall_Rule_Configuration|Firewall Rule Configuration]] for more discussion on how to actually manipulate firewall rules.

== Disaster Recovery ==

TODO: need backup f/w and instructions on how to move cables.

Here's what you need to put in /etc/rc.conf to get a firewall going (as far as routes and IPs)

TODO

Here's the config on the live firewall:

<pre>kern_securelevel_enable="NO"
portmap_enable="NO"
sendmail_enable="NO"
usbd_enable="YES"
gateway_enable="YES"

xntpd_enable="YES"
nfs_client_enable="YES"
nfs_reserved_port_only="YES"
inetd_enable="YES"
inetd_flags="-wW -a 10.1.2.2"

fsck_y_enable="YES"
background_fsck="NO"

defaultrouter="66.181.18.2"
hostname="firewall2.johncompanies.com"
ifconfig_bge0="inet 66.181.18.3 netmask 255.255.255.224"
ifconfig_bge1="inet 69.55.229.1 netmask 255.255.255.0"
ifconfig_bge1_alias0="inet 69.55.231.1 netmask 255.255.255.0"
ifconfig_bge1_alias1="inet 65.50.228.1 netmask 255.255.255.0"
ifconfig_bge1_alias2="inet 65.50.229.1 netmask 255.255.255.0"
ifconfig_bge1_alias3="inet 65.50.230.1 netmask 255.255.255.0"
ifconfig_bge1_alias4="inet 65.50.231.1 netmask 255.255.255.0"
ifconfig_bge1_alias5="inet 65.50.232.1 netmask 255.255.255.0"
ifconfig_bge1_alias6="inet 65.50.233.1 netmask 255.255.255.0"
ifconfig_bge1_alias7="inet 65.50.234.1 netmask 255.255.255.0"
ifconfig_bge1_alias8="inet 65.50.235.1 netmask 255.255.255.0"
ifconfig_fxp0="inet 10.1.2.2 netmask 255.255.255.0"
sshd_enable="YES"
usbd_enable="YES"
</pre>

== Cronjobs ==
30 3 * * * /usr/local/etc/rsync.backup
Backup to backup3

0 0 1 * * /sbin/ipfw zero
0 0 1 * * /sbin/ipfw del 3 4 5
Reset counters and remove pipe rules on the 1st of the month. Pay attention when setting up a rule as 3 4 5 (that's not a temporary traffic cap).

59 23 30 * * /sbin/ipfw show > /tmp/ipfw_count
3 0 30 * * /sbin/ipfw show > /tmp/ipfw_count
Capture counts periodically

0 3 * * * /etc/makefwrules.pl; /etc/makepiperules.pl;
This script will create <tt>/etc/firewall.sh</tt> which contains all the firewall and pipe rules in place at the time the script was run.

*/5 * * * * /usr/local/sbin/lsiraidchk
Checking the health of the RAID array

== DOS attacks ==

See [[FreeBSD_Reference#Handling_a_DoS_attack|Handling a DoS attack]] regarding how to handle a DOS attack.

== build ==

<pre>partition map:
/ 58g
swap 4g
/var 512m
/tmp 512m
/usr 5.5g

4. edit /etc/make.conf
echo "WITHOUT_X11=yes \
KERNCONF=firewall2 \
BOOT_COMCONSOLE_SPEED=115200" >> /etc/make.conf

5. add settings to /boot/loader.conf and /boot.config

echo "-Dh" >> /boot.config

echo 'console="comconsole,vidconsole" \
boot_multicons="YES" \
boot_serial="YES" \
comconsole_speed="115200"' >> /boot/loader.conf

6. turn off all ttyv's except 0 and 1 in /etc/ttys
also turn on ttyd0, change type to vt100:
vi /etc/ttys

ttyv2 "/usr/libexec/getty Pc" cons25 off secure
ttyv3 "/usr/libexec/getty Pc" cons25 off secure
ttyv4 "/usr/libexec/getty Pc" cons25 off secure
ttyv5 "/usr/libexec/getty Pc" cons25 off secure
ttyv6 "/usr/libexec/getty Pc" cons25 off secure
ttyv7 "/usr/libexec/getty Pc" cons25 off secure
# Serial terminals
# The 'dialup' keyword identifies dialin lines to login, fingerd etc.
ttyd0 "/usr/libexec/getty std.9600" vt100 on secure

kill -1 1

on console server:
vi /etc/remote
(rename port to jail8 depending on where and which digi plugged into)
test serial console

7. populate hosts
echo "69.55.230.10 backup2" >> /etc/hosts
echo "69.55.230.11 backup1" >> /etc/hosts
echo "10.1.2.3 backup3" >> /etc/hosts

8. put key in authorized_keys on backup3
cd
ssh-keygen -t dsa -b 1024
(default location, leave password blank)

Punch a hole in firewall1 to allow traffic to backup servers @ castle:

ipfw add 99 allow ip from 66.181.18.0/27 to 69.55.230.10 22
ipfw add 95 allow ip from 66.181.18.0/27 to 69.55.230.11 22

cat /root/.ssh/id_dsa.pub | ssh backup3 'cat - >> /root/.ssh/authorized_keys'
cat /root/.ssh/id_dsa.pub | ssh backup2 'cat - >> /root/.ssh/authorized_keys'
cat /root/.ssh/id_dsa.pub | ssh backup1 'cat - >> /root/.ssh/authorized_keys'

confirm that you can ssh to backup3 and backup 2 without getting a login prompt

ssh backup3 hostname

ssh backup2 hostname

ssh backup1 hostname

10. edit root's path and login script:
vi /root/.cshrc

Change alias entries (add G):
alias la ls -aG
alias lf ls -FAG
alias ll ls -lAG
alias ls ls -AG
alias mbm mb mount
alias mbu mb umount

and alter the prompt, set the following:
set prompt = "`/bin/hostname -s` %/# "

11. install cvsup
cd /usr/ports/net/cvsup-without-gui
make install clean; rehash; mail -s 'cvs installed' dave.boodman@vtext.com < /dev/null

12. get latest sources for this release:
cd /usr/src
echo "*default host=cvsup4.freebsd.org\
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_6_4\
*default delete use-rel-suffix\
*default compress\
src-all" > sup

cvsup sup ; mail -s 'cvs sup done' dave.boodman@vtext.com < /dev/null

13. configure new kernel.

cd /usr/src/sys/i386/conf
scp backup2:/mnt/data4/build/freebsd/firewall2-6.4 ./firewall2

15. build, install kernel and world

cd /boot

mv kernel kernel.GENERIC
cd kernel.GENERIC
cd /usr/src
make buildkernel installkernel

make buildworld ; mail -s 'buildworld done' dave.boodman@vtext.com < /dev/null
(2450: 1:56min, supermicro: 59mins, 2950: 38mins)
make installworld
(2450: 3min, supermicro: 1min, 2950: :34)
mergemaster -i

17. populate /etc/rc.conf with IPs and NFS settings
vi /etc/rc.conf

kern_securelevel_enable="NO"
portmap_enable="NO"
sendmail_enable="NO"
usbd_enable="YES"
gateway_enable="YES"

xntpd_enable="YES"
nfs_client_enable="YES"
nfs_reserved_port_only="YES"
inetd_enable="YES"
inetd_flags="-wW -a 10.1.2.2"

ifconfig_bce1="inet 10.1.2.2 netmask 255.255.255.0"
fsck_y_enable="YES"
background_fsck="NO"

defaultrouter="66.181.18.2"
hostname="firewall2.johncompanies.com"
ifconfig_bge0="inet 66.181.18.3 netmask 255.255.255.224"
ifconfig_bge1="inet 69.55.229.1 netmask 255.255.255.0"
ifconfig_fxp0="inet 10.1.2.2 netmask 255.255.255.0"
sshd_enable="YES"
usbd_enable="YES"

20. reboot. Confirm new kernel is loaded

uname -a

21. update ports:
cd /usr/ports
echo "*default host=cvsup4.FreeBSD.org\
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_6_4\
*default delete use-rel-suffix\
*default compress\
ports-all tag=." > sup

cvsup sup; mail -s 'cvs sup ports done' dave.boodman@vtext.com < /dev/null

22. Install raid mgmt tool

# linux base
cd /usr/ports/devel/libtool22
make install base

cd /usr/ports/emulators/linux_base-fc4
make install clean

#linux-megamgr-5.20
cd /usr/ports/sysutils/linux-megamgr
make install clean

# megarc-1.51
cd /usr/ports/sysutils/megarc
make install clean

Test:
rehash; megarc -ldInfo -a0 -l0

23. install rsync from ports
cd /usr/ports/net/rsync
make install clean

choose default options

25. install bb client
adduser
Username: bb
Full name: bb
Uid (Leave empty for default): 1984
Login group [bb]:
Login group is bb. Invite bb into other groups? []:
Login class [default]:
Shell (sh csh tcsh nologin) [sh]:
Home directory [/home/bb]:
Use password-based authentication? [yes]:
Use an empty password? (yes/no) [no]:
Use a random password? (yes/no) [no]: yes
Lock out the account after creation? [no]:
Username : bb
Password : <random>
Full Name : bb
Uid : 1984
Class :
Groups : bb
Home : /home/bb
Shell : /bin/sh
Locked : no
OK? (yes/no): yes

cd /usr/home/bb
scp backup2:/mnt/data4/build/bb/bb-freebsd.tar .
tar xvf bb-freebsd.tar

edit /home/bb/bbc1.9e-btf/etc/bb-hosts with something like:
echo "69.55.230.2 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh \
10.1.2.1 firewall2.johncompanies.com # ssh" > /home/bb/bbc1.9e-btf/etc/bb-hosts

vi /home/bb/bbc1.9e-btf/ext/openfiles
MACHINE="firewall2,johncompanies,com" # HAS TO BE IN A,B,C FORM

cd /usr/home/bb/bbc1.9e-btf/etc
./bbchkcfg.sh
(y to questions)
./bbchkhosts.sh
(ignore ssh errors)
cd ../..
chown -R bb .
su bb
cd
cd bbc1.9e-btf/src
make; make install
cd ..

vi /usr/home/bb/bbc1.9e-btf/bin/bb-cpu.sh
$1 $TOPARGS > $BBTMP/TOP.$$
# /usr/local/jail/bin/jtop > $BBTMP/TOP.$$

./runbb.sh start
more BBOUT
(look for errors)
exit

echo 'su - bb -c "cd /home/bb/bbc1.9e-btf; ./runbb.sh start"' > /usr/local/etc/rc.d/bb.sh
chmod +x /usr/local/etc/rc.d/bb.sh

Punch a hole in the firewall to allow it to communicate with bb monitor:

ipfw add 00096 allow ip from 66.181.18.0/27 to 69.55.230.2

27. configure bb on mail:
vi /usr/home/bb/bbsrc/bb1.9e-btf/etc/bb-hosts
66.181.18.3 firewall2.johncompanies.com # ssh

su bb
cd
bbsrc/bb/runbb.sh restart ; exit

29. configure ntp
echo "server 10.1.2.1" > /etc/ntp.conf

/usr/sbin/ntpd -p /var/run/ntpd.pid
sleep 2; ntpq -p
(confirm it’s able to reach our time server)

echo '/usr/sbin/ntpd -c /etc/ntp.conf -p /var/run/ntpd.pid -f /var/db/ntpd.drift' > /usr/local/etc/rc.d/ntp.sh
chmod 0700 /usr/local/etc/rc.d/ntp.sh

30. fwd and reverse lookups on ns1c
vr johncompanies.com
(edit the PTR too)

33. setup backups
echo '#\!/bin/sh\
backupdir=/data/firewall2/current\
\
## ENTRY /etc ' > /usr/local/etc/backup.config

on backup3:
setup backup dirs:
ssh backup3 mkdir -p /data/firewall2/current

on backup3, add the system to
vi /usr/local/sbin/snapshot_archive

scp 69.55.230.2:/usr/local/etc/rsync.backup /usr/local/etc/rsync.backup
vi /usr/local/etc/rsync.backup
backup1 > backup3

crontab -e
1 0 * * * /usr/local/etc/rsync.backup

34. mkdir /root/logs

35. edit sshd_config for security
vi /etc/ssh/sshd_config
ListenAddress 66.181.18.3
ListenAddress 10.1.2.1

kill -1 `cat /var/run/sshd.pid`

35. raid chk

cat > /usr/local/sbin/lsiraidchk
#!/usr/bin/perl

my @out = split "\n", `megarc -ldInfo -a0 -Lall|grep Status:`;

foreach (@out) {
if ($_ =~ /DEGRADED/) { $date = `date`; chomp $date; `echo "$date: RAID ARRAY DEGRADED" >> /var/log/messages`; }
#print $_;
}

36. add crontab entries
crontab -e
30 3 * * * /usr/local/etc/rsync.backup
0 0 1 * * /sbin/ipfw zero
0 0 1 * * /sbin/ipfw del 3 4 5
59 23 30 * * /sbin/ipfw show > /tmp/ipfw_count
3 0 30 * * /sbin/ipfw show > /tmp/ipfw_count
0 3 * * * /etc/makefwrules.pl; /etc/makepiperules.pl;
*/5 * * * * /usr/local/sbin/lsiraidchk

#10 0 * * * rm /var/spool/clientmqueue/*

scp /etc/makefwrules.pl user@64.163.14.48:~
scp /etc/makepiperules.pl user@64.163.14.48:~
mv /home/user/makefwrules.pl /etc
mv /home/user/makepiperules.pl /etc
touch /etc/firewall.sh
mkdir /etc/oldrules/

other binaries

scp /usr/local/bin/rulemaker user@64.163.14.48:~
mv ~user/rulemaker /usr/local/sbin
scp ~user/Sendmail.pm user@64.163.14.48:~
scp ~user/doswatch.pl user@64.163.14.48:~

Setup basic ruleset

ipfw add 00009 count udp from any to any
ipfw add 00010 allow tcp from any to any established
ipfw add 00012 deny tcp from any to any tcpflags syn tcpoptions !mss
ipfw add 00012 deny icmp from any to any icmptypes 4,5,9,10,12,13,14,15,16,17,18
ipfw add 00012 deny tcp from any to any tcpflags syn,fin
ipfw add 00012 deny tcp from any to any tcpflags fin,psh,rst,urg
ipfw add 00012 allow icmp from any to any
ipfw add 00014 deny tcp from any to any dst-port 135
ipfw add 00150 skipto 65535 ip from any to any via em1 in

IPKVM3:
00098 allow ip from { 69.55.230.6 or 69.55.230.7 } to 69.55.230.10 dst-port 139
00098 deny ip from any to 69.55.230.10 dst-port 139
</pre>

= firewall3 =

== Summary ==

This machine is the backup firewall for the network at i2b.

* Location: i2b, cab ?
* OS: FreeBSD 9.1 amd64
* Networking: Priv IP: 10.1.2.5, Pub IPs: 66.181.18.3 (external), 69.55.229.1 & 69.55.231.1 (internal). It has 3 network connections (2 onboard, 1 PCI) connecting to the external, internal and private networks.

<pre>

The internal network NIC is the left one on the motherboard (69.55.229.1/24, ...).
The external network NIC is the right one on the motherboard (66.181.18.3/28).
The PCI ethernet card is connected to our private network (10.1.2.5/24).

</pre>

* Hardware: 2 SCSI SCA drive bays (2 columns of 3, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: 160 GB (2 x 160GB) RAID1 array running on an LSI MegaRAID SCSI 320 PCI RAID card.

== Services Provided ==
* firewall (ipfw)
* bigbrother

== Firewall Rule Configuration ==

See [[FreeBSD_Reference#Firewall_Rule_Configuration|Firewall Rule Configuration]] for more discussion on how to actually manipulate firewall rules.

== Disaster Recovery ==

'''To put the backup firewall3 into service:'''

<pre>
Move the internal cable (to our networks) from firewall2 to em1 which is the left most ethernet port (69.55.229.1).
Move the external cable (to outside world) from firewall2 to em0 which is the port to the right on the motherboard (66.181.18.3).
The PCI ethernet port (fxp0) should already be connected to private network (10.1.2.5).
</pre>

Here's what you need to put in /etc/rc.conf to get a firewall going (as far as routes and IPs)

<pre>
kern_securelevel_enable="NO"
portmap_enable="NO"
sendmail_enable="NO"
usbd_enable="YES"
gateway_enable="YES"

xntpd_enable="YES"
nfs_client_enable="YES"
nfs_reserved_port_only="YES"
inetd_enable="YES"
inetd_flags="-wW -a 10.1.2.5"

fsck_y_enable="YES"
background_fsck="NO"

defaultrouter="66.181.18.2"
hostname="firewall3.johncompanies.com"
ifconfig_em0="inet 66.181.18.3 netmask 255.255.255.224"

ifconfig_em1="inet 69.55.229.1 netmask 255.255.255.0"
ifconfig_em1_alias0="inet 69.55.231.1 netmask 255.255.255.0"

# ifconfig_em1_alias1="inet 65.50.228.1 netmask 255.255.255.0"
# ifconfig_em1_alias2="inet 65.50.229.1 netmask 255.255.255.0"
# ifconfig_em1_alias3="inet 65.50.230.1 netmask 255.255.255.0"
# ifconfig_em1_alias4="inet 65.50.231.1 netmask 255.255.255.0"
# ifconfig_em1_alias5="inet 65.50.232.1 netmask 255.255.255.0"
# ifconfig_em1_alias6="inet 65.50.233.1 netmask 255.255.255.0"
# ifconfig_em1_alias7="inet 65.50.234.1 netmask 255.255.255.0"
# ifconfig_em1_alias8="inet 65.50.235.1 netmask 255.255.255.0"

ifconfig_fxp0="inet 10.1.2.5 netmask 255.255.255.0"

sshd_enable="YES"
usbd_enable="YES"
</pre>

== Cronjobs ==
30 3 * * * /usr/local/etc/rsync.backup
Backup to backup3

0 0 1 * * /sbin/ipfw zero
0 0 1 * * /sbin/ipfw del 3 4 5
Reset counters and remove pipe rules on the 1st of the month. Pay attention when setting up a rule as 3 4 5 (that's not a temporary traffic cap).

59 23 30 * * /sbin/ipfw show > /tmp/ipfw_count
3 0 30 * * /sbin/ipfw show > /tmp/ipfw_count
Capture counts periodically

0 3 * * * /etc/makefwrules.pl; /etc/makepiperules.pl;
This script will create <tt>/etc/firewall.sh</tt> which contains all the firewall and pipe rules in place at the time the script was run.

*/5 * * * * /usr/local/sbin/lsiraidchk
Checking the health of the RAID array

== DOS attacks ==

See [[FreeBSD_Reference#Handling_a_DoS_attack|Handling a DoS attack]] regarding how to handle a DOS attack.

== build ==

<pre>partition map:
/ 58g
swap 4g
/var 512m
/tmp 512m
/usr 5.5g

4. edit /etc/make.conf
echo "WITHOUT_X11=yes \
KERNCONF=firewall3 \
BOOT_COMCONSOLE_SPEED=115200" >> /etc/make.conf

5. add settings to /boot/loader.conf and /boot.config

echo "-Dh" >> /boot.config

echo 'console="comconsole,vidconsole" \
boot_multicons="YES" \
boot_serial="YES" \
comconsole_speed="115200"' >> /boot/loader.conf

6. turn off all ttyv's except 0 and 1 in /etc/ttys
also turn on ttyd0, change type to vt100:
vi /etc/ttys

ttyv2 "/usr/libexec/getty Pc" cons25 off secure
ttyv3 "/usr/libexec/getty Pc" cons25 off secure
ttyv4 "/usr/libexec/getty Pc" cons25 off secure
ttyv5 "/usr/libexec/getty Pc" cons25 off secure
ttyv6 "/usr/libexec/getty Pc" cons25 off secure
ttyv7 "/usr/libexec/getty Pc" cons25 off secure
# Serial terminals
# The 'dialup' keyword identifies dialin lines to login, fingerd etc.
ttyd0 "/usr/libexec/getty std.9600" vt100 on secure

kill -1 1

on console server:
vi /etc/remote
(rename port to jail8 depending on where and which digi plugged into)
test serial console

7. populate hosts
echo "69.55.230.10 backup2" >> /etc/hosts
echo "69.55.230.11 backup1" >> /etc/hosts
echo "10.1.2.3 backup3" >> /etc/hosts

8. put key in authorized_keys on backup3
cd
ssh-keygen -t dsa -b 1024
(default location, leave password blank)

Punch a hole in firewall1 to allow traffic to backup servers @ castle:

ipfw add 99 allow ip from 66.181.18.0/27 to 69.55.230.10 22
ipfw add 95 allow ip from 66.181.18.0/27 to 69.55.230.11 22

cat /root/.ssh/id_dsa.pub | ssh backup3 'cat - >> /root/.ssh/authorized_keys'
cat /root/.ssh/id_dsa.pub | ssh backup2 'cat - >> /root/.ssh/authorized_keys'
cat /root/.ssh/id_dsa.pub | ssh backup1 'cat - >> /root/.ssh/authorized_keys'

confirm that you can ssh to backup3 and backup 2 without getting a login prompt

ssh backup3 hostname

ssh backup2 hostname

ssh backup1 hostname

10. edit root's path and login script:
vi /root/.cshrc

Change alias entries (add G):
alias la ls -aG
alias lf ls -FAG
alias ll ls -lAG
alias ls ls -AG
alias mbm mb mount
alias mbu mb umount

and alter the prompt, set the following:
set prompt = "`/bin/hostname -s` %/# "

11. install cvsup
cd /usr/ports/net/cvsup-without-gui
make install clean; rehash; mail -s 'cvs installed' 8583619553@vtext.com < /dev/null

12. get latest sources for this release:
cd /usr/src
echo "*default host=cvsup4.freebsd.org\
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_9_1\
*default delete use-rel-suffix\
*default compress\
src-all" > sup

cvsup sup ; mail -s 'cvs sup done' 8583619553@vtext.com < /dev/null

13. configure new kernel.

cd /usr/src/sys/amd64/conf
scp backup2:/mnt/data4/build/freebsd/firewall3-9.1 ./firewall3

15. build, install kernel and world

cd /boot

mv kernel kernel.GENERIC
cd kernel.GENERIC
cd /usr/src
make buildkernel installkernel

make buildworld ; mail -s 'buildworld done' 8583619553@vtext.com < /dev/null
(supermicro: 2:15 mins, 2950: 38? mins)
make installworld
(2450: 3min, supermicro: 1min, 2950: :34)
mergemaster -i

17. populate /etc/rc.conf with IPs and NFS settings
vi /etc/rc.conf

kern_securelevel_enable="NO"
portmap_enable="NO"
sendmail_enable="NO"
usbd_enable="YES"
gateway_enable="YES"

xntpd_enable="YES"
nfs_client_enable="YES"
nfs_reserved_port_only="YES"

fsck_y_enable="YES"
background_fsck="NO"

hostname="firewall3.johncompanies.com"
# external network
ifconfig_em0="inet 66.181.18.3 netmask 255.255.255.224"

# internal network
ifconfig_em1="inet 69.55.229.1 netmask 255.255.255.0"
ifconfig_em1_alias0="inet 69.55.231.1 netmask 255.255.255.0"

ifconfig_em1_alias1="inet 65.50.228.1 netmask 255.255.255.0"
ifconfig_em1_alias2="inet 65.50.229.1 netmask 255.255.255.0"
ifconfig_em1_alias3="inet 65.50.230.1 netmask 255.255.255.0"
ifconfig_em1_alias4="inet 65.50.231.1 netmask 255.255.255.0"
ifconfig_em1_alias5="inet 65.50.232.1 netmask 255.255.255.0"
ifconfig_em1_alias6="inet 65.50.233.1 netmask 255.255.255.0"
ifconfig_em1_alias7="inet 65.50.234.1 netmask 255.255.255.0"
ifconfig_em1_alias8="inet 65.50.235.1 netmask 255.255.255.0"

defaultrouter="66.181.18.2"

# private network
ifconfig_fxp0="inet 10.1.2.5 netmask 255.255.255.0"

inetd_enable="YES"
inetd_flags="-wW -a 10.1.2.5"

sshd_enable="YES"
usbd_enable="YES"
ntpd_enable="YES"
# powerd_enable="YES"

20. reboot. Confirm new kernel is loaded

uname -a

21. update ports:
cd /usr/ports

echo "*default host=cvsup4.FreeBSD.org\
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_9_1\
*default delete use-rel-suffix\
*default compress\
ports-all tag=." > sup

cvsup sup; mail -s 'cvs sup ports done' 8583619553@vtext.com < /dev/null

22. Install raid mgmt tool

# linux base
cd /usr/ports/devel/libtool22
make install base

cd /usr/ports/emulators/linux_base-fc4
make install clean

scp backup2:/d4/build/3ware/tw_cli-freebsd-x86_64-9.5.0.1.tgz /usr/local/sbin
cd /usr/local/sbin
tar xzvf tw_cli-freebsd-x86_64-9.5.0.1.tgz
rm tw_cli-freebsd-x86_64-9.5.0.1.tgz

23. install rsync from ports
cd /usr/ports/net/rsync
make install clean

choose default options

25. install bb client
adduser
Username: bb
Full name: bb
Uid (Leave empty for default): 1984
Login group [bb]:
Login group is bb. Invite bb into other groups? []:
Login class [default]:
Shell (sh csh tcsh nologin) [sh]:
Home directory [/home/bb]:
Use password-based authentication? [yes]:
Use an empty password? (yes/no) [no]:
Use a random password? (yes/no) [no]: yes
Lock out the account after creation? [no]:
Username : bb
Password : <random>
Full Name : bb
Uid : 1984
Class :
Groups : bb
Home : /home/bb
Shell : /bin/sh
Locked : no
OK? (yes/no): yes

cd /usr/home/bb
scp backup2:/mnt/data4/build/bb/bb-freebsd.tar .
tar xvf bb-freebsd.tar

edit /home/bb/bbc1.9e-btf/etc/bb-hosts with something like:
echo "69.55.230.2 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh \
10.1.2.5 firewall3.johncompanies.com # ssh" > /home/bb/bbc1.9e-btf/etc/bb-hosts

vi /home/bb/bbc1.9e-btf/ext/openfiles
MACHINE="firewall3,johncompanies,com" # HAS TO BE IN A,B,C FORM

cd /usr/home/bb/bbc1.9e-btf/etc
./bbchkcfg.sh
(y to questions)
./bbchkhosts.sh
(ignore ssh errors)
cd ../..
chown -R bb .
su bb
cd
cd bbc1.9e-btf/src
make; make install
cd ..

vi /usr/home/bb/bbc1.9e-btf/bin/bb-cpu.sh
$1 $TOPARGS > $BBTMP/TOP.$$
# /usr/local/jail/bin/jtop > $BBTMP/TOP.$$

./runbb.sh start
more BBOUT
(look for errors)
exit

echo 'su - bb -c "cd /home/bb/bbc1.9e-btf; ./runbb.sh start"' > /usr/local/etc/rc.d/bb.sh
chmod +x /usr/local/etc/rc.d/bb.sh

Punch a hole in the firewall to allow it to communicate with bb monitor:

ipfw add 00096 allow ip from 66.181.18.0/27 to 69.55.230.2

27. configure bb on mail:
vi /usr/home/bb/bbsrc/bb1.9e-btf/etc/bb-hosts
66.181.18.3 firewall3.johncompanies.com # ssh

su bb
cd
bbsrc/bb/runbb.sh restart ; exit

29. configure ntp
echo "server 10.1.2.1" > /etc/ntp.conf

/usr/sbin/ntpd -p /var/run/ntpd.pid
sleep 2; ntpq -p
(confirm it’s able to reach our time server)

echo '/usr/sbin/ntpd -c /etc/ntp.conf -p /var/run/ntpd.pid -f /var/db/ntpd.drift' > /usr/local/etc/rc.d/ntp.sh
chmod 0700 /usr/local/etc/rc.d/ntp.sh

30. fwd and reverse lookups on ns1c
vr johncompanies.com
(edit the PTR too)

33. setup backups
echo '#\!/bin/sh\
backupdir=/data/firewall2/current\
\
## ENTRY /etc ' > /usr/local/etc/backup.config

on backup3:
setup backup dirs:
ssh backup3 mkdir -p /data/firewall2/current

on backup3, add the system to
vi /usr/local/sbin/snapshot_archive

scp 69.55.230.2:/usr/local/etc/rsync.backup /usr/local/etc/rsync.backup
vi /usr/local/etc/rsync.backup
backup1 > backup3

crontab -e
1 0 * * * /usr/local/etc/rsync.backup

34. mkdir /root/logs

35. edit sshd_config for security
vi /etc/ssh/sshd_config
ListenAddress 66.181.18.3
ListenAddress 10.1.2.5

kill -1 `cat /var/run/sshd.pid`

35. raid chk

cat > /usr/local/sbin/lsiraidchk
#!/usr/bin/perl

my @out = split "\n", `megarc -ldInfo -a0 -Lall|grep Status:`;

foreach (@out) {
if ($_ =~ /DEGRADED/) { $date = `date`; chomp $date; `echo "$date: RAID ARRAY DEGRADED" >> /var/log/messages`; }
#print $_;
}

36. add crontab entries
crontab -e
30 3 * * * /usr/local/etc/rsync.backup
0 0 1 * * /sbin/ipfw zero
0 0 1 * * /sbin/ipfw del 3 4 5
59 23 30 * * /sbin/ipfw show > /tmp/ipfw_count
3 0 30 * * /sbin/ipfw show > /tmp/ipfw_count
0 3 * * * /etc/makefwrules.pl; /etc/makepiperules.pl;
*/5 * * * * /usr/local/sbin/lsiraidchk

#10 0 * * * rm /var/spool/clientmqueue/*

scp /etc/makefwrules.pl user@64.163.14.48:~
scp /etc/makepiperules.pl user@64.163.14.48:~
mv /home/user/makefwrules.pl /etc
mv /home/user/makepiperules.pl /etc
touch /etc/firewall.sh
mkdir /etc/oldrules/

other binaries

scp /usr/local/bin/rulemaker user@64.163.14.48:~
mv ~user/rulemaker /usr/local/sbin
scp ~user/Sendmail.pm user@64.163.14.48:~
scp ~user/doswatch.pl user@64.163.14.48:~

Setup basic ruleset

ipfw add 00009 count udp from any to any
ipfw add 00010 allow tcp from any to any established
ipfw add 00012 deny tcp from any to any tcpflags syn tcpoptions !mss
ipfw add 00012 deny icmp from any to any icmptypes 4,5,9,10,12,13,14,15,16,17,18
ipfw add 00012 deny tcp from any to any tcpflags syn,fin
ipfw add 00012 deny tcp from any to any tcpflags fin,psh,rst,urg
ipfw add 00012 allow icmp from any to any
ipfw add 00014 deny tcp from any to any dst-port 135
ipfw add 00150 skipto 65535 ip from any to any via em1 in

IPKVM3:
00098 allow ip from { 69.55.230.6 or 69.55.230.7 } to 69.55.230.10 dst-port 139
00098 deny ip from any to 69.55.230.10 dst-port 139
</pre>

= wiki =

The wiki (mediawiki) runs on nat2 in a jail running off 69.55.229.8

The backup wiki lives on virt13 in CT 5 / 69.55.230.18

== Setup jail ==

<pre>
mkdir /mnt/data1/wiki-dir
cd /usr/src
make installworld DESTDIR=/mnt/data1/wiki-dir
cd etc
make distribution DESTDIR=/mnt/data1/wiki-dir

mount -t devfs devfs /mnt/data1/wiki-dir/dev
devfs -m /mnt/data1/wiki-dir/dev rule -s 3 applyset

cd /mnt/data1/wiki-dir

ln -sf dev/null kernel

scp jail9:/usr/local/sbin/jkill /mnt/data1/wiki-dir/sbin

jail /mnt/data1/wiki-dir wiki.johncompanies.com 69.55.229.8 /bin/sh
csh

touch /etc/fstab
echo 'network_interfaces=""\
hostname="wiki.johncompanies.com"\
kern_securelevel_enable="NO"\
sendmail_enable="YES"\
sshd_enable="YES"' > /etc/rc.conf

echo "nameserver 69.55.229.3\
nameserver 69.55.225.225" >> /etc/resolv.conf

vi /etc/crontab

(remove the adjkerntz lines )

vi /etc/periodic/security/100.chksetuid

replace: MP=`mount -t ufs | grep -v " nosuid" | awk '{ print $3 }' | sort`
with: MP='/' (use single quotes)

mkdir -p /usr/compat/linux/dev

adduser

Username: user
Full name: user
Uid (Leave empty for default):
Login group [user]:
Login group is user. Invite user into other groups? []: wheel
Login class [default]:
Shell (sh csh tcsh nologin) [sh]:
Home directory [/home/user]:
Home directory permissions (Leave empty for default):
Use password-based authentication? [yes]:
Use an empty password? (yes/no) [no]:
Use a random password? (yes/no) [no]: y
Lock out the account after creation? [no]:
Username : user
Password : <random>
Full Name : user
Uid : 1001
Class :
Groups : user
Home : /home/user
Home Mode :
Shell : /bin/sh
Locked : no
OK? (yes/no): y
adduser: INFO: Successfully added (user) to the user database.
adduser: INFO: Password for (user) is: 901gmYjO
Add another user? (yes/no): n
Goodbye!

vi /usr/home/user/.profile
TERM=vt100; export TERM

tzsetup

newaliases

rm /sbin/halt /sbin/reboot
ln /sbin/jkill /sbin/halt
ln /sbin/jkill /sbin/reboot

vi /etc/syslog.conf
#*.err;kern.warning;auth.notice;mail.crit /dev/console
*.err;kern.warning;auth.notice;mail.crit /var/log/messages

exit
exit

cd libexec
chflags noschg ld-elf32.so.1
chflags noschg ld-elf.so.1
mv ld-elf32.so.1 ld-elf32.so.1-orig
ln ld-elf.so.1 ld-elf32.so.1
chflags schg ld-elf.so.1
chflags schg ld-elf32.so.1

cp -r /usr/ports /mnt/data1/wiki-dir/usr

cat > /usr/local/etc/rc.d/wiki.sh
mount -t devfs devfs /mnt/data1/wiki-dir/dev/
devfs -m /mnt/data1/wiki-dir/dev rule -s 3 applyset
jail /mnt/data1/wiki-dir wiki.johncompanies.com 69.55.229.8 /bin/sh /etc/rc

chmod 0700 /usr/local/etc/rc.d/wiki.sh
</pre>

== mediawiki setup ==

<pre>

cd /usr/ports/net/rsync
make install clean

cd /usr/ports/distfiles/
fetch http://downloads.mysql.com/archives/mysql-5.5/mysql-5.5.4-m3.tar.gz
cd /usr/ports/databases/mysql55-server
make install clean

cd /usr/ports/distfiles/
fetch http://downloads.php.net/johannes/php-5.3.2.tar.bz2
cd /usr/ports/lang/php52
make install clean
(build apache module)

cd /usr/ports/lang/php5-extensions
make install clean

cd /usr/ports/www/apache22
make install clean

cd /usr/local/www/
fetch http://download.wikimedia.org/mediawiki/1.19/mediawiki-1.19.2.tar.gz
tar xzf mediawiki-1.19.2.tar.gz
mv mediawiki-1.19.2 wiki

vi /usr/local/etc/apache22/httpd.conf
DocumentRoot "/usr/local/www/"

Include etc/apache22/extra/vhost-wiki.conf
Listen 443

<IfModule mod_php5.c>
<FilesMatch "\.ph(p3?|tml)$">
SetHandler application/x-httpd-php
</FilesMatch>
<FilesMatch "\.phps$">
SetHandler application/x-httpd-php-source
</FilesMatch>
# To re-enable php in user directories comment the following lines
# (from <IfModule ...> to </IfModule>.) Do NOT set it to On as it
# prevents .htaccess files from disabling it.
<IfModule mod_userdir.c>
<Directory /home/*/public_html>
php_admin_value engine Off
</Directory>
</IfModule>
</IfModule>

cat > /usr/local/etc/apache22/extra/vhost-wiki.conf
<VirtualHost *:443>
ServerAdmin support@johncompanies.com

DocumentRoot /usr/local/www/wiki
# <Directory />
# Options FollowSymLinks
# AllowOverride None
# Order deny,allow
# </Directory>
<Directory /usr/local/www/wiki>
Options Indexes FollowSymLinks MultiViews
Deny from all
AllowOverride AuthConfig
Order allow,deny
DirectoryIndex index.php
#Allow from 69.55.233.195
#Allow from boody.dyndns.org
</Directory>

ScriptAlias /cgi-bin/ /usr/lib/cgi-bin/
<Directory "/usr/lib/cgi-bin">
AllowOverride None
Options +ExecCGI -MultiViews +SymLinksIfOwnerMatch
Order allow,deny
Allow from all
</Directory>

ErrorLog /var/log/httpd-error.log

# Possible values include: debug, info, notice, warn, error, crit,
# alert, emerg.
LogLevel warn

CustomLog /var/log/httpd-access.log combined

Alias /doc/ "/usr/share/doc/"
<Directory "/usr/share/doc/">
Options Indexes MultiViews FollowSymLinks
AllowOverride None
Order deny,allow
Deny from all
Allow from 127.0.0.0/255.0.0.0 ::1/128
</Directory>

SSLEngine on
SSLCertificateFile /usr/local/etc/apache22/ssl/server.crt
SSLCertificateKeyFile /usr/local/etc/apache22/ssl/server.key

</VirtualHost>

mkdir ssl
cd ssl

openssl req -days 1999 -new -x509 -nodes -out server.crt -keyout server.key
US
CA
San Diego
johncompanies.com
johncompanies.com
wiki.johncompanies.com
support@johncompanies.com

cat > /usr/local/www/wiki/.htaccess
AuthType Basic
AuthUserFile /usr/local/etc/apache22/wiki.passwd
AuthName wiki
require valid-user
satisfy any

cd /usr/local/etc/apache22
htpasswd -c wiki.passwd admin

https://69.55.229.8/index.php

use mysql (innodb)
wiki name: JCWiki
Support / (mail pass) / support@johncompanies.com

cat > /usr/local/www/wiki/LocalSettings.php

<?php
# This file was automatically generated by the MediaWiki 1.19.2
# installer. If you make manual changes, please keep track in case you
# need to recreate them later.
#
# See includes/DefaultSettings.php for all configurable settings
# and their default values, but don't forget to make changes in _this_
# file, not there.
#
# Further documentation for configuration settings may be found at:
# http://www.mediawiki.org/wiki/Manual:Configuration_settings

# Protect against web entry
if ( !defined( 'MEDIAWIKI' ) ) {
exit;
}

## Uncomment this to disable output compression
# $wgDisableOutputCompression = true;

$wgSitename = "JCWiki";

## The URL base path to the directory containing the wiki;
## defaults for all runtime URL paths are based off of this.
## For more information on customizing the URLs please see:
## http://www.mediawiki.org/wiki/Manual:Short_URL
$wgScriptPath = "";
$wgScriptExtension = ".php";

## The protocol and server name to use in fully-qualified URLs
$wgServer = "https://69.55.229.8";

## The relative URL path to the skins directory
$wgStylePath = "$wgScriptPath/skins";

## The relative URL path to the logo. Make sure you change this from the default,
## or else you'll overwrite your logo when you upgrade!
#$wgLogo = "$wgStylePath/common/images/wiki.png";
$wgLogo = "$wgStylePath/common/images/jclogo.gif";

## UPO means: this is also a user preference option

$wgEnableEmail = true;
$wgEnableUserEmail = true; # UPO

$wgEmergencyContact = "apache@69.55.229.8";
$wgPasswordSender = "apache@69.55.229.8";

$wgEnotifUserTalk = false; # UPO
$wgEnotifWatchlist = false; # UPO
$wgEmailAuthentication = true;

## Database settings
$wgDBtype = "mysql";
$wgDBserver = "localhost";
$wgDBname = "my_wiki";
$wgDBuser = "root";
$wgDBpassword = "";

# MySQL specific settings
$wgDBprefix = "";

# MySQL table options to use during installation or update
$wgDBTableOptions = "ENGINE=InnoDB, DEFAULT CHARSET=binary";

# Experimental charset support for MySQL 5.0.
$wgDBmysql5 = false;

## Shared memory settings
$wgMainCacheType = CACHE_NONE;
$wgMemCachedServers = array();

## To enable image uploads, make sure the 'images' directory
## is writable, then set this to true:
$wgEnableUploads = false;
#$wgUseImageMagick = true;
#$wgImageMagickConvertCommand = "/usr/bin/convert";

# InstantCommons allows wiki to use images from http://commons.wikimedia.org
$wgUseInstantCommons = false;

## If you use ImageMagick (or any other shell command) on a
## Linux server, this will need to be set to the name of an
## available UTF-8 locale
$wgShellLocale = "en_US.utf8";

## If you want to use image uploads under safe mode,
## create the directories images/archive, images/thumb and
## images/temp, and make them all writable. Then uncomment
## this, if it's not already uncommented:
#$wgHashedUploadDirectory = false;

## Set $wgCacheDirectory to a writable directory on the web server
## to make your wiki go slightly faster. The directory should not
## be publically accessible from the web.
#$wgCacheDirectory = "$IP/cache";

# Site language code, should be one of the list in ./languages/Names.php
$wgLanguageCode = "en";

$wgSecretKey = "abc699ef26890b49b4055430f8ebbd25e84cce21a7e53aeaec4d4313af4c9739";

# Site upgrade key. Must be set to a string (default provided) to turn on the
# web installer while LocalSettings.php is in place
$wgUpgradeKey = "3196710f4a7d7332";

## Default skin: you can change the default skin. Use the internal symbolic
## names, ie 'standard', 'nostalgia', 'cologneblue', 'monobook', 'vector':
$wgDefaultSkin = "vector";

## For attaching licensing metadata to pages, and displaying an
## appropriate copyright notice / icon. GNU Free Documentation
## License and Creative Commons licenses are supported so far.
$wgRightsPage = ""; # Set to the title of a wiki page that describes your license/copyright
$wgRightsUrl = "";
$wgRightsText = "";
$wgRightsIcon = "";

# Path to the GNU diff3 utility. Used for conflict resolution.
$wgDiff3 = "/usr/bin/diff3";

# Query string length limit for ResourceLoader. You should only set this if
# your web server has a query string length limit (then set it to that limit),
# or if you have suhosin.get.max_value_length set in php.ini (then set it to
# that value)
$wgResourceLoaderMaxQueryLength = -1;

# End of automatically generated settings.
# Add more configuration options below.

</pre>

== copy/backup wiki ==
on main/primary wiki:
<pre>
/usr/local/etc/rc.d/mysql-server stop
ssh 69.55.230.18 "/etc/init.d/mysql stop"
rsync -av /var/db/mysql/my_wiki/ 69.55.230.18:/var/lib/mysql/my_wiki/
rsync -av /var/db/mysql/ib* 69.55.230.18:/var/lib/mysql/
/usr/local/etc/rc.d/mysql-server start
ssh 69.55.230.18 "/etc/init.d/mysql start"

</pre>

Infrastructure Machines

2013-10-17T17:18:51Z

24.165.90.109: /* Notes */

= jails =

== jail1 ==

* Location: castle, cab 3-8
* OS: FreeBSD 6.2 i386
* Networking: Priv IP: 10.1.4.101 (PCI nic), Pub IP: 69.55.230.107 (onboard)
* Hardware: Dell 2450. 4 SCSI SCA drive bays (2 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: two 74 GB (4 x 74GB) RAID1 arrays running on an Adaptec-based, Dell-branded (perc) RAID card.
* Remote management: none
* Disk accounting: gvinum

== jail2 ==

* Location: castle, cab 3-5
* OS: FreeBSD 7.2 amd64
* Networking: Priv IP: 10.1.4.102, Pub IP: 69.55.228.53 (2 onboard nics)
* Hardware: Dell 2950. 6 SATA/SAS drive bays (3 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: one 146 GB (2 x 146GB) RAID1 array, two 300 GB (4 x 300GB) RAID1 arrays running on an LSI-based, Dell-branded (PERC 6/i) RAID card.
* Remote management: [[DRAC/RMM|DRAC]] @ 10.1.4.232
* Disk accounting: md

== jail3 ==
* Location: i2b, cab 6
* OS: FreeBSD 8.3 amd64
* Networking: Priv IP: 10.1.2.103, Pub IP: 69.55.229.7 (2 onboard nics)
* Hardware: Supermicro (custom build). 6 SATA drive bays (2 columns of 3, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: two 300 GB (2 x 300GB) RAID1 array running on a 3ware 8006-2LP RAID card.
* Remote management: none
* Disk accounting: md

=== Notes ===
* '''We should not add users to this server since it is at I2B'''
* must be ssh'd to from nat2
* is a super jail for customer col01737

== jail4 ==

* Location: castle, cab 3-8
* OS: FreeBSD 9.1 x86_64
* Networking: Priv IP: 10.1.4.104, Pub IP: 69.55.228.104 (2 onboard nics)
* Hardware: Dell 2850. 6 SATA/SAS drive bays (3 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: one 1.4 TB RAID 5 array Dell-branded (PERC 4e)LSI megarc RAID card.
* Remote management: None
* Disk accounting: md

=== Notes ===
Only FreeBSD 9.1 jail

== jail7 ==

* Location: castle, cab 3-7
* OS: FreeBSD 6.3 i386
* Networking: Priv IP: 10.1.4.107, Pub IP: 69.55.230.108 (2 onboard nics)
* Hardware: Dell 2950. 6 SATA/SAS drive bays (3 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: two 146 GB (4 x 146GB) RAID1 arrays, one 74 GB (2 x 74GB) RAID1 array running on an LSI-based, Dell-branded (PERC 6/i) RAID card.
* Remote management: [[DRAC/RMM|DRAC]] @ 10.1.4.237
* Disk accounting: gvinum

=== Notes ===
Do not run a verify while OS/jails running, will crash.

== jail8 ==

* Location: castle, cab 3-6
* OS: FreeBSD 8.0 amd64
* Networking: Priv IP: 10.1.4.108, Pub IP: 69.55.234.2 (2 onboard nics)
* Hardware: Dell 2950. 6 SATA/SAS drive bays (3 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: one 146 GB (2 x 146GB) RAID1 array, one 300 GB (2 x 300GB) RAID1 array, one 400 GB (2 x 400GB) RAID1 array, running on an LSI-based, Dell-branded (PERC 6/i) RAID card.
* Remote management: [[DRAC/RMM|DRAC]] @ 10.1.4.238
* Disk accounting: md

== jail9 ==

* Location: castle, cab 3-6
* OS: FreeBSD 8.2 amd64
* Networking: Priv IP: 10.1.4.109, Pub IP: 69.55.232.36 (2 onboard nics)
* Hardware: Dell 2950. 6 SATA/SAS drive bays (3 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: one 146 GB (2 x 146GB) RAID1 array, one 400 GB (2 x 300GB) RAID1 array running on an LSI-based, Dell-branded (PERC 5/i) RAID card.
* Remote management: [[DRAC/RMM|DRAC]] @ 10.1.4.239
* Disk accounting: md

== jail11 ==

* Location: castle, cab 3-7
* OS: FreeBSD 4.7 i386
* Networking: Priv IP: 10.1.4.111 (PCI nic), Pub IP: 69.55.236.92 (onboard)
* Hardware: Dell 2450. 4 SCSI SCA drive bays (2 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: two 74 GB (4 x 74GB) RAID1 arrays running on an Adaptec-based, Dell-branded (perc) RAID card.
* Remote management: none
* Disk accounting: vinum

== mx1 ==

* Location: castle, cab 3-7
* OS: FreeBSD 4.11 i386
* Networking: Priv IP: 10.1.4.201 (PCI nic), Pub IP: 69.55.237.3 (onboard)
* Hardware: Dell 2450. 4 SCSI SCA drive bays (2 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: one 36 GB (2 x 36GB) RAID1 array, one 74 GB (2 x 74GB) RAID1 array running on an Adaptec-based, Dell-branded (perc) RAID card.
* Remote management: none
* Disk accounting: vinum

=== Notes ===
* is our (old) backup mail/dns vps service host

== mx2 ==

* Location: castle, cab 3-5
* OS: FreeBSD 7.1 i386
* Networking: Priv IP: 10.1.4.202 (PCI nic), Pub IP: 69.55.237.90 (onboard)
* Hardware: Dell 2450. 4 SCSI SCA drive bays (2 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: two 74 GB (4 x 74GB) RAID1 arrays running on an Adaptec-based, Dell-branded (perc) RAID card.
* Disk accounting: gvinum

=== Notes ===
* is our latest backup mail/dns vps service host

== jail17 ==
* Location: castle, cab 3-8
* OS: FreeBSD 4.10 i386
* Networking: Priv IP: 10.1.4.117 (PCI nic), Pub IP: 69.55.228.2 (onboard nics)
* Hardware: Supermicro (custom build). 6 SCA SCSI drive bays (2 columns of 3, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: two 74 GB (4 x 74GB) RAID1 arrays, one 146 GB (2 x 146GB) RAID1 array, running on an Adaptec 2120S RAID card.
* Remote management: none
* Disk accounting: vinum

== jail18 ==
* Location: castle, cab 3-7
* OS: FreeBSD 4.10 i386
* Networking: Priv IP: 10.1.4.117 (PCI nic), Pub IP: 69.55.228.2 (onboard nics)
* Hardware: Supermicro (custom build). 6 SCA SCSI drive bays (2 columns of 3, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: two 74 GB (4 x 74GB) RAID1 arrays, one 146 GB (2 x 146GB) RAID1 array, running on an Adaptec 2120S RAID card.
* Remote management: none
* Disk accounting: vinum

== jail19 ==
* Location: castle, cab 3-6
* OS: FreeBSD 6.1 i386
* Networking: Priv IP: 10.1.4.119 (PCI nic), Pub IP: 69.55.228.200 (onboard nics)
* Hardware: Supermicro (custom build). 6 SCA SCSI drive bays (2 columns of 3, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: one 74 GB (2 x 74GB) RAID1 array, one 146 GB (2 x 146GB) RAID1 array, running on an Adaptec 2120S RAID card.
* Remote management: none
* Disk accounting: gvinum

= virts =

== quar1 ==

* Location: castle, cab 3-8
* OS: RedHat 7.3 x86
* Networking: Priv IP: 10.1.4.151 (PCI nic), Pub IP: 69.55.227.2 (onboard nic)
* Hardware: Dell 2450. 4 SCSI SCA drive bays (2 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: one 36 GB (2 x 36GB) RAID1 array, one 74 GB (2 x 74GB) RAID1 array, running on an Adaptec-based, Dell-branded (perc) RAID card.
* Remote management: none
* Virtuozzo version: 2.6.1
* VZ license: hwid=23C0.C0E1.6FDD.08BA.8971.8E1C.EBD5.1EDC serial=0DE6.903E.E239.E23F.470C.4369.4104.A5A4

=== Notes ===
* used to be the home of customers who's VE's would just run out of control/badly
* has a max of 10 VE's allowed to run

== virt9 ==

* Location: castle, cab 3-7
* OS: RedHat 7.3 x86
* Networking: Priv IP: 10.1.4.59 (PCI nic), Pub IP: 69.55.226.161 (onboard nic)
* Hardware: Dell 2450. 4 SCSI SCA drive bays (2 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: two 74 GB (2 x 74GB) RAID1 arrays running on an Adaptec-based, Dell-branded (perc) RAID card.
* Remote management: none
* Virtuozzo version: 2.6.1
* VZ license: hwid=BC15.B4D6.0D25.A5FE.F3BA.D518.E351.AE3F serial=F6AD.B6B4.5650.8869.C97C.73EE.AF65.FA8B

== virt11 ==

* Location: castle, cab 3-6
* OS: CentOS 5.4 x86
* Networking: Priv IP: 10.1.4.61, Pub IP: 69.55.238.3, 2 onboard nics
* Hardware: Dell 2950. 6 SATA/SAS drive bays (3 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: one 146 GB (2 x 146GB) RAID1 array, one 400 GB (2 x 400GB) RAID1 array, one 146 GB (2 x 146GB) RAID1 array, running on an LSI-based, Dell-branded (perc 6/i) RAID card.
* Remote management: [[DRAC/RMM|DRAC]] @ 10.1.4.211
* Virtuozzo version: 4.0.0
* VZ license: hwid="029D.A187.78E1.480F.49E3.E20A.7389.7F79" serial="163C.F3E2.195F.96B5.2D38.8937.9600.4A05"

== virt12 ==

* Location: castle, cab 3-6
* OS: CentOS 5.2 x86
* Networking: Priv IP: 10.1.4.62, Pub IP: 69.55.227.70, 2 onboard nics
* Hardware: Dell 2950. 6 SATA/SAS drive bays (3 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: two 300 GB (2 x 300GB) RAID1 arrays one 400 GB (2 x 400GB) RAID1 array, running on an LSI-based, Dell-branded (perc 6/i) RAID card.
* Remote management: [[DRAC/RMM|DRAC]] @ 10.1.4.212
* Virtuozzo version: 4.0.0
* VZ license: hwid="0C53.A413.E095.B4F4.51BC.D740.6919.A77B" serial="84E5.9498.3759.E683.E24B.2514.CA72.DC31"

== virt13 ==

* Location: castle, cab 3-5
* OS: CentOS 6.2 x86_64
* Networking: Priv IP: 10.1.4.63, Pub IP: 69.55.226.2, 2 onboard nics
* Hardware: Dell 2950. 6 SATA/SAS drive bays (3 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: one 146 GB (2 x 146GB) RAID1 array, one 600 GB (2 x 600GB) RAID1 array, running on an LSI-based, Dell-branded (perc 6/i) RAID card.
* Remote management: [[DRAC/RMM|DRAC]] @ 10.1.4.213
* Virtuozzo version: 4.7.0
* VZ license: hwid="7D07.93BE.0B1F.7D2B.B039.4B5B.48B6.453B" serial="60A4.A94C.44BB.DCD6.8D03.1778.605B.10FE"

=== Notes ===
* home to our latest/current signups
* currently the only 64bit vz host

== virt14 ==

* Location: castle, cab 3-?
* OS: CentOS 6.4x86_64
* Networking: Priv IP: 10.1.4.64 Pub IP: 69.55.225.14 2 onboard nics
* Hardware: Dell 2950. 6 SATA/SAS drive bays (3 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: one 600 GB (2 x 600GB) RAID1 array, running on an LSI-based, Dell-branded (perc 6/i) RAID card.
* Remote management: [[DRAC/RMM|DRAC]] @ 10.1.4.214
* Virtuozzo version: 4.7.0
* VZ license: hwid="" serial=""

=== Notes ===
* new home to our latest/current signups
* virt 13 and 14 currently the only 64bit vz hosts

== virt15 ==

* Location: castle, cab 3-5
* OS: RedHat 9 x86
* Networking: Priv IP: 10.1.4.65, Pub IP: 69.55.232.160 (2 onboard nics)
* Hardware: Supermicro (custom build). 6 SCSI SCA drive bays (2 columns of 3, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: one 74 GB (2 x 74GB) RAID1 array, two 146 GB (2 x 146GB) RAID1 arrays, running on an LSI MegaRAID SCSI 320-1 RAID card.
* Remote management: none
* Virtuozzo version: 2.6.2
* VZ license: hwid=A90F.6F48.E723.D8BA.3025.184A.5B73.D11E serial=E94B.5164.C1E6.A67F.67D1.7D96.0B6C.5524

== virt16 ==

* Location: castle, cab 3-7
* OS: Fedora Core 4 x86
* Networking: Priv IP: 10.1.4.66, Pub IP: 69.55.232.2 (2 onboard nics)
* Hardware: Supermicro (custom build). 6 SCSI SCA drive bays (2 columns of 3, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: one 74 GB (2 x 74GB) RAID1 array, one 146 GB (2 x 146GB) RAID1 array, running on an LSI MegaRAID SCSI 320-1 RAID card.
* Remote management: none
* Virtuozzo version: 3.0.0
* VZ license: hwid=DEFA.A325.7230.BBC8.9715.8B52.3FD7.27BE serial=66C0.41EA.3FBB.11D3.9CC6.55C7.09AE.14AB

== virt17 ==

* Location: castle, cab 3-6
* OS: CentOS 4.4 x86
* Networking: Priv IP: 10.1.4.67, Pub IP: 69.55.232.162, 2 onboard nics
* Hardware: Dell 2950. 6 SATA/SAS drive bays (3 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: two 146 GB (2 x 146GB) RAID1 arrays running on an LSI-based, Dell-branded (perc 5/i) RAID card.
* Remote management: [[DRAC/RMM|DRAC]] @ 10.1.4.217
* Virtuozzo version: 3.0.0
* VZ license: hwid=2E14.AED9.70B8.C26E.D99F.B0D3.BCD2.229C serial=2A11.DAD0.61DB.E889.8DF4.9AF7.CF82.3C37

== virt19 ==

* Location: castle, cab 3-6
* OS: CentOS 5.2 x86
* Networking: Priv IP: 10.1.4.69, Pub IP: 69.55.236.2, 2 onboard nics
* Hardware: Dell 2950. 6 SATA/SAS drive bays (3 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: one 146 GB (2 x 146GB) RAID1 array, one 300 GB (2 x 300GB) RAID1 array, running on an LSI-based, Dell-branded (perc 5/i) RAID card.
* Remote management: [[DRAC/RMM|DRAC]] @ 10.1.4.219
* Virtuozzo version: 3.0.0
* VZ license: hwid=3968.13F7.B2AC.8952.8E19.13A9.6EF5.5822 serial=061D.84CD.CCE5.B213.15B5.C061.D6A7.B034

= mail =
== Summary ==
This machine (mail) is the swiss army knife of the company, playing host to many services and functions.

* Location: castle, cab 3-8
* OS: FreeBSD 4.10 x86
* Networking: Priv IP: 10.1.4.5, Pub IPs: 69.55.230.2, 69.55.225.225 (ns1c jail), 69.55.230.9. 1 onboard and 1 PCI
* Hardware: Dell 2450. 4 SCSI SCA drive bays (2 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: two 36 GB (2 x 36GB) RAID1 arrays running on an Adaptec-based, Dell-branded (perc) RAID card.

== Services Provided ==
* mail
* web
* mysql
* bigbrother server/pager
* snmp
* named in a jail (ns1c)

== email ==
This server hosts mail for johncompanies.com (mail.johncompanies.com). Sendmail 8.13.6/8.13.6 is listening on 69.55.230.2 port 25 for incoming mail. Relaying is allowed per /etc/mail/relay-domains
Other addresses (aliases) are defined per /etc/mail/aliases

The following active users have mail hosted on this server:
* dave
* linux
* support
* payments
* sales
* tech1
* info

Traditionally, mail is checked via shell apps (pine). qpopper (pop3s) is running to allow mail downloading. Checking mail in this way causes an opened INBOX in pine to lock read-only. For this reason, we tee incoming mail to support and linux to tech1.

Procmail rules are setup to filter spam and send text messages. They are enabled for info, support, linux, tech1, dave and can be found in ~/Procmail/, for example:
<pre># more ~support/Procmail/rc.emergency
:0c # use c only if you want to forward a copy and file the original later
* ^Subject:.*\<emergency\>
* ! ^Subject:\<re\>
{
:0h
FROMANDSUBJECT=|formail -XFrom: -XSubject:

:0fwh
| /usr/local/bin/formail -I"Subject: " -I"To: pager@johncompanies.com" ; echo $FROMANDSUBJECT ; echo

:0
! -t
}

</pre>

control: <tt>cd /etc/mail; make stop</tt> (stop), <tt>cd /etc/mail; make start</tt> (start)

The following aliases are also in place:

<pre>debian: linux
jobs: info
careers: info
#reboot: 6128102202@txt.att.net
#reboot: 8582298897@vtext.com
reboot: pager
#pager: 8582298897@vtext.com
pager: 4158718324@txt.att.net
tech1on: "| /usr/local/sbin/tech1on.sh"
tech1off: "| /usr/local/sbin/tech1off.sh"</pre>

To change them, edit <tt>/etc/aliases</tt> and then run <tt>newaliases</tt>

Note on tech1: this address was setup as a read-only address to be mirrored on all email coming into support and linux. We set this up so we could easily check support mail via a pop client- popping email locks out the user in pine so checking support/linux directly via pop was not an option. When checking and responding to email that comes into tech1, care should be taken to make sure it is sent as/under an address other than tech1. This is cause tech1 is not monitored by support staff as closely as email to support/linux. Further, the tech on call may not be checking tech1. Lastly, because of the nature of the copying, you will sometimes notice certain automated email/notices are received 2x in support- this is because of/related to the tech1 mirror.

To enable it (on mail, run):

<tt>~support/tech1on.sh</tt>

To disable

<tt>~support/tech1off.sh</tt>

Or via email:

<tt>tech1on@johncompanies.com
tech1off@johncompanies.com</tt>

== web ==

See [[Management_System_/_Public_Website_/_Signup|Management System / Public Website / Signup]]

== mysql ==
mysql 4.1.22 is running on port 3306

* datadir: <tt>/mnt/data1/db/mysql/</tt>
* config: <tt>/etc/my.cnf</tt>
* database: <tt>jc</tt>
* control: <tt>/usr/local/etc/rc.d/mysql-server.sh stop</tt> (stop), <tt>/usr/local/etc/rc.d/mysql-server.sh start</tt> (start)

== bigbrother ==
There is a client running on mail (which monitors the services running on mail and mail itself), installed under <tt>/usr/home/bb/bbc1.9e-btf</tt><br>
And the big brother pager/server (which displays information gathered from all bb-monitored machines, including mail) is installed under <tt>/usr/home/bb/bbsrc/bb1.9i-btf</tt>

Both are running under the user <tt>bb</tt>

Refer to [[BigBrother]] for more about use.

== DNS (ns1c.johncompanies.com) ==
ns1c is a jail running on the mail server, who's IP is 69.55.225.225

It's running from <tt>/mnt/data1/ns1c-dir</tt>

See [[DNS]] for more details

== Usage and Notes ==
* always mounted to backup1 and backup2 via nfs:
<pre>backup2:/mnt/data1 on /backup (nfs)
backup2:/mnt/data2 on /backup2 (nfs)
backup2:/mnt/data3 on /backup3 (nfs)
backup2:/mnt/data4 on /backup4 (nfs)
backup1:/data on /backup1 (nfs)
</pre>

== Cronjobs ==
* * * * * /usr/local/www/mgmt/mrtg/mrtg.sh > /dev/null 2>&1
Gathers up data for our mrtg/load graphs

*/5 * * * * /usr/local/bin/rsync -a root@nat2:/mnt/data1/mrtg/data/ /usr/local/www/mgmt/mrtg/data/
Gathers up data from i2b servers for our mrtg/load graphs

40 0 * * * /usr/local/bin/rsync -a root@nat2:"/mnt/data1/mrtg/*.cfg" /usr/local/www/mgmt/mrtg
Gathers up mrtg configuration (port names) from i2b switches for our mrtg/load graphs

41 0 * * * for f in `grep -l "mnt\/data1" /usr/local/www/mgmt/mrtg/switch-p*.cfg`; do cat $f | sed s#\/mnt\/data1#\/usr\/local\/www\/mgmt# > $f.new; mv $f.new $f; done
Gathers up mrtg configuration (port names) from castle switches for our mrtg/load graphs

1 0 1 * * cp /usr/local/www/mgmt/html/top20ip /usr/local/www/mgmt/html/top20ip_last
1 0 1 * * cp /usr/local/www/mgmt/html/top20customers /usr/local/www/mgmt/html/top20customers_last
2 * * * * /usr/local/www/cronjobs/top20ip.pl > /dev/null 2>&1
15 * * * * /usr/local/www/cronjobs/top20customer.pl > /dev/null 2>&1
1 0 1 * * rm /usr/local/www/mgmt/html/bandtrack
Archiving and generation of bandwidth statistics presented in mgmt -> Reference -> Bandwidth

1 0 * * * /usr/local/etc/rsync.backup
Nightly backup script

0 1 * * * /usr/local/www/mgmt/awstats/wwwroot/cgi-bin/awstats.pl -config=jcpub -update
Public web traffic stats

15 0 * * * rm /usr/local/www/mgmt/bwgraphs/*.png
16 0 * * * rm /usr/local/www/am/bwgraphs/*
Cleanup for graph-related temp data generated by customers using the bandwidth reports via the AM

10 0 1 * * /usr/local/www/cronjobs/monthly_bandwidth_report.pl
Monthly bandwidth overage report

*/3 * * * * /usr/local/www/cronjobs/bbcheck.pl
Updates mgmt with bb monitoring issues

5 0 * * * /usr/local/www/cronjobs/shutdownreminder.pl
Emails customers reminding them of upcoming shutdown date

7 0 * * * /usr/local/www/cronjobs/invoice_email.pl
Emails customers who have invoices and are set to auto-email (currently no customer gets these)

8 */4 * * * /usr/local/www/cronjobs/mysqlrepchk.pl
Checking that we are properly replicating (mysql) traffic data from bwdb to backup1

16 0 1 * * /usr/local/www/cronjobs/purge_traffic.pl
Removed old traffic data from the traffic database (running on backup1)

*/5 * * * * chmod 0700 /usr/local/www/ccard_orders/* && mv /usr/local/www/ccard_orders/* /usr/local/www/ccard_orders/done
Secure credit card data: set root-read-only

25 0 * * * /usr/local/www/cronjobs/biller.pl
Enters service charges in customer billing ledgers

10 13 * * * /usr/local/www/cronjobs/pfp_batch_gather.pl
Looks for customers with balance due and active credit card on file, prepares a payflow batch

10 14 * * * /usr/local/www/cronjobs/pfp_batch_process.pl
Tries to collect ccard funds for items in payflow batch - communicates with payflow

15 13 * * * /usr/local/www/cronjobs/pb_batch_gather.pl
Looks for customers with balance due and active paypal billing agreement on file, prepares a paypal batch

15 14 * * * /usr/local/www/cronjobs/pb_batch_process.pl
Tries to collect paypal funds for items in paypal batch - communicates with paypal

0 7 * * 1 /usr/local/www/cronjobs/email_pmt_reminder.pl
Emails customers in arrears, reminding them to pay

0 0 1 * * /usr/bin/mail -s 'archive sent mail in pine' support@johncompanies.com < /dev/null
Reminds us to archive sent mail

0 3 * * * /usr/local/bin/rsync -a isys.e-monitoring.net:/var/mail /backup2/isys; /usr/local/bin/rsync -a isys.e-monitoring.net:/usr/home /backup2/isys
Backup data on isys

== Regular maintenance ==
*[[Routine_Maintenance#Adaptec_Controllers|Check RAID array]]

= nat =
== Summary ==
This is the main machine to which we ssh and runs all our screen sessions. Further, it's ip runs in a special block which is not routed through the firewall and this is somewhat immune to DoS attacks which hobble our firewall. Lastly, it acts as a nat server for certain/random devices on the private network.

* Location: castle, cab 3-7
* OS: FreeBSD 9.1 i386
* Networking: Priv IP: 10.1.4.1, Pub IPs: 69.55.233.195, 69.55.233.196, 69.55.233.197, 69.55.233.198, 69.55.233.199. 1 onboard and 1 PCI
* Hardware: Custom 1U. single power supply.
* Drives: one 8 GB IDE drive

== Services Provided ==
* nat

== nat control ==
All rules are contained in and look like:

<pre>cat /etc/ipnat.rules
# www (was 69.55.230.12)
# virt19
#bimap fxp0 10.1.4.209/32 -> 69.55.233.198/32
# virt18
#bimap fxp0 10.1.4.208/32 -> 69.55.233.196/32
# virt13
#bimap fxp0 10.1.4.213/32 -> 69.55.233.196/32
# virt12
#bimap fxp0 10.1.4.212/32 -> 69.55.233.196/32
# virt17
bimap fxp0 10.1.4.217/32 -> 69.55.233.196/32
# virt11
#bimap fxp0 10.1.4.211/32 -> 69.55.233.196/32
# ASA
#bimap fxp0 10.1.4.172/32 -> 69.55.233.196/32
# P1A
bimap fxp0 10.1.4.240/32 -> 69.55.233.197/32
#bimap fxp0 10.1.4.238/32 -> 69.55.233.197/32
# developer (was 69.55.230.17)
# jail2
#bimap fxp0 10.1.4.232/32 -> 69.55.233.198/32
# jail8
#bimap fxp0 10.1.4.238/32 -> 69.55.233.198/32
# jail9
#bimap fxp0 10.1.4.239/32 -> 69.55.233.198/32
# POLL
#BIMAP EM0 10.1.6.134/32 -> 69.55.230.20/32
# 1U SUN
#BIMAP EM0 10.1.4.4/32 -> 69.55.227.46/32
# ??
#BIMAP EM0 10.1.6.3/32 -> 69.55.230.100/32
# random machine
#bimap fxp0 10.1.6.13/32 -> 69.55.233.199/32
#bimap fxp0 10.1.4.232/32 -> 69.55.233.199/32
# OFFICE OUTBOUND TRAFFIC
#map fxp0 10.1.6.0/24 -> 0.0.0.0/32 proxy port ftp ftp/tcp
#map fxp0 10.1.6.0/24 -> 0.0.0.0/32</pre>

A simple entry looks like:
bimap fxp0 10.1.4.240/32 -> 69.55.233.197/32
Which essentially means make private IP 10.1.4.240 reachable on 69.55.233.197 and allow 10.1.4.240 to communicate with the public internet via 69.55.233.197

To reload new rule config:
ipnat -C -F -f /etc/ipnat.rules

You may want to setup natting, as above, when you need to reach a DRAC card's web interface, wherin the DRAC card only has a private IP.

= nat2 =
== Summary ==
This is the main machine to which we ssh and runs all our screen sessions at i2b, and runs ns3c (this is kind of the what mail is to castle). Further, it's ip runs in IP space provided by i2b: 66.181.18.1 - 66.181.18.30, which is not routed through the firewall and this is somewhat immune to DoS attacks which hobble our firewall. Lastly, it acts as a nat server for certain/random devices on the private network.

* Location: i2b, cab 6
* OS: FreeBSD 6.4 x86
* Networking: Priv IP: 10.1.2.1, Pub IPs: 69.55.229.2, 69.55.229.3, 66.181.18.4, 66.181.18.5, 66.181.18.6, 66.181.18.7, 66.181.18.8, 66.181.18.9, 66.181.18.10, 66.181.18.11, 66.181.18.12, 66.181.18.13, 66.181.18.14 1 onboard and 1 PCI
* Hardware: Custom 2U. 6 drive bays, non-hot-swappable. single power supply.
* Drives: one 150 GB (2 x 150GB) RAID1 array running on a 3ware 8006 RAID card.

== Services Provided ==
* nat
* bigbrother
* ns3c (jail)
* ntp

== nat config ==
Here's what's currently nat'd on nat2:
<pre>cat /etc/ipnat.rules
# sample entry
#ATS-9
bimap em0 10.1.2.79/32 -> 66.181.18.14/32
#ATS-8
bimap em0 10.1.2.78/32 -> 66.181.18.13/32
#ATS-7
bimap em0 10.1.2.77/32 -> 66.181.18.12/32
#ATS-6
bimap em0 10.1.2.76/32 -> 66.181.18.6/32
#ATS-5
bimap em0 10.1.2.75/32 -> 66.181.18.7/32
#ATS-4
bimap em0 10.1.2.74/32 -> 66.181.18.8/32
#ATS-3
bimap em0 10.1.2.73/32 -> 66.181.18.9/32
#ATS-2
bimap em0 10.1.2.72/32 -> 66.181.18.10/32
#ATS-1
bimap em0 10.1.2.71/32 -> 66.181.18.11/32
#bwdb2
bimap em0 10.1.2.4/32 -> 66.181.18.5/32

# spare

map em0 10.1.2.0/24 -> 0.0.0.0/32 proxy port ftp ftp/tcp

#bimap fxp0 10.1.6.49/32 -> 10.1.1.2/32
#bimap fxp0 10.1.4.208/32 -> 69.55.233.196/32
</pre>

== build ==

* partition map:
<pre>/ 512m
swap 1G
/var 256m
/tmp 256m
/usr 5g
/mnt/data1 ~</pre>

* edit /etc/make.conf
<pre>echo "WITHOUT_X11=yes \
KERNCONF=nat2 \
BOOT_COMCONSOLE_SPEED=115200" >> /etc/make.conf</pre>

* add settings to /boot/loader.conf and /boot.config

<pre>echo "-Dh" >> /boot.config

echo 'console="comconsole,vidconsole" \
boot_multicons="YES" \
boot_serial="YES" \
comconsole_speed="115200"' >> /boot/loader.conf</pre>

* turn off all ttyv's except 0 and 1 in /etc/ttys
also turn on ttyd0, change type to vt100:
<pre>vi /etc/ttys

ttyv2 "/usr/libexec/getty Pc" cons25 off secure
ttyv3 "/usr/libexec/getty Pc" cons25 off secure
ttyv4 "/usr/libexec/getty Pc" cons25 off secure
ttyv5 "/usr/libexec/getty Pc" cons25 off secure
ttyv6 "/usr/libexec/getty Pc" cons25 off secure
ttyv7 "/usr/libexec/getty Pc" cons25 off secure
# Serial terminals
# The 'dialup' keyword identifies dialin lines to login, fingerd etc.
ttyd0 "/usr/libexec/getty std.9600" vt100 on secure

kill -1 1</pre>

on console server:
vi /etc/remote
(rename port to jail8 depending on where and which digi plugged into)
test serial console

* populate hosts
<pre>echo "69.55.230.10 backup2" >> /etc/hosts
echo "69.55.230.11 backup1" >> /etc/hosts
echo "10.1.2.4 bwdb2" >> /etc/hosts
echo "10.1.2.3 backup3" >> /etc/hosts</pre>

* put key in authorized_keys on backup3
<pre>cd
ssh-keygen -t dsa -b 1024</pre>
(default location, leave password blank)

<pre>cat /root/.ssh/id_dsa.pub | ssh backup3 'cat - >> /root/.ssh/authorized_keys'
cat /root/.ssh/id_dsa.pub | ssh backup2 'cat - >> /root/.ssh/authorized_keys'
cat /root/.ssh/id_dsa.pub | ssh backup1 'cat - >> /root/.ssh/authorized_keys'</pre>

confirm that you can ssh to backup3 and backup 2 without getting a login prompt

ssh backup3 hostname

ssh backup2 hostname

ssh backup1 hostname

* edit root's path and login script:
vi /root/.cshrc

Change alias entries (add G):
<pre>alias la ls -aG
alias lf ls -FAG
alias ll ls -lAG
alias ls ls -AG
alias mbm mb mount
alias mbu mb umount</pre>

and alter the prompt, set the following:
set prompt = "`/bin/hostname -s` %/# "

* install cvsup
<pre>cd /usr/ports/net/cvsup-without-gui
make install clean; rehash; mail -s 'cvs installed' dave.boodman@vtext.com < /dev/null</pre>

* get latest sources for this release:
<pre>cd /usr/src
echo "*default host=cvsup4.freebsd.org\
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_6_4\
*default delete use-rel-suffix\
*default compress\
src-all" > sup

cvsup sup ; mail -s 'cvs sup done' dave.boodman@vtext.com < /dev/null</pre>

* configure new kernel.

cd /usr/src/sys/i386/conf
scp backup2:/mnt/data4/build/freebsd/nat2-6.4 ./nat2

* build, install kernel and world

<pre>cd /boot

mv kernel kernel.GENERIC
cd kernel.GENERIC
cd /usr/src
make buildkernel installkernel

make buildworld ; mail -s 'buildworld done' support@johncompanies.com < /dev/null
make installworld
mergemaster -i
</pre>

* populate /etc/rc.conf with IPs and NFS settings
<pre>vi /etc/rc.conf

hostname="nat2.johncompanies.com"
kern_securelevel_enable="NO"
portmap_enable="NO"
sendmail_enable="NO"
usbd_enable="YES"

xntpd_enable="YES"
xntpd_flags="-A -p /var/run/ntpd.pid"

nfs_client_enable="YES"
nfs_reserved_port_only="YES"
ifconfig_em0="inet 10.1.6.50 netmask 255.255.255.0"
#ifconfig_em0="inet 69.55.229.2 netmask 255.255.255.0"
#ifconfig_em0_alias0="inet 69.55.229.229 netmask 255.255.255.255"
ifconfig_fxp0="inet 69.55.229.2 netmask 255.255.255.0"
ifconfig_fxp0_alias0="inet 69.55.229.3 netmask 255.255.255.255"
ifconfig_fxp1="inet 10.1.2.1 netmask 255.255.255.0"
defaultrouter="10.1.6.1"
#defaultrouter=" 66.181.14.250"
snmpd_enable="YES"
ipnat_enable="YES"
ipnat_rules="/etc/ipnat.rules"
gateway_enable="YES"

inetd_enable="YES"
inetd_flags="-wW -a 10.1.2.1"
fsck_y_enable="YES"
background_fsck="NO"
sshd_enable="YES"</pre>

* reboot. Confirm new kernel is loaded

uname -a

* update ports:
<pre>cd /usr/ports
echo "*default host=cvsup4.FreeBSD.org\
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_6_4\
*default delete use-rel-suffix\
*default compress\
ports-all tag=." > sup

cvsup sup; mail -s 'cvs sup ports done' dave.boodman@vtext.com < /dev/null</pre>

* Install raid mgmt tool

<pre>cd /usr/local/sbin
fetch http://3ware.com/download/Escalade9690SA-Series/9.5.3/tw_cli-freebsd-x86-9.5.3.tgz
tar xzf tw_cli-freebsd-x86-9.5.3.tgz
rm tw_cli-freebsd-x86-9.5.3.tgz
chmod 0700 tw_cli</pre>

Test:
./tw_cli info c0

* install rsync from ports
<pre>cd /usr/ports/net/rsync
make install clean</pre>
choose default options

* install perl from ports
<pre>cd /usr/ports/lang/perl5.8
make install clean</pre>

* install screen from ports
<pre>cd /usr/ports/sysutils/screen
make install clean</pre>

* install bb client
<pre>adduser
Username: bb
Full name: bb
Uid (Leave empty for default): 1984
Login group [bb]:
Login group is bb. Invite bb into other groups? []:
Login class [default]:
Shell (sh csh tcsh nologin) [sh]:
Home directory [/home/bb]:
Use password-based authentication? [yes]:
Use an empty password? (yes/no) [no]:
Use a random password? (yes/no) [no]: yes
Lock out the account after creation? [no]:
Username : bb
Password : <random>
Full Name : bb
Uid : 1984
Class :
Groups : bb
Home : /home/bb
Shell : /bin/sh
Locked : no
OK? (yes/no): yes

cd /usr/home/bb
scp backup2:/mnt/data4/build/bb/bb-freebsd.tar .
tar xvf bb-freebsd.tar</pre>
edit /home/bb/bbc1.9e-btf/etc/bb-hosts with something like:
<pre>echo "69.55.230.2 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh \
10.1.2.1 nat2.johncompanies.com # ssh" > /home/bb/bbc1.9e-btf/etc/bb-hosts
vi /home/bb/bbc1.9e-btf/ext/openfiles
MACHINE="nat2,johncompanies,com" # HAS TO BE IN A,B,C FORM

cd /usr/home/bb/bbc1.9e-btf/etc
./bbchkcfg.sh
(y to questions)
./bbchkhosts.sh
(ignore ssh errors)
cd ../..
chown -R bb .
su bb
cd
cd bbc1.9e-btf/src
make; make install
cd ..

vi /usr/home/bb/bbc1.9e-btf/bin/bb-cpu.sh
$1 $TOPARGS > $BBTMP/TOP.$$
# /usr/local/jail/bin/jtop > $BBTMP/TOP.$$

./runbb.sh start
more BBOUT
(look for errors)
exit

echo 'su - bb -c "cd /home/bb/bbc1.9e-btf; ./runbb.sh start"' > /usr/local/etc/rc.d/bb.sh
chmod +x /usr/local/etc/rc.d/bb.sh
</pre>

Punch a hole in the firewall to allow it to communicate with bb monitor (probably already exists):

ipfw add 96 allow ip from 66.181.18.0/27 to 69.55.230.2

* configure bb on mail:
<pre>vi /usr/home/bb/bbsrc/bb1.9e-btf/etc/bb-hosts
64.163.14.48 nat2.johncompanies.com # ssh

su bb
cd
bbsrc/bb/runbb.sh restart ; exit</pre>

* configure ntp
<pre>echo "server 69.55.230.2
server 0.pool.ntp.org
server 1.pool.ntp.org
server 2.pool.ntp.org
server 3.pool.ntp.org
server 66.187.233.4
server 217.204.76.170
server 64.112.189.11
server 66.69.112.130
server 80.85.129.25
server 80.237.234.15
server 130.60.7.44
server 134.99.176.3
server 198.144.202.250
server 202.74.170.194
server 204.17.42.199
server 204.87.183.6
server 213.15.3.1
server 213.239.178.33
server 217.114.97.97
server 69.55.230.2" > /etc/ntp.conf</pre>

<pre>/usr/sbin/ntpd -A -p /var/run/ntpd.pid
sleep 2; ntpq -p</pre>
(confirm it’s able to reach our time server)

echo '/usr/sbin/ntpd -c /etc/ntp.conf -p /var/run/ntpd.pid -f /var/db/ntpd.drift' > /usr/local/etc/rc.d/ntp.sh
chmod 0700 /usr/local/etc/rc.d/ntp.sh

* fwd and reverse lookups on ns1c
vr johncompanies.com
(edit the PTR too)

* setup backups, nfs mount

<pre>mkdir /backup3
echo 'backup3:/data /backup3 nfs rw,bg 0 0' >> /etc/fstab

echo '#\!/bin/sh\
backupdir=/data/nat2/current\
\
## ENTRY /etc ' > /usr/local/etc/backup.config</pre>

on backup3:
setup backup dirs:
ssh backup3 mkdir -p /data/nat2/current

on backup3, add the system to
vi /usr/local/sbin/snapshot_archive

<pre>scp 69.55.230.2:/usr/local/etc/rsync.backup /usr/local/etc/rsync.backup
vi /usr/local/etc/rsync.backup
backup1 > backup3</pre>

crontab -e
1 0 * * * /usr/local/etc/rsync.backup

* edit sshd_config for security
<pre>vi /etc/ssh/sshd_config
ListenAddress 66.181.18.1
ListenAddress 69.55.229.2
ListenAddress 10.1.2.1

kill -1 `cat /var/run/sshd.pid`</pre>

* raid chk

<pre>cat > /usr/local/sbin/lsiraidchk
#!/usr/bin/perl

my @out = split "\n", `megarc -ldInfo -a0 -Lall|grep Status:`;

foreach (@out) {
if ($_ =~ /DEGRADED/) { $date = `date`; chomp $date; `echo "$date: RAID ARRAY DEGRADED" >> /var/log/messages`; }
#print $_;
}</pre>

* netflow stuff
add crontab entries
<pre>crontab -e
30 3 * * * /usr/local/etc/rsync.backup
0 0 1 * * /sbin/ipfw zero
0 0 1 * * /sbin/ipfw del 3 4 5
59 23 30 * * /sbin/ipfw show > /tmp/ipfw_count
3 0 30 * * /sbin/ipfw show > /tmp/ipfw_count
0 3 * * * /etc/makefwrules.pl; /etc/makepiperules.pl;
*/5 * * * * /usr/local/sbin/lsiraidchk

#10 0 * * * rm /var/spool/clientmqueue/*</pre>

<pre>scp /etc/makefwrules.pl user@64.163.14.48:~
scp /etc/makepiperules.pl user@64.163.14.48:~
mv /home/user/makefwrules.pl /etc
mv /home/user/makepiperules.pl /etc
touch /etc/firewall.sh
mkdir /etc/oldrules/</pre>

other binaries

<pre>scp /usr/local/bin/rulemaker user@64.163.14.48:~
mv ~user/rulemaker /usr/local/sbin
scp ~user/Sendmail.pm user@64.163.14.48:~
scp ~user/doswatch.pl user@64.163.14.48:~</pre>

* add nat rules
<pre>vi /etc/ipnat.rules
# sample entry
bimap fxp0 10.1.6.70/32 -> 10.1.6.59/32
#bimap fxp0 10.1.4.208/32 -> 69.55.233.196/32

ipnat -C -f /etc/ipnat.rules</pre>

* shell for user
<pre>cp /root/.cshrc ~user/
vi ~user/</pre>
change # to $

* mrtg

<pre>cd /usr/ports/net-mgmt/mrtg
make install clean</pre>
(no FONTCONFIG, v3)

this didn't work cause of libtool incompat

so manually moved files:

<pre>scp /usr/local/bin/cfgmaker user@nat2:/usr/local/bin/cfgmaker
scp /usr/local/lib/perl5/site_perl/5.6.1/MRTG_lib.pm root@nat2:/usr/local/lib/perl5/site_perl/5.8.9/
scp /usr/local/lib/perl5/site_perl/5.6.1/SNMP_util.pm root@nat2:/usr/local/lib/perl5/site_perl/5.8.9/
scp /usr/local/lib/perl5/site_perl/5.6.1/BER.pm root@nat2:/usr/local/lib/perl5/site_perl/5.8.9/
scp /usr/local/lib/perl5/site_perl/5.6.1/SNMP_Session.pm root@nat2:/usr/local/lib/perl5/site_perl/5.8.9/
scp /usr/local/bin/mrtg root@nat2:/usr/local/bin/mrtg
scp /usr/local/lib/perl5/site_perl/5.6.1/locales_mrtg.pm root@nat2:/usr/local/lib/perl5/site_perl/5.8.9/locales_mrtg.pm
scp /usr/local/bin/rrdtool root@nat2:/usr/local/bin/rrdtool
scp /usr/local/lib/perl5/site_perl/5.6.1/mach/RRDs.pm root@nat2:/usr/local/lib/perl5/site_perl/5.8.9/mach/RRDs.pm
rsync -av /usr/local/lib/perl5/site_perl/5.6.1/mach/auto/RRDs/ root@nat2:/usr/local/lib/perl5/site_perl/5.8.9/mach/auto/RRDs/
scp /usr/lib/libz.so.2 root@nat2:/usr/lib/libz.so.2
scp /usr/lib/libm.so.2 root@nat2:/usr/lib/libm.so.2
rsync -av /usr/local/lib/librrd* root@nat2:/usr/local/lib/
scp /usr/lib/libc.so.4 root@nat2:/usr/lib/libc.so.4

rsync -av /usr/ports/net/rrdtool root@nat2:/usr/ports/net
cd /usr/ports/net/rrdtool
make install

mkdir -p /mnt/data1/mrtg/data
scp /usr/local/www/mgmt/mrtg/template.pl root@nat2:/mnt/data1/mrtg/
scp /usr/local/www/mgmt/mrtg/host.pl root@nat2:/mnt/data1/mrtg/

cfgmaker --if-template=template.pl --show-op-down --global "options[_]: growright,bits" --global 'WorkDir: /mnt/data1/mrtg/data' --global 'Interval: 1' --global 'LogFormat: rrdtool' --global 'PathAdd: /usr/local/bin' --global 'LibAdd: /usr/local/lib' --host-template=host.pl jc292401@10.1.2.50 --output=switch-p20.cfg

cat > /mnt/data1/mrtg/mrtg.sh
#!/bin/sh
/usr/local/bin/mrtg /mnt/data1/mrtg/switch-p20.cfg

chmod 0700 /mnt/data1/mrtg/mrtg.sh

crontab -e
* * * * * /mnt/data1/mrtg/mrtg.sh 2>&1 > /dev/null</pre>

* snmp firewall block
<pre>cat > /usr/local/etc/rc.d/boot.sh
ipfw add 10 allow udp from 69.55.230.2 to any 161
ipfw add 10 allow udp from 10.1.2.1 to any 161
ipfw add 11 deny udp from any to any 161
chmod 0700 /usr/local/etc/rc.d/boot.sh</pre>

= bwdb =
== Summary ==
This machine tracks and stores network traffic (netflow) at castle. It is our means to monitor customer bandwidth usage.

* Location: castle, cab 3-7
* OS: FreeBSD 4.10 x86
* Networking: Priv IP: 10.1.4.203 There are 2 onboard nic's, one of which is the "listener"
* Hardware: Custom 1U. Single power supply.
* Drives: one 250 GB (2 x 250GB) RAID1 array running on a Promise IDE RAID card.

== Services Provided ==
* netflow
* mysql
* bigbrother
* snmp

== netflow ==

The main function of this server is to run netflow on an eth device in promiscuous mode so as to hear everything happening on the port (wherein all network traffic is mirrored to that port via the cisco swith). Every 15min, it creates a flow file under <tt>/usr/home/flows/</tt> (organized by date). The flow file contains all traffic data for a 15min increment of time.

A cronjob moves that flow file (or files if there are multiple due to some delay)
1,16,31,46 * * * * /usr/home/flowbin/queue.pl

into a processing queue:
<tt>/usr/home/working</tt>

Then a separate file processes whatever flow files it finds there, inserting the data into the local mysql database:
2,17,32,47 * * * * /usr/home/flowbin/processflows.pl

== mysql ==

The database storing all the traffic data is named <tt>traffic</tt>
Tables:
<pre>mysql> show tables;
+---------------------------+
| Tables_in_traffic |
+---------------------------+
| dailyIpTotals_69_55_224 |
| dailyIpTotals_69_55_225 |
| dailyIpTotals_69_55_226 |
| dailyIpTotals_69_55_227 |
| dailyIpTotals_69_55_228 |
| dailyIpTotals_69_55_229 |
| dailyIpTotals_69_55_230 |
| dailyIpTotals_69_55_231 |
| dailyIpTotals_69_55_232 |
| dailyIpTotals_69_55_233 |
| dailyIpTotals_69_55_234 |
| dailyIpTotals_69_55_235 |
| dailyIpTotals_69_55_236 |
| dailyIpTotals_69_55_237 |
| dailyIpTotals_69_55_238 |
| dailyIpTotals_69_55_239 |
| dailyPortTotals_69_55_224 |
| dailyPortTotals_69_55_225 |
| dailyPortTotals_69_55_226 |
| dailyPortTotals_69_55_227 |
| dailyPortTotals_69_55_228 |
| dailyPortTotals_69_55_229 |
| dailyPortTotals_69_55_230 |
| dailyPortTotals_69_55_231 |
| dailyPortTotals_69_55_232 |
| dailyPortTotals_69_55_233 |
| dailyPortTotals_69_55_234 |
| dailyPortTotals_69_55_235 |
| dailyPortTotals_69_55_236 |
| dailyPortTotals_69_55_237 |
| dailyPortTotals_69_55_238 |
| dailyPortTotals_69_55_239 |
| ipTotals_69_55_224 |
| ipTotals_69_55_225 |
| ipTotals_69_55_226 |
| ipTotals_69_55_227 |
| ipTotals_69_55_228 |
| ipTotals_69_55_229 |
| ipTotals_69_55_230 |
| ipTotals_69_55_231 |
| ipTotals_69_55_232 |
| ipTotals_69_55_233 |
| ipTotals_69_55_234 |
| ipTotals_69_55_235 |
| ipTotals_69_55_236 |
| ipTotals_69_55_237 |
| ipTotals_69_55_238 |
| ipTotals_69_55_239 |
| portTotals_69_55_224 |
| portTotals_69_55_225 |
| portTotals_69_55_226 |
| portTotals_69_55_227 |
| portTotals_69_55_228 |
| portTotals_69_55_229 |
| portTotals_69_55_230 |
| portTotals_69_55_231 |
| portTotals_69_55_232 |
| portTotals_69_55_233 |
| portTotals_69_55_234 |
| portTotals_69_55_235 |
| portTotals_69_55_236 |
| portTotals_69_55_237 |
| portTotals_69_55_238 |
| portTotals_69_55_239 |
+---------------------------+
</pre>

So as you see we store each class-C block in its own table, for efficiency. Further, we store and organize data in 4 ways: "daily" tables and 15-minute granularity tables, and for each of those we track simple IP traffic and port-specific traffic. The daily tables contains 2 entries (one for each direction) for each IP for each day. For the current day, the row data is incremented as the day goes on.

<pre>mysql> describe dailyIpTotals_69_55_224;
+-----------+-------------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+-----------+-------------+------+-----+---------+-------+
| id | varchar(23) | | PRI | | |
| date | date | YES | | NULL | |
| ip | varchar(15) | YES | MUL | NULL | |
| direction | tinyint(1) | YES | | NULL | |
| octets | bigint(12) | YES | | NULL | |
| packets | int(11) | YES | | NULL | |
+-----------+-------------+------+-----+---------+-------+

mysql> select * from dailyIpTotals_69_55_224 limit 1\G
*************************** 1. row ***************************
id: 6955224194-20100917-1
date: 2010-09-17
ip: 69.55.224.194
direction: 1
octets: 8821
packets: 91
</pre>

The <tt>id</tt> is a unique identifier (key), <tt>direction</tt> indicates incoming or outgoing traffic (outbound = 2, inbound = 1), <tt>octets</tt> are the amount of traffic in kilobytes, and <tt>packets</tt> is the total number of packets.

The 15-minute table has similar information, but it's organized in 15 minute increments:

<pre>mysql> describe ipTotals_69_55_224;
+-----------+------------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+-----------+------------+------+-----+---------+-------+
| date | datetime | YES | | NULL | |
| ip | char(15) | YES | MUL | NULL | |
| direction | tinyint(1) | YES | | NULL | |
| octets | bigint(20) | YES | | NULL | |
| packets | int(11) | YES | | NULL | |
+-----------+------------+------+-----+---------+-------+

mysql> select * from ipTotals_69_55_224 limit 2\G
*************************** 1. row ***************************
date: 2010-01-11 19:30:00
ip: 69.55.224.13
direction: 1
octets: 288
packets: 6
*************************** 2. row ***************************
date: 2010-01-11 19:30:00
ip: 69.55.224.12
direction: 1
octets: 216
packets: 4</pre>

So for a given IP, there will be 192 rows in a given day: 4 rows per hour, *2 for 2 directions, *24 for 24hours in a day. Obviously this table is large which is why we broke it down into a daily table for quick, easy, daily-summary access.

That covers the simple traffic tabulation tables. We also track traffic by port:

<pre>mysql> describe dailyPortTotals_69_55_224;
+-----------+-------------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+-----------+-------------+------+-----+---------+-------+
| id | varchar(28) | | PRI | | |
| date | date | YES | | NULL | |
| ip | varchar(15) | YES | MUL | NULL | |
| direction | tinyint(1) | YES | | NULL | |
| protocol | smallint(3) | YES | | NULL | |
| port | int(11) | YES | | NULL | |
| octets | bigint(11) | YES | | NULL | |
| packets | int(11) | YES | | NULL | |
+-----------+-------------+------+-----+---------+-------+
8 rows in set (0.00 sec)

mysql> select * from dailyPortTotals_69_55_224 limit 1\G
*************************** 1. row ***************************
id: 695522496-20091218-1-6-23
date: 2009-12-18
ip: 69.55.224.96
direction: 1
protocol: 6
port: 23
octets: 1796
packets: 30

mysql> select * from portTotals_69_55_224 limit 1\G
*************************** 1. row ***************************
date: 2010-09-07 18:45:00
ip: 69.55.224.254
direction: 1
protocol: 6
port: 99999
octets: 144
packets: 3

</pre>

This is largely the same with 2 more additions: <tt>protocol</tt> (1=ICMP, 6=TCP, 17=UDP), and <tt>port</tt> which we set to 99999 if the traffic is return traffic and the port is above 1024. Obviously the potential for number of rows grows quickly when you consider the addition of port and protocol tracking per IP.

== Regular maintenance ==
*[[Routine_Maintenance#Adaptec_Controllers|Check RAID array]]
* archive data from database
archive_daily.pl 2012 09
This will archive data for the given year and month from the daily summary tables. Generally we want to have a year of history in the database.

archive_15min.pl 2012 09
This will archive data for the given year and month from the 15min-increment tables. Generally, we want to have 6 months of history in the database.

* if space becomes tight, move flow files and exported data to a backup server, both located in <tt>/usr/home/flowbin/archive</tt> and <tt>/usr/home/exported</tt>, respectively

== Slaving ==

If we were going to setup traffic database slaving (we don't do this anymore), perhaps cause the bwdb machine gets busy and it cannot handle traffic requests and netflow, here's how it's done:

On the traffic master:

GRANT REPLICATION SLAVE ON *.* TO 'repl'@'10.1.4.8' IDENTIFIED BY 'qERUG8wf';

in my.cnf:
<pre>bin-log
server-id=1
max_binlog_size=500M
expire_logs_days = 3</pre>

on slave:
in my.cnf:
<pre>server-id = 2
master-host = 10.1.4.203
master-user = repl
master-password = qERUG8wf
master-connect-retry=60
replicate-wild-do-table=traffic.daily%
max_relay_log_size=500M
expire_logs_days = 3

replicate-wild-do-table=traffic.%</pre>

on master:
touch /usr/home/working/.lock
(make sure processflows not running)

<pre>FLUSH TABLES WITH READ LOCK;
cd /usr/home/database/traffic
tar -czf mysql-traffic-snapshot.tgz ./daily*
(~1G)
SHOW MASTER STATUS;
+-----------------+-----------+--------------+------------------+
| File | Position | Binlog_Do_DB | Binlog_Ignore_DB |
+-----------------+-----------+--------------+------------------+
| bwdb-bin.000039 | 154432615 | | |
+-----------------+-----------+--------------+------------------+

(write down info)
UNLOCK TABLES;
scp mysql-traffic-snapshot.tgz 10.1.4.5:/mnt/data1/db/mysql/traffic/</pre>

on slave:
<pre>mkdir /mnt/data1/db/mysql/traffic
cd /mnt/data1/db/mysql/traffic/
tar xzvf mysql-traffic-snapshot.tgz
(restart mysql)
CHANGE MASTER TO MASTER_HOST='10.1.4.203',MASTER_USER='repl',MASTER_PASSWORD='qERUG8wf',MASTER_LOG_FILE='bwdb-bin.000059',MASTER_LOG_POS=482502186;
START SLAVE;</pre>

<pre>cd /usr/home/database/traffic
scp *</pre>

<pre>optimize table dailyPortTotals_69_55_224;
optimize table dailyPortTotals_69_55_225;
optimize table dailyPortTotals_69_55_226;
optimize table dailyPortTotals_69_55_227;
optimize table dailyPortTotals_69_55_228;
optimize table dailyPortTotals_69_55_229;
optimize table dailyPortTotals_69_55_230;
optimize table dailyPortTotals_69_55_231;
optimize table dailyPortTotals_69_55_232;
optimize table dailyPortTotals_69_55_233;
optimize table dailyPortTotals_69_55_234;
optimize table dailyPortTotals_69_55_235;
optimize table dailyPortTotals_69_55_236;
optimize table dailyPortTotals_69_55_237;
optimize table dailyPortTotals_69_55_238;
optimize table dailyPortTotals_69_55_239;</pre>

== Build ==

=== BIOS Config ===
disable quiet boot

set to last state after power loss

set date/time to GMT

enable serial console output (baud rate 115200)

=== Install OS ===

Install FreeBSD 8.3 amd64

* partition map:
<pre>/ 500m
swap 4096m
/var 256m
/tmp 256m
/usr ~</pre>

* edit /etc/make.conf
Castle:
<pre>echo "WITHOUT_X11=yes \
KERNCONF=bwdb \
BOOT_COMCONSOLE_SPEED=115200" >> /etc/make.conf</pre>

i2b:
<pre>echo "WITHOUT_X11=yes \
KERNCONF=bwdb2 \
BOOT_COMCONSOLE_SPEED=115200" >> /etc/make.conf</pre>

* add settings to /boot/loader.conf and /boot.config

<pre>echo "-Dh" >> /boot.config

echo 'console="comconsole,vidconsole" \
boot_multicons="YES" \
boot_serial="YES" \
comconsole_speed="115200"' >> /boot/loader.conf</pre>

* turn off all ttyv's except 0 and 1 in /etc/ttys
also turn on ttyu0, change type to vt100:
<pre>vi /etc/ttys

ttyv2 "/usr/libexec/getty Pc" cons25 off secure
ttyv3 "/usr/libexec/getty Pc" cons25 off secure
ttyv4 "/usr/libexec/getty Pc" cons25 off secure
ttyv5 "/usr/libexec/getty Pc" cons25 off secure
ttyv6 "/usr/libexec/getty Pc" cons25 off secure
ttyv7 "/usr/libexec/getty Pc" cons25 off secure
# Serial terminals
# The 'dialup' keyword identifies dialin lines to login, fingerd etc.
ttyu0 "/usr/libexec/getty std.9600" vt100 on secure

kill -1 1</pre>

on console server:
vi /etc/remote
(rename port to jail8 depending on where and which digi plugged into)
test serial console

* populate hosts
i2b:
<pre>echo "69.55.230.10 backup2" >> /etc/hosts
echo "69.55.230.11 backup1" >> /etc/hosts
echo "10.1.2.3 backup3" >> /etc/hosts</pre>

castle:
<pre>echo "10.1.4.3 backup2 backup2.johncompanies.com" >> /etc/hosts
echo "10.1.4.8 backup1 backup1.johncompanies.com" >> /etc/hosts
echo "10.1.4.4 mail mail.johncompanies.com" >> /etc/hosts
</pre>

* put key in authorized_keys on backup1 and backup2
cd
ssh-keygen -t dsa -b 1024
(default location, leave password blank)

castle:
cat /root/.ssh/id_dsa.pub | ssh backup1 'cat - >> /root/.ssh/authorized_keys'
cat /root/.ssh/id_dsa.pub | ssh backup2 'cat - >> /root/.ssh/authorized_keys'

i2b:
cat /root/.ssh/id_dsa.pub | ssh backup1 'cat - >> /root/.ssh/authorized_keys'
cat /root/.ssh/id_dsa.pub | ssh backup2 'cat - >> /root/.ssh/authorized_keys'
cat /root/.ssh/id_dsa.pub | ssh backup3 'cat - >> /root/.ssh/authorized_keys'

confirm that you can ssh to backup2 and backup1 (and backup3 if at i2b) without getting a login prompt

ssh backup1 hostname
ssh backup2 hostname

* edit root's path and login script:
vi /root/.cshrc

Change alias entries (add G):
<pre>alias la ls -aG
alias lf ls -FAG
alias ll ls -lAG
alias ls ls -AG
</pre>

and alter the prompt, set the following:
set prompt = "`/bin/hostname -s` %/# "

* install cvsup
cd /usr/ports/net/cvsup-without-gui
make install clean; rehash; mail -s 'cvs installed' support@johncompanies.com < /dev/null

* get latest sources for this release:
<pre>cd /usr/src
echo "*default host=cvsup4.freebsd.org\
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_8_3\
*default delete use-rel-suffix\
*default compress\
src-all" > sup

cvsup sup ; mail -s 'cvs sup done' support@johncompanies.com < /dev/null</pre>

* configure new kernel

cd /usr/src/sys/amd64/conf
scp backup2:/mnt/data4/build/freebsd/kern_config-bwdb-8.3-amd64 ./bwdb

Edit config and change name:
vi bwdb
ident bwdb

* build, install kernel and world

<pre>cd /boot

mv kernel kernel.GENERIC
cd kernel.GENERIC
cd /usr/src
make buildkernel installkernel

make buildworld ; mail -s 'buildworld done' support@johncompanies.com < /dev/null
(2450: 1:56min, supermicro: 59mins, 2950: 38mins)
make installworld
(2450: 3min, supermicro: 1min, 2950: :34)
mergemaster -i</pre>

* populate /etc/rc.conf with IPs and NFS settings

castle:
<pre>vi /etc/rc.conf

hostname="bwdb.johncompanies.com"
kern_securelevel_enable="NO"
portmap_enable="NO"
sendmail_enable="NO"
usbd_enable="YES"

xntpd_enable="YES"
nfs_client_enable="YES"
nfs_reserved_port_only="YES"
ifconfig_fxp0="inet 10.1.4.203 netmask 255.255.255.0"
ifconfig_em0="up promisc"
defaultrouter="10.1.4.1"
snmpd_enable="YES"

inetd_enable="YES"
inetd_flags="-wW -a 10.1.4.203"
fsck_y_enable="YES"
background_fsck="NO"
sshd_enable="YES"
ipfw_load="YES"</pre>

i2b:
<pre>vi /etc/rc.conf

hostname="bwdb2.johncompanies.com"
kern_securelevel_enable="NO"
portmap_enable="NO"
sendmail_enable="NO"
usbd_enable="YES"

xntpd_enable="YES"
nfs_client_enable="YES"
nfs_reserved_port_only="YES"
ifconfig_fxp0="inet 10.1.2.4 netmask 255.255.255.0"
ifconfig_em0="up promisc"
defaultrouter="10.1.2.1"
snmpd_enable="YES"

inetd_enable="YES"
inetd_flags="-wW -a 10.1.2.4"
fsck_y_enable="YES"
background_fsck="NO"
sshd_enable="YES"
ipfw_load="YES"</pre>

* reboot. Confirm new kernel is loaded

uname -a

* update ports:
<pre>cd /usr/ports
echo "*default host=cvsup4.FreeBSD.org\
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_8_3\
*default delete use-rel-suffix\
*default compress\
ports-all tag=." > sup

cvsup sup; mail -s 'cvs sup ports done' support@johncompanies.com < /dev/null</pre>

* Install raid mgmt tool

<pre>cd /usr/local/sbin
scp backup2:/d4/build/3ware/tw_cli-freebsd-x86_64-9.5.0.1.tgz .
tar xzf tw_cli-freebsd-x86_64-9.5.0.1.tgz
rm tw_cli-freebsd-x86_64-9.5.0.1.tgz
chmod 0700 tw_cli</pre>

Test:
./tw_cli info c0

Grab raid check script:
scp backup1:/usr/local/sbin/3wraidchk /usr/local/etc

Setup cronjob:
<pre>crontab -e
*/5 * * * * /usr/local/etc/3wraidchk</pre>

* install rsync from ports
cd /usr/ports/net/rsync
make install clean

choose default options

* install perl from ports
cd /usr/ports/lang/perl5.8
make install clean

choose default options

* install bb client

Compiling from source on AMD64 will not work. So, we use a linux-compiled version and rely on linux compat. Linux compat won't install on 8.x - libtool 2.4 need. So, instead we copy(ed) over linux:
rsync -aSHv --exclude=proc --exclude=sys 10.1.4.108:/usr/compat/linux/ /usr/compat/linux/

adduser

Output/response:

<pre>Username: bb
Full name: bb
Uid (Leave empty for default): 1984
Login group [bb]:
Login group is bb. Invite bb into other groups? []:
Login class [default]:
Shell (sh csh tcsh nologin) [sh]:
Home directory [/home/bb]:
Use password-based authentication? [yes]:
Use an empty password? (yes/no) [no]:
Use a random password? (yes/no) [no]: yes
Lock out the account after creation? [no]:
Username : bb
Password : <random>
Full Name : bb
Uid : 1984
Class :
Groups : bb
Home : /home/bb
Shell : /bin/sh
Locked : no
OK? (yes/no): yes</pre>

cd /usr/home/bb
scp backup2:/mnt/data4/build/bb/bb-freebsd_linuxcompat.tgz .
tar xzf bb-freebsd_linuxcompat.tgz

edit /home/bb/bbc1.9e-btf/etc/bb-hosts with something like:

echo "10.1.4.5 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh \
10.1.4.203 bwdb.johncompanies.com # ssh" > /home/bb/bbc1.9e-btf/etc/bb-hosts

Edit for machine name and private IP.

if this machine is at i2b:
echo "69.55.230.2 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh \
10.1.2.4 bwdb2.johncompanies.com # ssh" > /home/bb/bbc1.9e-btf/etc/bb-hosts

<pre>vi /home/bb/bbc1.9e-btf/ext/openfiles

MACHINE="bwdb,johncompanies,com" # HAS TO BE IN A,B,C FORM</pre>
Edit for machine name.

Have bb watch for flow-capture, mysql
<pre>cat >> /home/bb/bbc1.9e-btf/etc/bb-proctab
localhost: flow-capture :
localhost: mysqld :</pre>

<pre>cd /usr/home/bb/bbc1.9e-btf/etc
./bbchkcfg.sh
(y to questions)
./bbchkhosts.sh
(ignore ssh errors)
cd ../..
chown -R bb .
su bb
cd
cd bbc1.9e-btf
./runbb.sh start
more BBOUT
(look for errors)
exit</pre>

Put in script to start bb @ boot:
echo 'su - bb -c "cd /home/bb/bbc1.9e-btf; ./runbb.sh start"' > /usr/local/etc/rc.d/bb.sh
chmod +x /usr/local/etc/rc.d/bb.sh

If this is at i2b, punch a hole in the firewall to allow it to communicate with bb monitor:

ipfw add 00096 allow tcp from 66.181.18.0/27 to 69.55.230.2

* configure bb on mail
<pre>vi /usr/home/bb/bbsrc/bb1.9e-btf/etc/bb-hosts
10.1.4.203 bwdb.johncompanies.com # ssh

su bb
cd
bbsrc/bb/runbb.sh restart ; exit</pre>

* configure ntp server
Castle:
echo "server 10.1.4.1" > /etc/ntp.conf

I2b:
echo "server 10.1.2.1" > /etc/ntp.conf

<pre>/usr/sbin/ntpd -p /var/run/ntpd.pid
sleep 2; ntpq -p</pre>
(confirm it’s able to reach our time server)

<pre>echo '/usr/sbin/ntpd -c /etc/ntp.conf -p /var/run/ntpd.pid -f /var/db/ntpd.drift' > /usr/local/etc/rc.d/ntp.sh
chmod 0700 /usr/local/etc/rc.d/ntp.sh</pre>

* fwd and reverse lookups on ns1c
vr johncompanies.com
(edit the PTR too)

* setup backups
<pre>echo '#\!/bin/sh\
backupdir=/data/bwdb/current\
server=backup1\
\
## ENTRY /etc\
## ENTRY /usr/home/flowbin\
## ENTRY /usr/home/database' > /usr/local/etc/backup.config</pre>

Castle:
setup backup dirs:
ssh backup1 mkdir -p /data/bwdb/current
on backup1, add the system to
vi /usr/local/sbin/snapshot_rotate

I2b:
setup backup dirs:
ssh backup3 mkdir -p /data/bwdb/current
on backup3, add the system to
vi /usr/local/sbin/snapshot_archive

Copy over the backup script:
scp backup2:/d4/bin/freebsd8.x/rsync.backup /usr/local/etc/

Edit rsync.backup and change <tt>config</tt> var to point to correct config file location: <tt>/usr/local/etc/backup.config</tt>

<pre>crontab -e
5 0 * * * /usr/local/etc/rsync.backup</pre>

* make /root/logs
mkdir /root/logs

* edit sshd_config for security

<pre>vi /etc/ssh/sshd_config
ListenAddress 10.1.4.203
PermitRootLogin yes

kill -1 `cat /var/run/sshd.pid`</pre>

Edit for private IP.

* snmp

(Before doing this you may need to take down the firewall and also add to resolv.conf 69.43.143.41)
<pre>
cd /usr/ports/net-mgmt/net-snmp
make install clean
(defaults)

cat >> /etc/rc.conf
snmpd_enable="YES"
snmpd_flags="-a"
snmpd_conffile="/usr/local/share/snmp/snmpd.conf"
snmptrapd_enable="YES"
snmptrapd_flags="-a -p /var/run/snmptrapd.pid"

cat > /usr/local/share/snmp/snmpd.conf
rocommunity jcread 10.1.4.5
rocommunity jcread 10.1.4.202
</pre>

=== netflow ===

Install flow tools:
<pre>cd /usr/ports/net-mgmt/flow-tools
make install clean</pre>
Defaults.

mkdir /usr/home/flows

Flow start script:
echo "/usr/local/bin/flow-capture -w /usr/home/flows -S5 -N -2 0/10.1.4.203/4444" > /usr/local/etc/rc.d/flow-capture.sh

chmod 0700 /usr/local/etc/rc.d/flow-capture.sh

Edit for private IP.

Netgraph start script:
<pre>
cat > /usr/local/etc/rc.d/netgraph.sh

/usr/sbin/ngctl -f- <<-SEQ
mkpeer em0: netflow lower iface0
name em0:lower netflow
connect em0: netflow: upper out0
mkpeer netflow: ksocket export inet/dgram/udp
msg netflow:export connect inet/10.1.4.203:4444
SEQ

#/usr/sbin/ngctl -f- <<-SEQ
#shutdown netflow:
#SEQ

chmod 0700 /usr/local/etc/rc.d/netgraph.sh</pre>
Edit for private IP.

Confirm netflow is running after running scripts:
<pre>newbwdb /usr/ports/net-mgmt/flow-tools# /usr/sbin/ngctl
Available commands:
config get or set configuration of node at <path>
connect Connects hook <peerhook> of the node at <relpath> to <hook>
debug Get/set debugging verbosity level
dot Produce a GraphViz (.dot) of the entire netgraph.
help Show command summary or get more help on a specific command
list Show information about all nodes
mkpeer Create and connect a new node to the node at "path"
msg Send a netgraph control message to the node at "path"
name Assign name <name> to the node at <path>
read Read and execute commands from a file
rmhook Disconnect hook "hook" of the node at "path"
show Show information about the node at <path>
shutdown Shutdown the node at <path>
status Get human readable status information from the node at <path>
types Show information about all installed node types
write Send a data packet down the hook named by "hook".
quit Exit program
+ show netflow:
Name: netflow Type: netflow ID: 00000004 Num hooks: 3
Local hook Peer name Peer type Peer ID Peer hook
---------- --------- --------- ------- ---------
export <unnamed> ksocket 00000005 inet/dgram/udp
out0 em0 ether 00000001 upper
iface0 em0 ether 00000001 lower
+
</pre>

We notice that sometimes flow-capture is failing due to swap exhaustion (even after adding more swap). So we crontab flow-capture to restart (it's ok to start if it's already running, it just quits):

<pre>
crontab -e
#restart flow-capture
*/15 * * * * /usr/local/etc/rc.d/flow-capture.sh
</pre>

==== process flow tools ====

<pre>mkdir /usr/home/flowbin
mkdir /usr/home/working</pre>

Install modules:
<pre>cd /usr/ports/devel/p5-Date-Calc
make install clean
cd /usr/ports/mail/p5-Mail-Sendmail
make install clean</pre>

Queue script:
<pre>
cat > /usr/home/flowbin/queue.pl
#!/usr/bin/perl

use strict;

BEGIN {
push @INC, "/usr/home/flowbin";
}

use date;

my $flowbase = "/usr/home/flows";
#my $flowqueue = "/usr/home/queue";
my $flowqueue = "/usr/home/working";

my ($date, $time) = date::CurrentDateTime();

my $flowdir = mkFlowDir($date);
`mv $flowdir/ft-* $flowqueue`;

if (date::DateWindow($date, $time, $date, "00:00:00", 600)) {
my $newdate = date::AddDays($date, -1);
my $flowdir = mkFlowDir($newdate);
`mv $flowdir/ft-* $flowqueue`;
}

sub mkFlowDir {
my $date = shift;
$date =~ /([0-9]{4}-[0-9]{2})/;
my $yearmonth = $1;
return "$flowbase/$yearmonth/$date";
}
</pre>

Date.pm module:
<pre>
cat > /usr/home/flowbin/date.pm
#!/usr/local/bin/perl
#
# $Header: /usr/cvs/newgw/lib/date.pm,v 1.2 2003/11/24 17:06:02 glenn Exp $
#
# Copyright (c) 2001, 2002, 2003
# e-Monitoring Networks, Inc. All rights reserved.
#
#
#
# date.pl - Higher level functions written on top of Date::Calc

package date;

use strict;
use Date::Calc qw(:all);

sub DayDiff { #calculate the difference in days from two dates
my $date1 = shift;
my $date2 = shift;
my ($year1, $month1, $day1) = &DateToymd($date1);
my ($year2, $month2, $day2) = &DateToymd($date2);
my $diff = &Delta_Days($year1, $month1, $day1, $year2, $month2, $day2);
return $diff;
}

sub AddDays { #adds specified number of days to the supplied date
my $date = shift;
my $days = shift;
my ($year, $month, $day) = &DateToymd($date);
my ($nyear, $nmonth, $nday) = &Add_Delta_Days($year, $month, $day, $days);
my $ndate = &ymdToDate($nyear, $nmonth, $nday);
return $ndate;
}

sub AddHours { #adds specified number of hours to the supplied date and time
my $date = shift;
my $time = shift;
my $addhours = shift;
my $adddays = 0;
if (abs($addhours / 24) >= 1) {
$adddays = int($addhours / 24);
$addhours -= $adddays * 24;
}
my ($year, $month, $day) = &DateToymd($date);
my ($hour, $minute, $second) = &TimeTohms($time);
my ($ny, $nm, $nd, $nh, $nmin, $ns) = &Add_Delta_DHMS($year, $month, $day,
$hour, $minute, $second,
$adddays, $addhours, 0, 0);
my $ndate = &ymdToDate($ny, $nm, $nd);
my $ntime = &hmsToTime($nh, $nmin, $ns);
return $ndate, $ntime;
}

sub AddMinutes {
my $date = shift;
my $time = shift;
my $minutes = shift;
my ($year, $month, $day) = &DateToymd($date);
my ($hour, $minute, $second) = &TimeTohms($time);
my ($ny, $nm, $nd, $nh, $nmin, $ns) = &Add_Delta_DHMS($year, $month, $day,
$hour, $minute, $second,
0, 0, $minutes, 0);
my $ndate = &ymdToDate($ny, $nm, $nd);
my $ntime = &hmsToTime($nh, $nmin, $ns);
return $ndate, $ntime;
}

sub CurrentDateTime { #return the current date and time
my ($y, $m, $d, $h, $min, $s, $z, $z, $z) = &System_Clock;
my $date = &ymdToDate($y, $m, $d);
my $time = &hmsToTime($h, $min, $s);
return $date, $time;
}

sub Currentymd { #return the current year, month and day as separate variables
my ($y, $m, $d, $h, $min, $s, $z, $z, $z) = &System_Clock;
return $y, $m, $d;
}

sub DateToymd { #takes a date and returns year, month, day as individual values
my $date = shift;
if ($date =~ /([0-9]{4})-([0-9]{2})-([0-9]{2})/) {
my $day = $3;
my $month = $2;
my $year = $1;
return $year, $month, $day;
}
return undef;
}

sub TimeTohms { #takes a time and return hours minutes and seconds as individual values
my $time = shift;
if ($time =~ /([0-9]{1,2}):([0-9]{1,2}):([0-9]{1,2})/) {
my $hour = $1;
my $minute = $2;
my $second = $3;
if ($hour !~ /[0-9]{2}/) { $hour = "0$hour"; }
if ($minute !~ /[0-9]{2}/) { $minute = "0$minute"; }
if ($second !~ /[0-9]{2}/) { $second = "0$second"; }
return $hour, $minute, $second;
}
return undef;
}

sub ymdToDate { #takes year, month, day and assembles them into our date format
my $year = shift;
my $month = shift;
my $day = shift;
if (defined($year) && defined($month) && defined ($day)) {
$month = sprintf("%02d", $month);
$day = sprintf("%02d", $day);
return "$year-$month-$day";
}
return undef;
}

sub hmsToTime { #takes hour minute and second and assembles them into our time format
my $hour = shift;
my $minute = shift;
my $second = shift;
if (defined($hour) && defined($minute) && defined ($second)) {
if ($hour !~ /[0-9]{2}/) { $hour = "0$hour"; }
if ($minute !~ /[0-9]{2}/) { $minute = "0$minute"; }
if ($second !~ /[0-9]{2}/) { $second = "0$second"; }
return sprintf ("%02d:%02d:%02d", $hour, $minute, $second);
}
return undef;
}

sub CompareDates { #compares two date and time pairs
my $date1 = shift;
my $time1 = shift;
my $date2 = shift;
my $time2 = shift;

my ($year1, $month1, $day1) = &DateToymd($date1);
my ($hour1, $minute1, $second1) = &TimeTohms($time1);
my ($year2, $month2, $day2) = &DateToymd($date2);
my ($hour2, $minute2, $second2) = &TimeTohms($time2);

# &debug("$year1, $month1, $day1, $year2, $month2, $day2");
my $days = &Delta_Days($year1, $month1, $day1, $year2, $month2, $day2);
if ($days > 0) { return 1;}
if ($days < 0) { return -1;}
if ($days == 0) { #same day, compare times
my $seconds1 = $second1 + (60 * $minute1) + (3600 * $hour1);
my $seconds2 = $second2 + (60 * $minute2) + (3600 * $hour2);
if ($seconds1 < $seconds2) { return 1;}
if ($seconds1 > $seconds2) { return -1;}
if ($seconds1 == $seconds2) { return 0;}
}
return undef;
}

sub DateWindow { #compares two date time pairs to see if they are < X seconds apart
my $date1 = shift;
my $time1 = shift;
my $date2 = shift;
my $time2 = shift;
my $window = shift;

my ($year1, $month1, $day1) = &DateToymd($date1);
my ($hour1, $minute1, $second1) = &TimeTohms($time1);
my ($year2, $month2, $day2) = &DateToymd($date2);
my ($hour2, $minute2, $second2) = &TimeTohms($time2);

my ($day, $hour, $minute, $second) =
&Delta_DHMS($year1, $month1, $day1, $hour1, $minute1, $second1,
$year2, $month2, $day2, $hour2, $minute2, $second2);
$minute *= 60;
$hour *= 3600;
$day *= 86400;
my $total = $second + $minute + $hour + $day;
if (abs($total) < $window) {
return 1;
}
return 0;
}

sub CheckDateOrder { #takes three dates/times, returns true if they are in chronological order
my $date1 = shift;
my $time1 = shift;
my $date2 = shift;
my $time2 = shift;
my $date3 = shift;
my $time3 = shift;
if (&CompareDates($date1, $time1, $date2, $time2) == -1) {
return 0;
}
if (&CompareDates($date2, $time2, $date3, $time3) == -1) {
return 0;
}
return 1;
}

sub EpochSeconds { #calculates number of seconds since the epoch for the given date/time
my $date = shift;
my $time = shift;
my ($year, $month, $day) = &DateToymd($date);
my ($hour, $minute, $second) = &TimeTohms($time);
my ($d, $h, $m, $s) = &Delta_DHMS(1970, 1, 1, 0, 0, 0,
$year, $month, $day, $hour, $minute, $second);
my $seconds = $s + (60 * $m) + (3600 * $h) + (86400 * $d);
return $seconds;
}

sub SecondsToDateTime { #converts seconds since epoch to date/time
my $seconds = shift;
my $days = int($seconds / 86400);
$seconds -= $days * 86400;
my $hours = int($seconds / 3600);
$seconds -= $hours * 3600;
my $minutes = int($seconds / 60);
$seconds -= $minutes * 60;
my ($year, $month, $day, $hour, $minute, $second) =
&Add_Delta_DHMS(1970, 1, 1, 0, 0, 0, $days, $hours, $minutes, $seconds);
$month = sprintf("%02d", $month);
$day = sprintf("%02d", $day);
$hour = sprintf("%02d", $hour);
$minute = sprintf("%02d", $minute);
$second = sprintf("%02d", $second);
return "$year-$month-$day", "$hour:$minute:$second";
}

sub DateToDayName {
my $date = shift;
my ($year, $month, $day) = &DateToymd($date);
my $name = &Day_of_Week_to_Text(&Day_of_Week($year, $month, $day));
$name =~ /^[A-Za-z]{3}/;
$name = $&;
return $name;
}

sub ValiDate {
return @_;

}

sub CheckBusinessDay { # checks to see if date is business day. 1=yes, 0=no
my $date = shift;
my ($year, $month, $day) = &DateToymd($date);
if (Day_of_Week($year,$month,$day) < 6) { return 1; }
else { return 0; }
}

1; #don't remove this line
</pre>

chmod 0700 /usr/home/flowbin/queue.pl

Setup cronjob:
<pre>crontab -e
#move flow data into the queue
1,16,31,46 * * * * /usr/home/flowbin/queue.pl</pre>

==== flow processing: i2b ====
<pre>cat > /usr/home/flowbin/processflows-sql.pl
#!/usr/bin/perl

#use strict;
#$debug=1;
#$dry=1;

my $log = '/usr/home/flowbin/discards.log';

use Data::Dumper;

BEGIN {
push @INC, "/usr/home/flowbin";
}

#my $queuedir = "/usr/home/queue";
my $queuedir = "/usr/home/working";
my $archivedir = "/usr/home/archive";
my $sqldir = "/usr/home/sql";
my $sqldirworking = "/usr/home/sql/tmp";

unless ($dry) {
if (-e "$queuedir/.lock") {
open(FILE, "$queuedir/.lock");
my $pid = <FILE>;
chomp($pid);
close(FILE);
if (kill(0, $pid)) {
#another process is using the queue, bail out
exit(0);
}
else {
#dead lock file, remove it
`rm $queuedir/.lock`;
}
}
open(FILE, "> $queuedir/.lock");
print FILE "$$\n";
close(FILE);
}

opendir(DIR, $queuedir);
my @files = readdir(DIR);
closedir(DIR);

foreach my $file (sort @files) {
unless($file =~ /^\./) {
$file =~ /([0-9]{4}-[0-9]{2}-[0-9]{2})\.([0-9]{2})([0-9]{2})([0-9]{2})/;
my $date = "$1 $2:$3:$4";
my $outfile = "$1-$2:$3.sql";
unless (open (SQL, "+> $sqldirworking/$outfile")) { die "cant open $sqldirworking/$outfile"; }
my $condensedDate = $1;
$condensedDate =~ s/-//g;
my $iptotal = {};
my $protototal = {};
my $porttotal = {};

&debug("started file $file at ");
&debug(`date`);
&debug("getting raw flow data (flow-print)");
`cat $queuedir/$file | /usr/local/bin/flow-print -f 5 > /usr/home/working/tmp-$file`;
&debug("aggregating data at ");
&debug(`date`);
unless (open(DATA, "/usr/home/working/tmp-$file")) { die "can't open: $!"; }
LOOP: while (my $line = readline DATA) {
my @d = split /[\s]+/, $line;
if ($d[0] ne '' && $d[0] ne 'Start') {
my $addr = 0;
my $port = 0;

#Start End Sif SrcIPaddress SrcP DIf DstIPaddress DstP P Fl Pkts Octets
#0 1 2 3 4 5 6 7 8 9 10 11
#|
# outbound = 2, inbound = 1

my (@src_ip) = split '\.', $d[3];
my (@dst_ip) = split '\.', $d[6];

if ($src_ip[0] == 69 && $src_ip[1] == 55 && ($src_ip[2] == 229 || $src_ip[2] == 231)) { # for i2b
$d[2] = 2;
# hack for outbound bulk traffic counted 2x
#if ($src_ip[2] == 231) { $d[11] /= 2; $d[10] /= 2; }
}
# note- this is where we filter out IPs only found at i2b
elsif ($dst_ip[0] == 69 && $dst_ip[1] == 55 && ($dst_ip[2] == 229 || $dst_ip[2] == 231)) { # for i2b
$d[2] = 1;
}
else { next LOOP; }

if ($d[2] == 2) {
$addr = $d[3];
# if the dst-port is low, store that
if ($d[7] <= 1024) { $port = $d[7]; }
# if the src-port is low, store that
elsif ($d[4] <= 1024) { $port = $d[4]; }
else { $port = 99999; }
}
elsif ($d[2] == 1) {
$addr = $d[6];
# if the dst-port is high, assume its return traffic, try to store src-port if low
if ($d[7] > 1024) {
if ($d[4] <= 1024) { $port = $d[4]; }
else { $port = 99999; }
} else {
$port = $d[7];
}
} else {
next LOOP;
}

my (@ip) = split '\.', $addr;
unless ($ip[0] == 69) { next LOOP; }
unless ($ip[1] == 55) { next LOOP; }
unless ($ip[2] == 229 || $ip[2] == 231) { next LOOP; }

my $classC = "$ip[0]_$ip[1]_$ip[2]";

# IP dir
# if ($d[10] < 2147483647) { $iptotal->{$classC}->{$addr}->{$d[2]}->{'pktTotal'} += $d[10]; }
# if ($d[11] < 2147483647) { $iptotal->{$classC}->{$addr}->{$d[2]}->{'octetTotal'} += $d[11]; }
#
# if ($d[10] < 2147483647) { $protototal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{'pktTotal'} += $d[10]; }
# if ($d[11] < 2147483647) { $protototal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{'octetTotal'} += $d[11]; }
#
# if ($d[10] < 2147483647) { $porttotal->{$classC}->{$addr}->{$d[2]}->{$port}->{'pktTotal'} += $d[10]; }
# if ($d[11] < 2147483647) { $porttotal->{$classC}->{$addr}->{$d[2]}->{$port}->{'octetTotal'} += $d[11]; }
$iptotal->{$classC}->{$addr}->{$d[2]}->{'pktTotal'} += $d[10];
$iptotal->{$classC}->{$addr}->{$d[2]}->{'octetTotal'} += $d[11];

$protototal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{'pktTotal'} += $d[10];
$protototal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{'octetTotal'} += $d[11];

$porttotal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{$port}->{'pktTotal'} += $d[10];
$porttotal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{$port}->{'octetTotal'} += $d[11];
}
}
close(DATA);
`rm /usr/home/working/tmp-$file`;
&debug("processing ip totals at ");
&debug(`date`);
foreach my $classC (keys(%{$iptotal})) {
my @values;
foreach my $ip (keys(%{$iptotal->{$classC}})) {
foreach my $dir (keys(%{$iptotal->{$classC}->{$ip}})) {
my $octets = $iptotal->{$classC}->{$ip}->{$dir}->{'octetTotal'};
my $packets = $iptotal->{$classC}->{$ip}->{$dir}->{'pktTotal'};
# $packets = $packets > 2147483647 ? 0 : $packets;
if ($octets > 2147483647) {
my $ddir = $dir==1 ? 'in' : 'out';
#print SQL "$date $ip $ddir $octets\n";
# $octets = 0;
}
# dailyIpTotals.id = ip(no .'s)-yyyymmdd-direction
my $id = "$ip-$condensedDate-$dir";
$id =~ s/\.//g;
push @values, "('$date', '$ip', $dir, $octets, $packets)";
my $sql = "insert into dailyIpTotals_$classC values ('$id', '$date', '$ip', $dir, $octets, $packets) ON DUPLICATE KEY UPDATE octets=octets+$octets, packets=packets+$packets";
print "$sql\n" if $dry;
print SQL "$sql;\n";
# $db->query("insert into ipTotals values ('$date', '$ip', $dir, $octets, $packets)");
}
}

# break inserts into 100 records at a time
&debug("inserting $#values +1 values");
while ($#values > 0) {
my $sql = "insert into ipTotals_$classC values ";
my $max_index = $#values > 100 ? 100 : $#values;
for (my $i=0; $i<=$max_index; $i++) {
$sql .= shift @values;
$sql .= ',';
}
chop $sql;
print "$sql\n" if $dry;
print SQL "$sql;\n";
}
}

# &debug("processing protocol totals at ");
# &debug(`date`);
# foreach my $classC (keys(%{$protototal})) {
# $db->query("lock tables dailyProtoTotals_$classC write") unless $dry;
# my @values;
# foreach my $ip (keys(%{$protototal->{$classC}})) {
# foreach my $dir (keys(%{$protototal->{$classC}->{$ip}})) {
# foreach my $proto (keys(%{$protototal->{$classC}->{$ip}->{$dir}})) {
# my $octets = $protototal->{$classC}->{$ip}->{$dir}->{$proto}->{'octetTotal'};
# my $packets = $protototal->{$classC}->{$ip}->{$dir}->{$proto}->{'pktTotal'};
# # $octets = $octets > 2147483647 ? 0 : $octets;
# # $packets = $packets > 2147483647 ? 0 : $packets;
# # dailyProtoTotals.id = ip(no .'s)-yyyymmdd-direction-proto
# my $id = "$ip-$condensedDate-$dir-$proto";
# $id =~ s/\.//g;
# push @values, "('$date', '$ip', $dir, $proto, $octets, $packets)";
# my $sql = "insert into dailyProtoTotals_$classC values ('$id', '$date', '$ip', $dir, $proto, $octets, $packets) ON DUPLICATE KEY UPDATE octets=octets+$octets, packets=packets+$packets";
# print "$sql\n" if $dry;
# $db->query($sql) unless $dry;
# # $db->query("insert into protoTotals values ('$date', '$ip', $dir, $proto, $octets, $packets)");
# }
# }
# }
# $db->query("unlock tables") unless $dry;
# my $sql = "insert into protoTotals_$classC values ";
# $sql .= join ',', @values;
# $db->query("lock tables protoTotals_$classC write") unless $dry;
# print "$sql\n" if $dry;
# $db->query($sql) unless $dry;
# $db->query("unlock tables") unless $dry;
# }

&debug("processing port totals at ");
&debug(`date`);
foreach my $classC (keys(%{$porttotal})) {
my @values;
foreach my $ip (keys(%{$porttotal->{$classC}})) {
foreach my $dir (keys(%{$porttotal->{$classC}->{$ip}})) {
foreach my $proto (keys(%{$porttotal->{$classC}->{$ip}->{$dir}})) {
foreach my $port (keys(%{$porttotal->{$classC}->{$ip}->{$dir}->{$proto}})) {
my $octets = $porttotal->{$classC}->{$ip}->{$dir}->{$proto}->{$port}->{'octetTotal'};
my $packets = $porttotal->{$classC}->{$ip}->{$dir}->{$proto}->{$port}->{'pktTotal'};
# $octets = $octets > 2147483647 ? 0 : $octets;
# $packets = $packets > 2147483647 ? 0 : $packets;

# dailyPortTotals.id = ip(no .'s)-yyyymmdd-direction-protocol-port
my $id = "$ip-$condensedDate-$dir-$proto-$port";
$id =~ s/\.//g;
push @values, "('$date', '$ip', $dir, $proto, $port, $octets, $packets)";
my $sql = "insert into dailyPortTotals_$classC values ('$id', '$date', '$ip', $dir, $proto, $port, $octets, $packets) ON DUPLICATE KEY UPDATE octets=octets+$octets, packets=packets+$packets";
print "$sql\n" if $dry;
print SQL "$sql;\n";

# $db->query("insert into portTotals values ('$date', '$ip', $dir, $port, $octets, $packets)");
}
}
}
}

# break inserts into 100 records at a time
&debug("inserting $#values +1 values");
while ($#values > 0) {
my $sql = "insert into portTotals_$classC values ";
my $max_index = $#values > 100 ? 100 : $#values;
for (my $i=0; $i<=$max_index; $i++) {
$sql .= shift @values;
$sql .= ',';
}
chop $sql;
print "$sql\n" if $dry;
print SQL "$sql;\n";
}
}

# 12 1 8 1 1= 23
# dailyIpTotals.id = ip(no .'s)-yyyymmdd-direction
# 12 1 8 1 1 3=26
# dailyProtoTotals.id = ip(no .'s)-yyyymmdd-direction-proto
# 12 1 8 1 1 5=28
# dailyPortTotals.id = ip(no .'s)-yyyymmdd-direction-port
#print "finished at ";
#print `date`;
`mv $queuedir/$file $archivedir` unless $dry;
close(SQL);
`bzip2 $sqldirworking/$outfile`;
`mv $sqldirworking/$outfile.bz2 $sqldir/`;
}
}
`rm $queuedir/.lock` unless $dry;

sub debug {
my $message = shift;
if ($debug) {
print "$message\n";
}
}

# var full during ft-v05.2005-03-28.084500-0800 and
# 2005-02-24 69.55.226

# all port/daily totals before 2005-04-07
</pre>

This script sends the sql files to the traffic server for processing:
<pre>cat > /usr/home/flowbin/sendsql.pl
#!/usr/bin/perl

#use strict;
#$debug=1;
#$dry=1;

my $remote = "69.55.233.199";
my $sqldir = "/usr/home/sql";
my $archive = "/usr/home/archive";
my $sqldirremote = "/data/bwdb2/pending/";
my @err;
unless ($dry) {
if (-e "$sqldir/.lock") {
open(FILE, "$sqldir/.lock");
my $pid = <FILE>;
chomp($pid);
close(FILE);
if (kill(0, $pid)) {
#another process is using the queue, bail out
exit(0);
}
else {
#dead lock file, remove it
`rm $sqldir/.lock`;
}
}
open(FILE, "> $sqldir/.lock");
print FILE "$$\n";
close(FILE);
}

opendir(DIR, $sqldir);
my @files = readdir(DIR);
closedir(DIR);

foreach my $file (sort @files) {
next unless $file =~ /bz2$/;

my $r = `scp -Cq $sqldir/$file $remote:$sqldirremote 2>&1`;
# print "scp $sqldir/$file $remote:$sqldirremote";
unless ($?==0) {
push @err, "scp -Cq $sqldir/$file $remote:$sqldirremote ($r)";
}
else {
`mv $sqldir/$file $archive`;
`ssh $remote mv $sqldirremote/$file $sqldirremote/${file}.done`;
}
}

`rm $sqldir/.lock` unless $dry;

if (@err) {
email_support('bwdb2: sendsql.pl error',join "\n", @err);
}

sub email_support {
my $subj=shift;
my $body=shift;
use Mail::Sendmail;

# prepare message
my %mail = (
To => 'support@johncompanies.com,dave@johncompanies.com',
From => 'support@johncompanies.com',
Subject => $subj,
Message => $body,
smtp => 'mail.johncompanies.com',
);
sendmail(%mail) || warn "Error: $Mail::Sendmail::error";
}

sub debug {
my $message = shift;
if ($debug) {
print "$message\n";
}
}

# var full during ft-v05.2005-03-28.084500-0800 and
# 2005-02-24 69.55.226

# all port/daily totals before 2005-04-07
</pre>

<pre>crontab -e
#process flows
2,17,32,47 * * * * /usr/home/flowbin/processflows-sql.pl
#move sql commands to traffic db
8,23,38,53 * * * * /usr/home/flowbin/sendsql.pl
</pre>

==== flow processing: castle ====
<pre>
cat > /usr/home/flowbin/processflows.pl

#!/usr/bin/perl

#use strict;
#$debug=1;
#$dry=1;

my $log = '/usr/home/flowbin/discards.log';

use Data::Dumper;

BEGIN {
push @INC, "/usr/home/flowbin";
}

use db;

#my $queuedir = "/usr/home/queue";
my $queuedir = "/usr/home/working";
my $archivedir = "/usr/home/archive";

unless ($dry) {
if (-e "$queuedir/.lock") {
open(FILE, "$queuedir/.lock");
my $pid = <FILE>;
chomp($pid);
close(FILE);
if (kill(0, $pid)) {
#another process is using the queue, bail out
exit(0);
}
else {
#dead lock file, remove it
`rm $queuedir/.lock`;
}
}
open(FILE, "> $queuedir/.lock");
print FILE "$$\n";
close(FILE);
}

my $db = db->new();
$db->connect('traffic', '', 'root', '5over3') || die $db->{'error'};

opendir(DIR, $queuedir);
my @files = readdir(DIR);
closedir(DIR);

foreach my $file (sort @files) {
unless($file =~ /^\./) {
$file =~ /([0-9]{4}-[0-9]{2}-[0-9]{2})\.([0-9]{2})([0-9]{2})([0-9]{2})/;
my $date = "$1 $2:$3:$4";
my $condensedDate = $1;
$condensedDate =~ s/-//g;
my $iptotal = {};
my $protototal = {};
my $porttotal = {};

&debug("started file $file at ");
&debug(`date`);
&debug("getting raw flow data (flow-print)");
`cat $queuedir/$file | /usr/local/bin/flow-print -f 5 > /usr/home/working/tmp-$file`;
&debug("aggregating data at ");
&debug(`date`);
unless (open(DATA, "/usr/home/working/tmp-$file")) { die "can't open: $!"; }
LOOP: while (my $line = readline DATA) {
my @d = split /[\s]+/, $line;
if ($d[0] ne '' && $d[0] ne 'Start') {
my $addr = 0;
my $port = 0;

#Start End Sif SrcIPaddress SrcP DIf DstIPaddress DstP P Fl Pkts Octets
#0 1 2 3 4 5 6 7 8 9 10 11
#|
# outbound = 2, inbound = 1

my (@src_ip) = split '\.', $d[3];
my (@dst_ip) = split '\.', $d[6];

if ($src_ip[0] == 69 && $src_ip[1] == 55 &&
$src_ip[2] >= 224 && $src_ip[2] <= 239 &&
$src_ip[2] != 229 && $src_ip[2] != 231) { # for castle
# if ($src_ip[0] == 69 && $src_ip[1] == 55 && $src_ip[2] == 229) { # for i2b
$d[2] = 2;
# hack for outbound bulk traffic counted 2x
if ($dst_ip[2] == 234) { $d[11] /= 2; $d[10] /= 2; }
}
elsif ($dst_ip[0] == 69 && $dst_ip[1] == 55 &&
$dst_ip[2] >= 224 && $dst_ip[2] <= 239 &&
$dst_ip[2] != 229 && $dst_ip[2] != 231) { # for castle
# elsif ($dst_ip[0] == 69 && $dst_ip[1] == 55 && $dst_ip[2] == 229) { # for i2b
$d[2] = 1;
}
else { next LOOP; }

if ($d[2] == 2) {
$addr = $d[3];
# if the dst-port is low, store that
if ($d[7] <= 1024) { $port = $d[7]; }
# if the src-port is low, store that
elsif ($d[4] <= 1024) { $port = $d[4]; }
else { $port = 99999; }
}
elsif ($d[2] == 1) {
$addr = $d[6];
# if the dst-port is high, assume its return traffic, try to store src-port if low
if ($d[7] > 1024) {
if ($d[4] <= 1024) { $port = $d[4]; }
else { $port = 99999; }
} else {
$port = $d[7];
}
} else {
next LOOP;
}

my (@ip) = split '\.', $addr;
unless ($ip[0] == 69) { next LOOP; }
unless ($ip[1] == 55) { next LOOP; }
unless ($ip[2] >= 224 && $ip[2] <= 239 && $ip[2] != 229 && $ip[2] != 231) { next LOOP; }
# unless ($ip[2] == 229) { next LOOP; }

my $classC = "$ip[0]_$ip[1]_$ip[2]";

# IP dir
# if ($d[10] < 2147483647) { $iptotal->{$classC}->{$addr}->{$d[2]}->{'pktTotal'} += $d[10]; }
# if ($d[11] < 2147483647) { $iptotal->{$classC}->{$addr}->{$d[2]}->{'octetTotal'} += $d[11]; }
#
# if ($d[10] < 2147483647) { $protototal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{'pktTotal'} += $d[10]; }
# if ($d[11] < 2147483647) { $protototal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{'octetTotal'} += $d[11]; }
#
# if ($d[10] < 2147483647) { $porttotal->{$classC}->{$addr}->{$d[2]}->{$port}->{'pktTotal'} += $d[10]; }
# if ($d[11] < 2147483647) { $porttotal->{$classC}->{$addr}->{$d[2]}->{$port}->{'octetTotal'} += $d[11]; }
$iptotal->{$classC}->{$addr}->{$d[2]}->{'pktTotal'} += $d[10];
$iptotal->{$classC}->{$addr}->{$d[2]}->{'octetTotal'} += $d[11];

$protototal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{'pktTotal'} += $d[10];
$protototal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{'octetTotal'} += $d[11];

$porttotal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{$port}->{'pktTotal'} += $d[10];
$porttotal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{$port}->{'octetTotal'} += $d[11];
}
}
close(DATA);
`rm /usr/home/working/tmp-$file`;
&debug("processing ip totals at ");
&debug(`date`);
foreach my $classC (keys(%{$iptotal})) {
$db->query("lock tables dailyIpTotals_$classC write") unless $dry;
my @values;
foreach my $ip (keys(%{$iptotal->{$classC}})) {
foreach my $dir (keys(%{$iptotal->{$classC}->{$ip}})) {
my $octets = $iptotal->{$classC}->{$ip}->{$dir}->{'octetTotal'};
my $packets = $iptotal->{$classC}->{$ip}->{$dir}->{'pktTotal'};
# $packets = $packets > 2147483647 ? 0 : $packets;
if ($octets > 2147483647) {
my $ddir = $dir==1 ? 'in' : 'out';
`echo "$date $ip $ddir $octets\n" >> $log`;
# $octets = 0;
}
# dailyIpTotals.id = ip(no .'s)-yyyymmdd-direction
my $id = "$ip-$condensedDate-$dir";
$id =~ s/\.//g;
push @values, "('$date', '$ip', $dir, $octets, $packets)";
my $sql = "insert into dailyIpTotals_$classC values ('$id', '$date', '$ip', $dir, $octets, $packets) ON DUPLICATE KEY UPDATE octets=octets+$octets, packets=packets+$packets";
print "$sql\n" if $dry;
$db->query($sql) unless $dry;
# $db->query("insert into ipTotals values ('$date', '$ip', $dir, $octets, $packets)");
}
}
$db->query("unlock tables") unless $dry;

$db->query("lock tables ipTotals_$classC write") unless $dry;
# break inserts into 100 records at a time
&debug("inserting $#values +1 values");
while ($#values > 0) {
my $sql = "insert into ipTotals_$classC values ";
my $max_index = $#values > 100 ? 100 : $#values;
for (my $i=0; $i<=$max_index; $i++) {
$sql .= shift @values;
$sql .= ',';
}
chop $sql;
print "$sql\n" if $dry;
$db->query($sql) unless $dry;
}
$db->query("unlock tables") unless $dry;
}

sleep 20;
# &debug("processing protocol totals at ");
# &debug(`date`);
# foreach my $classC (keys(%{$protototal})) {
# $db->query("lock tables dailyProtoTotals_$classC write") unless $dry;
# my @values;
# foreach my $ip (keys(%{$protototal->{$classC}})) {
# foreach my $dir (keys(%{$protototal->{$classC}->{$ip}})) {
# foreach my $proto (keys(%{$protototal->{$classC}->{$ip}->{$dir}})) {
# my $octets = $protototal->{$classC}->{$ip}->{$dir}->{$proto}->{'octetTotal'};
# my $packets = $protototal->{$classC}->{$ip}->{$dir}->{$proto}->{'pktTotal'};
# # $octets = $octets > 2147483647 ? 0 : $octets;
# # $packets = $packets > 2147483647 ? 0 : $packets;
# # dailyProtoTotals.id = ip(no .'s)-yyyymmdd-direction-proto
# my $id = "$ip-$condensedDate-$dir-$proto";
# $id =~ s/\.//g;
# push @values, "('$date', '$ip', $dir, $proto, $octets, $packets)";
# my $sql = "insert into dailyProtoTotals_$classC values ('$id', '$date', '$ip', $dir, $proto, $octets, $packets) ON DUPLICATE KEY UPDATE octets=octets+$octets, packets=packets+$packets";
# print "$sql\n" if $dry;
# $db->query($sql) unless $dry;
# # $db->query("insert into protoTotals values ('$date', '$ip', $dir, $proto, $octets, $packets)");
# }
# }
# }
# $db->query("unlock tables") unless $dry;
# my $sql = "insert into protoTotals_$classC values ";
# $sql .= join ',', @values;
# $db->query("lock tables protoTotals_$classC write") unless $dry;
# print "$sql\n" if $dry;
# $db->query($sql) unless $dry;
# $db->query("unlock tables") unless $dry;
# }

&debug("processing port totals at ");
&debug(`date`);
foreach my $classC (keys(%{$porttotal})) {
$db->query("lock tables dailyPortTotals_$classC write") unless $dry;
my @values;
foreach my $ip (keys(%{$porttotal->{$classC}})) {
foreach my $dir (keys(%{$porttotal->{$classC}->{$ip}})) {
foreach my $proto (keys(%{$porttotal->{$classC}->{$ip}->{$dir}})) {
foreach my $port (keys(%{$porttotal->{$classC}->{$ip}->{$dir}->{$proto}})) {
my $octets = $porttotal->{$classC}->{$ip}->{$dir}->{$proto}->{$port}->{'octetTotal'};
my $packets = $porttotal->{$classC}->{$ip}->{$dir}->{$proto}->{$port}->{'pktTotal'};
# $octets = $octets > 2147483647 ? 0 : $octets;
# $packets = $packets > 2147483647 ? 0 : $packets;

# dailyPortTotals.id = ip(no .'s)-yyyymmdd-direction-protocol-port
my $id = "$ip-$condensedDate-$dir-$proto-$port";
$id =~ s/\.//g;
push @values, "('$date', '$ip', $dir, $proto, $port, $octets, $packets)";
my $sql = "insert into dailyPortTotals_$classC values ('$id', '$date', '$ip', $dir, $proto, $port, $octets, $packets) ON DUPLICATE KEY UPDATE octets=octets+$octets, packets=packets+$packets";
print "$sql\n" if $dry;
$db->query($sql) unless $dry;
# $db->query("insert into portTotals values ('$date', '$ip', $dir, $port, $octets, $packets)");
}
}
}
}
$db->query("unlock tables") unless $dry;

$db->query("lock tables portTotals_$classC write") unless $dry;
# break inserts into 100 records at a time
&debug("inserting $#values +1 values");
while ($#values > 0) {
my $sql = "insert into portTotals_$classC values ";
my $max_index = $#values > 100 ? 100 : $#values;
for (my $i=0; $i<=$max_index; $i++) {
$sql .= shift @values;
$sql .= ',';
}
chop $sql;
print "$sql\n" if $dry;
$db->query($sql) unless $dry;
}
$db->query("unlock tables") unless $dry;
sleep 10;
}

# 12 1 8 1 1= 23
# dailyIpTotals.id = ip(no .'s)-yyyymmdd-direction
# 12 1 8 1 1 3=26
# dailyProtoTotals.id = ip(no .'s)-yyyymmdd-direction-proto
# 12 1 8 1 1 5=28
# dailyPortTotals.id = ip(no .'s)-yyyymmdd-direction-port
#print "finished at ";
#print `date`;
`mv $queuedir/$file $archivedir` unless $dry;
}
}
`rm $queuedir/.lock` unless $dry;

sub debug {
my $message = shift;
if ($debug) {
print "$message\n";
}
}

# var full during ft-v05.2005-03-28.084500-0800 and
# 2005-02-24 69.55.226

# all port/daily totals before 2005-04-07
</pre>

<pre>
cat > /usr/home/flowbin/db.pm

#!/usr/bin/perl
#
# $Header: /usr/cvs/newgw/lib/db.pm,v 1.4 2003/06/05 18:20:01 glenn Exp $
#
# Copyright (c) 2003
# e-Monitoring Networks, Inc. All rights reserved.
#
#
package db;

use strict;
use DBI;

sub new {
my $class = shift;
my $self = {};

$self->{'debug'} = 0;
bless $self, $class;

return $self;
}

sub connect {
my $self = shift;
my $dbname = shift;
my $dbhost = shift;
my $dbuser = shift;
my $dbpass = shift;

my $host = '';
if (defined($dbhost)) {
$host = ";host=$dbhost";
}

eval {
$self->debug("connecting to: DBI:mysql:database=$dbname;$host", 1);
$self->{'dbh'} = DBI->connect("DBI:mysql:database=$dbname;$host", $dbuser, $dbpass);
};
if ($self->{'dbh'}) {
return 1;
}
$self->{'error'} = "Error connecting to database $@";
$self->debug("Error connecting to database $@");
return 0;
}

sub query {
my $self = shift;
my $query = shift;

$self->debug($query, 1);
my $sth;
eval {
$sth = $self->{'dbh'}->prepare($query);
};
unless ($sth) {
$self->{'error'} = "error preparing query $@";
$self->debug("error preparing query $@");
return undef;
}
my $qty;
eval {
$qty = $sth->execute;
};
unless ($qty) {
$self->{'error'} = "error executing query $@";
warn "error executing query $@ $query";
return undef;
}
$self->debug("returning $qty, $sth from query", 6);
return ($qty, $sth);
}

sub disconnect {
my $self = shift;

$self->{'dbh'}->disconnect;
return 0;
}

sub debug {
my $self = shift;
my $msg = shift;
my $level = shift || 0;

if ($level < $self->{'debug'}) {
print "$msg\n";
}
return 0;
}
1;
</pre>

mkdir /usr/home/archive
mkdir -p /usr/home/sql/tmp

<pre>crontab -e
#process flows
2,17,32,47 * * * * /usr/home/flowbin/processflows.pl</pre>

==== setup traffic db ====
* Install mysql:
<pre>cd /usr/ports/databases/mysql50-server
make install clean</pre>

cat >> /etc/rc.conf
mysql_enable="YES"

Move db data dir:
/usr/local/etc/rc.d/mysql-server stop
mkdir /usr/home/database/
mv /var/db/mysql/* /usr/home/database/
chown -R mysql:mysql /usr/home/database

Edit database location in startup script:
vi /usr/local/etc/rc.d/mysql-server
# : ${mysql_dbdir="/var/db/mysql"}
: ${mysql_dbdir="/usr/home/database"}

/usr/local/etc/rc.d/mysql-server start

* Install mysql perl database modules:
<pre>
cd /usr/ports/databases/p5-DBI
make install clean
cd /usr/ports/databases/p5-DBD-mysql50
make install clean
(no to SSL support)
</pre>

* Setting up database
<pre>
rehash
/usr/local/etc/rc.d/mysql-server start
mysql -u root
create database traffic;
grant all on *.* to root@localhost identified by '5over3';
grant all on traffic.* to jc@10.1.4.5 identified by '2gMKY3Wt';

</pre>

If this was a new server we'd setup new tables. See [[#mysql_2|mysql]] for how those tables would be setup.

We are assuming here we are moving data from an existing db, here's how that's done (from the current traffic db):
rsync -av --progress /usr/home/database/traffic/ 10.1.4.203:/usr/home/database/traffic/

When you're ready to do the cutover, shut down mysql on both hosts and do one last sync.

==== process flows from bwdb2 ====
On traffic database server (bwdb):

<pre>crontab -e
#import sql from bwdb2
10,25,40,55 * * * * /usr/home/flowbin/processsql.pl</pre>

Add access to mysql:
<pre>mysql -u root -p
grant all on traffic.* to bwdb2@localhost identified by 's1lver4d';
</pre>

<pre>cat > /usr/home/flowbin/processsql.pl

#!/usr/bin/perl

#use strict;
#$debug=1;
#$dry=1;

my $sqldir = "/usr/home/bwdb2/pending";
my $mysql = '/usr/local/bin/mysql';
my @err;
unless ($dry) {
if (-e "$sqldir/.lock") {
open(FILE, "$sqldir/.lock");
my $pid = <FILE>;
chomp($pid);
close(FILE);
if (kill(0, $pid)) {
#another process is using the queue, bail out
exit(0);
}
else {
#dead lock file, remove it
`rm $sqldir/.lock`;
}
}
open(FILE, "> $sqldir/.lock");
print FILE "$$\n";
close(FILE);
}

opendir(DIR, $sqldir);
my @files = readdir(DIR);
closedir(DIR);

foreach my $file (sort @files) {
next unless $file =~ /done$/;
my $r = `bzcat $sqldir/$file | $mysql -u bwdb2 -ps1lver4d traffic`;
unless ($?==0) {
push @err, "bzcat $sqldir/$file | $mysql -u bwdb2 -pxxxxx traffic ($r)";
}
else {
`rm $sqldir/$file`;
}
}

`rm $sqldir/.lock` unless $dry;

if (@err) {
email_support('bwdb: processsql.pl error',join "\n", @err);
}

sub email_support {
my $subj=shift;
my $body=shift;
use Mail::Sendmail;

# prepare message
my %mail = (
To => 'dave@johncompanies.com',
From => 'support@johncompanies.com',
Subject => $subj,
Message => $body,
smtp => 'mail.johncompanies.com',
);
sendmail(%mail) || warn "Error: $Mail::Sendmail::error";
}

sub debug {
my $message = shift;
if ($debug) {
print "$message\n";
}
}
</pre>

chmod 0700 /usr/home/flowbin/processsql.pl

Make sure bwdb is reachable from the outside only to bwdb2:

On nat, add to <tt>/etc/ipnat.rules</tt>
<pre># bwdb
bimap fxp0 10.1.4.203/32 -> 69.55.233.199/32</pre>

Reload:
ipnat -C -F -f /etc/ipnat.rules

Setup firewall rule on firewall:
ipfw add 00094 allow ip from 66.181.18.5 to 69.55.233.199 22
ipfw add 00094 deny ip from any to 69.55.233.199

Setup firewall on bwdb to restrict access now that it's nat'd:
<pre>
cat >> /usr/local/etc/rc.d/boot.sh
ipfw add 1 allow tcp from any to any established
ipfw add 2 allow ip from 10.1.4.0/24,66.181.18.5,69.55.233.195 to me 22
ipfw add 3 allow ip from 10.1.4.5 to me 3306
ipfw add 4 allow ip from 69.55.225.225 53 to me
ipfw add 5 allow ip from 69.55.230.2 25 to me
ipfw add 6 allow ip from me to me 4444
ipfw add 7 allow icmp from any to me
ipfw add 8 allow udp from 10.1.4.203 to 10.1.4.203 dst-port 4444
ipfw add 9 allow udp from 10.1.4.5 to me 161
ipfw add 100 deny ip from any to me
</pre>

chmod 0700 /usr/local/etc/rc.d/boot.sh

From bwdb2, add ssh key:
cat /root/.ssh/id_dsa.pub | ssh 69.55.233.199 'cat - >> /root/.ssh/authorized_keys'

Confirm no password access:
ssh 69.55.233.199 hostname

= bwdb2 =
== Summary ==
This machine tracks and stores network traffic (netflow) at i2b. It is our means to monitor customer bandwidth usage.

* Location: i2b, cab6
* OS: FreeBSD 6.4 x86
* Networking: Priv IP: 10.1.2.4 There are 2 onboard nic's, one of which is the "listener"
* Hardware: Custom 2U. Single power supply.
* Drives: two 150 GB (2 x 150GB) RAID1 arrays running on a 3ware 7006 RAID card.

== Services Provided ==
* netflow
* bigbrother

== netflow ==

The main function of this server is to run netflow on an eth device in promiscuous mode so as to hear everything happening on the port (wherein all network traffic is mirrored to that port via the cisco swith). Every 15min, it creates a flow file under <tt>/usr/home/flows/</tt> (organized by date). The flow file contains all traffic data for a 15min increment of time.

A cronjob moves that flow file (or files if there are multiple due to some delay)
1,16,31,46 * * * * /usr/home/flowbin/queue.pl

into a processing queue:
<tt>/usr/home/working</tt>

Then a separate file processes whatever flow files it finds there, and builds sql files ready for insertion into the traffic database:
2,17,32,47 * * * * /usr/home/flowbin/processflows-sql.pl

Then yet another process copies the sql files to the traffic database server for processing and insertion into the mysql database:
8,23,38,53 * * * * /usr/home/flowbin/sendsql.pl

== Regular maintenance ==
*[[Routine_Maintenance#Adaptec_Controllers|Check RAID array]]

* if space becomes tight, move sql files and flow files to backup server, both located in <tt>/usr/home/flowbin/archive</tt>

= firewall (newgateway) =

== Summary ==

This machine is the primary (only) firewall for the entire network at castle.

* Location: castle, cab 3-8
* OS: FreeBSD 4.11 x86
* Networking: Priv IP: 10.1.4.223, Pub IPs: 69.55.233.164 (external), 69.55.233.156 (internal). It has 3 network connections (2 onboard, 1 PCI) connecting to the external, internal and private networks. If you're looking at the back of the server, the internal-network-facing nic is on the right (em1), and the external-facing-network (3750) is on the left (em0).
* Hardware: 6 SCSI SCA drive bays (2 columns of 3, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: 36 GB (2 x 36GB) RAID1 array running on an Adaptec 2120S PCI RAID card.

== Services Provided ==
* firewall (ipfw)
* snmp
* bigbrother

== Firewall Rule Configuration ==

See [[FreeBSD_Reference#Firewall_Rule_Configuration|Firewall Rule Configuration]] for more discussion on how to actually manipulate firewall rules.

== Disaster Recovery ==

If there is ever an outage with the firewall, the old firewall "gate" is located just below and is running with the proper network configuration, but with no firewall rules in place (to facilitate good throughput). Have castle move the cable on the left on the current firewall to the left port in the old firewall and the right cable to the right port.

Here's what you need to put in /etc/rc.conf to get a firewall going (as far as routes and IPs)

<pre>hostname="newgateway.johncompanies.com"
firewall_script="/etc/firewall.sh"
firewall_enable="NO"
sendmail_enable="NONE"
sshd_enable="YES"
inetd_enable="NO"
xntpd_enable="YES"
snmpd_enable="YES"
#snmpd_flags="-as -p /var/run/snmpd.pid"
#ipnat_enable="YES"
#ipnat_rules="/etc/ipnat.rules"
gateway_enable="YES"

defaultrouter="69.55.233.161"

ifconfig_xl0="inet 10.1.4.223 netmask 255.255.255.0"
ifconfig_em0="inet 69.55.233.164 netmask 255.255.255.248"

#
# Original JohnCompanies 69.55.224.0/20
#
ifconfig_em1="inet 69.55.233.156 netmask 255.255.255.248"

static_routes="route1 route2 route3 route4 route5 route6 route7 route8 route9 route10 route11 route1
2 route13 route14 route15 route16 route17 route18"

route_route1="-net 69.55.224.0 69.55.233.153"
route_route2="-net 69.55.225.0 69.55.233.153"
route_route3="-net 69.55.226.0 69.55.233.153"
route_route4="-net 69.55.227.0 69.55.233.153"
route_route5="-net 69.55.228.0 69.55.233.153"
route_route6="-net 69.55.229.0 69.55.233.153"
route_route7="-net 69.55.230.0 69.55.233.153"
route_route8="-net 69.55.231.0 69.55.233.153"
route_route9="-net 69.55.232.0 69.55.233.153"
route_route10="-net 69.55.233.0 69.55.233.153"
route_route11="-net 69.55.234.0 69.55.233.153"
route_route12="-net 69.55.235.0 69.55.233.153"
route_route13="-net 69.55.236.0 69.55.233.153"
route_route14="-net 69.55.237.0 69.55.233.153"
route_route15="-net 69.55.238.0 69.55.233.153"
route_route16="-net 69.55.239.0 69.55.233.153"
route_route17="-net 10.1.5.0 10.1.4.2"
route_route18="-net 10.1.6.0 10.1.4.2"

#In case of 3750 failure:
#defaultrouter="69.43.128.81"
#ifconfig_em0="inet 69.43.129.84 netmask 255.255.255.248"

#bind .1's here:
#ifconfig_em1="inet 69.55.224.1 netmask 255.255.255.0"
#ifconfig_em1_alias0="inet 69.55.225.1 netmask 255.255.255.0"
#ifconfig_em1_alias1="inet 69.55.226.1 netmask 255.255.255.0"
#ifconfig_em1_alias2="inet 69.55.227.1 netmask 255.255.255.0"
#ifconfig_em1_alias3="inet 69.55.228.1 netmask 255.255.255.0"
#ifconfig_em1_alias4="inet 69.55.229.1 netmask 255.255.255.0"
#ifconfig_em1_alias5="inet 69.55.230.1 netmask 255.255.255.0"
#ifconfig_em1_alias6="inet 69.55.231.1 netmask 255.255.255.0"
#ifconfig_em1_alias7="inet 69.55.232.1 netmask 255.255.255.0"
#ifconfig_em1_alias8="inet 69.55.233.1 netmask 255.255.255.0"
#ifconfig_em1_alias9="inet 69.55.234.1 netmask 255.255.255.0"
#ifconfig_em1_alias10="inet 69.55.235.1 netmask 255.255.255.0"
#ifconfig_em1_alias11="inet 69.55.236.1 netmask 255.255.255.0"
#ifconfig_em1_alias12="inet 69.55.237.1 netmask 255.255.255.0"
#ifconfig_em1_alias13="inet 69.55.238.1 netmask 255.255.255.0"
#ifconfig_em1_alias14="inet 69.55.239.1 netmask 255.255.255.0"

#bulk:
# reassign 69.55.231.1 to the int iface on the firewall
# set the DG on the firewall to 69.43.138.9
# set the ext firewall IP to 69.43.138.12, NM: 255.255.255.248</pre>

== Cronjobs ==
1 0 * * * /usr/local/etc/rsync.backup
Backup to backup1

0 0 1 * * /sbin/ipfw zero
0 0 1 * * /sbin/ipfw del 3 4 5 17331
Reset counters and remove pipe rules on the 1st of the month. Pay attention when setting up a rule as 3 4 5 (that's not a temporary traffic cap).

Inside <tt>/etc/daily.local</tt> you will see a call to <tt>/etc/makepiperules.pl</tt>
This script will create <tt>/etc/firewall.sh</tt> which contains all the firewall and pipe rules in place at the time the script was run.

== DOS attacks ==

See [[FreeBSD_Reference#Handling_a_DoS_attack|Handling a DoS attack]] regarding how to handle a DOS attack.

Theres a background process (running from user shell) that monitors the firewall for incoming UDP DoS attacks. When it notices packets above a certain level it will
# enter a rule that allows all UDP to go through
# send an emergency email to support and indicating an attack is in progress
# send an email to castle (nocstaff@castleaccess.com and jcsupport@castleaccess.com) telling them to investigate and put up a null if warranted
# wait for a couple minutes to see if the attack subsides- if so it will remove the pass-all UDP rule, if not it will repeat the process from #1
This file lives under /usr/home/user/doswatch.pl
To run:
cd /usr/home/user
./doswatch.pl &

To kill;
fg
^C

It writes its findings to /usr/home/user/doswatch.log

= backup1 =

== Summary ==

This machine acts as the primary backup location for all VPS-based customers. No customer directly accesses this server to perform their backups. We also store cancelled customers on this server.

* Location: castle, cab 3-8
* OS: Ubuntu 8.04.1 server x86
* Networking: Priv IP: 10.1.4.8, Pub IP: 69.55.230.11 (firewalled from all but JC infrastructure @ i2b)
* Hardware: 6 SATA drive bays (2 columns of 3, drive 0 top left, drive 1 just below) all hot-swap. Single power supply.
* Drives: 4.5 TB (6 x 1TB) RAID5 array running on a 3ware 9650SE-8LPML (8-port) card

== Services provided ==
* backup via rsync
* mysql - traffic data
* nfs server - for backups
* snmp client - for big brother
* bigbrother client

== Usage and Notes ==
* all data is stored under /data
* virtually all jc infrastructure, and all VPS machines are setup to mount to backup1 via nfs (mountpoint: <tt>/backup1</tt>), and they all have their ssh keys setup to allow passwordless rsync's
* each virt or jail backs up each evening to backup1. Each server has it's own directory (named for the server). Under those directories are 7 daily snapshots (0-6)
* at the time of writing, the mysql server running here is replicating from (slave to) the mysql instance on bwdb. Requests for bandwidth data usage for customers (coming from management, account manager, and accounting scripts running on mail) all direct towards the database "traffic" running on this server.
* cancelled customer systems are compressed and stored under <tt>/data/deprecated</tt>
* archived bwdb2 flow files are stored under <tt>/data/bwdb2</tt>
* critical files from backup2 are stored under <tt>/data/backup2</tt>

== Cronjobs ==
<pre>
00 5 * * * /usr/local/sbin/backupwatch.pl 2>&1 > /dev/null
35 5 * * * /usr/local/sbin/usage_check; /usr/local/sbin/snapshot_archive; /usr/local/sbin/snapshot_rotate /data/backuplog.log</pre>
this runs daily the scripts to report on how much disk space each customer system occupies and how long their backups took. Then it rotates backups for each system, removing the oldest backup. It will email support@johncompanies.com at it’s conclusion. This email can be deleted, however note when it begins to take significantly longer to complete, ie runs past 2200 pm – this usually indicates a problem on the backup server.

<pre>10,25,40,55 * * * * /usr/local/sbin/processsql.pl
</pre>
this processes prepared sql command files sent from/by bwdb2 (@ i2b) and imports them into the traffic database.
<pre>0 0 * * * /usr/local/sbin/3wraidchk
</pre>
checks the health of the RAID array

== Regular maintenance ==
*[[Routine_Maintenance#Free_up_space_on_backup1|Remove old backups]]
*[[Routine_Maintenance#3ware|Check on auto-verify]]

== build ==

<pre>Setup raid5 with a boot vol of 12G 5.45tb
12G boot
4664 GB

Install ubuntu 8.04

Swap 4G

Don’t format data drive

http://www.unixgods.org/~tilo/linux_larger_2TB.html

parted /dev/sdb
print
mklabel gpt
print

#Disk /dev/sdb: 4987GB
#Sector size (logical/physical): 512B/512B
#Partition Table: gpt

#Number Start End Size File system Name Flags

mkpart primary ext3 0 4987GB
print

#Disk /dev/sdb: 5987GB
#Sector size (logical/physical): 512B/512B
#Partition Table: gpt

#Number Start End Size File system Name Flags
# 1 17.4kB 4987GB 4987GB primary

quit

mkfs.ext3 /dev/sdb1
#mke2fs 1.40.8 (13-Mar-2008)
#Filesystem label=
#OS type: Linux
#Block size=4096 (log=2)
#Fragment size=4096 (log=2)
#304390144 inodes, 1217544183 blocks
#60877209 blocks (5.00%) reserved for the super user
#First data block=0
#Maximum filesystem blocks=0
#37157 block groups
#32768 blocks per group, 32768 fragments per group
#8192 inodes per group
#Superblock backups stored on blocks:
# 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
# 4096000, 7962624, 11239424, 20480000, 23887872, 71663616, 78675968,
# 102400000, 214990848, 512000000, 550731776, 644972544
#
#Writing inode tables: 967/37157

mkdir /data

#root@backup1:~# df -h
#Filesystem Size Used Avail Use% Mounted on
#/dev/sda2 8.3G 540M 7.3G 7% /
#varrun 1013M 40K 1013M 1% /var/run
#varlock 1013M 0 1013M 0% /var/lock
#udev 1013M 56K 1013M 1% /dev
#devshm 1013M 0 1013M 0% /dev/shm
#/dev/sdb1 4.5T 192M 4.3T 1% /data

apt-get update
apt-get upgrade
apt-get install snmp snmpd ntp nfs-kernel-server

echo "\"\e[5~\": history-search-backward" >> ~/.inputrc
echo "\"\e[6~\": history-search-forward" >> ~/.inputrc

vi /etc/ntp.conf
server 10.1.4.5

scp root@10.1.4.3:/root/.ssh/authorized_keys /root/.ssh/
cd /root/
ssh-keygen -t dsa
echo "10.1.4.3 backup2" >> /etc/hosts

cat .ssh/id_dsa.pub | ssh backup2 'cat - >> /root/.ssh/authorized_keys'

ssh backup2

vi /root/.bashrc
export PS1="[\u@\h \w]# "
alias h='history'
alias vi='vim'
alias j='jobs'
export PS1="[\u@\h \w]# "
alias dr='screen -dr'
export EDITOR=vim
export GREP_OPTIONS='--color=auto'
export HISTFILESIZE=1000

source /root/.bashrc

echo "# ttyS0 - getty
#
# This service maintains a getty on ttyS0 from the point the system is
# started until it is shut down again.

start on runlevel 2
start on runlevel 3
start on runlevel 4
start on runlevel 5

stop on runlevel 0
stop on runlevel 1
stop on runlevel 6

respawn
exec /sbin/getty 38400 ttyS0" > /etc/event.d/ttyS0

vi /boot/grub/menu.lst

serial --unit=0 --speed=38400 --word=8 --parity=no --stop=1
terminal --timeout=15 serial console

append to kernel lines:
console=tty0 console=ttyS0,38400n8

show menu:
#hiddenmenu

echo 'rocommunity jcread 10.1.4.5
rocommunity jcread 10.1.4.3
agentaddress 10.1.4.8:161' > /etc/snmp/snmpd.conf

# to see which iface it is, on backup2:

snmpwalk -v 1 -c jcread 10.1.4.8 interface

echo "bb:x:1984:1984:Big Brother:/home/bb:/bin/bash" >> /etc/passwd

echo "bb:x:1984:" >> /etc/group

pwconv

mkdir /home/bb
chown bb.bb /home/bb

cd ~bb
scp backup2:/mnt/data4/build/bb/bb-linux.tar .

tar xf bb-linux.tar

cd /home/bb/bbc1.9e-btf/etc

echo "10.1.4.5 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh" > /home/bb/bbc1.9e-btf/etc/bb-hosts
echo "10.1.4.8 backup1.johncompanies.com # ssh" >> /home/bb/bbc1.9e-btf/etc/bb-hosts

echo "/:90:95
/var:90:95
/data:85:99" > /home/bb/bbc1.9e-btf/etc/bb-dftab

vi /home/bb/bbc1.9e-btf/bin/bb-disk.sh
(remove all | SORT xxxx)

chmod +r /var/log/messages

./bbchkcfg.sh
#(y to questions)
./bbchkhosts.sh
#(ignore ssh errors)
cd ../..
chown -R bb .
su bb
cd
cd bbc1.9e-btf/src

#make; make install
cd ..
./runbb.sh start
more BBOUT
(look for errors)
exit

vi /etc/rc.local
su - bb -c "cd /home/bb/bbc1.9e-btf; ./runbb.sh start"

echo '/data 10.1.4.0/24(rw, no_root_squash,async,no_subtree_check)' >> /etc/exports

/etc/init.d/nfs-kernel-server restart

echo 'chmod o+r /var/log/messages' >> /etc/cron.weekly/sysklogd

echo '10.1.4.8 backup1' >> /etc/hosts
echo '/dev/sdb1 /data ext3 rw,noatime 0 0' >> /etc/fstab

to install digi drivers:

wget http://ftp1.digi.com/support/driver/40002086_n.tgz
apt-get install linux-image-2.6.24-19-server
apt-get install linux-source-2.6.24 (not needed?)
apt-get install linux-headers-2.6.24-19-server
apt-get install make
apt-get install gcc
apt-get install g++
apt-get install libncurses5-dev
apt-get install expect
apt-get install libdbi-perl libdate-calc-perl libdbd-mysql-perl

cd /usr/src; ln -s linux-headers-2.6.24-19-server linux
./configure
make all
make install
make postinstall

/usr/bin/dgrp_cfg_node -v -v init el 65.116.11.2 8

apt-get install mysql

mkdir /data/mysql
chown mysql:mysql /data/mysql
/etc/init.d/mysql stop
mv /var/lib/mysql/* /data/mysql/
mv /data/mysql/ib_* /var/lib/mysql/
vi /etc/mysql/my.cnf
(change datadir to /data/mysql)
vi /etc/apparmor.d/usr.sbin.mysqld
add:
/data/mysql/ r,
/data/mysql/** rwk,
Comment out:
# /var/lib/mysql/ r,
# /var/lib/mysql/** rwk,

/etc/init.d/apparmor restart
/etc/init.d/mysql start

tw_cli /c0/u0 set ignoreECC=on
tw_cli /c0/u0 set storsave=balance
tw_cli /c0/u0 set cache=on

0 0 * * * /usr/local/sbin/3wraidchk
</pre>

= backup2 =

== Summary ==

This machine is used for archiving data and is a backup server for colo customers. It was the former primary backup location for all VPS-based customers before backup1 was installed. Only dedicated customers directly accesses this server to perform their backups. NOTE: power button is broken, so the reset button (paper clip) was rewired to be the power button.

* Location: castle, cab 3-7
* OS: FreeBSD 6.1 x86
* Networking: Priv IP: 10.1.4.3, Pub IP: 69.55.230.10 (firewalled from all but JC infrastructure @ i2b)
* Hardware: 16 IDE drive bays (4 columns of 4, drive 0-0 top left, drive 0-1 just to the right TODO) all hot-swap. Triple power supply.
* Drives:
**3ware 7500-8:
***200 GB JBOD (1 x 200G) labeled 0-0
***500 GB RAID5 (3 x 250G) 0-1 thru 0-3
***700 GB RAID5 (4 x 250G) 0-4 thru 0-7
**3ware 7500-8:
***700 GB RAID5 (4 x 250G) 1-0 thru 1-3
***700 GB RAID5 (4 x 250G) 1-4 thru 1-7

All drives MUST be western digital IDE drives. Other brands will not fit.

In case of an outage, nfs will hang on all connected servers until the nfs service returns. If you can't get backup2 back online, you can get nfs running elsewhere and fake backup2's MAC's: priv: 00:0e:0c:59:c1:a6, pub: 00:07:e9:5b:c6:45

To configure:
ifconfig fxp0 link 00:90:27:f9:0a:d9

== Services provided ==
* backup via rsync and nfs
* samba
* nfs
* snmp
* bigbrother

== Usage ==
* all data is stored under 4 mount points, corresponding to the 4 large RAID5 arrays: <tt>/mnt/data1 /mnt/data2 /mnt/data3 /mnt/data4</tt>
* iso images provided for customers wanting to mount an ISO as a CDROM via the IPKVM are provided via samba on this server. Images live under <tt>/mnt/data2/iso</tt>
* this used to be our primary backup server so you will see old backups from virt and jails around- missing customer data though, just the machine's data
* this server serves as an archive for exported db data from bwdb and old flow files.
* isys backs up here
* customers are nfs-moutned under /mnt/data3/customers as file-backed md devices
* in <tt>/mnt/data4</tt> there are lots of useful things used for building our vps servers, customer servers, and management scripts:
** <tt>/bin</tt>: the master repository of scripts and custom binaries we use on jails and virts. Each night every virt and jail rsync's what's in here to update the local files. So any global updates to scripts would need to be made here (or will be overwritten with what's in here)
** <tt>/build</tt>: files we use for setting up big brother, 3ware cli and scripts for colo's, vzcp customized setup files and so on
** <tt>/vzrpms</tt>: contains the OS templates for many-to-most of the OS's we offer on vz systems

== Cronjobs ==
* backs itself up nightly to nfs-mounted backup1 (mountpoint: <tt>/backup2</tt>)

== Regular maintenance ==
*[[Routine_Maintenance#3ware|Check on health]]

= backup3 =
== Summary ==
This machine is used for archiving data, is a backup server for colo customers, runs a samba server to make available iso's to the IPKVMs, and allows us to connect to the digi serial multiplexer at i2b. Only dedicated customers directly accesses this server to perform their backups.

* Location: i2b, cab 6
* OS: Ubuntu 10.04.1 server amd64
* Networking: Priv IP: 10.1.2.3, Pub IPs: 69.55.229.4 AND 69.55.231.2
* Hardware: 16 drive SATA bays (4 columns of 4, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: 5 TB (6 x 1TB) RAID5 array running on an Areca Technology Corp. ARC-1160 16-Port

== Services provided ==
* backup via rsync and nfs
* samba
* nfs
* digi realport
* snmp
* bigbrother

== Usage ==
* all data is stored under /data
* iso images provided for customers wanting to mount an ISO as a CDROM via the IPKVM are provided via samba on this server. Images live under <tt>/data/iso</tt>
* this server serves as an archive for exported db data from bwdb and old flow files.
* inftrastructure machines at i2b back up here
* customers are nfs-moutned under /data/customers as file-backed loopback devices

== management scripts ==
* mkbackups

== Cronjobs ==
0 0 * * * /usr/local/sbin/arecaraidchk
RAID checks

35 4 * * * /usr/local/sbin/snapshot_archive
Rotate daily snapshots for infrastructure machine backups

== Regular maintenance ==
*[[Routine_Maintenance#Areca|Check on RAID health]]

== Build ==

=== BIOS Config ===
disable quiet boot

set to last state after power loss

set date/time to GMT

enable serial console output (baud rate 115200)

=== Install OS ===
<pre>Ubuntu 10.04.1 amd64 (couldn't get 12.04 to load cause the H/W was incompat)
10G / ext3
2G swap
~ /data ext4

Install packages:
openssh
samba</pre>

=== DNS and private IP ===

echo "nameserver 69.55.225.225" >> /etc/resolv.conf

Add a 2nd IP to eth0 and setup priv net
<pre>vi /etc/network/interfaces

auto eth0
iface eth0 inet static
address 69.55.229.4
netmask 255.255.255.0
network 69.55.229.0
broadcast 69.55.229.255
gateway 69.55.229.1
# dns-* options are implemented by the resolvconf package, if installed
dns-nameservers 69.55.229.3 66.181.0.2
dns-search johncompanies.com

auto eth0:1
iface eth0:1 inet static
address 69.55.231.2
netmask 255.255.255.0
network 69.55.231.0
broadcast 69.55.231.255

auto eth1
iface eth1 inet static
address 10.1.2.3
netmask 255.255.255.0
network 10.1.2.0
broadcast 10.1.2.255

</pre>

=== Install packages ===
<pre>apt-get update
apt-get upgrade
apt-get install gcc
apt-get install libssl-dev
apt-get install libncurses5-dev
apt-get install cu
apt-get install unzip
apt-get install snmp snmpd ntp nfs-kernel-server</pre>

=== tweak grub, enable serial ===

<pre>vi /etc/default/grub
#GRUB_HIDDEN_TIMEOUT=0
GRUB_CMDLINE_LINUX_DEFAULT="max_loop=64"
GRUB_CMDLINE_LINUX="console=tty0 console=ttyS0"
update-grub</pre>

<pre>echo "start on stopped rc RUNLEVEL=[2345]
stop on runlevel [!2345]
respawn
exec /sbin/getty -L ttyS0 38400 vt102" > /etc/init/ttyS0.conf</pre>

=== install realport (digi) driver ===

give the digi an ip with DgIpServ.exe

<pre>cd /usr/src/
wget ftp://ftp1.digi.com/support/beta/linux/dgrp/dgrp-1.9.tgz
tar xzf dgrp-1.9.tgz
cd dgrp-1.9/
./configure
make
make install
make postinstall
update-rc.d dgrp_daemon defaults</pre>

configure ports:
dgrp_cfg_node init el 10.1.2.10 16

try connecting with:
cu -l /dev/ttyel00 -s 38400

=== shell, ntp, ssh key, hosts ===

Shell autocompletion search:
<pre>echo "\"\e[5~\": history-search-backward" >> ~/.inputrc
echo "\"\e[6~\": history-search-forward" >> ~/.inputrc</pre>

Setup ntp:
<pre>vi /etc/ntp.conf
server 10.1.2.1
server ntp.ubuntu.com</pre>

Generate ssh keys:
<pre>cd /root/
ssh-keygen -t dsa</pre>
Defaults, no password

Setup hosts:
<pre>echo "69.55.230.10 backup2" >> /etc/hosts
echo "69.55.230.11 backup1" >> /etc/hosts
echo "10.1.2.4 bwdb2" >> /etc/hosts
echo "10.1.2.3 backup3" >> /etc/hosts</pre>

Copy keys to servers where we need passwordless login:
<pre>cat .ssh/id_dsa.pub | ssh backup2 'cat - >> /root/.ssh/authorized_keys'
cat .ssh/id_dsa.pub | ssh backup1 'cat - >> /root/.ssh/authorized_keys'</pre>

Setup shell:
<pre>vi /root/.bashrc
(add to bottom)
alias h='history'
alias vi='vim'
alias j='jobs'
export PS1="[\u@\h \w]# "
alias dr='screen -dr'
export EDITOR=vim
export GREP_OPTIONS='--color=auto'
export HISTFILESIZE=1000

alias tip-switch-p20='cu -l ttyel00 -s 9600'
alias tip-switch-p21='cu -l ttyel15 -s 9600'
alias tip-switch-p22='cu -l ttyel14 -s 9600'
alias tip-switch-p23='cu -l ttyel05 -s 9600'
alias tip-switch-p24='cu -l ttyel06 -s 9600'
alias tip-switch-p25='cu -l ttyel09 -s 9600'
alias tip-switch-p26='cu -l ttyel07 -s 9600'
alias tip-switch-p27='cu -l ttyel08 -s 9600'
alias tip-firewall2='cu -l ttyel01 -s 115200'
alias tip-nat2='cu -l /dev/ttyel02 -s 115200'
alias tip-backup3='cu -l ttyel04 -s 38400'
alias tip-bwdb2='cu -l ttyel03 -s 115200'
alias tip-backup4='cu -l ttyel13 -s 115200'
alias tip-jail3='cu -l ttyel11 -s 115200'

Load new shell:
source /root/.bashrc

Setup snmpd (this is only valid for a server at castle):
echo 'rocommunity jcread 10.1.4.5
rocommunity jcread 10.1.4.3
agentaddress 10.1.4.8:161' > /etc/snmp/snmpd.conf

to see which iface it is, on backup2:

snmpwalk -v 1 -c jcread 10.1.4.8 interface

=== nfs ===

Allow mounts from private net:
echo '/data 10.1.2.0/24(rw,no_root_squash,async,no_subtree_check)' >> /etc/exports

Restart nfsd:
/etc/init.d/nfs-kernel-server restart

=== bb ===

Add user, group:
echo "bb:x:1984:1984:Big Brother:/home/bb:/bin/bash" >> /etc/passwd
echo "bb:x:1984:" >> /etc/group
pwconv

Create home:
mkdir /home/bb
chown bb.bb /home/bb
cd ~bb

Copy over and install files:
<pre>scp backup2:/mnt/data4/build/bb/bb-linux.tar .
tar xf bb-linux.tar
cd /home/bb/bbc1.9e-btf/etc</pre>

Configure main bb server:
echo "69.55.230.2 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh" > /home/bb/bbc1.9e-btf/etc/bb-hosts
echo "10.1.2.3 backup3.johncompanies.com # ssh" >> /home/bb/bbc1.9e-btf/etc/bb-hosts

Configure low disk alerts:
<pre>echo "/:90:95
/var:90:95
/data:85:99" > /home/bb/bbc1.9e-btf/etc/bb-dftab</pre>

vi /home/bb/bbc1.9e-btf/bin/bb-disk.sh
(remove all | SORT xxxx since SORT is broken)

chmod +r /var/log/messages

./bbchkcfg.sh
(y to questions)
./bbchkhosts.sh
(ignore ssh errors)
<pre>cd ../..
chown -R bb .
su bb
cd
cd bbc1.9e-btf/src</pre>

<pre>make; make install
cd ..
./runbb.sh start
more BBOUT</pre>
(look for errors)
exit

<pre>vi /etc/rc.local
su - bb -c "cd /home/bb/bbc1.9e-btf; ./runbb.sh start"
</pre>
(before the exit 0)

echo 'chmod o+r /var/log/messages' >> /etc/cron.weekly/sysklogd

Add f/w rule:
ipfw add 00096 allow ip from { 69.55.229.4 or 69.55.229.3 } to 69.55.230.2 1984

<pre>vi ~bb/bbc1.9e-btf/etc/bbdef-client.sh
DFWARN=199
DFPANIC=199</pre>

=== raid check ===

==== 3ware ====
<pre>
scp backup1:/usr/local/sbin/tw_cli /usr/local/sbin/tw_cli
scp backup1:/usr/local/sbin/checkraid.sh /usr/local/sbin/checkraid.sh
scp backup1:/usr/local/sbin/3wraidchk /usr/local/sbin/3wraidchk
vi /usr/local/sbin/checkraid.sh
:%s/c0/c2/g

crontab -e
0 0 * * * /usr/local/sbin/3wraidchk</pre>

==== areca ====
<pre>
cd /tmp
wget http://www.areca.us/support/s_linux/cli/linuxcli_V1.10.0_120815.zip
unzip linuxcli_V1.10.0_120815.zip
cp linuxcli_V1.10.0_120815/x86_64/cli64 /usr/local/sbin/
chmod 0700 /usr/local/sbin/cli64
cli64 rsf info
</pre>

<pre>scp backup2:/data4/bin/arecaraidchk /usr/local/sbin
scp backup1:/usr/local/sbin/Sendmail.pm /usr/local/sbin

crontab -e
0 0 * * * /usr/local/sbin/arecaraidchk
</pre>

<pre>cat > /root/verify.sh
cli64 vsf info
cli64 rsf info
cli64 disk info
cli64 event info
echo press enter when ready to run verify ; read x

cli64 vsf check vol=1
</pre>

=== misc binaries ===

scp backup1:/usr/local/sbin/snapshot_archive /usr/local/sbin/snapshot_archive
vi /usr/local/sbin/snapshot_archive
(remove entries)

crontab -e
35 4 * * * /usr/local/sbin/snapshot_archive

scp backup1:/usr/local/sbin/pagedave /usr/local/sbin/pagedave
scp backup1:/usr/local/sbin/taskdone /usr/local/sbin/taskdone

Since installing /bin/mail requires all sorts of packages (lame) we write a simple one here...which can only email johncompanies.com addr's unless you add relaying for this host:

<pre>
cat > /bin/mail
#!/usr/bin/perl
use strict;
use warnings;

use lib '/usr/local/sbin';
use Sendmail qw(sendmail);

my $sub = $ARGV[1];
my $to = $ARGV[2];

my %mail = (
To => $to,
From => $to,
Subject => $sub,
Message => '',
smtp => 'mail.johncompanies.com'
);
sendmail(%mail) || print "Error: $Sendmail::error";

</pre>

chmod 0700 /bin/mail

=== mkbackup ===

mkdir /data/customers

<pre>cat > /usr/local/sbin/mkbackup
#!/bin/sh

if test $1; then
cid=$1
else
echo "ERROR: Usage: mkbackup cid GB ip Terminating."
exit
fi

if test $2; then
gb=$2
else
echo "ERROR: Usage: mkbackup cid GB ip Terminating."
exit
fi

if test $3; then
ip=$3
else
echo "ERROR: Usage: mkbackup cid GB ip Terminating."
exit
fi

if test -e /data/customers/${cid}-file; then
echo "ERROR: /data/customers/${cid}-file exists"
exit
else
echo "touch /data/customers/${cid}-file"
touch /data/customers/${cid}-file
count=`echo $gb|awk '{print $1*1000}'`
echo "dd if=/dev/zero of=/data/customers/${cid}-file bs=1024K count=$count"
dd if=/dev/zero of=/data/customers/${cid}-file bs=1024K count=$count
echo "/sbin/mkfs -t ext3 -F -j -q /data/customers/${cid}-file"
/sbin/mkfs -t ext3 -F -j -q /data/customers/${cid}-file
fi

if test -e /data/customers/$cid; then
echo "ERROR: /data/customers/$cid exists"
exit
else
echo "mkdir /data/customers/${cid}"
mkdir /data/customers/${cid}
echo "mount -o loop /data/customers/${cid}-file /data/customers/$cid"
mount -o loop /data/customers/${cid}-file /data/customers/$cid
df -h /data/customers/$cid

echo "fsck -y /data/customers/${cid}-file" >> /etc/nfs_backup_mounts.sh
echo "mount -o loop /data/customers/${cid}-file /data/customers/$cid" >> /etc/nfs_backup_mounts.sh
echo "" >> /etc/nfs_backup_mounts.sh

echo "/data/customers/$cid $ip/32(rw,no_root_squash,async,no_subtree_check)" >> /etc/exports
/etc/init.d/nfs-kernel-server restart
tail /var/log/messages
fi</pre>

chmod 0700 /usr/local/sbin/mkbackup

vi /etc/rc.local
add:
/etc/nfs_backup_mounts.sh

=== samba ===

apt-get install samba

vi /etc/samba/smb.conf

; comment out any mounts, add:

<pre>[data]
read only = yes
locking = no
path = /data/iso
guest ok = yes</pre>

/etc/init.d/smbd restart

mkdir /data/iso

Bring over some stuff from backup2

<pre>cd /data/iso
scp backup2:/d2/iso/3wfirmware.iso .
scp backup2:/d2/iso/MD5SUMS .
scp backup2:/d2/iso/bootimg.iso .
scp backup2:/d2/iso/systemrescuecd-x86-0.2.19.iso .
scp backup2:/d2/iso/win98bootcd.iso .
scp backup2:/d2/iso/acronis_bootdisk.iso .
scp backup2:/d2/iso/memtest86-3.2.iso .</pre>

=== Moving from one server to another ===

Here are the steps you would take to move settings and data from one server to a new backup server:

* rsync over all /data/customers (we do this cause if we didn't use *-file it would copy over the files AND the data in the mountpoint)
rsync -av --progress --ignore-times *-file root@10.1.2.33:/data/customers/
after umounting all the customers, copy over the (empty) directories separately:
for f in `find . -type d`; do rsync -av $f root@69.55.229.25:/data/customers; done

* copy mount script
[root@backup3 /data/customers]# scp /etc/nfs_backup_mounts.sh root@69.55.229.25:/etc/nfs_backup_mounts.sh

* copy rc.local
[root@backup3 /data/customers]# scp /etc/rc.local root@69.55.229.25:/etc/rc.local

* copy /etc/exports
[root@backup3 /data/customers]# scp /etc/exports root@69.55.229.25:/etc/exports

* edit /etc/hostname on both machines (set current to oldbackup3)

* edit /etc/network/interfaces (swap IPs).

* stop mounts from mounting on old and new servers so it doesnt start with reboot right away:
chmod 000 /etc/nfs_backup_mounts.sh

* reboot both servers @ same time

* check everything out

* run /etc/nfs_backup_mounts.sh on new server

* if switch port changed update mrtg to reflect correct port pub nic is on (on p20):
vi /usr/local/www/mgmt/mrtg/mrtg1.cfg

= console =

== Summary ==
This box's only purpose is to serve as a means to connect to the digi serial multiplexer boxes at castle. Connect to it using the blue (cisco) ribbon cable with the beige RJ-45 to serial connector, 9600 8N1.

* Location: castle, cab 3-8
* OS: SunOS 5.8 (solaris)
* Networking: Priv IP: 10.1.4.4
* Hardware: Sun Netra

To connect to consoles, ssh in as user 'console' and use the <tt>tip</tt> command to connect to devices listed in <tt>/etc/remote</tt>

i.e.
tip switch-p1
tip jail1

== Configuring digi/ports ==

=== /etc/remote ===
This is where the configuration/mapping for ports and custom names which we use along with the tip command to connect to various ports on the digi switches.

We have 2 digi's at castle we connect to:

#3-7 10.1.4.10
virt15:dv=/dev/dty/CO001s:br#38400:el=^C^S^Q^U^D:ie=%$:oe=^D:
virt13:dv=/dev/dty/CO002s:br#115200:el=^C^S^Q^U^D:ie=%$:oe=^D:hf:

and

#3-6 10.1.4.11
jail4:dv=/dev/dty/CP001s:br#9600:el=^C^S^Q^U^D:ie=%$:oe=^D:hf:
jail16:dv=/dev/dty/CP002s:br#9600:el=^C^S^Q^U^D:ie=%$:oe=^D:hf:

The only things you need to edit are the first part (i.e. <tt>jail4</tt>) and the speed (i.e. <tt>9600</tt>). You can decipher which port on the digi each line corresponds to by the <tt>CP001s or CO001s</tt> (port 1 on digi1 and digi2), <tt>CP002s or CO002s</tt> (port 2 on digi1 and digi2)

=== drpadmin ===
The tool you use to configure a device to a digi box is drpadmin:

<pre>bash-2.03$ su
Password:
# drpadmin

Please select an option (a)dd (d)elete (s)how (r)eset (q)uit : s
0 10.1.4.10 32 CO 771 never 1027
1 10.1.4.11 32 CP 771 never 1027
2 65.116.11.2 8 el 771 never 1027

Please select an option (a)dd (d)elete (s)how (r)eset (q)uit :</pre>

Use those commands above to modify the devices available.

== Switching IP/hostname ==

Edit:
/etc/defaultrouter
/etc/hosts
/etc/hostname.hme0
/etc/nodename
Maybe needed to run: # ifconfig hme0 10.1.4.4 up

= devweb =

We do web development on devweb.johncompanies.com

Currently this is a jail running on jail17 / 69.55.230.8

If the jail is restarted, you will need to manually restart the web service with:
httpsdctl restart

All website development work should be done here first. It works exactly like and is setup like our [[Management_System_/_Public_Website_/_Signup_/_Account_Manager|main site]].

= firewall2 =

== Summary ==

This machine is the primary firewall for the entire network at i2b. firewall3 is a hot standby replacement for
firewall2. Both firewall2 and firewall3 should not be connected at the same time since they use the same internal
and external IP addresses.

* Location: i2b, cab 6
* OS: FreeBSD 6.4 x86
* Networking: Priv IP: 10.1.2.2, Pub IPs: 66.181.18.3 (external), 69.55.229.1 & 69.55.231.1 (internal). It has 3 network connections (2 onboard, 1 PCI) connecting to the external, internal and private networks. TODO: describe NIC location/orientation

* Hardware: 2 SCSI SCA drive bays (2 columns of 3, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: 73 GB (2 x 73GB) RAID1 array running on an LSI MegaRAID SCSI 320 PCI RAID card.

== Services Provided ==
* firewall (ipfw)
* bigbrother for customer machines

== Firewall Rule Configuration ==

See [[FreeBSD_Reference#Firewall_Rule_Configuration|Firewall Rule Configuration]] for more discussion on how to actually manipulate firewall rules.

== Disaster Recovery ==

TODO: need backup f/w and instructions on how to move cables.

Here's what you need to put in /etc/rc.conf to get a firewall going (as far as routes and IPs)

TODO

Here's the config on the live firewall:

<pre>kern_securelevel_enable="NO"
portmap_enable="NO"
sendmail_enable="NO"
usbd_enable="YES"
gateway_enable="YES"

xntpd_enable="YES"
nfs_client_enable="YES"
nfs_reserved_port_only="YES"
inetd_enable="YES"
inetd_flags="-wW -a 10.1.2.2"

fsck_y_enable="YES"
background_fsck="NO"

defaultrouter="66.181.18.2"
hostname="firewall2.johncompanies.com"
ifconfig_bge0="inet 66.181.18.3 netmask 255.255.255.224"
ifconfig_bge1="inet 69.55.229.1 netmask 255.255.255.0"
ifconfig_bge1_alias0="inet 69.55.231.1 netmask 255.255.255.0"
ifconfig_bge1_alias1="inet 65.50.228.1 netmask 255.255.255.0"
ifconfig_bge1_alias2="inet 65.50.229.1 netmask 255.255.255.0"
ifconfig_bge1_alias3="inet 65.50.230.1 netmask 255.255.255.0"
ifconfig_bge1_alias4="inet 65.50.231.1 netmask 255.255.255.0"
ifconfig_bge1_alias5="inet 65.50.232.1 netmask 255.255.255.0"
ifconfig_bge1_alias6="inet 65.50.233.1 netmask 255.255.255.0"
ifconfig_bge1_alias7="inet 65.50.234.1 netmask 255.255.255.0"
ifconfig_bge1_alias8="inet 65.50.235.1 netmask 255.255.255.0"
ifconfig_fxp0="inet 10.1.2.2 netmask 255.255.255.0"
sshd_enable="YES"
usbd_enable="YES"
</pre>

== Cronjobs ==
30 3 * * * /usr/local/etc/rsync.backup
Backup to backup3

0 0 1 * * /sbin/ipfw zero
0 0 1 * * /sbin/ipfw del 3 4 5
Reset counters and remove pipe rules on the 1st of the month. Pay attention when setting up a rule as 3 4 5 (that's not a temporary traffic cap).

59 23 30 * * /sbin/ipfw show > /tmp/ipfw_count
3 0 30 * * /sbin/ipfw show > /tmp/ipfw_count
Capture counts periodically

0 3 * * * /etc/makefwrules.pl; /etc/makepiperules.pl;
This script will create <tt>/etc/firewall.sh</tt> which contains all the firewall and pipe rules in place at the time the script was run.

*/5 * * * * /usr/local/sbin/lsiraidchk
Checking the health of the RAID array

== DOS attacks ==

See [[FreeBSD_Reference#Handling_a_DoS_attack|Handling a DoS attack]] regarding how to handle a DOS attack.

== build ==

<pre>partition map:
/ 58g
swap 4g
/var 512m
/tmp 512m
/usr 5.5g

4. edit /etc/make.conf
echo "WITHOUT_X11=yes \
KERNCONF=firewall2 \
BOOT_COMCONSOLE_SPEED=115200" >> /etc/make.conf

5. add settings to /boot/loader.conf and /boot.config

echo "-Dh" >> /boot.config

echo 'console="comconsole,vidconsole" \
boot_multicons="YES" \
boot_serial="YES" \
comconsole_speed="115200"' >> /boot/loader.conf

6. turn off all ttyv's except 0 and 1 in /etc/ttys
also turn on ttyd0, change type to vt100:
vi /etc/ttys

ttyv2 "/usr/libexec/getty Pc" cons25 off secure
ttyv3 "/usr/libexec/getty Pc" cons25 off secure
ttyv4 "/usr/libexec/getty Pc" cons25 off secure
ttyv5 "/usr/libexec/getty Pc" cons25 off secure
ttyv6 "/usr/libexec/getty Pc" cons25 off secure
ttyv7 "/usr/libexec/getty Pc" cons25 off secure
# Serial terminals
# The 'dialup' keyword identifies dialin lines to login, fingerd etc.
ttyd0 "/usr/libexec/getty std.9600" vt100 on secure

kill -1 1

on console server:
vi /etc/remote
(rename port to jail8 depending on where and which digi plugged into)
test serial console

7. populate hosts
echo "69.55.230.10 backup2" >> /etc/hosts
echo "69.55.230.11 backup1" >> /etc/hosts
echo "10.1.2.3 backup3" >> /etc/hosts

8. put key in authorized_keys on backup3
cd
ssh-keygen -t dsa -b 1024
(default location, leave password blank)

Punch a hole in firewall1 to allow traffic to backup servers @ castle:

ipfw add 99 allow ip from 66.181.18.0/27 to 69.55.230.10 22
ipfw add 95 allow ip from 66.181.18.0/27 to 69.55.230.11 22

cat /root/.ssh/id_dsa.pub | ssh backup3 'cat - >> /root/.ssh/authorized_keys'
cat /root/.ssh/id_dsa.pub | ssh backup2 'cat - >> /root/.ssh/authorized_keys'
cat /root/.ssh/id_dsa.pub | ssh backup1 'cat - >> /root/.ssh/authorized_keys'

confirm that you can ssh to backup3 and backup 2 without getting a login prompt

ssh backup3 hostname

ssh backup2 hostname

ssh backup1 hostname

10. edit root's path and login script:
vi /root/.cshrc

Change alias entries (add G):
alias la ls -aG
alias lf ls -FAG
alias ll ls -lAG
alias ls ls -AG
alias mbm mb mount
alias mbu mb umount

and alter the prompt, set the following:
set prompt = "`/bin/hostname -s` %/# "

11. install cvsup
cd /usr/ports/net/cvsup-without-gui
make install clean; rehash; mail -s 'cvs installed' dave.boodman@vtext.com < /dev/null

12. get latest sources for this release:
cd /usr/src
echo "*default host=cvsup4.freebsd.org\
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_6_4\
*default delete use-rel-suffix\
*default compress\
src-all" > sup

cvsup sup ; mail -s 'cvs sup done' dave.boodman@vtext.com < /dev/null

13. configure new kernel.

cd /usr/src/sys/i386/conf
scp backup2:/mnt/data4/build/freebsd/firewall2-6.4 ./firewall2

15. build, install kernel and world

cd /boot

mv kernel kernel.GENERIC
cd kernel.GENERIC
cd /usr/src
make buildkernel installkernel

make buildworld ; mail -s 'buildworld done' dave.boodman@vtext.com < /dev/null
(2450: 1:56min, supermicro: 59mins, 2950: 38mins)
make installworld
(2450: 3min, supermicro: 1min, 2950: :34)
mergemaster -i

17. populate /etc/rc.conf with IPs and NFS settings
vi /etc/rc.conf

kern_securelevel_enable="NO"
portmap_enable="NO"
sendmail_enable="NO"
usbd_enable="YES"
gateway_enable="YES"

xntpd_enable="YES"
nfs_client_enable="YES"
nfs_reserved_port_only="YES"
inetd_enable="YES"
inetd_flags="-wW -a 10.1.2.2"

ifconfig_bce1="inet 10.1.2.2 netmask 255.255.255.0"
fsck_y_enable="YES"
background_fsck="NO"

defaultrouter="66.181.18.2"
hostname="firewall2.johncompanies.com"
ifconfig_bge0="inet 66.181.18.3 netmask 255.255.255.224"
ifconfig_bge1="inet 69.55.229.1 netmask 255.255.255.0"
ifconfig_fxp0="inet 10.1.2.2 netmask 255.255.255.0"
sshd_enable="YES"
usbd_enable="YES"

20. reboot. Confirm new kernel is loaded

uname -a

21. update ports:
cd /usr/ports
echo "*default host=cvsup4.FreeBSD.org\
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_6_4\
*default delete use-rel-suffix\
*default compress\
ports-all tag=." > sup

cvsup sup; mail -s 'cvs sup ports done' dave.boodman@vtext.com < /dev/null

22. Install raid mgmt tool

# linux base
cd /usr/ports/devel/libtool22
make install base

cd /usr/ports/emulators/linux_base-fc4
make install clean

#linux-megamgr-5.20
cd /usr/ports/sysutils/linux-megamgr
make install clean

# megarc-1.51
cd /usr/ports/sysutils/megarc
make install clean

Test:
rehash; megarc -ldInfo -a0 -l0

23. install rsync from ports
cd /usr/ports/net/rsync
make install clean

choose default options

25. install bb client
adduser
Username: bb
Full name: bb
Uid (Leave empty for default): 1984
Login group [bb]:
Login group is bb. Invite bb into other groups? []:
Login class [default]:
Shell (sh csh tcsh nologin) [sh]:
Home directory [/home/bb]:
Use password-based authentication? [yes]:
Use an empty password? (yes/no) [no]:
Use a random password? (yes/no) [no]: yes
Lock out the account after creation? [no]:
Username : bb
Password : <random>
Full Name : bb
Uid : 1984
Class :
Groups : bb
Home : /home/bb
Shell : /bin/sh
Locked : no
OK? (yes/no): yes

cd /usr/home/bb
scp backup2:/mnt/data4/build/bb/bb-freebsd.tar .
tar xvf bb-freebsd.tar

edit /home/bb/bbc1.9e-btf/etc/bb-hosts with something like:
echo "69.55.230.2 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh \
10.1.2.1 firewall2.johncompanies.com # ssh" > /home/bb/bbc1.9e-btf/etc/bb-hosts

vi /home/bb/bbc1.9e-btf/ext/openfiles
MACHINE="firewall2,johncompanies,com" # HAS TO BE IN A,B,C FORM

cd /usr/home/bb/bbc1.9e-btf/etc
./bbchkcfg.sh
(y to questions)
./bbchkhosts.sh
(ignore ssh errors)
cd ../..
chown -R bb .
su bb
cd
cd bbc1.9e-btf/src
make; make install
cd ..

vi /usr/home/bb/bbc1.9e-btf/bin/bb-cpu.sh
$1 $TOPARGS > $BBTMP/TOP.$$
# /usr/local/jail/bin/jtop > $BBTMP/TOP.$$

./runbb.sh start
more BBOUT
(look for errors)
exit

echo 'su - bb -c "cd /home/bb/bbc1.9e-btf; ./runbb.sh start"' > /usr/local/etc/rc.d/bb.sh
chmod +x /usr/local/etc/rc.d/bb.sh

Punch a hole in the firewall to allow it to communicate with bb monitor:

ipfw add 00096 allow ip from 66.181.18.0/27 to 69.55.230.2

27. configure bb on mail:
vi /usr/home/bb/bbsrc/bb1.9e-btf/etc/bb-hosts
66.181.18.3 firewall2.johncompanies.com # ssh

su bb
cd
bbsrc/bb/runbb.sh restart ; exit

29. configure ntp
echo "server 10.1.2.1" > /etc/ntp.conf

/usr/sbin/ntpd -p /var/run/ntpd.pid
sleep 2; ntpq -p
(confirm it’s able to reach our time server)

echo '/usr/sbin/ntpd -c /etc/ntp.conf -p /var/run/ntpd.pid -f /var/db/ntpd.drift' > /usr/local/etc/rc.d/ntp.sh
chmod 0700 /usr/local/etc/rc.d/ntp.sh

30. fwd and reverse lookups on ns1c
vr johncompanies.com
(edit the PTR too)

33. setup backups
echo '#\!/bin/sh\
backupdir=/data/firewall2/current\
\
## ENTRY /etc ' > /usr/local/etc/backup.config

on backup3:
setup backup dirs:
ssh backup3 mkdir -p /data/firewall2/current

on backup3, add the system to
vi /usr/local/sbin/snapshot_archive

scp 69.55.230.2:/usr/local/etc/rsync.backup /usr/local/etc/rsync.backup
vi /usr/local/etc/rsync.backup
backup1 > backup3

crontab -e
1 0 * * * /usr/local/etc/rsync.backup

34. mkdir /root/logs

35. edit sshd_config for security
vi /etc/ssh/sshd_config
ListenAddress 66.181.18.3
ListenAddress 10.1.2.1

kill -1 `cat /var/run/sshd.pid`

35. raid chk

cat > /usr/local/sbin/lsiraidchk
#!/usr/bin/perl

my @out = split "\n", `megarc -ldInfo -a0 -Lall|grep Status:`;

foreach (@out) {
if ($_ =~ /DEGRADED/) { $date = `date`; chomp $date; `echo "$date: RAID ARRAY DEGRADED" >> /var/log/messages`; }
#print $_;
}

36. add crontab entries
crontab -e
30 3 * * * /usr/local/etc/rsync.backup
0 0 1 * * /sbin/ipfw zero
0 0 1 * * /sbin/ipfw del 3 4 5
59 23 30 * * /sbin/ipfw show > /tmp/ipfw_count
3 0 30 * * /sbin/ipfw show > /tmp/ipfw_count
0 3 * * * /etc/makefwrules.pl; /etc/makepiperules.pl;
*/5 * * * * /usr/local/sbin/lsiraidchk

#10 0 * * * rm /var/spool/clientmqueue/*

scp /etc/makefwrules.pl user@64.163.14.48:~
scp /etc/makepiperules.pl user@64.163.14.48:~
mv /home/user/makefwrules.pl /etc
mv /home/user/makepiperules.pl /etc
touch /etc/firewall.sh
mkdir /etc/oldrules/

other binaries

scp /usr/local/bin/rulemaker user@64.163.14.48:~
mv ~user/rulemaker /usr/local/sbin
scp ~user/Sendmail.pm user@64.163.14.48:~
scp ~user/doswatch.pl user@64.163.14.48:~

Setup basic ruleset

ipfw add 00009 count udp from any to any
ipfw add 00010 allow tcp from any to any established
ipfw add 00012 deny tcp from any to any tcpflags syn tcpoptions !mss
ipfw add 00012 deny icmp from any to any icmptypes 4,5,9,10,12,13,14,15,16,17,18
ipfw add 00012 deny tcp from any to any tcpflags syn,fin
ipfw add 00012 deny tcp from any to any tcpflags fin,psh,rst,urg
ipfw add 00012 allow icmp from any to any
ipfw add 00014 deny tcp from any to any dst-port 135
ipfw add 00150 skipto 65535 ip from any to any via em1 in

IPKVM3:
00098 allow ip from { 69.55.230.6 or 69.55.230.7 } to 69.55.230.10 dst-port 139
00098 deny ip from any to 69.55.230.10 dst-port 139
</pre>

= firewall3 =

== Summary ==

This machine is the backup firewall for the network at i2b.

* Location: i2b, cab ?
* OS: FreeBSD 9.1 amd64
* Networking: Priv IP: 10.1.2.5, Pub IPs: 66.181.18.3 (external), 69.55.229.1 & 69.55.231.1 (internal). It has 3 network connections (2 onboard, 1 PCI) connecting to the external, internal and private networks.

<pre>

The internal network NIC is the left one on the motherboard (69.55.229.1/24, ...).
The external network NIC is the right one on the motherboard (66.181.18.3/28).
The PCI ethernet card is connected to our private network (10.1.2.5/24).

</pre>

* Hardware: 2 SCSI SCA drive bays (2 columns of 3, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: 160 GB (2 x 160GB) RAID1 array running on an LSI MegaRAID SCSI 320 PCI RAID card.

== Services Provided ==
* firewall (ipfw)
* bigbrother

== Firewall Rule Configuration ==

See [[FreeBSD_Reference#Firewall_Rule_Configuration|Firewall Rule Configuration]] for more discussion on how to actually manipulate firewall rules.

== Disaster Recovery ==

'''To put the backup firewall3 into service:'''

<pre>
Move the internal cable (to our networks) from firewall2 to em1 which is the left most ethernet port (69.55.229.1).
Move the external cable (to outside world) from firewall2 to em0 which is the port to the right on the motherboard (66.181.18.3).
The PCI ethernet port (fxp0) should already be connected to private network (10.1.2.5).
</pre>

Here's what you need to put in /etc/rc.conf to get a firewall going (as far as routes and IPs)

<pre>
kern_securelevel_enable="NO"
portmap_enable="NO"
sendmail_enable="NO"
usbd_enable="YES"
gateway_enable="YES"

xntpd_enable="YES"
nfs_client_enable="YES"
nfs_reserved_port_only="YES"
inetd_enable="YES"
inetd_flags="-wW -a 10.1.2.5"

fsck_y_enable="YES"
background_fsck="NO"

defaultrouter="66.181.18.2"
hostname="firewall3.johncompanies.com"
ifconfig_em0="inet 66.181.18.3 netmask 255.255.255.224"

ifconfig_em1="inet 69.55.229.1 netmask 255.255.255.0"
ifconfig_em1_alias0="inet 69.55.231.1 netmask 255.255.255.0"

# ifconfig_em1_alias1="inet 65.50.228.1 netmask 255.255.255.0"
# ifconfig_em1_alias2="inet 65.50.229.1 netmask 255.255.255.0"
# ifconfig_em1_alias3="inet 65.50.230.1 netmask 255.255.255.0"
# ifconfig_em1_alias4="inet 65.50.231.1 netmask 255.255.255.0"
# ifconfig_em1_alias5="inet 65.50.232.1 netmask 255.255.255.0"
# ifconfig_em1_alias6="inet 65.50.233.1 netmask 255.255.255.0"
# ifconfig_em1_alias7="inet 65.50.234.1 netmask 255.255.255.0"
# ifconfig_em1_alias8="inet 65.50.235.1 netmask 255.255.255.0"

ifconfig_fxp0="inet 10.1.2.5 netmask 255.255.255.0"

sshd_enable="YES"
usbd_enable="YES"
</pre>

== Cronjobs ==
30 3 * * * /usr/local/etc/rsync.backup
Backup to backup3

0 0 1 * * /sbin/ipfw zero
0 0 1 * * /sbin/ipfw del 3 4 5
Reset counters and remove pipe rules on the 1st of the month. Pay attention when setting up a rule as 3 4 5 (that's not a temporary traffic cap).

59 23 30 * * /sbin/ipfw show > /tmp/ipfw_count
3 0 30 * * /sbin/ipfw show > /tmp/ipfw_count
Capture counts periodically

0 3 * * * /etc/makefwrules.pl; /etc/makepiperules.pl;
This script will create <tt>/etc/firewall.sh</tt> which contains all the firewall and pipe rules in place at the time the script was run.

*/5 * * * * /usr/local/sbin/lsiraidchk
Checking the health of the RAID array

== DOS attacks ==

See [[FreeBSD_Reference#Handling_a_DoS_attack|Handling a DoS attack]] regarding how to handle a DOS attack.

== build ==

<pre>partition map:
/ 58g
swap 4g
/var 512m
/tmp 512m
/usr 5.5g

4. edit /etc/make.conf
echo "WITHOUT_X11=yes \
KERNCONF=firewall3 \
BOOT_COMCONSOLE_SPEED=115200" >> /etc/make.conf

5. add settings to /boot/loader.conf and /boot.config

echo "-Dh" >> /boot.config

echo 'console="comconsole,vidconsole" \
boot_multicons="YES" \
boot_serial="YES" \
comconsole_speed="115200"' >> /boot/loader.conf

6. turn off all ttyv's except 0 and 1 in /etc/ttys
also turn on ttyd0, change type to vt100:
vi /etc/ttys

ttyv2 "/usr/libexec/getty Pc" cons25 off secure
ttyv3 "/usr/libexec/getty Pc" cons25 off secure
ttyv4 "/usr/libexec/getty Pc" cons25 off secure
ttyv5 "/usr/libexec/getty Pc" cons25 off secure
ttyv6 "/usr/libexec/getty Pc" cons25 off secure
ttyv7 "/usr/libexec/getty Pc" cons25 off secure
# Serial terminals
# The 'dialup' keyword identifies dialin lines to login, fingerd etc.
ttyd0 "/usr/libexec/getty std.9600" vt100 on secure

kill -1 1

on console server:
vi /etc/remote
(rename port to jail8 depending on where and which digi plugged into)
test serial console

7. populate hosts
echo "69.55.230.10 backup2" >> /etc/hosts
echo "69.55.230.11 backup1" >> /etc/hosts
echo "10.1.2.3 backup3" >> /etc/hosts

8. put key in authorized_keys on backup3
cd
ssh-keygen -t dsa -b 1024
(default location, leave password blank)

Punch a hole in firewall1 to allow traffic to backup servers @ castle:

ipfw add 99 allow ip from 66.181.18.0/27 to 69.55.230.10 22
ipfw add 95 allow ip from 66.181.18.0/27 to 69.55.230.11 22

cat /root/.ssh/id_dsa.pub | ssh backup3 'cat - >> /root/.ssh/authorized_keys'
cat /root/.ssh/id_dsa.pub | ssh backup2 'cat - >> /root/.ssh/authorized_keys'
cat /root/.ssh/id_dsa.pub | ssh backup1 'cat - >> /root/.ssh/authorized_keys'

confirm that you can ssh to backup3 and backup 2 without getting a login prompt

ssh backup3 hostname

ssh backup2 hostname

ssh backup1 hostname

10. edit root's path and login script:
vi /root/.cshrc

Change alias entries (add G):
alias la ls -aG
alias lf ls -FAG
alias ll ls -lAG
alias ls ls -AG
alias mbm mb mount
alias mbu mb umount

and alter the prompt, set the following:
set prompt = "`/bin/hostname -s` %/# "

11. install cvsup
cd /usr/ports/net/cvsup-without-gui
make install clean; rehash; mail -s 'cvs installed' 8583619553@vtext.com < /dev/null

12. get latest sources for this release:
cd /usr/src
echo "*default host=cvsup4.freebsd.org\
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_9_1\
*default delete use-rel-suffix\
*default compress\
src-all" > sup

cvsup sup ; mail -s 'cvs sup done' 8583619553@vtext.com < /dev/null

13. configure new kernel.

cd /usr/src/sys/amd64/conf
scp backup2:/mnt/data4/build/freebsd/firewall3-9.1 ./firewall3

15. build, install kernel and world

cd /boot

mv kernel kernel.GENERIC
cd kernel.GENERIC
cd /usr/src
make buildkernel installkernel

make buildworld ; mail -s 'buildworld done' 8583619553@vtext.com < /dev/null
(supermicro: 2:15 mins, 2950: 38? mins)
make installworld
(2450: 3min, supermicro: 1min, 2950: :34)
mergemaster -i

17. populate /etc/rc.conf with IPs and NFS settings
vi /etc/rc.conf

kern_securelevel_enable="NO"
portmap_enable="NO"
sendmail_enable="NO"
usbd_enable="YES"
gateway_enable="YES"

xntpd_enable="YES"
nfs_client_enable="YES"
nfs_reserved_port_only="YES"

fsck_y_enable="YES"
background_fsck="NO"

hostname="firewall3.johncompanies.com"
# external network
ifconfig_em0="inet 66.181.18.3 netmask 255.255.255.224"

# internal network
ifconfig_em1="inet 69.55.229.1 netmask 255.255.255.0"
ifconfig_em1_alias0="inet 69.55.231.1 netmask 255.255.255.0"

ifconfig_em1_alias1="inet 65.50.228.1 netmask 255.255.255.0"
ifconfig_em1_alias2="inet 65.50.229.1 netmask 255.255.255.0"
ifconfig_em1_alias3="inet 65.50.230.1 netmask 255.255.255.0"
ifconfig_em1_alias4="inet 65.50.231.1 netmask 255.255.255.0"
ifconfig_em1_alias5="inet 65.50.232.1 netmask 255.255.255.0"
ifconfig_em1_alias6="inet 65.50.233.1 netmask 255.255.255.0"
ifconfig_em1_alias7="inet 65.50.234.1 netmask 255.255.255.0"
ifconfig_em1_alias8="inet 65.50.235.1 netmask 255.255.255.0"

defaultrouter="66.181.18.2"

# private network
ifconfig_fxp0="inet 10.1.2.5 netmask 255.255.255.0"

inetd_enable="YES"
inetd_flags="-wW -a 10.1.2.5"

sshd_enable="YES"
usbd_enable="YES"
ntpd_enable="YES"
# powerd_enable="YES"

20. reboot. Confirm new kernel is loaded

uname -a

21. update ports:
cd /usr/ports

echo "*default host=cvsup4.FreeBSD.org\
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_9_1\
*default delete use-rel-suffix\
*default compress\
ports-all tag=." > sup

cvsup sup; mail -s 'cvs sup ports done' 8583619553@vtext.com < /dev/null

22. Install raid mgmt tool

# linux base
cd /usr/ports/devel/libtool22
make install base

cd /usr/ports/emulators/linux_base-fc4
make install clean

scp backup2:/d4/build/3ware/tw_cli-freebsd-x86_64-9.5.0.1.tgz /usr/local/sbin
cd /usr/local/sbin
tar xzvf tw_cli-freebsd-x86_64-9.5.0.1.tgz
rm tw_cli-freebsd-x86_64-9.5.0.1.tgz

23. install rsync from ports
cd /usr/ports/net/rsync
make install clean

choose default options

25. install bb client
adduser
Username: bb
Full name: bb
Uid (Leave empty for default): 1984
Login group [bb]:
Login group is bb. Invite bb into other groups? []:
Login class [default]:
Shell (sh csh tcsh nologin) [sh]:
Home directory [/home/bb]:
Use password-based authentication? [yes]:
Use an empty password? (yes/no) [no]:
Use a random password? (yes/no) [no]: yes
Lock out the account after creation? [no]:
Username : bb
Password : <random>
Full Name : bb
Uid : 1984
Class :
Groups : bb
Home : /home/bb
Shell : /bin/sh
Locked : no
OK? (yes/no): yes

cd /usr/home/bb
scp backup2:/mnt/data4/build/bb/bb-freebsd.tar .
tar xvf bb-freebsd.tar

edit /home/bb/bbc1.9e-btf/etc/bb-hosts with something like:
echo "69.55.230.2 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh \
10.1.2.5 firewall3.johncompanies.com # ssh" > /home/bb/bbc1.9e-btf/etc/bb-hosts

vi /home/bb/bbc1.9e-btf/ext/openfiles
MACHINE="firewall3,johncompanies,com" # HAS TO BE IN A,B,C FORM

cd /usr/home/bb/bbc1.9e-btf/etc
./bbchkcfg.sh
(y to questions)
./bbchkhosts.sh
(ignore ssh errors)
cd ../..
chown -R bb .
su bb
cd
cd bbc1.9e-btf/src
make; make install
cd ..

vi /usr/home/bb/bbc1.9e-btf/bin/bb-cpu.sh
$1 $TOPARGS > $BBTMP/TOP.$$
# /usr/local/jail/bin/jtop > $BBTMP/TOP.$$

./runbb.sh start
more BBOUT
(look for errors)
exit

echo 'su - bb -c "cd /home/bb/bbc1.9e-btf; ./runbb.sh start"' > /usr/local/etc/rc.d/bb.sh
chmod +x /usr/local/etc/rc.d/bb.sh

Punch a hole in the firewall to allow it to communicate with bb monitor:

ipfw add 00096 allow ip from 66.181.18.0/27 to 69.55.230.2

27. configure bb on mail:
vi /usr/home/bb/bbsrc/bb1.9e-btf/etc/bb-hosts
66.181.18.3 firewall3.johncompanies.com # ssh

su bb
cd
bbsrc/bb/runbb.sh restart ; exit

29. configure ntp
echo "server 10.1.2.1" > /etc/ntp.conf

/usr/sbin/ntpd -p /var/run/ntpd.pid
sleep 2; ntpq -p
(confirm it’s able to reach our time server)

echo '/usr/sbin/ntpd -c /etc/ntp.conf -p /var/run/ntpd.pid -f /var/db/ntpd.drift' > /usr/local/etc/rc.d/ntp.sh
chmod 0700 /usr/local/etc/rc.d/ntp.sh

30. fwd and reverse lookups on ns1c
vr johncompanies.com
(edit the PTR too)

33. setup backups
echo '#\!/bin/sh\
backupdir=/data/firewall2/current\
\
## ENTRY /etc ' > /usr/local/etc/backup.config

on backup3:
setup backup dirs:
ssh backup3 mkdir -p /data/firewall2/current

on backup3, add the system to
vi /usr/local/sbin/snapshot_archive

scp 69.55.230.2:/usr/local/etc/rsync.backup /usr/local/etc/rsync.backup
vi /usr/local/etc/rsync.backup
backup1 > backup3

crontab -e
1 0 * * * /usr/local/etc/rsync.backup

34. mkdir /root/logs

35. edit sshd_config for security
vi /etc/ssh/sshd_config
ListenAddress 66.181.18.3
ListenAddress 10.1.2.5

kill -1 `cat /var/run/sshd.pid`

35. raid chk

cat > /usr/local/sbin/lsiraidchk
#!/usr/bin/perl

my @out = split "\n", `megarc -ldInfo -a0 -Lall|grep Status:`;

foreach (@out) {
if ($_ =~ /DEGRADED/) { $date = `date`; chomp $date; `echo "$date: RAID ARRAY DEGRADED" >> /var/log/messages`; }
#print $_;
}

36. add crontab entries
crontab -e
30 3 * * * /usr/local/etc/rsync.backup
0 0 1 * * /sbin/ipfw zero
0 0 1 * * /sbin/ipfw del 3 4 5
59 23 30 * * /sbin/ipfw show > /tmp/ipfw_count
3 0 30 * * /sbin/ipfw show > /tmp/ipfw_count
0 3 * * * /etc/makefwrules.pl; /etc/makepiperules.pl;
*/5 * * * * /usr/local/sbin/lsiraidchk

#10 0 * * * rm /var/spool/clientmqueue/*

scp /etc/makefwrules.pl user@64.163.14.48:~
scp /etc/makepiperules.pl user@64.163.14.48:~
mv /home/user/makefwrules.pl /etc
mv /home/user/makepiperules.pl /etc
touch /etc/firewall.sh
mkdir /etc/oldrules/

other binaries

scp /usr/local/bin/rulemaker user@64.163.14.48:~
mv ~user/rulemaker /usr/local/sbin
scp ~user/Sendmail.pm user@64.163.14.48:~
scp ~user/doswatch.pl user@64.163.14.48:~

Setup basic ruleset

ipfw add 00009 count udp from any to any
ipfw add 00010 allow tcp from any to any established
ipfw add 00012 deny tcp from any to any tcpflags syn tcpoptions !mss
ipfw add 00012 deny icmp from any to any icmptypes 4,5,9,10,12,13,14,15,16,17,18
ipfw add 00012 deny tcp from any to any tcpflags syn,fin
ipfw add 00012 deny tcp from any to any tcpflags fin,psh,rst,urg
ipfw add 00012 allow icmp from any to any
ipfw add 00014 deny tcp from any to any dst-port 135
ipfw add 00150 skipto 65535 ip from any to any via em1 in

IPKVM3:
00098 allow ip from { 69.55.230.6 or 69.55.230.7 } to 69.55.230.10 dst-port 139
00098 deny ip from any to 69.55.230.10 dst-port 139
</pre>

= wiki =

The wiki (mediawiki) runs on nat2 in a jail running off 69.55.229.8

The backup wiki lives on virt13 in CT 5 / 69.55.230.18

== Setup jail ==

<pre>
mkdir /mnt/data1/wiki-dir
cd /usr/src
make installworld DESTDIR=/mnt/data1/wiki-dir
cd etc
make distribution DESTDIR=/mnt/data1/wiki-dir

mount -t devfs devfs /mnt/data1/wiki-dir/dev
devfs -m /mnt/data1/wiki-dir/dev rule -s 3 applyset

cd /mnt/data1/wiki-dir

ln -sf dev/null kernel

scp jail9:/usr/local/sbin/jkill /mnt/data1/wiki-dir/sbin

jail /mnt/data1/wiki-dir wiki.johncompanies.com 69.55.229.8 /bin/sh
csh

touch /etc/fstab
echo 'network_interfaces=""\
hostname="wiki.johncompanies.com"\
kern_securelevel_enable="NO"\
sendmail_enable="YES"\
sshd_enable="YES"' > /etc/rc.conf

echo "nameserver 69.55.229.3\
nameserver 69.55.225.225" >> /etc/resolv.conf

vi /etc/crontab

(remove the adjkerntz lines )

vi /etc/periodic/security/100.chksetuid

replace: MP=`mount -t ufs | grep -v " nosuid" | awk '{ print $3 }' | sort`
with: MP='/' (use single quotes)

mkdir -p /usr/compat/linux/dev

adduser

Username: user
Full name: user
Uid (Leave empty for default):
Login group [user]:
Login group is user. Invite user into other groups? []: wheel
Login class [default]:
Shell (sh csh tcsh nologin) [sh]:
Home directory [/home/user]:
Home directory permissions (Leave empty for default):
Use password-based authentication? [yes]:
Use an empty password? (yes/no) [no]:
Use a random password? (yes/no) [no]: y
Lock out the account after creation? [no]:
Username : user
Password : <random>
Full Name : user
Uid : 1001
Class :
Groups : user
Home : /home/user
Home Mode :
Shell : /bin/sh
Locked : no
OK? (yes/no): y
adduser: INFO: Successfully added (user) to the user database.
adduser: INFO: Password for (user) is: 901gmYjO
Add another user? (yes/no): n
Goodbye!

vi /usr/home/user/.profile
TERM=vt100; export TERM

tzsetup

newaliases

rm /sbin/halt /sbin/reboot
ln /sbin/jkill /sbin/halt
ln /sbin/jkill /sbin/reboot

vi /etc/syslog.conf
#*.err;kern.warning;auth.notice;mail.crit /dev/console
*.err;kern.warning;auth.notice;mail.crit /var/log/messages

exit
exit

cd libexec
chflags noschg ld-elf32.so.1
chflags noschg ld-elf.so.1
mv ld-elf32.so.1 ld-elf32.so.1-orig
ln ld-elf.so.1 ld-elf32.so.1
chflags schg ld-elf.so.1
chflags schg ld-elf32.so.1

cp -r /usr/ports /mnt/data1/wiki-dir/usr

cat > /usr/local/etc/rc.d/wiki.sh
mount -t devfs devfs /mnt/data1/wiki-dir/dev/
devfs -m /mnt/data1/wiki-dir/dev rule -s 3 applyset
jail /mnt/data1/wiki-dir wiki.johncompanies.com 69.55.229.8 /bin/sh /etc/rc

chmod 0700 /usr/local/etc/rc.d/wiki.sh
</pre>

== mediawiki setup ==

<pre>

cd /usr/ports/net/rsync
make install clean

cd /usr/ports/distfiles/
fetch http://downloads.mysql.com/archives/mysql-5.5/mysql-5.5.4-m3.tar.gz
cd /usr/ports/databases/mysql55-server
make install clean

cd /usr/ports/distfiles/
fetch http://downloads.php.net/johannes/php-5.3.2.tar.bz2
cd /usr/ports/lang/php52
make install clean
(build apache module)

cd /usr/ports/lang/php5-extensions
make install clean

cd /usr/ports/www/apache22
make install clean

cd /usr/local/www/
fetch http://download.wikimedia.org/mediawiki/1.19/mediawiki-1.19.2.tar.gz
tar xzf mediawiki-1.19.2.tar.gz
mv mediawiki-1.19.2 wiki

vi /usr/local/etc/apache22/httpd.conf
DocumentRoot "/usr/local/www/"

Include etc/apache22/extra/vhost-wiki.conf
Listen 443

<IfModule mod_php5.c>
<FilesMatch "\.ph(p3?|tml)$">
SetHandler application/x-httpd-php
</FilesMatch>
<FilesMatch "\.phps$">
SetHandler application/x-httpd-php-source
</FilesMatch>
# To re-enable php in user directories comment the following lines
# (from <IfModule ...> to </IfModule>.) Do NOT set it to On as it
# prevents .htaccess files from disabling it.
<IfModule mod_userdir.c>
<Directory /home/*/public_html>
php_admin_value engine Off
</Directory>
</IfModule>
</IfModule>

cat > /usr/local/etc/apache22/extra/vhost-wiki.conf
<VirtualHost *:443>
ServerAdmin support@johncompanies.com

DocumentRoot /usr/local/www/wiki
# <Directory />
# Options FollowSymLinks
# AllowOverride None
# Order deny,allow
# </Directory>
<Directory /usr/local/www/wiki>
Options Indexes FollowSymLinks MultiViews
Deny from all
AllowOverride AuthConfig
Order allow,deny
DirectoryIndex index.php
#Allow from 69.55.233.195
#Allow from boody.dyndns.org
</Directory>

ScriptAlias /cgi-bin/ /usr/lib/cgi-bin/
<Directory "/usr/lib/cgi-bin">
AllowOverride None
Options +ExecCGI -MultiViews +SymLinksIfOwnerMatch
Order allow,deny
Allow from all
</Directory>

ErrorLog /var/log/httpd-error.log

# Possible values include: debug, info, notice, warn, error, crit,
# alert, emerg.
LogLevel warn

CustomLog /var/log/httpd-access.log combined

Alias /doc/ "/usr/share/doc/"
<Directory "/usr/share/doc/">
Options Indexes MultiViews FollowSymLinks
AllowOverride None
Order deny,allow
Deny from all
Allow from 127.0.0.0/255.0.0.0 ::1/128
</Directory>

SSLEngine on
SSLCertificateFile /usr/local/etc/apache22/ssl/server.crt
SSLCertificateKeyFile /usr/local/etc/apache22/ssl/server.key

</VirtualHost>

mkdir ssl
cd ssl

openssl req -days 1999 -new -x509 -nodes -out server.crt -keyout server.key
US
CA
San Diego
johncompanies.com
johncompanies.com
wiki.johncompanies.com
support@johncompanies.com

cat > /usr/local/www/wiki/.htaccess
AuthType Basic
AuthUserFile /usr/local/etc/apache22/wiki.passwd
AuthName wiki
require valid-user
satisfy any

cd /usr/local/etc/apache22
htpasswd -c wiki.passwd admin

https://69.55.229.8/index.php

use mysql (innodb)
wiki name: JCWiki
Support / (mail pass) / support@johncompanies.com

cat > /usr/local/www/wiki/LocalSettings.php

<?php
# This file was automatically generated by the MediaWiki 1.19.2
# installer. If you make manual changes, please keep track in case you
# need to recreate them later.
#
# See includes/DefaultSettings.php for all configurable settings
# and their default values, but don't forget to make changes in _this_
# file, not there.
#
# Further documentation for configuration settings may be found at:
# http://www.mediawiki.org/wiki/Manual:Configuration_settings

# Protect against web entry
if ( !defined( 'MEDIAWIKI' ) ) {
exit;
}

## Uncomment this to disable output compression
# $wgDisableOutputCompression = true;

$wgSitename = "JCWiki";

## The URL base path to the directory containing the wiki;
## defaults for all runtime URL paths are based off of this.
## For more information on customizing the URLs please see:
## http://www.mediawiki.org/wiki/Manual:Short_URL
$wgScriptPath = "";
$wgScriptExtension = ".php";

## The protocol and server name to use in fully-qualified URLs
$wgServer = "https://69.55.229.8";

## The relative URL path to the skins directory
$wgStylePath = "$wgScriptPath/skins";

## The relative URL path to the logo. Make sure you change this from the default,
## or else you'll overwrite your logo when you upgrade!
#$wgLogo = "$wgStylePath/common/images/wiki.png";
$wgLogo = "$wgStylePath/common/images/jclogo.gif";

## UPO means: this is also a user preference option

$wgEnableEmail = true;
$wgEnableUserEmail = true; # UPO

$wgEmergencyContact = "apache@69.55.229.8";
$wgPasswordSender = "apache@69.55.229.8";

$wgEnotifUserTalk = false; # UPO
$wgEnotifWatchlist = false; # UPO
$wgEmailAuthentication = true;

## Database settings
$wgDBtype = "mysql";
$wgDBserver = "localhost";
$wgDBname = "my_wiki";
$wgDBuser = "root";
$wgDBpassword = "";

# MySQL specific settings
$wgDBprefix = "";

# MySQL table options to use during installation or update
$wgDBTableOptions = "ENGINE=InnoDB, DEFAULT CHARSET=binary";

# Experimental charset support for MySQL 5.0.
$wgDBmysql5 = false;

## Shared memory settings
$wgMainCacheType = CACHE_NONE;
$wgMemCachedServers = array();

## To enable image uploads, make sure the 'images' directory
## is writable, then set this to true:
$wgEnableUploads = false;
#$wgUseImageMagick = true;
#$wgImageMagickConvertCommand = "/usr/bin/convert";

# InstantCommons allows wiki to use images from http://commons.wikimedia.org
$wgUseInstantCommons = false;

## If you use ImageMagick (or any other shell command) on a
## Linux server, this will need to be set to the name of an
## available UTF-8 locale
$wgShellLocale = "en_US.utf8";

## If you want to use image uploads under safe mode,
## create the directories images/archive, images/thumb and
## images/temp, and make them all writable. Then uncomment
## this, if it's not already uncommented:
#$wgHashedUploadDirectory = false;

## Set $wgCacheDirectory to a writable directory on the web server
## to make your wiki go slightly faster. The directory should not
## be publically accessible from the web.
#$wgCacheDirectory = "$IP/cache";

# Site language code, should be one of the list in ./languages/Names.php
$wgLanguageCode = "en";

$wgSecretKey = "abc699ef26890b49b4055430f8ebbd25e84cce21a7e53aeaec4d4313af4c9739";

# Site upgrade key. Must be set to a string (default provided) to turn on the
# web installer while LocalSettings.php is in place
$wgUpgradeKey = "3196710f4a7d7332";

## Default skin: you can change the default skin. Use the internal symbolic
## names, ie 'standard', 'nostalgia', 'cologneblue', 'monobook', 'vector':
$wgDefaultSkin = "vector";

## For attaching licensing metadata to pages, and displaying an
## appropriate copyright notice / icon. GNU Free Documentation
## License and Creative Commons licenses are supported so far.
$wgRightsPage = ""; # Set to the title of a wiki page that describes your license/copyright
$wgRightsUrl = "";
$wgRightsText = "";
$wgRightsIcon = "";

# Path to the GNU diff3 utility. Used for conflict resolution.
$wgDiff3 = "/usr/bin/diff3";

# Query string length limit for ResourceLoader. You should only set this if
# your web server has a query string length limit (then set it to that limit),
# or if you have suhosin.get.max_value_length set in php.ini (then set it to
# that value)
$wgResourceLoaderMaxQueryLength = -1;

# End of automatically generated settings.
# Add more configuration options below.

</pre>

== copy/backup wiki ==
on main/primary wiki:
<pre>
/usr/local/etc/rc.d/mysql-server stop
ssh 69.55.230.18 "/etc/init.d/mysql stop"
rsync -av /var/db/mysql/my_wiki/ 69.55.230.18:/var/lib/mysql/my_wiki/
rsync -av /var/db/mysql/ib* 69.55.230.18:/var/lib/mysql/
/usr/local/etc/rc.d/mysql-server start
ssh 69.55.230.18 "/etc/init.d/mysql start"

</pre>

Infrastructure Machines

2013-10-17T17:17:41Z

24.165.90.109: /* Notes */

= jails =

== jail1 ==

* Location: castle, cab 3-8
* OS: FreeBSD 6.2 i386
* Networking: Priv IP: 10.1.4.101 (PCI nic), Pub IP: 69.55.230.107 (onboard)
* Hardware: Dell 2450. 4 SCSI SCA drive bays (2 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: two 74 GB (4 x 74GB) RAID1 arrays running on an Adaptec-based, Dell-branded (perc) RAID card.
* Remote management: none
* Disk accounting: gvinum

== jail2 ==

* Location: castle, cab 3-5
* OS: FreeBSD 7.2 amd64
* Networking: Priv IP: 10.1.4.102, Pub IP: 69.55.228.53 (2 onboard nics)
* Hardware: Dell 2950. 6 SATA/SAS drive bays (3 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: one 146 GB (2 x 146GB) RAID1 array, two 300 GB (4 x 300GB) RAID1 arrays running on an LSI-based, Dell-branded (PERC 6/i) RAID card.
* Remote management: [[DRAC/RMM|DRAC]] @ 10.1.4.232
* Disk accounting: md

== jail3 ==
* Location: i2b, cab 6
* OS: FreeBSD 8.3 amd64
* Networking: Priv IP: 10.1.2.103, Pub IP: 69.55.229.7 (2 onboard nics)
* Hardware: Supermicro (custom build). 6 SATA drive bays (2 columns of 3, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: two 300 GB (2 x 300GB) RAID1 array running on a 3ware 8006-2LP RAID card.
* Remote management: none
* Disk accounting: md

=== Notes ===
* '''We should not add users to this server since it is at I2B'''
* must be ssh'd to from nat2
* is a super jail for customer col01737

== jail4 ==

* Location: castle, cab 3-8
* OS: FreeBSD 9.1 x86_64
* Networking: Priv IP: 10.1.4.104, Pub IP: 69.55.228.104 (2 onboard nics)
* Hardware: Dell 2850. 6 SATA/SAS drive bays (3 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: one 1.4 TB RAID 5 array Dell-branded (PERC 4e)LSI megarc RAID card.
* Remote management: None
* Disk accounting: md

=== Notes ===
New Server not yet in production

== jail7 ==

* Location: castle, cab 3-7
* OS: FreeBSD 6.3 i386
* Networking: Priv IP: 10.1.4.107, Pub IP: 69.55.230.108 (2 onboard nics)
* Hardware: Dell 2950. 6 SATA/SAS drive bays (3 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: two 146 GB (4 x 146GB) RAID1 arrays, one 74 GB (2 x 74GB) RAID1 array running on an LSI-based, Dell-branded (PERC 6/i) RAID card.
* Remote management: [[DRAC/RMM|DRAC]] @ 10.1.4.237
* Disk accounting: gvinum

=== Notes ===
Do not run a verify while OS/jails running, will crash.

== jail8 ==

* Location: castle, cab 3-6
* OS: FreeBSD 8.0 amd64
* Networking: Priv IP: 10.1.4.108, Pub IP: 69.55.234.2 (2 onboard nics)
* Hardware: Dell 2950. 6 SATA/SAS drive bays (3 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: one 146 GB (2 x 146GB) RAID1 array, one 300 GB (2 x 300GB) RAID1 array, one 400 GB (2 x 400GB) RAID1 array, running on an LSI-based, Dell-branded (PERC 6/i) RAID card.
* Remote management: [[DRAC/RMM|DRAC]] @ 10.1.4.238
* Disk accounting: md

== jail9 ==

* Location: castle, cab 3-6
* OS: FreeBSD 8.2 amd64
* Networking: Priv IP: 10.1.4.109, Pub IP: 69.55.232.36 (2 onboard nics)
* Hardware: Dell 2950. 6 SATA/SAS drive bays (3 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: one 146 GB (2 x 146GB) RAID1 array, one 400 GB (2 x 300GB) RAID1 array running on an LSI-based, Dell-branded (PERC 5/i) RAID card.
* Remote management: [[DRAC/RMM|DRAC]] @ 10.1.4.239
* Disk accounting: md

== jail11 ==

* Location: castle, cab 3-7
* OS: FreeBSD 4.7 i386
* Networking: Priv IP: 10.1.4.111 (PCI nic), Pub IP: 69.55.236.92 (onboard)
* Hardware: Dell 2450. 4 SCSI SCA drive bays (2 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: two 74 GB (4 x 74GB) RAID1 arrays running on an Adaptec-based, Dell-branded (perc) RAID card.
* Remote management: none
* Disk accounting: vinum

== mx1 ==

* Location: castle, cab 3-7
* OS: FreeBSD 4.11 i386
* Networking: Priv IP: 10.1.4.201 (PCI nic), Pub IP: 69.55.237.3 (onboard)
* Hardware: Dell 2450. 4 SCSI SCA drive bays (2 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: one 36 GB (2 x 36GB) RAID1 array, one 74 GB (2 x 74GB) RAID1 array running on an Adaptec-based, Dell-branded (perc) RAID card.
* Remote management: none
* Disk accounting: vinum

=== Notes ===
* is our (old) backup mail/dns vps service host

== mx2 ==

* Location: castle, cab 3-5
* OS: FreeBSD 7.1 i386
* Networking: Priv IP: 10.1.4.202 (PCI nic), Pub IP: 69.55.237.90 (onboard)
* Hardware: Dell 2450. 4 SCSI SCA drive bays (2 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: two 74 GB (4 x 74GB) RAID1 arrays running on an Adaptec-based, Dell-branded (perc) RAID card.
* Disk accounting: gvinum

=== Notes ===
* is our latest backup mail/dns vps service host

== jail17 ==
* Location: castle, cab 3-8
* OS: FreeBSD 4.10 i386
* Networking: Priv IP: 10.1.4.117 (PCI nic), Pub IP: 69.55.228.2 (onboard nics)
* Hardware: Supermicro (custom build). 6 SCA SCSI drive bays (2 columns of 3, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: two 74 GB (4 x 74GB) RAID1 arrays, one 146 GB (2 x 146GB) RAID1 array, running on an Adaptec 2120S RAID card.
* Remote management: none
* Disk accounting: vinum

== jail18 ==
* Location: castle, cab 3-7
* OS: FreeBSD 4.10 i386
* Networking: Priv IP: 10.1.4.117 (PCI nic), Pub IP: 69.55.228.2 (onboard nics)
* Hardware: Supermicro (custom build). 6 SCA SCSI drive bays (2 columns of 3, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: two 74 GB (4 x 74GB) RAID1 arrays, one 146 GB (2 x 146GB) RAID1 array, running on an Adaptec 2120S RAID card.
* Remote management: none
* Disk accounting: vinum

== jail19 ==
* Location: castle, cab 3-6
* OS: FreeBSD 6.1 i386
* Networking: Priv IP: 10.1.4.119 (PCI nic), Pub IP: 69.55.228.200 (onboard nics)
* Hardware: Supermicro (custom build). 6 SCA SCSI drive bays (2 columns of 3, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: one 74 GB (2 x 74GB) RAID1 array, one 146 GB (2 x 146GB) RAID1 array, running on an Adaptec 2120S RAID card.
* Remote management: none
* Disk accounting: gvinum

= virts =

== quar1 ==

* Location: castle, cab 3-8
* OS: RedHat 7.3 x86
* Networking: Priv IP: 10.1.4.151 (PCI nic), Pub IP: 69.55.227.2 (onboard nic)
* Hardware: Dell 2450. 4 SCSI SCA drive bays (2 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: one 36 GB (2 x 36GB) RAID1 array, one 74 GB (2 x 74GB) RAID1 array, running on an Adaptec-based, Dell-branded (perc) RAID card.
* Remote management: none
* Virtuozzo version: 2.6.1
* VZ license: hwid=23C0.C0E1.6FDD.08BA.8971.8E1C.EBD5.1EDC serial=0DE6.903E.E239.E23F.470C.4369.4104.A5A4

=== Notes ===
* used to be the home of customers who's VE's would just run out of control/badly
* has a max of 10 VE's allowed to run

== virt9 ==

* Location: castle, cab 3-7
* OS: RedHat 7.3 x86
* Networking: Priv IP: 10.1.4.59 (PCI nic), Pub IP: 69.55.226.161 (onboard nic)
* Hardware: Dell 2450. 4 SCSI SCA drive bays (2 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: two 74 GB (2 x 74GB) RAID1 arrays running on an Adaptec-based, Dell-branded (perc) RAID card.
* Remote management: none
* Virtuozzo version: 2.6.1
* VZ license: hwid=BC15.B4D6.0D25.A5FE.F3BA.D518.E351.AE3F serial=F6AD.B6B4.5650.8869.C97C.73EE.AF65.FA8B

== virt11 ==

* Location: castle, cab 3-6
* OS: CentOS 5.4 x86
* Networking: Priv IP: 10.1.4.61, Pub IP: 69.55.238.3, 2 onboard nics
* Hardware: Dell 2950. 6 SATA/SAS drive bays (3 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: one 146 GB (2 x 146GB) RAID1 array, one 400 GB (2 x 400GB) RAID1 array, one 146 GB (2 x 146GB) RAID1 array, running on an LSI-based, Dell-branded (perc 6/i) RAID card.
* Remote management: [[DRAC/RMM|DRAC]] @ 10.1.4.211
* Virtuozzo version: 4.0.0
* VZ license: hwid="029D.A187.78E1.480F.49E3.E20A.7389.7F79" serial="163C.F3E2.195F.96B5.2D38.8937.9600.4A05"

== virt12 ==

* Location: castle, cab 3-6
* OS: CentOS 5.2 x86
* Networking: Priv IP: 10.1.4.62, Pub IP: 69.55.227.70, 2 onboard nics
* Hardware: Dell 2950. 6 SATA/SAS drive bays (3 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: two 300 GB (2 x 300GB) RAID1 arrays one 400 GB (2 x 400GB) RAID1 array, running on an LSI-based, Dell-branded (perc 6/i) RAID card.
* Remote management: [[DRAC/RMM|DRAC]] @ 10.1.4.212
* Virtuozzo version: 4.0.0
* VZ license: hwid="0C53.A413.E095.B4F4.51BC.D740.6919.A77B" serial="84E5.9498.3759.E683.E24B.2514.CA72.DC31"

== virt13 ==

* Location: castle, cab 3-5
* OS: CentOS 6.2 x86_64
* Networking: Priv IP: 10.1.4.63, Pub IP: 69.55.226.2, 2 onboard nics
* Hardware: Dell 2950. 6 SATA/SAS drive bays (3 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: one 146 GB (2 x 146GB) RAID1 array, one 600 GB (2 x 600GB) RAID1 array, running on an LSI-based, Dell-branded (perc 6/i) RAID card.
* Remote management: [[DRAC/RMM|DRAC]] @ 10.1.4.213
* Virtuozzo version: 4.7.0
* VZ license: hwid="7D07.93BE.0B1F.7D2B.B039.4B5B.48B6.453B" serial="60A4.A94C.44BB.DCD6.8D03.1778.605B.10FE"

=== Notes ===
* home to our latest/current signups
* currently the only 64bit vz host

== virt14 ==

* Location: castle, cab 3-?
* OS: CentOS 6.4x86_64
* Networking: Priv IP: 10.1.4.64 Pub IP: 69.55.225.14 2 onboard nics
* Hardware: Dell 2950. 6 SATA/SAS drive bays (3 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: one 600 GB (2 x 600GB) RAID1 array, running on an LSI-based, Dell-branded (perc 6/i) RAID card.
* Remote management: [[DRAC/RMM|DRAC]] @ 10.1.4.214
* Virtuozzo version: 4.7.0
* VZ license: hwid="" serial=""

=== Notes ===
* new home to our latest/current signups
* virt 13 and 14 currently the only 64bit vz hosts

== virt15 ==

* Location: castle, cab 3-5
* OS: RedHat 9 x86
* Networking: Priv IP: 10.1.4.65, Pub IP: 69.55.232.160 (2 onboard nics)
* Hardware: Supermicro (custom build). 6 SCSI SCA drive bays (2 columns of 3, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: one 74 GB (2 x 74GB) RAID1 array, two 146 GB (2 x 146GB) RAID1 arrays, running on an LSI MegaRAID SCSI 320-1 RAID card.
* Remote management: none
* Virtuozzo version: 2.6.2
* VZ license: hwid=A90F.6F48.E723.D8BA.3025.184A.5B73.D11E serial=E94B.5164.C1E6.A67F.67D1.7D96.0B6C.5524

== virt16 ==

* Location: castle, cab 3-7
* OS: Fedora Core 4 x86
* Networking: Priv IP: 10.1.4.66, Pub IP: 69.55.232.2 (2 onboard nics)
* Hardware: Supermicro (custom build). 6 SCSI SCA drive bays (2 columns of 3, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: one 74 GB (2 x 74GB) RAID1 array, one 146 GB (2 x 146GB) RAID1 array, running on an LSI MegaRAID SCSI 320-1 RAID card.
* Remote management: none
* Virtuozzo version: 3.0.0
* VZ license: hwid=DEFA.A325.7230.BBC8.9715.8B52.3FD7.27BE serial=66C0.41EA.3FBB.11D3.9CC6.55C7.09AE.14AB

== virt17 ==

* Location: castle, cab 3-6
* OS: CentOS 4.4 x86
* Networking: Priv IP: 10.1.4.67, Pub IP: 69.55.232.162, 2 onboard nics
* Hardware: Dell 2950. 6 SATA/SAS drive bays (3 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: two 146 GB (2 x 146GB) RAID1 arrays running on an LSI-based, Dell-branded (perc 5/i) RAID card.
* Remote management: [[DRAC/RMM|DRAC]] @ 10.1.4.217
* Virtuozzo version: 3.0.0
* VZ license: hwid=2E14.AED9.70B8.C26E.D99F.B0D3.BCD2.229C serial=2A11.DAD0.61DB.E889.8DF4.9AF7.CF82.3C37

== virt19 ==

* Location: castle, cab 3-6
* OS: CentOS 5.2 x86
* Networking: Priv IP: 10.1.4.69, Pub IP: 69.55.236.2, 2 onboard nics
* Hardware: Dell 2950. 6 SATA/SAS drive bays (3 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: one 146 GB (2 x 146GB) RAID1 array, one 300 GB (2 x 300GB) RAID1 array, running on an LSI-based, Dell-branded (perc 5/i) RAID card.
* Remote management: [[DRAC/RMM|DRAC]] @ 10.1.4.219
* Virtuozzo version: 3.0.0
* VZ license: hwid=3968.13F7.B2AC.8952.8E19.13A9.6EF5.5822 serial=061D.84CD.CCE5.B213.15B5.C061.D6A7.B034

= mail =
== Summary ==
This machine (mail) is the swiss army knife of the company, playing host to many services and functions.

* Location: castle, cab 3-8
* OS: FreeBSD 4.10 x86
* Networking: Priv IP: 10.1.4.5, Pub IPs: 69.55.230.2, 69.55.225.225 (ns1c jail), 69.55.230.9. 1 onboard and 1 PCI
* Hardware: Dell 2450. 4 SCSI SCA drive bays (2 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: two 36 GB (2 x 36GB) RAID1 arrays running on an Adaptec-based, Dell-branded (perc) RAID card.

== Services Provided ==
* mail
* web
* mysql
* bigbrother server/pager
* snmp
* named in a jail (ns1c)

== email ==
This server hosts mail for johncompanies.com (mail.johncompanies.com). Sendmail 8.13.6/8.13.6 is listening on 69.55.230.2 port 25 for incoming mail. Relaying is allowed per /etc/mail/relay-domains
Other addresses (aliases) are defined per /etc/mail/aliases

The following active users have mail hosted on this server:
* dave
* linux
* support
* payments
* sales
* tech1
* info

Traditionally, mail is checked via shell apps (pine). qpopper (pop3s) is running to allow mail downloading. Checking mail in this way causes an opened INBOX in pine to lock read-only. For this reason, we tee incoming mail to support and linux to tech1.

Procmail rules are setup to filter spam and send text messages. They are enabled for info, support, linux, tech1, dave and can be found in ~/Procmail/, for example:
<pre># more ~support/Procmail/rc.emergency
:0c # use c only if you want to forward a copy and file the original later
* ^Subject:.*\<emergency\>
* ! ^Subject:\<re\>
{
:0h
FROMANDSUBJECT=|formail -XFrom: -XSubject:

:0fwh
| /usr/local/bin/formail -I"Subject: " -I"To: pager@johncompanies.com" ; echo $FROMANDSUBJECT ; echo

:0
! -t
}

</pre>

control: <tt>cd /etc/mail; make stop</tt> (stop), <tt>cd /etc/mail; make start</tt> (start)

The following aliases are also in place:

<pre>debian: linux
jobs: info
careers: info
#reboot: 6128102202@txt.att.net
#reboot: 8582298897@vtext.com
reboot: pager
#pager: 8582298897@vtext.com
pager: 4158718324@txt.att.net
tech1on: "| /usr/local/sbin/tech1on.sh"
tech1off: "| /usr/local/sbin/tech1off.sh"</pre>

To change them, edit <tt>/etc/aliases</tt> and then run <tt>newaliases</tt>

Note on tech1: this address was setup as a read-only address to be mirrored on all email coming into support and linux. We set this up so we could easily check support mail via a pop client- popping email locks out the user in pine so checking support/linux directly via pop was not an option. When checking and responding to email that comes into tech1, care should be taken to make sure it is sent as/under an address other than tech1. This is cause tech1 is not monitored by support staff as closely as email to support/linux. Further, the tech on call may not be checking tech1. Lastly, because of the nature of the copying, you will sometimes notice certain automated email/notices are received 2x in support- this is because of/related to the tech1 mirror.

To enable it (on mail, run):

<tt>~support/tech1on.sh</tt>

To disable

<tt>~support/tech1off.sh</tt>

Or via email:

<tt>tech1on@johncompanies.com
tech1off@johncompanies.com</tt>

== web ==

See [[Management_System_/_Public_Website_/_Signup|Management System / Public Website / Signup]]

== mysql ==
mysql 4.1.22 is running on port 3306

* datadir: <tt>/mnt/data1/db/mysql/</tt>
* config: <tt>/etc/my.cnf</tt>
* database: <tt>jc</tt>
* control: <tt>/usr/local/etc/rc.d/mysql-server.sh stop</tt> (stop), <tt>/usr/local/etc/rc.d/mysql-server.sh start</tt> (start)

== bigbrother ==
There is a client running on mail (which monitors the services running on mail and mail itself), installed under <tt>/usr/home/bb/bbc1.9e-btf</tt><br>
And the big brother pager/server (which displays information gathered from all bb-monitored machines, including mail) is installed under <tt>/usr/home/bb/bbsrc/bb1.9i-btf</tt>

Both are running under the user <tt>bb</tt>

Refer to [[BigBrother]] for more about use.

== DNS (ns1c.johncompanies.com) ==
ns1c is a jail running on the mail server, who's IP is 69.55.225.225

It's running from <tt>/mnt/data1/ns1c-dir</tt>

See [[DNS]] for more details

== Usage and Notes ==
* always mounted to backup1 and backup2 via nfs:
<pre>backup2:/mnt/data1 on /backup (nfs)
backup2:/mnt/data2 on /backup2 (nfs)
backup2:/mnt/data3 on /backup3 (nfs)
backup2:/mnt/data4 on /backup4 (nfs)
backup1:/data on /backup1 (nfs)
</pre>

== Cronjobs ==
* * * * * /usr/local/www/mgmt/mrtg/mrtg.sh > /dev/null 2>&1
Gathers up data for our mrtg/load graphs

*/5 * * * * /usr/local/bin/rsync -a root@nat2:/mnt/data1/mrtg/data/ /usr/local/www/mgmt/mrtg/data/
Gathers up data from i2b servers for our mrtg/load graphs

40 0 * * * /usr/local/bin/rsync -a root@nat2:"/mnt/data1/mrtg/*.cfg" /usr/local/www/mgmt/mrtg
Gathers up mrtg configuration (port names) from i2b switches for our mrtg/load graphs

41 0 * * * for f in `grep -l "mnt\/data1" /usr/local/www/mgmt/mrtg/switch-p*.cfg`; do cat $f | sed s#\/mnt\/data1#\/usr\/local\/www\/mgmt# > $f.new; mv $f.new $f; done
Gathers up mrtg configuration (port names) from castle switches for our mrtg/load graphs

1 0 1 * * cp /usr/local/www/mgmt/html/top20ip /usr/local/www/mgmt/html/top20ip_last
1 0 1 * * cp /usr/local/www/mgmt/html/top20customers /usr/local/www/mgmt/html/top20customers_last
2 * * * * /usr/local/www/cronjobs/top20ip.pl > /dev/null 2>&1
15 * * * * /usr/local/www/cronjobs/top20customer.pl > /dev/null 2>&1
1 0 1 * * rm /usr/local/www/mgmt/html/bandtrack
Archiving and generation of bandwidth statistics presented in mgmt -> Reference -> Bandwidth

1 0 * * * /usr/local/etc/rsync.backup
Nightly backup script

0 1 * * * /usr/local/www/mgmt/awstats/wwwroot/cgi-bin/awstats.pl -config=jcpub -update
Public web traffic stats

15 0 * * * rm /usr/local/www/mgmt/bwgraphs/*.png
16 0 * * * rm /usr/local/www/am/bwgraphs/*
Cleanup for graph-related temp data generated by customers using the bandwidth reports via the AM

10 0 1 * * /usr/local/www/cronjobs/monthly_bandwidth_report.pl
Monthly bandwidth overage report

*/3 * * * * /usr/local/www/cronjobs/bbcheck.pl
Updates mgmt with bb monitoring issues

5 0 * * * /usr/local/www/cronjobs/shutdownreminder.pl
Emails customers reminding them of upcoming shutdown date

7 0 * * * /usr/local/www/cronjobs/invoice_email.pl
Emails customers who have invoices and are set to auto-email (currently no customer gets these)

8 */4 * * * /usr/local/www/cronjobs/mysqlrepchk.pl
Checking that we are properly replicating (mysql) traffic data from bwdb to backup1

16 0 1 * * /usr/local/www/cronjobs/purge_traffic.pl
Removed old traffic data from the traffic database (running on backup1)

*/5 * * * * chmod 0700 /usr/local/www/ccard_orders/* && mv /usr/local/www/ccard_orders/* /usr/local/www/ccard_orders/done
Secure credit card data: set root-read-only

25 0 * * * /usr/local/www/cronjobs/biller.pl
Enters service charges in customer billing ledgers

10 13 * * * /usr/local/www/cronjobs/pfp_batch_gather.pl
Looks for customers with balance due and active credit card on file, prepares a payflow batch

10 14 * * * /usr/local/www/cronjobs/pfp_batch_process.pl
Tries to collect ccard funds for items in payflow batch - communicates with payflow

15 13 * * * /usr/local/www/cronjobs/pb_batch_gather.pl
Looks for customers with balance due and active paypal billing agreement on file, prepares a paypal batch

15 14 * * * /usr/local/www/cronjobs/pb_batch_process.pl
Tries to collect paypal funds for items in paypal batch - communicates with paypal

0 7 * * 1 /usr/local/www/cronjobs/email_pmt_reminder.pl
Emails customers in arrears, reminding them to pay

0 0 1 * * /usr/bin/mail -s 'archive sent mail in pine' support@johncompanies.com < /dev/null
Reminds us to archive sent mail

0 3 * * * /usr/local/bin/rsync -a isys.e-monitoring.net:/var/mail /backup2/isys; /usr/local/bin/rsync -a isys.e-monitoring.net:/usr/home /backup2/isys
Backup data on isys

== Regular maintenance ==
*[[Routine_Maintenance#Adaptec_Controllers|Check RAID array]]

= nat =
== Summary ==
This is the main machine to which we ssh and runs all our screen sessions. Further, it's ip runs in a special block which is not routed through the firewall and this is somewhat immune to DoS attacks which hobble our firewall. Lastly, it acts as a nat server for certain/random devices on the private network.

* Location: castle, cab 3-7
* OS: FreeBSD 9.1 i386
* Networking: Priv IP: 10.1.4.1, Pub IPs: 69.55.233.195, 69.55.233.196, 69.55.233.197, 69.55.233.198, 69.55.233.199. 1 onboard and 1 PCI
* Hardware: Custom 1U. single power supply.
* Drives: one 8 GB IDE drive

== Services Provided ==
* nat

== nat control ==
All rules are contained in and look like:

<pre>cat /etc/ipnat.rules
# www (was 69.55.230.12)
# virt19
#bimap fxp0 10.1.4.209/32 -> 69.55.233.198/32
# virt18
#bimap fxp0 10.1.4.208/32 -> 69.55.233.196/32
# virt13
#bimap fxp0 10.1.4.213/32 -> 69.55.233.196/32
# virt12
#bimap fxp0 10.1.4.212/32 -> 69.55.233.196/32
# virt17
bimap fxp0 10.1.4.217/32 -> 69.55.233.196/32
# virt11
#bimap fxp0 10.1.4.211/32 -> 69.55.233.196/32
# ASA
#bimap fxp0 10.1.4.172/32 -> 69.55.233.196/32
# P1A
bimap fxp0 10.1.4.240/32 -> 69.55.233.197/32
#bimap fxp0 10.1.4.238/32 -> 69.55.233.197/32
# developer (was 69.55.230.17)
# jail2
#bimap fxp0 10.1.4.232/32 -> 69.55.233.198/32
# jail8
#bimap fxp0 10.1.4.238/32 -> 69.55.233.198/32
# jail9
#bimap fxp0 10.1.4.239/32 -> 69.55.233.198/32
# POLL
#BIMAP EM0 10.1.6.134/32 -> 69.55.230.20/32
# 1U SUN
#BIMAP EM0 10.1.4.4/32 -> 69.55.227.46/32
# ??
#BIMAP EM0 10.1.6.3/32 -> 69.55.230.100/32
# random machine
#bimap fxp0 10.1.6.13/32 -> 69.55.233.199/32
#bimap fxp0 10.1.4.232/32 -> 69.55.233.199/32
# OFFICE OUTBOUND TRAFFIC
#map fxp0 10.1.6.0/24 -> 0.0.0.0/32 proxy port ftp ftp/tcp
#map fxp0 10.1.6.0/24 -> 0.0.0.0/32</pre>

A simple entry looks like:
bimap fxp0 10.1.4.240/32 -> 69.55.233.197/32
Which essentially means make private IP 10.1.4.240 reachable on 69.55.233.197 and allow 10.1.4.240 to communicate with the public internet via 69.55.233.197

To reload new rule config:
ipnat -C -F -f /etc/ipnat.rules

You may want to setup natting, as above, when you need to reach a DRAC card's web interface, wherin the DRAC card only has a private IP.

= nat2 =
== Summary ==
This is the main machine to which we ssh and runs all our screen sessions at i2b, and runs ns3c (this is kind of the what mail is to castle). Further, it's ip runs in IP space provided by i2b: 66.181.18.1 - 66.181.18.30, which is not routed through the firewall and this is somewhat immune to DoS attacks which hobble our firewall. Lastly, it acts as a nat server for certain/random devices on the private network.

* Location: i2b, cab 6
* OS: FreeBSD 6.4 x86
* Networking: Priv IP: 10.1.2.1, Pub IPs: 69.55.229.2, 69.55.229.3, 66.181.18.4, 66.181.18.5, 66.181.18.6, 66.181.18.7, 66.181.18.8, 66.181.18.9, 66.181.18.10, 66.181.18.11, 66.181.18.12, 66.181.18.13, 66.181.18.14 1 onboard and 1 PCI
* Hardware: Custom 2U. 6 drive bays, non-hot-swappable. single power supply.
* Drives: one 150 GB (2 x 150GB) RAID1 array running on a 3ware 8006 RAID card.

== Services Provided ==
* nat
* bigbrother
* ns3c (jail)
* ntp

== nat config ==
Here's what's currently nat'd on nat2:
<pre>cat /etc/ipnat.rules
# sample entry
#ATS-9
bimap em0 10.1.2.79/32 -> 66.181.18.14/32
#ATS-8
bimap em0 10.1.2.78/32 -> 66.181.18.13/32
#ATS-7
bimap em0 10.1.2.77/32 -> 66.181.18.12/32
#ATS-6
bimap em0 10.1.2.76/32 -> 66.181.18.6/32
#ATS-5
bimap em0 10.1.2.75/32 -> 66.181.18.7/32
#ATS-4
bimap em0 10.1.2.74/32 -> 66.181.18.8/32
#ATS-3
bimap em0 10.1.2.73/32 -> 66.181.18.9/32
#ATS-2
bimap em0 10.1.2.72/32 -> 66.181.18.10/32
#ATS-1
bimap em0 10.1.2.71/32 -> 66.181.18.11/32
#bwdb2
bimap em0 10.1.2.4/32 -> 66.181.18.5/32

# spare

map em0 10.1.2.0/24 -> 0.0.0.0/32 proxy port ftp ftp/tcp

#bimap fxp0 10.1.6.49/32 -> 10.1.1.2/32
#bimap fxp0 10.1.4.208/32 -> 69.55.233.196/32
</pre>

== build ==

* partition map:
<pre>/ 512m
swap 1G
/var 256m
/tmp 256m
/usr 5g
/mnt/data1 ~</pre>

* edit /etc/make.conf
<pre>echo "WITHOUT_X11=yes \
KERNCONF=nat2 \
BOOT_COMCONSOLE_SPEED=115200" >> /etc/make.conf</pre>

* add settings to /boot/loader.conf and /boot.config

<pre>echo "-Dh" >> /boot.config

echo 'console="comconsole,vidconsole" \
boot_multicons="YES" \
boot_serial="YES" \
comconsole_speed="115200"' >> /boot/loader.conf</pre>

* turn off all ttyv's except 0 and 1 in /etc/ttys
also turn on ttyd0, change type to vt100:
<pre>vi /etc/ttys

ttyv2 "/usr/libexec/getty Pc" cons25 off secure
ttyv3 "/usr/libexec/getty Pc" cons25 off secure
ttyv4 "/usr/libexec/getty Pc" cons25 off secure
ttyv5 "/usr/libexec/getty Pc" cons25 off secure
ttyv6 "/usr/libexec/getty Pc" cons25 off secure
ttyv7 "/usr/libexec/getty Pc" cons25 off secure
# Serial terminals
# The 'dialup' keyword identifies dialin lines to login, fingerd etc.
ttyd0 "/usr/libexec/getty std.9600" vt100 on secure

kill -1 1</pre>

on console server:
vi /etc/remote
(rename port to jail8 depending on where and which digi plugged into)
test serial console

* populate hosts
<pre>echo "69.55.230.10 backup2" >> /etc/hosts
echo "69.55.230.11 backup1" >> /etc/hosts
echo "10.1.2.4 bwdb2" >> /etc/hosts
echo "10.1.2.3 backup3" >> /etc/hosts</pre>

* put key in authorized_keys on backup3
<pre>cd
ssh-keygen -t dsa -b 1024</pre>
(default location, leave password blank)

<pre>cat /root/.ssh/id_dsa.pub | ssh backup3 'cat - >> /root/.ssh/authorized_keys'
cat /root/.ssh/id_dsa.pub | ssh backup2 'cat - >> /root/.ssh/authorized_keys'
cat /root/.ssh/id_dsa.pub | ssh backup1 'cat - >> /root/.ssh/authorized_keys'</pre>

confirm that you can ssh to backup3 and backup 2 without getting a login prompt

ssh backup3 hostname

ssh backup2 hostname

ssh backup1 hostname

* edit root's path and login script:
vi /root/.cshrc

Change alias entries (add G):
<pre>alias la ls -aG
alias lf ls -FAG
alias ll ls -lAG
alias ls ls -AG
alias mbm mb mount
alias mbu mb umount</pre>

and alter the prompt, set the following:
set prompt = "`/bin/hostname -s` %/# "

* install cvsup
<pre>cd /usr/ports/net/cvsup-without-gui
make install clean; rehash; mail -s 'cvs installed' dave.boodman@vtext.com < /dev/null</pre>

* get latest sources for this release:
<pre>cd /usr/src
echo "*default host=cvsup4.freebsd.org\
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_6_4\
*default delete use-rel-suffix\
*default compress\
src-all" > sup

cvsup sup ; mail -s 'cvs sup done' dave.boodman@vtext.com < /dev/null</pre>

* configure new kernel.

cd /usr/src/sys/i386/conf
scp backup2:/mnt/data4/build/freebsd/nat2-6.4 ./nat2

* build, install kernel and world

<pre>cd /boot

mv kernel kernel.GENERIC
cd kernel.GENERIC
cd /usr/src
make buildkernel installkernel

make buildworld ; mail -s 'buildworld done' support@johncompanies.com < /dev/null
make installworld
mergemaster -i
</pre>

* populate /etc/rc.conf with IPs and NFS settings
<pre>vi /etc/rc.conf

hostname="nat2.johncompanies.com"
kern_securelevel_enable="NO"
portmap_enable="NO"
sendmail_enable="NO"
usbd_enable="YES"

xntpd_enable="YES"
xntpd_flags="-A -p /var/run/ntpd.pid"

nfs_client_enable="YES"
nfs_reserved_port_only="YES"
ifconfig_em0="inet 10.1.6.50 netmask 255.255.255.0"
#ifconfig_em0="inet 69.55.229.2 netmask 255.255.255.0"
#ifconfig_em0_alias0="inet 69.55.229.229 netmask 255.255.255.255"
ifconfig_fxp0="inet 69.55.229.2 netmask 255.255.255.0"
ifconfig_fxp0_alias0="inet 69.55.229.3 netmask 255.255.255.255"
ifconfig_fxp1="inet 10.1.2.1 netmask 255.255.255.0"
defaultrouter="10.1.6.1"
#defaultrouter=" 66.181.14.250"
snmpd_enable="YES"
ipnat_enable="YES"
ipnat_rules="/etc/ipnat.rules"
gateway_enable="YES"

inetd_enable="YES"
inetd_flags="-wW -a 10.1.2.1"
fsck_y_enable="YES"
background_fsck="NO"
sshd_enable="YES"</pre>

* reboot. Confirm new kernel is loaded

uname -a

* update ports:
<pre>cd /usr/ports
echo "*default host=cvsup4.FreeBSD.org\
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_6_4\
*default delete use-rel-suffix\
*default compress\
ports-all tag=." > sup

cvsup sup; mail -s 'cvs sup ports done' dave.boodman@vtext.com < /dev/null</pre>

* Install raid mgmt tool

<pre>cd /usr/local/sbin
fetch http://3ware.com/download/Escalade9690SA-Series/9.5.3/tw_cli-freebsd-x86-9.5.3.tgz
tar xzf tw_cli-freebsd-x86-9.5.3.tgz
rm tw_cli-freebsd-x86-9.5.3.tgz
chmod 0700 tw_cli</pre>

Test:
./tw_cli info c0

* install rsync from ports
<pre>cd /usr/ports/net/rsync
make install clean</pre>
choose default options

* install perl from ports
<pre>cd /usr/ports/lang/perl5.8
make install clean</pre>

* install screen from ports
<pre>cd /usr/ports/sysutils/screen
make install clean</pre>

* install bb client
<pre>adduser
Username: bb
Full name: bb
Uid (Leave empty for default): 1984
Login group [bb]:
Login group is bb. Invite bb into other groups? []:
Login class [default]:
Shell (sh csh tcsh nologin) [sh]:
Home directory [/home/bb]:
Use password-based authentication? [yes]:
Use an empty password? (yes/no) [no]:
Use a random password? (yes/no) [no]: yes
Lock out the account after creation? [no]:
Username : bb
Password : <random>
Full Name : bb
Uid : 1984
Class :
Groups : bb
Home : /home/bb
Shell : /bin/sh
Locked : no
OK? (yes/no): yes

cd /usr/home/bb
scp backup2:/mnt/data4/build/bb/bb-freebsd.tar .
tar xvf bb-freebsd.tar</pre>
edit /home/bb/bbc1.9e-btf/etc/bb-hosts with something like:
<pre>echo "69.55.230.2 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh \
10.1.2.1 nat2.johncompanies.com # ssh" > /home/bb/bbc1.9e-btf/etc/bb-hosts
vi /home/bb/bbc1.9e-btf/ext/openfiles
MACHINE="nat2,johncompanies,com" # HAS TO BE IN A,B,C FORM

cd /usr/home/bb/bbc1.9e-btf/etc
./bbchkcfg.sh
(y to questions)
./bbchkhosts.sh
(ignore ssh errors)
cd ../..
chown -R bb .
su bb
cd
cd bbc1.9e-btf/src
make; make install
cd ..

vi /usr/home/bb/bbc1.9e-btf/bin/bb-cpu.sh
$1 $TOPARGS > $BBTMP/TOP.$$
# /usr/local/jail/bin/jtop > $BBTMP/TOP.$$

./runbb.sh start
more BBOUT
(look for errors)
exit

echo 'su - bb -c "cd /home/bb/bbc1.9e-btf; ./runbb.sh start"' > /usr/local/etc/rc.d/bb.sh
chmod +x /usr/local/etc/rc.d/bb.sh
</pre>

Punch a hole in the firewall to allow it to communicate with bb monitor (probably already exists):

ipfw add 96 allow ip from 66.181.18.0/27 to 69.55.230.2

* configure bb on mail:
<pre>vi /usr/home/bb/bbsrc/bb1.9e-btf/etc/bb-hosts
64.163.14.48 nat2.johncompanies.com # ssh

su bb
cd
bbsrc/bb/runbb.sh restart ; exit</pre>

* configure ntp
<pre>echo "server 69.55.230.2
server 0.pool.ntp.org
server 1.pool.ntp.org
server 2.pool.ntp.org
server 3.pool.ntp.org
server 66.187.233.4
server 217.204.76.170
server 64.112.189.11
server 66.69.112.130
server 80.85.129.25
server 80.237.234.15
server 130.60.7.44
server 134.99.176.3
server 198.144.202.250
server 202.74.170.194
server 204.17.42.199
server 204.87.183.6
server 213.15.3.1
server 213.239.178.33
server 217.114.97.97
server 69.55.230.2" > /etc/ntp.conf</pre>

<pre>/usr/sbin/ntpd -A -p /var/run/ntpd.pid
sleep 2; ntpq -p</pre>
(confirm it’s able to reach our time server)

echo '/usr/sbin/ntpd -c /etc/ntp.conf -p /var/run/ntpd.pid -f /var/db/ntpd.drift' > /usr/local/etc/rc.d/ntp.sh
chmod 0700 /usr/local/etc/rc.d/ntp.sh

* fwd and reverse lookups on ns1c
vr johncompanies.com
(edit the PTR too)

* setup backups, nfs mount

<pre>mkdir /backup3
echo 'backup3:/data /backup3 nfs rw,bg 0 0' >> /etc/fstab

echo '#\!/bin/sh\
backupdir=/data/nat2/current\
\
## ENTRY /etc ' > /usr/local/etc/backup.config</pre>

on backup3:
setup backup dirs:
ssh backup3 mkdir -p /data/nat2/current

on backup3, add the system to
vi /usr/local/sbin/snapshot_archive

<pre>scp 69.55.230.2:/usr/local/etc/rsync.backup /usr/local/etc/rsync.backup
vi /usr/local/etc/rsync.backup
backup1 > backup3</pre>

crontab -e
1 0 * * * /usr/local/etc/rsync.backup

* edit sshd_config for security
<pre>vi /etc/ssh/sshd_config
ListenAddress 66.181.18.1
ListenAddress 69.55.229.2
ListenAddress 10.1.2.1

kill -1 `cat /var/run/sshd.pid`</pre>

* raid chk

<pre>cat > /usr/local/sbin/lsiraidchk
#!/usr/bin/perl

my @out = split "\n", `megarc -ldInfo -a0 -Lall|grep Status:`;

foreach (@out) {
if ($_ =~ /DEGRADED/) { $date = `date`; chomp $date; `echo "$date: RAID ARRAY DEGRADED" >> /var/log/messages`; }
#print $_;
}</pre>

* netflow stuff
add crontab entries
<pre>crontab -e
30 3 * * * /usr/local/etc/rsync.backup
0 0 1 * * /sbin/ipfw zero
0 0 1 * * /sbin/ipfw del 3 4 5
59 23 30 * * /sbin/ipfw show > /tmp/ipfw_count
3 0 30 * * /sbin/ipfw show > /tmp/ipfw_count
0 3 * * * /etc/makefwrules.pl; /etc/makepiperules.pl;
*/5 * * * * /usr/local/sbin/lsiraidchk

#10 0 * * * rm /var/spool/clientmqueue/*</pre>

<pre>scp /etc/makefwrules.pl user@64.163.14.48:~
scp /etc/makepiperules.pl user@64.163.14.48:~
mv /home/user/makefwrules.pl /etc
mv /home/user/makepiperules.pl /etc
touch /etc/firewall.sh
mkdir /etc/oldrules/</pre>

other binaries

<pre>scp /usr/local/bin/rulemaker user@64.163.14.48:~
mv ~user/rulemaker /usr/local/sbin
scp ~user/Sendmail.pm user@64.163.14.48:~
scp ~user/doswatch.pl user@64.163.14.48:~</pre>

* add nat rules
<pre>vi /etc/ipnat.rules
# sample entry
bimap fxp0 10.1.6.70/32 -> 10.1.6.59/32
#bimap fxp0 10.1.4.208/32 -> 69.55.233.196/32

ipnat -C -f /etc/ipnat.rules</pre>

* shell for user
<pre>cp /root/.cshrc ~user/
vi ~user/</pre>
change # to $

* mrtg

<pre>cd /usr/ports/net-mgmt/mrtg
make install clean</pre>
(no FONTCONFIG, v3)

this didn't work cause of libtool incompat

so manually moved files:

<pre>scp /usr/local/bin/cfgmaker user@nat2:/usr/local/bin/cfgmaker
scp /usr/local/lib/perl5/site_perl/5.6.1/MRTG_lib.pm root@nat2:/usr/local/lib/perl5/site_perl/5.8.9/
scp /usr/local/lib/perl5/site_perl/5.6.1/SNMP_util.pm root@nat2:/usr/local/lib/perl5/site_perl/5.8.9/
scp /usr/local/lib/perl5/site_perl/5.6.1/BER.pm root@nat2:/usr/local/lib/perl5/site_perl/5.8.9/
scp /usr/local/lib/perl5/site_perl/5.6.1/SNMP_Session.pm root@nat2:/usr/local/lib/perl5/site_perl/5.8.9/
scp /usr/local/bin/mrtg root@nat2:/usr/local/bin/mrtg
scp /usr/local/lib/perl5/site_perl/5.6.1/locales_mrtg.pm root@nat2:/usr/local/lib/perl5/site_perl/5.8.9/locales_mrtg.pm
scp /usr/local/bin/rrdtool root@nat2:/usr/local/bin/rrdtool
scp /usr/local/lib/perl5/site_perl/5.6.1/mach/RRDs.pm root@nat2:/usr/local/lib/perl5/site_perl/5.8.9/mach/RRDs.pm
rsync -av /usr/local/lib/perl5/site_perl/5.6.1/mach/auto/RRDs/ root@nat2:/usr/local/lib/perl5/site_perl/5.8.9/mach/auto/RRDs/
scp /usr/lib/libz.so.2 root@nat2:/usr/lib/libz.so.2
scp /usr/lib/libm.so.2 root@nat2:/usr/lib/libm.so.2
rsync -av /usr/local/lib/librrd* root@nat2:/usr/local/lib/
scp /usr/lib/libc.so.4 root@nat2:/usr/lib/libc.so.4

rsync -av /usr/ports/net/rrdtool root@nat2:/usr/ports/net
cd /usr/ports/net/rrdtool
make install

mkdir -p /mnt/data1/mrtg/data
scp /usr/local/www/mgmt/mrtg/template.pl root@nat2:/mnt/data1/mrtg/
scp /usr/local/www/mgmt/mrtg/host.pl root@nat2:/mnt/data1/mrtg/

cfgmaker --if-template=template.pl --show-op-down --global "options[_]: growright,bits" --global 'WorkDir: /mnt/data1/mrtg/data' --global 'Interval: 1' --global 'LogFormat: rrdtool' --global 'PathAdd: /usr/local/bin' --global 'LibAdd: /usr/local/lib' --host-template=host.pl jc292401@10.1.2.50 --output=switch-p20.cfg

cat > /mnt/data1/mrtg/mrtg.sh
#!/bin/sh
/usr/local/bin/mrtg /mnt/data1/mrtg/switch-p20.cfg

chmod 0700 /mnt/data1/mrtg/mrtg.sh

crontab -e
* * * * * /mnt/data1/mrtg/mrtg.sh 2>&1 > /dev/null</pre>

* snmp firewall block
<pre>cat > /usr/local/etc/rc.d/boot.sh
ipfw add 10 allow udp from 69.55.230.2 to any 161
ipfw add 10 allow udp from 10.1.2.1 to any 161
ipfw add 11 deny udp from any to any 161
chmod 0700 /usr/local/etc/rc.d/boot.sh</pre>

= bwdb =
== Summary ==
This machine tracks and stores network traffic (netflow) at castle. It is our means to monitor customer bandwidth usage.

* Location: castle, cab 3-7
* OS: FreeBSD 4.10 x86
* Networking: Priv IP: 10.1.4.203 There are 2 onboard nic's, one of which is the "listener"
* Hardware: Custom 1U. Single power supply.
* Drives: one 250 GB (2 x 250GB) RAID1 array running on a Promise IDE RAID card.

== Services Provided ==
* netflow
* mysql
* bigbrother
* snmp

== netflow ==

The main function of this server is to run netflow on an eth device in promiscuous mode so as to hear everything happening on the port (wherein all network traffic is mirrored to that port via the cisco swith). Every 15min, it creates a flow file under <tt>/usr/home/flows/</tt> (organized by date). The flow file contains all traffic data for a 15min increment of time.

A cronjob moves that flow file (or files if there are multiple due to some delay)
1,16,31,46 * * * * /usr/home/flowbin/queue.pl

into a processing queue:
<tt>/usr/home/working</tt>

Then a separate file processes whatever flow files it finds there, inserting the data into the local mysql database:
2,17,32,47 * * * * /usr/home/flowbin/processflows.pl

== mysql ==

The database storing all the traffic data is named <tt>traffic</tt>
Tables:
<pre>mysql> show tables;
+---------------------------+
| Tables_in_traffic |
+---------------------------+
| dailyIpTotals_69_55_224 |
| dailyIpTotals_69_55_225 |
| dailyIpTotals_69_55_226 |
| dailyIpTotals_69_55_227 |
| dailyIpTotals_69_55_228 |
| dailyIpTotals_69_55_229 |
| dailyIpTotals_69_55_230 |
| dailyIpTotals_69_55_231 |
| dailyIpTotals_69_55_232 |
| dailyIpTotals_69_55_233 |
| dailyIpTotals_69_55_234 |
| dailyIpTotals_69_55_235 |
| dailyIpTotals_69_55_236 |
| dailyIpTotals_69_55_237 |
| dailyIpTotals_69_55_238 |
| dailyIpTotals_69_55_239 |
| dailyPortTotals_69_55_224 |
| dailyPortTotals_69_55_225 |
| dailyPortTotals_69_55_226 |
| dailyPortTotals_69_55_227 |
| dailyPortTotals_69_55_228 |
| dailyPortTotals_69_55_229 |
| dailyPortTotals_69_55_230 |
| dailyPortTotals_69_55_231 |
| dailyPortTotals_69_55_232 |
| dailyPortTotals_69_55_233 |
| dailyPortTotals_69_55_234 |
| dailyPortTotals_69_55_235 |
| dailyPortTotals_69_55_236 |
| dailyPortTotals_69_55_237 |
| dailyPortTotals_69_55_238 |
| dailyPortTotals_69_55_239 |
| ipTotals_69_55_224 |
| ipTotals_69_55_225 |
| ipTotals_69_55_226 |
| ipTotals_69_55_227 |
| ipTotals_69_55_228 |
| ipTotals_69_55_229 |
| ipTotals_69_55_230 |
| ipTotals_69_55_231 |
| ipTotals_69_55_232 |
| ipTotals_69_55_233 |
| ipTotals_69_55_234 |
| ipTotals_69_55_235 |
| ipTotals_69_55_236 |
| ipTotals_69_55_237 |
| ipTotals_69_55_238 |
| ipTotals_69_55_239 |
| portTotals_69_55_224 |
| portTotals_69_55_225 |
| portTotals_69_55_226 |
| portTotals_69_55_227 |
| portTotals_69_55_228 |
| portTotals_69_55_229 |
| portTotals_69_55_230 |
| portTotals_69_55_231 |
| portTotals_69_55_232 |
| portTotals_69_55_233 |
| portTotals_69_55_234 |
| portTotals_69_55_235 |
| portTotals_69_55_236 |
| portTotals_69_55_237 |
| portTotals_69_55_238 |
| portTotals_69_55_239 |
+---------------------------+
</pre>

So as you see we store each class-C block in its own table, for efficiency. Further, we store and organize data in 4 ways: "daily" tables and 15-minute granularity tables, and for each of those we track simple IP traffic and port-specific traffic. The daily tables contains 2 entries (one for each direction) for each IP for each day. For the current day, the row data is incremented as the day goes on.

<pre>mysql> describe dailyIpTotals_69_55_224;
+-----------+-------------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+-----------+-------------+------+-----+---------+-------+
| id | varchar(23) | | PRI | | |
| date | date | YES | | NULL | |
| ip | varchar(15) | YES | MUL | NULL | |
| direction | tinyint(1) | YES | | NULL | |
| octets | bigint(12) | YES | | NULL | |
| packets | int(11) | YES | | NULL | |
+-----------+-------------+------+-----+---------+-------+

mysql> select * from dailyIpTotals_69_55_224 limit 1\G
*************************** 1. row ***************************
id: 6955224194-20100917-1
date: 2010-09-17
ip: 69.55.224.194
direction: 1
octets: 8821
packets: 91
</pre>

The <tt>id</tt> is a unique identifier (key), <tt>direction</tt> indicates incoming or outgoing traffic (outbound = 2, inbound = 1), <tt>octets</tt> are the amount of traffic in kilobytes, and <tt>packets</tt> is the total number of packets.

The 15-minute table has similar information, but it's organized in 15 minute increments:

<pre>mysql> describe ipTotals_69_55_224;
+-----------+------------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+-----------+------------+------+-----+---------+-------+
| date | datetime | YES | | NULL | |
| ip | char(15) | YES | MUL | NULL | |
| direction | tinyint(1) | YES | | NULL | |
| octets | bigint(20) | YES | | NULL | |
| packets | int(11) | YES | | NULL | |
+-----------+------------+------+-----+---------+-------+

mysql> select * from ipTotals_69_55_224 limit 2\G
*************************** 1. row ***************************
date: 2010-01-11 19:30:00
ip: 69.55.224.13
direction: 1
octets: 288
packets: 6
*************************** 2. row ***************************
date: 2010-01-11 19:30:00
ip: 69.55.224.12
direction: 1
octets: 216
packets: 4</pre>

So for a given IP, there will be 192 rows in a given day: 4 rows per hour, *2 for 2 directions, *24 for 24hours in a day. Obviously this table is large which is why we broke it down into a daily table for quick, easy, daily-summary access.

That covers the simple traffic tabulation tables. We also track traffic by port:

<pre>mysql> describe dailyPortTotals_69_55_224;
+-----------+-------------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+-----------+-------------+------+-----+---------+-------+
| id | varchar(28) | | PRI | | |
| date | date | YES | | NULL | |
| ip | varchar(15) | YES | MUL | NULL | |
| direction | tinyint(1) | YES | | NULL | |
| protocol | smallint(3) | YES | | NULL | |
| port | int(11) | YES | | NULL | |
| octets | bigint(11) | YES | | NULL | |
| packets | int(11) | YES | | NULL | |
+-----------+-------------+------+-----+---------+-------+
8 rows in set (0.00 sec)

mysql> select * from dailyPortTotals_69_55_224 limit 1\G
*************************** 1. row ***************************
id: 695522496-20091218-1-6-23
date: 2009-12-18
ip: 69.55.224.96
direction: 1
protocol: 6
port: 23
octets: 1796
packets: 30

mysql> select * from portTotals_69_55_224 limit 1\G
*************************** 1. row ***************************
date: 2010-09-07 18:45:00
ip: 69.55.224.254
direction: 1
protocol: 6
port: 99999
octets: 144
packets: 3

</pre>

This is largely the same with 2 more additions: <tt>protocol</tt> (1=ICMP, 6=TCP, 17=UDP), and <tt>port</tt> which we set to 99999 if the traffic is return traffic and the port is above 1024. Obviously the potential for number of rows grows quickly when you consider the addition of port and protocol tracking per IP.

== Regular maintenance ==
*[[Routine_Maintenance#Adaptec_Controllers|Check RAID array]]
* archive data from database
archive_daily.pl 2012 09
This will archive data for the given year and month from the daily summary tables. Generally we want to have a year of history in the database.

archive_15min.pl 2012 09
This will archive data for the given year and month from the 15min-increment tables. Generally, we want to have 6 months of history in the database.

* if space becomes tight, move flow files and exported data to a backup server, both located in <tt>/usr/home/flowbin/archive</tt> and <tt>/usr/home/exported</tt>, respectively

== Slaving ==

If we were going to setup traffic database slaving (we don't do this anymore), perhaps cause the bwdb machine gets busy and it cannot handle traffic requests and netflow, here's how it's done:

On the traffic master:

GRANT REPLICATION SLAVE ON *.* TO 'repl'@'10.1.4.8' IDENTIFIED BY 'qERUG8wf';

in my.cnf:
<pre>bin-log
server-id=1
max_binlog_size=500M
expire_logs_days = 3</pre>

on slave:
in my.cnf:
<pre>server-id = 2
master-host = 10.1.4.203
master-user = repl
master-password = qERUG8wf
master-connect-retry=60
replicate-wild-do-table=traffic.daily%
max_relay_log_size=500M
expire_logs_days = 3

replicate-wild-do-table=traffic.%</pre>

on master:
touch /usr/home/working/.lock
(make sure processflows not running)

<pre>FLUSH TABLES WITH READ LOCK;
cd /usr/home/database/traffic
tar -czf mysql-traffic-snapshot.tgz ./daily*
(~1G)
SHOW MASTER STATUS;
+-----------------+-----------+--------------+------------------+
| File | Position | Binlog_Do_DB | Binlog_Ignore_DB |
+-----------------+-----------+--------------+------------------+
| bwdb-bin.000039 | 154432615 | | |
+-----------------+-----------+--------------+------------------+

(write down info)
UNLOCK TABLES;
scp mysql-traffic-snapshot.tgz 10.1.4.5:/mnt/data1/db/mysql/traffic/</pre>

on slave:
<pre>mkdir /mnt/data1/db/mysql/traffic
cd /mnt/data1/db/mysql/traffic/
tar xzvf mysql-traffic-snapshot.tgz
(restart mysql)
CHANGE MASTER TO MASTER_HOST='10.1.4.203',MASTER_USER='repl',MASTER_PASSWORD='qERUG8wf',MASTER_LOG_FILE='bwdb-bin.000059',MASTER_LOG_POS=482502186;
START SLAVE;</pre>

<pre>cd /usr/home/database/traffic
scp *</pre>

<pre>optimize table dailyPortTotals_69_55_224;
optimize table dailyPortTotals_69_55_225;
optimize table dailyPortTotals_69_55_226;
optimize table dailyPortTotals_69_55_227;
optimize table dailyPortTotals_69_55_228;
optimize table dailyPortTotals_69_55_229;
optimize table dailyPortTotals_69_55_230;
optimize table dailyPortTotals_69_55_231;
optimize table dailyPortTotals_69_55_232;
optimize table dailyPortTotals_69_55_233;
optimize table dailyPortTotals_69_55_234;
optimize table dailyPortTotals_69_55_235;
optimize table dailyPortTotals_69_55_236;
optimize table dailyPortTotals_69_55_237;
optimize table dailyPortTotals_69_55_238;
optimize table dailyPortTotals_69_55_239;</pre>

== Build ==

=== BIOS Config ===
disable quiet boot

set to last state after power loss

set date/time to GMT

enable serial console output (baud rate 115200)

=== Install OS ===

Install FreeBSD 8.3 amd64

* partition map:
<pre>/ 500m
swap 4096m
/var 256m
/tmp 256m
/usr ~</pre>

* edit /etc/make.conf
Castle:
<pre>echo "WITHOUT_X11=yes \
KERNCONF=bwdb \
BOOT_COMCONSOLE_SPEED=115200" >> /etc/make.conf</pre>

i2b:
<pre>echo "WITHOUT_X11=yes \
KERNCONF=bwdb2 \
BOOT_COMCONSOLE_SPEED=115200" >> /etc/make.conf</pre>

* add settings to /boot/loader.conf and /boot.config

<pre>echo "-Dh" >> /boot.config

echo 'console="comconsole,vidconsole" \
boot_multicons="YES" \
boot_serial="YES" \
comconsole_speed="115200"' >> /boot/loader.conf</pre>

* turn off all ttyv's except 0 and 1 in /etc/ttys
also turn on ttyu0, change type to vt100:
<pre>vi /etc/ttys

ttyv2 "/usr/libexec/getty Pc" cons25 off secure
ttyv3 "/usr/libexec/getty Pc" cons25 off secure
ttyv4 "/usr/libexec/getty Pc" cons25 off secure
ttyv5 "/usr/libexec/getty Pc" cons25 off secure
ttyv6 "/usr/libexec/getty Pc" cons25 off secure
ttyv7 "/usr/libexec/getty Pc" cons25 off secure
# Serial terminals
# The 'dialup' keyword identifies dialin lines to login, fingerd etc.
ttyu0 "/usr/libexec/getty std.9600" vt100 on secure

kill -1 1</pre>

on console server:
vi /etc/remote
(rename port to jail8 depending on where and which digi plugged into)
test serial console

* populate hosts
i2b:
<pre>echo "69.55.230.10 backup2" >> /etc/hosts
echo "69.55.230.11 backup1" >> /etc/hosts
echo "10.1.2.3 backup3" >> /etc/hosts</pre>

castle:
<pre>echo "10.1.4.3 backup2 backup2.johncompanies.com" >> /etc/hosts
echo "10.1.4.8 backup1 backup1.johncompanies.com" >> /etc/hosts
echo "10.1.4.4 mail mail.johncompanies.com" >> /etc/hosts
</pre>

* put key in authorized_keys on backup1 and backup2
cd
ssh-keygen -t dsa -b 1024
(default location, leave password blank)

castle:
cat /root/.ssh/id_dsa.pub | ssh backup1 'cat - >> /root/.ssh/authorized_keys'
cat /root/.ssh/id_dsa.pub | ssh backup2 'cat - >> /root/.ssh/authorized_keys'

i2b:
cat /root/.ssh/id_dsa.pub | ssh backup1 'cat - >> /root/.ssh/authorized_keys'
cat /root/.ssh/id_dsa.pub | ssh backup2 'cat - >> /root/.ssh/authorized_keys'
cat /root/.ssh/id_dsa.pub | ssh backup3 'cat - >> /root/.ssh/authorized_keys'

confirm that you can ssh to backup2 and backup1 (and backup3 if at i2b) without getting a login prompt

ssh backup1 hostname
ssh backup2 hostname

* edit root's path and login script:
vi /root/.cshrc

Change alias entries (add G):
<pre>alias la ls -aG
alias lf ls -FAG
alias ll ls -lAG
alias ls ls -AG
</pre>

and alter the prompt, set the following:
set prompt = "`/bin/hostname -s` %/# "

* install cvsup
cd /usr/ports/net/cvsup-without-gui
make install clean; rehash; mail -s 'cvs installed' support@johncompanies.com < /dev/null

* get latest sources for this release:
<pre>cd /usr/src
echo "*default host=cvsup4.freebsd.org\
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_8_3\
*default delete use-rel-suffix\
*default compress\
src-all" > sup

cvsup sup ; mail -s 'cvs sup done' support@johncompanies.com < /dev/null</pre>

* configure new kernel

cd /usr/src/sys/amd64/conf
scp backup2:/mnt/data4/build/freebsd/kern_config-bwdb-8.3-amd64 ./bwdb

Edit config and change name:
vi bwdb
ident bwdb

* build, install kernel and world

<pre>cd /boot

mv kernel kernel.GENERIC
cd kernel.GENERIC
cd /usr/src
make buildkernel installkernel

make buildworld ; mail -s 'buildworld done' support@johncompanies.com < /dev/null
(2450: 1:56min, supermicro: 59mins, 2950: 38mins)
make installworld
(2450: 3min, supermicro: 1min, 2950: :34)
mergemaster -i</pre>

* populate /etc/rc.conf with IPs and NFS settings

castle:
<pre>vi /etc/rc.conf

hostname="bwdb.johncompanies.com"
kern_securelevel_enable="NO"
portmap_enable="NO"
sendmail_enable="NO"
usbd_enable="YES"

xntpd_enable="YES"
nfs_client_enable="YES"
nfs_reserved_port_only="YES"
ifconfig_fxp0="inet 10.1.4.203 netmask 255.255.255.0"
ifconfig_em0="up promisc"
defaultrouter="10.1.4.1"
snmpd_enable="YES"

inetd_enable="YES"
inetd_flags="-wW -a 10.1.4.203"
fsck_y_enable="YES"
background_fsck="NO"
sshd_enable="YES"
ipfw_load="YES"</pre>

i2b:
<pre>vi /etc/rc.conf

hostname="bwdb2.johncompanies.com"
kern_securelevel_enable="NO"
portmap_enable="NO"
sendmail_enable="NO"
usbd_enable="YES"

xntpd_enable="YES"
nfs_client_enable="YES"
nfs_reserved_port_only="YES"
ifconfig_fxp0="inet 10.1.2.4 netmask 255.255.255.0"
ifconfig_em0="up promisc"
defaultrouter="10.1.2.1"
snmpd_enable="YES"

inetd_enable="YES"
inetd_flags="-wW -a 10.1.2.4"
fsck_y_enable="YES"
background_fsck="NO"
sshd_enable="YES"
ipfw_load="YES"</pre>

* reboot. Confirm new kernel is loaded

uname -a

* update ports:
<pre>cd /usr/ports
echo "*default host=cvsup4.FreeBSD.org\
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_8_3\
*default delete use-rel-suffix\
*default compress\
ports-all tag=." > sup

cvsup sup; mail -s 'cvs sup ports done' support@johncompanies.com < /dev/null</pre>

* Install raid mgmt tool

<pre>cd /usr/local/sbin
scp backup2:/d4/build/3ware/tw_cli-freebsd-x86_64-9.5.0.1.tgz .
tar xzf tw_cli-freebsd-x86_64-9.5.0.1.tgz
rm tw_cli-freebsd-x86_64-9.5.0.1.tgz
chmod 0700 tw_cli</pre>

Test:
./tw_cli info c0

Grab raid check script:
scp backup1:/usr/local/sbin/3wraidchk /usr/local/etc

Setup cronjob:
<pre>crontab -e
*/5 * * * * /usr/local/etc/3wraidchk</pre>

* install rsync from ports
cd /usr/ports/net/rsync
make install clean

choose default options

* install perl from ports
cd /usr/ports/lang/perl5.8
make install clean

choose default options

* install bb client

Compiling from source on AMD64 will not work. So, we use a linux-compiled version and rely on linux compat. Linux compat won't install on 8.x - libtool 2.4 need. So, instead we copy(ed) over linux:
rsync -aSHv --exclude=proc --exclude=sys 10.1.4.108:/usr/compat/linux/ /usr/compat/linux/

adduser

Output/response:

<pre>Username: bb
Full name: bb
Uid (Leave empty for default): 1984
Login group [bb]:
Login group is bb. Invite bb into other groups? []:
Login class [default]:
Shell (sh csh tcsh nologin) [sh]:
Home directory [/home/bb]:
Use password-based authentication? [yes]:
Use an empty password? (yes/no) [no]:
Use a random password? (yes/no) [no]: yes
Lock out the account after creation? [no]:
Username : bb
Password : <random>
Full Name : bb
Uid : 1984
Class :
Groups : bb
Home : /home/bb
Shell : /bin/sh
Locked : no
OK? (yes/no): yes</pre>

cd /usr/home/bb
scp backup2:/mnt/data4/build/bb/bb-freebsd_linuxcompat.tgz .
tar xzf bb-freebsd_linuxcompat.tgz

edit /home/bb/bbc1.9e-btf/etc/bb-hosts with something like:

echo "10.1.4.5 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh \
10.1.4.203 bwdb.johncompanies.com # ssh" > /home/bb/bbc1.9e-btf/etc/bb-hosts

Edit for machine name and private IP.

if this machine is at i2b:
echo "69.55.230.2 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh \
10.1.2.4 bwdb2.johncompanies.com # ssh" > /home/bb/bbc1.9e-btf/etc/bb-hosts

<pre>vi /home/bb/bbc1.9e-btf/ext/openfiles

MACHINE="bwdb,johncompanies,com" # HAS TO BE IN A,B,C FORM</pre>
Edit for machine name.

Have bb watch for flow-capture, mysql
<pre>cat >> /home/bb/bbc1.9e-btf/etc/bb-proctab
localhost: flow-capture :
localhost: mysqld :</pre>

<pre>cd /usr/home/bb/bbc1.9e-btf/etc
./bbchkcfg.sh
(y to questions)
./bbchkhosts.sh
(ignore ssh errors)
cd ../..
chown -R bb .
su bb
cd
cd bbc1.9e-btf
./runbb.sh start
more BBOUT
(look for errors)
exit</pre>

Put in script to start bb @ boot:
echo 'su - bb -c "cd /home/bb/bbc1.9e-btf; ./runbb.sh start"' > /usr/local/etc/rc.d/bb.sh
chmod +x /usr/local/etc/rc.d/bb.sh

If this is at i2b, punch a hole in the firewall to allow it to communicate with bb monitor:

ipfw add 00096 allow tcp from 66.181.18.0/27 to 69.55.230.2

* configure bb on mail
<pre>vi /usr/home/bb/bbsrc/bb1.9e-btf/etc/bb-hosts
10.1.4.203 bwdb.johncompanies.com # ssh

su bb
cd
bbsrc/bb/runbb.sh restart ; exit</pre>

* configure ntp server
Castle:
echo "server 10.1.4.1" > /etc/ntp.conf

I2b:
echo "server 10.1.2.1" > /etc/ntp.conf

<pre>/usr/sbin/ntpd -p /var/run/ntpd.pid
sleep 2; ntpq -p</pre>
(confirm it’s able to reach our time server)

<pre>echo '/usr/sbin/ntpd -c /etc/ntp.conf -p /var/run/ntpd.pid -f /var/db/ntpd.drift' > /usr/local/etc/rc.d/ntp.sh
chmod 0700 /usr/local/etc/rc.d/ntp.sh</pre>

* fwd and reverse lookups on ns1c
vr johncompanies.com
(edit the PTR too)

* setup backups
<pre>echo '#\!/bin/sh\
backupdir=/data/bwdb/current\
server=backup1\
\
## ENTRY /etc\
## ENTRY /usr/home/flowbin\
## ENTRY /usr/home/database' > /usr/local/etc/backup.config</pre>

Castle:
setup backup dirs:
ssh backup1 mkdir -p /data/bwdb/current
on backup1, add the system to
vi /usr/local/sbin/snapshot_rotate

I2b:
setup backup dirs:
ssh backup3 mkdir -p /data/bwdb/current
on backup3, add the system to
vi /usr/local/sbin/snapshot_archive

Copy over the backup script:
scp backup2:/d4/bin/freebsd8.x/rsync.backup /usr/local/etc/

Edit rsync.backup and change <tt>config</tt> var to point to correct config file location: <tt>/usr/local/etc/backup.config</tt>

<pre>crontab -e
5 0 * * * /usr/local/etc/rsync.backup</pre>

* make /root/logs
mkdir /root/logs

* edit sshd_config for security

<pre>vi /etc/ssh/sshd_config
ListenAddress 10.1.4.203
PermitRootLogin yes

kill -1 `cat /var/run/sshd.pid`</pre>

Edit for private IP.

* snmp

(Before doing this you may need to take down the firewall and also add to resolv.conf 69.43.143.41)
<pre>
cd /usr/ports/net-mgmt/net-snmp
make install clean
(defaults)

cat >> /etc/rc.conf
snmpd_enable="YES"
snmpd_flags="-a"
snmpd_conffile="/usr/local/share/snmp/snmpd.conf"
snmptrapd_enable="YES"
snmptrapd_flags="-a -p /var/run/snmptrapd.pid"

cat > /usr/local/share/snmp/snmpd.conf
rocommunity jcread 10.1.4.5
rocommunity jcread 10.1.4.202
</pre>

=== netflow ===

Install flow tools:
<pre>cd /usr/ports/net-mgmt/flow-tools
make install clean</pre>
Defaults.

mkdir /usr/home/flows

Flow start script:
echo "/usr/local/bin/flow-capture -w /usr/home/flows -S5 -N -2 0/10.1.4.203/4444" > /usr/local/etc/rc.d/flow-capture.sh

chmod 0700 /usr/local/etc/rc.d/flow-capture.sh

Edit for private IP.

Netgraph start script:
<pre>
cat > /usr/local/etc/rc.d/netgraph.sh

/usr/sbin/ngctl -f- <<-SEQ
mkpeer em0: netflow lower iface0
name em0:lower netflow
connect em0: netflow: upper out0
mkpeer netflow: ksocket export inet/dgram/udp
msg netflow:export connect inet/10.1.4.203:4444
SEQ

#/usr/sbin/ngctl -f- <<-SEQ
#shutdown netflow:
#SEQ

chmod 0700 /usr/local/etc/rc.d/netgraph.sh</pre>
Edit for private IP.

Confirm netflow is running after running scripts:
<pre>newbwdb /usr/ports/net-mgmt/flow-tools# /usr/sbin/ngctl
Available commands:
config get or set configuration of node at <path>
connect Connects hook <peerhook> of the node at <relpath> to <hook>
debug Get/set debugging verbosity level
dot Produce a GraphViz (.dot) of the entire netgraph.
help Show command summary or get more help on a specific command
list Show information about all nodes
mkpeer Create and connect a new node to the node at "path"
msg Send a netgraph control message to the node at "path"
name Assign name <name> to the node at <path>
read Read and execute commands from a file
rmhook Disconnect hook "hook" of the node at "path"
show Show information about the node at <path>
shutdown Shutdown the node at <path>
status Get human readable status information from the node at <path>
types Show information about all installed node types
write Send a data packet down the hook named by "hook".
quit Exit program
+ show netflow:
Name: netflow Type: netflow ID: 00000004 Num hooks: 3
Local hook Peer name Peer type Peer ID Peer hook
---------- --------- --------- ------- ---------
export <unnamed> ksocket 00000005 inet/dgram/udp
out0 em0 ether 00000001 upper
iface0 em0 ether 00000001 lower
+
</pre>

We notice that sometimes flow-capture is failing due to swap exhaustion (even after adding more swap). So we crontab flow-capture to restart (it's ok to start if it's already running, it just quits):

<pre>
crontab -e
#restart flow-capture
*/15 * * * * /usr/local/etc/rc.d/flow-capture.sh
</pre>

==== process flow tools ====

<pre>mkdir /usr/home/flowbin
mkdir /usr/home/working</pre>

Install modules:
<pre>cd /usr/ports/devel/p5-Date-Calc
make install clean
cd /usr/ports/mail/p5-Mail-Sendmail
make install clean</pre>

Queue script:
<pre>
cat > /usr/home/flowbin/queue.pl
#!/usr/bin/perl

use strict;

BEGIN {
push @INC, "/usr/home/flowbin";
}

use date;

my $flowbase = "/usr/home/flows";
#my $flowqueue = "/usr/home/queue";
my $flowqueue = "/usr/home/working";

my ($date, $time) = date::CurrentDateTime();

my $flowdir = mkFlowDir($date);
`mv $flowdir/ft-* $flowqueue`;

if (date::DateWindow($date, $time, $date, "00:00:00", 600)) {
my $newdate = date::AddDays($date, -1);
my $flowdir = mkFlowDir($newdate);
`mv $flowdir/ft-* $flowqueue`;
}

sub mkFlowDir {
my $date = shift;
$date =~ /([0-9]{4}-[0-9]{2})/;
my $yearmonth = $1;
return "$flowbase/$yearmonth/$date";
}
</pre>

Date.pm module:
<pre>
cat > /usr/home/flowbin/date.pm
#!/usr/local/bin/perl
#
# $Header: /usr/cvs/newgw/lib/date.pm,v 1.2 2003/11/24 17:06:02 glenn Exp $
#
# Copyright (c) 2001, 2002, 2003
# e-Monitoring Networks, Inc. All rights reserved.
#
#
#
# date.pl - Higher level functions written on top of Date::Calc

package date;

use strict;
use Date::Calc qw(:all);

sub DayDiff { #calculate the difference in days from two dates
my $date1 = shift;
my $date2 = shift;
my ($year1, $month1, $day1) = &DateToymd($date1);
my ($year2, $month2, $day2) = &DateToymd($date2);
my $diff = &Delta_Days($year1, $month1, $day1, $year2, $month2, $day2);
return $diff;
}

sub AddDays { #adds specified number of days to the supplied date
my $date = shift;
my $days = shift;
my ($year, $month, $day) = &DateToymd($date);
my ($nyear, $nmonth, $nday) = &Add_Delta_Days($year, $month, $day, $days);
my $ndate = &ymdToDate($nyear, $nmonth, $nday);
return $ndate;
}

sub AddHours { #adds specified number of hours to the supplied date and time
my $date = shift;
my $time = shift;
my $addhours = shift;
my $adddays = 0;
if (abs($addhours / 24) >= 1) {
$adddays = int($addhours / 24);
$addhours -= $adddays * 24;
}
my ($year, $month, $day) = &DateToymd($date);
my ($hour, $minute, $second) = &TimeTohms($time);
my ($ny, $nm, $nd, $nh, $nmin, $ns) = &Add_Delta_DHMS($year, $month, $day,
$hour, $minute, $second,
$adddays, $addhours, 0, 0);
my $ndate = &ymdToDate($ny, $nm, $nd);
my $ntime = &hmsToTime($nh, $nmin, $ns);
return $ndate, $ntime;
}

sub AddMinutes {
my $date = shift;
my $time = shift;
my $minutes = shift;
my ($year, $month, $day) = &DateToymd($date);
my ($hour, $minute, $second) = &TimeTohms($time);
my ($ny, $nm, $nd, $nh, $nmin, $ns) = &Add_Delta_DHMS($year, $month, $day,
$hour, $minute, $second,
0, 0, $minutes, 0);
my $ndate = &ymdToDate($ny, $nm, $nd);
my $ntime = &hmsToTime($nh, $nmin, $ns);
return $ndate, $ntime;
}

sub CurrentDateTime { #return the current date and time
my ($y, $m, $d, $h, $min, $s, $z, $z, $z) = &System_Clock;
my $date = &ymdToDate($y, $m, $d);
my $time = &hmsToTime($h, $min, $s);
return $date, $time;
}

sub Currentymd { #return the current year, month and day as separate variables
my ($y, $m, $d, $h, $min, $s, $z, $z, $z) = &System_Clock;
return $y, $m, $d;
}

sub DateToymd { #takes a date and returns year, month, day as individual values
my $date = shift;
if ($date =~ /([0-9]{4})-([0-9]{2})-([0-9]{2})/) {
my $day = $3;
my $month = $2;
my $year = $1;
return $year, $month, $day;
}
return undef;
}

sub TimeTohms { #takes a time and return hours minutes and seconds as individual values
my $time = shift;
if ($time =~ /([0-9]{1,2}):([0-9]{1,2}):([0-9]{1,2})/) {
my $hour = $1;
my $minute = $2;
my $second = $3;
if ($hour !~ /[0-9]{2}/) { $hour = "0$hour"; }
if ($minute !~ /[0-9]{2}/) { $minute = "0$minute"; }
if ($second !~ /[0-9]{2}/) { $second = "0$second"; }
return $hour, $minute, $second;
}
return undef;
}

sub ymdToDate { #takes year, month, day and assembles them into our date format
my $year = shift;
my $month = shift;
my $day = shift;
if (defined($year) && defined($month) && defined ($day)) {
$month = sprintf("%02d", $month);
$day = sprintf("%02d", $day);
return "$year-$month-$day";
}
return undef;
}

sub hmsToTime { #takes hour minute and second and assembles them into our time format
my $hour = shift;
my $minute = shift;
my $second = shift;
if (defined($hour) && defined($minute) && defined ($second)) {
if ($hour !~ /[0-9]{2}/) { $hour = "0$hour"; }
if ($minute !~ /[0-9]{2}/) { $minute = "0$minute"; }
if ($second !~ /[0-9]{2}/) { $second = "0$second"; }
return sprintf ("%02d:%02d:%02d", $hour, $minute, $second);
}
return undef;
}

sub CompareDates { #compares two date and time pairs
my $date1 = shift;
my $time1 = shift;
my $date2 = shift;
my $time2 = shift;

my ($year1, $month1, $day1) = &DateToymd($date1);
my ($hour1, $minute1, $second1) = &TimeTohms($time1);
my ($year2, $month2, $day2) = &DateToymd($date2);
my ($hour2, $minute2, $second2) = &TimeTohms($time2);

# &debug("$year1, $month1, $day1, $year2, $month2, $day2");
my $days = &Delta_Days($year1, $month1, $day1, $year2, $month2, $day2);
if ($days > 0) { return 1;}
if ($days < 0) { return -1;}
if ($days == 0) { #same day, compare times
my $seconds1 = $second1 + (60 * $minute1) + (3600 * $hour1);
my $seconds2 = $second2 + (60 * $minute2) + (3600 * $hour2);
if ($seconds1 < $seconds2) { return 1;}
if ($seconds1 > $seconds2) { return -1;}
if ($seconds1 == $seconds2) { return 0;}
}
return undef;
}

sub DateWindow { #compares two date time pairs to see if they are < X seconds apart
my $date1 = shift;
my $time1 = shift;
my $date2 = shift;
my $time2 = shift;
my $window = shift;

my ($year1, $month1, $day1) = &DateToymd($date1);
my ($hour1, $minute1, $second1) = &TimeTohms($time1);
my ($year2, $month2, $day2) = &DateToymd($date2);
my ($hour2, $minute2, $second2) = &TimeTohms($time2);

my ($day, $hour, $minute, $second) =
&Delta_DHMS($year1, $month1, $day1, $hour1, $minute1, $second1,
$year2, $month2, $day2, $hour2, $minute2, $second2);
$minute *= 60;
$hour *= 3600;
$day *= 86400;
my $total = $second + $minute + $hour + $day;
if (abs($total) < $window) {
return 1;
}
return 0;
}

sub CheckDateOrder { #takes three dates/times, returns true if they are in chronological order
my $date1 = shift;
my $time1 = shift;
my $date2 = shift;
my $time2 = shift;
my $date3 = shift;
my $time3 = shift;
if (&CompareDates($date1, $time1, $date2, $time2) == -1) {
return 0;
}
if (&CompareDates($date2, $time2, $date3, $time3) == -1) {
return 0;
}
return 1;
}

sub EpochSeconds { #calculates number of seconds since the epoch for the given date/time
my $date = shift;
my $time = shift;
my ($year, $month, $day) = &DateToymd($date);
my ($hour, $minute, $second) = &TimeTohms($time);
my ($d, $h, $m, $s) = &Delta_DHMS(1970, 1, 1, 0, 0, 0,
$year, $month, $day, $hour, $minute, $second);
my $seconds = $s + (60 * $m) + (3600 * $h) + (86400 * $d);
return $seconds;
}

sub SecondsToDateTime { #converts seconds since epoch to date/time
my $seconds = shift;
my $days = int($seconds / 86400);
$seconds -= $days * 86400;
my $hours = int($seconds / 3600);
$seconds -= $hours * 3600;
my $minutes = int($seconds / 60);
$seconds -= $minutes * 60;
my ($year, $month, $day, $hour, $minute, $second) =
&Add_Delta_DHMS(1970, 1, 1, 0, 0, 0, $days, $hours, $minutes, $seconds);
$month = sprintf("%02d", $month);
$day = sprintf("%02d", $day);
$hour = sprintf("%02d", $hour);
$minute = sprintf("%02d", $minute);
$second = sprintf("%02d", $second);
return "$year-$month-$day", "$hour:$minute:$second";
}

sub DateToDayName {
my $date = shift;
my ($year, $month, $day) = &DateToymd($date);
my $name = &Day_of_Week_to_Text(&Day_of_Week($year, $month, $day));
$name =~ /^[A-Za-z]{3}/;
$name = $&;
return $name;
}

sub ValiDate {
return @_;

}

sub CheckBusinessDay { # checks to see if date is business day. 1=yes, 0=no
my $date = shift;
my ($year, $month, $day) = &DateToymd($date);
if (Day_of_Week($year,$month,$day) < 6) { return 1; }
else { return 0; }
}

1; #don't remove this line
</pre>

chmod 0700 /usr/home/flowbin/queue.pl

Setup cronjob:
<pre>crontab -e
#move flow data into the queue
1,16,31,46 * * * * /usr/home/flowbin/queue.pl</pre>

==== flow processing: i2b ====
<pre>cat > /usr/home/flowbin/processflows-sql.pl
#!/usr/bin/perl

#use strict;
#$debug=1;
#$dry=1;

my $log = '/usr/home/flowbin/discards.log';

use Data::Dumper;

BEGIN {
push @INC, "/usr/home/flowbin";
}

#my $queuedir = "/usr/home/queue";
my $queuedir = "/usr/home/working";
my $archivedir = "/usr/home/archive";
my $sqldir = "/usr/home/sql";
my $sqldirworking = "/usr/home/sql/tmp";

unless ($dry) {
if (-e "$queuedir/.lock") {
open(FILE, "$queuedir/.lock");
my $pid = <FILE>;
chomp($pid);
close(FILE);
if (kill(0, $pid)) {
#another process is using the queue, bail out
exit(0);
}
else {
#dead lock file, remove it
`rm $queuedir/.lock`;
}
}
open(FILE, "> $queuedir/.lock");
print FILE "$$\n";
close(FILE);
}

opendir(DIR, $queuedir);
my @files = readdir(DIR);
closedir(DIR);

foreach my $file (sort @files) {
unless($file =~ /^\./) {
$file =~ /([0-9]{4}-[0-9]{2}-[0-9]{2})\.([0-9]{2})([0-9]{2})([0-9]{2})/;
my $date = "$1 $2:$3:$4";
my $outfile = "$1-$2:$3.sql";
unless (open (SQL, "+> $sqldirworking/$outfile")) { die "cant open $sqldirworking/$outfile"; }
my $condensedDate = $1;
$condensedDate =~ s/-//g;
my $iptotal = {};
my $protototal = {};
my $porttotal = {};

&debug("started file $file at ");
&debug(`date`);
&debug("getting raw flow data (flow-print)");
`cat $queuedir/$file | /usr/local/bin/flow-print -f 5 > /usr/home/working/tmp-$file`;
&debug("aggregating data at ");
&debug(`date`);
unless (open(DATA, "/usr/home/working/tmp-$file")) { die "can't open: $!"; }
LOOP: while (my $line = readline DATA) {
my @d = split /[\s]+/, $line;
if ($d[0] ne '' && $d[0] ne 'Start') {
my $addr = 0;
my $port = 0;

#Start End Sif SrcIPaddress SrcP DIf DstIPaddress DstP P Fl Pkts Octets
#0 1 2 3 4 5 6 7 8 9 10 11
#|
# outbound = 2, inbound = 1

my (@src_ip) = split '\.', $d[3];
my (@dst_ip) = split '\.', $d[6];

if ($src_ip[0] == 69 && $src_ip[1] == 55 && ($src_ip[2] == 229 || $src_ip[2] == 231)) { # for i2b
$d[2] = 2;
# hack for outbound bulk traffic counted 2x
#if ($src_ip[2] == 231) { $d[11] /= 2; $d[10] /= 2; }
}
# note- this is where we filter out IPs only found at i2b
elsif ($dst_ip[0] == 69 && $dst_ip[1] == 55 && ($dst_ip[2] == 229 || $dst_ip[2] == 231)) { # for i2b
$d[2] = 1;
}
else { next LOOP; }

if ($d[2] == 2) {
$addr = $d[3];
# if the dst-port is low, store that
if ($d[7] <= 1024) { $port = $d[7]; }
# if the src-port is low, store that
elsif ($d[4] <= 1024) { $port = $d[4]; }
else { $port = 99999; }
}
elsif ($d[2] == 1) {
$addr = $d[6];
# if the dst-port is high, assume its return traffic, try to store src-port if low
if ($d[7] > 1024) {
if ($d[4] <= 1024) { $port = $d[4]; }
else { $port = 99999; }
} else {
$port = $d[7];
}
} else {
next LOOP;
}

my (@ip) = split '\.', $addr;
unless ($ip[0] == 69) { next LOOP; }
unless ($ip[1] == 55) { next LOOP; }
unless ($ip[2] == 229 || $ip[2] == 231) { next LOOP; }

my $classC = "$ip[0]_$ip[1]_$ip[2]";

# IP dir
# if ($d[10] < 2147483647) { $iptotal->{$classC}->{$addr}->{$d[2]}->{'pktTotal'} += $d[10]; }
# if ($d[11] < 2147483647) { $iptotal->{$classC}->{$addr}->{$d[2]}->{'octetTotal'} += $d[11]; }
#
# if ($d[10] < 2147483647) { $protototal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{'pktTotal'} += $d[10]; }
# if ($d[11] < 2147483647) { $protototal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{'octetTotal'} += $d[11]; }
#
# if ($d[10] < 2147483647) { $porttotal->{$classC}->{$addr}->{$d[2]}->{$port}->{'pktTotal'} += $d[10]; }
# if ($d[11] < 2147483647) { $porttotal->{$classC}->{$addr}->{$d[2]}->{$port}->{'octetTotal'} += $d[11]; }
$iptotal->{$classC}->{$addr}->{$d[2]}->{'pktTotal'} += $d[10];
$iptotal->{$classC}->{$addr}->{$d[2]}->{'octetTotal'} += $d[11];

$protototal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{'pktTotal'} += $d[10];
$protototal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{'octetTotal'} += $d[11];

$porttotal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{$port}->{'pktTotal'} += $d[10];
$porttotal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{$port}->{'octetTotal'} += $d[11];
}
}
close(DATA);
`rm /usr/home/working/tmp-$file`;
&debug("processing ip totals at ");
&debug(`date`);
foreach my $classC (keys(%{$iptotal})) {
my @values;
foreach my $ip (keys(%{$iptotal->{$classC}})) {
foreach my $dir (keys(%{$iptotal->{$classC}->{$ip}})) {
my $octets = $iptotal->{$classC}->{$ip}->{$dir}->{'octetTotal'};
my $packets = $iptotal->{$classC}->{$ip}->{$dir}->{'pktTotal'};
# $packets = $packets > 2147483647 ? 0 : $packets;
if ($octets > 2147483647) {
my $ddir = $dir==1 ? 'in' : 'out';
#print SQL "$date $ip $ddir $octets\n";
# $octets = 0;
}
# dailyIpTotals.id = ip(no .'s)-yyyymmdd-direction
my $id = "$ip-$condensedDate-$dir";
$id =~ s/\.//g;
push @values, "('$date', '$ip', $dir, $octets, $packets)";
my $sql = "insert into dailyIpTotals_$classC values ('$id', '$date', '$ip', $dir, $octets, $packets) ON DUPLICATE KEY UPDATE octets=octets+$octets, packets=packets+$packets";
print "$sql\n" if $dry;
print SQL "$sql;\n";
# $db->query("insert into ipTotals values ('$date', '$ip', $dir, $octets, $packets)");
}
}

# break inserts into 100 records at a time
&debug("inserting $#values +1 values");
while ($#values > 0) {
my $sql = "insert into ipTotals_$classC values ";
my $max_index = $#values > 100 ? 100 : $#values;
for (my $i=0; $i<=$max_index; $i++) {
$sql .= shift @values;
$sql .= ',';
}
chop $sql;
print "$sql\n" if $dry;
print SQL "$sql;\n";
}
}

# &debug("processing protocol totals at ");
# &debug(`date`);
# foreach my $classC (keys(%{$protototal})) {
# $db->query("lock tables dailyProtoTotals_$classC write") unless $dry;
# my @values;
# foreach my $ip (keys(%{$protototal->{$classC}})) {
# foreach my $dir (keys(%{$protototal->{$classC}->{$ip}})) {
# foreach my $proto (keys(%{$protototal->{$classC}->{$ip}->{$dir}})) {
# my $octets = $protototal->{$classC}->{$ip}->{$dir}->{$proto}->{'octetTotal'};
# my $packets = $protototal->{$classC}->{$ip}->{$dir}->{$proto}->{'pktTotal'};
# # $octets = $octets > 2147483647 ? 0 : $octets;
# # $packets = $packets > 2147483647 ? 0 : $packets;
# # dailyProtoTotals.id = ip(no .'s)-yyyymmdd-direction-proto
# my $id = "$ip-$condensedDate-$dir-$proto";
# $id =~ s/\.//g;
# push @values, "('$date', '$ip', $dir, $proto, $octets, $packets)";
# my $sql = "insert into dailyProtoTotals_$classC values ('$id', '$date', '$ip', $dir, $proto, $octets, $packets) ON DUPLICATE KEY UPDATE octets=octets+$octets, packets=packets+$packets";
# print "$sql\n" if $dry;
# $db->query($sql) unless $dry;
# # $db->query("insert into protoTotals values ('$date', '$ip', $dir, $proto, $octets, $packets)");
# }
# }
# }
# $db->query("unlock tables") unless $dry;
# my $sql = "insert into protoTotals_$classC values ";
# $sql .= join ',', @values;
# $db->query("lock tables protoTotals_$classC write") unless $dry;
# print "$sql\n" if $dry;
# $db->query($sql) unless $dry;
# $db->query("unlock tables") unless $dry;
# }

&debug("processing port totals at ");
&debug(`date`);
foreach my $classC (keys(%{$porttotal})) {
my @values;
foreach my $ip (keys(%{$porttotal->{$classC}})) {
foreach my $dir (keys(%{$porttotal->{$classC}->{$ip}})) {
foreach my $proto (keys(%{$porttotal->{$classC}->{$ip}->{$dir}})) {
foreach my $port (keys(%{$porttotal->{$classC}->{$ip}->{$dir}->{$proto}})) {
my $octets = $porttotal->{$classC}->{$ip}->{$dir}->{$proto}->{$port}->{'octetTotal'};
my $packets = $porttotal->{$classC}->{$ip}->{$dir}->{$proto}->{$port}->{'pktTotal'};
# $octets = $octets > 2147483647 ? 0 : $octets;
# $packets = $packets > 2147483647 ? 0 : $packets;

# dailyPortTotals.id = ip(no .'s)-yyyymmdd-direction-protocol-port
my $id = "$ip-$condensedDate-$dir-$proto-$port";
$id =~ s/\.//g;
push @values, "('$date', '$ip', $dir, $proto, $port, $octets, $packets)";
my $sql = "insert into dailyPortTotals_$classC values ('$id', '$date', '$ip', $dir, $proto, $port, $octets, $packets) ON DUPLICATE KEY UPDATE octets=octets+$octets, packets=packets+$packets";
print "$sql\n" if $dry;
print SQL "$sql;\n";

# $db->query("insert into portTotals values ('$date', '$ip', $dir, $port, $octets, $packets)");
}
}
}
}

# break inserts into 100 records at a time
&debug("inserting $#values +1 values");
while ($#values > 0) {
my $sql = "insert into portTotals_$classC values ";
my $max_index = $#values > 100 ? 100 : $#values;
for (my $i=0; $i<=$max_index; $i++) {
$sql .= shift @values;
$sql .= ',';
}
chop $sql;
print "$sql\n" if $dry;
print SQL "$sql;\n";
}
}

# 12 1 8 1 1= 23
# dailyIpTotals.id = ip(no .'s)-yyyymmdd-direction
# 12 1 8 1 1 3=26
# dailyProtoTotals.id = ip(no .'s)-yyyymmdd-direction-proto
# 12 1 8 1 1 5=28
# dailyPortTotals.id = ip(no .'s)-yyyymmdd-direction-port
#print "finished at ";
#print `date`;
`mv $queuedir/$file $archivedir` unless $dry;
close(SQL);
`bzip2 $sqldirworking/$outfile`;
`mv $sqldirworking/$outfile.bz2 $sqldir/`;
}
}
`rm $queuedir/.lock` unless $dry;

sub debug {
my $message = shift;
if ($debug) {
print "$message\n";
}
}

# var full during ft-v05.2005-03-28.084500-0800 and
# 2005-02-24 69.55.226

# all port/daily totals before 2005-04-07
</pre>

This script sends the sql files to the traffic server for processing:
<pre>cat > /usr/home/flowbin/sendsql.pl
#!/usr/bin/perl

#use strict;
#$debug=1;
#$dry=1;

my $remote = "69.55.233.199";
my $sqldir = "/usr/home/sql";
my $archive = "/usr/home/archive";
my $sqldirremote = "/data/bwdb2/pending/";
my @err;
unless ($dry) {
if (-e "$sqldir/.lock") {
open(FILE, "$sqldir/.lock");
my $pid = <FILE>;
chomp($pid);
close(FILE);
if (kill(0, $pid)) {
#another process is using the queue, bail out
exit(0);
}
else {
#dead lock file, remove it
`rm $sqldir/.lock`;
}
}
open(FILE, "> $sqldir/.lock");
print FILE "$$\n";
close(FILE);
}

opendir(DIR, $sqldir);
my @files = readdir(DIR);
closedir(DIR);

foreach my $file (sort @files) {
next unless $file =~ /bz2$/;

my $r = `scp -Cq $sqldir/$file $remote:$sqldirremote 2>&1`;
# print "scp $sqldir/$file $remote:$sqldirremote";
unless ($?==0) {
push @err, "scp -Cq $sqldir/$file $remote:$sqldirremote ($r)";
}
else {
`mv $sqldir/$file $archive`;
`ssh $remote mv $sqldirremote/$file $sqldirremote/${file}.done`;
}
}

`rm $sqldir/.lock` unless $dry;

if (@err) {
email_support('bwdb2: sendsql.pl error',join "\n", @err);
}

sub email_support {
my $subj=shift;
my $body=shift;
use Mail::Sendmail;

# prepare message
my %mail = (
To => 'support@johncompanies.com,dave@johncompanies.com',
From => 'support@johncompanies.com',
Subject => $subj,
Message => $body,
smtp => 'mail.johncompanies.com',
);
sendmail(%mail) || warn "Error: $Mail::Sendmail::error";
}

sub debug {
my $message = shift;
if ($debug) {
print "$message\n";
}
}

# var full during ft-v05.2005-03-28.084500-0800 and
# 2005-02-24 69.55.226

# all port/daily totals before 2005-04-07
</pre>

<pre>crontab -e
#process flows
2,17,32,47 * * * * /usr/home/flowbin/processflows-sql.pl
#move sql commands to traffic db
8,23,38,53 * * * * /usr/home/flowbin/sendsql.pl
</pre>

==== flow processing: castle ====
<pre>
cat > /usr/home/flowbin/processflows.pl

#!/usr/bin/perl

#use strict;
#$debug=1;
#$dry=1;

my $log = '/usr/home/flowbin/discards.log';

use Data::Dumper;

BEGIN {
push @INC, "/usr/home/flowbin";
}

use db;

#my $queuedir = "/usr/home/queue";
my $queuedir = "/usr/home/working";
my $archivedir = "/usr/home/archive";

unless ($dry) {
if (-e "$queuedir/.lock") {
open(FILE, "$queuedir/.lock");
my $pid = <FILE>;
chomp($pid);
close(FILE);
if (kill(0, $pid)) {
#another process is using the queue, bail out
exit(0);
}
else {
#dead lock file, remove it
`rm $queuedir/.lock`;
}
}
open(FILE, "> $queuedir/.lock");
print FILE "$$\n";
close(FILE);
}

my $db = db->new();
$db->connect('traffic', '', 'root', '5over3') || die $db->{'error'};

opendir(DIR, $queuedir);
my @files = readdir(DIR);
closedir(DIR);

foreach my $file (sort @files) {
unless($file =~ /^\./) {
$file =~ /([0-9]{4}-[0-9]{2}-[0-9]{2})\.([0-9]{2})([0-9]{2})([0-9]{2})/;
my $date = "$1 $2:$3:$4";
my $condensedDate = $1;
$condensedDate =~ s/-//g;
my $iptotal = {};
my $protototal = {};
my $porttotal = {};

&debug("started file $file at ");
&debug(`date`);
&debug("getting raw flow data (flow-print)");
`cat $queuedir/$file | /usr/local/bin/flow-print -f 5 > /usr/home/working/tmp-$file`;
&debug("aggregating data at ");
&debug(`date`);
unless (open(DATA, "/usr/home/working/tmp-$file")) { die "can't open: $!"; }
LOOP: while (my $line = readline DATA) {
my @d = split /[\s]+/, $line;
if ($d[0] ne '' && $d[0] ne 'Start') {
my $addr = 0;
my $port = 0;

#Start End Sif SrcIPaddress SrcP DIf DstIPaddress DstP P Fl Pkts Octets
#0 1 2 3 4 5 6 7 8 9 10 11
#|
# outbound = 2, inbound = 1

my (@src_ip) = split '\.', $d[3];
my (@dst_ip) = split '\.', $d[6];

if ($src_ip[0] == 69 && $src_ip[1] == 55 &&
$src_ip[2] >= 224 && $src_ip[2] <= 239 &&
$src_ip[2] != 229 && $src_ip[2] != 231) { # for castle
# if ($src_ip[0] == 69 && $src_ip[1] == 55 && $src_ip[2] == 229) { # for i2b
$d[2] = 2;
# hack for outbound bulk traffic counted 2x
if ($dst_ip[2] == 234) { $d[11] /= 2; $d[10] /= 2; }
}
elsif ($dst_ip[0] == 69 && $dst_ip[1] == 55 &&
$dst_ip[2] >= 224 && $dst_ip[2] <= 239 &&
$dst_ip[2] != 229 && $dst_ip[2] != 231) { # for castle
# elsif ($dst_ip[0] == 69 && $dst_ip[1] == 55 && $dst_ip[2] == 229) { # for i2b
$d[2] = 1;
}
else { next LOOP; }

if ($d[2] == 2) {
$addr = $d[3];
# if the dst-port is low, store that
if ($d[7] <= 1024) { $port = $d[7]; }
# if the src-port is low, store that
elsif ($d[4] <= 1024) { $port = $d[4]; }
else { $port = 99999; }
}
elsif ($d[2] == 1) {
$addr = $d[6];
# if the dst-port is high, assume its return traffic, try to store src-port if low
if ($d[7] > 1024) {
if ($d[4] <= 1024) { $port = $d[4]; }
else { $port = 99999; }
} else {
$port = $d[7];
}
} else {
next LOOP;
}

my (@ip) = split '\.', $addr;
unless ($ip[0] == 69) { next LOOP; }
unless ($ip[1] == 55) { next LOOP; }
unless ($ip[2] >= 224 && $ip[2] <= 239 && $ip[2] != 229 && $ip[2] != 231) { next LOOP; }
# unless ($ip[2] == 229) { next LOOP; }

my $classC = "$ip[0]_$ip[1]_$ip[2]";

# IP dir
# if ($d[10] < 2147483647) { $iptotal->{$classC}->{$addr}->{$d[2]}->{'pktTotal'} += $d[10]; }
# if ($d[11] < 2147483647) { $iptotal->{$classC}->{$addr}->{$d[2]}->{'octetTotal'} += $d[11]; }
#
# if ($d[10] < 2147483647) { $protototal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{'pktTotal'} += $d[10]; }
# if ($d[11] < 2147483647) { $protototal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{'octetTotal'} += $d[11]; }
#
# if ($d[10] < 2147483647) { $porttotal->{$classC}->{$addr}->{$d[2]}->{$port}->{'pktTotal'} += $d[10]; }
# if ($d[11] < 2147483647) { $porttotal->{$classC}->{$addr}->{$d[2]}->{$port}->{'octetTotal'} += $d[11]; }
$iptotal->{$classC}->{$addr}->{$d[2]}->{'pktTotal'} += $d[10];
$iptotal->{$classC}->{$addr}->{$d[2]}->{'octetTotal'} += $d[11];

$protototal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{'pktTotal'} += $d[10];
$protototal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{'octetTotal'} += $d[11];

$porttotal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{$port}->{'pktTotal'} += $d[10];
$porttotal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{$port}->{'octetTotal'} += $d[11];
}
}
close(DATA);
`rm /usr/home/working/tmp-$file`;
&debug("processing ip totals at ");
&debug(`date`);
foreach my $classC (keys(%{$iptotal})) {
$db->query("lock tables dailyIpTotals_$classC write") unless $dry;
my @values;
foreach my $ip (keys(%{$iptotal->{$classC}})) {
foreach my $dir (keys(%{$iptotal->{$classC}->{$ip}})) {
my $octets = $iptotal->{$classC}->{$ip}->{$dir}->{'octetTotal'};
my $packets = $iptotal->{$classC}->{$ip}->{$dir}->{'pktTotal'};
# $packets = $packets > 2147483647 ? 0 : $packets;
if ($octets > 2147483647) {
my $ddir = $dir==1 ? 'in' : 'out';
`echo "$date $ip $ddir $octets\n" >> $log`;
# $octets = 0;
}
# dailyIpTotals.id = ip(no .'s)-yyyymmdd-direction
my $id = "$ip-$condensedDate-$dir";
$id =~ s/\.//g;
push @values, "('$date', '$ip', $dir, $octets, $packets)";
my $sql = "insert into dailyIpTotals_$classC values ('$id', '$date', '$ip', $dir, $octets, $packets) ON DUPLICATE KEY UPDATE octets=octets+$octets, packets=packets+$packets";
print "$sql\n" if $dry;
$db->query($sql) unless $dry;
# $db->query("insert into ipTotals values ('$date', '$ip', $dir, $octets, $packets)");
}
}
$db->query("unlock tables") unless $dry;

$db->query("lock tables ipTotals_$classC write") unless $dry;
# break inserts into 100 records at a time
&debug("inserting $#values +1 values");
while ($#values > 0) {
my $sql = "insert into ipTotals_$classC values ";
my $max_index = $#values > 100 ? 100 : $#values;
for (my $i=0; $i<=$max_index; $i++) {
$sql .= shift @values;
$sql .= ',';
}
chop $sql;
print "$sql\n" if $dry;
$db->query($sql) unless $dry;
}
$db->query("unlock tables") unless $dry;
}

sleep 20;
# &debug("processing protocol totals at ");
# &debug(`date`);
# foreach my $classC (keys(%{$protototal})) {
# $db->query("lock tables dailyProtoTotals_$classC write") unless $dry;
# my @values;
# foreach my $ip (keys(%{$protototal->{$classC}})) {
# foreach my $dir (keys(%{$protototal->{$classC}->{$ip}})) {
# foreach my $proto (keys(%{$protototal->{$classC}->{$ip}->{$dir}})) {
# my $octets = $protototal->{$classC}->{$ip}->{$dir}->{$proto}->{'octetTotal'};
# my $packets = $protototal->{$classC}->{$ip}->{$dir}->{$proto}->{'pktTotal'};
# # $octets = $octets > 2147483647 ? 0 : $octets;
# # $packets = $packets > 2147483647 ? 0 : $packets;
# # dailyProtoTotals.id = ip(no .'s)-yyyymmdd-direction-proto
# my $id = "$ip-$condensedDate-$dir-$proto";
# $id =~ s/\.//g;
# push @values, "('$date', '$ip', $dir, $proto, $octets, $packets)";
# my $sql = "insert into dailyProtoTotals_$classC values ('$id', '$date', '$ip', $dir, $proto, $octets, $packets) ON DUPLICATE KEY UPDATE octets=octets+$octets, packets=packets+$packets";
# print "$sql\n" if $dry;
# $db->query($sql) unless $dry;
# # $db->query("insert into protoTotals values ('$date', '$ip', $dir, $proto, $octets, $packets)");
# }
# }
# }
# $db->query("unlock tables") unless $dry;
# my $sql = "insert into protoTotals_$classC values ";
# $sql .= join ',', @values;
# $db->query("lock tables protoTotals_$classC write") unless $dry;
# print "$sql\n" if $dry;
# $db->query($sql) unless $dry;
# $db->query("unlock tables") unless $dry;
# }

&debug("processing port totals at ");
&debug(`date`);
foreach my $classC (keys(%{$porttotal})) {
$db->query("lock tables dailyPortTotals_$classC write") unless $dry;
my @values;
foreach my $ip (keys(%{$porttotal->{$classC}})) {
foreach my $dir (keys(%{$porttotal->{$classC}->{$ip}})) {
foreach my $proto (keys(%{$porttotal->{$classC}->{$ip}->{$dir}})) {
foreach my $port (keys(%{$porttotal->{$classC}->{$ip}->{$dir}->{$proto}})) {
my $octets = $porttotal->{$classC}->{$ip}->{$dir}->{$proto}->{$port}->{'octetTotal'};
my $packets = $porttotal->{$classC}->{$ip}->{$dir}->{$proto}->{$port}->{'pktTotal'};
# $octets = $octets > 2147483647 ? 0 : $octets;
# $packets = $packets > 2147483647 ? 0 : $packets;

# dailyPortTotals.id = ip(no .'s)-yyyymmdd-direction-protocol-port
my $id = "$ip-$condensedDate-$dir-$proto-$port";
$id =~ s/\.//g;
push @values, "('$date', '$ip', $dir, $proto, $port, $octets, $packets)";
my $sql = "insert into dailyPortTotals_$classC values ('$id', '$date', '$ip', $dir, $proto, $port, $octets, $packets) ON DUPLICATE KEY UPDATE octets=octets+$octets, packets=packets+$packets";
print "$sql\n" if $dry;
$db->query($sql) unless $dry;
# $db->query("insert into portTotals values ('$date', '$ip', $dir, $port, $octets, $packets)");
}
}
}
}
$db->query("unlock tables") unless $dry;

$db->query("lock tables portTotals_$classC write") unless $dry;
# break inserts into 100 records at a time
&debug("inserting $#values +1 values");
while ($#values > 0) {
my $sql = "insert into portTotals_$classC values ";
my $max_index = $#values > 100 ? 100 : $#values;
for (my $i=0; $i<=$max_index; $i++) {
$sql .= shift @values;
$sql .= ',';
}
chop $sql;
print "$sql\n" if $dry;
$db->query($sql) unless $dry;
}
$db->query("unlock tables") unless $dry;
sleep 10;
}

# 12 1 8 1 1= 23
# dailyIpTotals.id = ip(no .'s)-yyyymmdd-direction
# 12 1 8 1 1 3=26
# dailyProtoTotals.id = ip(no .'s)-yyyymmdd-direction-proto
# 12 1 8 1 1 5=28
# dailyPortTotals.id = ip(no .'s)-yyyymmdd-direction-port
#print "finished at ";
#print `date`;
`mv $queuedir/$file $archivedir` unless $dry;
}
}
`rm $queuedir/.lock` unless $dry;

sub debug {
my $message = shift;
if ($debug) {
print "$message\n";
}
}

# var full during ft-v05.2005-03-28.084500-0800 and
# 2005-02-24 69.55.226

# all port/daily totals before 2005-04-07
</pre>

<pre>
cat > /usr/home/flowbin/db.pm

#!/usr/bin/perl
#
# $Header: /usr/cvs/newgw/lib/db.pm,v 1.4 2003/06/05 18:20:01 glenn Exp $
#
# Copyright (c) 2003
# e-Monitoring Networks, Inc. All rights reserved.
#
#
package db;

use strict;
use DBI;

sub new {
my $class = shift;
my $self = {};

$self->{'debug'} = 0;
bless $self, $class;

return $self;
}

sub connect {
my $self = shift;
my $dbname = shift;
my $dbhost = shift;
my $dbuser = shift;
my $dbpass = shift;

my $host = '';
if (defined($dbhost)) {
$host = ";host=$dbhost";
}

eval {
$self->debug("connecting to: DBI:mysql:database=$dbname;$host", 1);
$self->{'dbh'} = DBI->connect("DBI:mysql:database=$dbname;$host", $dbuser, $dbpass);
};
if ($self->{'dbh'}) {
return 1;
}
$self->{'error'} = "Error connecting to database $@";
$self->debug("Error connecting to database $@");
return 0;
}

sub query {
my $self = shift;
my $query = shift;

$self->debug($query, 1);
my $sth;
eval {
$sth = $self->{'dbh'}->prepare($query);
};
unless ($sth) {
$self->{'error'} = "error preparing query $@";
$self->debug("error preparing query $@");
return undef;
}
my $qty;
eval {
$qty = $sth->execute;
};
unless ($qty) {
$self->{'error'} = "error executing query $@";
warn "error executing query $@ $query";
return undef;
}
$self->debug("returning $qty, $sth from query", 6);
return ($qty, $sth);
}

sub disconnect {
my $self = shift;

$self->{'dbh'}->disconnect;
return 0;
}

sub debug {
my $self = shift;
my $msg = shift;
my $level = shift || 0;

if ($level < $self->{'debug'}) {
print "$msg\n";
}
return 0;
}
1;
</pre>

mkdir /usr/home/archive
mkdir -p /usr/home/sql/tmp

<pre>crontab -e
#process flows
2,17,32,47 * * * * /usr/home/flowbin/processflows.pl</pre>

==== setup traffic db ====
* Install mysql:
<pre>cd /usr/ports/databases/mysql50-server
make install clean</pre>

cat >> /etc/rc.conf
mysql_enable="YES"

Move db data dir:
/usr/local/etc/rc.d/mysql-server stop
mkdir /usr/home/database/
mv /var/db/mysql/* /usr/home/database/
chown -R mysql:mysql /usr/home/database

Edit database location in startup script:
vi /usr/local/etc/rc.d/mysql-server
# : ${mysql_dbdir="/var/db/mysql"}
: ${mysql_dbdir="/usr/home/database"}

/usr/local/etc/rc.d/mysql-server start

* Install mysql perl database modules:
<pre>
cd /usr/ports/databases/p5-DBI
make install clean
cd /usr/ports/databases/p5-DBD-mysql50
make install clean
(no to SSL support)
</pre>

* Setting up database
<pre>
rehash
/usr/local/etc/rc.d/mysql-server start
mysql -u root
create database traffic;
grant all on *.* to root@localhost identified by '5over3';
grant all on traffic.* to jc@10.1.4.5 identified by '2gMKY3Wt';

</pre>

If this was a new server we'd setup new tables. See [[#mysql_2|mysql]] for how those tables would be setup.

We are assuming here we are moving data from an existing db, here's how that's done (from the current traffic db):
rsync -av --progress /usr/home/database/traffic/ 10.1.4.203:/usr/home/database/traffic/

When you're ready to do the cutover, shut down mysql on both hosts and do one last sync.

==== process flows from bwdb2 ====
On traffic database server (bwdb):

<pre>crontab -e
#import sql from bwdb2
10,25,40,55 * * * * /usr/home/flowbin/processsql.pl</pre>

Add access to mysql:
<pre>mysql -u root -p
grant all on traffic.* to bwdb2@localhost identified by 's1lver4d';
</pre>

<pre>cat > /usr/home/flowbin/processsql.pl

#!/usr/bin/perl

#use strict;
#$debug=1;
#$dry=1;

my $sqldir = "/usr/home/bwdb2/pending";
my $mysql = '/usr/local/bin/mysql';
my @err;
unless ($dry) {
if (-e "$sqldir/.lock") {
open(FILE, "$sqldir/.lock");
my $pid = <FILE>;
chomp($pid);
close(FILE);
if (kill(0, $pid)) {
#another process is using the queue, bail out
exit(0);
}
else {
#dead lock file, remove it
`rm $sqldir/.lock`;
}
}
open(FILE, "> $sqldir/.lock");
print FILE "$$\n";
close(FILE);
}

opendir(DIR, $sqldir);
my @files = readdir(DIR);
closedir(DIR);

foreach my $file (sort @files) {
next unless $file =~ /done$/;
my $r = `bzcat $sqldir/$file | $mysql -u bwdb2 -ps1lver4d traffic`;
unless ($?==0) {
push @err, "bzcat $sqldir/$file | $mysql -u bwdb2 -pxxxxx traffic ($r)";
}
else {
`rm $sqldir/$file`;
}
}

`rm $sqldir/.lock` unless $dry;

if (@err) {
email_support('bwdb: processsql.pl error',join "\n", @err);
}

sub email_support {
my $subj=shift;
my $body=shift;
use Mail::Sendmail;

# prepare message
my %mail = (
To => 'dave@johncompanies.com',
From => 'support@johncompanies.com',
Subject => $subj,
Message => $body,
smtp => 'mail.johncompanies.com',
);
sendmail(%mail) || warn "Error: $Mail::Sendmail::error";
}

sub debug {
my $message = shift;
if ($debug) {
print "$message\n";
}
}
</pre>

chmod 0700 /usr/home/flowbin/processsql.pl

Make sure bwdb is reachable from the outside only to bwdb2:

On nat, add to <tt>/etc/ipnat.rules</tt>
<pre># bwdb
bimap fxp0 10.1.4.203/32 -> 69.55.233.199/32</pre>

Reload:
ipnat -C -F -f /etc/ipnat.rules

Setup firewall rule on firewall:
ipfw add 00094 allow ip from 66.181.18.5 to 69.55.233.199 22
ipfw add 00094 deny ip from any to 69.55.233.199

Setup firewall on bwdb to restrict access now that it's nat'd:
<pre>
cat >> /usr/local/etc/rc.d/boot.sh
ipfw add 1 allow tcp from any to any established
ipfw add 2 allow ip from 10.1.4.0/24,66.181.18.5,69.55.233.195 to me 22
ipfw add 3 allow ip from 10.1.4.5 to me 3306
ipfw add 4 allow ip from 69.55.225.225 53 to me
ipfw add 5 allow ip from 69.55.230.2 25 to me
ipfw add 6 allow ip from me to me 4444
ipfw add 7 allow icmp from any to me
ipfw add 8 allow udp from 10.1.4.203 to 10.1.4.203 dst-port 4444
ipfw add 9 allow udp from 10.1.4.5 to me 161
ipfw add 100 deny ip from any to me
</pre>

chmod 0700 /usr/local/etc/rc.d/boot.sh

From bwdb2, add ssh key:
cat /root/.ssh/id_dsa.pub | ssh 69.55.233.199 'cat - >> /root/.ssh/authorized_keys'

Confirm no password access:
ssh 69.55.233.199 hostname

= bwdb2 =
== Summary ==
This machine tracks and stores network traffic (netflow) at i2b. It is our means to monitor customer bandwidth usage.

* Location: i2b, cab6
* OS: FreeBSD 6.4 x86
* Networking: Priv IP: 10.1.2.4 There are 2 onboard nic's, one of which is the "listener"
* Hardware: Custom 2U. Single power supply.
* Drives: two 150 GB (2 x 150GB) RAID1 arrays running on a 3ware 7006 RAID card.

== Services Provided ==
* netflow
* bigbrother

== netflow ==

The main function of this server is to run netflow on an eth device in promiscuous mode so as to hear everything happening on the port (wherein all network traffic is mirrored to that port via the cisco swith). Every 15min, it creates a flow file under <tt>/usr/home/flows/</tt> (organized by date). The flow file contains all traffic data for a 15min increment of time.

A cronjob moves that flow file (or files if there are multiple due to some delay)
1,16,31,46 * * * * /usr/home/flowbin/queue.pl

into a processing queue:
<tt>/usr/home/working</tt>

Then a separate file processes whatever flow files it finds there, and builds sql files ready for insertion into the traffic database:
2,17,32,47 * * * * /usr/home/flowbin/processflows-sql.pl

Then yet another process copies the sql files to the traffic database server for processing and insertion into the mysql database:
8,23,38,53 * * * * /usr/home/flowbin/sendsql.pl

== Regular maintenance ==
*[[Routine_Maintenance#Adaptec_Controllers|Check RAID array]]

* if space becomes tight, move sql files and flow files to backup server, both located in <tt>/usr/home/flowbin/archive</tt>

= firewall (newgateway) =

== Summary ==

This machine is the primary (only) firewall for the entire network at castle.

* Location: castle, cab 3-8
* OS: FreeBSD 4.11 x86
* Networking: Priv IP: 10.1.4.223, Pub IPs: 69.55.233.164 (external), 69.55.233.156 (internal). It has 3 network connections (2 onboard, 1 PCI) connecting to the external, internal and private networks. If you're looking at the back of the server, the internal-network-facing nic is on the right (em1), and the external-facing-network (3750) is on the left (em0).
* Hardware: 6 SCSI SCA drive bays (2 columns of 3, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: 36 GB (2 x 36GB) RAID1 array running on an Adaptec 2120S PCI RAID card.

== Services Provided ==
* firewall (ipfw)
* snmp
* bigbrother

== Firewall Rule Configuration ==

See [[FreeBSD_Reference#Firewall_Rule_Configuration|Firewall Rule Configuration]] for more discussion on how to actually manipulate firewall rules.

== Disaster Recovery ==

If there is ever an outage with the firewall, the old firewall "gate" is located just below and is running with the proper network configuration, but with no firewall rules in place (to facilitate good throughput). Have castle move the cable on the left on the current firewall to the left port in the old firewall and the right cable to the right port.

Here's what you need to put in /etc/rc.conf to get a firewall going (as far as routes and IPs)

<pre>hostname="newgateway.johncompanies.com"
firewall_script="/etc/firewall.sh"
firewall_enable="NO"
sendmail_enable="NONE"
sshd_enable="YES"
inetd_enable="NO"
xntpd_enable="YES"
snmpd_enable="YES"
#snmpd_flags="-as -p /var/run/snmpd.pid"
#ipnat_enable="YES"
#ipnat_rules="/etc/ipnat.rules"
gateway_enable="YES"

defaultrouter="69.55.233.161"

ifconfig_xl0="inet 10.1.4.223 netmask 255.255.255.0"
ifconfig_em0="inet 69.55.233.164 netmask 255.255.255.248"

#
# Original JohnCompanies 69.55.224.0/20
#
ifconfig_em1="inet 69.55.233.156 netmask 255.255.255.248"

static_routes="route1 route2 route3 route4 route5 route6 route7 route8 route9 route10 route11 route1
2 route13 route14 route15 route16 route17 route18"

route_route1="-net 69.55.224.0 69.55.233.153"
route_route2="-net 69.55.225.0 69.55.233.153"
route_route3="-net 69.55.226.0 69.55.233.153"
route_route4="-net 69.55.227.0 69.55.233.153"
route_route5="-net 69.55.228.0 69.55.233.153"
route_route6="-net 69.55.229.0 69.55.233.153"
route_route7="-net 69.55.230.0 69.55.233.153"
route_route8="-net 69.55.231.0 69.55.233.153"
route_route9="-net 69.55.232.0 69.55.233.153"
route_route10="-net 69.55.233.0 69.55.233.153"
route_route11="-net 69.55.234.0 69.55.233.153"
route_route12="-net 69.55.235.0 69.55.233.153"
route_route13="-net 69.55.236.0 69.55.233.153"
route_route14="-net 69.55.237.0 69.55.233.153"
route_route15="-net 69.55.238.0 69.55.233.153"
route_route16="-net 69.55.239.0 69.55.233.153"
route_route17="-net 10.1.5.0 10.1.4.2"
route_route18="-net 10.1.6.0 10.1.4.2"

#In case of 3750 failure:
#defaultrouter="69.43.128.81"
#ifconfig_em0="inet 69.43.129.84 netmask 255.255.255.248"

#bind .1's here:
#ifconfig_em1="inet 69.55.224.1 netmask 255.255.255.0"
#ifconfig_em1_alias0="inet 69.55.225.1 netmask 255.255.255.0"
#ifconfig_em1_alias1="inet 69.55.226.1 netmask 255.255.255.0"
#ifconfig_em1_alias2="inet 69.55.227.1 netmask 255.255.255.0"
#ifconfig_em1_alias3="inet 69.55.228.1 netmask 255.255.255.0"
#ifconfig_em1_alias4="inet 69.55.229.1 netmask 255.255.255.0"
#ifconfig_em1_alias5="inet 69.55.230.1 netmask 255.255.255.0"
#ifconfig_em1_alias6="inet 69.55.231.1 netmask 255.255.255.0"
#ifconfig_em1_alias7="inet 69.55.232.1 netmask 255.255.255.0"
#ifconfig_em1_alias8="inet 69.55.233.1 netmask 255.255.255.0"
#ifconfig_em1_alias9="inet 69.55.234.1 netmask 255.255.255.0"
#ifconfig_em1_alias10="inet 69.55.235.1 netmask 255.255.255.0"
#ifconfig_em1_alias11="inet 69.55.236.1 netmask 255.255.255.0"
#ifconfig_em1_alias12="inet 69.55.237.1 netmask 255.255.255.0"
#ifconfig_em1_alias13="inet 69.55.238.1 netmask 255.255.255.0"
#ifconfig_em1_alias14="inet 69.55.239.1 netmask 255.255.255.0"

#bulk:
# reassign 69.55.231.1 to the int iface on the firewall
# set the DG on the firewall to 69.43.138.9
# set the ext firewall IP to 69.43.138.12, NM: 255.255.255.248</pre>

== Cronjobs ==
1 0 * * * /usr/local/etc/rsync.backup
Backup to backup1

0 0 1 * * /sbin/ipfw zero
0 0 1 * * /sbin/ipfw del 3 4 5 17331
Reset counters and remove pipe rules on the 1st of the month. Pay attention when setting up a rule as 3 4 5 (that's not a temporary traffic cap).

Inside <tt>/etc/daily.local</tt> you will see a call to <tt>/etc/makepiperules.pl</tt>
This script will create <tt>/etc/firewall.sh</tt> which contains all the firewall and pipe rules in place at the time the script was run.

== DOS attacks ==

See [[FreeBSD_Reference#Handling_a_DoS_attack|Handling a DoS attack]] regarding how to handle a DOS attack.

Theres a background process (running from user shell) that monitors the firewall for incoming UDP DoS attacks. When it notices packets above a certain level it will
# enter a rule that allows all UDP to go through
# send an emergency email to support and indicating an attack is in progress
# send an email to castle (nocstaff@castleaccess.com and jcsupport@castleaccess.com) telling them to investigate and put up a null if warranted
# wait for a couple minutes to see if the attack subsides- if so it will remove the pass-all UDP rule, if not it will repeat the process from #1
This file lives under /usr/home/user/doswatch.pl
To run:
cd /usr/home/user
./doswatch.pl &

To kill;
fg
^C

It writes its findings to /usr/home/user/doswatch.log

= backup1 =

== Summary ==

This machine acts as the primary backup location for all VPS-based customers. No customer directly accesses this server to perform their backups. We also store cancelled customers on this server.

* Location: castle, cab 3-8
* OS: Ubuntu 8.04.1 server x86
* Networking: Priv IP: 10.1.4.8, Pub IP: 69.55.230.11 (firewalled from all but JC infrastructure @ i2b)
* Hardware: 6 SATA drive bays (2 columns of 3, drive 0 top left, drive 1 just below) all hot-swap. Single power supply.
* Drives: 4.5 TB (6 x 1TB) RAID5 array running on a 3ware 9650SE-8LPML (8-port) card

== Services provided ==
* backup via rsync
* mysql - traffic data
* nfs server - for backups
* snmp client - for big brother
* bigbrother client

== Usage and Notes ==
* all data is stored under /data
* virtually all jc infrastructure, and all VPS machines are setup to mount to backup1 via nfs (mountpoint: <tt>/backup1</tt>), and they all have their ssh keys setup to allow passwordless rsync's
* each virt or jail backs up each evening to backup1. Each server has it's own directory (named for the server). Under those directories are 7 daily snapshots (0-6)
* at the time of writing, the mysql server running here is replicating from (slave to) the mysql instance on bwdb. Requests for bandwidth data usage for customers (coming from management, account manager, and accounting scripts running on mail) all direct towards the database "traffic" running on this server.
* cancelled customer systems are compressed and stored under <tt>/data/deprecated</tt>
* archived bwdb2 flow files are stored under <tt>/data/bwdb2</tt>
* critical files from backup2 are stored under <tt>/data/backup2</tt>

== Cronjobs ==
<pre>
00 5 * * * /usr/local/sbin/backupwatch.pl 2>&1 > /dev/null
35 5 * * * /usr/local/sbin/usage_check; /usr/local/sbin/snapshot_archive; /usr/local/sbin/snapshot_rotate /data/backuplog.log</pre>
this runs daily the scripts to report on how much disk space each customer system occupies and how long their backups took. Then it rotates backups for each system, removing the oldest backup. It will email support@johncompanies.com at it’s conclusion. This email can be deleted, however note when it begins to take significantly longer to complete, ie runs past 2200 pm – this usually indicates a problem on the backup server.

<pre>10,25,40,55 * * * * /usr/local/sbin/processsql.pl
</pre>
this processes prepared sql command files sent from/by bwdb2 (@ i2b) and imports them into the traffic database.
<pre>0 0 * * * /usr/local/sbin/3wraidchk
</pre>
checks the health of the RAID array

== Regular maintenance ==
*[[Routine_Maintenance#Free_up_space_on_backup1|Remove old backups]]
*[[Routine_Maintenance#3ware|Check on auto-verify]]

== build ==

<pre>Setup raid5 with a boot vol of 12G 5.45tb
12G boot
4664 GB

Install ubuntu 8.04

Swap 4G

Don’t format data drive

http://www.unixgods.org/~tilo/linux_larger_2TB.html

parted /dev/sdb
print
mklabel gpt
print

#Disk /dev/sdb: 4987GB
#Sector size (logical/physical): 512B/512B
#Partition Table: gpt

#Number Start End Size File system Name Flags

mkpart primary ext3 0 4987GB
print

#Disk /dev/sdb: 5987GB
#Sector size (logical/physical): 512B/512B
#Partition Table: gpt

#Number Start End Size File system Name Flags
# 1 17.4kB 4987GB 4987GB primary

quit

mkfs.ext3 /dev/sdb1
#mke2fs 1.40.8 (13-Mar-2008)
#Filesystem label=
#OS type: Linux
#Block size=4096 (log=2)
#Fragment size=4096 (log=2)
#304390144 inodes, 1217544183 blocks
#60877209 blocks (5.00%) reserved for the super user
#First data block=0
#Maximum filesystem blocks=0
#37157 block groups
#32768 blocks per group, 32768 fragments per group
#8192 inodes per group
#Superblock backups stored on blocks:
# 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
# 4096000, 7962624, 11239424, 20480000, 23887872, 71663616, 78675968,
# 102400000, 214990848, 512000000, 550731776, 644972544
#
#Writing inode tables: 967/37157

mkdir /data

#root@backup1:~# df -h
#Filesystem Size Used Avail Use% Mounted on
#/dev/sda2 8.3G 540M 7.3G 7% /
#varrun 1013M 40K 1013M 1% /var/run
#varlock 1013M 0 1013M 0% /var/lock
#udev 1013M 56K 1013M 1% /dev
#devshm 1013M 0 1013M 0% /dev/shm
#/dev/sdb1 4.5T 192M 4.3T 1% /data

apt-get update
apt-get upgrade
apt-get install snmp snmpd ntp nfs-kernel-server

echo "\"\e[5~\": history-search-backward" >> ~/.inputrc
echo "\"\e[6~\": history-search-forward" >> ~/.inputrc

vi /etc/ntp.conf
server 10.1.4.5

scp root@10.1.4.3:/root/.ssh/authorized_keys /root/.ssh/
cd /root/
ssh-keygen -t dsa
echo "10.1.4.3 backup2" >> /etc/hosts

cat .ssh/id_dsa.pub | ssh backup2 'cat - >> /root/.ssh/authorized_keys'

ssh backup2

vi /root/.bashrc
export PS1="[\u@\h \w]# "
alias h='history'
alias vi='vim'
alias j='jobs'
export PS1="[\u@\h \w]# "
alias dr='screen -dr'
export EDITOR=vim
export GREP_OPTIONS='--color=auto'
export HISTFILESIZE=1000

source /root/.bashrc

echo "# ttyS0 - getty
#
# This service maintains a getty on ttyS0 from the point the system is
# started until it is shut down again.

start on runlevel 2
start on runlevel 3
start on runlevel 4
start on runlevel 5

stop on runlevel 0
stop on runlevel 1
stop on runlevel 6

respawn
exec /sbin/getty 38400 ttyS0" > /etc/event.d/ttyS0

vi /boot/grub/menu.lst

serial --unit=0 --speed=38400 --word=8 --parity=no --stop=1
terminal --timeout=15 serial console

append to kernel lines:
console=tty0 console=ttyS0,38400n8

show menu:
#hiddenmenu

echo 'rocommunity jcread 10.1.4.5
rocommunity jcread 10.1.4.3
agentaddress 10.1.4.8:161' > /etc/snmp/snmpd.conf

# to see which iface it is, on backup2:

snmpwalk -v 1 -c jcread 10.1.4.8 interface

echo "bb:x:1984:1984:Big Brother:/home/bb:/bin/bash" >> /etc/passwd

echo "bb:x:1984:" >> /etc/group

pwconv

mkdir /home/bb
chown bb.bb /home/bb

cd ~bb
scp backup2:/mnt/data4/build/bb/bb-linux.tar .

tar xf bb-linux.tar

cd /home/bb/bbc1.9e-btf/etc

echo "10.1.4.5 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh" > /home/bb/bbc1.9e-btf/etc/bb-hosts
echo "10.1.4.8 backup1.johncompanies.com # ssh" >> /home/bb/bbc1.9e-btf/etc/bb-hosts

echo "/:90:95
/var:90:95
/data:85:99" > /home/bb/bbc1.9e-btf/etc/bb-dftab

vi /home/bb/bbc1.9e-btf/bin/bb-disk.sh
(remove all | SORT xxxx)

chmod +r /var/log/messages

./bbchkcfg.sh
#(y to questions)
./bbchkhosts.sh
#(ignore ssh errors)
cd ../..
chown -R bb .
su bb
cd
cd bbc1.9e-btf/src

#make; make install
cd ..
./runbb.sh start
more BBOUT
(look for errors)
exit

vi /etc/rc.local
su - bb -c "cd /home/bb/bbc1.9e-btf; ./runbb.sh start"

echo '/data 10.1.4.0/24(rw, no_root_squash,async,no_subtree_check)' >> /etc/exports

/etc/init.d/nfs-kernel-server restart

echo 'chmod o+r /var/log/messages' >> /etc/cron.weekly/sysklogd

echo '10.1.4.8 backup1' >> /etc/hosts
echo '/dev/sdb1 /data ext3 rw,noatime 0 0' >> /etc/fstab

to install digi drivers:

wget http://ftp1.digi.com/support/driver/40002086_n.tgz
apt-get install linux-image-2.6.24-19-server
apt-get install linux-source-2.6.24 (not needed?)
apt-get install linux-headers-2.6.24-19-server
apt-get install make
apt-get install gcc
apt-get install g++
apt-get install libncurses5-dev
apt-get install expect
apt-get install libdbi-perl libdate-calc-perl libdbd-mysql-perl

cd /usr/src; ln -s linux-headers-2.6.24-19-server linux
./configure
make all
make install
make postinstall

/usr/bin/dgrp_cfg_node -v -v init el 65.116.11.2 8

apt-get install mysql

mkdir /data/mysql
chown mysql:mysql /data/mysql
/etc/init.d/mysql stop
mv /var/lib/mysql/* /data/mysql/
mv /data/mysql/ib_* /var/lib/mysql/
vi /etc/mysql/my.cnf
(change datadir to /data/mysql)
vi /etc/apparmor.d/usr.sbin.mysqld
add:
/data/mysql/ r,
/data/mysql/** rwk,
Comment out:
# /var/lib/mysql/ r,
# /var/lib/mysql/** rwk,

/etc/init.d/apparmor restart
/etc/init.d/mysql start

tw_cli /c0/u0 set ignoreECC=on
tw_cli /c0/u0 set storsave=balance
tw_cli /c0/u0 set cache=on

0 0 * * * /usr/local/sbin/3wraidchk
</pre>

= backup2 =

== Summary ==

This machine is used for archiving data and is a backup server for colo customers. It was the former primary backup location for all VPS-based customers before backup1 was installed. Only dedicated customers directly accesses this server to perform their backups. NOTE: power button is broken, so the reset button (paper clip) was rewired to be the power button.

* Location: castle, cab 3-7
* OS: FreeBSD 6.1 x86
* Networking: Priv IP: 10.1.4.3, Pub IP: 69.55.230.10 (firewalled from all but JC infrastructure @ i2b)
* Hardware: 16 IDE drive bays (4 columns of 4, drive 0-0 top left, drive 0-1 just to the right TODO) all hot-swap. Triple power supply.
* Drives:
**3ware 7500-8:
***200 GB JBOD (1 x 200G) labeled 0-0
***500 GB RAID5 (3 x 250G) 0-1 thru 0-3
***700 GB RAID5 (4 x 250G) 0-4 thru 0-7
**3ware 7500-8:
***700 GB RAID5 (4 x 250G) 1-0 thru 1-3
***700 GB RAID5 (4 x 250G) 1-4 thru 1-7

All drives MUST be western digital IDE drives. Other brands will not fit.

In case of an outage, nfs will hang on all connected servers until the nfs service returns. If you can't get backup2 back online, you can get nfs running elsewhere and fake backup2's MAC's: priv: 00:0e:0c:59:c1:a6, pub: 00:07:e9:5b:c6:45

To configure:
ifconfig fxp0 link 00:90:27:f9:0a:d9

== Services provided ==
* backup via rsync and nfs
* samba
* nfs
* snmp
* bigbrother

== Usage ==
* all data is stored under 4 mount points, corresponding to the 4 large RAID5 arrays: <tt>/mnt/data1 /mnt/data2 /mnt/data3 /mnt/data4</tt>
* iso images provided for customers wanting to mount an ISO as a CDROM via the IPKVM are provided via samba on this server. Images live under <tt>/mnt/data2/iso</tt>
* this used to be our primary backup server so you will see old backups from virt and jails around- missing customer data though, just the machine's data
* this server serves as an archive for exported db data from bwdb and old flow files.
* isys backs up here
* customers are nfs-moutned under /mnt/data3/customers as file-backed md devices
* in <tt>/mnt/data4</tt> there are lots of useful things used for building our vps servers, customer servers, and management scripts:
** <tt>/bin</tt>: the master repository of scripts and custom binaries we use on jails and virts. Each night every virt and jail rsync's what's in here to update the local files. So any global updates to scripts would need to be made here (or will be overwritten with what's in here)
** <tt>/build</tt>: files we use for setting up big brother, 3ware cli and scripts for colo's, vzcp customized setup files and so on
** <tt>/vzrpms</tt>: contains the OS templates for many-to-most of the OS's we offer on vz systems

== Cronjobs ==
* backs itself up nightly to nfs-mounted backup1 (mountpoint: <tt>/backup2</tt>)

== Regular maintenance ==
*[[Routine_Maintenance#3ware|Check on health]]

= backup3 =
== Summary ==
This machine is used for archiving data, is a backup server for colo customers, runs a samba server to make available iso's to the IPKVMs, and allows us to connect to the digi serial multiplexer at i2b. Only dedicated customers directly accesses this server to perform their backups.

* Location: i2b, cab 6
* OS: Ubuntu 10.04.1 server amd64
* Networking: Priv IP: 10.1.2.3, Pub IPs: 69.55.229.4 AND 69.55.231.2
* Hardware: 16 drive SATA bays (4 columns of 4, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: 5 TB (6 x 1TB) RAID5 array running on an Areca Technology Corp. ARC-1160 16-Port

== Services provided ==
* backup via rsync and nfs
* samba
* nfs
* digi realport
* snmp
* bigbrother

== Usage ==
* all data is stored under /data
* iso images provided for customers wanting to mount an ISO as a CDROM via the IPKVM are provided via samba on this server. Images live under <tt>/data/iso</tt>
* this server serves as an archive for exported db data from bwdb and old flow files.
* inftrastructure machines at i2b back up here
* customers are nfs-moutned under /data/customers as file-backed loopback devices

== management scripts ==
* mkbackups

== Cronjobs ==
0 0 * * * /usr/local/sbin/arecaraidchk
RAID checks

35 4 * * * /usr/local/sbin/snapshot_archive
Rotate daily snapshots for infrastructure machine backups

== Regular maintenance ==
*[[Routine_Maintenance#Areca|Check on RAID health]]

== Build ==

=== BIOS Config ===
disable quiet boot

set to last state after power loss

set date/time to GMT

enable serial console output (baud rate 115200)

=== Install OS ===
<pre>Ubuntu 10.04.1 amd64 (couldn't get 12.04 to load cause the H/W was incompat)
10G / ext3
2G swap
~ /data ext4

Install packages:
openssh
samba</pre>

=== DNS and private IP ===

echo "nameserver 69.55.225.225" >> /etc/resolv.conf

Add a 2nd IP to eth0 and setup priv net
<pre>vi /etc/network/interfaces

auto eth0
iface eth0 inet static
address 69.55.229.4
netmask 255.255.255.0
network 69.55.229.0
broadcast 69.55.229.255
gateway 69.55.229.1
# dns-* options are implemented by the resolvconf package, if installed
dns-nameservers 69.55.229.3 66.181.0.2
dns-search johncompanies.com

auto eth0:1
iface eth0:1 inet static
address 69.55.231.2
netmask 255.255.255.0
network 69.55.231.0
broadcast 69.55.231.255

auto eth1
iface eth1 inet static
address 10.1.2.3
netmask 255.255.255.0
network 10.1.2.0
broadcast 10.1.2.255

</pre>

=== Install packages ===
<pre>apt-get update
apt-get upgrade
apt-get install gcc
apt-get install libssl-dev
apt-get install libncurses5-dev
apt-get install cu
apt-get install unzip
apt-get install snmp snmpd ntp nfs-kernel-server</pre>

=== tweak grub, enable serial ===

<pre>vi /etc/default/grub
#GRUB_HIDDEN_TIMEOUT=0
GRUB_CMDLINE_LINUX_DEFAULT="max_loop=64"
GRUB_CMDLINE_LINUX="console=tty0 console=ttyS0"
update-grub</pre>

<pre>echo "start on stopped rc RUNLEVEL=[2345]
stop on runlevel [!2345]
respawn
exec /sbin/getty -L ttyS0 38400 vt102" > /etc/init/ttyS0.conf</pre>

=== install realport (digi) driver ===

give the digi an ip with DgIpServ.exe

<pre>cd /usr/src/
wget ftp://ftp1.digi.com/support/beta/linux/dgrp/dgrp-1.9.tgz
tar xzf dgrp-1.9.tgz
cd dgrp-1.9/
./configure
make
make install
make postinstall
update-rc.d dgrp_daemon defaults</pre>

configure ports:
dgrp_cfg_node init el 10.1.2.10 16

try connecting with:
cu -l /dev/ttyel00 -s 38400

=== shell, ntp, ssh key, hosts ===

Shell autocompletion search:
<pre>echo "\"\e[5~\": history-search-backward" >> ~/.inputrc
echo "\"\e[6~\": history-search-forward" >> ~/.inputrc</pre>

Setup ntp:
<pre>vi /etc/ntp.conf
server 10.1.2.1
server ntp.ubuntu.com</pre>

Generate ssh keys:
<pre>cd /root/
ssh-keygen -t dsa</pre>
Defaults, no password

Setup hosts:
<pre>echo "69.55.230.10 backup2" >> /etc/hosts
echo "69.55.230.11 backup1" >> /etc/hosts
echo "10.1.2.4 bwdb2" >> /etc/hosts
echo "10.1.2.3 backup3" >> /etc/hosts</pre>

Copy keys to servers where we need passwordless login:
<pre>cat .ssh/id_dsa.pub | ssh backup2 'cat - >> /root/.ssh/authorized_keys'
cat .ssh/id_dsa.pub | ssh backup1 'cat - >> /root/.ssh/authorized_keys'</pre>

Setup shell:
<pre>vi /root/.bashrc
(add to bottom)
alias h='history'
alias vi='vim'
alias j='jobs'
export PS1="[\u@\h \w]# "
alias dr='screen -dr'
export EDITOR=vim
export GREP_OPTIONS='--color=auto'
export HISTFILESIZE=1000

alias tip-switch-p20='cu -l ttyel00 -s 9600'
alias tip-switch-p21='cu -l ttyel15 -s 9600'
alias tip-switch-p22='cu -l ttyel14 -s 9600'
alias tip-switch-p23='cu -l ttyel05 -s 9600'
alias tip-switch-p24='cu -l ttyel06 -s 9600'
alias tip-switch-p25='cu -l ttyel09 -s 9600'
alias tip-switch-p26='cu -l ttyel07 -s 9600'
alias tip-switch-p27='cu -l ttyel08 -s 9600'
alias tip-firewall2='cu -l ttyel01 -s 115200'
alias tip-nat2='cu -l /dev/ttyel02 -s 115200'
alias tip-backup3='cu -l ttyel04 -s 38400'
alias tip-bwdb2='cu -l ttyel03 -s 115200'
alias tip-backup4='cu -l ttyel13 -s 115200'
alias tip-jail3='cu -l ttyel11 -s 115200'

Load new shell:
source /root/.bashrc

Setup snmpd (this is only valid for a server at castle):
echo 'rocommunity jcread 10.1.4.5
rocommunity jcread 10.1.4.3
agentaddress 10.1.4.8:161' > /etc/snmp/snmpd.conf

to see which iface it is, on backup2:

snmpwalk -v 1 -c jcread 10.1.4.8 interface

=== nfs ===

Allow mounts from private net:
echo '/data 10.1.2.0/24(rw,no_root_squash,async,no_subtree_check)' >> /etc/exports

Restart nfsd:
/etc/init.d/nfs-kernel-server restart

=== bb ===

Add user, group:
echo "bb:x:1984:1984:Big Brother:/home/bb:/bin/bash" >> /etc/passwd
echo "bb:x:1984:" >> /etc/group
pwconv

Create home:
mkdir /home/bb
chown bb.bb /home/bb
cd ~bb

Copy over and install files:
<pre>scp backup2:/mnt/data4/build/bb/bb-linux.tar .
tar xf bb-linux.tar
cd /home/bb/bbc1.9e-btf/etc</pre>

Configure main bb server:
echo "69.55.230.2 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh" > /home/bb/bbc1.9e-btf/etc/bb-hosts
echo "10.1.2.3 backup3.johncompanies.com # ssh" >> /home/bb/bbc1.9e-btf/etc/bb-hosts

Configure low disk alerts:
<pre>echo "/:90:95
/var:90:95
/data:85:99" > /home/bb/bbc1.9e-btf/etc/bb-dftab</pre>

vi /home/bb/bbc1.9e-btf/bin/bb-disk.sh
(remove all | SORT xxxx since SORT is broken)

chmod +r /var/log/messages

./bbchkcfg.sh
(y to questions)
./bbchkhosts.sh
(ignore ssh errors)
<pre>cd ../..
chown -R bb .
su bb
cd
cd bbc1.9e-btf/src</pre>

<pre>make; make install
cd ..
./runbb.sh start
more BBOUT</pre>
(look for errors)
exit

<pre>vi /etc/rc.local
su - bb -c "cd /home/bb/bbc1.9e-btf; ./runbb.sh start"
</pre>
(before the exit 0)

echo 'chmod o+r /var/log/messages' >> /etc/cron.weekly/sysklogd

Add f/w rule:
ipfw add 00096 allow ip from { 69.55.229.4 or 69.55.229.3 } to 69.55.230.2 1984

<pre>vi ~bb/bbc1.9e-btf/etc/bbdef-client.sh
DFWARN=199
DFPANIC=199</pre>

=== raid check ===

==== 3ware ====
<pre>
scp backup1:/usr/local/sbin/tw_cli /usr/local/sbin/tw_cli
scp backup1:/usr/local/sbin/checkraid.sh /usr/local/sbin/checkraid.sh
scp backup1:/usr/local/sbin/3wraidchk /usr/local/sbin/3wraidchk
vi /usr/local/sbin/checkraid.sh
:%s/c0/c2/g

crontab -e
0 0 * * * /usr/local/sbin/3wraidchk</pre>

==== areca ====
<pre>
cd /tmp
wget http://www.areca.us/support/s_linux/cli/linuxcli_V1.10.0_120815.zip
unzip linuxcli_V1.10.0_120815.zip
cp linuxcli_V1.10.0_120815/x86_64/cli64 /usr/local/sbin/
chmod 0700 /usr/local/sbin/cli64
cli64 rsf info
</pre>

<pre>scp backup2:/data4/bin/arecaraidchk /usr/local/sbin
scp backup1:/usr/local/sbin/Sendmail.pm /usr/local/sbin

crontab -e
0 0 * * * /usr/local/sbin/arecaraidchk
</pre>

<pre>cat > /root/verify.sh
cli64 vsf info
cli64 rsf info
cli64 disk info
cli64 event info
echo press enter when ready to run verify ; read x

cli64 vsf check vol=1
</pre>

=== misc binaries ===

scp backup1:/usr/local/sbin/snapshot_archive /usr/local/sbin/snapshot_archive
vi /usr/local/sbin/snapshot_archive
(remove entries)

crontab -e
35 4 * * * /usr/local/sbin/snapshot_archive

scp backup1:/usr/local/sbin/pagedave /usr/local/sbin/pagedave
scp backup1:/usr/local/sbin/taskdone /usr/local/sbin/taskdone

Since installing /bin/mail requires all sorts of packages (lame) we write a simple one here...which can only email johncompanies.com addr's unless you add relaying for this host:

<pre>
cat > /bin/mail
#!/usr/bin/perl
use strict;
use warnings;

use lib '/usr/local/sbin';
use Sendmail qw(sendmail);

my $sub = $ARGV[1];
my $to = $ARGV[2];

my %mail = (
To => $to,
From => $to,
Subject => $sub,
Message => '',
smtp => 'mail.johncompanies.com'
);
sendmail(%mail) || print "Error: $Sendmail::error";

</pre>

chmod 0700 /bin/mail

=== mkbackup ===

mkdir /data/customers

<pre>cat > /usr/local/sbin/mkbackup
#!/bin/sh

if test $1; then
cid=$1
else
echo "ERROR: Usage: mkbackup cid GB ip Terminating."
exit
fi

if test $2; then
gb=$2
else
echo "ERROR: Usage: mkbackup cid GB ip Terminating."
exit
fi

if test $3; then
ip=$3
else
echo "ERROR: Usage: mkbackup cid GB ip Terminating."
exit
fi

if test -e /data/customers/${cid}-file; then
echo "ERROR: /data/customers/${cid}-file exists"
exit
else
echo "touch /data/customers/${cid}-file"
touch /data/customers/${cid}-file
count=`echo $gb|awk '{print $1*1000}'`
echo "dd if=/dev/zero of=/data/customers/${cid}-file bs=1024K count=$count"
dd if=/dev/zero of=/data/customers/${cid}-file bs=1024K count=$count
echo "/sbin/mkfs -t ext3 -F -j -q /data/customers/${cid}-file"
/sbin/mkfs -t ext3 -F -j -q /data/customers/${cid}-file
fi

if test -e /data/customers/$cid; then
echo "ERROR: /data/customers/$cid exists"
exit
else
echo "mkdir /data/customers/${cid}"
mkdir /data/customers/${cid}
echo "mount -o loop /data/customers/${cid}-file /data/customers/$cid"
mount -o loop /data/customers/${cid}-file /data/customers/$cid
df -h /data/customers/$cid

echo "fsck -y /data/customers/${cid}-file" >> /etc/nfs_backup_mounts.sh
echo "mount -o loop /data/customers/${cid}-file /data/customers/$cid" >> /etc/nfs_backup_mounts.sh
echo "" >> /etc/nfs_backup_mounts.sh

echo "/data/customers/$cid $ip/32(rw,no_root_squash,async,no_subtree_check)" >> /etc/exports
/etc/init.d/nfs-kernel-server restart
tail /var/log/messages
fi</pre>

chmod 0700 /usr/local/sbin/mkbackup

vi /etc/rc.local
add:
/etc/nfs_backup_mounts.sh

=== samba ===

apt-get install samba

vi /etc/samba/smb.conf

; comment out any mounts, add:

<pre>[data]
read only = yes
locking = no
path = /data/iso
guest ok = yes</pre>

/etc/init.d/smbd restart

mkdir /data/iso

Bring over some stuff from backup2

<pre>cd /data/iso
scp backup2:/d2/iso/3wfirmware.iso .
scp backup2:/d2/iso/MD5SUMS .
scp backup2:/d2/iso/bootimg.iso .
scp backup2:/d2/iso/systemrescuecd-x86-0.2.19.iso .
scp backup2:/d2/iso/win98bootcd.iso .
scp backup2:/d2/iso/acronis_bootdisk.iso .
scp backup2:/d2/iso/memtest86-3.2.iso .</pre>

=== Moving from one server to another ===

Here are the steps you would take to move settings and data from one server to a new backup server:

* rsync over all /data/customers (we do this cause if we didn't use *-file it would copy over the files AND the data in the mountpoint)
rsync -av --progress --ignore-times *-file root@10.1.2.33:/data/customers/
after umounting all the customers, copy over the (empty) directories separately:
for f in `find . -type d`; do rsync -av $f root@69.55.229.25:/data/customers; done

* copy mount script
[root@backup3 /data/customers]# scp /etc/nfs_backup_mounts.sh root@69.55.229.25:/etc/nfs_backup_mounts.sh

* copy rc.local
[root@backup3 /data/customers]# scp /etc/rc.local root@69.55.229.25:/etc/rc.local

* copy /etc/exports
[root@backup3 /data/customers]# scp /etc/exports root@69.55.229.25:/etc/exports

* edit /etc/hostname on both machines (set current to oldbackup3)

* edit /etc/network/interfaces (swap IPs).

* stop mounts from mounting on old and new servers so it doesnt start with reboot right away:
chmod 000 /etc/nfs_backup_mounts.sh

* reboot both servers @ same time

* check everything out

* run /etc/nfs_backup_mounts.sh on new server

* if switch port changed update mrtg to reflect correct port pub nic is on (on p20):
vi /usr/local/www/mgmt/mrtg/mrtg1.cfg

= console =

== Summary ==
This box's only purpose is to serve as a means to connect to the digi serial multiplexer boxes at castle. Connect to it using the blue (cisco) ribbon cable with the beige RJ-45 to serial connector, 9600 8N1.

* Location: castle, cab 3-8
* OS: SunOS 5.8 (solaris)
* Networking: Priv IP: 10.1.4.4
* Hardware: Sun Netra

To connect to consoles, ssh in as user 'console' and use the <tt>tip</tt> command to connect to devices listed in <tt>/etc/remote</tt>

i.e.
tip switch-p1
tip jail1

== Configuring digi/ports ==

=== /etc/remote ===
This is where the configuration/mapping for ports and custom names which we use along with the tip command to connect to various ports on the digi switches.

We have 2 digi's at castle we connect to:

#3-7 10.1.4.10
virt15:dv=/dev/dty/CO001s:br#38400:el=^C^S^Q^U^D:ie=%$:oe=^D:
virt13:dv=/dev/dty/CO002s:br#115200:el=^C^S^Q^U^D:ie=%$:oe=^D:hf:

and

#3-6 10.1.4.11
jail4:dv=/dev/dty/CP001s:br#9600:el=^C^S^Q^U^D:ie=%$:oe=^D:hf:
jail16:dv=/dev/dty/CP002s:br#9600:el=^C^S^Q^U^D:ie=%$:oe=^D:hf:

The only things you need to edit are the first part (i.e. <tt>jail4</tt>) and the speed (i.e. <tt>9600</tt>). You can decipher which port on the digi each line corresponds to by the <tt>CP001s or CO001s</tt> (port 1 on digi1 and digi2), <tt>CP002s or CO002s</tt> (port 2 on digi1 and digi2)

=== drpadmin ===
The tool you use to configure a device to a digi box is drpadmin:

<pre>bash-2.03$ su
Password:
# drpadmin

Please select an option (a)dd (d)elete (s)how (r)eset (q)uit : s
0 10.1.4.10 32 CO 771 never 1027
1 10.1.4.11 32 CP 771 never 1027
2 65.116.11.2 8 el 771 never 1027

Please select an option (a)dd (d)elete (s)how (r)eset (q)uit :</pre>

Use those commands above to modify the devices available.

== Switching IP/hostname ==

Edit:
/etc/defaultrouter
/etc/hosts
/etc/hostname.hme0
/etc/nodename
Maybe needed to run: # ifconfig hme0 10.1.4.4 up

= devweb =

We do web development on devweb.johncompanies.com

Currently this is a jail running on jail17 / 69.55.230.8

If the jail is restarted, you will need to manually restart the web service with:
httpsdctl restart

All website development work should be done here first. It works exactly like and is setup like our [[Management_System_/_Public_Website_/_Signup_/_Account_Manager|main site]].

= firewall2 =

== Summary ==

This machine is the primary firewall for the entire network at i2b. firewall3 is a hot standby replacement for
firewall2. Both firewall2 and firewall3 should not be connected at the same time since they use the same internal
and external IP addresses.

* Location: i2b, cab 6
* OS: FreeBSD 6.4 x86
* Networking: Priv IP: 10.1.2.2, Pub IPs: 66.181.18.3 (external), 69.55.229.1 & 69.55.231.1 (internal). It has 3 network connections (2 onboard, 1 PCI) connecting to the external, internal and private networks. TODO: describe NIC location/orientation

* Hardware: 2 SCSI SCA drive bays (2 columns of 3, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: 73 GB (2 x 73GB) RAID1 array running on an LSI MegaRAID SCSI 320 PCI RAID card.

== Services Provided ==
* firewall (ipfw)
* bigbrother for customer machines

== Firewall Rule Configuration ==

See [[FreeBSD_Reference#Firewall_Rule_Configuration|Firewall Rule Configuration]] for more discussion on how to actually manipulate firewall rules.

== Disaster Recovery ==

TODO: need backup f/w and instructions on how to move cables.

Here's what you need to put in /etc/rc.conf to get a firewall going (as far as routes and IPs)

TODO

Here's the config on the live firewall:

<pre>kern_securelevel_enable="NO"
portmap_enable="NO"
sendmail_enable="NO"
usbd_enable="YES"
gateway_enable="YES"

xntpd_enable="YES"
nfs_client_enable="YES"
nfs_reserved_port_only="YES"
inetd_enable="YES"
inetd_flags="-wW -a 10.1.2.2"

fsck_y_enable="YES"
background_fsck="NO"

defaultrouter="66.181.18.2"
hostname="firewall2.johncompanies.com"
ifconfig_bge0="inet 66.181.18.3 netmask 255.255.255.224"
ifconfig_bge1="inet 69.55.229.1 netmask 255.255.255.0"
ifconfig_bge1_alias0="inet 69.55.231.1 netmask 255.255.255.0"
ifconfig_bge1_alias1="inet 65.50.228.1 netmask 255.255.255.0"
ifconfig_bge1_alias2="inet 65.50.229.1 netmask 255.255.255.0"
ifconfig_bge1_alias3="inet 65.50.230.1 netmask 255.255.255.0"
ifconfig_bge1_alias4="inet 65.50.231.1 netmask 255.255.255.0"
ifconfig_bge1_alias5="inet 65.50.232.1 netmask 255.255.255.0"
ifconfig_bge1_alias6="inet 65.50.233.1 netmask 255.255.255.0"
ifconfig_bge1_alias7="inet 65.50.234.1 netmask 255.255.255.0"
ifconfig_bge1_alias8="inet 65.50.235.1 netmask 255.255.255.0"
ifconfig_fxp0="inet 10.1.2.2 netmask 255.255.255.0"
sshd_enable="YES"
usbd_enable="YES"
</pre>

== Cronjobs ==
30 3 * * * /usr/local/etc/rsync.backup
Backup to backup3

0 0 1 * * /sbin/ipfw zero
0 0 1 * * /sbin/ipfw del 3 4 5
Reset counters and remove pipe rules on the 1st of the month. Pay attention when setting up a rule as 3 4 5 (that's not a temporary traffic cap).

59 23 30 * * /sbin/ipfw show > /tmp/ipfw_count
3 0 30 * * /sbin/ipfw show > /tmp/ipfw_count
Capture counts periodically

0 3 * * * /etc/makefwrules.pl; /etc/makepiperules.pl;
This script will create <tt>/etc/firewall.sh</tt> which contains all the firewall and pipe rules in place at the time the script was run.

*/5 * * * * /usr/local/sbin/lsiraidchk
Checking the health of the RAID array

== DOS attacks ==

See [[FreeBSD_Reference#Handling_a_DoS_attack|Handling a DoS attack]] regarding how to handle a DOS attack.

== build ==

<pre>partition map:
/ 58g
swap 4g
/var 512m
/tmp 512m
/usr 5.5g

4. edit /etc/make.conf
echo "WITHOUT_X11=yes \
KERNCONF=firewall2 \
BOOT_COMCONSOLE_SPEED=115200" >> /etc/make.conf

5. add settings to /boot/loader.conf and /boot.config

echo "-Dh" >> /boot.config

echo 'console="comconsole,vidconsole" \
boot_multicons="YES" \
boot_serial="YES" \
comconsole_speed="115200"' >> /boot/loader.conf

6. turn off all ttyv's except 0 and 1 in /etc/ttys
also turn on ttyd0, change type to vt100:
vi /etc/ttys

ttyv2 "/usr/libexec/getty Pc" cons25 off secure
ttyv3 "/usr/libexec/getty Pc" cons25 off secure
ttyv4 "/usr/libexec/getty Pc" cons25 off secure
ttyv5 "/usr/libexec/getty Pc" cons25 off secure
ttyv6 "/usr/libexec/getty Pc" cons25 off secure
ttyv7 "/usr/libexec/getty Pc" cons25 off secure
# Serial terminals
# The 'dialup' keyword identifies dialin lines to login, fingerd etc.
ttyd0 "/usr/libexec/getty std.9600" vt100 on secure

kill -1 1

on console server:
vi /etc/remote
(rename port to jail8 depending on where and which digi plugged into)
test serial console

7. populate hosts
echo "69.55.230.10 backup2" >> /etc/hosts
echo "69.55.230.11 backup1" >> /etc/hosts
echo "10.1.2.3 backup3" >> /etc/hosts

8. put key in authorized_keys on backup3
cd
ssh-keygen -t dsa -b 1024
(default location, leave password blank)

Punch a hole in firewall1 to allow traffic to backup servers @ castle:

ipfw add 99 allow ip from 66.181.18.0/27 to 69.55.230.10 22
ipfw add 95 allow ip from 66.181.18.0/27 to 69.55.230.11 22

cat /root/.ssh/id_dsa.pub | ssh backup3 'cat - >> /root/.ssh/authorized_keys'
cat /root/.ssh/id_dsa.pub | ssh backup2 'cat - >> /root/.ssh/authorized_keys'
cat /root/.ssh/id_dsa.pub | ssh backup1 'cat - >> /root/.ssh/authorized_keys'

confirm that you can ssh to backup3 and backup 2 without getting a login prompt

ssh backup3 hostname

ssh backup2 hostname

ssh backup1 hostname

10. edit root's path and login script:
vi /root/.cshrc

Change alias entries (add G):
alias la ls -aG
alias lf ls -FAG
alias ll ls -lAG
alias ls ls -AG
alias mbm mb mount
alias mbu mb umount

and alter the prompt, set the following:
set prompt = "`/bin/hostname -s` %/# "

11. install cvsup
cd /usr/ports/net/cvsup-without-gui
make install clean; rehash; mail -s 'cvs installed' dave.boodman@vtext.com < /dev/null

12. get latest sources for this release:
cd /usr/src
echo "*default host=cvsup4.freebsd.org\
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_6_4\
*default delete use-rel-suffix\
*default compress\
src-all" > sup

cvsup sup ; mail -s 'cvs sup done' dave.boodman@vtext.com < /dev/null

13. configure new kernel.

cd /usr/src/sys/i386/conf
scp backup2:/mnt/data4/build/freebsd/firewall2-6.4 ./firewall2

15. build, install kernel and world

cd /boot

mv kernel kernel.GENERIC
cd kernel.GENERIC
cd /usr/src
make buildkernel installkernel

make buildworld ; mail -s 'buildworld done' dave.boodman@vtext.com < /dev/null
(2450: 1:56min, supermicro: 59mins, 2950: 38mins)
make installworld
(2450: 3min, supermicro: 1min, 2950: :34)
mergemaster -i

17. populate /etc/rc.conf with IPs and NFS settings
vi /etc/rc.conf

kern_securelevel_enable="NO"
portmap_enable="NO"
sendmail_enable="NO"
usbd_enable="YES"
gateway_enable="YES"

xntpd_enable="YES"
nfs_client_enable="YES"
nfs_reserved_port_only="YES"
inetd_enable="YES"
inetd_flags="-wW -a 10.1.2.2"

ifconfig_bce1="inet 10.1.2.2 netmask 255.255.255.0"
fsck_y_enable="YES"
background_fsck="NO"

defaultrouter="66.181.18.2"
hostname="firewall2.johncompanies.com"
ifconfig_bge0="inet 66.181.18.3 netmask 255.255.255.224"
ifconfig_bge1="inet 69.55.229.1 netmask 255.255.255.0"
ifconfig_fxp0="inet 10.1.2.2 netmask 255.255.255.0"
sshd_enable="YES"
usbd_enable="YES"

20. reboot. Confirm new kernel is loaded

uname -a

21. update ports:
cd /usr/ports
echo "*default host=cvsup4.FreeBSD.org\
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_6_4\
*default delete use-rel-suffix\
*default compress\
ports-all tag=." > sup

cvsup sup; mail -s 'cvs sup ports done' dave.boodman@vtext.com < /dev/null

22. Install raid mgmt tool

# linux base
cd /usr/ports/devel/libtool22
make install base

cd /usr/ports/emulators/linux_base-fc4
make install clean

#linux-megamgr-5.20
cd /usr/ports/sysutils/linux-megamgr
make install clean

# megarc-1.51
cd /usr/ports/sysutils/megarc
make install clean

Test:
rehash; megarc -ldInfo -a0 -l0

23. install rsync from ports
cd /usr/ports/net/rsync
make install clean

choose default options

25. install bb client
adduser
Username: bb
Full name: bb
Uid (Leave empty for default): 1984
Login group [bb]:
Login group is bb. Invite bb into other groups? []:
Login class [default]:
Shell (sh csh tcsh nologin) [sh]:
Home directory [/home/bb]:
Use password-based authentication? [yes]:
Use an empty password? (yes/no) [no]:
Use a random password? (yes/no) [no]: yes
Lock out the account after creation? [no]:
Username : bb
Password : <random>
Full Name : bb
Uid : 1984
Class :
Groups : bb
Home : /home/bb
Shell : /bin/sh
Locked : no
OK? (yes/no): yes

cd /usr/home/bb
scp backup2:/mnt/data4/build/bb/bb-freebsd.tar .
tar xvf bb-freebsd.tar

edit /home/bb/bbc1.9e-btf/etc/bb-hosts with something like:
echo "69.55.230.2 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh \
10.1.2.1 firewall2.johncompanies.com # ssh" > /home/bb/bbc1.9e-btf/etc/bb-hosts

vi /home/bb/bbc1.9e-btf/ext/openfiles
MACHINE="firewall2,johncompanies,com" # HAS TO BE IN A,B,C FORM

cd /usr/home/bb/bbc1.9e-btf/etc
./bbchkcfg.sh
(y to questions)
./bbchkhosts.sh
(ignore ssh errors)
cd ../..
chown -R bb .
su bb
cd
cd bbc1.9e-btf/src
make; make install
cd ..

vi /usr/home/bb/bbc1.9e-btf/bin/bb-cpu.sh
$1 $TOPARGS > $BBTMP/TOP.$$
# /usr/local/jail/bin/jtop > $BBTMP/TOP.$$

./runbb.sh start
more BBOUT
(look for errors)
exit

echo 'su - bb -c "cd /home/bb/bbc1.9e-btf; ./runbb.sh start"' > /usr/local/etc/rc.d/bb.sh
chmod +x /usr/local/etc/rc.d/bb.sh

Punch a hole in the firewall to allow it to communicate with bb monitor:

ipfw add 00096 allow ip from 66.181.18.0/27 to 69.55.230.2

27. configure bb on mail:
vi /usr/home/bb/bbsrc/bb1.9e-btf/etc/bb-hosts
66.181.18.3 firewall2.johncompanies.com # ssh

su bb
cd
bbsrc/bb/runbb.sh restart ; exit

29. configure ntp
echo "server 10.1.2.1" > /etc/ntp.conf

/usr/sbin/ntpd -p /var/run/ntpd.pid
sleep 2; ntpq -p
(confirm it’s able to reach our time server)

echo '/usr/sbin/ntpd -c /etc/ntp.conf -p /var/run/ntpd.pid -f /var/db/ntpd.drift' > /usr/local/etc/rc.d/ntp.sh
chmod 0700 /usr/local/etc/rc.d/ntp.sh

30. fwd and reverse lookups on ns1c
vr johncompanies.com
(edit the PTR too)

33. setup backups
echo '#\!/bin/sh\
backupdir=/data/firewall2/current\
\
## ENTRY /etc ' > /usr/local/etc/backup.config

on backup3:
setup backup dirs:
ssh backup3 mkdir -p /data/firewall2/current

on backup3, add the system to
vi /usr/local/sbin/snapshot_archive

scp 69.55.230.2:/usr/local/etc/rsync.backup /usr/local/etc/rsync.backup
vi /usr/local/etc/rsync.backup
backup1 > backup3

crontab -e
1 0 * * * /usr/local/etc/rsync.backup

34. mkdir /root/logs

35. edit sshd_config for security
vi /etc/ssh/sshd_config
ListenAddress 66.181.18.3
ListenAddress 10.1.2.1

kill -1 `cat /var/run/sshd.pid`

35. raid chk

cat > /usr/local/sbin/lsiraidchk
#!/usr/bin/perl

my @out = split "\n", `megarc -ldInfo -a0 -Lall|grep Status:`;

foreach (@out) {
if ($_ =~ /DEGRADED/) { $date = `date`; chomp $date; `echo "$date: RAID ARRAY DEGRADED" >> /var/log/messages`; }
#print $_;
}

36. add crontab entries
crontab -e
30 3 * * * /usr/local/etc/rsync.backup
0 0 1 * * /sbin/ipfw zero
0 0 1 * * /sbin/ipfw del 3 4 5
59 23 30 * * /sbin/ipfw show > /tmp/ipfw_count
3 0 30 * * /sbin/ipfw show > /tmp/ipfw_count
0 3 * * * /etc/makefwrules.pl; /etc/makepiperules.pl;
*/5 * * * * /usr/local/sbin/lsiraidchk

#10 0 * * * rm /var/spool/clientmqueue/*

scp /etc/makefwrules.pl user@64.163.14.48:~
scp /etc/makepiperules.pl user@64.163.14.48:~
mv /home/user/makefwrules.pl /etc
mv /home/user/makepiperules.pl /etc
touch /etc/firewall.sh
mkdir /etc/oldrules/

other binaries

scp /usr/local/bin/rulemaker user@64.163.14.48:~
mv ~user/rulemaker /usr/local/sbin
scp ~user/Sendmail.pm user@64.163.14.48:~
scp ~user/doswatch.pl user@64.163.14.48:~

Setup basic ruleset

ipfw add 00009 count udp from any to any
ipfw add 00010 allow tcp from any to any established
ipfw add 00012 deny tcp from any to any tcpflags syn tcpoptions !mss
ipfw add 00012 deny icmp from any to any icmptypes 4,5,9,10,12,13,14,15,16,17,18
ipfw add 00012 deny tcp from any to any tcpflags syn,fin
ipfw add 00012 deny tcp from any to any tcpflags fin,psh,rst,urg
ipfw add 00012 allow icmp from any to any
ipfw add 00014 deny tcp from any to any dst-port 135
ipfw add 00150 skipto 65535 ip from any to any via em1 in

IPKVM3:
00098 allow ip from { 69.55.230.6 or 69.55.230.7 } to 69.55.230.10 dst-port 139
00098 deny ip from any to 69.55.230.10 dst-port 139
</pre>

= firewall3 =

== Summary ==

This machine is the backup firewall for the network at i2b.

* Location: i2b, cab ?
* OS: FreeBSD 9.1 amd64
* Networking: Priv IP: 10.1.2.5, Pub IPs: 66.181.18.3 (external), 69.55.229.1 & 69.55.231.1 (internal). It has 3 network connections (2 onboard, 1 PCI) connecting to the external, internal and private networks.

<pre>

The internal network NIC is the left one on the motherboard (69.55.229.1/24, ...).
The external network NIC is the right one on the motherboard (66.181.18.3/28).
The PCI ethernet card is connected to our private network (10.1.2.5/24).

</pre>

* Hardware: 2 SCSI SCA drive bays (2 columns of 3, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: 160 GB (2 x 160GB) RAID1 array running on an LSI MegaRAID SCSI 320 PCI RAID card.

== Services Provided ==
* firewall (ipfw)
* bigbrother

== Firewall Rule Configuration ==

See [[FreeBSD_Reference#Firewall_Rule_Configuration|Firewall Rule Configuration]] for more discussion on how to actually manipulate firewall rules.

== Disaster Recovery ==

'''To put the backup firewall3 into service:'''

<pre>
Move the internal cable (to our networks) from firewall2 to em1 which is the left most ethernet port (69.55.229.1).
Move the external cable (to outside world) from firewall2 to em0 which is the port to the right on the motherboard (66.181.18.3).
The PCI ethernet port (fxp0) should already be connected to private network (10.1.2.5).
</pre>

Here's what you need to put in /etc/rc.conf to get a firewall going (as far as routes and IPs)

<pre>
kern_securelevel_enable="NO"
portmap_enable="NO"
sendmail_enable="NO"
usbd_enable="YES"
gateway_enable="YES"

xntpd_enable="YES"
nfs_client_enable="YES"
nfs_reserved_port_only="YES"
inetd_enable="YES"
inetd_flags="-wW -a 10.1.2.5"

fsck_y_enable="YES"
background_fsck="NO"

defaultrouter="66.181.18.2"
hostname="firewall3.johncompanies.com"
ifconfig_em0="inet 66.181.18.3 netmask 255.255.255.224"

ifconfig_em1="inet 69.55.229.1 netmask 255.255.255.0"
ifconfig_em1_alias0="inet 69.55.231.1 netmask 255.255.255.0"

# ifconfig_em1_alias1="inet 65.50.228.1 netmask 255.255.255.0"
# ifconfig_em1_alias2="inet 65.50.229.1 netmask 255.255.255.0"
# ifconfig_em1_alias3="inet 65.50.230.1 netmask 255.255.255.0"
# ifconfig_em1_alias4="inet 65.50.231.1 netmask 255.255.255.0"
# ifconfig_em1_alias5="inet 65.50.232.1 netmask 255.255.255.0"
# ifconfig_em1_alias6="inet 65.50.233.1 netmask 255.255.255.0"
# ifconfig_em1_alias7="inet 65.50.234.1 netmask 255.255.255.0"
# ifconfig_em1_alias8="inet 65.50.235.1 netmask 255.255.255.0"

ifconfig_fxp0="inet 10.1.2.5 netmask 255.255.255.0"

sshd_enable="YES"
usbd_enable="YES"
</pre>

== Cronjobs ==
30 3 * * * /usr/local/etc/rsync.backup
Backup to backup3

0 0 1 * * /sbin/ipfw zero
0 0 1 * * /sbin/ipfw del 3 4 5
Reset counters and remove pipe rules on the 1st of the month. Pay attention when setting up a rule as 3 4 5 (that's not a temporary traffic cap).

59 23 30 * * /sbin/ipfw show > /tmp/ipfw_count
3 0 30 * * /sbin/ipfw show > /tmp/ipfw_count
Capture counts periodically

0 3 * * * /etc/makefwrules.pl; /etc/makepiperules.pl;
This script will create <tt>/etc/firewall.sh</tt> which contains all the firewall and pipe rules in place at the time the script was run.

*/5 * * * * /usr/local/sbin/lsiraidchk
Checking the health of the RAID array

== DOS attacks ==

See [[FreeBSD_Reference#Handling_a_DoS_attack|Handling a DoS attack]] regarding how to handle a DOS attack.

== build ==

<pre>partition map:
/ 58g
swap 4g
/var 512m
/tmp 512m
/usr 5.5g

4. edit /etc/make.conf
echo "WITHOUT_X11=yes \
KERNCONF=firewall3 \
BOOT_COMCONSOLE_SPEED=115200" >> /etc/make.conf

5. add settings to /boot/loader.conf and /boot.config

echo "-Dh" >> /boot.config

echo 'console="comconsole,vidconsole" \
boot_multicons="YES" \
boot_serial="YES" \
comconsole_speed="115200"' >> /boot/loader.conf

6. turn off all ttyv's except 0 and 1 in /etc/ttys
also turn on ttyd0, change type to vt100:
vi /etc/ttys

ttyv2 "/usr/libexec/getty Pc" cons25 off secure
ttyv3 "/usr/libexec/getty Pc" cons25 off secure
ttyv4 "/usr/libexec/getty Pc" cons25 off secure
ttyv5 "/usr/libexec/getty Pc" cons25 off secure
ttyv6 "/usr/libexec/getty Pc" cons25 off secure
ttyv7 "/usr/libexec/getty Pc" cons25 off secure
# Serial terminals
# The 'dialup' keyword identifies dialin lines to login, fingerd etc.
ttyd0 "/usr/libexec/getty std.9600" vt100 on secure

kill -1 1

on console server:
vi /etc/remote
(rename port to jail8 depending on where and which digi plugged into)
test serial console

7. populate hosts
echo "69.55.230.10 backup2" >> /etc/hosts
echo "69.55.230.11 backup1" >> /etc/hosts
echo "10.1.2.3 backup3" >> /etc/hosts

8. put key in authorized_keys on backup3
cd
ssh-keygen -t dsa -b 1024
(default location, leave password blank)

Punch a hole in firewall1 to allow traffic to backup servers @ castle:

ipfw add 99 allow ip from 66.181.18.0/27 to 69.55.230.10 22
ipfw add 95 allow ip from 66.181.18.0/27 to 69.55.230.11 22

cat /root/.ssh/id_dsa.pub | ssh backup3 'cat - >> /root/.ssh/authorized_keys'
cat /root/.ssh/id_dsa.pub | ssh backup2 'cat - >> /root/.ssh/authorized_keys'
cat /root/.ssh/id_dsa.pub | ssh backup1 'cat - >> /root/.ssh/authorized_keys'

confirm that you can ssh to backup3 and backup 2 without getting a login prompt

ssh backup3 hostname

ssh backup2 hostname

ssh backup1 hostname

10. edit root's path and login script:
vi /root/.cshrc

Change alias entries (add G):
alias la ls -aG
alias lf ls -FAG
alias ll ls -lAG
alias ls ls -AG
alias mbm mb mount
alias mbu mb umount

and alter the prompt, set the following:
set prompt = "`/bin/hostname -s` %/# "

11. install cvsup
cd /usr/ports/net/cvsup-without-gui
make install clean; rehash; mail -s 'cvs installed' 8583619553@vtext.com < /dev/null

12. get latest sources for this release:
cd /usr/src
echo "*default host=cvsup4.freebsd.org\
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_9_1\
*default delete use-rel-suffix\
*default compress\
src-all" > sup

cvsup sup ; mail -s 'cvs sup done' 8583619553@vtext.com < /dev/null

13. configure new kernel.

cd /usr/src/sys/amd64/conf
scp backup2:/mnt/data4/build/freebsd/firewall3-9.1 ./firewall3

15. build, install kernel and world

cd /boot

mv kernel kernel.GENERIC
cd kernel.GENERIC
cd /usr/src
make buildkernel installkernel

make buildworld ; mail -s 'buildworld done' 8583619553@vtext.com < /dev/null
(supermicro: 2:15 mins, 2950: 38? mins)
make installworld
(2450: 3min, supermicro: 1min, 2950: :34)
mergemaster -i

17. populate /etc/rc.conf with IPs and NFS settings
vi /etc/rc.conf

kern_securelevel_enable="NO"
portmap_enable="NO"
sendmail_enable="NO"
usbd_enable="YES"
gateway_enable="YES"

xntpd_enable="YES"
nfs_client_enable="YES"
nfs_reserved_port_only="YES"

fsck_y_enable="YES"
background_fsck="NO"

hostname="firewall3.johncompanies.com"
# external network
ifconfig_em0="inet 66.181.18.3 netmask 255.255.255.224"

# internal network
ifconfig_em1="inet 69.55.229.1 netmask 255.255.255.0"
ifconfig_em1_alias0="inet 69.55.231.1 netmask 255.255.255.0"

ifconfig_em1_alias1="inet 65.50.228.1 netmask 255.255.255.0"
ifconfig_em1_alias2="inet 65.50.229.1 netmask 255.255.255.0"
ifconfig_em1_alias3="inet 65.50.230.1 netmask 255.255.255.0"
ifconfig_em1_alias4="inet 65.50.231.1 netmask 255.255.255.0"
ifconfig_em1_alias5="inet 65.50.232.1 netmask 255.255.255.0"
ifconfig_em1_alias6="inet 65.50.233.1 netmask 255.255.255.0"
ifconfig_em1_alias7="inet 65.50.234.1 netmask 255.255.255.0"
ifconfig_em1_alias8="inet 65.50.235.1 netmask 255.255.255.0"

defaultrouter="66.181.18.2"

# private network
ifconfig_fxp0="inet 10.1.2.5 netmask 255.255.255.0"

inetd_enable="YES"
inetd_flags="-wW -a 10.1.2.5"

sshd_enable="YES"
usbd_enable="YES"
ntpd_enable="YES"
# powerd_enable="YES"

20. reboot. Confirm new kernel is loaded

uname -a

21. update ports:
cd /usr/ports

echo "*default host=cvsup4.FreeBSD.org\
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_9_1\
*default delete use-rel-suffix\
*default compress\
ports-all tag=." > sup

cvsup sup; mail -s 'cvs sup ports done' 8583619553@vtext.com < /dev/null

22. Install raid mgmt tool

# linux base
cd /usr/ports/devel/libtool22
make install base

cd /usr/ports/emulators/linux_base-fc4
make install clean

scp backup2:/d4/build/3ware/tw_cli-freebsd-x86_64-9.5.0.1.tgz /usr/local/sbin
cd /usr/local/sbin
tar xzvf tw_cli-freebsd-x86_64-9.5.0.1.tgz
rm tw_cli-freebsd-x86_64-9.5.0.1.tgz

23. install rsync from ports
cd /usr/ports/net/rsync
make install clean

choose default options

25. install bb client
adduser
Username: bb
Full name: bb
Uid (Leave empty for default): 1984
Login group [bb]:
Login group is bb. Invite bb into other groups? []:
Login class [default]:
Shell (sh csh tcsh nologin) [sh]:
Home directory [/home/bb]:
Use password-based authentication? [yes]:
Use an empty password? (yes/no) [no]:
Use a random password? (yes/no) [no]: yes
Lock out the account after creation? [no]:
Username : bb
Password : <random>
Full Name : bb
Uid : 1984
Class :
Groups : bb
Home : /home/bb
Shell : /bin/sh
Locked : no
OK? (yes/no): yes

cd /usr/home/bb
scp backup2:/mnt/data4/build/bb/bb-freebsd.tar .
tar xvf bb-freebsd.tar

edit /home/bb/bbc1.9e-btf/etc/bb-hosts with something like:
echo "69.55.230.2 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh \
10.1.2.5 firewall3.johncompanies.com # ssh" > /home/bb/bbc1.9e-btf/etc/bb-hosts

vi /home/bb/bbc1.9e-btf/ext/openfiles
MACHINE="firewall3,johncompanies,com" # HAS TO BE IN A,B,C FORM

cd /usr/home/bb/bbc1.9e-btf/etc
./bbchkcfg.sh
(y to questions)
./bbchkhosts.sh
(ignore ssh errors)
cd ../..
chown -R bb .
su bb
cd
cd bbc1.9e-btf/src
make; make install
cd ..

vi /usr/home/bb/bbc1.9e-btf/bin/bb-cpu.sh
$1 $TOPARGS > $BBTMP/TOP.$$
# /usr/local/jail/bin/jtop > $BBTMP/TOP.$$

./runbb.sh start
more BBOUT
(look for errors)
exit

echo 'su - bb -c "cd /home/bb/bbc1.9e-btf; ./runbb.sh start"' > /usr/local/etc/rc.d/bb.sh
chmod +x /usr/local/etc/rc.d/bb.sh

Punch a hole in the firewall to allow it to communicate with bb monitor:

ipfw add 00096 allow ip from 66.181.18.0/27 to 69.55.230.2

27. configure bb on mail:
vi /usr/home/bb/bbsrc/bb1.9e-btf/etc/bb-hosts
66.181.18.3 firewall3.johncompanies.com # ssh

su bb
cd
bbsrc/bb/runbb.sh restart ; exit

29. configure ntp
echo "server 10.1.2.1" > /etc/ntp.conf

/usr/sbin/ntpd -p /var/run/ntpd.pid
sleep 2; ntpq -p
(confirm it’s able to reach our time server)

echo '/usr/sbin/ntpd -c /etc/ntp.conf -p /var/run/ntpd.pid -f /var/db/ntpd.drift' > /usr/local/etc/rc.d/ntp.sh
chmod 0700 /usr/local/etc/rc.d/ntp.sh

30. fwd and reverse lookups on ns1c
vr johncompanies.com
(edit the PTR too)

33. setup backups
echo '#\!/bin/sh\
backupdir=/data/firewall2/current\
\
## ENTRY /etc ' > /usr/local/etc/backup.config

on backup3:
setup backup dirs:
ssh backup3 mkdir -p /data/firewall2/current

on backup3, add the system to
vi /usr/local/sbin/snapshot_archive

scp 69.55.230.2:/usr/local/etc/rsync.backup /usr/local/etc/rsync.backup
vi /usr/local/etc/rsync.backup
backup1 > backup3

crontab -e
1 0 * * * /usr/local/etc/rsync.backup

34. mkdir /root/logs

35. edit sshd_config for security
vi /etc/ssh/sshd_config
ListenAddress 66.181.18.3
ListenAddress 10.1.2.5

kill -1 `cat /var/run/sshd.pid`

35. raid chk

cat > /usr/local/sbin/lsiraidchk
#!/usr/bin/perl

my @out = split "\n", `megarc -ldInfo -a0 -Lall|grep Status:`;

foreach (@out) {
if ($_ =~ /DEGRADED/) { $date = `date`; chomp $date; `echo "$date: RAID ARRAY DEGRADED" >> /var/log/messages`; }
#print $_;
}

36. add crontab entries
crontab -e
30 3 * * * /usr/local/etc/rsync.backup
0 0 1 * * /sbin/ipfw zero
0 0 1 * * /sbin/ipfw del 3 4 5
59 23 30 * * /sbin/ipfw show > /tmp/ipfw_count
3 0 30 * * /sbin/ipfw show > /tmp/ipfw_count
0 3 * * * /etc/makefwrules.pl; /etc/makepiperules.pl;
*/5 * * * * /usr/local/sbin/lsiraidchk

#10 0 * * * rm /var/spool/clientmqueue/*

scp /etc/makefwrules.pl user@64.163.14.48:~
scp /etc/makepiperules.pl user@64.163.14.48:~
mv /home/user/makefwrules.pl /etc
mv /home/user/makepiperules.pl /etc
touch /etc/firewall.sh
mkdir /etc/oldrules/

other binaries

scp /usr/local/bin/rulemaker user@64.163.14.48:~
mv ~user/rulemaker /usr/local/sbin
scp ~user/Sendmail.pm user@64.163.14.48:~
scp ~user/doswatch.pl user@64.163.14.48:~

Setup basic ruleset

ipfw add 00009 count udp from any to any
ipfw add 00010 allow tcp from any to any established
ipfw add 00012 deny tcp from any to any tcpflags syn tcpoptions !mss
ipfw add 00012 deny icmp from any to any icmptypes 4,5,9,10,12,13,14,15,16,17,18
ipfw add 00012 deny tcp from any to any tcpflags syn,fin
ipfw add 00012 deny tcp from any to any tcpflags fin,psh,rst,urg
ipfw add 00012 allow icmp from any to any
ipfw add 00014 deny tcp from any to any dst-port 135
ipfw add 00150 skipto 65535 ip from any to any via em1 in

IPKVM3:
00098 allow ip from { 69.55.230.6 or 69.55.230.7 } to 69.55.230.10 dst-port 139
00098 deny ip from any to 69.55.230.10 dst-port 139
</pre>

= wiki =

The wiki (mediawiki) runs on nat2 in a jail running off 69.55.229.8

The backup wiki lives on virt13 in CT 5 / 69.55.230.18

== Setup jail ==

<pre>
mkdir /mnt/data1/wiki-dir
cd /usr/src
make installworld DESTDIR=/mnt/data1/wiki-dir
cd etc
make distribution DESTDIR=/mnt/data1/wiki-dir

mount -t devfs devfs /mnt/data1/wiki-dir/dev
devfs -m /mnt/data1/wiki-dir/dev rule -s 3 applyset

cd /mnt/data1/wiki-dir

ln -sf dev/null kernel

scp jail9:/usr/local/sbin/jkill /mnt/data1/wiki-dir/sbin

jail /mnt/data1/wiki-dir wiki.johncompanies.com 69.55.229.8 /bin/sh
csh

touch /etc/fstab
echo 'network_interfaces=""\
hostname="wiki.johncompanies.com"\
kern_securelevel_enable="NO"\
sendmail_enable="YES"\
sshd_enable="YES"' > /etc/rc.conf

echo "nameserver 69.55.229.3\
nameserver 69.55.225.225" >> /etc/resolv.conf

vi /etc/crontab

(remove the adjkerntz lines )

vi /etc/periodic/security/100.chksetuid

replace: MP=`mount -t ufs | grep -v " nosuid" | awk '{ print $3 }' | sort`
with: MP='/' (use single quotes)

mkdir -p /usr/compat/linux/dev

adduser

Username: user
Full name: user
Uid (Leave empty for default):
Login group [user]:
Login group is user. Invite user into other groups? []: wheel
Login class [default]:
Shell (sh csh tcsh nologin) [sh]:
Home directory [/home/user]:
Home directory permissions (Leave empty for default):
Use password-based authentication? [yes]:
Use an empty password? (yes/no) [no]:
Use a random password? (yes/no) [no]: y
Lock out the account after creation? [no]:
Username : user
Password : <random>
Full Name : user
Uid : 1001
Class :
Groups : user
Home : /home/user
Home Mode :
Shell : /bin/sh
Locked : no
OK? (yes/no): y
adduser: INFO: Successfully added (user) to the user database.
adduser: INFO: Password for (user) is: 901gmYjO
Add another user? (yes/no): n
Goodbye!

vi /usr/home/user/.profile
TERM=vt100; export TERM

tzsetup

newaliases

rm /sbin/halt /sbin/reboot
ln /sbin/jkill /sbin/halt
ln /sbin/jkill /sbin/reboot

vi /etc/syslog.conf
#*.err;kern.warning;auth.notice;mail.crit /dev/console
*.err;kern.warning;auth.notice;mail.crit /var/log/messages

exit
exit

cd libexec
chflags noschg ld-elf32.so.1
chflags noschg ld-elf.so.1
mv ld-elf32.so.1 ld-elf32.so.1-orig
ln ld-elf.so.1 ld-elf32.so.1
chflags schg ld-elf.so.1
chflags schg ld-elf32.so.1

cp -r /usr/ports /mnt/data1/wiki-dir/usr

cat > /usr/local/etc/rc.d/wiki.sh
mount -t devfs devfs /mnt/data1/wiki-dir/dev/
devfs -m /mnt/data1/wiki-dir/dev rule -s 3 applyset
jail /mnt/data1/wiki-dir wiki.johncompanies.com 69.55.229.8 /bin/sh /etc/rc

chmod 0700 /usr/local/etc/rc.d/wiki.sh
</pre>

== mediawiki setup ==

<pre>

cd /usr/ports/net/rsync
make install clean

cd /usr/ports/distfiles/
fetch http://downloads.mysql.com/archives/mysql-5.5/mysql-5.5.4-m3.tar.gz
cd /usr/ports/databases/mysql55-server
make install clean

cd /usr/ports/distfiles/
fetch http://downloads.php.net/johannes/php-5.3.2.tar.bz2
cd /usr/ports/lang/php52
make install clean
(build apache module)

cd /usr/ports/lang/php5-extensions
make install clean

cd /usr/ports/www/apache22
make install clean

cd /usr/local/www/
fetch http://download.wikimedia.org/mediawiki/1.19/mediawiki-1.19.2.tar.gz
tar xzf mediawiki-1.19.2.tar.gz
mv mediawiki-1.19.2 wiki

vi /usr/local/etc/apache22/httpd.conf
DocumentRoot "/usr/local/www/"

Include etc/apache22/extra/vhost-wiki.conf
Listen 443

<IfModule mod_php5.c>
<FilesMatch "\.ph(p3?|tml)$">
SetHandler application/x-httpd-php
</FilesMatch>
<FilesMatch "\.phps$">
SetHandler application/x-httpd-php-source
</FilesMatch>
# To re-enable php in user directories comment the following lines
# (from <IfModule ...> to </IfModule>.) Do NOT set it to On as it
# prevents .htaccess files from disabling it.
<IfModule mod_userdir.c>
<Directory /home/*/public_html>
php_admin_value engine Off
</Directory>
</IfModule>
</IfModule>

cat > /usr/local/etc/apache22/extra/vhost-wiki.conf
<VirtualHost *:443>
ServerAdmin support@johncompanies.com

DocumentRoot /usr/local/www/wiki
# <Directory />
# Options FollowSymLinks
# AllowOverride None
# Order deny,allow
# </Directory>
<Directory /usr/local/www/wiki>
Options Indexes FollowSymLinks MultiViews
Deny from all
AllowOverride AuthConfig
Order allow,deny
DirectoryIndex index.php
#Allow from 69.55.233.195
#Allow from boody.dyndns.org
</Directory>

ScriptAlias /cgi-bin/ /usr/lib/cgi-bin/
<Directory "/usr/lib/cgi-bin">
AllowOverride None
Options +ExecCGI -MultiViews +SymLinksIfOwnerMatch
Order allow,deny
Allow from all
</Directory>

ErrorLog /var/log/httpd-error.log

# Possible values include: debug, info, notice, warn, error, crit,
# alert, emerg.
LogLevel warn

CustomLog /var/log/httpd-access.log combined

Alias /doc/ "/usr/share/doc/"
<Directory "/usr/share/doc/">
Options Indexes MultiViews FollowSymLinks
AllowOverride None
Order deny,allow
Deny from all
Allow from 127.0.0.0/255.0.0.0 ::1/128
</Directory>

SSLEngine on
SSLCertificateFile /usr/local/etc/apache22/ssl/server.crt
SSLCertificateKeyFile /usr/local/etc/apache22/ssl/server.key

</VirtualHost>

mkdir ssl
cd ssl

openssl req -days 1999 -new -x509 -nodes -out server.crt -keyout server.key
US
CA
San Diego
johncompanies.com
johncompanies.com
wiki.johncompanies.com
support@johncompanies.com

cat > /usr/local/www/wiki/.htaccess
AuthType Basic
AuthUserFile /usr/local/etc/apache22/wiki.passwd
AuthName wiki
require valid-user
satisfy any

cd /usr/local/etc/apache22
htpasswd -c wiki.passwd admin

https://69.55.229.8/index.php

use mysql (innodb)
wiki name: JCWiki
Support / (mail pass) / support@johncompanies.com

cat > /usr/local/www/wiki/LocalSettings.php

<?php
# This file was automatically generated by the MediaWiki 1.19.2
# installer. If you make manual changes, please keep track in case you
# need to recreate them later.
#
# See includes/DefaultSettings.php for all configurable settings
# and their default values, but don't forget to make changes in _this_
# file, not there.
#
# Further documentation for configuration settings may be found at:
# http://www.mediawiki.org/wiki/Manual:Configuration_settings

# Protect against web entry
if ( !defined( 'MEDIAWIKI' ) ) {
exit;
}

## Uncomment this to disable output compression
# $wgDisableOutputCompression = true;

$wgSitename = "JCWiki";

## The URL base path to the directory containing the wiki;
## defaults for all runtime URL paths are based off of this.
## For more information on customizing the URLs please see:
## http://www.mediawiki.org/wiki/Manual:Short_URL
$wgScriptPath = "";
$wgScriptExtension = ".php";

## The protocol and server name to use in fully-qualified URLs
$wgServer = "https://69.55.229.8";

## The relative URL path to the skins directory
$wgStylePath = "$wgScriptPath/skins";

## The relative URL path to the logo. Make sure you change this from the default,
## or else you'll overwrite your logo when you upgrade!
#$wgLogo = "$wgStylePath/common/images/wiki.png";
$wgLogo = "$wgStylePath/common/images/jclogo.gif";

## UPO means: this is also a user preference option

$wgEnableEmail = true;
$wgEnableUserEmail = true; # UPO

$wgEmergencyContact = "apache@69.55.229.8";
$wgPasswordSender = "apache@69.55.229.8";

$wgEnotifUserTalk = false; # UPO
$wgEnotifWatchlist = false; # UPO
$wgEmailAuthentication = true;

## Database settings
$wgDBtype = "mysql";
$wgDBserver = "localhost";
$wgDBname = "my_wiki";
$wgDBuser = "root";
$wgDBpassword = "";

# MySQL specific settings
$wgDBprefix = "";

# MySQL table options to use during installation or update
$wgDBTableOptions = "ENGINE=InnoDB, DEFAULT CHARSET=binary";

# Experimental charset support for MySQL 5.0.
$wgDBmysql5 = false;

## Shared memory settings
$wgMainCacheType = CACHE_NONE;
$wgMemCachedServers = array();

## To enable image uploads, make sure the 'images' directory
## is writable, then set this to true:
$wgEnableUploads = false;
#$wgUseImageMagick = true;
#$wgImageMagickConvertCommand = "/usr/bin/convert";

# InstantCommons allows wiki to use images from http://commons.wikimedia.org
$wgUseInstantCommons = false;

## If you use ImageMagick (or any other shell command) on a
## Linux server, this will need to be set to the name of an
## available UTF-8 locale
$wgShellLocale = "en_US.utf8";

## If you want to use image uploads under safe mode,
## create the directories images/archive, images/thumb and
## images/temp, and make them all writable. Then uncomment
## this, if it's not already uncommented:
#$wgHashedUploadDirectory = false;

## Set $wgCacheDirectory to a writable directory on the web server
## to make your wiki go slightly faster. The directory should not
## be publically accessible from the web.
#$wgCacheDirectory = "$IP/cache";

# Site language code, should be one of the list in ./languages/Names.php
$wgLanguageCode = "en";

$wgSecretKey = "abc699ef26890b49b4055430f8ebbd25e84cce21a7e53aeaec4d4313af4c9739";

# Site upgrade key. Must be set to a string (default provided) to turn on the
# web installer while LocalSettings.php is in place
$wgUpgradeKey = "3196710f4a7d7332";

## Default skin: you can change the default skin. Use the internal symbolic
## names, ie 'standard', 'nostalgia', 'cologneblue', 'monobook', 'vector':
$wgDefaultSkin = "vector";

## For attaching licensing metadata to pages, and displaying an
## appropriate copyright notice / icon. GNU Free Documentation
## License and Creative Commons licenses are supported so far.
$wgRightsPage = ""; # Set to the title of a wiki page that describes your license/copyright
$wgRightsUrl = "";
$wgRightsText = "";
$wgRightsIcon = "";

# Path to the GNU diff3 utility. Used for conflict resolution.
$wgDiff3 = "/usr/bin/diff3";

# Query string length limit for ResourceLoader. You should only set this if
# your web server has a query string length limit (then set it to that limit),
# or if you have suhosin.get.max_value_length set in php.ini (then set it to
# that value)
$wgResourceLoaderMaxQueryLength = -1;

# End of automatically generated settings.
# Add more configuration options below.

</pre>

== copy/backup wiki ==
on main/primary wiki:
<pre>
/usr/local/etc/rc.d/mysql-server stop
ssh 69.55.230.18 "/etc/init.d/mysql stop"
rsync -av /var/db/mysql/my_wiki/ 69.55.230.18:/var/lib/mysql/my_wiki/
rsync -av /var/db/mysql/ib* 69.55.230.18:/var/lib/mysql/
/usr/local/etc/rc.d/mysql-server start
ssh 69.55.230.18 "/etc/init.d/mysql start"

</pre>

Jail Server Install

2013-09-25T17:48:42Z

24.165.90.109: /* populate /etc/rc.conf with IPs and service settings */

= FreeBSD 9.x =

== Assumptions ==

Setup instructions below assume this is DELL 2950 with an LSI-based SAS RAID card.

Server is at castle, connected to pub, private, serial and DRAC

Assuming OS loading done via IPKVM with ISO mounted via USB

Assumes at 4 drives, 2 mirrors

== Configure server BIOS ==

setup console redirect, speed 115200

set LCD string to name of server "jail8"

set date to GMT

go into RAID bios and setup mirrors

== Setup DRAC ==
[[DRAC/RMM#DRAC_setup|DRAC setup]]

== Install OS (sysinstall) ==

boot to bootonly disk for AMD version of FreeBSD, i.e. <tt>FreeBSD-8.3-RELEASE-amd64-bootonly.iso</tt>

when the install menu appears, choose <tt>custom install</tt>

=== partition menu ===
move cursor to mfid0, hit space (takes you to partition map screen). If there is only 1 mirror, there will be no option to select a specifit drive: mfid0 will be selected.

type 'a' to use entire disk<br>
type 'q' to quit and save<br>
choose 'freebsd standard mbr'

space to '''un'''select mfid0<br>
cursor down to mfid1<br>
hit space

type 'a' to use entire disk<br>
type 'q' to quit and save<br>
choose 'none' for boot mgr (leave untouched)

cursor over mfid0<br>
space<br>
(takes you into part. Screen again) 'q' to exit<br>
none for boot mgr<br>

Make sure both drives (mfid0 and mfid1) are checked and tab to ok

=== Label menu ===

Make sure mfid0 is highlighted at the top of the screen, setup the following partitions

/ 512M<br>
swap 6G<br>
/var 1G <br>
/tmp 256M<br>
/usr 8G<br>
/mnt/data1 remaining space

All partitions except / should be setup for soft updates. If not, type 's' to enable for soft updates on all except for / (should look like UFS2+S Y under the Newfs column)

move cursor to mfid1 at the top of the screen<br>
swap 8G (or 4G if there’s a 3rd drive)<br>
/mnt/data2 remaining space <br>
'q' to save and exit

=== distributions ===
Choose the following distribudions

* developer (ok to install ports)
* custom -> lib32
exit

=== media ===
if you are installing via a cd, no need to enter this menu or change anything. Otherwise, choose ftp to install via ftp. You will be prompted to setup networking. You will need to choose a nic (typically bce0 or bce1). Say no to DHCP and IPv6. Hopefully the public nic cable was installed in bce0 so start with that nic and provide the hostname, (public) IP, netmask, gateway and DNS. When configured, it should start pinging. If it doesn't, have the NOC swap cables. Select any FTP server, usually Main or ftp4.

=== commit ===

this usually takes 12mins<br>
during the process you may need to select a new ftp mirror, this is not a problem.<br>
at the conclution of the install you will be prompted to enter the root password (2x) and returned to the configuration menu.

=== add user ===

'''Add''' user 'user'. Defaults for everything is fine, just remember to enter 'wheel' in the member group field.
Do set the password.

=== Setup timezone ===

PT

=== Networking ===
page down to the bottom and enable '[X]' sshd

If you installed via cd, you will need to visit:<br>
interfaces->bce0-><br>
No IPV6<br>
dhcp=no<br>
Set hostname, IP, DNS, gateway<br>
(i.e. setup the nic as indicated above)

Exit the install and if you installed via CD, take it out and let the machine reboot

== Configure OS, kernel, userland, jail ==

=== double check the date/time ===

date

=== populate /etc/resolv.conf ===

echo "search johncompanies.com \
nameserver 69.55.225.225 \
nameserver 69.55.230.3 \
nameserver 69.55.229.3" > /etc/resolv.conf

=== edit /etc/make.conf ===
echo "WITHOUT_X11=yes \
KERNCONF=jail4 \
BOOT_COMCONSOLE_SPEED=115200" >> /etc/make.conf

=== setup bootloader for console, etc ===

add settings to /boot/loader.conf and /boot.config:

echo "-Dh" >> /boot.config

echo 'console="comconsole,vidconsole" \
boot_multicons="YES" \
boot_serial="YES" \
mfi_linux_load="YES" \
comconsole_speed="115200"' >> /boot/loader.conf

=== enable login via serial console ===
turn off all ttyv's except 0 and 1 in /etc/ttys and turn on ttyd0, change type to vt100:

vi /etc/ttys

The changed lines should look like:

ttyv2 "/usr/libexec/getty Pc" cons25 off secure
ttyv3 "/usr/libexec/getty Pc" cons25 off secure
ttyv4 "/usr/libexec/getty Pc" cons25 off secure
ttyv5 "/usr/libexec/getty Pc" cons25 off secure
ttyv6 "/usr/libexec/getty Pc" cons25 off secure
ttyv7 "/usr/libexec/getty Pc" cons25 off secure
# Serial terminals
# The 'dialup' keyword identifies dialin lines to login, fingerd etc.
ttyu0 "/usr/libexec/getty std.9600" vt100 on secure

Restart init
kill -1 1

At this point you should have a login on console.

To configure serial console access, login to the console server as root and run:

# vi /etc/remote

following examples there, rename port to server's hostname, depending on where and which digi box this server is plugged into. Make sure to get speed right too: 115200

=== populate hosts ===

If server is at castle:
echo "10.1.4.3 backup2" >> /etc/hosts
echo "10.1.4.8 backup1" >> /etc/hosts

If server is at i2b:
echo "69.55.230.10 backup2" >> /etc/hosts
echo "10.1.2.3 backup3" >> /etc/hosts
echo "69.55.230.11 backup1" >> /etc/hosts

=== create ssh key, upload to backup servers ===
cd
ssh-keygen -t dsa -b 1024
(default location, leave password blank)

If server is at castle:
cat /root/.ssh/id_dsa.pub | ssh backup2 'cat - >> /root/.ssh/authorized_keys'
cat /root/.ssh/id_dsa.pub | ssh backup1 'cat - >> /root/.ssh/authorized_keys'

If server is at i2b:
cat /root/.ssh/id_dsa.pub | ssh backup3 'cat - >> /root/.ssh/authorized_keys'
cat /root/.ssh/id_dsa.pub | ssh backup2 'cat - >> /root/.ssh/authorized_keys'
cat /root/.ssh/id_dsa.pub | ssh backup1 'cat - >> /root/.ssh/authorized_keys'

confirm that you can ssh to backup2 and backup1 without getting a login prompt

ssh backup2 hostname
ssh backup1 hostname
ssh backup3 hostname

=== create & populate binaries/scripts dirs ===
<pre>mkdir -p /usr/local/jail/bin
mkdir -p /usr/local/jail/rc.d
mkdir -p /usr/local/jail/template/
mkdir /mnt/data1
mkdir /mnt/data2
scp backup2:"/mnt/data4/bin/freebsd9.x/*" /usr/local/jail/bin
cd /usr/local/jail/rc.d/
touch quad1
touch deprecated
chmod +x *
cd /usr/local/jail/bin
ln -s /usr/local/jail/rc.d/quad1 quad1
ln -s /usr/local/jail/bin/jailmake_md jailmake
ln -s /usr/local/jail/bin/js_md js
ln -s /usr/local/jail/bin/canceljail_md canceljail
ln -s /usr/local/jail/bin/jailmakeempty_md jailmakeempty
ln -s /usr/local/jail/bin/postboot_md postboot
ln -s /usr/local/jail/bin/preboot_md preboot
ln -s /usr/local/jail/bin/startjail_md startjail
ln -s /usr/local/jail/bin/stopjail_md stopjail

rehash</pre>

=== edit root's path and login script ===

vi /root/.cshrc

Change alias entries (add G):

<pre>alias la ls -aG
alias lf ls -FAG
alias ll ls -lAG
alias ls ls -AG
alias mbm mb mount
alias mbu mb umount
alias cjb cd /usr/local/jail/bin
alias cd1 cd /mnt/data1
alias cd2 cd /mnt/data2
alias cd3 cd /mnt/data3
alias jtop jtop lj
alias j jobs</pre>

add to path be careful to leave a space after bin and make sure the wrapping isn't broken:
/usr/local/jail/bin

alter the prompt, set the following:
set prompt = "$user@`/bin/hostname -s` %/# "

at the bottom of the file add:
<pre>set sshtty=`who am i|awk '{print $2}'`
/usr/sbin/rtprio 3 -`psj | grep $sshtty | awk '{print $2}'`

set shortty=`who am i | awk '{print $2}' | sed -E 's/.*(..)$/\1/'`
foreach x (`psj | grep sh | grep $shortty | awk '{print $2}'`)
/usr/sbin/rtprio 2 -$x
end</pre>

Make the new settings active in current shell:
source /root/.cshrc

=== update ports ===

portsnap fetch
portsnap extract

To update later on:
portsnap fetch
portsnap update

=== install svn ===

<pre>
setenv PACKAGESITE "ftp://ftp4.freebsd.org/pub/FreeBSD/ports/amd64/packages-9-current/Latest/"
pkg_add subversion
</pre>

=== get latest sources for this release ===

<pre>
cd /usr

# mv src/ src.orig
tar cvzf src.orig.tgz src
rm -fr src/*
svn checkout svn://svn.FreeBSD.org/base/stable/9 /usr/src
</pre>

To update:
make update SVN_UPDATE=yes

=== configure new kernel ===

Pull down the kernel config we are using for this distribution. In this case we use an 8.2 kernel config on 8.3, which is valid. The local file should be the same name as host- <tt>jail4</tt> in this example

<pre>cd /usr/src/sys/amd64/conf
scp backup2:/mnt/data4/build/freebsd/kern_config-9.1-amd64 ./jail4</pre>

edit the kernel config and change ident to be the name of the jail:
vi jail4
ident jail4

Optional, edit <tt>/sys/conf/newvers.sh</tt> to add <tt>–jc2</tt> to the end of the BRANCH string (RELEASE-jc2)
vi /sys/conf/newvers.sh

notes on kernel configuring: http://www.FreeBSD.org/doc/en_US.ISO8859-1/books/handbook/kernelconfig-config.html

=== install patches ===

We don't have any patches right now. Refer to older FreeBSD version build docs on how that is/was done.

=== build, install kernel and world ===

Rename current generic kernel so it will always be available to boot from. Save room by removing non-needed kernel modules:

<pre>cd /boot
mv kernel kernel.GENERIC
cd kernel.GENERIC
mkdir hold
mv mfi_linux.ko hold/
mv linux.ko hold/
mv linprocfs.ko hold/
mv linsysfs.ko hold/
mv geom_vinum.ko hold/
mv geom_concat.ko hold/
mv zfs.* hold/
mv opensolaris* hold/

rm *.ko
rm *.symbols
mv hold/* .
rmdir hold/

Note on -DWITHOUT_CLANG: try to do it without including that directive, it may work for you.

cd /usr/src
make buildworld KERNCONF=jail4; mail -s 'buildworld done' dave.boodman@vtext.com < /dev/null

make -DWITHOUT_CLANG buildworld KERNCONF=jail4; mail -s 'buildworld done' dave.boodman@vtext.com < /dev/null
</pre>
~4.5hr

cd /usr/src
make buildkernel installkernel

mergemaster -p
You will be prompted to merge, replace or ignore files changed by the src update. In most cases you can '''d'''elete the temp (new) files.

make -DWITHOUT_CLANG installworld
~34min

ONLY if this will be a zfs system (not currently used in 8.x):
<pre>cd /sys/modules/zfs
make
make install
cd /sys/modules/opensolaris
make
make install</pre>

=== populate devfs ruleset ===
scp backup2:/mnt/data4/build/freebsd/devfs.rules.8x /etc/devfs.rules

=== populate /etc/rc.conf with IPs and service settings ===

<pre>vi /etc/rc.conf

kern_securelevel_enable="NO"
portmap_enable="NO"
sendmail_enable="NO"
usbd_enable="YES"

nfs_client_enable="YES"
nfs_reserved_port_only="YES"
inetd_enable="YES"
inetd_flags="-wW -a 10.1.4.XXX"
devfs_system_ruleset="devfsrules_show_all"

ifconfig_bce1="inet 10.1.4.XXX netmask 255.255.255.0"
ifconfig_bce0="inet 69.55.2XX.XXX netmask 255.255.255.0"
#ifconfig_bce0_alias0="inet 69.55.2XX.XXX netmask 255.255.255.0"

fsck_y_enable="YES"
background_fsck="NO"
#rc_mfi_raid_tty_log="YES"
#zfs_enable="YES"</pre>

Modify IPs, hostname, gateway for this box.

=== make sure sysctls are set and preserved upon boot ===

<pre>echo "kern.consmute=0\
kern.ipc.shm_use_phys=1\
kern.ipc.shmall=131070\
kern.ipc.shmmax=134217728\
net.inet.tcp.syncookies=0\
kern.maxfiles=32768\
kern.fallback_elf_brand=3\
kern.maxprocperuid=4000\
security.jail.sysvipc_allowed=1\
security.jail.allow_raw_sockets=1\
security.jail.socket_unixiproute_only=1\
security.jail.chflags_allowed=0\
dev.amr.0.allow_volume_configure=1\
compat.linux.osrelease=2.6.12\
vm.pmap.shpgperproc=500\
security.bsd.unprivileged_read_msgbuf=0\
kern.maxvnodes=400000" >> /etc/sysctl.conf</pre>

Tuning note: watch vfs.numvnodes while the server is live to get guidance on where to set maxvnodes

=== mount procfs ===
echo "proc /proc procfs rw 0 0" >> /etc/fstab
echo "linprocfs /usr/compat/linux/proc linprocfs rw 0 0" >> /etc/fstab
echo "linsysfs /usr/compat/linux/sys linsysfs rw 0 0" >> /etc/fstab
mkdir -p /usr/compat/linux/proc
mkdir -p /usr/compat/linux/sys

=== enable noatime option ===

grep data /etc/fstab

data1 and data2 should look something like (add ',noatime' after 'rw'):

/dev/mfid0s1g /mnt/data1 ufs rw,noatime 2 2
/dev/mfid1s1d /mnt/data2 ufs rw,noatime 2 2

=== reboot. Confirm new kernel is loaded, devfs in place ===

uname -a

Check devfs rules
devfs rule showsets
devfs rule -s 3 show

Should see:
<pre># devfs rule showsets
1
2
3
4

# devfs rule -s 3 show
100 include 1
207 path pts* unhide
217 path fd unhide
218 path fd/* unhide
300 path ttyp* unhide
301 path ttyq* unhide
302 path ttyr* unhide
303 path ttys* unhide
304 path ttyP* unhide
305 path ttyQ* unhide
306 path ttyR* unhide
307 path ttyS* unhide
400 path null unhide
500 path zero unhide
600 path random unhide
610 path urandom unhide
700 path mem unhide
710 path kmem unhide
810 path mdctl unhide
900 path stdin unhide
910 path stdout unhide
920 path stderr unhide</pre>

=== Install raid mgmt tool ===

==== Perc5/i, 6/i ====
Pull over cli from previous system (jail9)
scp root@10.1.4.109:"/usr/local/sbin/mega*" /usr/local/sbin/
scp root@10.1.4.109:/usr/local/libexec/MegaCli /usr/local/libexec/MegaCli

These are linux-based tools. This will require linux base...which you might install via:

pkg_add -r linux_base

Test:
rehash; megacli ldinfo lall a0
or
megarc -ldInfo -a0 -Lall
(2850)

however, linux does seem to be installed already so we don't need to pkg_add or port install or rsync anything over from a current system.

==== 2850 PERC 4e/Di- no linux ====
<pre>cd /usr/ports/distfiles/
fetch http://backup01.best-hosting.ru/pub/FreeBSD/ports/distfiles/dr_freebsd_1.51.zip
cd /usr/ports/sysutils/megarc
make install clean
megarc -dispCfg -a0</pre>

=== install rsync from ports ===
cd /usr/ports/net/rsync
make install clean

choose default options

=== configure inetd to respond to mrtg load queries ===
echo "load stream tcp nowait user /usr/local/jail/bin/load.pl load.pl" >> /etc/inetd.conf
echo "load 12384/tcp" >> /etc/services

=== install perl ===
cd /usr/ports/lang/perl5.12
make install clean

choose defaults

=== install bb client ===

Compiling from source on AMD64 will not work. So, we use a linux-compiled version and rely on linux compat.

adduser

Output/response:

<pre>Username: bb
Full name: bb
Uid (Leave empty for default): 1984
Login group [bb]:
Login group is bb. Invite bb into other groups? []:
Login class [default]:
Shell (sh csh tcsh nologin) [sh]:
Home directory [/home/bb]:
Use password-based authentication? [yes]:
Use an empty password? (yes/no) [no]:
Use a random password? (yes/no) [no]: yes
Lock out the account after creation? [no]:
Username : bb
Password : <random>
Full Name : bb
Uid : 1984
Class :
Groups : bb
Home : /home/bb
Shell : /bin/sh
Locked : no
OK? (yes/no): yes</pre>

cd /usr/home/bb
scp backup2:/mnt/data4/build/bb/bb-freebsd_linuxcompat.tgz .
tar xzf bb-freebsd_linuxcompat.tgz

edit /home/bb/bbc1.9e-btf/etc/bb-hosts with something like:

echo "10.1.4.5 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh \
10.1.4.'''104 jail4'''.johncompanies.com # ssh" > /home/bb/bbc1.9e-btf/etc/bb-hosts

Edit for machine name and private IP.

if this machine is at i2b:
echo "69.55.230.2 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh \
69.55.228.104 jail4.johncompanies.com # ssh" > /home/bb/bbc1.9e-btf/etc/bb-hosts

<pre>vi /home/bb/bbc1.9e-btf/ext/openfiles

MACHINE="jail4,johncompanies,com" # HAS TO BE IN A,B,C FORM</pre>
Edit for machine name.

<pre>cd /usr/home/bb/bbc1.9e-btf/etc
./bbchkcfg.sh
(y to questions)
./bbchkhosts.sh
(ignore ssh errors)
cd ../..
chown -R bb .
su bb
cd
cd bbc1.9e-btf
./runbb.sh start
more BBOUT
(look for errors)
exit</pre>

Put in script to start bb @ boot:
echo 'su - bb -c "cd /home/bb/bbc1.9e-btf; ./runbb.sh start"' > /usr/local/etc/rc.d/bb.sh
chmod +x /usr/local/etc/rc.d/bb.sh

=== remove reserve space ===
<pre>cd
umount /mnt/data1
umount /mnt/data2
tunefs -m 0 /mnt/data1
tunefs -m 0 /mnt/data2
mount -a</pre>

=== setup rdate ===
cd /usr/ports/sysutils/rdate
make install clean

crontab -e
0 0 * * * /usr/local/sbin/rdate -s utcnist.colorado.edu

/usr/local/sbin/rdate -s utcnist.colorado.edu

We used to use ntpd, however it listens on jail IPs which is a security risk. So we stopped. Here's the old instructions:
install new ntp from ports
/usr/ports/net/ntp
make install clean

<pre>echo "server 10.1.4.5" > /etc/ntp.conf
/usr/sbin/ntpd -p /var/run/ntpd.pid
sleep 2; ntpq -p</pre>(confirm it’s able to reach our time server)

=== make a data partition ===

IF you didn't format the data partition during sysinstall:

Create a g partition on 2nd mirror – bsdlabel no longer works (below shows d partition made with sysinstall):

<pre>jail8 /usr/home/bb# gpart show
=> 63 285474735 mfid0 MBR (136G)
63 285458922 1 freebsd [active] (136G)
285458985 15813 - free - (7.7M)

=> 0 285458922 mfid0s1 BSD (136G)
0 524288 1 freebsd-ufs (256M)
524288 12582912 2 freebsd-swap (6.0G)
13107200 524288 4 freebsd-ufs (256M)
13631488 524288 5 freebsd-ufs (256M)
14155776 8388608 6 freebsd-ufs (4.0G)
22544384 262914538 7 freebsd-ufs (125G)

=> 63 584843175 mfid1 MBR (279G)
63 584830197 1 freebsd [active] (279G)
584830260 12978 - free - (6.3M)

=> 0 584830197 mfid1s1 BSD (279G)
0 16777216 2 freebsd-swap (8.0G)
16777216 568052981 4 freebsd-ufs (271G)

jail8 /usr/home/bb# gpart show mfid1s1
=> 0 584830197 mfid1s1 BSD (279G)
0 16777216 2 freebsd-swap (8.0G)
16777216 568052981 4 freebsd-ufs (271G)

# gpart list mfid1s1

Geom name: mfid1s1
fwheads: 255
fwsectors: 63
last: 584830196
first: 0
entries: 8
scheme: BSD
Providers:
1. Name: mfid1s1b
Mediasize: 8589934592 (8.0G)
Sectorsize: 512
Mode: r1w1e0
rawtype: 1
length: 8589934592
offset: 0
type: freebsd-swap
index: 2
end: 16777215
start: 0
2. Name: mfid1s1d
Mediasize: 290843126272 (271G)
Sectorsize: 512
Mode: r0w0e0
rawtype: 7
length: 290843126272
offset: 8589934592
type: freebsd-ufs
index: 4
end: 584830196
start: 16777216
Consumers:
1. Name: mfid1s1
Mediasize: 299433060864 (279G)
Sectorsize: 512
Mode: r1w1e1

# gpart delete -i 4 mfid1s1
mfid1s1d deleted
jail8 /usr/home/bb# gpart list mfid1s1
Geom name: mfid1s1
fwheads: 255
fwsectors: 63
last: 584830196
first: 0
entries: 8
scheme: BSD
Providers:
1. Name: mfid1s1b
Mediasize: 8589934592 (8.0G)
Sectorsize: 512
Mode: r1w1e0
rawtype: 1
length: 8589934592
offset: 0
type: freebsd-swap
index: 2
end: 16777215
start: 0
Consumers:
1. Name: mfid1s1
Mediasize: 299433060864 (279G)
Sectorsize: 512
Mode: r1w1e1

# gpart show mfid1s1
=> 0 584830197 mfid1s1 BSD (279G)
0 16777216 2 freebsd-swap (8.0G)
16777216 568052981 - free - (271G)

# gpart add -t freebsd-ufs -i 7 mfid1s1
mfid1s1g added

# gpart show mfid1s1
=> 0 584830197 mfid1s1 BSD (279G)
0 16777216 2 freebsd-swap (8.0G)
16777216 568052981 7 freebsd-ufs (271G)</pre>

Here's how we USED to do it with bsdlabel:

bsdlabel -e /dev/mfid0s1

given:
<pre># /dev/aacd0s1:
8 partitions:
# size offset fstype [fsize bsize bps/cpg]
a: 262144 0 4.2BSD 2048 16384 16392
b: 4194304 262144 swap
c: 143363997 0 unused 0 0 # "raw" part, don't edit
d: 524288 4456448 4.2BSD 2048 16384 32776
e: 524288 4980736 4.2BSD 2048 16384 32776
f: 6291456 5505024 4.2BSD 2048 16384 28552</pre>

new offset = 6291456 + 5505024 = 11796480<br>
new size is size for 'c' partition minus the new start from above<br>
143363997 - 11796480 = 131567517

So:<br>
g: 131567517 11796480 unused 0 0

=== create the jail template ===

cd /usr/ports/sysutils/jailutils
make install clean

Create an md device to hold the jail:
<pre>touch /mnt/data1/jail-template20g
mdconfig -a -t vnode -s 20g -f /mnt/data1/jail-template20g -u 0
newfs -O 1 /dev/md0
mkdir /mnt/data1/jail-DIR
mount /dev/md0 /mnt/data1/jail-DIR</pre>

Build world into the jail:
<pre>cd /usr/src
make world DESTDIR=/mnt/data1/jail-DIR; taskdone</pre>
~4.5 hr

Make /etc into the jail, mount dev, copy in jkill:
<pre>cd etc
make distribution DESTDIR=/mnt/data1/jail-DIR
mount -t devfs devfs /mnt/data1/jail-DIR/dev
devfs -m /mnt/data1/jail-DIR/dev rule -s 3 applyset
cd /mnt/data1/jail-DIR
ln -sf dev/null kernel
cp /usr/local/sbin/jkill /mnt/data1/jail-DIR/sbin</pre>

Enter into jail to do configuration:
jail /mnt/data1/jail-DIR testhostname 192.168.11.100 /bin/sh
csh

Create fstab:
<pre>touch /etc/fstab
echo 'network_interfaces=""\
hostname="newsystem"\
kern_securelevel_enable="NO"\
sendmail_enable="YES"\
sshd_enable="YES"' > /etc/rc.conf

echo "nameserver 69.55.225.225\
nameserver 69.55.230.3" >> /etc/resolv.conf</pre>

Edit crontab:
vi /etc/crontab
remove the adjkerntz lines
comment out periodic’s and put this line above them:
# IF YOU UNCOMMENT THESE, PLEASE ADJUST THEIR RUN TIME

rm -rf /etc/periodic/daily/400.status-disks

check and remove any crap in /tmp

vi /etc/periodic/security/100.chksetuid
replace: <tt>MP=`mount -t ufs | grep -v " nosuid" | awk '{ print $3 }' | sort`</tt><br>
with: <tt>MP='/'</tt> (use single quotes)

mkdir -p /usr/compat/linux/dev

Add account for user. Output/response:
<pre>adduser

Username: user
Full name: user
Uid (Leave empty for default):
Login group [user]:
Login group is user. Invite user into other groups? []: wheel
Login class [default]:
Shell (sh csh tcsh nologin) [sh]:
Home directory [/home/user]:
Home directory permissions (Leave empty for default):
Use password-based authentication? [yes]:
Use an empty password? (yes/no) [no]:
Use a random password? (yes/no) [no]: y
Lock out the account after creation? [no]:
Username : user
Password : <random>
Full Name : user
Uid : 1001
Class :
Groups : user
Home : /home/user
Home Mode :
Shell : /bin/sh
Locked : no
OK? (yes/no): y
adduser: INFO: Successfully added (user) to the user database.
adduser: INFO: Password for (user) is: 901gmYjO
Add another user? (yes/no): n
Goodbye!</pre>

set TERM:
vi /usr/home/user/.profile
TERM=vt100; export TERM

Set time zone to PT:
tzsetup

Reload aliases:
newaliases

Replace reboot/halt:
rm /sbin/halt /sbin/reboot
ln /sbin/jkill /sbin/halt
ln /sbin/jkill /sbin/reboot

Redirect console output, comment out console and move to /var/log/messages:
vi /etc/syslog.conf
#*.err;kern.warning;auth.notice;mail.crit /dev/console
*.err;kern.warning;auth.notice;mail.crit /var/log/messages

exit
exit

Lib32 compat library:
cd libexec
chflags noschg ld-elf32.so.1
chflags noschg ld-elf.so.1
mv ld-elf32.so.1 ld-elf32.so.1-orig
ln ld-elf.so.1 ld-elf32.so.1
chflags schg ld-elf.so.1
chflags schg ld-elf32.so.1

Replace traceroute:
mv /mnt/data1/jail-DIR/usr/sbin/traceroute /mnt/data1/jail-DIR/usr/sbin/_traceroute
echo '#\!/bin/sh\
/usr/sbin/_traceroute -i bce0 $1' >> /mnt/data1/jail-DIR/usr/sbin/traceroute
chmod +x /mnt/data1/jail-DIR/usr/sbin/traceroute

Modify 'bce0' to reflect whichever nic is public on this hardware.

Clean out ports before copying into the jail (~30mins):
cd /usr/ports
make -DNOCLEANDEPENDS clean
rm -fr /usr/ports/distfiles/*
cp -r /usr/ports /mnt/data1/jail-DIR/usr

rm /mnt/data1/jail-DIR/root/.history

Umount the jail and dump it:
<pre>cd /
umount /mnt/data1/jail-DIR/dev
dump -0a -f /usr/local/jail/template/template /dev/md0
umount /dev/md0
rmdir /mnt/data1/jail-DIR
mdconfig -d -u 0</pre>

=== setup backups ===
<pre>echo '#\!/bin/sh\
backupdir=/data/jail3\
server=backup1\
\
## ENTRY /etc\
## ENTRY /usr/local/etc\
## ENTRY /usr/local/jail\
## ENTRY /root/logs' > /usr/local/jail/bin/backup.config</pre>

Edit to reflect backup server and jail hostname

On backup server, setup backup dirs:
ssh backup1 mkdir -p /data/jail3/0

On backup server, setup backup dirs:
backup1# vi /usr/local/sbin/snapshot_rotate

Edit /usr/local/jail/bin/backup.xxx to use the right drives and copy to /usr/local/jail/bin/backup
vi /usr/local/jail/bin/backup.md
adjust df so it includes all relevant drives, currently <tt>df > /etc/df.bak</tt> is fine. Also, make sure the binary/script source is pulling from the right dir on backup2:
/usr/local/bin/rsync -a backup2:/mnt/data4/bin/freebsd8.x/ /usr/local/jail/bin/

cp /usr/local/jail/bin/backup.md /usr/local/jail/bin/backup

=== create /root/logs ===
mkdir /root/logs

=== edit sshd_config ===
vi /etc/ssh/sshd_config
ListenAddress 69.55.'''229.7'''
ListenAddress 10.1.4.'''103'''

Adjust to pub/private IP for jail.

Restart sshd:
kill -1 `cat /var/run/sshd.pid`

=== add crontab entries ===
<pre>crontab -e
5 0 * * * /usr/local/jail/bin/backup
1 0 1 * * /usr/local/jail/bin/ipfwreset
0 18 * * * /usr/local/jail/bin/ipfwbackup
4,9,14,19,24,29,34,39,44,49,55,59 * * * * /usr/local/jail/bin/trafstats
0 0,6,12,18 * * * /usr/local/jail/bin/sync_jail_names</pre>

On 2950:
*/5 * * * * /usr/local/jail/bin/perc5iraidchk

On 3ware-based servers:
0 0 * * * /usr/local/jail/bin/3wraidchk

=== Reboot notify script ===
ln -s /usr/local/jail/bin/notify.sh /usr/local/etc/rc.d/notify.sh

=== patch jail against starting jails with rtprio ===

<pre>mv /usr/sbin/jail /usr/sbin/jail_
echo '#\!/bin/sh\
/usr/sbin/rtprio -t /usr/sbin/jail_ $*' > /usr/sbin/jail
chmod +x /usr/sbin/jail</pre>

=== make sure mail works ===
If there are map errors:
cd /etc/mail; make maps

=== recover space on /usr (optional) ===

rm -fr /usr/obj

=== wrapper for jps ===

mv /usr/local/sbin/jps /usr/local/sbin/jps_

=== wrapper for jls ===

mv /usr/sbin/jls /usr/sbin/jls_

=== wrapper for jexec ===

mv /usr/sbin/jexec /usr/sbin/jexec_

=== install jtop ===

cd /usr/ports/sysutils/jtop
make install clean

=== block jails from reaching private net ===
echo 'ipfw add 1 deny ip from 69.55.224.0/20 to 10.1.4.0/24' > /usr/local/etc/rc.d/ipfw.sh
chmod 0700 /usr/local/etc/rc.d/ipfw.sh

== add to management infrastructure ==

=== mail ===

==== add to management db ====

tables: jc.ref_machines and jc.ref_templates

on jail run:
uname -r

Which shows something like:
8.3-RELEASE-p2

Insert into db:

<pre>insert into ref_machines values (null,'jail3','mx3.johncompanies.com',0,'f8');
select machine_id from ref_machines where host='jail3';
+------------+
| machine_id |
+------------+
| 35 |
+------------+
insert into ref_templates values ('','8.3-RELEASE-jc2',35,'FreeBSD 8.3',1);</pre>

==== add to bb server ====
vi /usr/home/bb/bbsrc/bb1.9i-btf/etc/bb-hosts
10.1.4.'''109 jail9'''.johncompanies.com # ssh

In the case of an i2b server, use real ip:
69.55.'''229.7 jail3'''.johncompanies.com # ssh

<pre>su bb
cd
bbsrc/bb/runbb.sh restart ; exit</pre>

==== Update backupgraph ====
vi /usr/local/www/mgmt/cgi/backupgraph.pl
(add hostname)

==== Update load mrtg ====
vi /usr/local/www/mgmt/mrtg/mrtg1.cfg
(add new entry to file following existing format)

=== ns1c ===
fwd and reverse lookups:
vr johncompanies.com

vi internal.johncompanies.com
rndc reload johncompanies.com IN private

ptr 69.55.227.x

=== wiki ===
add to server/cabinet map

=== firewall ===
add an outside blocking rule to the firewall, so this machine can only be reached from inside the firewall. Follow example already in firewall jail17 is:

00117 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 or 69.55.238.150 } to 69.55.228.200
00117 deny ip from any to 69.55.228.200

jail19 would be 00119...
ipfw add 00109 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 } to 69.55.232.3
ipfw add 00109 deny ip from any to 69.55.232.3

=== select some customers for castle probe map ===

= FreeBSD 8.x =

== Assumptions ==

Setup instructions below assume this is DELL 2950 with an LSI-based SAS RAID card.

Server is at castle, connected to pub, private, serial and DRAC

Assuming OS loading done via IPKVM with ISO mounted via USB

Assumes at 4 drives, 2 mirrors

== Configure server BIOS ==

setup console redirect, speed 115200

set LCD string to name of server "jail8"

set date to GMT

go into RAID bios and setup mirrors

configure DRAC: TODO

== Install OS (sysinstall) ==

boot to bootonly disk for AMD version of FreeBSD, i.e. <tt>FreeBSD-8.3-RELEASE-amd64-bootonly.iso</tt>

when the install menu appears, choose <tt>custom install</tt>

=== partition menu ===
move cursor to mfid0, hit space (takes you to partition map screen). If there is only 1 mirror, there will be no option to select a specifit drive: mfid0 will be selected.

type 'a' to use entire disk<br>
type 'q' to quit and save<br>
choose 'freebsd standard mbr'

space to '''un'''select mfid0<br>
cursor down to mfid1<br>
hit space

type 'a' to use entire disk<br>
type 'q' to quit and save<br>
choose 'none' for boot mgr (leave untouched)

cursor over mfid0<br>
space<br>
(takes you into part. Screen again) 'q' to exit<br>
none for boot mgr<br>

Make sure both drives (mfid0 and mfid1) are checked and tab to ok

=== Label menu ===

Make sure mfid0 is highlighted at the top of the screen, setup the following partitions

/ 512M<br>
swap 6G<br>
/var 256M <br>
/tmp 256M<br>
/usr 5G<br>
/mnt/data1 remaining space

All partitions except / should be setup for soft updates. If not, type 's' to enable for soft updates on all except for / (should look like UFS2+S Y under the Newfs column)

move cursor to mfid1 at the top of the screen<br>
swap 8G (or 4G if there’s a 3rd drive)<br>
/mnt/data2 remaining space <br>
'q' to save and exit

=== distributions ===
Choose the following distribudions

* developer (ok to install ports)
* custom -> lib32
exit

=== media ===
if you are installing via a cd, no need to enter this menu or change anything. Otherwise, choose ftp to install via ftp. You will be prompted to setup networking. You will need to choose a nic (typically bce0 or bce1). Say no to DHCP and IPv6. Hopefully the public nic cable was installed in bce0 so start with that nic and provide the hostname, (public) IP, netmask, gateway and DNS. When configured, it should start pinging. If it doesn't, have the NOC swap cables. Select any FTP server, usually Main or ftp4.

=== commit ===

this usually takes 12mins<br>
during the process you may need to select a new ftp mirror, this is not a problem.<br>
at the conclution of the install you will be prompted to enter the root password (2x) and returned to the configuration menu.

=== add user ===

'''Add''' user 'user'. Defaults for everything is fine, just remember to enter 'wheel' in the member group field.
Do set the password.

=== Setup timezone ===

PT

=== Networking ===
page down to the bottom and enable '[X]' sshd

If you installed via cd, you will need to visit:<br>
interfaces->bce0-><br>
No IPV6<br>
dhcp=no<br>
Set hostname, IP, DNS, gateway<br>
(i.e. setup the nic as indicated above)

Exit the install and if you installed via CD, take it out and let the machine reboot

== Configure OS, kernel, userland, jail ==

=== double check the date/time ===

=== populate /etc/resolv.conf ===

echo "nameserver 69.55.225.225\
nameserver 69.55.230.3
nameserver 69.55.229.3" > /etc/resolv.conf

=== edit /etc/make.conf ===
echo "WITHOUT_X11=yes \
KERNCONF=jail3 \
BOOT_COMCONSOLE_SPEED=115200" >> /etc/make.conf

=== setup bootloader for console, etc ===

add settings to /boot/loader.conf and /boot.config:

echo "-Dh" >> /boot.config

echo 'console="comconsole,vidconsole" \
boot_multicons="YES" \
boot_serial="YES" \
mfi_linux_load="YES" \
comconsole_speed="115200"' >> /boot/loader.conf

=== enable login via serial console ===
turn off all ttyv's except 0 and 1 in /etc/ttys and turn on ttyd0, change type to vt100:

vi /etc/ttys

The changed lines should look like:

ttyv2 "/usr/libexec/getty Pc" cons25 off secure
ttyv3 "/usr/libexec/getty Pc" cons25 off secure
ttyv4 "/usr/libexec/getty Pc" cons25 off secure
ttyv5 "/usr/libexec/getty Pc" cons25 off secure
ttyv6 "/usr/libexec/getty Pc" cons25 off secure
ttyv7 "/usr/libexec/getty Pc" cons25 off secure
# Serial terminals
# The 'dialup' keyword identifies dialin lines to login, fingerd etc.
ttyu0 "/usr/libexec/getty std.9600" vt100 on secure

Restart init
kill -1 1

At this point you should have a login on console.

To configure serial console access, login to the console server as root and run:

# vi /etc/remote

following examples there, rename port to server's hostname, depending on where and which digi box this server is plugged into. Make sure to get speed right too: 115200

=== populate hosts ===

If server is at castle:
echo "10.1.4.3 backup2" >> /etc/hosts
echo "10.1.4.8 backup1" >> /etc/hosts

If server is at i2b:
echo "69.55.230.10 backup2" >> /etc/hosts
echo "10.1.2.3 backup3" >> /etc/hosts
echo "69.55.230.11 backup1" >> /etc/hosts

=== create ssh key, upload to backup servers ===
cd
ssh-keygen -t dsa -b 1024
(default location, leave password blank)

If server is at castle:
cat /root/.ssh/id_dsa.pub | ssh backup2 'cat - >> /root/.ssh/authorized_keys'
cat /root/.ssh/id_dsa.pub | ssh backup1 'cat - >> /root/.ssh/authorized_keys'

If server is at i2b:
cat /root/.ssh/id_dsa.pub | ssh backup3 'cat - >> /root/.ssh/authorized_keys'
cat /root/.ssh/id_dsa.pub | ssh backup2 'cat - >> /root/.ssh/authorized_keys'
cat /root/.ssh/id_dsa.pub | ssh backup1 'cat - >> /root/.ssh/authorized_keys'

confirm that you can ssh to backup2 and backup1 without getting a login prompt

ssh backup2 hostname
ssh backup1 hostname
ssh backup3 hostname

=== create & populate binaries/scripts dirs ===
<pre>mkdir -p /usr/local/jail/bin
mkdir -p /usr/local/jail/rc.d
mkdir -p /usr/local/jail/template/
mkdir /mnt/data1
mkdir /mnt/data2
scp backup2:"/mnt/data4/bin/freebsd8.x/*" /usr/local/jail/bin
cd /usr/local/jail/rc.d/
touch quad1
touch deprecated
chmod +x *
cd /usr/local/jail/bin
ln -s /usr/local/jail/rc.d/quad1 quad1
ln -s /usr/local/jail/bin/jailmake_md jailmake
ln -s /usr/local/jail/bin/js_md js
ln -s /usr/local/jail/bin/canceljail_md canceljail
ln -s /usr/local/jail/bin/jailmakeempty_md jailmakeempty
ln -s /usr/local/jail/bin/postboot_md postboot
ln -s /usr/local/jail/bin/preboot_md preboot
ln -s /usr/local/jail/bin/startjail_md startjail
ln -s /usr/local/jail/bin/stopjail_md stopjail

rehash</pre>

=== edit root's path and login script ===

vi /root/.cshrc

Change alias entries (add G):

<pre>alias la ls -aG
alias lf ls -FAG
alias ll ls -lAG
alias ls ls -AG
alias mbm mb mount
alias mbu mb umount
alias cjb cd /usr/local/jail/bin
alias cd1 cd /mnt/data1
alias cd2 cd /mnt/data2
alias cd3 cd /mnt/data3
alias jtop jtop lj
alias j jobs</pre>

add to path be careful to leave a space after bin and make sure the wrapping isn't broken:
/usr/local/jail/bin

alter the prompt, set the following:
set prompt = "`/bin/hostname -s` %/# "

at the bottom of the file add:
<pre>set sshtty=`who am i|awk '{print $2}'`
/usr/sbin/rtprio 3 -`psj | grep $sshtty | awk '{print $2}'`

set shortty=`who am i | awk '{print $2}' | sed -E 's/.*(..)$/\1/'`
foreach x (`psj | grep sh | grep $shortty | awk '{print $2}'`)
/usr/sbin/rtprio 2 -$x
end</pre>

Make the new settings active in current shell:
source /root/.cshrc

=== install cvsup ===

<pre>cd /usr/ports/net/cvsup-without-gui
make install clean; rehash; mail -s 'cvs installed' support@johncompanies.com < /dev/null</pre>
stand by for gettext options (use defaults). this process takes approx 22mins- hence the email/page notice above.

=== get latest sources for this release ===

<pre>cd /usr/src
echo "*default host=cvsup4.freebsd.org\
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_8_3\
*default delete use-rel-suffix\
*default compress\
src-all" > sup</pre>

If you need to run stable (cause release is broken or some other reason) make the sup file look like:

<pre>echo "*default host=cvsup4.freebsd.org\
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_8\
*default delete use-rel-suffix\
*default compress\
src-all" > sup</pre>

cvsup sup ; mail -s 'cvs sup done' support@johncompanies.com < /dev/null

time varies, 10-20mins

=== configure new kernel ===

Pull down the kernel config we are using for this distribution. In this case we use an 8.2 kernel config on 8.3, which is valid. The local file should be the same name as host- <tt>jail3</tt> in this example

<pre>cd /usr/src/sys/amd64/conf
scp backup2:/mnt/data4/build/freebsd/kern_config-8.2-amd64 ./jail3</pre>

edit the kernel config and change ident to be the name of the jail:
vi jail3
ident jail3

Optional, edit <tt>/sys/conf/newvers.sh</tt> to add <tt>–jc2</tt> to the end of the BRANCH string (RELEASE-jc2)
vi /sys/conf/newvers.sh

notes on kernel configuring: http://www.FreeBSD.org/doc/en_US.ISO8859-1/books/handbook/kernelconfig-config.html

=== install patches ===

The various patches are in /mnt/data4/build/freebsd/patches on backup2. There are dirs for each version. Not all dirs are populated, but patches for later versions work on older ones unless there is a new patch in the older dir.

there are no patches we use for 8.x, but here would be the commands:

cd /usr/src
scp backup2:"/mnt/data4/build/freebsd/patches/8.0/*" .

Apply patches, i.e. the <tt>jls-patch</tt>:
patch -l < jls-patch

=== build, install kernel and world ===

Rename current generic kernel so it will always be available to boot from. Save room by removing non-needed kernel modules:

<pre>cd /boot
mv kernel kernel.GENERIC
cd kernel.GENERIC
mkdir hold
mv mfi_linux.ko hold/
mv linux.ko hold/
mv linprocfs.ko hold/
mv linsysfs.ko hold/
mv geom_vinum.ko hold/
mv geom_concat.ko hold/
mv zfs.* hold/
mv opensolaris* hold/

rm *.ko
rm *.symbols
mv hold/* .
rmdir hold/

cd /usr/src
make buildkernel installkernel

make buildworld ; mail -s 'buildworld done' dave.boodman@vtext.com < /dev/null</pre>
~38mins

make installworld
~34min

mergemaster -i
You will be prompted to merge, replace or ignore files changed by the src update. In most cases you can '''d'''elete the temp (new) files.

ONLY if this will be a zfs system (not currently used in 8.x):
<pre>cd /sys/modules/zfs
make
make install
cd /sys/modules/opensolaris
make
make install</pre>

=== populate devfs ruleset ===
scp backup2:/mnt/data4/build/freebsd/devfs.rules.8x /etc/devfs.rules

=== populate /etc/rc.conf with IPs and service settings ===

<pre>vi /etc/rc.conf

kern_securelevel_enable="NO"
portmap_enable="NO"
sendmail_enable="NO"
usbd_enable="YES"

nfs_client_enable="YES"
nfs_reserved_port_only="YES"
inetd_enable="YES"
inetd_flags="-wW -a 10.1.4.103"
devfs_system_ruleset="devfsrules_show_all"

ifconfig_bce1="inet 10.1.4.103 netmask 255.255.255.0"
ifconfig_bce0="inet 69.55.229.7 netmask 255.255.255.0"
#ifconfig_bce0_alias0="inet 69.55.2xx.xx netmask 255.255.255.0"

fsck_y_enable="YES"
background_fsck="NO"
#rc_mfi_raid_tty_log="YES"
#zfs_enable="YES"</pre>

Modify IPs, hostname, gateway for this box.

=== make sure sysctls are set and preserved upon boot ===

<pre>echo "kern.consmute=0\
kern.ipc.shm_use_phys=1\
kern.ipc.shmall=131070\
kern.ipc.shmmax=134217728\
net.inet.tcp.syncookies=0\
kern.maxfiles=32768\
kern.fallback_elf_brand=3\
kern.maxprocperuid=4000\
security.jail.sysvipc_allowed=1\
security.jail.allow_raw_sockets=1\
security.jail.socket_unixiproute_only=1\
security.jail.chflags_allowed=0\
dev.amr.0.allow_volume_configure=1\
compat.linux.osrelease=2.6.12\
vm.pmap.shpgperproc=500\
security.bsd.unprivileged_read_msgbuf=0\
kern.maxvnodes=400000" >> /etc/sysctl.conf</pre>

Tuning note: watch vfs.numvnodes while the server is live to get guidance on where to set maxvnodes

=== mount procfs ===
echo "proc /proc procfs rw 0 0" >> /etc/fstab
echo "linprocfs /usr/compat/linux/proc linprocfs rw 0 0" >> /etc/fstab
echo "linsysfs /usr/compat/linux/sys linsysfs rw 0 0" >> /etc/fstab
mkdir -p /usr/compat/linux/proc
mkdir -p /usr/compat/linux/sys

=== enable noatime option ===
data1 and data2 should look something like (add ',noatime' after 'rw'):
/dev/mfid0s1g /mnt/data1 ufs rw,noatime 2 2
/dev/mfid1s1d /mnt/data2 ufs rw,noatime 2 2

=== reboot. Confirm new kernel is loaded, devfs in place ===

uname -a

Check devfs rules
devfs rule showsets
devfs rule -s 3 show

Should see:
<pre># devfs rule showsets
1
2
3
4

# devfs rule -s 3 show
100 include 1
207 path pts* unhide
217 path fd unhide
218 path fd/* unhide
300 path ttyp* unhide
301 path ttyq* unhide
302 path ttyr* unhide
303 path ttys* unhide
304 path ttyP* unhide
305 path ttyQ* unhide
306 path ttyR* unhide
307 path ttyS* unhide
400 path null unhide
500 path zero unhide
600 path random unhide
610 path urandom unhide
700 path mem unhide
710 path kmem unhide
810 path mdctl unhide
900 path stdin unhide
910 path stdout unhide
920 path stderr unhide</pre>

=== update ports ===

<pre>cd /usr/ports
echo "*default host=cvsup4.FreeBSD.org\
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_8_3\
*default delete use-rel-suffix\
*default compress\
ports-all tag=." > sup

cvsup sup; mail -s 'cvs sup ports done' support@johncompanies.com < /dev/null</pre>
~24mins

=== Install raid mgmt tool ===

==== Perc5/i, 6/i ====
Pull over cli from previous system (jail9)
scp /usr/local/sbin/mega* root@10.1.4.109:/usr/local/sbin/
scp /usr/local/libexec/MegaCli root@10.1.4.109:/usr/local/libexec/MegaCli
These are linux-based tools. This will require linux base which we would normally install from ports, but since it's failing lately, we just pull in the compat libraries from another system:

rsync -aSHv --exclude=proc --exclude=sys 10.1.4.109:/usr/compat/linux/ /usr/compat/linux/

Test:
rehash; megacli ldinfo lall a0

DEPRECATED: Assuming it worked, here's how we ''used to'' install linux_base:
----
cd /usr/ports/emulators/linux_base-fc4
make install clean
(didnt succeed due to libtool requirement)

cd /usr/ports/distfiles
fetch http://www.lsi.com/DistributionSystem/AssetDocument/support/downloads/megaraid/miscellaneous/linux/2.00.15_Linux_MegaCLI.zip
cd /usr/ports/sysutils/linux-megacli
make install clean
(also failed due to libtool)
----

==== 2850 PERC 4e/Di- no linux ====
<pre>cd /usr/ports/distfiles/
fetch http://backup01.best-hosting.ru/pub/FreeBSD/ports/distfiles/dr_freebsd_1.51.zip
cd /usr/ports/sysutils/megarc
make install clean
megarc -dispCfg -a0</pre>

=== install rsync from ports ===
cd /usr/ports/net/rsync
make install clean

choose default options

=== configure inetd to respond to mrtg load queries ===
echo "load stream tcp nowait user /usr/local/jail/bin/load.pl load.pl" >> /etc/inetd.conf
echo "load 12384/tcp" >> /etc/services

=== install perl ===
cd /usr/ports/lang/perl5.12
make install clean

choose defaults

=== install bb client ===

Compiling from source on AMD64 will not work. So, we use a linux-compiled version and rely on linux compat. Per [[#Perc5.2Fi.2C_6.2Fi|above]], linux compat won't install on 8.x - libtool 2.4 need. So, instead we copy(ed) over linux:
rsync -aSHv --exclude=proc --exclude=sys 10.1.4.108:/usr/compat/linux/ /usr/compat/linux/

adduser

Output/response:

<pre>Username: bb
Full name: bb
Uid (Leave empty for default): 1984
Login group [bb]:
Login group is bb. Invite bb into other groups? []:
Login class [default]:
Shell (sh csh tcsh nologin) [sh]:
Home directory [/home/bb]:
Use password-based authentication? [yes]:
Use an empty password? (yes/no) [no]:
Use a random password? (yes/no) [no]: yes
Lock out the account after creation? [no]:
Username : bb
Password : <random>
Full Name : bb
Uid : 1984
Class :
Groups : bb
Home : /home/bb
Shell : /bin/sh
Locked : no
OK? (yes/no): yes</pre>

cd /usr/home/bb
scp backup2:/mnt/data4/build/bb/bb-freebsd_linuxcompat.tgz .
tar xzf bb-freebsd_linuxcompat.tgz

edit /home/bb/bbc1.9e-btf/etc/bb-hosts with something like:

echo "10.1.4.5 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh \
10.1.4.'''103 jail3'''.johncompanies.com # ssh" > /home/bb/bbc1.9e-btf/etc/bb-hosts

Edit for machine name and private IP.

if this machine is at i2b:
echo "69.55.230.2 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh \
69.55.229.7 jail3.johncompanies.com # ssh" > /home/bb/bbc1.9e-btf/etc/bb-hosts

<pre>vi /home/bb/bbc1.9e-btf/ext/openfiles

MACHINE="jail3,johncompanies,com" # HAS TO BE IN A,B,C FORM</pre>
Edit for machine name.

<pre>cd /usr/home/bb/bbc1.9e-btf/etc
./bbchkcfg.sh
(y to questions)
./bbchkhosts.sh
(ignore ssh errors)
cd ../..
chown -R bb .
su bb
cd
cd bbc1.9e-btf
./runbb.sh start
more BBOUT
(look for errors)
exit</pre>

Put in script to start bb @ boot:
echo 'su - bb -c "cd /home/bb/bbc1.9e-btf; ./runbb.sh start"' > /usr/local/etc/rc.d/bb.sh
chmod +x /usr/local/etc/rc.d/bb.sh

=== remove reserve space ===
<pre>cd
umount /mnt/data1
umount /mnt/data2
tunefs -m 0 /mnt/data1
tunefs -m 0 /mnt/data2
mount -a</pre>

=== setup rdate ===
cd /usr/ports/sysutils/rdate
make install clean

crontab -e
0 0 * * * /usr/local/sbin/rdate -s utcnist.colorado.edu

/usr/local/sbin/rdate -s utcnist.colorado.edu

We used to use ntpd, however it listens on jail IPs which is a security risk. So we stopped. Here's the old instructions:
install new ntp from ports
/usr/ports/net/ntp
make install clean

<pre>echo "server 10.1.4.5" > /etc/ntp.conf
/usr/sbin/ntpd -p /var/run/ntpd.pid
sleep 2; ntpq -p</pre>(confirm it’s able to reach our time server)

=== make a data partition ===

IF you didn't format the data partition during sysinstall:

Create a g partition on 2nd mirror – bsdlabel no longer works (below shows d partition made with sysinstall):

<pre>jail8 /usr/home/bb# gpart show
=> 63 285474735 mfid0 MBR (136G)
63 285458922 1 freebsd [active] (136G)
285458985 15813 - free - (7.7M)

=> 0 285458922 mfid0s1 BSD (136G)
0 524288 1 freebsd-ufs (256M)
524288 12582912 2 freebsd-swap (6.0G)
13107200 524288 4 freebsd-ufs (256M)
13631488 524288 5 freebsd-ufs (256M)
14155776 8388608 6 freebsd-ufs (4.0G)
22544384 262914538 7 freebsd-ufs (125G)

=> 63 584843175 mfid1 MBR (279G)
63 584830197 1 freebsd [active] (279G)
584830260 12978 - free - (6.3M)

=> 0 584830197 mfid1s1 BSD (279G)
0 16777216 2 freebsd-swap (8.0G)
16777216 568052981 4 freebsd-ufs (271G)

jail8 /usr/home/bb# gpart show mfid1s1
=> 0 584830197 mfid1s1 BSD (279G)
0 16777216 2 freebsd-swap (8.0G)
16777216 568052981 4 freebsd-ufs (271G)

# gpart list mfid1s1

Geom name: mfid1s1
fwheads: 255
fwsectors: 63
last: 584830196
first: 0
entries: 8
scheme: BSD
Providers:
1. Name: mfid1s1b
Mediasize: 8589934592 (8.0G)
Sectorsize: 512
Mode: r1w1e0
rawtype: 1
length: 8589934592
offset: 0
type: freebsd-swap
index: 2
end: 16777215
start: 0
2. Name: mfid1s1d
Mediasize: 290843126272 (271G)
Sectorsize: 512
Mode: r0w0e0
rawtype: 7
length: 290843126272
offset: 8589934592
type: freebsd-ufs
index: 4
end: 584830196
start: 16777216
Consumers:
1. Name: mfid1s1
Mediasize: 299433060864 (279G)
Sectorsize: 512
Mode: r1w1e1

# gpart delete -i 4 mfid1s1
mfid1s1d deleted
jail8 /usr/home/bb# gpart list mfid1s1
Geom name: mfid1s1
fwheads: 255
fwsectors: 63
last: 584830196
first: 0
entries: 8
scheme: BSD
Providers:
1. Name: mfid1s1b
Mediasize: 8589934592 (8.0G)
Sectorsize: 512
Mode: r1w1e0
rawtype: 1
length: 8589934592
offset: 0
type: freebsd-swap
index: 2
end: 16777215
start: 0
Consumers:
1. Name: mfid1s1
Mediasize: 299433060864 (279G)
Sectorsize: 512
Mode: r1w1e1

# gpart show mfid1s1
=> 0 584830197 mfid1s1 BSD (279G)
0 16777216 2 freebsd-swap (8.0G)
16777216 568052981 - free - (271G)

# gpart add -t freebsd-ufs -i 7 mfid1s1
mfid1s1g added

# gpart show mfid1s1
=> 0 584830197 mfid1s1 BSD (279G)
0 16777216 2 freebsd-swap (8.0G)
16777216 568052981 7 freebsd-ufs (271G)</pre>

Here's how we USED to do it with bsdlabel:

bsdlabel -e /dev/mfid0s1

given:
<pre># /dev/aacd0s1:
8 partitions:
# size offset fstype [fsize bsize bps/cpg]
a: 262144 0 4.2BSD 2048 16384 16392
b: 4194304 262144 swap
c: 143363997 0 unused 0 0 # "raw" part, don't edit
d: 524288 4456448 4.2BSD 2048 16384 32776
e: 524288 4980736 4.2BSD 2048 16384 32776
f: 6291456 5505024 4.2BSD 2048 16384 28552</pre>

new offset = 6291456 + 5505024 = 11796480<br>
new size is size for 'c' partition minus the new start from above<br>
143363997 - 11796480 = 131567517

So:<br>
g: 131567517 11796480 unused 0 0

=== create the jail template ===

cd /usr/ports/sysutils/jailutils
make install clean

Create an md device to hold the jail:
<pre>touch /mnt/data1/jail-template20g
mdconfig -a -t vnode -s 20g -f /mnt/data1/jail-template20g -u 0
newfs -O 1 /dev/md0
mkdir /mnt/data1/jail-DIR
mount /dev/md0 /mnt/data1/jail-DIR</pre>

Build world into the jail:
<pre>cd /usr/src
make world DESTDIR=/mnt/data1/jail-DIR; taskdone</pre>
~1hr

Make /etc into the jail, mount dev, copy in jkill:
<pre>cd etc
make distribution DESTDIR=/mnt/data1/jail-DIR
mount -t devfs devfs /mnt/data1/jail-DIR/dev
devfs -m /mnt/data1/jail-DIR/dev rule -s 3 applyset
cd /mnt/data1/jail-DIR
ln -sf dev/null kernel
cp /usr/local/sbin/jkill /mnt/data1/jail-DIR/sbin</pre>

Enter into jail to do configuration:
jail /mnt/data1/jail-DIR testhostname 192.168.11.100 /bin/sh
csh

Create fstab:
<pre>touch /etc/fstab
echo 'network_interfaces=""\
hostname="newsystem"\
kern_securelevel_enable="NO"\
sendmail_enable="YES"\
sshd_enable="YES"' > /etc/rc.conf

echo "nameserver 69.55.225.225\
nameserver 69.55.230.3" >> /etc/resolv.conf</pre>

Edit crontab:
vi /etc/crontab
remove the adjkerntz lines
comment out periodic’s and put this line above them:
# IF YOU UNCOMMENT THESE, PLEASE ADJUST THEIR RUN TIME

rm -rf /etc/periodic/daily/400.status-disks

check and remove any crap in /tmp

vi /etc/periodic/security/100.chksetuid
replace: <tt>MP=`mount -t ufs | grep -v " nosuid" | awk '{ print $3 }' | sort`</tt><br>
with: <tt>MP='/'</tt> (use single quotes)

mkdir -p /usr/compat/linux/dev

Add account for user. Output/response:
<pre>adduser

Username: user
Full name: user
Uid (Leave empty for default):
Login group [user]:
Login group is user. Invite user into other groups? []: wheel
Login class [default]:
Shell (sh csh tcsh nologin) [sh]:
Home directory [/home/user]:
Home directory permissions (Leave empty for default):
Use password-based authentication? [yes]:
Use an empty password? (yes/no) [no]:
Use a random password? (yes/no) [no]: y
Lock out the account after creation? [no]:
Username : user
Password : <random>
Full Name : user
Uid : 1001
Class :
Groups : user
Home : /home/user
Home Mode :
Shell : /bin/sh
Locked : no
OK? (yes/no): y
adduser: INFO: Successfully added (user) to the user database.
adduser: INFO: Password for (user) is: 901gmYjO
Add another user? (yes/no): n
Goodbye!</pre>

set TERM:
vi /usr/home/user/.profile
TERM=vt100; export TERM

Set time zone to PT:
tzsetup

Reload aliases:
newaliases

Replace reboot/halt:
rm /sbin/halt /sbin/reboot
ln /sbin/jkill /sbin/halt
ln /sbin/jkill /sbin/reboot

Redirect console output, comment out console and move to /var/log/messages:
vi /etc/syslog.conf
#*.err;kern.warning;auth.notice;mail.crit /dev/console
*.err;kern.warning;auth.notice;mail.crit /var/log/messages

exit
exit

Lib32 compat library:
cd libexec
chflags noschg ld-elf32.so.1
chflags noschg ld-elf.so.1
mv ld-elf32.so.1 ld-elf32.so.1-orig
ln ld-elf.so.1 ld-elf32.so.1
chflags schg ld-elf.so.1
chflags schg ld-elf32.so.1

Replace traceroute:
mv /mnt/data1/jail-DIR/usr/sbin/traceroute /mnt/data1/jail-DIR/usr/sbin/_traceroute
echo '#\!/bin/sh\
/usr/sbin/_traceroute -i bce0 $1' >> /mnt/data1/jail-DIR/usr/sbin/traceroute
chmod +x /mnt/data1/jail-DIR/usr/sbin/traceroute

Modify 'bce0' to reflect whichever nic is public on this hardware.

Clean out ports before copying into the jail (~30mins):
cd /usr/ports
make -DNOCLEANDEPENDS clean
rm -fr /usr/ports/distfiles/*
cp -r /usr/ports /mnt/data1/jail-DIR/usr

rm /mnt/data1/jail-DIR/root/.history

Umount the jail and dump it:
<pre>cd /
umount /mnt/data1/jail-DIR/dev
dump -0a -f /usr/local/jail/template/template /dev/md0
umount /dev/md0
rmdir /mnt/data1/jail-DIR
mdconfig -d -u 0</pre>

=== setup backups ===
<pre>echo '#\!/bin/sh\
backupdir=/data/jail3\
server=backup1\
\
## ENTRY /etc\
## ENTRY /usr/local/etc\
## ENTRY /usr/local/jail\
## ENTRY /root/logs' > /usr/local/jail/bin/backup.config</pre>

Edit to reflect backup server and jail hostname

On backup server, setup backup dirs:
ssh backup1 mkdir -p /data/jail3/0

On backup server, setup backup dirs:
backup1# vi /usr/local/sbin/snapshot_rotate

Edit /usr/local/jail/bin/backup.xxx to use the right drives and copy to /usr/local/jail/bin/backup
vi /usr/local/jail/bin/backup.md
adjust df so it includes all relevant drives, currently <tt>df > /etc/df.bak</tt> is fine. Also, make sure the binary/script source is pulling from the right dir on backup2:
/usr/local/bin/rsync -a backup2:/mnt/data4/bin/freebsd8.x/ /usr/local/jail/bin/

cp /usr/local/jail/bin/backup.md /usr/local/jail/bin/backup

=== create /root/logs ===
mkdir /root/logs

=== edit sshd_config ===
vi /etc/ssh/sshd_config
ListenAddress 69.55.229.7
ListenAddress 10.1.4.103

'''''Adjust to pub/private IP for jail.'''
''
Restart sshd:
kill -1 `cat /var/run/sshd.pid`

=== add crontab entries ===
<pre>crontab -e
5 0 * * * /usr/local/jail/bin/backup
1 0 1 * * /usr/local/jail/bin/ipfwreset
0 18 * * * /usr/local/jail/bin/ipfwbackup
4,9,14,19,24,29,34,39,44,49,55,59 * * * * /usr/local/jail/bin/trafstats
0 0,6,12,18 * * * /usr/local/jail/bin/sync_jail_names</pre>

On 2950:
*/5 * * * * /usr/local/jail/bin/perc5iraidchk

On 3ware-based servers:
0 0 * * * /usr/local/jail/bin/3wraidchk

=== Reboot notify script ===
ln -s /usr/local/jail/bin/notify.sh /usr/local/etc/rc.d/notify.sh

=== patch jail against starting jails with rtprio ===

<pre>mv /usr/sbin/jail /usr/sbin/jail_
echo '#\!/bin/sh\
/usr/sbin/rtprio -t /usr/sbin/jail_ $*' > /usr/sbin/jail
chmod +x /usr/sbin/jail</pre>

=== make sure mail works ===
If there are map errors:
cd /etc/mail; make maps

=== recover space on /usr (optional) ===

rm -fr /usr/obj

=== wrapper for jps ===

mv /usr/local/sbin/jps /usr/local/sbin/jps_

=== wrapper for jls ===

mv /usr/sbin/jls /usr/sbin/jls_

=== wrapper for jexec ===

mv /usr/sbin/jexec /usr/sbin/jexec_

=== install jtop ===

cd /usr/ports/sysutils/jtop
make install clean

=== block jails from reaching private net ===
echo 'ipfw add 1 deny ip from 69.55.224.0/20 to 10.1.4.0/24' > /usr/local/etc/rc.d/ipfw.sh
chmod 0700 /usr/local/etc/rc.d/ipfw.sh

== add to management infrastructure ==

=== mail ===

==== add to management db ====

tables: jc.ref_machines and jc.ref_templates

on jail run:
uname -r

Which shows something like:
8.3-RELEASE-p2

Insert into db:

<pre>insert into ref_machines values (null,'jail3','mx3.johncompanies.com',0,'f8');
select machine_id from ref_machines where host='jail3';
+------------+
| machine_id |
+------------+
| 35 |
+------------+
insert into ref_templates values ('','8.3-RELEASE-jc2',35,'FreeBSD 8.3',1);</pre>

==== add to bb server ====
vi /usr/home/bb/bbsrc/bb1.9i-btf/etc/bb-hosts
10.1.4.109 jail9.johncompanies.com # ssh

In the case of an i2b server, use real ip:
69.55.229.7 jail3.johncompanies.com # ssh

<pre>su bb
cd
bbsrc/bb/runbb.sh restart ; exit</pre>

==== Update backupgraph ====
vi /usr/local/www/mgmt/cgi/backupgraph.pl
(add hostname)

==== Update load mrtg ====
vi /usr/local/www/mgmt/mrtg/mrtg1.cfg
(add new entry to file following existing format)

=== ns1c ===
fwd and reverse lookups:
vr johncompanies.com

vi internal.johncompanies.com
rndc reload johncompanies.com IN private

ptr 69.55.227.x

=== wiki ===
add to server/cabinet map

=== firewall ===
add an outside blocking rule to the firewall, so this machine can only be reached from inside the firewall. Follow example already in firewall jail17 is:

00117 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 or 69.55.238.150 } to 69.55.228.200
00117 deny ip from any to 69.55.228.200

jail19 would be 00119...
ipfw add 00109 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 } to 69.55.232.3
ipfw add 00109 deny ip from any to 69.55.232.3

=== select some customers for castle probe map ===

= FreeBSD 4.11 =

<pre>
4.11
Last updated 2006-1-26

All time extimates below assume disks aren’t scrubbing

1. make sure bios is setup for bios console redirect
Supermicro:
Console redirection:
Com port addr: on-board COM A
Baud: 38400
Console type: vt100
Flow control: none
Console connection: direct
Continue cr after post: off

2450:
Make sure running bios A09
Console Redirection: VT100/VT220

2. assuming mirrors (or at least disks) created (if not, refer to this), boot to disk 1 of 4.11
skip kernel config (enter)
custom install
partition
move cursor to aacd0, hit space (takes you to partition map screen)
a for entire disk
q to quit and save
standard mbr (no boot manager)
space to unselect aacd0
cursor over aacd1
space
a for entire disk
q to quit and save
none (leave untouched)

cursor over aacd0
space
(takes you into part. Screen again) q to exit
none
Make sure both are checked and tab to ok

label
Make sure aacd0 is highlighted
a to start with defaults

john likes:
/1g
/var 256m

Glenn likes:
/128
/var 256
/usr 3g
/tmp 256

We do:
delete everything
/ 128M
swap 4G
/var 256M
/tmp 256M
/usr 3G
/mnt/data1 remaining space

Make sure to toggle S for soft updates on all (should look like UFS+S Y under the Newfs column)
Set the noatime option

move cursor to aacd1
swap 4G
/mnt/data2 remaining space
Set the noatime option

q to save and exit

distributions
developer
yes to install ports
exit
media
cd
commit
yes
(2450: 16mins, supermicro: 11mins)
yes to "visit general config"
Set root pwd
Add user ‘user’ member group is wheel, set password
Set tz
Networking->interfaces->Fxp0
No IPV6
dhcp=yes
Set hostname & domain

Startup services:
Disble usbd

exit...
exit install
yes
take the cd out and let the machine reboot

3. put some temp settings in /etc/rc.conf:
usbd_enable="NO"
sendmail_enable="NO"

4. reboot

5. double check the date/time

6. edit /etc/make.conf (only add the console speed line if this is a supermicro capable of outputting BIOS redirect at that speed – 2450’s can only do 9600)
cat >> /etc/make.conf
WITHOUT_X11=yes
KERNCONF=jail18
BOOT_COMCONSOLE_SPEED=38400

7. install cvsup
cd /usr/ports/net/cvsup-without-gui
make install clean (stay close for gettext options, 2450: 21mins, supermicro: 14mins)
rehash

8. get latest sources for this release:
cd /usr/src
cat > sup
*default host=cvsup4.freebsd.org
*default base=/usr
*default prefix=/usr
*default release=cvs tag=RELENG_4_11
*default delete use-rel-suffix
*default compress
src-all

cvsup sup (2450, 4.10: 13mins, supermicro, 4.11: 11mins)

9. populate hosts
cat >> /etc/hosts
10.1.4.3 backup2

10. put key in authorized_keys on backup2
ssh-keygen -t dsa -b 1024 (default location, leave password blank)
scp /root/.ssh/id_dsa.pub user@backup2:/tmp/jail18pub
on backup2:
cat /tmp/jail18pub >> /root/.ssh/authorized_keys

confirm that you can ssh to backup2 without getting a login prompt

11. configure new kernel. Get config from similar machine or there may be a master copy somewhere under /mnt/data4/build (name the kernel config the same as the jail, ex jail4):
cd /usr/src/sys/i386/conf
scp backup2:/mnt/data4/build/freebsd/kern_config-4.11 ./jail18

edit the kernel config and change ident to be the name of the jail:
ident jail4

IMPORTANT CUSTOMIZATION:
for machines with >4G RAM, add to the config:
options PAE

for supermicro mobo’s with broadcom nics, add to the config:
device bge # Broadcom BCM570x (``Tigon III'')

for machines where lots of postgres might be running, change SHMMAXPGS:
options SHMMAXPGS=40960

edit /sys/conf/newvers.sh to add –jc2 to the end of the BRANCH string (RELEASE-p9-jc2)

12. bring over patches from backup2
The various patches are in /mnt/data4/build/freebsd/patches on backup2. There are dirs for each version. Not all dirs are populated, but patches for later versions work on older ones unless there is a new patch in the older dir.
cd /usr/src
scp backup2:"/mnt/data4/build/freebsd/patches/4.11/*" .
jail_proc_patch 100% |*************************************************| 2593 00:00
restore-patch 100% |*************************************************| 3295 00:00
411ps-jail-patch 100% |*************************************************| 2602 00:00
jail_rtprio_patch 100% |*************************************************| 301 00:00
udp-patch 100% |*************************************************| 594 00:00

Apply patches:
patch < 411ps-jail-patch
patch < jail_proc_patch
patch < restore-patch
patch < jail_rtprio_patch
patch < udp-patch

13. build, install kernel and world
make buildworld buildkernel installkernel (2450: 48min, supermicro: 20mins)
(Any compile errors can be looked up in /usr/include/sys/signal.h, other errors, do a rm -R /usr/obj/*)
make installworld (2450: 2min, supermicro: 1mins)
mergemaster -i
(answer no to most of it)

14. reboot. Confirm new kernel is loaded (uname -a)

15. update ports:
cd /usr/ports
cat > sup
*default host=cvsup4.FreeBSD.org
*default base=/usr
*default prefix=/usr
*default release=cvs tag=RELENG_4
*default delete use-rel-suffix
*default compress
ports-all tag=.

cvsup sup (2450: 26mins, supermicro: 26mins)

16. add console="comconsole" to /boot/loader.conf
cat >> /boot/loader.conf
console="comconsole"

17. edit /etc/ttys and turn off all ttyv's except 0 and 1
also turn on ttyd0, change type to vt100:
ttyd0 "/usr/libexec/getty std.9600" vt100 on secure

kill -1 1

on console:
vi /etc/remote
(rename port to jail4 depending on where and which digi plugged into)
test serial console

18. install linux_base:
cd /usr/ports/emulators/linux_base
make install clean (2450: 7min, supermicro: 2mins)

reibuild rpmdb cause we had probs installing aacapps
cd /compat/linux/bin
./rpm --initdb
./rpm --rebuilddb

19. install aacapps-4.1-0.i386.rpm
scp backup2:/mnt/data4/build/freebsd/aacapps-4.1-0.i386.rpm /tmp/.
/compat/linux/bin/rpm -ivh /tmp/aacapps-4.1-0.i386.rpm
cd /dev
sh MAKEDEV aac0

test out /compat/linux/usr/sbin/aaccli

20. edit root's path and login script:
vi /root/.cshrc
add to path: /usr/local/jail/bin /compat/linux/usr/sbin

Change alias entries (add G):
alias la ls -aG
alias lf ls -FAG
alias ll ls -lAG
alias ls ls -AG
alias mbm mb mount
alias mbu mb umount

and alter the prompt, set the following:
set prompt = "`/bin/hostname -s` %/# "

at the bottom of the file add:
set sshtty=`who am i|awk '{print $2}'`
/usr/sbin/rtprio 3 -`ps auxwJ | grep $sshtty | awk '{print $2}'`

set shortty=`who am i | awk '{print $2}' | sed -E 's/.*(..)$/\1/'`
foreach x (`ps cauxJ | grep sh | grep $shortty | awk '{print $2}'`)
/usr/sbin/rtprio 2 -$x
end

21. install rsync from ports
cd /usr/ports/net/rsync
make install clean

22. create & populate binaries/scripts dirs
mkdir -p /usr/local/jail/bin
mkdir -p /usr/local/jail/rc.d
scp backup2:"/mnt/data4/bin/freebsd/*" /usr/local/jail/bin
cd /usr/local/jail/rc.d/
touch quad1
touch quad2
touch quad3
touch quad4
touch safe1
touch safe2
touch safe3
touch safe4
chmod +x *
cd /usr/local/jail/bin
ln -s /usr/local/jail/rc.d/quad1 quad1
ln -s /usr/local/jail/rc.d/quad2 quad2
ln -s /usr/local/jail/rc.d/quad3 quad3
ln -s /usr/local/jail/rc.d/quad4 quad4
ln -s /usr/local/jail/rc.d/safe1 safe1
ln -s /usr/local/jail/rc.d/safe2 safe2
ln -s /usr/local/jail/rc.d/safe3 safe3
ln -s /usr/local/jail/rc.d/safe4 safe4
rehash

23. configure inetd to respond to mrtg load queries
cat >> /etc/inetd.conf
load stream tcp nowait user /usr/local/jail/bin/load.pl load.pl

cat >> /etc/services
load 12384/tcp

kill -HUP `cat /var/run/inetd.pid`

24. configure load mrtg, on mail
vi /usr/local/etc/mrtg/mrtg1.cfg
(add new entry to file following existing format)

25. configure bb on mail:
vi /usr/home/bb/bbsrc/bb1.9e-btf/etc/bb-hosts
10.1.4.104 jail4.johncompanies.com # ssh

su bb
cd /usr/home/bb/bbsrc/bb1.9e-btf
./runbb.sh stop
./runbb.sh start
exit

26. install bb client
adduser -group 1984 -shell /bin/csh -uid 1984 bb
cd /usr/home/bb
scp backup2:/mnt/data4/build/bb/bb-freebsd.tar .
tar xvf bb-freebsd.tar

cat > /home/bb/bbc1.9e-btf/etc/bb-hosts with something like:
10.1.4.105 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh
10.1.4.118 jail18.johncompanies.com # ssh

vi /home/bb/bbc1.9e-btf/ext/openfiles and change:
MACHINE="jail18,johncompanies,com" # HAS TO BE IN A,B,C FORM

cd /usr/home/bb/bbc1.9e-btf/etc
./bbchkcfg.sh
./bbchkhosts.sh (ignore ssh errors)
cd ../..
chown -R bb .
su bb
cd bbc1.9e-btf/
./runbb.sh start
more BBOUT (look for errors)
exit

cat > /usr/local/etc/rc.d/bb.sh
su - bb -c "cd /home/bb/bbc1.9e-btf; ./runbb.sh start"

chmod +x /usr/local/etc/rc.d/bb.sh

27. remove reserve space, enable softupdates (probably already set, so not necessary)
cd
umount /mnt/data1
umount /mnt/data2
tunefs -m 0 /mnt/data1
tunefs -m 0 /mnt/data2

mount -a

28. configure ntp
cat > /etc/ntp.conf
server 10.1.4.105

/usr/sbin/ntpd -p /var/run/ntpd.pid
ntpq -p
(confirm it’s able to reach our time server)

29. mrtg switch graphs

31. fwd and reverse lookups on ns1c
vi johncompanies.com
rr johncompanies.com
vi internal.johncompanies.com
rndc reload johncompanies.com IN private
(edit the PTR too)

32. create all /dev/vn and /dev/pty files in /dev
cat > /tmp/runme.sh
#!/bin/sh
cd /dev
for i in 1 2 3 4 5 6 7 ; do sh MAKEDEV pty$i ; done
for i in 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 ; do sh MAKEDEV vn$i ; done

NOTE: there are only 4 lines in the above output - however the lines may be wrapped on your terminal - make sure that there are only four lines in the script you make.

chmod +x /tmp/runme.sh
/tmp/runme.sh

rm /tmp/runme.sh

ls /dev/vn*|wc -l (make sure the output is a high number – larger than 128)
ls /dev/pty*|wc -l (make sure the output is exactly 256)

33. populate /etc/rc.conf with IPs and NFS settings
vi /etc/rc.conf

kern_securelevel_enable="NO"
portmap_enable="NO"
sendmail_enable="NO"
usbd_enable="NO"

xntpd_enable="YES"
nfs_client_enable="YES"
nfs_reserved_port_only="YES"
sshd_enable="YES"
inetd_enable="YES"
inetd_flags="-wW -a 10.1.4.104"

ifconfig_xl0="inet 10.1.4.104 netmask 255.255.255.0"
ifconfig_fxp0="inet 69.55.228.101 netmask 255.255.255.0"
defaultrouter="69.55.228.1"
ifconfig_fxp0_alias0="inet 69.55.2xx.xx netmask 255.255.255.0"

static_routes="t1 office"
route_t1="-net 10.1.5 10.1.4.2"
route_office="-net 10.1.6 10.1.4.2"

34. make sure sysctls are set and preserved after reboot
cat >> /etc/sysctl.conf
kern.consmute=0
jail.sysvipc_allowed=1
kern.ipc.shm_use_phys=1
kern.ipc.shmall=65535
kern.ipc.shmmax=134217728
net.inet.tcp.syncookies=0
kern.maxfiles=32768
kern.fallback_elf_brand=3
kern.maxprocperuid=4000
jail.max_procs_per_jail: 1026

35. reboot

36. create the jail template
vnconfig -T -S 1g -Z -s labels -c /dev/vn1 /mnt/data1/jail
disklabel -r -w vn1 auto
newfs /dev/vn1c
cd /usr/src
mkdir -p /mnt/data1/jail-DIR
mount /dev/vn1c /mnt/data1/jail-DIR
make world DESTDIR=/mnt/data1/jail-DIR (2450: 45mins, supermicro: 19mins)
cd etc
make distribution DESTDIR=/mnt/data1/jail-DIR -DNO_MAKEDEV_RUN
cd /mnt/data1/jail-DIR/dev
sh MAKEDEV jail
cd /mnt/data1/jail-DIR
ln -sf dev/null kernel

jail /mnt/data1/jail-DIR testhostname 192.168.11.100 /bin/sh
csh
touch /etc/fstab
cat > /etc/rc.conf
portmap_enable="NO"
network_interfaces=""
hostname="newsystem"
kern_securelevel_enable="NO"
sendmail_enable="YES"
sshd_enable="YES"

cat >> /etc/resolv.conf
nameserver 69.55.225.225
nameserver 69.55.230.3

vi /etc/crontab
remove the adjkerntz lines
comment out periodic’s and put this line above them:
# DO NOT UNCOMMENT THESE

rm -rf /etc/periodic/daily/400.status-disks

mv /bin/df /bin/df_

cat > /bin/df
#!/bin/sh
/bin/df_ $* .

chmod +x /bin/df

cat > /sbin/mount
echo `df | tail -1 | awk '{print $1 " on " $6 " (ufs, local)"}'`

check /tmp for crap

vi /etc/periodic/security/100.chksetuid
replace: MP=`mount -t ufs | grep -v " nosuid" | awk '{ print $3 }' | sort`
with: MP='/'

mkdir -p /usr/compat/linux/dev

adduser (Add account for user)

put user in wheel group
vi /etc/group

vi /usr/home/user/.profile (and add to the file):
TERM=vt100; export TERM

cd /etc
vipw -d .
root:$1$krszPxhk$xkCepSnz3mIikT3vCtJCt0:0:0::0:0:Charlie &:/root:/bin/csh
user:$1$Mx9p5Npk$QdMU6c8YQqp2FW2M3irEh/:1001:1001::0:0:User &:/home/user:/bin/sh

tzsetup

newaliases

chflags schg /dev/*mem

cd /dev
rm console
ln -s null console

exit
exit

cd /usr/ports
make -DNOCLEANDEPENDS clean (2450: 47mins , supermicro: 17mins)
rm -fr /usr/ports/distfiles/*
cp -r /usr/ports /mnt/data1/jail-DIR/usr

cd /mnt/data1/jail-DIR/dev
rm kmem
mknod kmem c 2 1 root:kmem
chmod 640 kmem
rm mem
mknod mem c 2 0 root:kmem
chmod 640 mem

rm /mnt/data1/jail-DIR/root/.history

sh
for i in 1 2 3 4 5 6 7 ; do sh MAKEDEV pty$i ; done
exit

cd /mnt/data1/jail-DIR/usr/compat/linux/dev
mknod null c 2 2
mknod random c 2 3

cd
mkdir /usr/local/jail/template/
dump -0a -f /usr/local/jail/template/411template /dev/vn1

umount /dev/vn1c
vnconfig -u /dev/vn1

rm /mnt/data1/jail
rm -fr /mnt/data1/jail-DIR

37. setup backups
cat > /usr/local/jail/bin/backup.config
#!/bin/sh
backupdir=/mnt/data2/jail18_rsync

## ENTRY /etc
## ENTRY /usr/local/etc
## ENTRY /usr/local/jail
## ENTRY /root/logs

on backup2:
setup backup dirs:
mkdir -p /mnt/data2/jail18_rsync/0_today
mkdir -p /mnt/data2/jail18_rsync/1_yesterday
mkdir -p /mnt/data2/jail18_rsync/2_two_day

add the system to /mnt/data1/bin/rsync_houseclean

on mail:
vi /usr/local/www/mgmt/cgi/backupgraph.pl
(add hostname)

38. mkdir /root/logs

39. edit sshd_config for security
vi /etc/ssh/sshd_config
ListenAddress 69.55.228.101
ListenAddress 10.1.4.104

kill -1 `cat /var/run/sshd.pid`

40. add crontab entries
crontab -e
5 0 * * * /usr/local/jail/bin/backup
1 0 1 * * /usr/local/jail/bin/ipfwreset
0 18 * * * /usr/local/jail/bin/ipfwbackup
4,9,14,19,24,29,34,39,44,49,55,59 * * * * /usr/local/jail/bin/trafstats
55 10,23 * * * /usr/local/jail/bin/trafficwatch.pl

41. Reboot notify script
cat > /usr/local/etc/rc.d/notify.sh
echo "`/bin/hostname` rebooted" | /usr/bin/mail reboot@johncompanies.com

chmod +x /usr/local/etc/rc.d/notify.sh

42. copy jailmake from prev system
scp user@10.1.4.118:/usr/local/jail/bin/jailmake /usr/local/jail/bin
rehash
NOTE: remove df altering code from jailmake since we put the correct df in the template, and make sure path to template file is right

43. add to templates via mgmt system

44. add to server/cabinet map. On mail:
vi /usr/local/www/mgmt/html/cabinetmap.html

45. add an outside blocking rule to the firewall, so this machine can only be reached from inside the firewall. Follow example already in firewall jail17 is:

00117 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 or 69.55.238.150 } to 69.55.228.2
00117 deny ip from any to 69.55.228.2

jail4 would be 00104...
ipfw add 00118 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 or 69.55.238.150 } to 69.55.228.140
ipfw add 00118 deny ip from any to 69.55.228.140

46. select customers for probe map
</pre>

= FreeBSD 6.1 =

Last updated 2006-05-09

<pre>All time extimates below assume disks aren’t scrubbing

1. make sure bios is setup for bios console redirect
Supermicro:
Console redirection:
Com port addr: on-board COM A
Baud: 38400
Console type: vt100
Flow control: none
Console connection: direct
Continue cr after post: off

2450:
Make sure running bios A09
Console Redirection: VT100/VT220

2. assuming mirrors (or at least disks) created (if not, refer to this), boot to disk 1 of 5.4
skip kernel config (enter)
custom install

partition ->
move cursor to amrd0, hit space (takes you to partition map screen)
a for entire disk
q to quit and save
standard mbr (no boot manager)
space to unselect aacd0
cursor over aacd1
space
a for entire disk
q to quit and save
none (leave untouched)

cursor over aacd0
space
(takes you into part. Screen again) q to exit
none
Make sure both are checked and tab to ok

Label ->
Make sure aacd0 is highlighted
a to start with defaults

john likes:
/1g
/var 256m

Glenn likes:
/128
/var 256
/usr 3g
/tmp 256

We do:
delete everything
/ 128M
swap 2G
/var 256M
/tmp 256M
/usr 3G
/mnt/data1 remaining space

Make sure to toggle S for soft updates on all (should look like UFS2+S Y under the Newfs column)

move cursor to aacd1
swap 2G
/mnt/data2 remaining space

q to save and exit

distributions ->
developer
yes to install ports
exit

media ->
cd

commit ->
yes
(2450: 14mins, supermicro: 12mins)

yes to "visit general config" ->
Set root pwd

Add user ‘user’ member group is wheel, set password

Set tz

Networking->interfaces->Fxp0 ->
No IPV6
dhcp=no
Set hostname & domain
Enable sshd

exit...
exit install ->
yes

take the cd out and let the machine reboot
when it comes back up, enter junk when it asks for key seed

3. double check the date/time

4. edit /etc/make.conf (only add the console speed line if this is a supermicro capable of outputting BIOS redirect at that speed – 2450’s can only do 9600)
echo "WITHOUT_X11=yes \
KERNCONF=jail19 \
BOOT_COMCONSOLE_SPEED=38400" >> /etc/make.conf

5. add console="comconsole" to /boot/loader.conf
echo "console=""comconsole""" >> /boot/loader.conf

6. turn off all ttyv's except 0 and 1 in /etc/ttys
also turn on ttyd0, change type to vt100:
vi /etc/ttys
ttyd0 "/usr/libexec/getty std.9600" vt100 on secure

kill -1 1

on console server:
vi /etc/remote
(rename port to jail18 depending on where and which digi plugged into)
test serial console

7. install cvsup
cd /usr/ports/net/cvsup-without-gui
make install clean; rehash
(stay close for gettext options, 2450: 27mins, supermicro: 17mins)

8. get latest sources for this release:
cd /usr/src
echo "*default host=cvsup4.freebsd.org\
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_6_1\
*default delete use-rel-suffix\
*default compress\
src-all" > sup

cvsup sup
(2450, ~12mins, supermicro, 27mins)

9. populate hosts
echo "10.1.4.3 backup2" >> /etc/hosts

10. put key in authorized_keys on backup2
cd
ssh-keygen -t dsa -b 1024
(default location, leave password blank)
scp /root/.ssh/id_dsa.pub user@backup2:/tmp/jailkey

cat /root/.ssh/id_dsa.pub | ssh backup2 'cat - >> /root/.ssh/authorized_keys'

confirm that you can ssh to backup2 without getting a login prompt

ssh backup2

11. configure new kernel. Get config from similar machine or there may be a master copy somewhere under /mnt/data4/build/freebsd (name the kernel config the same as the jail, ex jail18):
cd /usr/src/sys/i386/conf
scp backup2:/mnt/data4/build/freebsd/kern_config-6.1 ./jail19

edit the kernel config and change ident to be the name of the jail:
vi jail14
ident jail14

edit /sys/conf/newvers.sh to add –jc1 to the end of the BRANCH string (RELEASE-jc1)
vi /sys/conf/newvers.sh

12. bring over patches from backup2
The various patches are in /mnt/data4/build/freebsd/patches on backup2. There are dirs for each version. Not all dirs are populated, but patches for later versions work on older ones unless there is a new patch in the older dir.
cd /usr/src
scp backup2:"/mnt/data4/build/freebsd/patches/6.1/*" .
jail_proc_patch 100% |*************************************************| 2593 00:00
restore-patch 100% |*************************************************| 3295 00:00
54ps-jail-patch 100% |*************************************************| 2602 00:00
jail_rtprio_patch 100% |*************************************************| 301 00:00
udp-patch 100% |*************************************************| 594 00:00

Apply patches:
patch < 54ps-jail-patch
patch < jail_proc_patch
patch < restore-patch
patch < jail_rtprio_patch
patch < udp-patch

13. build, install kernel and world
cd /usr/src
make buildworld buildkernel installkernel
(2450: 1:56min, supermicro::59mins)
make installworld
(2450: 3min, supermicro: 1min)
mergemaster -i
delete /var/tmp/temproot
delete bsnmpd
delete temporary ./etc/hosts
delete temporary ./etc/motd
delete /var/tmp/temproot

14. reboot. Confirm new kernel is loaded

uname –a

15. update ports:
cd /usr/ports
echo "*default host=cvsup4.FreeBSD.org\
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_6_1\
*default delete use-rel-suffix\
*default compress\
ports-all tag=." > sup

cvsup sup
(2450: 18mins, supermicro: 19mins)

18. (only applies if adaptec card installed)
install linux_base:
cd /usr/ports/emulators/linux_base
make install clean
(2450: 7min, supermicro: 3mins)

reibuild rpmdb cause we had probs installing aacapps
cd /compat/linux/bin
./rpm --initdb
./rpm --rebuilddb

install aacapps-4.1-0.i386.rpm
scp backup2:/mnt/data4/build/freebsd/aacapps-4.1-0.i386.rpm /tmp/.
/compat/linux/bin/rpm -ivh --excludepath=/dev /tmp/aacapps-4.1-0.i386.rpm

test out;
/compat/linux/usr/sbin/aaccli

20. edit root's path and login script:
vi /root/.cshrc
add to path:
/usr/local/jail/bin
(if adaptec card installed, also add /compat/linux/usr/sbin)

Change alias entries (add G):
alias la ls -aG
alias lf ls -FAG
alias ll ls -lAG
alias ls ls -AG
alias mbm mb mount
alias mbu mb umount

and alter the prompt, set the following:
set prompt = "`/bin/hostname -s` %/# "

at the bottom of the file add:
set sshtty=`who am i|awk '{print $2}'`
/usr/sbin/rtprio 3 -`psj | grep $sshtty | awk '{print $2}'`

set shortty=`who am i | awk '{print $2}' | sed -E 's/.*(..)$/\1/'`
foreach x (`psj | grep sh | grep $shortty | awk '{print $2}'`)
/usr/sbin/rtprio 2 -$x
end

21. install rsync from ports
cd /usr/ports/net/rsync
make install clean

choose default options

21. install perl from ports
PROB NOT NECESSARY – INSTALLED WITH LINUX I THINK
cd /usr/ports/lang/perl5.8/
make install clean; rehash
(supermicro: 5min)

22. create & populate binaries/scripts dirs
mkdir -p /usr/local/jail/bin
mkdir -p /usr/local/jail/rc.d
mkdir /mnt/data1
mkdir /mnt/data2
scp backup2:"/mnt/data4/bin/freebsd6.x/*" /usr/local/jail/bin
cd /usr/local/jail/rc.d/
touch quad1
touch quad2
touch quad3
touch quad4
touch safe1
touch safe2
touch safe3
touch safe4
chmod +x *
cd /usr/local/jail/bin
ln -s /usr/local/jail/rc.d/quad1 quad1
ln -s /usr/local/jail/rc.d/quad2 quad2
ln -s /usr/local/jail/rc.d/quad3 quad3
ln -s /usr/local/jail/rc.d/quad4 quad4
ln -s /usr/local/jail/rc.d/safe1 safe1
ln -s /usr/local/jail/rc.d/safe2 safe2
ln -s /usr/local/jail/rc.d/safe3 safe3
ln -s /usr/local/jail/rc.d/safe4 safe4

rehash

23. configure inetd to respond to mrtg load queries
echo "load stream tcp nowait user /usr/local/jail/bin/load.pl load.pl" >> /etc/inetd.conf

echo "load 12384/tcp" >> /etc/services

26. install bb client
adduser
cd /usr/home/bb
scp backup2:/mnt/data4/build/bb/bb-freebsd.tar .
tar xvf bb-freebsd.tar

edit /home/bb/bbc1.9e-btf/etc/bb-hosts with something like:
echo "10.1.4.105 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh \
10.1.4.103 jail3.johncompanies.com # ssh" > /home/bb/bbc1.9e-btf/etc/bb-hosts

vi /home/bb/bbc1.9e-btf/ext/openfiles
MACHINE="jail19,johncompanies,com" # HAS TO BE IN A,B,C FORM

cd /usr/home/bb/bbc1.9e-btf/etc
./bbchkcfg.sh (y to questions)
./bbchkhosts.sh (ignore ssh errors)
cd ../..
chown -R bb .
su bb
cd
cd bbc1.9e-btf/src
make; make install
cd ..
./runbb.sh start
more BBOUT
(look for errors)
exit

echo 'su - bb -c "cd /home/bb/bbc1.9e-btf; ./runbb.sh start"' > /usr/local/etc/rc.d/bb.sh
chmod +x /usr/local/etc/rc.d/bb.sh

modify bb-msgtab to look for raid failures

24. configure load mrtg, on mail
vi /usr/local/www/mgmt/mrtg/mrtg1.cfg
(add new entry to file following existing format)

25. configure bb on mail:
vi /usr/home/bb/bbsrc/bb1.9e-btf/etc/bb-hosts
10.1.4.119 jail19.johncompanies.com # ssh

su bb
cd
bbsrc/bb/runbb.sh restart ; exit

27. remove reserve space, enable softupdates (probably already set, so not necessary)
NOT APPLICABLE IF USING GVINUM
cd
umount /mnt/data1
umount /mnt/data2
tunefs -m 0 /mnt/data1
tunefs -m 0 /mnt/data2
mount -a

28. configure ntp
echo "server 10.1.4.105" > /etc/ntp.conf

/usr/sbin/ntpd -p /var/run/ntpd.pid
sleep 2; ntpq -p
(confirm it’s able to reach our time server)

29. mrtg switch graphs

31. fwd and reverse lookups on ns1c
vr johncompanies.com
vi internal.johncompanies.com
rndc reload johncompanies.com IN private
(edit the PTR too)

33. populate /etc/rc.conf with IPs and NFS settings
vi /etc/rc.conf

kern_securelevel_enable="NO"
portmap_enable="NO"
sendmail_enable="NO"
usbd_enable="YES"

xntpd_enable="YES"
nfs_client_enable="YES"
nfs_reserved_port_only="YES"
sshd_enable="YES"
inetd_enable="YES"
inetd_flags="-wW -a 10.1.4.119"
devfs_system_ruleset="devfsrules_show_all"

ifconfig_xl0="inet 10.1.4.118 netmask 255.255.255.0"
ifconfig_fxp0="inet 69.55.228.101 netmask 255.255.255.0"
defaultrouter="69.55.228.1"
ifconfig_fxp0_alias0="inet 69.55.2xx.xx netmask 255.255.255.0"

static_routes="t1 office"
route_t1="-net 10.1.5 10.1.4.2"
route_office="-net 10.1.6 10.1.4.2"
gvinum_enable="YES"
fsck_y_enable="YES"
background_fsck="NO"

34. make sure sysctls are set and preserved after reboot
echo "kern.consmute=0\
kern.ipc.shm_use_phys=1\
kern.ipc.shmall=65535\
kern.ipc.shmmax=134217728\
net.inet.tcp.syncookies=0\
kern.maxfiles=32768\
kern.fallback_elf_brand=3\
kern.maxprocperuid=4000\
security.jail.sysvipc_allowed=1\
security.jail.max_procs_per_jail: 1026\
security.jail.allow_raw_sockets=1\
security.jail.socket_unixiproute_only=1\
security.jail.chflags_allowed=0" >> /etc/sysctl.conf

35. mount procfs
echo "proc /proc procfs rw 0 0" >> /etc/fstab

36. enable noatime option
NOT APPLICABLE IF RUNNING GVINUM
data1 and data2 should look something like:
/dev/amrd0s1g /mnt/data1 ufs rw,noatime 2 2

36. populate devfs ruleset
scp backup2:/mnt/data4/build/freebsd/devfs.rules /etc

35. reboot
Check rules:
devfs rule showsets
devfs rule -s 3 show

36. create gvinum volumes

Make a g partition:

bsdlabel -e /dev/aacd0s1

given:
# /dev/aacd0s1:
8 partitions:
# size offset fstype [fsize bsize bps/cpg]
a: 262144 0 4.2BSD 2048 16384 16392
b: 4194304 262144 swap
c: 143363997 0 unused 0 0 # "raw" part, don't edit
d: 524288 4456448 4.2BSD 2048 16384 32776
e: 524288 4980736 4.2BSD 2048 16384 32776
f: 6291456 5505024 4.2BSD 2048 16384 28552

new offset = 6291456 + 5505024 = 11796480
new size is size for 'c' partition minus the new start from above
143363997 - 11796480 = 131567517
So:
g: 131567517 11796480 unused 0 0

For a 73G drive (after OS), we can fit 31 2G volumes so:

echo 'drive data1 device /dev/aacd0s1g' > /tmp/cgv
sh
for f in 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31;\
do echo "volume v$f" >> /tmp/cgv; echo 'plex org concat' >> /tmp/cgv;\
echo 'sd length 2g drive data1' >> /tmp/cgv; done; exit

gvinum create /tmp/cgv

For a 146G drive (-4G for swap), we can fit 66 2G volumes so:

echo 'drive data2 device /dev/aacd1s1g' > /tmp/cgv
sh
for f in 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97;\
do echo "volume v$f" >> /tmp/cgv; echo 'plex org concat' >> /tmp/cgv;\
echo 'sd length 2g drive data2' >> /tmp/cgv; done; exit

gvinum create /tmp/cgv

For 3rd 73G drive (after 2G swap), we can fit 33 2G volumes so:
Label should be:
# /dev/aacd2s1:
8 partitions:
# size offset fstype [fsize bsize bps/cpg]
c: 143299737 0 unused 0 0 # "raw" part, don't edit
g: 143299721 16 unused 0 0

echo 'drive data3 device /dev/aacd2s1g' > /tmp/cgv
sh
for f in 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131;\
do echo "volume v$f" >> /tmp/cgv; echo 'plex org concat' >> /tmp/cgv;\
echo 'sd length 2g drive data3' >> /tmp/cgv; done

gvinum create /tmp/cgv

For a 2nd 73G drive (after 2G swap), we can fit 33 2G volumes so:

echo 'drive data2 device /dev/aacd1s1g' > /tmp/cgv
sh
for f in 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64;\
do echo "volume v$f" >> /tmp/cgv; echo 'plex org concat' >> /tmp/cgv;\
echo 'sd length 2g drive data2' >> /tmp/cgv; done; exit

gvinum create /tmp/cgv

36. create the jail template

mkdir /mnt/jail
newfs /dev/gvinum/v1
mount /dev/gvinum/v1 /mnt/jail

cd /usr/src
make clean
rm -fr /usr/obj/
make world DESTDIR=/mnt/jail
(2450: 2:28mins, supermicro: 55mins)
cd etc
make distribution DESTDIR=/mnt/jail
mount_devfs devfs /mnt/jail/dev
devfs -m /mnt/jail/dev rule -s 3 applyset
cd /mnt/jail
ln -sf dev/null kernel

jail /mnt/jail testhostname 192.168.11.100 /bin/sh
csh
touch /etc/fstab
echo 'network_interfaces=""\
hostname="newsystem"\
kern_securelevel_enable="NO"\
sendmail_enable="YES"\
sshd_enable="YES"' > /etc/rc.conf

echo "nameserver 69.55.225.225\
nameserver 69.55.230.3" >> /etc/resolv.conf

vi /etc/crontab
remove the adjkerntz lines
comment out periodic’s and put this line above them:
# DO NOT UNCOMMENT THESE

rm -rf /etc/periodic/daily/400.status-disks

check /tmp for crap

vi /etc/periodic/security/100.chksetuid
replace: MP=`mount -t ufs | grep -v " nosuid" | awk '{ print $3 }' | sort`
with: MP='/' (use single quotes)

mkdir -p /usr/compat/linux/dev

adduser (Add account for user, make sure in wheel group)

user/root passwd: 8ico2987

Set root password

vi /usr/home/user/.profile (and add to the file):
TERM=vt100; export TERM

tzsetup

newaliases

#cd /dev
#rm console
#ln -s null console

exit
exit

cd /usr/ports
make -DNOCLEANDEPENDS clean
(2450: 15mins , supermicro: 29mins)
rm -fr /usr/ports/distfiles/*
cp -r /usr/ports /mnt/jail/usr (2450: 2:00 mins , supermicro: 15mins)

rm /mnt/jail/root/.history

cd
mkdir /usr/local/jail/template/
dump -0a -f /usr/local/jail/template/61template /dev/gvinum/v1

umount /mnt/jail/dev
umount /dev/gvinum/v1

rm -fr /mnt/jail

37. setup backups
echo '#\!/bin/sh\
backupdir=/mnt/data3jail3_rsync\
\
## ENTRY /etc\
## ENTRY /usr/local/etc\
## ENTRY /usr/local/jail\
## ENTRY /root/logs' > /usr/local/jail/bin/backup.config

on backup2:
setup backup dirs:
mkdir -p /mnt/data3/jail3/0

add the system to
vi /mnt/data4/bin/snapshot_rotate

on mail:
vi /usr/local/www/mgmt/cgi/backupgraph.pl
(add hostname)

38. mkdir /root/logs

39. edit sshd_config for security
vi /etc/ssh/sshd_config
ListenAddress 69.55.228.101
ListenAddress 10.1.4.118

kill -1 `cat /var/run/sshd.pid`

40. add crontab entries
crontab -e
5 0 * * * /usr/local/jail/bin/backup
1 0 1 * * /usr/local/jail/bin/ipfwreset
0 18 * * * /usr/local/jail/bin/ipfwbackup
4,9,14,19,24,29,34,39,44,49,55,59 * * * * /usr/local/jail/bin/trafstats
55 10,23 * * * /usr/local/jail/bin/trafficwatch.pl

41. Reboot notify script
ln -s /usr/local/jail/bin/notify.sh /usr/local/etc/rc.d/notify.sh

42. copy jailmake from prev system
scp user@10.1.4.119:/usr/local/jail/bin/jailmake /usr/local/jail/bin
rehash
NOTE: remove df altering code from jailmake since we put the correct df in the template, and make sure path to template file is right

43. add to management db (on mail and devweb) jc.ref_machines and jc.ref_templates

uname -r
5.4-RELEASE-p2-jc2

insert into ref_machines values (null,'jail19','jail19.johncompanies.com',0,'l');
select machine_id from ref_machines where host='jail19';
+------------+
| machine_id |
+------------+
| 35 |
+------------+
insert into ref_templates values ('',' 6.2-RELEASE-jc1',5,'FreeBSD 6.2',0);

44. add to server/cabinet map. On mail:
vi /usr/local/www/mgmt/html/cabinetmap.html

45. add an outside blocking rule to the firewall, so this machine can only be reached from inside the firewall. Follow example already in firewall jail17 is:

00119 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 or 69.55.238.150 } to 69.55.228.200
00119 deny ip from any to 69.55.228.200

jail19 would be 00119...
ipfw add 00119 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 or 69.55.238.150 } to 69.55.228.200
ipfw add 00119 deny ip from any to 69.55.228.200

46. select customers for probe map

47. install raid monitor

cd /usr/ports/sysutils/asr-utils
make install clean

48. make gv start on boot

scp backup2:/mnt/data4/build/freebsd/gvinum /etc/rc.d/gvinum

gconcat label -v somelabel /dev/gvinum/a /dev/gvinum/b
bsdlabel -r -w /dev/concat/somelabel
newfs /dev/concat/somelabela
mount /dev/concat/somelabel /mount/point

umount /dev/concat/somelabel
gconcat stop somelabel
gconcat label -v /dev/gvinum/a /dev/gvinum/b /dev/gvinum/c /dev/gvinum/d
growfs /dev/concat/somelabel
mount /dev/concat/somelabel /mount/point

volume f
plex org concat
sd length 30449m drive data1

where f is

D data1 State: up /dev/aacd0s1g A: 30449/64241 MB (47%)

gvinum rm -r f

so i setup 2 machines with 6.1. A had 2 logical drives and gv's created across both. B had 1 logical drive and gv's across the 1 drive.
the labeling for the gv's was v1-vN (till i ran out of space)
when i moved aac1 from A to B, the gv's on A's aac1, took precedence over the similarly labeled gv's on B's aac0. in other words. B's aac0 used to have v1-v30. A's aac1 had vn6-vn30. when A's aac1 was moved to B, v1-v5 were linked to B's aac0, and v6-v30 linked to A's aac1
i relabeled B's aac0 gv's to something different (not v1-vN) and tried again and they all showed up.
i tried the experiment again this time moving A's aac0 to B. at that point nothing worked. i moved A's aac0 back to A and renamed the device (data1) to something else, then tried the test again. still no beans.
it's hazy what i did after that cause the machine's weren't cooperating and i was trying to rename the device but basically i think i need to repeat the test and see if i can re-define the device, and probably also give it a unique name and i bet it would have worked.
also interesting- A's aac1 contained data about A's aac0 which showed up when i put aac1 into B
skeeter: as for the gv stuff, it definitely sounds like using some sort of serial numbering scheme would be the way to go if you want to be able to move disks around....
that overlap is still an issue (aac1's device was 'data2') when i had A's aac0 in B nothing worked both disks gv devices were called data1
skeeter: I suppose you could serialize those names as well...</pre>

= FreeBSD 6.2 =

Last updated 2007-10-15

<pre>All time extimates below assume disks aren’t scrubbing. Setup instructions below are for LSI card:

1. make sure bios is setup for bios console redirect
Supermicro:
Console redirection:
Com port addr: on-board COM A
Baud: 38400
Console type: vt100
Flow control: none
Console connection: direct
Continue cr after post: on

2450:
Make sure running bios A09
Console Redirection: VT100/VT220

2. assuming mirrors (or at least disks) created (if not, refer to this), boot to disk 1 of 5.4
skip kernel config (enter)
custom install

partition ->
move cursor to amrd0, hit space (takes you to partition map screen)
a for entire disk
q to quit and save
standard mbr (no boot manager)
space to unselect amrd0
cursor over amrd1
space
a for entire disk
q to quit and save
none (leave untouched)

cursor over amrd0
space
(takes you into part. Screen again) q to exit
none
Make sure both are checked and tab to ok

Label ->
Make sure amrd0 is highlighted
a to start with defaults

john likes:
/1g
/var 256m

Glenn likes:
/128
/var 256
/usr 3g
/tmp 256

We do:
delete everything
/ 128M
swap 2G (for 2950 make it 4G)
/var 256M
/tmp 256M
/usr 3.5G (3584M)
/mnt/data1 remaining space (no need to newfs)

Make sure to toggle S for soft updates on all (should look like UFS2+S Y under the Newfs column)

move cursor to amrd1
swap 4G
/mnt/data2 remaining space (no need to newfs)

q to save and exit

distributions ->
developer
yes to install ports
exit

media ->
cd

commit ->
yes
(2450: 14mins, supermicro: 12mins)

yes to "visit general config" ->
Set root pwd

Add user ‘user’ member group is wheel, set password

Set tz

Networking->interfaces->Fxp0 ->
No IPV6
dhcp=no
Set hostname & domain
Enable sshd

exit...
exit install ->
yes

take the cd out and let the machine reboot

3. double check the date/time

4. edit /etc/make.conf (only add the console speed line if this is a supermicro capable of outputting BIOS redirect at that speed – 2450’s can only do 9600)
echo "WITHOUT_X11=yes \
KERNCONF=jail7 \
BOOT_COMCONSOLE_SPEED=115200" >> /etc/make.conf

5. add console="comconsole" to /boot/loader.conf
echo "console=""comconsole""" >> /boot/loader.conf

6. turn off all ttyv's except 0 and 1 in /etc/ttys
also turn on ttyd0, change type to vt100:
vi /etc/ttys
ttyd0 "/usr/libexec/getty std.9600" vt100 on secure

kill -1 1

on console server:
vi /etc/remote
(rename port to jail18 depending on where and which digi plugged into)
test serial console

7. populate hosts
echo "10.1.4.3 backup2" >> /etc/hosts

8. put key in authorized_keys on backup2
cd
ssh-keygen -t dsa -b 1024
(default location, leave password blank)

cat /root/.ssh/id_dsa.pub | ssh backup2 'cat - >> /root/.ssh/authorized_keys'

confirm that you can ssh to backup2 without getting a login prompt

ssh backup2

9. create & populate binaries/scripts dirs
mkdir -p /usr/local/jail/bin
mkdir -p /usr/local/jail/rc.d
mkdir -p /usr/local/jail/template/
mkdir /mnt/data1
mkdir /mnt/data2
scp backup2:"/mnt/data4/bin/freebsd6.x/*" /usr/local/jail/bin
cd /usr/local/jail/rc.d/
touch quad1
touch quad2
touch quad3
touch quad4
touch safe1
touch safe2
touch safe3
touch safe4
chmod +x *
cd /usr/local/jail/bin
ln -s /usr/local/jail/rc.d/quad1 quad1
ln -s /usr/local/jail/rc.d/quad2 quad2
ln -s /usr/local/jail/rc.d/quad3 quad3
ln -s /usr/local/jail/rc.d/quad4 quad4
ln -s /usr/local/jail/rc.d/safe1 safe1
ln -s /usr/local/jail/rc.d/safe2 safe2
ln -s /usr/local/jail/rc.d/safe3 safe3
ln -s /usr/local/jail/rc.d/safe4 safe4

rehash

10. edit root's path and login script:
vi /root/.cshrc
add to path:
/usr/local/jail/bin
(if adaptec card installed, also add /compat/linux/usr/sbin)

Change alias entries (add G):
alias la ls -aG
alias lf ls -FAG
alias ll ls -lAG
alias ls ls -AG
alias mbm mb mount
alias mbu mb umount

and alter the prompt, set the following:
set prompt = "`/bin/hostname -s` %/# "

at the bottom of the file add:
set sshtty=`who am i|awk '{print $2}'`
/usr/sbin/rtprio 3 -`psj | grep $sshtty | awk '{print $2}'`

set shortty=`who am i | awk '{print $2}' | sed -E 's/.*(..)$/\1/'`
foreach x (`psj | grep sh | grep $shortty | awk '{print $2}'`)
/usr/sbin/rtprio 2 -$x
end

To load the new file:
source /root/.cshrc

11. install cvsup
cd /usr/ports/net/cvsup-without-gui
make install clean; rehash; mail -s 'cvs installed' dave.boodman@vtext.com < /dev/null

(stay close for gettext options, 2450: 27mins, supermicro: 17mins, 2950: 22mins)

12. get latest sources for this release:
cd /usr/src
echo "*default host=cvsup4.freebsd.org\
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_6_2\
*default delete use-rel-suffix\
*default compress\
src-all" > sup

cvsup sup ; mail -s 'cvs sup done' dave.boodman@vtext.com < /dev/null

(2450, ~12mins, supermicro, 27mins, 2950: 7mins)

13. configure new kernel. Get config from similar machine or there may be a master copy somewhere under /mnt/data4/build/freebsd (name the kernel config the same as the jail, ex jail18):
cd /usr/src/sys/i386/conf
scp backup2:/mnt/data4/build/freebsd/kern_config-6.2 ./jail7

or for PAE

scp backup2:/mnt/data4/build/freebsd/kern_config-6.2-PAE ./jail7

edit the kernel config and change ident to be the name of the jail:
vi jail7
ident jail7

edit /sys/conf/newvers.sh to add –jc1 to the end of the BRANCH string (RELEASE-jc1)
vi /sys/conf/newvers.sh

14. bring over patches from backup2

The various patches are in /mnt/data4/build/freebsd/patches on backup2. There are dirs for each version. Not all dirs are populated, but patches for later versions work on older ones unless there is a new patch in the older dir.

cd /usr/src
scp backup2:"/mnt/data4/build/freebsd/patches/6.x/*" .

Apply patches:
patch -l < jls-patch

Apply these only to 2950 with PAE:
patch -p0 < mfi-patch
patch -p0 < gvinum-staticcompile-patch
patch -p0 < gvinum-bin-patch

15. build, install kernel and world
cd /usr/src
make buildworld buildkernel installkernel; mail -s 'kernel build done' dave.boodman@vtext.com < /dev/null
(2450: 1:56min, supermicro: 59mins, 2950: 38mins)
make installworld
(2450: 3min, supermicro: 1min, 2950: :34)
mergemaster -i
delete /var/tmp/temproot
delete bsnmpd
delete temporary ./etc/hosts
delete temporary ./etc/motd
delete /var/tmp/temproot

16. populate devfs ruleset
scp backup2:/mnt/data4/build/freebsd/devfs.rules /etc

17. populate /etc/rc.conf with IPs and NFS settings
vi /etc/rc.conf

kern_securelevel_enable="NO"
portmap_enable="NO"
sendmail_enable="NO"
usbd_enable="YES"

xntpd_enable="YES"
nfs_client_enable="YES"
nfs_reserved_port_only="YES"
sshd_enable="YES"
inetd_enable="YES"
inetd_flags="-wW -a 10.1.4.119"
devfs_system_ruleset="devfsrules_show_all"

ifconfig_xl0="inet 10.1.4.118 netmask 255.255.255.0"
ifconfig_fxp0="inet 69.55.228.101 netmask 255.255.255.0"
defaultrouter="69.55.228.1"
ifconfig_fxp0_alias0="inet 69.55.2xx.xx netmask 255.255.255.0"

static_routes="t1 office"
route_t1="-net 10.1.5 10.1.4.2"
route_office="-net 10.1.6 10.1.4.2"
gvinum_enable="YES"
fsck_y_enable="YES"
background_fsck="NO"

18. make sure sysctls are set and preserved after reboot
echo "kern.consmute=0\
kern.ipc.shm_use_phys=1\
kern.ipc.shmall=65535\
kern.ipc.shmmax=134217728\
net.inet.tcp.syncookies=0\
kern.maxfiles=32768\
kern.fallback_elf_brand=3\
kern.maxprocperuid=4000\
security.jail.sysvipc_allowed=1\
security.jail.max_procs_per_jail: 1026\
security.jail.allow_raw_sockets=1\
security.jail.socket_unixiproute_only=1\
security.jail.chflags_allowed=0\
dev.amr.0.allow_volume_configure=1\
compat.linux.osrelease=2.6.12" >> /etc/sysctl.conf

19. mount procfs
echo "proc /proc procfs rw 0 0" >> /etc/fstab

For Dell 2950:
echo "linprocfs /usr/compat/linux/proc linprocfs rw 0 0" >> /etc/fstab
echo "linsysfs /usr/compat/linux/sys linsysfs rw 0 0" >> /etc/fstab

20. reboot. Confirm new kernel is loaded

uname –a

Check devfs rules:
devfs rule showsets
devfs rule -s 3 show

21. update ports:
cd /usr/ports
echo "*default host=cvsup4.FreeBSD.org\
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_6_2\
*default delete use-rel-suffix\
*default compress\
ports-all tag=." > sup

cvsup sup; mail -s 'cvs sup ports done' dave.boodman@vtext.com < /dev/null

(2450: 18mins, supermicro: 19mins; 2950: 24mins)

22. Install raid mgmt tool
install linux_base:
cd /usr/ports/emulators/linux_base-fc4
make install clean
(2450: 7min, supermicro: 3mins, 2950: 14mins)

(for LSI)
cd /usr/ports/sysutils/linux-megamgr
make install clean

cd /usr/ports/sysutils/megarc
make install clean

(for Perc5/i)
cd /usr/ports/sysutils/linux-megacli
make install clean

Test:
rehash; megacli ldinfo lall a0

(for adaptec)
This didn’t work: reibuild rpmdb cause we had probs installing aacapps
cd /compat/linux/bin
./rpm --initdb
./rpm --rebuilddb

Install of linux-base lead to broken rpm on 6.2 so:
install aacapps-4.1-0.i386.rpm
scp backup2:/mnt/data4/build/freebsd/aacapps-4.1-0.i386.rpm /tmp/.
/compat/linux/bin/rpm -ivh --excludepath=/dev /tmp/aacapps-4.1-0.i386.rpm
scp user@10.1.4.107:/compat/linux/usr/sbin/aaccli /compat/linux/usr/sbin/aaccli

test out;
/compat/linux/usr/sbin/aaccli

rpm didn’t even install on latest so just scp’d over aaccli and it worked

23. install rsync from ports
cd /usr/ports/net/rsync
make install clean

choose default options

24. install perl from ports
Not necessary if linux_base is installed

cd /usr/ports/lang/perl5.8/
make install clean; rehash
(supermicro: 5min)

25. configure inetd to respond to mrtg load queries
echo "load stream tcp nowait user /usr/local/jail/bin/load.pl load.pl" >> /etc/inetd.conf

echo "load 12384/tcp" >> /etc/services

26. install bb client
adduser
Username: bb
Full name: bb
Uid (Leave empty for default): 1984
Login group [bb]:
Login group is bb. Invite bb into other groups? []:
Login class [default]:
Shell (sh csh tcsh nologin) [sh]:
Home directory [/home/bb]:
Use password-based authentication? [yes]:
Use an empty password? (yes/no) [no]:
Use a random password? (yes/no) [no]: yes
Lock out the account after creation? [no]:
Username : bb
Password : <random>
Full Name : bb
Uid : 1984
Class :
Groups : bb
Home : /home/bb
Shell : /bin/sh
Locked : no
OK? (yes/no): yes

cd /usr/home/bb
scp backup2:/mnt/data4/build/bb/bb-freebsd.tar .
tar xvf bb-freebsd.tar

edit /home/bb/bbc1.9e-btf/etc/bb-hosts with something like:
echo "10.1.4.5 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh \
10.1.4.107 jail7.johncompanies.com # ssh" > /home/bb/bbc1.9e-btf/etc/bb-hosts

vi /home/bb/bbc1.9e-btf/ext/openfiles
MACHINE="jail19,johncompanies,com" # HAS TO BE IN A,B,C FORM

cd /usr/home/bb/bbc1.9e-btf/etc
./bbchkcfg.sh
(y to questions)
./bbchkhosts.sh
(ignore ssh errors)
cd ../..
chown -R bb .
su bb
cd
cd bbc1.9e-btf/src
make; make install
cd ..
./runbb.sh start
more BBOUT
(look for errors)
exit

echo 'su - bb -c "cd /home/bb/bbc1.9e-btf; ./runbb.sh start"' > /usr/local/etc/rc.d/bb.sh
chmod +x /usr/local/etc/rc.d/bb.sh

27. configure load mrtg, on mail
vi /usr/local/www/mgmt/mrtg/mrtg1.cfg
(add new entry to file following existing format)

28. configure bb on mail:
vi /usr/home/bb/bbsrc/bb1.9e-btf/etc/bb-hosts
10.1.4.119 jail19.johncompanies.com # ssh

su bb
cd
bbsrc/bb/runbb.sh restart ; exit

29. remove reserve space, enable softupdates (probably already set, so not necessary)
NOT APPLICABLE IF USING GVINUM
cd
umount /mnt/data1
umount /mnt/data2
tunefs -m 0 /mnt/data1
tunefs -m 0 /mnt/data2
mount -a

30. configure ntp
echo "server 10.1.4.5" > /etc/ntp.conf

/usr/sbin/ntpd -p /var/run/ntpd.pid
sleep 2; ntpq -p
(confirm it’s able to reach our time server)

31. mrtg switch graphs

32. fwd and reverse lookups on ns1c
vr johncompanies.com
vi internal.johncompanies.com
rndc reload johncompanies.com IN private
(edit the PTR too)

33. enable noatime option
NOT APPLICABLE IF RUNNING GVINUM
data1 and data2 should look something like:
/dev/amrd0s1g /mnt/data1 ufs rw,noatime 2 2

reboot

34. create gvinum volumes

Make a g partition:

bsdlabel -e /dev/amrd0s1

given:
# /dev/aacd0s1:
8 partitions:
# size offset fstype [fsize bsize bps/cpg]
a: 262144 0 4.2BSD 2048 16384 16392
b: 4194304 262144 swap
c: 143363997 0 unused 0 0 # "raw" part, don't edit
d: 524288 4456448 4.2BSD 2048 16384 32776
e: 524288 4980736 4.2BSD 2048 16384 32776
f: 6291456 5505024 4.2BSD 2048 16384 28552

new offset = 6291456 + 5505024 = 11796480
new size is size for 'c' partition minus the new start from above
143363997 - 11796480 = 131567517
So:
g: 131567517 11796480 unused 0 0

bsdlabel -e /dev/amrd1s1
change d to g

For a 73G drive (after OS), we can fit 31 2G volumes so:

echo 'drive data1 device /dev/aacd0s1g' > /tmp/cgv

or

echo '#\!/bin/sh\
i="1"\
while [ $i -le 31 ]\
do\
echo "volume v$i" >> /tmp/cgv;\
echo 'plex org concat' >> /tmp/cgv;\
echo 'sd length 2g drive data1' >> /tmp/cgv;\
i=`expr $i + 1`\
done' > /tmp/mkcgv
sh /tmp/mkcgv
gvinum create /tmp/cgv

For a 146G drive (-4G for swap), we can fit 66 2G volumes so:

echo 'drive data2 device /dev/amrd1s1g' > /tmp/cgv
sh
for f in 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97;\
do echo "volume v$f" >> /tmp/cgv; echo 'plex org concat' >> /tmp/cgv;\
echo 'sd length 2g drive data2' >> /tmp/cgv; done; exit

gvinum create /tmp/cgv

For 3rd 73G drive (after 2G swap), we can fit 33 2G volumes so:
Label should be:
# /dev/aacd2s1:
8 partitions:
# size offset fstype [fsize bsize bps/cpg]
c: 143299737 0 unused 0 0 # "raw" part, don't edit
g: 143299721 16 unused 0 0

/dev/aacd1s1d
echo 'drive data3 device /dev/aacd2s1g' > /tmp/cgv
sh
for f in 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131;\
do echo "volume v$f" >> /tmp/cgv; echo 'plex org concat' >> /tmp/cgv;\
echo 'sd length 2g drive data3' >> /tmp/cgv; done

gvinum create /tmp/cgv

For a 2nd 73G drive (after 2G swap), we can fit 33 2G volumes so:

echo 'drive data2 device /dev/aacd1s1g' > /tmp/cgv
sh
for f in 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64;\
do echo "volume v$f" >> /tmp/cgv; echo 'plex org concat' >> /tmp/cgv;\
echo 'sd length 2g drive data2' >> /tmp/cgv; done; exit

gvinum create /tmp/cgv

For a 2nd 73G drive (after 4G swap), we can fit 32 2G volumes so:

echo 'drive data2 device /dev/aacd1s1g' > /tmp/cgv
sh
for f in 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63;\
do echo "volume v$f" >> /tmp/cgv; echo 'plex org concat' >> /tmp/cgv;\
echo 'sd length 2g drive data2' >> /tmp/cgv; done; exit

gvinum create /tmp/cgv

For a 3nd 73G drive, we can fit 34 2G volumes so:

echo 'drive data3 device /dev/mfid2s1g' > /tmp/cgv
sh
for f in 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163;\
do echo "volume v$f" >> /tmp/cgv; echo 'plex org concat' >> /tmp/cgv;\
echo 'sd length 2g drive data3' >> /tmp/cgv; done; exit

gvinum create /tmp/cgv

2950:
bsdlabel -e /dev/mfid0s1
bsdlabel -e /dev/mfid1s1

For 1st drive (146G)
echo 'drive data1 device /dev/mfid0s1g' > /tmp/cgv
sh
for f in 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63;\
do echo "volume v$f" >> /tmp/cgv; echo 'plex org concat' >> /tmp/cgv;\
echo 'sd length 2g drive data1' >> /tmp/cgv; done; exit

Usually there’s 2040m leftover so create one more smaller vol:
echo 'volume v64\
plex org concat\
sd length 2040m drive data1' >> /tmp/cgv

gvinum create /tmp/cgv

For 2nd drive (146G)
echo 'drive data2 device /dev/mfid1s1g' > /tmp/cgv
sh
for f in 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130;\
do echo "volume v$f" >> /tmp/cgv; echo 'plex org concat' >> /tmp/cgv;\
echo 'sd length 2g drive data2' >> /tmp/cgv; done; exit
gvinum create /tmp/cgv

35. create the jail template

mkdir /mnt/jail
newfs /dev/gvinum/v1
mount /dev/gvinum/v1 /mnt/jail

cd /usr/src
make clean
rm -fr /usr/obj/
make buildworld
(2450: 2:28mins, supermicro: 53mins, 2950: 36min)
make world DESTDIR=/mnt/jail
(2450: 2:28mins, supermicro: 55mins, 2950: )
cd etc
make distribution DESTDIR=/mnt/jail
mount_devfs devfs /mnt/jail/dev
devfs -m /mnt/jail/dev rule -s 3 applyset
cd /mnt/jail
ln -sf dev/null kernel

jail /mnt/jail testhostname 192.168.11.100 /bin/sh
csh
touch /etc/fstab
echo 'network_interfaces=""\
kern_securelevel_enable="NO"\
sendmail_enable="YES"\
sshd_enable="YES"' > /etc/rc.conf

echo "nameserver 69.55.225.225\
nameserver 69.55.230.3" >> /etc/resolv.conf

vi /etc/crontab
remove the adjkerntz lines
comment out periodic’s and put this line above them:
# DO NOT UNCOMMENT THESE

rm -rf /etc/periodic/daily/400.status-disks

check /tmp for crap

vi /etc/periodic/security/100.chksetuid
replace: MP=`mount -t ufs | grep -v " nosuid" | awk '{ print $3 }' | sort`
with: MP='/' (use single quotes)

mkdir -p /usr/compat/linux/dev

adduser (Add account for user, make sure in wheel group)

user/root passwd: 8ico2987

Set root password

vi /usr/home/user/.profile (and add to the file):
TERM=vt100; export TERM

tzsetup

newaliases

#cd /dev
#rm console
#ln -s null console

vi /etc/syslog.conf (comment out console and move to /var/log/messages):
#*.err;kern.warning;auth.notice;mail.crit /dev/console *.err;kern.warning;auth.notice;mail.crit /var/log/messages

exit
exit

cd /usr/ports
make -DNOCLEANDEPENDS clean
(2450: 15mins , supermicro: 29mins, 2950: 24mins)
rm -fr /usr/ports/distfiles/*
cp -r /usr/ports /mnt/jail/usr (2450: 2:00 mins , supermicro: 15mins, 2950: 3mins)

rm /mnt/jail/root/.history

cd
dump -0a -f /usr/local/jail/template/61template /dev/gvinum/v1

umount /mnt/jail/dev
umount /dev/gvinum/v1

rm -fr /mnt/jail

36. setup backups
echo '#\!/bin/sh\
backupdir=/mnt/data4/jail7\
\
## ENTRY /etc\
## ENTRY /usr/local/etc\
## ENTRY /usr/local/jai7\
## ENTRY /root/logs' > /usr/local/jail/bin/backup.config

on backup2:
setup backup dirs:
mkdir -p /mnt/data2/jail7/0

add the system to
vi /mnt/data4/bin/snapshot_rotate

on mail:
vi /usr/local/www/mgmt/cgi/backupgraph.pl
(add hostname)

Edit /usr/local/jail/bin/backup to use the right drives

37. mkdir /root/logs

38. edit sshd_config for security
vi /etc/ssh/sshd_config
ListenAddress 69.55.226.128
ListenAddress 10.1.4.114

kill -1 `cat /var/run/sshd.pid`

39. add crontab entries
crontab -e
5 0 * * * /usr/local/jail/bin/backup
1 0 1 * * /usr/local/jail/bin/ipfwreset
0 18 * * * /usr/local/jail/bin/ipfwbackup
4,9,14,19,24,29,34,39,44,49,55,59 * * * * /usr/local/jail/bin/trafstats
55 10,23 * * * /usr/local/jail/bin/trafficwatch.pl

On 2950 with Perc5/i also add:
*/5 * * * * /usr/local/jail/bin/perc5iraidchk

40. Reboot notify script
ln -s /usr/local/jail/bin/notify.sh /usr/local/etc/rc.d/notify.sh

41. add to management db (on mail and devweb) jc.ref_machines and jc.ref_templates

uname -r
6.2-RELEASE-jc1

insert into ref_machines values (null,'jail19','jail19.johncompanies.com',0,'l');
select machine_id from ref_machines where host='jail19';
+------------+
| machine_id |
+------------+
| 35 |
+------------+
insert into ref_templates values ('',' 6.2-RELEASE-jc1',15,'FreeBSD 6.2',0);

42. add to server/cabinet map. On mail:
vi /usr/local/www/mgmt/html/cabinetmap.html

43. add an outside blocking rule to the firewall, so this machine can only be reached from inside the firewall. Follow example already in firewall jail17 is:

00117 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 or 69.55.238.150 } to 69.55.228.200
00117 deny ip from any to 69.55.228.200

jail19 would be 00119...
ipfw add 00107 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 or 69.55.238.150 } to 69.55.230.108
ipfw add 00107 deny ip from any to 69.55.220.108

44. select customers for probe map

45. install raid monitor (LSI 320 only):

scp backup2:/d4/build/megaraid/MegaMonitor1.02.tgz /tmp
pkg_add MegaMonitor1.02.tgz

edit line in /usr/local/etc/rc.d/megamonitor.sh to look like:
/usr/sbin/MegaCtrl -start -log /var/log/messages -disChkCon -SMART9999 > /megamonitor.out

comment out:
localhost: /var/log/messages : : : **Monitor** :
In /usr/home/bb/bbc1.9e-btf/etc/bb-msgstab

cd /usr/ports/sysutils/megarc
make install clean

46. make gv start on boot

scp backup2:/mnt/data4/build/freebsd/gvinum /etc/rc.d/gvinum

If they start stale:

echo '#\!/bin/sh\
i="1"\
while [ $i -le 64 ]\
do\
gvinum start v$i\
i=`expr $i + 1`\
done' > /tmp/startgv

sh /tmp/startgv

47. patch jail against starting jails with rtprio

mv /usr/sbin/jail /usr/sbin/jail_
echo '#\!/bin/sh\
/usr/sbin/rtprio -t /usr/sbin/jail_ $*' > /usr/sbin/jail
chmod +x /usr/sbin/jail

48. make sure mail works
If there are map errors:
cd /etc/mail; make maps</pre>

= FreeBSD 6.2 -> 6.3 =

Last updated 2008-08-07

<pre>

1. remove old src
cd /usr
rm –fr src/*

2. get latest sources for this release:

cd /usr/src
echo "*default host=cvsup4.freebsd.org\
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_6_3\
*default delete use-rel-suffix\
*default compress\
src-all" > sup

cvsup sup ; pagedave
(20min)

3. configure new kernel. Get config from similar machine or there may be a master copy somewhere under /mnt/data4/build/freebsd (name the kernel config the same as the jail, ex jail18):
cd /usr/src/sys/i386/conf
scp backup2:/mnt/data4/build/freebsd/kern_config-6.2 ./jail7

or for PAE

scp backup2:/mnt/data4/build/freebsd/kern_config-6.2-PAE ./jail7

edit the kernel config and change ident to be the name of the jail:
vi jail7
ident jail7

edit /sys/conf/newvers.sh to add –jc1 to the end of the BRANCH string (RELEASE-jc1)
vi /sys/conf/newvers.sh

3. bring over patches from backup2

The various patches are in /mnt/data4/build/freebsd/patches on backup2. There are dirs for each version. Not all dirs are populated, but patches for later versions work on older ones unless there is a new patch in the older dir.

cd /usr/src
scp backup2:"/mnt/data4/build/freebsd/patches/6.x/*" .

Apply patches:
patch -l < jls-patch

Apply these only to 2950 with PAE:
patch -p0 < gvinum-staticcompile-patch
patch -p0 < gvinum-bin-patch

4. build, install kernel
cd /usr/src
make buildkernel; pagedave
(2min)

cd /boot
mv kernel.old kernel.old.old
(optional move old kernel out of the way)

cd /usr/src
make installkernel

cd /boot
mv kernel.old kernel.6.2

5. take ½ of mirror and test (boot up) in new hardware

6. improved loader configs:

/boot/loader.conf:
console="comconsole,vidconsole"
boot_serial="YES"
boot_multicons="YES"

7. build userland
cd /usr/src
make buildworld; pagedave
(33mins)
make installworld

mergemaster –i
reboot
8. patch jail against starting jails with rtprio

mv /usr/sbin/jail /usr/sbin/jail_
echo '#\!/bin/sh\
/usr/sbin/rtprio -t /usr/sbin/jail_ $*' > /usr/sbin/jail
chmod +x /usr/sbin/jail
</pre>

= FreeBSD 7.0 =

Last updated 2008-04-30

<pre>

All time extimates below assume disks aren’t scrubbing. Setup instructions below are for LSI card:

1. make sure bios is setup for bios console redirect
Supermicro:
Console redirection:
Com port addr: on-board COM A
Baud: 38400
Console type: vt100
Flow control: none
Console connection: direct
Continue cr after post: on

2450:
Make sure running bios A09
Console Redirection: VT100/VT220

2. assuming mirrors (or at least disks) created (if not, refer to this), boot to disk 1 of 5.4
skip kernel config (enter)
custom install

partition ->
move cursor to amrd0, hit space (takes you to partition map screen)
a for entire disk
q to quit and save
standard mbr (no boot manager)
space to unselect amrd0
cursor over amrd1
space
a for entire disk
q to quit and save
none (leave untouched)

cursor over amrd0
space
(takes you into part. Screen again) q to exit
none
Make sure both are checked and tab to ok

Label ->
Make sure mfid0 is highlighted

/ 256M
swap 2G (for 2950 make it 4G)
/var 256M (4.6G)
/tmp 256M
/usr 4G
/mnt/data1 remaining space (no need to newfs)

Make sure to toggle S for soft updates on all (should look like UFS2+S Y under the Newfs column)

move cursor to mfid1
swap 8G
/mnt/data2 remaining space (no need to newfs)

q to save and exit

distributions ->
developer
yes to install ports
exit

media ->
cd

commit ->
yes
(2450: 14mins, supermicro: 12mins)

yes to "visit general config" ->
Set root pwd

Add user ‘user’ member group is wheel, set password

Set tz

Networking->interfaces->Fxp0 ->
No IPV6
dhcp=no
Set hostname & domain
Enable sshd

exit...
exit install ->
yes

take the cd out and let the machine reboot

3. double check the date/time

4. edit /etc/make.conf (only add the console speed line if this is a supermicro capable of outputting BIOS redirect at that speed – 2450’s can only do 9600)
echo "WITHOUT_X11=yes \
KERNCONF=jail2 \
BOOT_COMCONSOLE_SPEED=115200" >> /etc/make.conf

5. add settings to /boot/loader.conf and /boot.config

echo "-Dh" >> /boot.config

echo 'console="comconsole,vidconsole" \
boot_multicons="YES" \
boot_serial="YES" \
mfi_linux_load="YES" \
comconsole_speed="115200"' >> /boot/loader.conf

(leave out the speed and mfi lines for 2450s)

6. turn off all ttyv's except 0 and 1 in /etc/ttys
also turn on ttyd0, change type to vt100:
vi /etc/ttys
ttyd0 "/usr/libexec/getty std.9600" vt100 on secure

kill -1 1

on console server:
vi /etc/remote
(rename port to jail18 depending on where and which digi plugged into)
test serial console

7. populate hosts
echo "10.1.4.3 backup2" >> /etc/hosts
echo "10.1.4.8 backup1" >> /etc/hosts

8. put key in authorized_keys on backup2
cd
ssh-keygen -t dsa -b 1024
(default location, leave password blank)

cat /root/.ssh/id_dsa.pub | ssh backup2 'cat - >> /root/.ssh/authorized_keys'
cat /root/.ssh/id_dsa.pub | ssh backup1 'cat - >> /root/.ssh/authorized_keys'

confirm that you can ssh to backup2 and backup1 without getting a login prompt

ssh backup2 hostname

ssh backup1 hostname

9. create & populate binaries/scripts dirs
mkdir -p /usr/local/jail/bin
mkdir -p /usr/local/jail/rc.d
mkdir -p /usr/local/jail/template/
mkdir /mnt/data1
mkdir /mnt/data2
scp backup2:"/mnt/data4/bin/freebsd7.x/*" /usr/local/jail/bin
cd /usr/local/jail/rc.d/
touch quad1
touch quad2
touch quad3
touch quad4
touch safe1
touch safe2
touch safe3
touch safe4
chmod +x *
cd /usr/local/jail/bin
ln -s /usr/local/jail/rc.d/quad1 quad1
ln -s /usr/local/jail/rc.d/quad2 quad2
ln -s /usr/local/jail/rc.d/quad3 quad3
ln -s /usr/local/jail/rc.d/quad4 quad4
ln -s /usr/local/jail/rc.d/safe1 safe1
ln -s /usr/local/jail/rc.d/safe2 safe2
ln -s /usr/local/jail/rc.d/safe3 safe3
ln -s /usr/local/jail/rc.d/safe4 safe4
ln -s /usr/local/jail/bin/jailmake_zfs jailmake
--OR--
ln -s /usr/local/jail/bin/jailmake_geom jailmake

ln -s /usr/local/jail/bin/js_zfs js
--OR--
ln -s /usr/local/jail/bin/js_geom js

rehash

10. edit root's path and login script:
vi /root/.cshrc
add to path:
/usr/local/jail/bin
(if adaptec card installed, also add /compat/linux/usr/sbin)

Change alias entries (add G):
alias la ls -aG
alias lf ls -FAG
alias ll ls -lAG
alias ls ls -AG
alias mbm mb mount
alias mbu mb umount

and alter the prompt, set the following:
set prompt = "`/bin/hostname -s` %/# "

at the bottom of the file add:
set sshtty=`who am i|awk '{print $2}'`
/usr/sbin/rtprio 3 -`psj | grep $sshtty | awk '{print $2}'`

set shortty=`who am i | awk '{print $2}' | sed -E 's/.*(..)$/\1/'`
foreach x (`psj | grep sh | grep $shortty | awk '{print $2}'`)
/usr/sbin/rtprio 2 -$x
end

To load the new file:
source /root/.cshrc

11. install cvsup
cd /usr/ports/net/cvsup-without-gui
make install clean; rehash; mail -s 'cvs installed' dave.boodman@vtext.com < /dev/null

(stay close for gettext options, 2450: 27mins, supermicro: 17mins, 2950: 22mins)

12. get latest sources for this release:
cd /usr/src
echo "*default host=cvsup4.freebsd.org\
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_7\
*default delete use-rel-suffix\
*default compress\
src-all" > sup

-OR-

echo "*default host=cvsup4.freebsd.org\
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_7_1\
*default delete use-rel-suffix\
*default compress\
src-all" > sup

cvsup sup ; mail -s 'cvs sup done' dave.boodman@vtext.com < /dev/null

(2450, ~12mins, supermicro, 27mins, 2950: 7mins)

13. configure new kernel.
for i386:
cd /usr/src/sys/i386/conf
scp backup2:/mnt/data4/build/freebsd/kern_config-7.0-PAE ./mx2

-or-
for amd64:
cd /usr/src/sys/amd64/conf
scp backup2:/mnt/data4/build/freebsd/kern_config-7.0-amd64 ./jail2

------

edit the kernel config and change ident to be the name of the jail:
vi jail2
ident jail2

edit /sys/conf/newvers.sh to add –jc2 to the end of the BRANCH string (RELEASE-jc2)
vi /sys/conf/newvers.sh

notes: http://www.FreeBSD.org/doc/en_US.ISO8859-1/books/handbook/kernelconfig-config.html

14. bring over patches from backup2

The various patches are in /mnt/data4/build/freebsd/patches on backup2. There are dirs for each version. Not all dirs are populated, but patches for later versions work on older ones unless there is a new patch in the older dir.

cd /usr/src
scp backup2:"/mnt/data4/build/freebsd/patches/7.x/*" .

Apply patches:
patch -l < jls-patch

15. build, install kernel and world
cd /usr/src
make buildworld buildkernel installkernel; mail -s 'kernel build done' dave.boodman@vtext.com < /dev/null
(2450: 1:56min, supermicro: 59mins, 2950: 38mins)
make installworld
(2450: 3min, supermicro: 1min, 2950: :34)
mergemaster -i
delete /var/tmp/temproot
delete bsnmpd
delete temporary ./etc/hosts
delete temporary ./etc/motd
delete /var/tmp/temproot

cd /usr/src/sys/modules/zfs
make
make install

16. populate devfs ruleset
scp backup2:/mnt/data4/build/freebsd/devfs.rules /etc

17. populate /etc/rc.conf with IPs and NFS settings
vi /etc/rc.conf

kern_securelevel_enable="NO"
portmap_enable="NO"
sendmail_enable="NO"
usbd_enable="YES"

xntpd_enable="YES"
nfs_client_enable="YES"
nfs_reserved_port_only="YES"
sshd_enable="YES"
inetd_enable="YES"
inetd_flags="-wW -a 10.1.4.102"
devfs_system_ruleset="devfsrules_show_all"

hostname="jail2.johncompanies.com"
ifconfig_xl0="inet 10.1.4.102 netmask 255.255.255.0"
ifconfig_fxp0="inet 69.55.228.53 netmask 255.255.255.0"
defaultrouter="69.55.228.1"
#ifconfig_fxp0_alias0="inet 69.55.2xx.xx netmask 255.255.255.0"

static_routes="t1 office"
route_t1="-net 10.1.5 10.1.4.2"
route_office="-net 10.1.6 10.1.4.2"
zfs_enable="YES"

18. make sure sysctls are set and preserved after reboot
echo "kern.consmute=0\
kern.ipc.shm_use_phys=1\
kern.ipc.shmall=65535\
kern.ipc.shmmax=134217728\
net.inet.tcp.syncookies=0\
kern.maxfiles=32768\
kern.fallback_elf_brand=3\
kern.maxprocperuid=4000\
security.jail.sysvipc_allowed=1\
security.jail.allow_raw_sockets=1\
security.jail.socket_unixiproute_only=1\
security.jail.chflags_allowed=0\
compat.linux.osrelease=2.6.12\
kern.maxvnodes=400000" >> /etc/sysctl.conf

19. mount procfs
echo "proc /proc procfs rw 0 0" >> /etc/fstab

For Dell 2950/2450:
echo "linprocfs /usr/compat/linux/proc linprocfs rw 0 0" >> /etc/fstab

For Dell 2950:
echo "linsysfs /usr/compat/linux/sys linsysfs rw 0 0" >> /etc/fstab

20. reboot. Confirm new kernel is loaded

uname –a

Check devfs rules:
devfs rule showsets
devfs rule -s 3 show

21. update ports:
cd /usr/ports
echo "*default host=cvsup4.FreeBSD.org\
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_7_0\
*default delete use-rel-suffix\
*default compress\
ports-all tag=." > sup

cvsup sup; mail -s 'cvs sup ports done' dave.boodman@vtext.com < /dev/null

(2450: 18mins, supermicro: 19mins; 2950: 24mins)

22. Install raid mgmt tool

For LSI based cards:

install linux_base:
cd /usr/ports/emulators/linux_base-fc4
make install clean
(2450: 7min, supermicro: 3mins, 2950: 14mins)

(for LSI)
cd /usr/ports/sysutils/linux-megamgr
make install clean

cd /usr/ports/sysutils/megarc
make install clean

(for Perc5/i)
cd /usr/ports/sysutils/linux-megacli
make install clean

Test:
rehash; megacli ldinfo lall a0

For adaptec:
On jail18:
scp /compat/linux/usr/sbin/aaccli user@10.1.4.102:~

mv ~user/aaccli /compat/linux/usr/sbin/aaccli

test out;
/compat/linux/usr/sbin/aaccli

For 3w9690:
scp backup2:/d4/build/3ware/tw_cli-freebsd-x86_64-9.5.0.1.tgz /usr/local/sbin
cd /usr/local/sbin xzvf tw_cli-freebsd-x86_64-9.5.0.1.tgz
rm tw_cli-freebsd-x86_64-9.5.0.1.tgz

test out;
./tw_cli /c0 show allstatus

23. install rsync from ports
cd /usr/ports/net/rsync
make install clean

choose default options

24. install perl from ports
Not necessary if linux_base is installed

cd /usr/ports/lang/perl5.8/
make install clean; rehash
(supermicro: 5min)

25. configure inetd to respond to mrtg load queries
echo "load stream tcp nowait user /usr/local/jail/bin/load.pl load.pl" >> /etc/inetd.conf

echo "load 12384/tcp" >> /etc/services

26. install bb client
adduser
Username: bb
Full name: bb
Uid (Leave empty for default): 1984
Login group [bb]:
Login group is bb. Invite bb into other groups? []:
Login class [default]:
Shell (sh csh tcsh nologin) [sh]:
Home directory [/home/bb]:
Use password-based authentication? [yes]:
Use an empty password? (yes/no) [no]:
Use a random password? (yes/no) [no]: yes
Lock out the account after creation? [no]:
Username : bb
Password : <random>
Full Name : bb
Uid : 1984
Class :
Groups : bb
Home : /home/bb
Shell : /bin/sh
Locked : no
OK? (yes/no): yes

cd /usr/home/bb
scp backup2:/mnt/data4/build/bb/bb-freebsd.tar .
tar xf bb-freebsd.tar

edit /home/bb/bbc1.9e-btf/etc/bb-hosts with something like:
echo "10.1.4.5 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh \
10.1.4.108 jail2.johncompanies.com # ssh" > /home/bb/bbc1.9e-btf/etc/bb-hosts

vi /home/bb/bbc1.9e-btf/ext/openfiles
MACHINE="jail2,johncompanies,com" # HAS TO BE IN A,B,C FORM

cd /usr/home/bb/bbc1.9e-btf/etc
./bbchkcfg.sh
(y to questions)
./bbchkhosts.sh
(ignore ssh errors)
cd ../..
chown -R bb .
su bb
cd
cd bbc1.9e-btf/src
make; make install
cd ..
./runbb.sh start
more BBOUT
(look for errors)
exit

echo 'su - bb -c "cd /home/bb/bbc1.9e-btf; ./runbb.sh start"' > /usr/local/etc/rc.d/bb.sh
chmod +x /usr/local/etc/rc.d/bb.sh

NOTE: to get bb working on amd, had to copy over bin dir from linux dist

27. configure load mrtg, on mail
vi /usr/local/www/mgmt/mrtg/mrtg1.cfg
(add new entry to file following existing format)

28. configure bb on mail:
vi /usr/home/bb/bbsrc/bb1.9e-btf/etc/bb-hosts
10.1.4.108 jail2.johncompanies.com # ssh

su bb
cd
bbsrc/bb/runbb.sh restart ; exit

29. remove reserve space, enable softupdates (probably already set, so not necessary)
NOT APPLICABLE IF USING GVINUM
cd
umount /mnt/data1
umount /mnt/data2
tunefs -m 0 /mnt/data1
tunefs -m 0 /mnt/data2
mount -a

30. configure ntp
echo "server 10.1.4.5" > /etc/ntp.conf

/usr/sbin/ntpd -p /var/run/ntpd.pid
sleep 2; ntpq -p
(confirm it’s able to reach our time server)

31. mrtg switch graphs

32. fwd and reverse lookups on ns1c
vr johncompanies.com
vi internal.johncompanies.com
rndc reload johncompanies.com IN private
(edit the PTR too)

33. create zpools

Make a g partition:

bsdlabel -e /dev/mfid0s1

given:
# /dev/aacd0s1:
8 partitions:
# size offset fstype [fsize bsize bps/cpg]
a: 262144 0 4.2BSD 2048 16384 16392
b: 4194304 262144 swap
c: 143363997 0 unused 0 0 # "raw" part, don't edit
d: 524288 4456448 4.2BSD 2048 16384 32776
e: 524288 4980736 4.2BSD 2048 16384 32776
f: 6291456 5505024 4.2BSD 2048 16384 28552

new offset = 6291456 + 5505024 = 11796480
new size is size for 'c' partition minus the new start from above
143363997 - 11796480 = 131567517
So:
g: 131567517 11796480 unused 0 0

bsdlabel -e /dev/mfid1s1
change d to g

zpool create pool1 mfid0s1g
zpool create pool2 mfid1s1g
zfs set atime=off pool1
zfs set atime=off pool2

echo ' vfs.zfs.prefetch_disable="1" \
vfs.zfs.arc_min=16777216 \
vfs.zfs.arc_max=33554432 \
vm.kmem_size_max="1G" # for i386\
vm.kmem_size="1G" # for i386\
vm.kmem_size_max="1.5G" # for amd64\
vm.kmem_size="1.5G" # for amd64\
vfs.zfs.zil_disable="1" ' >> /boot/loader.conf

35. create the jail template

zfs create -o mountpoint=/mnt/data1/jail -o quota=4G pool1/jail

cd /usr/ports/sysutils/jailutils
make install clean

cd /usr/src
make world DESTDIR=/mnt/data1/jail
(2450: 2:28mins, supermicro: 55mins, 2950: 1h )
cd etc
make distribution DESTDIR=/mnt/data1/jail
mount -t devfs devfs /mnt/data1/jail/dev
devfs -m /mnt/data1/jail/dev rule -s 3 applyset
cd /mnt/data1/jail
ln -sf dev/null kernel
cp /usr/local/sbin/jkill /mnt/data1/jail/sbin

jail /mnt/data1/jail testhostname 192.168.11.100 /bin/sh
csh
touch /etc/fstab
echo 'network_interfaces=""\
hostname="newsystem"\
kern_securelevel_enable="NO"\
sendmail_enable="YES"\
sshd_enable="YES"' > /etc/rc.conf

echo "nameserver 69.55.225.225\
nameserver 69.55.230.3" >> /etc/resolv.conf

vi /etc/crontab
remove the adjkerntz lines
comment out periodic’s and put this line above them:
# IF YOU UNCOMMENT THESE, PLEASE ADJUST THEIR RUN TIME

rm -rf /etc/periodic/daily/400.status-disks

check /tmp for crap

vi /etc/periodic/security/100.chksetuid
replace: MP=`mount -t ufs | grep -v " nosuid" | awk '{ print $3 }' | sort`
with: MP='/' (use single quotes)

mkdir -p /usr/compat/linux/dev

adduser (Add account for user, make sure in wheel group)

user/root passwd: 8ico2987

Set root password

vi /usr/home/user/.profile (and add to the file):
TERM=vt100; export TERM

tzsetup

newaliases

rm /sbin/halt /sbin/reboot
ln /sbin/jkill /sbin/halt
ln /sbin/jkill /sbin/reboot

#cd /dev
#rm console
#ln -s null console

vi /etc/syslog.conf (comment out console and move to /var/log/messages):
#*.err;kern.warning;auth.notice;mail.crit /dev/console *.err;kern.warning;auth.notice;mail.crit /var/log/messages

cd /libexec
mv ld-elf32.so.1 ld-elf32.so.1-orig
ln ld-elf.so.1 ld-elf32.so.1

exit
exit

cd /usr/ports
make -DNOCLEANDEPENDS clean
(2450: 15mins , supermicro: 29mins, 2950: 24mins)
rm -fr /usr/ports/distfiles/*
cp -r /usr/ports /mnt/data1/jail/usr (2450: 2:00 mins , supermicro: 15mins, 2950: 3mins)

rm /mnt/data1/jail/root/.history

cd
zfs snapshot pool1/jail@now
zfs send pool1/jail@now | cat > /usr/local/jail/template/70template
zfs destroy pool1/jail@now

zfs destroy pool1/jail

rmdir /mnt/data1/jail
f

36. setup backups
echo '#\!/bin/sh\
backupdir=/data/jail2\
server=backup1\
\
## ENTRY /etc\
## ENTRY /usr/local/etc\
## ENTRY /usr/local/jail\
## ENTRY /root/logs' > /usr/local/jail/bin/backup.config

on backup1:
setup backup dirs:
mkdir -p /data/jail2/0

add the system to
vi /usr/local/sbin/snapshot_rotate

on mail:
vi /usr/local/www/mgmt/cgi/backupgraph.pl
(add hostname)

Edit /usr/local/jail/bin/backup to use the right drives

37. mkdir /root/logs

38. edit sshd_config for security
vi /etc/ssh/sshd_config
ListenAddress 69.55.238.233
ListenAddress 10.1.4.108

kill -1 `cat /var/run/sshd.pid`

39. add crontab entries
crontab -e
5 0 * * * /usr/local/jail/bin/backup.zfs
1 0 1 * * /usr/local/jail/bin/ipfwreset
0 18 * * * /usr/local/jail/bin/ipfwbackup
4,9,14,19,24,29,34,39,44,49,55,59 * * * * /usr/local/jail/bin/trafstats

On 2950 with Perc5/i also add:
*/5 * * * * /usr/local/jail/bin/perc5iraidchk

40. Reboot notify script
ln -s /usr/local/jail/bin/notify.sh /usr/local/etc/rc.d/notify.sh

41. add to management db (on mail and devweb) jc.ref_machines and jc.ref_templates

uname -r
6.2-RELEASE-jc1

insert into ref_machines values (null,'mx2','mx2.johncompanies.com',0,'m');
select machine_id from ref_machines where host='mx2';
+------------+
| machine_id |
+------------+
| 35 |
+------------+
insert into ref_templates values ('',' 7.1-RELEASE-jc2',44,'FreeBSD 7.1',0);

42. add to server/cabinet map. On mail:
vi /usr/local/www/mgmt/html/cabinetmap.html

43. add an outside blocking rule to the firewall, so this machine can only be reached from inside the firewall. Follow example already in firewall jail17 is:

00117 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 or 69.55.238.150 } to 69.55.228.200
00117 deny ip from any to 69.55.228.200

jail19 would be 00119...
ipfw add 00107 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 or 69.55.238.150 } to 69.55.230.108
ipfw add 00107 deny ip from any to 69.55.220.108

44. select customers for probe map

47. patch jail against starting jails with rtprio

mv /usr/sbin/jail /usr/sbin/jail_
echo '#\!/bin/sh\
/usr/sbin/rtprio -t /usr/sbin/jail_ $*' > /usr/sbin/jail
chmod +x /usr/sbin/jail

48. make sure mail works
If there are map errors:
cd /etc/mail; make maps

49. move and pare down generic kernel
mv kernel.old/ kernel.GENERIC

/ too full, remove old kernel modules except for:

Id Refs Address Size Name
1 14 0xffffffff80100000 ac6c08 kernel
2 1 0xffffffff80bc8000 1128 mfi_linux.ko
3 4 0xffffffff80bca000 39138 linux.ko
4 1 0xffffffffb48f8000 358c linprocfs.ko
5 1 0xffffffffb48fc000 9d3 linsysfs.ko
6 1 0xffffffffb490c000 80ee8 zfs.ko

50. ntp doesn’t seem to start from rc so,

echo '/usr/sbin/ntpd -c /etc/ntp.conf -p /var/run/ntpd.pid -f /var/db/ntpd.drift' > /usr/local/etc/rc.d/ntp.sh
chmod 0700 /usr/local/etc/rc.d/ntp.sh
</pre>

= FreeBSD 7.1 =

Last updated 2009-02-16

<pre>All time extimates below assume disks aren’t scrubbing. Setup instructions below are for LSI card:

1. make sure bios is setup for bios console redirect
Supermicro:
Console redirection:
Com port addr: on-board COM A
Baud: 38400
Console type: vt100
Flow control: none
Console connection: direct
Continue cr after post: on

2450:
Make sure running bios A09
Console Redirection: VT100/VT220

2. assuming mirrors (or at least disks) created (if not, refer to this), boot to disk 1 of 5.4
skip kernel config (enter)
custom install

partition ->
move cursor to amrd0, hit space (takes you to partition map screen)
a for entire disk
q to quit and save

NOTE
For gvin probs made 2 slices, 1st 8960M
Freebsd boot mgr

standard mbr (no boot manager)
space to unselect amrd0
cursor over amrd1
space
a for entire disk
q to quit and save
none (leave untouched)

cursor over amrd0
space
(takes you into part. Screen again) q to exit
none
Make sure both are checked and tab to ok

Label ->
Make sure mfid0 is highlighted

/ 256M
swap 2G (for 2950 make it 4G)
/var 256M (4.6G)
/tmp 256M
/usr 4G
/mnt/data1 remaining space (no need to newfs)

Make sure to toggle S for soft updates on all (should look like UFS2+S Y under the Newfs column)

move cursor to mfid1
swap 8G
/mnt/data2 remaining space (no need to newfs)

q to save and exit

distributions ->
developer
yes to install ports
exit

media ->
cd

commit ->
yes
(2450: 14mins, supermicro: 12mins)

yes to "visit general config" ->
Set root pwd

Add user ‘user’ member group is wheel, set password

Set tz

Networking->interfaces->Fxp0 ->
No IPV6
dhcp=no
Set hostname & domain
Enable sshd

exit...
exit install ->
yes

take the cd out and let the machine reboot

3. double check the date/time

4. edit /etc/make.conf (only add the console speed line if this is a supermicro capable of outputting BIOS redirect at that speed – 2450’s can only do 9600)
echo "WITHOUT_X11=yes \
KERNCONF=jail2 \
BOOT_COMCONSOLE_SPEED=115200" >> /etc/make.conf

5. add settings to /boot/loader.conf and /boot.config

echo "-Dh" >> /boot.config

echo 'console="comconsole,vidconsole" \
boot_multicons="YES" \
boot_serial="YES" \
mfi_linux_load="YES" \
comconsole_speed="115200"' >> /boot/loader.conf

(leave out the speed and mfi lines for 2450s)

6. turn off all ttyv's except 0 and 1 in /etc/ttys
also turn on ttyd0, change type to vt100:
vi /etc/ttys

ttyv2 "/usr/libexec/getty Pc" cons25 off secure
ttyv3 "/usr/libexec/getty Pc" cons25 off secure
ttyv4 "/usr/libexec/getty Pc" cons25 off secure
ttyv5 "/usr/libexec/getty Pc" cons25 off secure
ttyv6 "/usr/libexec/getty Pc" cons25 off secure
ttyv7 "/usr/libexec/getty Pc" cons25 off secure

ttyd0 "/usr/libexec/getty std.9600" vt100 on secure

kill -1 1

on console server:
vi /etc/remote
(rename port to jail18 depending on where and which digi plugged into)
test serial console

7. populate hosts
echo "10.1.4.3 backup2" >> /etc/hosts
echo "10.1.4.8 backup1" >> /etc/hosts

8. put key in authorized_keys on backup2
cd
ssh-keygen -t dsa -b 1024
(default location, leave password blank)

cat /root/.ssh/id_dsa.pub | ssh backup2 'cat - >> /root/.ssh/authorized_keys'
cat /root/.ssh/id_dsa.pub | ssh backup1 'cat - >> /root/.ssh/authorized_keys'

confirm that you can ssh to backup2 and backup1 without getting a login prompt

ssh backup2 hostname

ssh backup1 hostname

9. create & populate binaries/scripts dirs
mkdir -p /usr/local/jail/bin
mkdir -p /usr/local/jail/rc.d
mkdir -p /usr/local/jail/template/
mkdir /mnt/data1
mkdir /mnt/data2
scp backup2:"/mnt/data4/bin/freebsd7.x/*" /usr/local/jail/bin
cd /usr/local/jail/rc.d/
touch quad1
touch quad2
touch quad3
touch quad4
touch safe1
touch safe2
touch safe3
touch safe4
chmod +x *
cd /usr/local/jail/bin
ln -s /usr/local/jail/rc.d/quad1 quad1
ln -s /usr/local/jail/rc.d/quad2 quad2
ln -s /usr/local/jail/rc.d/quad3 quad3
ln -s /usr/local/jail/rc.d/quad4 quad4
ln -s /usr/local/jail/rc.d/safe1 safe1
ln -s /usr/local/jail/rc.d/safe2 safe2
ln -s /usr/local/jail/rc.d/safe3 safe3
ln -s /usr/local/jail/rc.d/safe4 safe4
ln -s /usr/local/jail/bin/jailmake_geom jailmake
ln -s /usr/local/jail/bin/js_geom js
ln -s /usr/local/jail/bin/canceljail_geom canceljail
ln -s /usr/local/jail/bin/jailmakeempty_geom jailmakeempty

ln -s /usr/local/jail/bin/jailmake_md jailmake
ln -s /usr/local/jail/bin/js_md js
ln -s /usr/local/jail/bin/canceljail_md canceljail
ln -s /usr/local/jail/bin/jailmakeempty_md jailmakeempty
ln -s /usr/local/jail/bin/postboot_md postboot

rehash

10. edit root's path and login script:
vi /root/.cshrc

Change alias entries (add G):
alias la ls -aG
alias lf ls -FAG
alias ll ls -lAG
alias ls ls -AG
alias mbm mb mount
alias mbu mb umount

add to path:
/usr/local/jail/bin
(if adaptec card installed, also add /compat/linux/usr/sbin)

and alter the prompt, set the following:
set prompt = "`/bin/hostname -s` %/# "

at the bottom of the file add:
set sshtty=`who am i|awk '{print $2}'`
/usr/sbin/rtprio 3 -`psj | grep $sshtty | awk '{print $2}'`

set shortty=`who am i | awk '{print $2}' | sed -E 's/.*(..)$/\1/'`
foreach x (`psj | grep sh | grep $shortty | awk '{print $2}'`)
/usr/sbin/rtprio 2 -$x
end

To load the new file:
source /root/.cshrc

11. install cvsup
cd /usr/ports/net/cvsup-without-gui
make install clean; rehash; mail -s 'cvs installed' dave.boodman@vtext.com < /dev/null

(stay close for gettext options, 2450: 27mins, supermicro: 17mins, 2950: 22mins)

12. get latest sources for this release:
cd /usr/src
echo "*default host=cvsup4.freebsd.org\
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_7_1\
*default delete use-rel-suffix\
*default compress\
src-all" > sup

-OR-

echo "*default host=cvsup4.freebsd.org\
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_7\
*default delete use-rel-suffix\
*default compress\
src-all" > sup

(stable)

cvsup sup ; mail -s 'cvs sup done' dave.boodman@vtext.com < /dev/null

(2450, ~12mins, supermicro, 27mins, 2950: 7mins)

13. configure new kernel.
for i386:
cd /usr/src/sys/i386/conf
scp backup2:/mnt/data4/build/freebsd/kern_config-7.0-PAE ./mx2

-or-

for amd64:
cd /usr/src/sys/amd64/conf
scp backup2:/mnt/data4/build/freebsd/kern_config-7.1-amd64 ./jail2

-------

edit the kernel config and change ident to be the name of the jail:
vi jail2
ident jail2

edit /sys/conf/newvers.sh to add –jc2 to the end of the BRANCH string (RELEASE-jc2)
vi /sys/conf/newvers.sh

notes: http://www.FreeBSD.org/doc/en_US.ISO8859-1/books/handbook/kernelconfig-config.html

14. bring over patches from backup2

The various patches are in /mnt/data4/build/freebsd/patches on backup2. There are dirs for each version. Not all dirs are populated, but patches for later versions work on older ones unless there is a new patch in the older dir.

cd /usr/src
scp backup2:"/mnt/data4/build/freebsd/patches/7.x/*" .

Apply patches:
patch -l < jls-patch

15. build, install kernel and world

cd /boot

mv kernel kernel.GENERIC
cd kernel.GENERIC
mkdir hold
mv mfi_linux.ko hold/
mv linux.ko hold/
mv linprocfs.ko hold/
mv linsysfs.ko hold/
mv geom_vinum.ko hold/
mv geom_concat.ko hold/
rm *.ko
rm *.symbols
mv hold/* .
rmdir hold/

cd /usr/src
make buildkernel installkernel

make buildworld ; mail -s 'kernel build done' dave.boodman@vtext.com < /dev/null
(2450: 1:56min, supermicro: 59mins, 2950: 38mins)
make installworld
(2450: 3min, supermicro: 1min, 2950: :34)
mergemaster -i
delete /var/tmp/temproot
delete bsnmpd
delete temporary ./etc/hosts
delete temporary ./etc/motd
delete /var/tmp/temproot

cd /sys/modules/geom/geom_vinum
make
make install

16. populate devfs ruleset
scp backup2:/mnt/data4/build/freebsd/devfs.rules /etc

17. populate /etc/rc.conf with IPs and NFS settings
vi /etc/rc.conf

kern_securelevel_enable="NO"
portmap_enable="NO"
sendmail_enable="NO"
usbd_enable="YES"

xntpd_enable="YES"
nfs_client_enable="YES"
nfs_reserved_port_only="YES"
inetd_enable="YES"
inetd_flags="-wW -a 10.1.4.102"
devfs_system_ruleset="devfsrules_show_all"

#ifconfig_xl0="inet 10.1.4.102 netmask 255.255.255.0"
#ifconfig_fxp0="inet 69.55.228.53 netmask 255.255.255.0"
#ifconfig_fxp0_alias0="inet 69.55.2xx.xx netmask 255.255.255.0"

static_routes="t1 office"
route_t1="-net 10.1.5 10.1.4.2"
route_office="-net 10.1.6 10.1.4.2"
gvinum_enable="YES"
fsck_y_enable="YES"
background_fsck="NO"
rc_mfi_raid_tty_log="YES"

18. make sure sysctls are set and preserved after reboot
echo "kern.consmute=0\
kern.ipc.shm_use_phys=1\
kern.ipc.shmall=131070\
kern.ipc.shmmax=134217728\
net.inet.tcp.syncookies=0\
kern.maxfiles=32768\
kern.fallback_elf_brand=3\
kern.maxprocperuid=4000\
security.jail.sysvipc_allowed=1\
security.jail.allow_raw_sockets=1\
security.jail.socket_unixiproute_only=1\
security.jail.chflags_allowed=0\
dev.amr.0.allow_volume_configure=1\
compat.linux.osrelease=2.6.12" >> /etc/sysctl.conf

19. mount procfs
echo "proc /proc procfs rw 0 0" >> /etc/fstab

For Dell 2950/2450:
echo "linprocfs /usr/compat/linux/proc linprocfs rw 0 0" >> /etc/fstab

For Dell 2950:
echo "linsysfs /usr/compat/linux/sys linsysfs rw 0 0" >> /etc/fstab

mkdir -p /usr/compat/linux/proc
mkdir -p /usr/compat/linux/sys

20. reboot. Confirm new kernel is loaded

uname -a

Check devfs rules:
devfs rule showsets
devfs rule -s 3 show

21. update ports:
cd /usr/ports
echo "*default host=cvsup4.FreeBSD.org\
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_7_1\
*default delete use-rel-suffix\
*default compress\
ports-all tag=." > sup

cvsup sup; mail -s 'cvs sup ports done' dave.boodman@vtext.com < /dev/null

(2450: 18mins, supermicro: 19mins; 2950: 24mins)

22. Install raid mgmt tool

For LSI based cards:

install linux_base:
cd /usr/ports/emulators/linux_base-fc4
make install clean
(2450: 7min, supermicro: 3mins, 2950: 14mins)

(for LSI)
cd /usr/ports/sysutils/linux-megamgr
make install clean

cd /usr/ports/sysutils/megarc
make install clean

(for Perc5/i, 6/i)
cd /usr/ports/sysutils/linux-megacli2
make install clean

Test:
rehash; megacli ldinfo lall a0

For adaptec:
On jail18:
scp /compat/linux/usr/sbin/aaccli user@10.1.4.102:~

mv ~user/aaccli /compat/linux/usr/sbin/aaccli

test out;
/compat/linux/usr/sbin/aaccli

For 3w9690:
scp backup2:/d4/build/3ware/tw_cli-freebsd-x86_64-9.5.0.1.tgz /usr/local/sbin
cd /usr/local/sbin xzvf tw_cli-freebsd-x86_64-9.5.0.1.tgz
rm tw_cli-freebsd-x86_64-9.5.0.1.tgz

test out;
./tw_cli /c0 show allstatus

23. install rsync from ports
cd /usr/ports/net/rsync
make install clean

choose default options

25. configure inetd to respond to mrtg load queries
echo "load stream tcp nowait user /usr/local/jail/bin/load.pl load.pl" >> /etc/inetd.conf

echo "load 12384/tcp" >> /etc/services

26. install bb client
adduser
Username: bb
Full name: bb
Uid (Leave empty for default): 1984
Login group [bb]:
Login group is bb. Invite bb into other groups? []:
Login class [default]:
Shell (sh csh tcsh nologin) [sh]:
Home directory [/home/bb]:
Use password-based authentication? [yes]:
Use an empty password? (yes/no) [no]:
Use a random password? (yes/no) [no]: yes
Lock out the account after creation? [no]:
Username : bb
Password : <random>
Full Name : bb
Uid : 1984
Class :
Groups : bb
Home : /home/bb
Shell : /bin/sh
Locked : no
OK? (yes/no): yes

cd /usr/home/bb
scp backup2:/mnt/data4/build/bb/bb-freebsd_linuxcompat.tgz .
tar xzf bb-freebsd_linuxcompat.tgz

edit /home/bb/bbc1.9e-btf/etc/bb-hosts with something like:
echo "10.1.4.5 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh \
10.1.4.102 jail2.johncompanies.com # ssh" > /home/bb/bbc1.9e-btf/etc/bb-hosts

vi /home/bb/bbc1.9e-btf/ext/openfiles
MACHINE="jail2,johncompanies,com" # HAS TO BE IN A,B,C FORM

cd /usr/home/bb/bbc1.9e-btf/etc
./bbchkcfg.sh
(y to questions)
./bbchkhosts.sh
(ignore ssh errors)
cd ../..
chown -R bb .
su bb
cd
cd bbc1.9e-btf
./runbb.sh start
more BBOUT
(look for errors)
exit

echo 'su - bb -c "cd /home/bb/bbc1.9e-btf; ./runbb.sh start"' > /usr/local/etc/rc.d/bb.sh
chmod +x /usr/local/etc/rc.d/bb.sh

NOTE: to get bb working on amd, had to copy over bin dir from linux dist

27. configure load mrtg, on mail
vi /usr/local/www/mgmt/mrtg/mrtg1.cfg
(add new entry to file following existing format)

28. configure bb on mail:
vi /usr/home/bb/bbsrc/bb1.9e-btf/etc/bb-hosts
10.1.4.102 jail2.johncompanies.com # ssh

su bb
cd
bbsrc/bb/runbb.sh restart ; exit

29. remove reserve space, enable softupdates (probably already set, so not necessary)
NOT APPLICABLE IF USING GVINUM
cd
umount /mnt/data1
umount /mnt/data2
tunefs -m 0 /mnt/data1
tunefs -m 0 /mnt/data2
mount -a

30. configure ntp
echo "server 10.1.4.5" > /etc/ntp.conf

/usr/sbin/ntpd -p /var/run/ntpd.pid
sleep 2; ntpq -p
(confirm it’s able to reach our time server)

32. fwd and reverse lookups on ns1c
vr johncompanies.com
vi internal.johncompanies.com
rndc reload johncompanies.com IN private
(edit the PTR too)

33. create gvinum volumes

Make a g partition:

2950:
bsdlabel -e /dev/mfid0s1
bsdlabel -e /dev/mfid1s1
bsdlabel -e /dev/mfid2s1

given:
# /dev/aacd0s1:
8 partitions:
# size offset fstype [fsize bsize bps/cpg]
a: 262144 0 4.2BSD 2048 16384 16392
b: 4194304 262144 swap
c: 143363997 0 unused 0 0 # "raw" part, don't edit
d: 524288 4456448 4.2BSD 2048 16384 32776
e: 524288 4980736 4.2BSD 2048 16384 32776
f: 6291456 5505024 4.2BSD 2048 16384 28552

new offset = 6291456 + 5505024 = 11796480
new size is size for 'c' partition minus the new start from above
143363997 - 11796480 = 131567517
So:
g: 131567517 11796480 unused 0 0

bsdlabel -e /dev/amrd1s1
change d to g

For a 146G drive (after OS and 4G swap), we can fit 127 1G volumes so:

echo 'drive data1 device /dev/mfid0s1g' > /tmp/cgv

echo '#\!/bin/sh\
i="1"\
while [ $i -le 127 ]\
do\
echo "volume v$i" >> /tmp/cgv;\
echo 'plex org concat' >> /tmp/cgv;\
echo 'sd length 1g drive data1' >> /tmp/cgv;\
i=`expr $i + 1`\
done' > /tmp/mkcgv
sh /tmp/mkcgv
gvinum create /tmp/cgv

volume 1
plex org concat
sd length 3g drive data1
volume 2
plex org concat
sd length 5g drive data1
volume 3
plex org concat
sd length 8g drive data1
volume 4
plex org concat
sd length 6g drive data1
volume 5
plex org concat
sd length 10g drive data1

for f in 1 2 3 4 5; do bsdlabel -rw /dev/gvinum/$f; done
for f in 1 2 3 4 5; do newfs /dev/gvinum/${f}a; done
for f in 1 2 3 4 5; do mkdir /mnt/data1/$f; mount /dev/gvinum/${f}a /mnt/data1/$f;done
for f in 1 2 3 4 5; do dd if=/dev/zero of=/mnt/data1/$f/this_is_$f bs=1m count=$f; done
for f in 1 2 3 4 5; do ll -h /mnt/data1/$f/; done

For a 300G drive (after OS and 4G swap), we can fit 273 1G volumes so:

echo 'drive data2 device /dev/mfid1s1g' > /tmp/cgv

echo '#\!/bin/sh\
i="128"\
while [ $i -le 401 ]\
do\
echo "volume v$i" >> /tmp/cgv;\
echo 'plex org concat' >> /tmp/cgv;\
echo 'sd length 1g drive data2' >> /tmp/cgv;\
i=`expr $i + 1`\
done' > /tmp/mkcgv
sh /tmp/mkcgv
gvinum create /tmp/cgv

For a 300G drive (after OS and 4G swap), we can fit 273 1G volumes so:

echo 'drive data3 device /dev/mfid2s1g' > /tmp/cgv

echo '#\!/bin/sh\
i="402"\
while [ $i -le 675 ]\
do\
echo "volume v$i" >> /tmp/cgv;\
echo 'plex org concat' >> /tmp/cgv;\
echo 'sd length 1g drive data3' >> /tmp/cgv;\
i=`expr $i + 1`\
done' > /tmp/mkcgv
sh /tmp/mkcgv
gvinum create /tmp/cgv

Sometimes there’s 2040m leftover so create one more smaller vol:
echo 'volume v64\
plex org concat\
sd length 2040m drive data1' >> /tmp/cgv

To delete:

echo '#\!/bin/sh\
i="1"\
while [ $i -le 127 ]\
do\
echo "gvinum rm -r v$i" >> /tmp/dgv;\
i=`expr $i + 1`\
done' > /tmp/mkdgv
sh /tmp/mkdgv
sh /tmp/dgv

dd if=/dev/zero of=/dev/sdb seek=285474816 bs=512
dd if=/dev/zero of=/dev/mfid0s2 bs=1m count=1000 (do this to all drives)

35. create the jail template

touch /mnt/data1/jail
mdconfig -a -t vnode -s 1g -f /mnt/data1/jail -u 0
bsdlabel -rw md0
newfs4x /dev/md0c
mkdir /mnt/data1/jail-DIR
mount /dev/md0c /mnt/data1/jail-DIR

cd /usr/ports/sysutils/jailutils
make install clean

cd /usr/src
make world DESTDIR=/mnt/data1/jail-DIR
(2450: 2:28mins, supermicro: 55mins, 2950: 1h )
cd etc
make distribution DESTDIR=/mnt/data1/jail-DIR
mount -t devfs devfs /mnt/data1/jail-DIR/dev
devfs -m /mnt/data1/jail-DIR/dev rule -s 3 applyset
cd /mnt/data1/jail-DIR
ln -sf dev/null kernel
cp /usr/local/sbin/jkill /mnt/data1/jail-DIR/sbin

jail /mnt/data1/jail-DIR testhostname 192.168.11.100 /bin/sh
csh
touch /etc/fstab
echo 'network_interfaces=""\
hostname="newsystem"\
kern_securelevel_enable="NO"\
sendmail_enable="YES"\
sshd_enable="YES"' > /etc/rc.conf

echo "nameserver 69.55.225.225\
nameserver 69.55.230.3" >> /etc/resolv.conf

vi /etc/crontab
remove the adjkerntz lines
comment out periodic’s and put this line above them:
# IF YOU UNCOMMENT THESE, PLEASE ADJUST THEIR RUN TIME

rm -rf /etc/periodic/daily/400.status-disks

check /tmp for crap

vi /etc/periodic/security/100.chksetuid
replace: MP=`mount -t ufs | grep -v " nosuid" | awk '{ print $3 }' | sort`
with: MP='/' (use single quotes)

mkdir -p /usr/compat/linux/dev

adduser (Add account for user, make sure in wheel group)

user/root passwd: 8ico2987

Set root password

vi /usr/home/user/.profile (and add to the file):
TERM=vt100; export TERM

tzsetup

newaliases

rm /sbin/halt /sbin/reboot
ln /sbin/jkill /sbin/halt
ln /sbin/jkill /sbin/reboot

#cd /dev
#rm console
#ln -s null console

vi /etc/syslog.conf (comment out console and move to /var/log/messages):
#*.err;kern.warning;auth.notice;mail.crit /dev/console *.err;kern.warning;auth.notice;mail.crit /var/log/messages

exit
exit

cd libexec
mv ld-elf32.so.1 ld-elf32.so.1-orig
ln ld-elf.so.1 ld-elf32.so.1

may have to run:
chflags noschg ld-elf32.so.1
chflags noschg ld-elf.so.1
then
chflags schg ld-elf.so.1
chflags schg ld-elf32.so.1

cd /usr/ports
make -DNOCLEANDEPENDS clean
(2450: 15mins , supermicro: 29mins, 2950: 18mins)
rm -fr /usr/ports/distfiles/*
cp -r /usr/ports /mnt/data1/jail-DIR/usr (2450: 2:00 mins , supermicro: 15mins, 2950: 3mins)

rm /mnt/data1/jail-DIR/root/.history

cd
dump -0a -f /usr/local/jail/template/71template /dev/md0c
umount /mnt/data1/jail-DIR/dev
umount /dev/md0c
rmdir /mnt/data1/jail-DIR

36. setup backups
echo '#\!/bin/sh\
backupdir=/data/jail2\
server=backup1\
\
## ENTRY /etc\
## ENTRY /usr/local/etc\
## ENTRY /usr/local/jail\
## ENTRY /root/logs' > /usr/local/jail/bin/backup.config

on backup1:
setup backup dirs:
ssh backup1 mkdir -p /data/jail2/0

add the system to
vi /usr/local/sbin/snapshot_rotate

on mail:
vi /usr/local/www/mgmt/cgi/backupgraph.pl
(add hostname)

Edit /usr/local/jail/bin/backup.xxx to use the right drives and copy to /usr/local/jail/bin/backup

37. mkdir /root/logs

38. edit sshd_config for security
vi /etc/ssh/sshd_config
ListenAddress 69.55.228.53
ListenAddress 10.1.4.102

kill -1 `cat /var/run/sshd.pid`

39. add crontab entries
crontab -e
5 0 * * * /usr/local/jail/bin/backup.md
1 0 1 * * /usr/local/jail/bin/ipfwreset
0 18 * * * /usr/local/jail/bin/ipfwbackup
4,9,14,19,24,29,34,39,44,49,55,59 * * * * /usr/local/jail/bin/trafstats

On 2950 with Perc5/i also add:
*/5 * * * * /usr/local/jail/bin/perc5iraidchk

40. Reboot notify script
ln -s /usr/local/jail/bin/notify.sh /usr/local/etc/rc.d/notify.sh

41. add to management db (on mail and devweb) jc.ref_machines and jc.ref_templates

uname -r
6.2-RELEASE-jc1

insert into ref_machines values (null,'mx2','mx2.johncompanies.com',0,'m');
select machine_id from ref_machines where host='mx2';
+------------+
| machine_id |
+------------+
| 35 |
+------------+
insert into ref_templates values ('',' 7.1-RELEASE-jc2',44,'FreeBSD 7.1',0);

42. add to server/cabinet map. On mail:
vi /usr/local/www/mgmt/html/cabinetmap.html

43. add an outside blocking rule to the firewall, so this machine can only be reached from inside the firewall. Follow example already in firewall jail17 is:

00117 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 or 69.55.238.150 } to 69.55.228.200
00117 deny ip from any to 69.55.228.200

jail19 would be 00119...
ipfw add 00107 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 or 69.55.238.150 } to 69.55.230.108
ipfw add 00107 deny ip from any to 69.55.220.108

44. select customers for probe map

46. make gv start on boot

scp backup2:/mnt/data4/build/freebsd/gvinum /etc/rc.d/gvinum

If they start stale:

echo '#\!/bin/sh\
i="1"\
while [ $i -le 127 ]\
do\
gvinum start v$i\
i=`expr $i + 1`\
done' > /tmp/startgv

sh /tmp/startgv

47. patch jail against starting jails with rtprio

mv /usr/sbin/jail /usr/sbin/jail_
echo '#\!/bin/sh\
/usr/sbin/rtprio -t /usr/sbin/jail_ $*' > /usr/sbin/jail
chmod +x /usr/sbin/jail

48. make sure mail works
If there are map errors:
cd /etc/mail; make maps

49. move and pare down generic kernel
mv kernel.old/ kernel.GENERIC

/ too full, remove old kernel modules except for:

Id Refs Address Size Name
1 14 0xffffffff80100000 ac6c08 kernel
2 1 0xffffffff80bc8000 1128 mfi_linux.ko
3 4 0xffffffff80bca000 39138 linux.ko
4 1 0xffffffffb48f8000 358c linprocfs.ko
5 1 0xffffffffb48fc000 9d3 linsysfs.ko
6 1 0xffffffffb490c000 80ee8 zfs.ko

50. ntp doesn’t seem to start from rc so,

echo '/usr/sbin/ntpd -c /etc/ntp.conf -p /var/run/ntpd.pid -f /var/db/ntpd.drift' > /usr/local/etc/rc.d/ntp.sh
chmod 0700 /usr/local/etc/rc.d/ntp.sh</pre>

= FreeBSD 7.2 =

Last updated 2009-06-18

<pre>All time extimates below assume disks aren’t scrubbing. Setup instructions below are for LSI card:

1. make sure bios is setup for bios console redirect
2950:
Console redirection:
LCD string..
Date to GMT

2. assuming mirrors (or at least disks) created (if not, refer to this), boot to disk 1 of 7.2
skip kernel config (enter)
custom install

partition ->
move cursor to mfid0, hit space (takes you to partition map screen)
a for entire disk
q to quit and save

standard mbr (no boot manager)
space to unselect mfid0
cursor over mfid1
space
a for entire disk
q to quit and save
none (leave untouched)

cursor over mfid0
space
(takes you into part. Screen again) q to exit
none
Make sure both are checked and tab to ok

Label ->
Make sure mfid0 is highlighted

/ 256M
swap 2G (for 2950 make it 4G)
/var 256M (4.6G)
/tmp 256M
/usr 4G
/mnt/data1 remaining space (no need to newfs)

Make sure to toggle S for soft updates on all (should look like UFS2+S Y under the Newfs column)

move cursor to mfid1
swap 8G (or 4G if there’s a 3rd drive)
/mnt/data2 remaining space (no need to newfs)

q to save and exit

distributions ->
developer
yes to install ports
exit

media ->
cd

commit ->
yes
(2450: 14mins, supermicro: 12mins)

yes to "visit general config" ->
Set root pwd

Add user ‘user’ member group is wheel, set password

Set tz

Networking->interfaces->Fxp0 ->
No IPV6
dhcp=no
Set hostname & domain
Enable sshd

exit...
exit install ->
yes

take the cd out and let the machine reboot

3. double check the date/time

4. edit /etc/make.conf
echo "WITHOUT_X11=yes \
KERNCONF=jail8 \
BOOT_COMCONSOLE_SPEED=115200" >> /etc/make.conf

5. add settings to /boot/loader.conf and /boot.config

echo "-Dh" >> /boot.config

echo 'console="comconsole,vidconsole" \
boot_multicons="YES" \
boot_serial="YES" \
mfi_linux_load="YES" \
comconsole_speed="115200"' >> /boot/loader.conf

6. turn off all ttyv's except 0 and 1 in /etc/ttys
also turn on ttyd0, change type to vt100:
vi /etc/ttys

ttyv2 "/usr/libexec/getty Pc" cons25 off secure
ttyv3 "/usr/libexec/getty Pc" cons25 off secure
ttyv4 "/usr/libexec/getty Pc" cons25 off secure
ttyv5 "/usr/libexec/getty Pc" cons25 off secure
ttyv6 "/usr/libexec/getty Pc" cons25 off secure
ttyv7 "/usr/libexec/getty Pc" cons25 off secure

ttyd0 "/usr/libexec/getty std.9600" vt100 on secure

kill -1 1

on console server:
vi /etc/remote
(rename port to jail18 depending on where and which digi plugged into)
test serial console

7. populate hosts
echo "10.1.4.3 backup2" >> /etc/hosts
echo "10.1.4.8 backup1" >> /etc/hosts

8. put key in authorized_keys on backup2
cd
ssh-keygen -t dsa -b 1024
(default location, leave password blank)

cat /root/.ssh/id_dsa.pub | ssh backup2 'cat - >> /root/.ssh/authorized_keys'
cat /root/.ssh/id_dsa.pub | ssh backup1 'cat - >> /root/.ssh/authorized_keys'

confirm that you can ssh to backup2 and backup1 without getting a login prompt

ssh backup2 hostname

ssh backup1 hostname

9. create & populate binaries/scripts dirs
mkdir -p /usr/local/jail/bin
mkdir -p /usr/local/jail/rc.d
mkdir -p /usr/local/jail/template/
mkdir /mnt/data1
mkdir /mnt/data2
scp backup2:"/mnt/data4/bin/freebsd7.x/*" /usr/local/jail/bin
cd /usr/local/jail/rc.d/
touch quad1
touch deprecated
chmod +x *
cd /usr/local/jail/bin
ln -s /usr/local/jail/rc.d/quad1 quad1
ln -s /usr/local/jail/bin/jailmake_md jailmake
ln -s /usr/local/jail/bin/js_md js
ln -s /usr/local/jail/bin/canceljail_md canceljail
ln -s /usr/local/jail/bin/jailmakeempty_md jailmakeempty
ln -s /usr/local/jail/bin/postboot_md postboot

rehash

10. edit root's path and login script:
vi /root/.cshrc

Change alias entries (add G):
alias la ls -aG
alias lf ls -FAG
alias ll ls -lAG
alias ls ls -AG
alias mbm mb mount
alias mbu mb umount

add to path:
/usr/local/jail/bin
(if adaptec card installed, also add /compat/linux/usr/sbin)

and alter the prompt, set the following:
set prompt = "`/bin/hostname -s` %/# "

at the bottom of the file add:
set sshtty=`who am i|awk '{print $2}'`
/usr/sbin/rtprio 3 -`psj | grep $sshtty | awk '{print $2}'`

set shortty=`who am i | awk '{print $2}' | sed -E 's/.*(..)$/\1/'`
foreach x (`psj | grep sh | grep $shortty | awk '{print $2}'`)
/usr/sbin/rtprio 2 -$x
end

To load the new file:
source /root/.cshrc

11. install cvsup
cd /usr/ports/net/cvsup-without-gui
make install clean; rehash; mail -s 'cvs installed' dave.boodman@vtext.com < /dev/null

(stay close for gettext options, 2450: 27mins, supermicro: 17mins, 2950: 22mins)

12. get latest sources for this release:
cd /usr/src
echo "*default host=cvsup4.freebsd.org\
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_7_2\
*default delete use-rel-suffix\
*default compress\
src-all" > sup

-OR-

echo "*default host=cvsup4.freebsd.org\
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_7\
*default delete use-rel-suffix\
*default compress\
src-all" > sup

(stable)

cvsup sup ; mail -s 'cvs sup done' dave.boodman@vtext.com < /dev/null

(2450, ~12mins, supermicro, 27mins, 2950: 7mins)

13. configure new kernel.

cd /usr/src/sys/amd64/conf
scp backup2:/mnt/data4/build/freebsd/kern_config-7.1-amd64 ./jail8

edit the kernel config and change ident to be the name of the jail:
vi jail8
ident jail8

edit /sys/conf/newvers.sh to add –jc2 to the end of the BRANCH string (RELEASE-jc2)
vi /sys/conf/newvers.sh

notes: http://www.FreeBSD.org/doc/en_US.ISO8859-1/books/handbook/kernelconfig-config.html

14. bring over patches from backup2

The various patches are in /mnt/data4/build/freebsd/patches on backup2. There are dirs for each version. Not all dirs are populated, but patches for later versions work on older ones unless there is a new patch in the older dir.

cd /usr/src
scp backup2:"/mnt/data4/build/freebsd/patches/7.2/*" .

Apply patches:
patch -l < jls-patch

15. build, install kernel and world

cd /boot

mv kernel kernel.GENERIC
cd kernel.GENERIC
mkdir hold
mv mfi_linux.ko hold/
mv linux.ko hold/
mv linprocfs.ko hold/
mv linsysfs.ko hold/
mv geom_vinum.ko hold/
mv geom_concat.ko hold/
rm *.ko
rm *.symbols
mv hold/* .
rmdir hold/

cd /usr/src
make buildkernel installkernel

make buildworld ; mail -s 'buildworld done' dave.boodman@vtext.com < /dev/null
(2450: 1:56min, supermicro: 59mins, 2950: 38mins)
make installworld
(2450: 3min, supermicro: 1min, 2950: :34)
mergemaster -i
delete /var/tmp/temproot
delete bsnmpd
delete temporary ./etc/hosts
delete temporary ./etc/motd
delete /var/tmp/temproot

cd /sys/modules/geom/geom_vinum
make
make install

16. populate devfs ruleset
scp backup2:/mnt/data4/build/freebsd/devfs.rules /etc

17. populate /etc/rc.conf with IPs and NFS settings
vi /etc/rc.conf

kern_securelevel_enable="NO"
portmap_enable="NO"
sendmail_enable="NO"
usbd_enable="YES"

xntpd_enable="YES"
nfs_client_enable="YES"
nfs_reserved_port_only="YES"
inetd_enable="YES"
inetd_flags="-wW -a 10.1.4.102"
devfs_system_ruleset="devfsrules_show_all"

#ifconfig_xl0="inet 10.1.4.102 netmask 255.255.255.0"
#ifconfig_fxp0="inet 69.55.228.53 netmask 255.255.255.0"
#ifconfig_fxp0_alias0="inet 69.55.2xx.xx netmask 255.255.255.0"

static_routes="t1 office"
route_t1="-net 10.1.5 10.1.4.2"
route_office="-net 10.1.6 10.1.4.2"
gvinum_enable="YES"
fsck_y_enable="YES"
background_fsck="NO"
rc_mfi_raid_tty_log="YES"

18. make sure sysctls are set and preserved after reboot
echo "kern.consmute=0\
kern.ipc.shm_use_phys=1\
kern.ipc.shmall=131070\
kern.ipc.shmmax=134217728\
net.inet.tcp.syncookies=0\
kern.maxfiles=32768\
kern.fallback_elf_brand=3\
kern.maxprocperuid=4000\
security.jail.sysvipc_allowed=1\
security.jail.allow_raw_sockets=1\
security.jail.socket_unixiproute_only=1\
security.jail.chflags_allowed=0\
dev.amr.0.allow_volume_configure=1\
compat.linux.osrelease=2.6.12\
vm.pmap.shpgperproc=300" >> /etc/sysctl.conf

19. mount procfs
echo "proc /proc procfs rw 0 0" >> /etc/fstab

For Dell 2950/2450:
echo "linprocfs /usr/compat/linux/proc linprocfs rw 0 0" >> /etc/fstab

For Dell 2950:
echo "linsysfs /usr/compat/linux/sys linsysfs rw 0 0" >> /etc/fstab

mkdir -p /usr/compat/linux/proc
mkdir -p /usr/compat/linux/sys

19. enable noatime option
NOT APPLICABLE IF RUNNING GVINUM
data1 and data2 should look something like:
/dev/amrd0s1g /mnt/data1 ufs rw,noatime 2 2

20. reboot. Confirm new kernel is loaded

uname -a

Check devfs rules:
devfs rule showsets
devfs rule -s 3 show

21. update ports:
cd /usr/ports
echo "*default host=cvsup4.FreeBSD.org\
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_7_2\
*default delete use-rel-suffix\
*default compress\
ports-all tag=." > sup

cvsup sup; mail -s 'cvs sup ports done' dave.boodman@vtext.com < /dev/null

(2450: 18mins, supermicro: 19mins; 2950: 24mins)

22. Install raid mgmt tool

For LSI based cards:

install linux_base:
cd /usr/ports/emulators/linux_base-fc4
make install clean
(2450: 7min, supermicro: 3mins, 2950: 14mins)

(for LSI)
cd /usr/ports/sysutils/linux-megamgr
make install clean

cd /usr/ports/sysutils/megarc
make install clean

(for Perc5/i, 6/i)
cd /usr/ports/sysutils/linux-megacli2
make install clean

Test:
rehash; megacli ldinfo lall a0

For adaptec:
On jail18:
scp /compat/linux/usr/sbin/aaccli user@10.1.4.102:~

mv ~user/aaccli /compat/linux/usr/sbin/aaccli

test out;
/compat/linux/usr/sbin/aaccli

For 3w9690:
scp backup2:/d4/build/3ware/tw_cli-freebsd-x86_64-9.5.0.1.tgz /usr/local/sbin
cd /usr/local/sbin xzvf tw_cli-freebsd-x86_64-9.5.0.1.tgz
rm tw_cli-freebsd-x86_64-9.5.0.1.tgz

test out;
./tw_cli /c0 show allstatus

23. install rsync from ports
cd /usr/ports/net/rsync
make install clean

choose default options

25. configure inetd to respond to mrtg load queries
echo "load stream tcp nowait user /usr/local/jail/bin/load.pl load.pl" >> /etc/inetd.conf

echo "load 12384/tcp" >> /etc/services

26. install bb client
adduser
Username: bb
Full name: bb
Uid (Leave empty for default): 1984
Login group [bb]:
Login group is bb. Invite bb into other groups? []:
Login class [default]:
Shell (sh csh tcsh nologin) [sh]:
Home directory [/home/bb]:
Use password-based authentication? [yes]:
Use an empty password? (yes/no) [no]:
Use a random password? (yes/no) [no]: yes
Lock out the account after creation? [no]:
Username : bb
Password : <random>
Full Name : bb
Uid : 1984
Class :
Groups : bb
Home : /home/bb
Shell : /bin/sh
Locked : no
OK? (yes/no): yes

cd /usr/home/bb
scp backup2:/mnt/data4/build/bb/bb-freebsd_linuxcompat.tgz .
tar xzf bb-freebsd_linuxcompat.tgz

edit /home/bb/bbc1.9e-btf/etc/bb-hosts with something like:
echo "10.1.4.5 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh \
10.1.4.108 jail8.johncompanies.com # ssh" > /home/bb/bbc1.9e-btf/etc/bb-hosts

vi /home/bb/bbc1.9e-btf/ext/openfiles
MACHINE="jail8,johncompanies,com" # HAS TO BE IN A,B,C FORM

cd /usr/home/bb/bbc1.9e-btf/etc
./bbchkcfg.sh
(y to questions)
./bbchkhosts.sh
(ignore ssh errors)
cd ../..
chown -R bb .
su bb
cd
cd bbc1.9e-btf
./runbb.sh start
more BBOUT
(look for errors)
exit

echo 'su - bb -c "cd /home/bb/bbc1.9e-btf; ./runbb.sh start"' > /usr/local/etc/rc.d/bb.sh
chmod +x /usr/local/etc/rc.d/bb.sh

NOTE: to get bb working on amd, had to copy over bin dir from linux dist

27. configure load mrtg, on mail
vi /usr/local/www/mgmt/mrtg/mrtg1.cfg
(add new entry to file following existing format)

28. configure bb on mail:
vi /usr/home/bb/bbsrc/bb1.9e-btf/etc/bb-hosts
10.1.4.102 jail2.johncompanies.com # ssh

su bb
cd
bbsrc/bb/runbb.sh restart ; exit

29. remove reserve space, enable softupdates (probably already set, so not necessary)
NOT APPLICABLE IF USING GVINUM
cd
umount /mnt/data1
umount /mnt/data2
tunefs -m 0 /mnt/data1
tunefs -m 0 /mnt/data2
mount -a

30. configure ntp
echo "server 10.1.4.5" > /etc/ntp.conf

/usr/sbin/ntpd -p /var/run/ntpd.pid
sleep 2; ntpq -p
(confirm it’s able to reach our time server)

32. fwd and reverse lookups on ns1c
vr johncompanies.com
vi internal.johncompanies.com
rndc reload johncompanies.com IN private
(edit the PTR too)

35. create the jail template

touch /mnt/data1/jail
mdconfig -a -t vnode -s 1g -f /mnt/data1/jail -u 0
bsdlabel -rw md0
newfs4x -i 4096 /dev/md0c
mkdir /mnt/data1/jail-DIR
mount /dev/md0c /mnt/data1/jail-DIR

cd /usr/ports/sysutils/jailutils
make install clean

cd /usr/src
make world DESTDIR=/mnt/data1/jail-DIR; pagedave
(2450: 2:28mins, supermicro: 55mins, 2950: 1h )
cd etc
make distribution DESTDIR=/mnt/data1/jail-DIR
mount -t devfs devfs /mnt/data1/jail-DIR/dev
devfs -m /mnt/data1/jail-DIR/dev rule -s 3 applyset
cd /mnt/data1/jail-DIR
ln -sf dev/null kernel
cp /usr/local/sbin/jkill /mnt/data1/jail-DIR/sbin

jail /mnt/data1/jail-DIR testhostname 192.168.11.100 /bin/sh
csh
touch /etc/fstab
echo 'network_interfaces=""\
hostname="newsystem"\
kern_securelevel_enable="NO"\
sendmail_enable="YES"\
sshd_enable="YES"' > /etc/rc.conf

echo "nameserver 69.55.225.225\
nameserver 69.55.230.3" >> /etc/resolv.conf

vi /etc/crontab
remove the adjkerntz lines
comment out periodic’s and put this line above them:
# IF YOU UNCOMMENT THESE, PLEASE ADJUST THEIR RUN TIME

rm -rf /etc/periodic/daily/400.status-disks

check /tmp for crap

vi /etc/periodic/security/100.chksetuid
replace: MP=`mount -t ufs | grep -v " nosuid" | awk '{ print $3 }' | sort`
with: MP='/' (use single quotes)

mkdir -p /usr/compat/linux/dev

adduser (Add account for user, make sure in wheel group)

Username: user
Full name: user
Uid (Leave empty for default):
Login group [user]:
Login group is user. Invite user into other groups? []: wheel
Login class [default]:
Shell (sh csh tcsh nologin) [sh]:
Home directory [/home/user]:
Home directory permissions (Leave empty for default):
Use password-based authentication? [yes]:
Use an empty password? (yes/no) [no]:
Use a random password? (yes/no) [no]: y
Lock out the account after creation? [no]:
Username : user
Password : <random>
Full Name : user
Uid : 1001
Class :
Groups : user
Home : /home/user
Home Mode :
Shell : /bin/sh
Locked : no
OK? (yes/no): y
adduser: INFO: Successfully added (user) to the user database.
adduser: INFO: Password for (user) is: 901gmYjO
Add another user? (yes/no): n
Goodbye!

vi /usr/home/user/.profile (and add to the file):
TERM=vt100; export TERM

tzsetup

newaliases

rm /sbin/halt /sbin/reboot
ln /sbin/jkill /sbin/halt
ln /sbin/jkill /sbin/reboot

#cd /dev
#rm console
#ln -s null console

vi /etc/syslog.conf (comment out console and move to /var/log/messages):
#*.err;kern.warning;auth.notice;mail.crit /dev/console *.err;kern.warning;auth.notice;mail.crit /var/log/messages

exit
exit

cd libexec
chflags noschg ld-elf32.so.1
chflags noschg ld-elf.so.1

mv ld-elf32.so.1 ld-elf32.so.1-orig
ln ld-elf.so.1 ld-elf32.so.1

chflags schg ld-elf.so.1
chflags schg ld-elf32.so.1

cd /usr/ports
make -DNOCLEANDEPENDS clean
(2450: 15mins , supermicro: 29mins, 2950: 18mins)
rm -fr /usr/ports/distfiles/*
cp -r /usr/ports /mnt/data1/jail-DIR/usr (2450: 2:00 mins , supermicro: 15mins, 2950: 3mins)

rm /mnt/data1/jail-DIR/root/.history

cd
umount /mnt/data1/jail-DIR/dev
dump -0a -f /usr/local/jail/template/72template /dev/md0c
umount /dev/md0c
rmdir /mnt/data1/jail-DIR

36. setup backups
echo '#\!/bin/sh\
backupdir=/data/jail2\
server=backup1\
\
## ENTRY /etc\
## ENTRY /usr/local/etc\
## ENTRY /usr/local/jail\
## ENTRY /root/logs' > /usr/local/jail/bin/backup.config

on backup1:
setup backup dirs:
ssh backup1 mkdir -p /data/jail2/0

add the system to
vi /usr/local/sbin/snapshot_rotate

on mail:
vi /usr/local/www/mgmt/cgi/backupgraph.pl
(add hostname)

Edit /usr/local/jail/bin/backup.xxx to use the right drives and copy to /usr/local/jail/bin/backup

37. mkdir /root/logs

38. edit sshd_config for security
vi /etc/ssh/sshd_config
ListenAddress 69.55.228.53
ListenAddress 10.1.4.102

kill -1 `cat /var/run/sshd.pid`

39. add crontab entries
crontab -e
5 0 * * * /usr/local/jail/bin/backup.md
1 0 1 * * /usr/local/jail/bin/ipfwreset
0 18 * * * /usr/local/jail/bin/ipfwbackup
4,9,14,19,24,29,34,39,44,49,55,59 * * * * /usr/local/jail/bin/trafstats

On 2950 with Perc5/i also add:
*/5 * * * * /usr/local/jail/bin/perc5iraidchk

40. Reboot notify script
ln -s /usr/local/jail/bin/notify.sh /usr/local/etc/rc.d/notify.sh

41. add to management db (on mail and devweb) jc.ref_machines and jc.ref_templates

uname -r
6.2-RELEASE-jc1

insert into ref_machines values (null,'mx2','mx2.johncompanies.com',0,'m');
select machine_id from ref_machines where host='mx2';
+------------+
| machine_id |
+------------+
| 35 |
+------------+
insert into ref_templates values ('',' 7.1-RELEASE-jc2',44,'FreeBSD 7.1',0);

42. add to server/cabinet map. On mail:
vi /usr/local/www/mgmt/html/cabinetmap.html

43. add an outside blocking rule to the firewall, so this machine can only be reached from inside the firewall. Follow example already in firewall jail17 is:

00117 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 or 69.55.238.150 } to 69.55.228.200
00117 deny ip from any to 69.55.228.200

jail19 would be 00119...
ipfw add 00107 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 or 69.55.238.150 } to 69.55.230.108
ipfw add 00107 deny ip from any to 69.55.220.108

44. select customers for probe map

47. patch jail against starting jails with rtprio

mv /usr/sbin/jail /usr/sbin/jail_
echo '#\!/bin/sh\
/usr/sbin/rtprio -t /usr/sbin/jail_ $*' > /usr/sbin/jail
chmod +x /usr/sbin/jail

48. make sure mail works
If there are map errors:
cd /etc/mail; make maps

50. ntp doesn’t seem to start from rc so,

echo '/usr/sbin/ntpd -c /etc/ntp.conf -p /var/run/ntpd.pid -f /var/db/ntpd.drift' > /usr/local/etc/rc.d/ntp.sh
chmod 0700 /usr/local/etc/rc.d/ntp.sh

51. recover space on /usr

rm -fr /usr/obj

52. setup fuse

cd /usr/ports/sysutils/fusefs-kmod/
make install

vi /etc/rc.conf
fusefs_enable="YES"

sysctl vfs.usermount=1

cd /usr/ports/sysutils/fusefs-sshfs
make install

sshfs 1005@usw-s009.rsync.net: /mnt/data1/69.55.234.68-col00001-DIR/mnt</pre>

= Stress testing a 6.x jail =

<pre>mkdir /mnt/jail
newfs /dev/gvinum/v1
mount /dev/gvinum/v1 /mnt/jail
cd /mnt/jail
restore -r -f /path/to/61stress .
cd ..
umount /mnt/jail

sh
for f in 1 2 3 4 5 6 7 8 9 10; do mkdir /mnt/data1/$f; done

gconcat label v3-v5 /dev/gvinum/v3 /dev/gvinum/v4 /dev/gvinum/v5
bsdlabel -r -w /dev/concat/v3-v5
newfs /dev/concat/v3-v5a
mount /dev/concat/v3-v5a /mnt/data1/1
bsdlabel -r -w /dev/gvinum/v10
newfs /dev/gvinum/v10a
mount /dev/gvinum/v10a /mnt/data1/2
gconcat label v23-v25 /dev/gvinum/v23 /dev/gvinum/v24 /dev/gvinum/v25
bsdlabel -r -w /dev/concat/v23-v25
newfs /dev/concat/v23-v25a
mount /dev/concat/v23-v25a /mnt/data1/3
bsdlabel -r -w /dev/gvinum/v50
newfs /dev/gvinum/v50a
mount /dev/gvinum/v50a /mnt/data1/4
gconcat label v63-v65 /dev/gvinum/v63 /dev/gvinum/v64 /dev/gvinum/v65
bsdlabel -r -w /dev/concat/v63-v65
newfs /dev/concat/v63-v65a
mount /dev/concat/v63-v65a /mnt/data1/5
bsdlabel -r -w /dev/gvinum/v70
newfs /dev/gvinum/v70a
mount /dev/gvinum/v70a /mnt/data1/6
gconcat label v83-v85 /dev/gvinum/v83 /dev/gvinum/v84 /dev/gvinum/v85
bsdlabel -r -w /dev/concat/v83-v85
newfs /dev/concat/v83-v85a
mount /dev/concat/v83-v85a /mnt/data1/7
bsdlabel -r -w /dev/gvinum/v100
newfs /dev/gvinum/v100a
mount /dev/gvinum//v100a /mnt/data1/8
gconcat label v123-v125 /dev/gvinum/v123 /dev/gvinum/v124 /dev/gvinum/v125
bsdlabel -r -w /dev/concat/v123-v125
mount /dev/concat/v123-v125a /mnt/data1/9
newfs /dev/concat/v123-v125a
mount /dev/concat/v123-v125a /mnt/data1/9
bsdlabel -r -w /dev/gvinum/v130
newfs /dev/gvinum/v130a
mount /dev/gvinum//v130a /mnt/data1/10

sh
for f in 1 2 3 4 5 6 7 8 9 10; do cd /mnt/data1/$f; dump -0a -f - /dev/gvinum/v1| restore -r -f - ; done

sh
for f in 1 2 3 4 5 6 7 8 9 10; do mount_devfs devfs /mnt/data1/$f/dev; devfs -m /mnt/data1/$f/dev rule -s 3 applyset; done

sh
for f in 15 16 17 18 19 20 21 22 23 24; do ifconfig bce0 alias 10.1.6.$f netmask 255.255.255.255; done

jail /mnt/data1/1 testhostname1 10.1.6.15 /bin/sh /etc/rc 2> /dev/null
jail /mnt/data1/2 testhostname2 10.1.6.16 /bin/sh /etc/rc 2> /dev/null
jail /mnt/data1/3 testhostname3 10.1.6.17 /bin/sh /etc/rc 2> /dev/null
jail /mnt/data1/4 testhostname4 10.1.6.18 /bin/sh /etc/rc 2> /dev/null
jail /mnt/data1/5 testhostname5 10.1.6.19 /bin/sh /etc/rc 2> /dev/null
jail /mnt/data1/6 testhostname6 10.1.6.20 /bin/sh /etc/rc 2> /dev/null
jail /mnt/data1/7 testhostname7 10.1.6.21 /bin/sh /etc/rc 2> /dev/null
jail /mnt/data1/8 testhostname8 10.1.6.22 /bin/sh /etc/rc 2> /dev/null
jail /mnt/data1/9 testhostname9 10.1.6.23 /bin/sh /etc/rc 2> /dev/null
jail /mnt/data1/10 testhostname10 10.1.6.24 /bin/sh /etc/rc 2> /dev/null

systat -vmstat</pre>

= Stress testing zfs =

<pre>zfs receive pool1/jail2 < /usr/local/jail/template/template
zfs set mountpoint=/mnt/data1/jail2 pool1/jail2
zfs set quota=4G pool1/jail2
zfs destroy pool1/jail2@now</pre>
install apps that will run when jail started

<pre>cp -r /usr/src /mnt/data1/jail2/usr
rm -fr /mnt/data1/jail2/usr/obj

mount -t devfs devfs /mnt/data1/jail2/dev; devfs -m /mnt/data1/jail2/dev rule -s 3 applyset;

jail /mnt/data1/jail2 stress-test 69.55.234.86 /bin/sh
csh
cd /usr/ports/benchmarks/bonnie
make install clean
cd /usr/ports/sysutils/stress
make install clean
cd /usr/ports/net/rsync
make install clean
cd /usr/ports/lang/perl5.8
make install clean

cd /usr/local/etc/rc.d/

cat >> boot.sh
sleep 30 && sh /usr/local/etc/rc.d/buildworld &
sleep 30 && sh /usr/local/etc/rc.d/portindex &
sleep 30 && sh /usr/local/etc/rc.d/stress &
sleep 30 && sh /usr/local/etc/rc.d/bonnie &

cat > buildworld
#!/bin/sh
while (true); do cd /usr/src; make buildworld; done

cat > portindex
while (true); do cd /usr/ports/; make index; done

cat > stress
/usr/local/bin/stress -c 1 -i 1 -m 1 -d 1 --vm-bytes 768M --hdd-bytes 128M

cat > bonnie
#!/bin/sh
while (true); do /usr/local/bin/bonnie -s 2g; done

chmod +x *

exit
exit

cd
zfs snapshot pool1/jail2@now

sh
for f in 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105; do zfs send pool1/jail2@now | zfs receive pool1/jail$f;
zfs set quota=10G pool1/jail$f;
zfs set mountpoint=/mnt/data1/jail$f pool1/jail$f;
done

for f in 106 107 108 114 115 116 117 118 119 120 121 122 123 125 126 127 128 129 130 131; do zfs send pool1/jail2@now | zfs receive pool2/jail$f
zfs set quota=10G pool2/jail$f;
zfs set mountpoint=/mnt/data2/jail$f pool2/jail$f;
done

for f in 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101; do chmod 0 /mnt/data1/jail$f/usr/local/etc/rc.d/bonnie; done
for f in 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105; do chmod 0 /mnt/data1/jail$f/usr/local/etc/rc.d/stress; done

for f in 115 116 117 118 119 120 121 122 123 125 126 127; do chmod 0 /mnt/data2/jail$f/usr/local/etc/rc.d/bonnie; done
for f in 119 120 121 122 123 125 126 127 128 129 130 131; do chmod 0 /mnt/data2/jail$f/usr/local/etc/rc.d/stress; done

for f in 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105; do mount -t devfs devfs /mnt/data1/jail$f/dev; devfs -m /mnt/data1/jail$f/dev rule -s 3 applyset; done

for f in 106 107 108 114 115 116 117 118 119 120 121 122 123 125 126 127 128 129 130 131; do mount -t devfs devfs /mnt/data2/jail$f/dev; devfs -m /mnt/data2/jail$f/dev rule -s 3 applyset; done

for f in 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105;
do jail /mnt/data1/jail$f testhostname$f 69.55.234.$f /bin/sh /etc/rc 2> /dev/null; done

for f in 106 107 108 114 115 116 117 118 119 120 121 122 123 125 126 127 128 129 130 131;
do jail /mnt/data2/jail$f testhostname$f 69.55.234.$f /bin/sh /etc/rc 2> /dev/null; done

systat -vmstat
sysctl vfs.numvnodes

for f in 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105; do umount /mnt/data1/jail$f/dev; zfs destroy -r pool1/jail$f; done

for f in 106 107 108 114 115 116 117 118 119 120 121 122 123 125 126 127 128 129 130 131; do umount /mnt/data2/jail$f/dev; zfs destroy -r pool2/jail$f; done

for f in 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105; do touch /mnt/data1/jail$f;
mdconfig -a -t vnode -s 10g -f /mnt/data1/jail$f -u $f;
bsdlabel -r -w md$f auto;
newfs -O 1 /dev/md${f}a;
mkdir /mnt/data1/jail$f-DIR;
mount /dev/md${f}a /mnt/data1/jail$f-DIR;
rsync -aSH /mnt/data2/jail2/ /mnt/data1/jail$f-DIR/;
done

for f in 106 107 108 114 115 116 117 118 119 120 121 122 123 125 126 127 128 129 130 131; do touch /mnt/data2/jail$f;
mdconfig -a -t vnode -s 10g -f /mnt/data2/jail$f -u $f;
bsdlabel -r -w md$f auto;
newfs -O 1 /dev/md${f}a;
mkdir /mnt/data2/jail$f-DIR;
mount /dev/md${f}a /mnt/data2/jail$f-DIR;
rsync -aSH /mnt/data1/jail86-DIR/ /mnt/data2/jail$f-DIR/;
done

for f in 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105;
do chmod 0700 /mnt/data1/jail$f-DIR/usr/local/etc/rc.d/bonnie; done
for f in 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105;
do chmod 0700 /mnt/data1/jail$f-DIR/usr/local/etc/rc.d/stress; done

for f in 106 107 108 114 115 116 117 118 119 120 121 122 123 125 126 127 128 129 130 131;
do chmod 0700 /mnt/data2/jail$f-DIR/usr/local/etc/rc.d/bonnie; done
for f in 106 107 108 114 115 116 117 118 119 120 121 122 123 125 126 127 128 129 130 131;
do chmod 0700 /mnt/data2/jail$f-DIR/usr/local/etc/rc.d/stress; done

for f in 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101; do chmod 0 /mnt/data1/jail$f-DIR/usr/local/etc/rc.d/bonnie; done
for f in 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105; do chmod 0 /mnt/data1/jail$f-DIR/usr/local/etc/rc.d/stress; done

for f in 115 116 117 118 119 120 121 122 123 125 126 127; do chmod 0 /mnt/data2/jail$f-DIR/usr/local/etc/rc.d/bonnie; done
for f in 119 120 121 122 123 125 126 127 128 129 130 131; do chmod 0 /mnt/data2/jail$f-DIR/usr/local/etc/rc.d/stress; done

for f in 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105; do mount -t devfs devfs /mnt/data1/jail$f-DIR/dev; devfs -m /mnt/data1/jail$f-DIR/dev rule -s 3 applyset; done

for f in 106 107 108 114 115 116 117 118 119 120 121 122 123 125 126 127 128 129 130 131; do mount -t devfs devfs /mnt/data2/jail$f-DIR/dev; devfs -m /mnt/data2/jail$f-DIR/dev rule -s 3 applyset; done

for f in 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105;
do jail /mnt/data1/jail$f-DIR testhostname$f 69.55.234.$f /bin/sh /etc/rc 2> /dev/null; done

for f in 106 107 108 114 115 116 117 118 119 120 121 122 123 125 126 127 128 129 130 131;
do jail /mnt/data2/jail$f-DIR testhostname$f 69.55.234.$f /bin/sh /etc/rc 2> /dev/null; done

for f in 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105;
do cp /mnt/data1/boot.sh /mnt/data1/jail$f-DIR/usr/local/etc/rc.d/;
cp /mnt/data1/stress /mnt/data1/jail$f-DIR/usr/local/etc/rc.d/;
cp /mnt/data1/bonnie /mnt/data1/jail$f-DIR/usr/local/etc/rc.d/;
done

for f in 106 107 108 114 115 116 117 118 119 120 121 122 123 125 126 127 128 129 130 131;
do cp /mnt/data1/boot.sh /mnt/data2/jail$f-DIR/usr/local/etc/rc.d/;
cp /mnt/data1/stress /mnt/data2/jail$f-DIR/usr/local/etc/rc.d/;
cp /mnt/data1/bonnie /mnt/data2/jail$f-DIR/usr/local/etc/rc.d/;
done

for f in 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105; do umount /mnt/data1/jail$f-DIR/dev; umount /mnt/data1/jail$f-DIR; mdconfig -d -u $f; done

for f in 106 107 108 114 115 116 117 118 119 120 121 122 123 125 126 127 128 129 130 131; do umount /mnt/data2/jail$f-DIR/dev; umount /mnt/data2/jail$f-DIR; mdconfig -d -u $f; done

for f in 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105; do
mkdir /mnt/data1/jail$f-DIR/;
cd /mnt/data1/jail$f-DIR/;
dump -0a -f - /dev/md0a | restore -r -f -;
done

for f in 106 107 108 114 115 116 117 118 119 120 121 122 123 125 126 127 128 129 130 131; do mkdir /mnt/data2/jail$f-DIR/;
cd /mnt/data2/jail$f-DIR/;
dump -0a -f - /dev/md0a | restore -r -f -;
done

for f in 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105; do chmod 0700 /mnt/data1/jail$f-DIR/usr/local/etc/rc.d/*; done

for f in 106 107 108 114 115 116 117 118 119 120 121 122 123 125 126 127 128 129 130 131; do chmod 0700 /mnt/data2/jail$f-DIR/usr/local/etc/rc.d/*; done

--vm-bytes 300M</pre>

= Reloading FreeBSD (while preserving customer data) =

<pre>2. assuming 2 mirrors, boot to disk 1 of 4.11
skip kernel config (enter)
custom install
(skip partition)
label
move cursor to aacd0, hit space (takes you to partition map screen)
given this as the pre-existing partition map:
Filesystem 1K-blocks Used Avail Capacity Mounted on
/dev/aacd0s1a 128990 36888 81784 31% /
/dev/aacd0s1h 30499106 4 28059174 0% /mnt/data1
/dev/aacd1s1e 69526550 4 63964422 0% /mnt/data2
/dev/aacd0s1f 257998 10 237350 0% /tmp
/dev/aacd0s1g 3096462 852082 1996664 30% /usr
/dev/aacd0s1e 257998 1198 236162 1% /var
procfs 4 4 0 100% /proc

relabel and all mount points according to prev settings

or, delete all partitions except for data parts, re-create as necessary

re-toggle newfs on all mount points except for /mnt/data1 - make sure under the Newfs column the data1 (and data2) partiotions look like: UFS+S Y

make sure under the Newfs column the data1 (and data2) partiotions look like
UFS+S N

space to unselect aacd0
cursor over aacd1
space

Space to select both drives and tab to ok

Continue installation as per normal
distributions
developer
yes to install ports
exit
media
cd
commit
yes
(2450: 16mins, supermicro: 11mins)
yes to "visit general config"
Set root pwd
Add user ‘user’ member group is wheel, set password
Set tz
Networking->interfaces->Fxp0
No IPV6
dhcp=yes
Set hostname & domain

exit...
exit install
yes
take the cd out and let the machine reboot</pre>

Jail Server Install

2013-09-25T17:47:18Z

24.165.90.109: /* populate /etc/rc.conf with IPs and service settings */

= FreeBSD 9.x =

== Assumptions ==

Setup instructions below assume this is DELL 2950 with an LSI-based SAS RAID card.

Server is at castle, connected to pub, private, serial and DRAC

Assuming OS loading done via IPKVM with ISO mounted via USB

Assumes at 4 drives, 2 mirrors

== Configure server BIOS ==

setup console redirect, speed 115200

set LCD string to name of server "jail8"

set date to GMT

go into RAID bios and setup mirrors

== Setup DRAC ==
[[DRAC/RMM#DRAC_setup|DRAC setup]]

== Install OS (sysinstall) ==

boot to bootonly disk for AMD version of FreeBSD, i.e. <tt>FreeBSD-8.3-RELEASE-amd64-bootonly.iso</tt>

when the install menu appears, choose <tt>custom install</tt>

=== partition menu ===
move cursor to mfid0, hit space (takes you to partition map screen). If there is only 1 mirror, there will be no option to select a specifit drive: mfid0 will be selected.

type 'a' to use entire disk<br>
type 'q' to quit and save<br>
choose 'freebsd standard mbr'

space to '''un'''select mfid0<br>
cursor down to mfid1<br>
hit space

type 'a' to use entire disk<br>
type 'q' to quit and save<br>
choose 'none' for boot mgr (leave untouched)

cursor over mfid0<br>
space<br>
(takes you into part. Screen again) 'q' to exit<br>
none for boot mgr<br>

Make sure both drives (mfid0 and mfid1) are checked and tab to ok

=== Label menu ===

Make sure mfid0 is highlighted at the top of the screen, setup the following partitions

/ 512M<br>
swap 6G<br>
/var 1G <br>
/tmp 256M<br>
/usr 8G<br>
/mnt/data1 remaining space

All partitions except / should be setup for soft updates. If not, type 's' to enable for soft updates on all except for / (should look like UFS2+S Y under the Newfs column)

move cursor to mfid1 at the top of the screen<br>
swap 8G (or 4G if there’s a 3rd drive)<br>
/mnt/data2 remaining space <br>
'q' to save and exit

=== distributions ===
Choose the following distribudions

* developer (ok to install ports)
* custom -> lib32
exit

=== media ===
if you are installing via a cd, no need to enter this menu or change anything. Otherwise, choose ftp to install via ftp. You will be prompted to setup networking. You will need to choose a nic (typically bce0 or bce1). Say no to DHCP and IPv6. Hopefully the public nic cable was installed in bce0 so start with that nic and provide the hostname, (public) IP, netmask, gateway and DNS. When configured, it should start pinging. If it doesn't, have the NOC swap cables. Select any FTP server, usually Main or ftp4.

=== commit ===

this usually takes 12mins<br>
during the process you may need to select a new ftp mirror, this is not a problem.<br>
at the conclution of the install you will be prompted to enter the root password (2x) and returned to the configuration menu.

=== add user ===

'''Add''' user 'user'. Defaults for everything is fine, just remember to enter 'wheel' in the member group field.
Do set the password.

=== Setup timezone ===

PT

=== Networking ===
page down to the bottom and enable '[X]' sshd

If you installed via cd, you will need to visit:<br>
interfaces->bce0-><br>
No IPV6<br>
dhcp=no<br>
Set hostname, IP, DNS, gateway<br>
(i.e. setup the nic as indicated above)

Exit the install and if you installed via CD, take it out and let the machine reboot

== Configure OS, kernel, userland, jail ==

=== double check the date/time ===

date

=== populate /etc/resolv.conf ===

echo "search johncompanies.com \
nameserver 69.55.225.225 \
nameserver 69.55.230.3 \
nameserver 69.55.229.3" > /etc/resolv.conf

=== edit /etc/make.conf ===
echo "WITHOUT_X11=yes \
KERNCONF=jail4 \
BOOT_COMCONSOLE_SPEED=115200" >> /etc/make.conf

=== setup bootloader for console, etc ===

add settings to /boot/loader.conf and /boot.config:

echo "-Dh" >> /boot.config

echo 'console="comconsole,vidconsole" \
boot_multicons="YES" \
boot_serial="YES" \
mfi_linux_load="YES" \
comconsole_speed="115200"' >> /boot/loader.conf

=== enable login via serial console ===
turn off all ttyv's except 0 and 1 in /etc/ttys and turn on ttyd0, change type to vt100:

vi /etc/ttys

The changed lines should look like:

ttyv2 "/usr/libexec/getty Pc" cons25 off secure
ttyv3 "/usr/libexec/getty Pc" cons25 off secure
ttyv4 "/usr/libexec/getty Pc" cons25 off secure
ttyv5 "/usr/libexec/getty Pc" cons25 off secure
ttyv6 "/usr/libexec/getty Pc" cons25 off secure
ttyv7 "/usr/libexec/getty Pc" cons25 off secure
# Serial terminals
# The 'dialup' keyword identifies dialin lines to login, fingerd etc.
ttyu0 "/usr/libexec/getty std.9600" vt100 on secure

Restart init
kill -1 1

At this point you should have a login on console.

To configure serial console access, login to the console server as root and run:

# vi /etc/remote

following examples there, rename port to server's hostname, depending on where and which digi box this server is plugged into. Make sure to get speed right too: 115200

=== populate hosts ===

If server is at castle:
echo "10.1.4.3 backup2" >> /etc/hosts
echo "10.1.4.8 backup1" >> /etc/hosts

If server is at i2b:
echo "69.55.230.10 backup2" >> /etc/hosts
echo "10.1.2.3 backup3" >> /etc/hosts
echo "69.55.230.11 backup1" >> /etc/hosts

=== create ssh key, upload to backup servers ===
cd
ssh-keygen -t dsa -b 1024
(default location, leave password blank)

If server is at castle:
cat /root/.ssh/id_dsa.pub | ssh backup2 'cat - >> /root/.ssh/authorized_keys'
cat /root/.ssh/id_dsa.pub | ssh backup1 'cat - >> /root/.ssh/authorized_keys'

If server is at i2b:
cat /root/.ssh/id_dsa.pub | ssh backup3 'cat - >> /root/.ssh/authorized_keys'
cat /root/.ssh/id_dsa.pub | ssh backup2 'cat - >> /root/.ssh/authorized_keys'
cat /root/.ssh/id_dsa.pub | ssh backup1 'cat - >> /root/.ssh/authorized_keys'

confirm that you can ssh to backup2 and backup1 without getting a login prompt

ssh backup2 hostname
ssh backup1 hostname
ssh backup3 hostname

=== create & populate binaries/scripts dirs ===
<pre>mkdir -p /usr/local/jail/bin
mkdir -p /usr/local/jail/rc.d
mkdir -p /usr/local/jail/template/
mkdir /mnt/data1
mkdir /mnt/data2
scp backup2:"/mnt/data4/bin/freebsd9.x/*" /usr/local/jail/bin
cd /usr/local/jail/rc.d/
touch quad1
touch deprecated
chmod +x *
cd /usr/local/jail/bin
ln -s /usr/local/jail/rc.d/quad1 quad1
ln -s /usr/local/jail/bin/jailmake_md jailmake
ln -s /usr/local/jail/bin/js_md js
ln -s /usr/local/jail/bin/canceljail_md canceljail
ln -s /usr/local/jail/bin/jailmakeempty_md jailmakeempty
ln -s /usr/local/jail/bin/postboot_md postboot
ln -s /usr/local/jail/bin/preboot_md preboot
ln -s /usr/local/jail/bin/startjail_md startjail
ln -s /usr/local/jail/bin/stopjail_md stopjail

rehash</pre>

=== edit root's path and login script ===

vi /root/.cshrc

Change alias entries (add G):

<pre>alias la ls -aG
alias lf ls -FAG
alias ll ls -lAG
alias ls ls -AG
alias mbm mb mount
alias mbu mb umount
alias cjb cd /usr/local/jail/bin
alias cd1 cd /mnt/data1
alias cd2 cd /mnt/data2
alias cd3 cd /mnt/data3
alias jtop jtop lj
alias j jobs</pre>

add to path be careful to leave a space after bin and make sure the wrapping isn't broken:
/usr/local/jail/bin

alter the prompt, set the following:
set prompt = "$user@`/bin/hostname -s` %/# "

at the bottom of the file add:
<pre>set sshtty=`who am i|awk '{print $2}'`
/usr/sbin/rtprio 3 -`psj | grep $sshtty | awk '{print $2}'`

set shortty=`who am i | awk '{print $2}' | sed -E 's/.*(..)$/\1/'`
foreach x (`psj | grep sh | grep $shortty | awk '{print $2}'`)
/usr/sbin/rtprio 2 -$x
end</pre>

Make the new settings active in current shell:
source /root/.cshrc

=== update ports ===

portsnap fetch
portsnap extract

To update later on:
portsnap fetch
portsnap update

=== install svn ===

<pre>
setenv PACKAGESITE "ftp://ftp4.freebsd.org/pub/FreeBSD/ports/amd64/packages-9-current/Latest/"
pkg_add subversion
</pre>

=== get latest sources for this release ===

<pre>
cd /usr

# mv src/ src.orig
tar cvzf src.orig.tgz src
rm -fr src/*
svn checkout svn://svn.FreeBSD.org/base/stable/9 /usr/src
</pre>

To update:
make update SVN_UPDATE=yes

=== configure new kernel ===

Pull down the kernel config we are using for this distribution. In this case we use an 8.2 kernel config on 8.3, which is valid. The local file should be the same name as host- <tt>jail4</tt> in this example

<pre>cd /usr/src/sys/amd64/conf
scp backup2:/mnt/data4/build/freebsd/kern_config-9.1-amd64 ./jail4</pre>

edit the kernel config and change ident to be the name of the jail:
vi jail4
ident jail4

Optional, edit <tt>/sys/conf/newvers.sh</tt> to add <tt>–jc2</tt> to the end of the BRANCH string (RELEASE-jc2)
vi /sys/conf/newvers.sh

notes on kernel configuring: http://www.FreeBSD.org/doc/en_US.ISO8859-1/books/handbook/kernelconfig-config.html

=== install patches ===

We don't have any patches right now. Refer to older FreeBSD version build docs on how that is/was done.

=== build, install kernel and world ===

Rename current generic kernel so it will always be available to boot from. Save room by removing non-needed kernel modules:

<pre>cd /boot
mv kernel kernel.GENERIC
cd kernel.GENERIC
mkdir hold
mv mfi_linux.ko hold/
mv linux.ko hold/
mv linprocfs.ko hold/
mv linsysfs.ko hold/
mv geom_vinum.ko hold/
mv geom_concat.ko hold/
mv zfs.* hold/
mv opensolaris* hold/

rm *.ko
rm *.symbols
mv hold/* .
rmdir hold/

Note on -DWITHOUT_CLANG: try to do it without including that directive, it may work for you.

cd /usr/src
make buildworld KERNCONF=jail4; mail -s 'buildworld done' dave.boodman@vtext.com < /dev/null

make -DWITHOUT_CLANG buildworld KERNCONF=jail4; mail -s 'buildworld done' dave.boodman@vtext.com < /dev/null
</pre>
~4.5hr

cd /usr/src
make buildkernel installkernel

mergemaster -p
You will be prompted to merge, replace or ignore files changed by the src update. In most cases you can '''d'''elete the temp (new) files.

make -DWITHOUT_CLANG installworld
~34min

ONLY if this will be a zfs system (not currently used in 8.x):
<pre>cd /sys/modules/zfs
make
make install
cd /sys/modules/opensolaris
make
make install</pre>

=== populate devfs ruleset ===
scp backup2:/mnt/data4/build/freebsd/devfs.rules.8x /etc/devfs.rules

=== populate /etc/rc.conf with IPs and service settings ===

<pre>vi /etc/rc.conf

kern_securelevel_enable="NO"
portmap_enable="NO"
sendmail_enable="NO"
usbd_enable="YES"

nfs_client_enable="YES"
nfs_reserved_port_only="YES"
inetd_enable="YES"
inetd_flags="-wW -a 10.1.4.XXX"
devfs_system_ruleset="devfsrules_show_all"

ifconfig_bce1="inet 10.1.4.XXX netmask 255.255.255.0"
ifconfig_bce0="inet 69.55.2XX.XXX netmask 255.255.255.0"
#ifconfig_bce0_alias0="inet 69.55.2XX.XXX netmask 255.255.255.0"

fsck_y_enable="YES"
background_fsck="NO"
#rc_mfi_raid_tty_log="YES"
#zfs_enable="YES"</pre>

Modify IPs, hostname, gateway for this box.

=== make sure sysctls are set and preserved upon boot ===

<pre>echo "kern.consmute=0\
kern.ipc.shm_use_phys=1\
kern.ipc.shmall=131070\
kern.ipc.shmmax=134217728\
net.inet.tcp.syncookies=0\
kern.maxfiles=32768\
kern.fallback_elf_brand=3\
kern.maxprocperuid=4000\
security.jail.sysvipc_allowed=1\
security.jail.allow_raw_sockets=1\
security.jail.socket_unixiproute_only=1\
security.jail.chflags_allowed=0\
dev.amr.0.allow_volume_configure=1\
compat.linux.osrelease=2.6.12\
vm.pmap.shpgperproc=500\
security.bsd.unprivileged_read_msgbuf=0\
kern.maxvnodes=400000" >> /etc/sysctl.conf</pre>

Tuning note: watch vfs.numvnodes while the server is live to get guidance on where to set maxvnodes

=== mount procfs ===
echo "proc /proc procfs rw 0 0" >> /etc/fstab
echo "linprocfs /usr/compat/linux/proc linprocfs rw 0 0" >> /etc/fstab
echo "linsysfs /usr/compat/linux/sys linsysfs rw 0 0" >> /etc/fstab
mkdir -p /usr/compat/linux/proc
mkdir -p /usr/compat/linux/sys

=== enable noatime option ===

grep data /etc/fstab

data1 and data2 should look something like (add ',noatime' after 'rw'):

/dev/mfid0s1g /mnt/data1 ufs rw,noatime 2 2
/dev/mfid1s1d /mnt/data2 ufs rw,noatime 2 2

=== reboot. Confirm new kernel is loaded, devfs in place ===

uname -a

Check devfs rules
devfs rule showsets
devfs rule -s 3 show

Should see:
<pre># devfs rule showsets
1
2
3
4

# devfs rule -s 3 show
100 include 1
207 path pts* unhide
217 path fd unhide
218 path fd/* unhide
300 path ttyp* unhide
301 path ttyq* unhide
302 path ttyr* unhide
303 path ttys* unhide
304 path ttyP* unhide
305 path ttyQ* unhide
306 path ttyR* unhide
307 path ttyS* unhide
400 path null unhide
500 path zero unhide
600 path random unhide
610 path urandom unhide
700 path mem unhide
710 path kmem unhide
810 path mdctl unhide
900 path stdin unhide
910 path stdout unhide
920 path stderr unhide</pre>

=== Install raid mgmt tool ===

==== Perc5/i, 6/i ====
Pull over cli from previous system (jail9)
scp root@10.1.4.109:"/usr/local/sbin/mega*" /usr/local/sbin/
scp root@10.1.4.109:/usr/local/libexec/MegaCli /usr/local/libexec/MegaCli

These are linux-based tools. This will require linux base...which you might install via:

pkg_add -r linux_base

Test:
rehash; megacli ldinfo lall a0
or
megarc -ldInfo -a0 -Lall
(2850)

however, linux does seem to be installed already so we don't need to pkg_add or port install or rsync anything over from a current system.

==== 2850 PERC 4e/Di- no linux ====
<pre>cd /usr/ports/distfiles/
fetch http://backup01.best-hosting.ru/pub/FreeBSD/ports/distfiles/dr_freebsd_1.51.zip
cd /usr/ports/sysutils/megarc
make install clean
megarc -dispCfg -a0</pre>

=== install rsync from ports ===
cd /usr/ports/net/rsync
make install clean

choose default options

=== configure inetd to respond to mrtg load queries ===
echo "load stream tcp nowait user /usr/local/jail/bin/load.pl load.pl" >> /etc/inetd.conf
echo "load 12384/tcp" >> /etc/services

=== install perl ===
cd /usr/ports/lang/perl5.12
make install clean

choose defaults

=== install bb client ===

Compiling from source on AMD64 will not work. So, we use a linux-compiled version and rely on linux compat.

adduser

Output/response:

<pre>Username: bb
Full name: bb
Uid (Leave empty for default): 1984
Login group [bb]:
Login group is bb. Invite bb into other groups? []:
Login class [default]:
Shell (sh csh tcsh nologin) [sh]:
Home directory [/home/bb]:
Use password-based authentication? [yes]:
Use an empty password? (yes/no) [no]:
Use a random password? (yes/no) [no]: yes
Lock out the account after creation? [no]:
Username : bb
Password : <random>
Full Name : bb
Uid : 1984
Class :
Groups : bb
Home : /home/bb
Shell : /bin/sh
Locked : no
OK? (yes/no): yes</pre>

cd /usr/home/bb
scp backup2:/mnt/data4/build/bb/bb-freebsd_linuxcompat.tgz .
tar xzf bb-freebsd_linuxcompat.tgz

edit /home/bb/bbc1.9e-btf/etc/bb-hosts with something like:

echo "10.1.4.5 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh \
10.1.4.'''104 jail4'''.johncompanies.com # ssh" > /home/bb/bbc1.9e-btf/etc/bb-hosts

Edit for machine name and private IP.

if this machine is at i2b:
echo "69.55.230.2 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh \
69.55.228.104 jail4.johncompanies.com # ssh" > /home/bb/bbc1.9e-btf/etc/bb-hosts

<pre>vi /home/bb/bbc1.9e-btf/ext/openfiles

MACHINE="jail4,johncompanies,com" # HAS TO BE IN A,B,C FORM</pre>
Edit for machine name.

<pre>cd /usr/home/bb/bbc1.9e-btf/etc
./bbchkcfg.sh
(y to questions)
./bbchkhosts.sh
(ignore ssh errors)
cd ../..
chown -R bb .
su bb
cd
cd bbc1.9e-btf
./runbb.sh start
more BBOUT
(look for errors)
exit</pre>

Put in script to start bb @ boot:
echo 'su - bb -c "cd /home/bb/bbc1.9e-btf; ./runbb.sh start"' > /usr/local/etc/rc.d/bb.sh
chmod +x /usr/local/etc/rc.d/bb.sh

=== remove reserve space ===
<pre>cd
umount /mnt/data1
umount /mnt/data2
tunefs -m 0 /mnt/data1
tunefs -m 0 /mnt/data2
mount -a</pre>

=== setup rdate ===
cd /usr/ports/sysutils/rdate
make install clean

crontab -e
0 0 * * * /usr/local/sbin/rdate -s utcnist.colorado.edu

/usr/local/sbin/rdate -s utcnist.colorado.edu

We used to use ntpd, however it listens on jail IPs which is a security risk. So we stopped. Here's the old instructions:
install new ntp from ports
/usr/ports/net/ntp
make install clean

<pre>echo "server 10.1.4.5" > /etc/ntp.conf
/usr/sbin/ntpd -p /var/run/ntpd.pid
sleep 2; ntpq -p</pre>(confirm it’s able to reach our time server)

=== make a data partition ===

IF you didn't format the data partition during sysinstall:

Create a g partition on 2nd mirror – bsdlabel no longer works (below shows d partition made with sysinstall):

<pre>jail8 /usr/home/bb# gpart show
=> 63 285474735 mfid0 MBR (136G)
63 285458922 1 freebsd [active] (136G)
285458985 15813 - free - (7.7M)

=> 0 285458922 mfid0s1 BSD (136G)
0 524288 1 freebsd-ufs (256M)
524288 12582912 2 freebsd-swap (6.0G)
13107200 524288 4 freebsd-ufs (256M)
13631488 524288 5 freebsd-ufs (256M)
14155776 8388608 6 freebsd-ufs (4.0G)
22544384 262914538 7 freebsd-ufs (125G)

=> 63 584843175 mfid1 MBR (279G)
63 584830197 1 freebsd [active] (279G)
584830260 12978 - free - (6.3M)

=> 0 584830197 mfid1s1 BSD (279G)
0 16777216 2 freebsd-swap (8.0G)
16777216 568052981 4 freebsd-ufs (271G)

jail8 /usr/home/bb# gpart show mfid1s1
=> 0 584830197 mfid1s1 BSD (279G)
0 16777216 2 freebsd-swap (8.0G)
16777216 568052981 4 freebsd-ufs (271G)

# gpart list mfid1s1

Geom name: mfid1s1
fwheads: 255
fwsectors: 63
last: 584830196
first: 0
entries: 8
scheme: BSD
Providers:
1. Name: mfid1s1b
Mediasize: 8589934592 (8.0G)
Sectorsize: 512
Mode: r1w1e0
rawtype: 1
length: 8589934592
offset: 0
type: freebsd-swap
index: 2
end: 16777215
start: 0
2. Name: mfid1s1d
Mediasize: 290843126272 (271G)
Sectorsize: 512
Mode: r0w0e0
rawtype: 7
length: 290843126272
offset: 8589934592
type: freebsd-ufs
index: 4
end: 584830196
start: 16777216
Consumers:
1. Name: mfid1s1
Mediasize: 299433060864 (279G)
Sectorsize: 512
Mode: r1w1e1

# gpart delete -i 4 mfid1s1
mfid1s1d deleted
jail8 /usr/home/bb# gpart list mfid1s1
Geom name: mfid1s1
fwheads: 255
fwsectors: 63
last: 584830196
first: 0
entries: 8
scheme: BSD
Providers:
1. Name: mfid1s1b
Mediasize: 8589934592 (8.0G)
Sectorsize: 512
Mode: r1w1e0
rawtype: 1
length: 8589934592
offset: 0
type: freebsd-swap
index: 2
end: 16777215
start: 0
Consumers:
1. Name: mfid1s1
Mediasize: 299433060864 (279G)
Sectorsize: 512
Mode: r1w1e1

# gpart show mfid1s1
=> 0 584830197 mfid1s1 BSD (279G)
0 16777216 2 freebsd-swap (8.0G)
16777216 568052981 - free - (271G)

# gpart add -t freebsd-ufs -i 7 mfid1s1
mfid1s1g added

# gpart show mfid1s1
=> 0 584830197 mfid1s1 BSD (279G)
0 16777216 2 freebsd-swap (8.0G)
16777216 568052981 7 freebsd-ufs (271G)</pre>

Here's how we USED to do it with bsdlabel:

bsdlabel -e /dev/mfid0s1

given:
<pre># /dev/aacd0s1:
8 partitions:
# size offset fstype [fsize bsize bps/cpg]
a: 262144 0 4.2BSD 2048 16384 16392
b: 4194304 262144 swap
c: 143363997 0 unused 0 0 # "raw" part, don't edit
d: 524288 4456448 4.2BSD 2048 16384 32776
e: 524288 4980736 4.2BSD 2048 16384 32776
f: 6291456 5505024 4.2BSD 2048 16384 28552</pre>

new offset = 6291456 + 5505024 = 11796480<br>
new size is size for 'c' partition minus the new start from above<br>
143363997 - 11796480 = 131567517

So:<br>
g: 131567517 11796480 unused 0 0

=== create the jail template ===

cd /usr/ports/sysutils/jailutils
make install clean

Create an md device to hold the jail:
<pre>touch /mnt/data1/jail-template20g
mdconfig -a -t vnode -s 20g -f /mnt/data1/jail-template20g -u 0
newfs -O 1 /dev/md0
mkdir /mnt/data1/jail-DIR
mount /dev/md0 /mnt/data1/jail-DIR</pre>

Build world into the jail:
<pre>cd /usr/src
make world DESTDIR=/mnt/data1/jail-DIR; taskdone</pre>
~4.5 hr

Make /etc into the jail, mount dev, copy in jkill:
<pre>cd etc
make distribution DESTDIR=/mnt/data1/jail-DIR
mount -t devfs devfs /mnt/data1/jail-DIR/dev
devfs -m /mnt/data1/jail-DIR/dev rule -s 3 applyset
cd /mnt/data1/jail-DIR
ln -sf dev/null kernel
cp /usr/local/sbin/jkill /mnt/data1/jail-DIR/sbin</pre>

Enter into jail to do configuration:
jail /mnt/data1/jail-DIR testhostname 192.168.11.100 /bin/sh
csh

Create fstab:
<pre>touch /etc/fstab
echo 'network_interfaces=""\
hostname="newsystem"\
kern_securelevel_enable="NO"\
sendmail_enable="YES"\
sshd_enable="YES"' > /etc/rc.conf

echo "nameserver 69.55.225.225\
nameserver 69.55.230.3" >> /etc/resolv.conf</pre>

Edit crontab:
vi /etc/crontab
remove the adjkerntz lines
comment out periodic’s and put this line above them:
# IF YOU UNCOMMENT THESE, PLEASE ADJUST THEIR RUN TIME

rm -rf /etc/periodic/daily/400.status-disks

check and remove any crap in /tmp

vi /etc/periodic/security/100.chksetuid
replace: <tt>MP=`mount -t ufs | grep -v " nosuid" | awk '{ print $3 }' | sort`</tt><br>
with: <tt>MP='/'</tt> (use single quotes)

mkdir -p /usr/compat/linux/dev

Add account for user. Output/response:
<pre>adduser

Username: user
Full name: user
Uid (Leave empty for default):
Login group [user]:
Login group is user. Invite user into other groups? []: wheel
Login class [default]:
Shell (sh csh tcsh nologin) [sh]:
Home directory [/home/user]:
Home directory permissions (Leave empty for default):
Use password-based authentication? [yes]:
Use an empty password? (yes/no) [no]:
Use a random password? (yes/no) [no]: y
Lock out the account after creation? [no]:
Username : user
Password : <random>
Full Name : user
Uid : 1001
Class :
Groups : user
Home : /home/user
Home Mode :
Shell : /bin/sh
Locked : no
OK? (yes/no): y
adduser: INFO: Successfully added (user) to the user database.
adduser: INFO: Password for (user) is: 901gmYjO
Add another user? (yes/no): n
Goodbye!</pre>

set TERM:
vi /usr/home/user/.profile
TERM=vt100; export TERM

Set time zone to PT:
tzsetup

Reload aliases:
newaliases

Replace reboot/halt:
rm /sbin/halt /sbin/reboot
ln /sbin/jkill /sbin/halt
ln /sbin/jkill /sbin/reboot

Redirect console output, comment out console and move to /var/log/messages:
vi /etc/syslog.conf
#*.err;kern.warning;auth.notice;mail.crit /dev/console
*.err;kern.warning;auth.notice;mail.crit /var/log/messages

exit
exit

Lib32 compat library:
cd libexec
chflags noschg ld-elf32.so.1
chflags noschg ld-elf.so.1
mv ld-elf32.so.1 ld-elf32.so.1-orig
ln ld-elf.so.1 ld-elf32.so.1
chflags schg ld-elf.so.1
chflags schg ld-elf32.so.1

Replace traceroute:
mv /mnt/data1/jail-DIR/usr/sbin/traceroute /mnt/data1/jail-DIR/usr/sbin/_traceroute
echo '#\!/bin/sh\
/usr/sbin/_traceroute -i bce0 $1' >> /mnt/data1/jail-DIR/usr/sbin/traceroute
chmod +x /mnt/data1/jail-DIR/usr/sbin/traceroute

Modify 'bce0' to reflect whichever nic is public on this hardware.

Clean out ports before copying into the jail (~30mins):
cd /usr/ports
make -DNOCLEANDEPENDS clean
rm -fr /usr/ports/distfiles/*
cp -r /usr/ports /mnt/data1/jail-DIR/usr

rm /mnt/data1/jail-DIR/root/.history

Umount the jail and dump it:
<pre>cd /
umount /mnt/data1/jail-DIR/dev
dump -0a -f /usr/local/jail/template/template /dev/md0
umount /dev/md0
rmdir /mnt/data1/jail-DIR
mdconfig -d -u 0</pre>

=== setup backups ===
<pre>echo '#\!/bin/sh\
backupdir=/data/jail3\
server=backup1\
\
## ENTRY /etc\
## ENTRY /usr/local/etc\
## ENTRY /usr/local/jail\
## ENTRY /root/logs' > /usr/local/jail/bin/backup.config</pre>

Edit to reflect backup server and jail hostname

On backup server, setup backup dirs:
ssh backup1 mkdir -p /data/jail3/0

On backup server, setup backup dirs:
backup1# vi /usr/local/sbin/snapshot_rotate

Edit /usr/local/jail/bin/backup.xxx to use the right drives and copy to /usr/local/jail/bin/backup
vi /usr/local/jail/bin/backup.md
adjust df so it includes all relevant drives, currently <tt>df > /etc/df.bak</tt> is fine. Also, make sure the binary/script source is pulling from the right dir on backup2:
/usr/local/bin/rsync -a backup2:/mnt/data4/bin/freebsd8.x/ /usr/local/jail/bin/

cp /usr/local/jail/bin/backup.md /usr/local/jail/bin/backup

=== create /root/logs ===
mkdir /root/logs

=== edit sshd_config ===
vi /etc/ssh/sshd_config
ListenAddress 69.55.'''229.7'''
ListenAddress 10.1.4.'''103'''

Adjust to pub/private IP for jail.

Restart sshd:
kill -1 `cat /var/run/sshd.pid`

=== add crontab entries ===
<pre>crontab -e
5 0 * * * /usr/local/jail/bin/backup
1 0 1 * * /usr/local/jail/bin/ipfwreset
0 18 * * * /usr/local/jail/bin/ipfwbackup
4,9,14,19,24,29,34,39,44,49,55,59 * * * * /usr/local/jail/bin/trafstats
0 0,6,12,18 * * * /usr/local/jail/bin/sync_jail_names</pre>

On 2950:
*/5 * * * * /usr/local/jail/bin/perc5iraidchk

On 3ware-based servers:
0 0 * * * /usr/local/jail/bin/3wraidchk

=== Reboot notify script ===
ln -s /usr/local/jail/bin/notify.sh /usr/local/etc/rc.d/notify.sh

=== patch jail against starting jails with rtprio ===

<pre>mv /usr/sbin/jail /usr/sbin/jail_
echo '#\!/bin/sh\
/usr/sbin/rtprio -t /usr/sbin/jail_ $*' > /usr/sbin/jail
chmod +x /usr/sbin/jail</pre>

=== make sure mail works ===
If there are map errors:
cd /etc/mail; make maps

=== recover space on /usr (optional) ===

rm -fr /usr/obj

=== wrapper for jps ===

mv /usr/local/sbin/jps /usr/local/sbin/jps_

=== wrapper for jls ===

mv /usr/sbin/jls /usr/sbin/jls_

=== wrapper for jexec ===

mv /usr/sbin/jexec /usr/sbin/jexec_

=== install jtop ===

cd /usr/ports/sysutils/jtop
make install clean

=== block jails from reaching private net ===
echo 'ipfw add 1 deny ip from 69.55.224.0/20 to 10.1.4.0/24' > /usr/local/etc/rc.d/ipfw.sh
chmod 0700 /usr/local/etc/rc.d/ipfw.sh

== add to management infrastructure ==

=== mail ===

==== add to management db ====

tables: jc.ref_machines and jc.ref_templates

on jail run:
uname -r

Which shows something like:
8.3-RELEASE-p2

Insert into db:

<pre>insert into ref_machines values (null,'jail3','mx3.johncompanies.com',0,'f8');
select machine_id from ref_machines where host='jail3';
+------------+
| machine_id |
+------------+
| 35 |
+------------+
insert into ref_templates values ('','8.3-RELEASE-jc2',35,'FreeBSD 8.3',1);</pre>

==== add to bb server ====
vi /usr/home/bb/bbsrc/bb1.9i-btf/etc/bb-hosts
10.1.4.'''109 jail9'''.johncompanies.com # ssh

In the case of an i2b server, use real ip:
69.55.'''229.7 jail3'''.johncompanies.com # ssh

<pre>su bb
cd
bbsrc/bb/runbb.sh restart ; exit</pre>

==== Update backupgraph ====
vi /usr/local/www/mgmt/cgi/backupgraph.pl
(add hostname)

==== Update load mrtg ====
vi /usr/local/www/mgmt/mrtg/mrtg1.cfg
(add new entry to file following existing format)

=== ns1c ===
fwd and reverse lookups:
vr johncompanies.com

vi internal.johncompanies.com
rndc reload johncompanies.com IN private

ptr 69.55.227.x

=== wiki ===
add to server/cabinet map

=== firewall ===
add an outside blocking rule to the firewall, so this machine can only be reached from inside the firewall. Follow example already in firewall jail17 is:

00117 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 or 69.55.238.150 } to 69.55.228.200
00117 deny ip from any to 69.55.228.200

jail19 would be 00119...
ipfw add 00109 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 } to 69.55.232.3
ipfw add 00109 deny ip from any to 69.55.232.3

=== select some customers for castle probe map ===

= FreeBSD 8.x =

== Assumptions ==

Setup instructions below assume this is DELL 2950 with an LSI-based SAS RAID card.

Server is at castle, connected to pub, private, serial and DRAC

Assuming OS loading done via IPKVM with ISO mounted via USB

Assumes at 4 drives, 2 mirrors

== Configure server BIOS ==

setup console redirect, speed 115200

set LCD string to name of server "jail8"

set date to GMT

go into RAID bios and setup mirrors

configure DRAC: TODO

== Install OS (sysinstall) ==

boot to bootonly disk for AMD version of FreeBSD, i.e. <tt>FreeBSD-8.3-RELEASE-amd64-bootonly.iso</tt>

when the install menu appears, choose <tt>custom install</tt>

=== partition menu ===
move cursor to mfid0, hit space (takes you to partition map screen). If there is only 1 mirror, there will be no option to select a specifit drive: mfid0 will be selected.

type 'a' to use entire disk<br>
type 'q' to quit and save<br>
choose 'freebsd standard mbr'

space to '''un'''select mfid0<br>
cursor down to mfid1<br>
hit space

type 'a' to use entire disk<br>
type 'q' to quit and save<br>
choose 'none' for boot mgr (leave untouched)

cursor over mfid0<br>
space<br>
(takes you into part. Screen again) 'q' to exit<br>
none for boot mgr<br>

Make sure both drives (mfid0 and mfid1) are checked and tab to ok

=== Label menu ===

Make sure mfid0 is highlighted at the top of the screen, setup the following partitions

/ 512M<br>
swap 6G<br>
/var 256M <br>
/tmp 256M<br>
/usr 5G<br>
/mnt/data1 remaining space

All partitions except / should be setup for soft updates. If not, type 's' to enable for soft updates on all except for / (should look like UFS2+S Y under the Newfs column)

move cursor to mfid1 at the top of the screen<br>
swap 8G (or 4G if there’s a 3rd drive)<br>
/mnt/data2 remaining space <br>
'q' to save and exit

=== distributions ===
Choose the following distribudions

* developer (ok to install ports)
* custom -> lib32
exit

=== media ===
if you are installing via a cd, no need to enter this menu or change anything. Otherwise, choose ftp to install via ftp. You will be prompted to setup networking. You will need to choose a nic (typically bce0 or bce1). Say no to DHCP and IPv6. Hopefully the public nic cable was installed in bce0 so start with that nic and provide the hostname, (public) IP, netmask, gateway and DNS. When configured, it should start pinging. If it doesn't, have the NOC swap cables. Select any FTP server, usually Main or ftp4.

=== commit ===

this usually takes 12mins<br>
during the process you may need to select a new ftp mirror, this is not a problem.<br>
at the conclution of the install you will be prompted to enter the root password (2x) and returned to the configuration menu.

=== add user ===

'''Add''' user 'user'. Defaults for everything is fine, just remember to enter 'wheel' in the member group field.
Do set the password.

=== Setup timezone ===

PT

=== Networking ===
page down to the bottom and enable '[X]' sshd

If you installed via cd, you will need to visit:<br>
interfaces->bce0-><br>
No IPV6<br>
dhcp=no<br>
Set hostname, IP, DNS, gateway<br>
(i.e. setup the nic as indicated above)

Exit the install and if you installed via CD, take it out and let the machine reboot

== Configure OS, kernel, userland, jail ==

=== double check the date/time ===

=== populate /etc/resolv.conf ===

echo "nameserver 69.55.225.225\
nameserver 69.55.230.3
nameserver 69.55.229.3" > /etc/resolv.conf

=== edit /etc/make.conf ===
echo "WITHOUT_X11=yes \
KERNCONF=jail3 \
BOOT_COMCONSOLE_SPEED=115200" >> /etc/make.conf

=== setup bootloader for console, etc ===

add settings to /boot/loader.conf and /boot.config:

echo "-Dh" >> /boot.config

echo 'console="comconsole,vidconsole" \
boot_multicons="YES" \
boot_serial="YES" \
mfi_linux_load="YES" \
comconsole_speed="115200"' >> /boot/loader.conf

=== enable login via serial console ===
turn off all ttyv's except 0 and 1 in /etc/ttys and turn on ttyd0, change type to vt100:

vi /etc/ttys

The changed lines should look like:

ttyv2 "/usr/libexec/getty Pc" cons25 off secure
ttyv3 "/usr/libexec/getty Pc" cons25 off secure
ttyv4 "/usr/libexec/getty Pc" cons25 off secure
ttyv5 "/usr/libexec/getty Pc" cons25 off secure
ttyv6 "/usr/libexec/getty Pc" cons25 off secure
ttyv7 "/usr/libexec/getty Pc" cons25 off secure
# Serial terminals
# The 'dialup' keyword identifies dialin lines to login, fingerd etc.
ttyu0 "/usr/libexec/getty std.9600" vt100 on secure

Restart init
kill -1 1

At this point you should have a login on console.

To configure serial console access, login to the console server as root and run:

# vi /etc/remote

following examples there, rename port to server's hostname, depending on where and which digi box this server is plugged into. Make sure to get speed right too: 115200

=== populate hosts ===

If server is at castle:
echo "10.1.4.3 backup2" >> /etc/hosts
echo "10.1.4.8 backup1" >> /etc/hosts

If server is at i2b:
echo "69.55.230.10 backup2" >> /etc/hosts
echo "10.1.2.3 backup3" >> /etc/hosts
echo "69.55.230.11 backup1" >> /etc/hosts

=== create ssh key, upload to backup servers ===
cd
ssh-keygen -t dsa -b 1024
(default location, leave password blank)

If server is at castle:
cat /root/.ssh/id_dsa.pub | ssh backup2 'cat - >> /root/.ssh/authorized_keys'
cat /root/.ssh/id_dsa.pub | ssh backup1 'cat - >> /root/.ssh/authorized_keys'

If server is at i2b:
cat /root/.ssh/id_dsa.pub | ssh backup3 'cat - >> /root/.ssh/authorized_keys'
cat /root/.ssh/id_dsa.pub | ssh backup2 'cat - >> /root/.ssh/authorized_keys'
cat /root/.ssh/id_dsa.pub | ssh backup1 'cat - >> /root/.ssh/authorized_keys'

confirm that you can ssh to backup2 and backup1 without getting a login prompt

ssh backup2 hostname
ssh backup1 hostname
ssh backup3 hostname

=== create & populate binaries/scripts dirs ===
<pre>mkdir -p /usr/local/jail/bin
mkdir -p /usr/local/jail/rc.d
mkdir -p /usr/local/jail/template/
mkdir /mnt/data1
mkdir /mnt/data2
scp backup2:"/mnt/data4/bin/freebsd8.x/*" /usr/local/jail/bin
cd /usr/local/jail/rc.d/
touch quad1
touch deprecated
chmod +x *
cd /usr/local/jail/bin
ln -s /usr/local/jail/rc.d/quad1 quad1
ln -s /usr/local/jail/bin/jailmake_md jailmake
ln -s /usr/local/jail/bin/js_md js
ln -s /usr/local/jail/bin/canceljail_md canceljail
ln -s /usr/local/jail/bin/jailmakeempty_md jailmakeempty
ln -s /usr/local/jail/bin/postboot_md postboot
ln -s /usr/local/jail/bin/preboot_md preboot
ln -s /usr/local/jail/bin/startjail_md startjail
ln -s /usr/local/jail/bin/stopjail_md stopjail

rehash</pre>

=== edit root's path and login script ===

vi /root/.cshrc

Change alias entries (add G):

<pre>alias la ls -aG
alias lf ls -FAG
alias ll ls -lAG
alias ls ls -AG
alias mbm mb mount
alias mbu mb umount
alias cjb cd /usr/local/jail/bin
alias cd1 cd /mnt/data1
alias cd2 cd /mnt/data2
alias cd3 cd /mnt/data3
alias jtop jtop lj
alias j jobs</pre>

add to path be careful to leave a space after bin and make sure the wrapping isn't broken:
/usr/local/jail/bin

alter the prompt, set the following:
set prompt = "`/bin/hostname -s` %/# "

at the bottom of the file add:
<pre>set sshtty=`who am i|awk '{print $2}'`
/usr/sbin/rtprio 3 -`psj | grep $sshtty | awk '{print $2}'`

set shortty=`who am i | awk '{print $2}' | sed -E 's/.*(..)$/\1/'`
foreach x (`psj | grep sh | grep $shortty | awk '{print $2}'`)
/usr/sbin/rtprio 2 -$x
end</pre>

Make the new settings active in current shell:
source /root/.cshrc

=== install cvsup ===

<pre>cd /usr/ports/net/cvsup-without-gui
make install clean; rehash; mail -s 'cvs installed' support@johncompanies.com < /dev/null</pre>
stand by for gettext options (use defaults). this process takes approx 22mins- hence the email/page notice above.

=== get latest sources for this release ===

<pre>cd /usr/src
echo "*default host=cvsup4.freebsd.org\
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_8_3\
*default delete use-rel-suffix\
*default compress\
src-all" > sup</pre>

If you need to run stable (cause release is broken or some other reason) make the sup file look like:

<pre>echo "*default host=cvsup4.freebsd.org\
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_8\
*default delete use-rel-suffix\
*default compress\
src-all" > sup</pre>

cvsup sup ; mail -s 'cvs sup done' support@johncompanies.com < /dev/null

time varies, 10-20mins

=== configure new kernel ===

Pull down the kernel config we are using for this distribution. In this case we use an 8.2 kernel config on 8.3, which is valid. The local file should be the same name as host- <tt>jail3</tt> in this example

<pre>cd /usr/src/sys/amd64/conf
scp backup2:/mnt/data4/build/freebsd/kern_config-8.2-amd64 ./jail3</pre>

edit the kernel config and change ident to be the name of the jail:
vi jail3
ident jail3

Optional, edit <tt>/sys/conf/newvers.sh</tt> to add <tt>–jc2</tt> to the end of the BRANCH string (RELEASE-jc2)
vi /sys/conf/newvers.sh

notes on kernel configuring: http://www.FreeBSD.org/doc/en_US.ISO8859-1/books/handbook/kernelconfig-config.html

=== install patches ===

The various patches are in /mnt/data4/build/freebsd/patches on backup2. There are dirs for each version. Not all dirs are populated, but patches for later versions work on older ones unless there is a new patch in the older dir.

there are no patches we use for 8.x, but here would be the commands:

cd /usr/src
scp backup2:"/mnt/data4/build/freebsd/patches/8.0/*" .

Apply patches, i.e. the <tt>jls-patch</tt>:
patch -l < jls-patch

=== build, install kernel and world ===

Rename current generic kernel so it will always be available to boot from. Save room by removing non-needed kernel modules:

<pre>cd /boot
mv kernel kernel.GENERIC
cd kernel.GENERIC
mkdir hold
mv mfi_linux.ko hold/
mv linux.ko hold/
mv linprocfs.ko hold/
mv linsysfs.ko hold/
mv geom_vinum.ko hold/
mv geom_concat.ko hold/
mv zfs.* hold/
mv opensolaris* hold/

rm *.ko
rm *.symbols
mv hold/* .
rmdir hold/

cd /usr/src
make buildkernel installkernel

make buildworld ; mail -s 'buildworld done' dave.boodman@vtext.com < /dev/null</pre>
~38mins

make installworld
~34min

mergemaster -i
You will be prompted to merge, replace or ignore files changed by the src update. In most cases you can '''d'''elete the temp (new) files.

ONLY if this will be a zfs system (not currently used in 8.x):
<pre>cd /sys/modules/zfs
make
make install
cd /sys/modules/opensolaris
make
make install</pre>

=== populate devfs ruleset ===
scp backup2:/mnt/data4/build/freebsd/devfs.rules.8x /etc/devfs.rules

=== populate /etc/rc.conf with IPs and service settings ===

<pre>vi /etc/rc.conf

kern_securelevel_enable="NO"
portmap_enable="NO"
sendmail_enable="NO"
usbd_enable="YES"

nfs_client_enable="YES"
nfs_reserved_port_only="YES"
inetd_enable="YES"
inetd_flags="-wW -a 10.1.4.'''103'''"
devfs_system_ruleset="devfsrules_show_all"

ifconfig_bce1="inet 10.1.4.'''103''' netmask 255.255.255.0"
ifconfig_bce0="inet 69.55.'''229.7''' netmask 255.255.255.0"
#ifconfig_bce0_alias0="inet 69.55.2xx.xx netmask 255.255.255.0"

fsck_y_enable="YES"
background_fsck="NO"
#rc_mfi_raid_tty_log="YES"
#zfs_enable="YES"</pre>

Modify IPs, hostname, gateway for this box.

=== make sure sysctls are set and preserved upon boot ===

<pre>echo "kern.consmute=0\
kern.ipc.shm_use_phys=1\
kern.ipc.shmall=131070\
kern.ipc.shmmax=134217728\
net.inet.tcp.syncookies=0\
kern.maxfiles=32768\
kern.fallback_elf_brand=3\
kern.maxprocperuid=4000\
security.jail.sysvipc_allowed=1\
security.jail.allow_raw_sockets=1\
security.jail.socket_unixiproute_only=1\
security.jail.chflags_allowed=0\
dev.amr.0.allow_volume_configure=1\
compat.linux.osrelease=2.6.12\
vm.pmap.shpgperproc=500\
security.bsd.unprivileged_read_msgbuf=0\
kern.maxvnodes=400000" >> /etc/sysctl.conf</pre>

Tuning note: watch vfs.numvnodes while the server is live to get guidance on where to set maxvnodes

=== mount procfs ===
echo "proc /proc procfs rw 0 0" >> /etc/fstab
echo "linprocfs /usr/compat/linux/proc linprocfs rw 0 0" >> /etc/fstab
echo "linsysfs /usr/compat/linux/sys linsysfs rw 0 0" >> /etc/fstab
mkdir -p /usr/compat/linux/proc
mkdir -p /usr/compat/linux/sys

=== enable noatime option ===
data1 and data2 should look something like (add ',noatime' after 'rw'):
/dev/mfid0s1g /mnt/data1 ufs rw,noatime 2 2
/dev/mfid1s1d /mnt/data2 ufs rw,noatime 2 2

=== reboot. Confirm new kernel is loaded, devfs in place ===

uname -a

Check devfs rules
devfs rule showsets
devfs rule -s 3 show

Should see:
<pre># devfs rule showsets
1
2
3
4

# devfs rule -s 3 show
100 include 1
207 path pts* unhide
217 path fd unhide
218 path fd/* unhide
300 path ttyp* unhide
301 path ttyq* unhide
302 path ttyr* unhide
303 path ttys* unhide
304 path ttyP* unhide
305 path ttyQ* unhide
306 path ttyR* unhide
307 path ttyS* unhide
400 path null unhide
500 path zero unhide
600 path random unhide
610 path urandom unhide
700 path mem unhide
710 path kmem unhide
810 path mdctl unhide
900 path stdin unhide
910 path stdout unhide
920 path stderr unhide</pre>

=== update ports ===

<pre>cd /usr/ports
echo "*default host=cvsup4.FreeBSD.org\
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_8_3\
*default delete use-rel-suffix\
*default compress\
ports-all tag=." > sup

cvsup sup; mail -s 'cvs sup ports done' support@johncompanies.com < /dev/null</pre>
~24mins

=== Install raid mgmt tool ===

==== Perc5/i, 6/i ====
Pull over cli from previous system (jail9)
scp /usr/local/sbin/mega* root@10.1.4.109:/usr/local/sbin/
scp /usr/local/libexec/MegaCli root@10.1.4.109:/usr/local/libexec/MegaCli
These are linux-based tools. This will require linux base which we would normally install from ports, but since it's failing lately, we just pull in the compat libraries from another system:

rsync -aSHv --exclude=proc --exclude=sys 10.1.4.109:/usr/compat/linux/ /usr/compat/linux/

Test:
rehash; megacli ldinfo lall a0

DEPRECATED: Assuming it worked, here's how we ''used to'' install linux_base:
----
cd /usr/ports/emulators/linux_base-fc4
make install clean
(didnt succeed due to libtool requirement)

cd /usr/ports/distfiles
fetch http://www.lsi.com/DistributionSystem/AssetDocument/support/downloads/megaraid/miscellaneous/linux/2.00.15_Linux_MegaCLI.zip
cd /usr/ports/sysutils/linux-megacli
make install clean
(also failed due to libtool)
----

==== 2850 PERC 4e/Di- no linux ====
<pre>cd /usr/ports/distfiles/
fetch http://backup01.best-hosting.ru/pub/FreeBSD/ports/distfiles/dr_freebsd_1.51.zip
cd /usr/ports/sysutils/megarc
make install clean
megarc -dispCfg -a0</pre>

=== install rsync from ports ===
cd /usr/ports/net/rsync
make install clean

choose default options

=== configure inetd to respond to mrtg load queries ===
echo "load stream tcp nowait user /usr/local/jail/bin/load.pl load.pl" >> /etc/inetd.conf
echo "load 12384/tcp" >> /etc/services

=== install perl ===
cd /usr/ports/lang/perl5.12
make install clean

choose defaults

=== install bb client ===

Compiling from source on AMD64 will not work. So, we use a linux-compiled version and rely on linux compat. Per [[#Perc5.2Fi.2C_6.2Fi|above]], linux compat won't install on 8.x - libtool 2.4 need. So, instead we copy(ed) over linux:
rsync -aSHv --exclude=proc --exclude=sys 10.1.4.108:/usr/compat/linux/ /usr/compat/linux/

adduser

Output/response:

<pre>Username: bb
Full name: bb
Uid (Leave empty for default): 1984
Login group [bb]:
Login group is bb. Invite bb into other groups? []:
Login class [default]:
Shell (sh csh tcsh nologin) [sh]:
Home directory [/home/bb]:
Use password-based authentication? [yes]:
Use an empty password? (yes/no) [no]:
Use a random password? (yes/no) [no]: yes
Lock out the account after creation? [no]:
Username : bb
Password : <random>
Full Name : bb
Uid : 1984
Class :
Groups : bb
Home : /home/bb
Shell : /bin/sh
Locked : no
OK? (yes/no): yes</pre>

cd /usr/home/bb
scp backup2:/mnt/data4/build/bb/bb-freebsd_linuxcompat.tgz .
tar xzf bb-freebsd_linuxcompat.tgz

edit /home/bb/bbc1.9e-btf/etc/bb-hosts with something like:

echo "10.1.4.5 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh \
10.1.4.'''103 jail3'''.johncompanies.com # ssh" > /home/bb/bbc1.9e-btf/etc/bb-hosts

Edit for machine name and private IP.

if this machine is at i2b:
echo "69.55.230.2 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh \
69.55.229.7 jail3.johncompanies.com # ssh" > /home/bb/bbc1.9e-btf/etc/bb-hosts

<pre>vi /home/bb/bbc1.9e-btf/ext/openfiles

MACHINE="jail3,johncompanies,com" # HAS TO BE IN A,B,C FORM</pre>
Edit for machine name.

<pre>cd /usr/home/bb/bbc1.9e-btf/etc
./bbchkcfg.sh
(y to questions)
./bbchkhosts.sh
(ignore ssh errors)
cd ../..
chown -R bb .
su bb
cd
cd bbc1.9e-btf
./runbb.sh start
more BBOUT
(look for errors)
exit</pre>

Put in script to start bb @ boot:
echo 'su - bb -c "cd /home/bb/bbc1.9e-btf; ./runbb.sh start"' > /usr/local/etc/rc.d/bb.sh
chmod +x /usr/local/etc/rc.d/bb.sh

=== remove reserve space ===
<pre>cd
umount /mnt/data1
umount /mnt/data2
tunefs -m 0 /mnt/data1
tunefs -m 0 /mnt/data2
mount -a</pre>

=== setup rdate ===
cd /usr/ports/sysutils/rdate
make install clean

crontab -e
0 0 * * * /usr/local/sbin/rdate -s utcnist.colorado.edu

/usr/local/sbin/rdate -s utcnist.colorado.edu

We used to use ntpd, however it listens on jail IPs which is a security risk. So we stopped. Here's the old instructions:
install new ntp from ports
/usr/ports/net/ntp
make install clean

<pre>echo "server 10.1.4.5" > /etc/ntp.conf
/usr/sbin/ntpd -p /var/run/ntpd.pid
sleep 2; ntpq -p</pre>(confirm it’s able to reach our time server)

=== make a data partition ===

IF you didn't format the data partition during sysinstall:

Create a g partition on 2nd mirror – bsdlabel no longer works (below shows d partition made with sysinstall):

<pre>jail8 /usr/home/bb# gpart show
=> 63 285474735 mfid0 MBR (136G)
63 285458922 1 freebsd [active] (136G)
285458985 15813 - free - (7.7M)

=> 0 285458922 mfid0s1 BSD (136G)
0 524288 1 freebsd-ufs (256M)
524288 12582912 2 freebsd-swap (6.0G)
13107200 524288 4 freebsd-ufs (256M)
13631488 524288 5 freebsd-ufs (256M)
14155776 8388608 6 freebsd-ufs (4.0G)
22544384 262914538 7 freebsd-ufs (125G)

=> 63 584843175 mfid1 MBR (279G)
63 584830197 1 freebsd [active] (279G)
584830260 12978 - free - (6.3M)

=> 0 584830197 mfid1s1 BSD (279G)
0 16777216 2 freebsd-swap (8.0G)
16777216 568052981 4 freebsd-ufs (271G)

jail8 /usr/home/bb# gpart show mfid1s1
=> 0 584830197 mfid1s1 BSD (279G)
0 16777216 2 freebsd-swap (8.0G)
16777216 568052981 4 freebsd-ufs (271G)

# gpart list mfid1s1

Geom name: mfid1s1
fwheads: 255
fwsectors: 63
last: 584830196
first: 0
entries: 8
scheme: BSD
Providers:
1. Name: mfid1s1b
Mediasize: 8589934592 (8.0G)
Sectorsize: 512
Mode: r1w1e0
rawtype: 1
length: 8589934592
offset: 0
type: freebsd-swap
index: 2
end: 16777215
start: 0
2. Name: mfid1s1d
Mediasize: 290843126272 (271G)
Sectorsize: 512
Mode: r0w0e0
rawtype: 7
length: 290843126272
offset: 8589934592
type: freebsd-ufs
index: 4
end: 584830196
start: 16777216
Consumers:
1. Name: mfid1s1
Mediasize: 299433060864 (279G)
Sectorsize: 512
Mode: r1w1e1

# gpart delete -i 4 mfid1s1
mfid1s1d deleted
jail8 /usr/home/bb# gpart list mfid1s1
Geom name: mfid1s1
fwheads: 255
fwsectors: 63
last: 584830196
first: 0
entries: 8
scheme: BSD
Providers:
1. Name: mfid1s1b
Mediasize: 8589934592 (8.0G)
Sectorsize: 512
Mode: r1w1e0
rawtype: 1
length: 8589934592
offset: 0
type: freebsd-swap
index: 2
end: 16777215
start: 0
Consumers:
1. Name: mfid1s1
Mediasize: 299433060864 (279G)
Sectorsize: 512
Mode: r1w1e1

# gpart show mfid1s1
=> 0 584830197 mfid1s1 BSD (279G)
0 16777216 2 freebsd-swap (8.0G)
16777216 568052981 - free - (271G)

# gpart add -t freebsd-ufs -i 7 mfid1s1
mfid1s1g added

# gpart show mfid1s1
=> 0 584830197 mfid1s1 BSD (279G)
0 16777216 2 freebsd-swap (8.0G)
16777216 568052981 7 freebsd-ufs (271G)</pre>

Here's how we USED to do it with bsdlabel:

bsdlabel -e /dev/mfid0s1

given:
<pre># /dev/aacd0s1:
8 partitions:
# size offset fstype [fsize bsize bps/cpg]
a: 262144 0 4.2BSD 2048 16384 16392
b: 4194304 262144 swap
c: 143363997 0 unused 0 0 # "raw" part, don't edit
d: 524288 4456448 4.2BSD 2048 16384 32776
e: 524288 4980736 4.2BSD 2048 16384 32776
f: 6291456 5505024 4.2BSD 2048 16384 28552</pre>

new offset = 6291456 + 5505024 = 11796480<br>
new size is size for 'c' partition minus the new start from above<br>
143363997 - 11796480 = 131567517

So:<br>
g: 131567517 11796480 unused 0 0

=== create the jail template ===

cd /usr/ports/sysutils/jailutils
make install clean

Create an md device to hold the jail:
<pre>touch /mnt/data1/jail-template20g
mdconfig -a -t vnode -s 20g -f /mnt/data1/jail-template20g -u 0
newfs -O 1 /dev/md0
mkdir /mnt/data1/jail-DIR
mount /dev/md0 /mnt/data1/jail-DIR</pre>

Build world into the jail:
<pre>cd /usr/src
make world DESTDIR=/mnt/data1/jail-DIR; taskdone</pre>
~1hr

Make /etc into the jail, mount dev, copy in jkill:
<pre>cd etc
make distribution DESTDIR=/mnt/data1/jail-DIR
mount -t devfs devfs /mnt/data1/jail-DIR/dev
devfs -m /mnt/data1/jail-DIR/dev rule -s 3 applyset
cd /mnt/data1/jail-DIR
ln -sf dev/null kernel
cp /usr/local/sbin/jkill /mnt/data1/jail-DIR/sbin</pre>

Enter into jail to do configuration:
jail /mnt/data1/jail-DIR testhostname 192.168.11.100 /bin/sh
csh

Create fstab:
<pre>touch /etc/fstab
echo 'network_interfaces=""\
hostname="newsystem"\
kern_securelevel_enable="NO"\
sendmail_enable="YES"\
sshd_enable="YES"' > /etc/rc.conf

echo "nameserver 69.55.225.225\
nameserver 69.55.230.3" >> /etc/resolv.conf</pre>

Edit crontab:
vi /etc/crontab
remove the adjkerntz lines
comment out periodic’s and put this line above them:
# IF YOU UNCOMMENT THESE, PLEASE ADJUST THEIR RUN TIME

rm -rf /etc/periodic/daily/400.status-disks

check and remove any crap in /tmp

vi /etc/periodic/security/100.chksetuid
replace: <tt>MP=`mount -t ufs | grep -v " nosuid" | awk '{ print $3 }' | sort`</tt><br>
with: <tt>MP='/'</tt> (use single quotes)

mkdir -p /usr/compat/linux/dev

Add account for user. Output/response:
<pre>adduser

Username: user
Full name: user
Uid (Leave empty for default):
Login group [user]:
Login group is user. Invite user into other groups? []: wheel
Login class [default]:
Shell (sh csh tcsh nologin) [sh]:
Home directory [/home/user]:
Home directory permissions (Leave empty for default):
Use password-based authentication? [yes]:
Use an empty password? (yes/no) [no]:
Use a random password? (yes/no) [no]: y
Lock out the account after creation? [no]:
Username : user
Password : <random>
Full Name : user
Uid : 1001
Class :
Groups : user
Home : /home/user
Home Mode :
Shell : /bin/sh
Locked : no
OK? (yes/no): y
adduser: INFO: Successfully added (user) to the user database.
adduser: INFO: Password for (user) is: 901gmYjO
Add another user? (yes/no): n
Goodbye!</pre>

set TERM:
vi /usr/home/user/.profile
TERM=vt100; export TERM

Set time zone to PT:
tzsetup

Reload aliases:
newaliases

Replace reboot/halt:
rm /sbin/halt /sbin/reboot
ln /sbin/jkill /sbin/halt
ln /sbin/jkill /sbin/reboot

Redirect console output, comment out console and move to /var/log/messages:
vi /etc/syslog.conf
#*.err;kern.warning;auth.notice;mail.crit /dev/console
*.err;kern.warning;auth.notice;mail.crit /var/log/messages

exit
exit

Lib32 compat library:
cd libexec
chflags noschg ld-elf32.so.1
chflags noschg ld-elf.so.1
mv ld-elf32.so.1 ld-elf32.so.1-orig
ln ld-elf.so.1 ld-elf32.so.1
chflags schg ld-elf.so.1
chflags schg ld-elf32.so.1

Replace traceroute:
mv /mnt/data1/jail-DIR/usr/sbin/traceroute /mnt/data1/jail-DIR/usr/sbin/_traceroute
echo '#\!/bin/sh\
/usr/sbin/_traceroute -i bce0 $1' >> /mnt/data1/jail-DIR/usr/sbin/traceroute
chmod +x /mnt/data1/jail-DIR/usr/sbin/traceroute

Modify 'bce0' to reflect whichever nic is public on this hardware.

Clean out ports before copying into the jail (~30mins):
cd /usr/ports
make -DNOCLEANDEPENDS clean
rm -fr /usr/ports/distfiles/*
cp -r /usr/ports /mnt/data1/jail-DIR/usr

rm /mnt/data1/jail-DIR/root/.history

Umount the jail and dump it:
<pre>cd /
umount /mnt/data1/jail-DIR/dev
dump -0a -f /usr/local/jail/template/template /dev/md0
umount /dev/md0
rmdir /mnt/data1/jail-DIR
mdconfig -d -u 0</pre>

=== setup backups ===
<pre>echo '#\!/bin/sh\
backupdir=/data/jail3\
server=backup1\
\
## ENTRY /etc\
## ENTRY /usr/local/etc\
## ENTRY /usr/local/jail\
## ENTRY /root/logs' > /usr/local/jail/bin/backup.config</pre>

Edit to reflect backup server and jail hostname

On backup server, setup backup dirs:
ssh backup1 mkdir -p /data/jail3/0

On backup server, setup backup dirs:
backup1# vi /usr/local/sbin/snapshot_rotate

Edit /usr/local/jail/bin/backup.xxx to use the right drives and copy to /usr/local/jail/bin/backup
vi /usr/local/jail/bin/backup.md
adjust df so it includes all relevant drives, currently <tt>df > /etc/df.bak</tt> is fine. Also, make sure the binary/script source is pulling from the right dir on backup2:
/usr/local/bin/rsync -a backup2:/mnt/data4/bin/freebsd8.x/ /usr/local/jail/bin/

cp /usr/local/jail/bin/backup.md /usr/local/jail/bin/backup

=== create /root/logs ===
mkdir /root/logs

=== edit sshd_config ===
vi /etc/ssh/sshd_config
ListenAddress 69.55.229.7
ListenAddress 10.1.4.103

'''''Adjust to pub/private IP for jail.'''
''
Restart sshd:
kill -1 `cat /var/run/sshd.pid`

=== add crontab entries ===
<pre>crontab -e
5 0 * * * /usr/local/jail/bin/backup
1 0 1 * * /usr/local/jail/bin/ipfwreset
0 18 * * * /usr/local/jail/bin/ipfwbackup
4,9,14,19,24,29,34,39,44,49,55,59 * * * * /usr/local/jail/bin/trafstats
0 0,6,12,18 * * * /usr/local/jail/bin/sync_jail_names</pre>

On 2950:
*/5 * * * * /usr/local/jail/bin/perc5iraidchk

On 3ware-based servers:
0 0 * * * /usr/local/jail/bin/3wraidchk

=== Reboot notify script ===
ln -s /usr/local/jail/bin/notify.sh /usr/local/etc/rc.d/notify.sh

=== patch jail against starting jails with rtprio ===

<pre>mv /usr/sbin/jail /usr/sbin/jail_
echo '#\!/bin/sh\
/usr/sbin/rtprio -t /usr/sbin/jail_ $*' > /usr/sbin/jail
chmod +x /usr/sbin/jail</pre>

=== make sure mail works ===
If there are map errors:
cd /etc/mail; make maps

=== recover space on /usr (optional) ===

rm -fr /usr/obj

=== wrapper for jps ===

mv /usr/local/sbin/jps /usr/local/sbin/jps_

=== wrapper for jls ===

mv /usr/sbin/jls /usr/sbin/jls_

=== wrapper for jexec ===

mv /usr/sbin/jexec /usr/sbin/jexec_

=== install jtop ===

cd /usr/ports/sysutils/jtop
make install clean

=== block jails from reaching private net ===
echo 'ipfw add 1 deny ip from 69.55.224.0/20 to 10.1.4.0/24' > /usr/local/etc/rc.d/ipfw.sh
chmod 0700 /usr/local/etc/rc.d/ipfw.sh

== add to management infrastructure ==

=== mail ===

==== add to management db ====

tables: jc.ref_machines and jc.ref_templates

on jail run:
uname -r

Which shows something like:
8.3-RELEASE-p2

Insert into db:

<pre>insert into ref_machines values (null,'jail3','mx3.johncompanies.com',0,'f8');
select machine_id from ref_machines where host='jail3';
+------------+
| machine_id |
+------------+
| 35 |
+------------+
insert into ref_templates values ('','8.3-RELEASE-jc2',35,'FreeBSD 8.3',1);</pre>

==== add to bb server ====
vi /usr/home/bb/bbsrc/bb1.9i-btf/etc/bb-hosts
10.1.4.109 jail9.johncompanies.com # ssh

In the case of an i2b server, use real ip:
69.55.229.7 jail3.johncompanies.com # ssh

<pre>su bb
cd
bbsrc/bb/runbb.sh restart ; exit</pre>

==== Update backupgraph ====
vi /usr/local/www/mgmt/cgi/backupgraph.pl
(add hostname)

==== Update load mrtg ====
vi /usr/local/www/mgmt/mrtg/mrtg1.cfg
(add new entry to file following existing format)

=== ns1c ===
fwd and reverse lookups:
vr johncompanies.com

vi internal.johncompanies.com
rndc reload johncompanies.com IN private

ptr 69.55.227.x

=== wiki ===
add to server/cabinet map

=== firewall ===
add an outside blocking rule to the firewall, so this machine can only be reached from inside the firewall. Follow example already in firewall jail17 is:

00117 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 or 69.55.238.150 } to 69.55.228.200
00117 deny ip from any to 69.55.228.200

jail19 would be 00119...
ipfw add 00109 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 } to 69.55.232.3
ipfw add 00109 deny ip from any to 69.55.232.3

=== select some customers for castle probe map ===

= FreeBSD 4.11 =

<pre>
4.11
Last updated 2006-1-26

All time extimates below assume disks aren’t scrubbing

1. make sure bios is setup for bios console redirect
Supermicro:
Console redirection:
Com port addr: on-board COM A
Baud: 38400
Console type: vt100
Flow control: none
Console connection: direct
Continue cr after post: off

2450:
Make sure running bios A09
Console Redirection: VT100/VT220

2. assuming mirrors (or at least disks) created (if not, refer to this), boot to disk 1 of 4.11
skip kernel config (enter)
custom install
partition
move cursor to aacd0, hit space (takes you to partition map screen)
a for entire disk
q to quit and save
standard mbr (no boot manager)
space to unselect aacd0
cursor over aacd1
space
a for entire disk
q to quit and save
none (leave untouched)

cursor over aacd0
space
(takes you into part. Screen again) q to exit
none
Make sure both are checked and tab to ok

label
Make sure aacd0 is highlighted
a to start with defaults

john likes:
/1g
/var 256m

Glenn likes:
/128
/var 256
/usr 3g
/tmp 256

We do:
delete everything
/ 128M
swap 4G
/var 256M
/tmp 256M
/usr 3G
/mnt/data1 remaining space

Make sure to toggle S for soft updates on all (should look like UFS+S Y under the Newfs column)
Set the noatime option

move cursor to aacd1
swap 4G
/mnt/data2 remaining space
Set the noatime option

q to save and exit

distributions
developer
yes to install ports
exit
media
cd
commit
yes
(2450: 16mins, supermicro: 11mins)
yes to "visit general config"
Set root pwd
Add user ‘user’ member group is wheel, set password
Set tz
Networking->interfaces->Fxp0
No IPV6
dhcp=yes
Set hostname & domain

Startup services:
Disble usbd

exit...
exit install
yes
take the cd out and let the machine reboot

3. put some temp settings in /etc/rc.conf:
usbd_enable="NO"
sendmail_enable="NO"

4. reboot

5. double check the date/time

6. edit /etc/make.conf (only add the console speed line if this is a supermicro capable of outputting BIOS redirect at that speed – 2450’s can only do 9600)
cat >> /etc/make.conf
WITHOUT_X11=yes
KERNCONF=jail18
BOOT_COMCONSOLE_SPEED=38400

7. install cvsup
cd /usr/ports/net/cvsup-without-gui
make install clean (stay close for gettext options, 2450: 21mins, supermicro: 14mins)
rehash

8. get latest sources for this release:
cd /usr/src
cat > sup
*default host=cvsup4.freebsd.org
*default base=/usr
*default prefix=/usr
*default release=cvs tag=RELENG_4_11
*default delete use-rel-suffix
*default compress
src-all

cvsup sup (2450, 4.10: 13mins, supermicro, 4.11: 11mins)

9. populate hosts
cat >> /etc/hosts
10.1.4.3 backup2

10. put key in authorized_keys on backup2
ssh-keygen -t dsa -b 1024 (default location, leave password blank)
scp /root/.ssh/id_dsa.pub user@backup2:/tmp/jail18pub
on backup2:
cat /tmp/jail18pub >> /root/.ssh/authorized_keys

confirm that you can ssh to backup2 without getting a login prompt

11. configure new kernel. Get config from similar machine or there may be a master copy somewhere under /mnt/data4/build (name the kernel config the same as the jail, ex jail4):
cd /usr/src/sys/i386/conf
scp backup2:/mnt/data4/build/freebsd/kern_config-4.11 ./jail18

edit the kernel config and change ident to be the name of the jail:
ident jail4

IMPORTANT CUSTOMIZATION:
for machines with >4G RAM, add to the config:
options PAE

for supermicro mobo’s with broadcom nics, add to the config:
device bge # Broadcom BCM570x (``Tigon III'')

for machines where lots of postgres might be running, change SHMMAXPGS:
options SHMMAXPGS=40960

edit /sys/conf/newvers.sh to add –jc2 to the end of the BRANCH string (RELEASE-p9-jc2)

12. bring over patches from backup2
The various patches are in /mnt/data4/build/freebsd/patches on backup2. There are dirs for each version. Not all dirs are populated, but patches for later versions work on older ones unless there is a new patch in the older dir.
cd /usr/src
scp backup2:"/mnt/data4/build/freebsd/patches/4.11/*" .
jail_proc_patch 100% |*************************************************| 2593 00:00
restore-patch 100% |*************************************************| 3295 00:00
411ps-jail-patch 100% |*************************************************| 2602 00:00
jail_rtprio_patch 100% |*************************************************| 301 00:00
udp-patch 100% |*************************************************| 594 00:00

Apply patches:
patch < 411ps-jail-patch
patch < jail_proc_patch
patch < restore-patch
patch < jail_rtprio_patch
patch < udp-patch

13. build, install kernel and world
make buildworld buildkernel installkernel (2450: 48min, supermicro: 20mins)
(Any compile errors can be looked up in /usr/include/sys/signal.h, other errors, do a rm -R /usr/obj/*)
make installworld (2450: 2min, supermicro: 1mins)
mergemaster -i
(answer no to most of it)

14. reboot. Confirm new kernel is loaded (uname -a)

15. update ports:
cd /usr/ports
cat > sup
*default host=cvsup4.FreeBSD.org
*default base=/usr
*default prefix=/usr
*default release=cvs tag=RELENG_4
*default delete use-rel-suffix
*default compress
ports-all tag=.

cvsup sup (2450: 26mins, supermicro: 26mins)

16. add console="comconsole" to /boot/loader.conf
cat >> /boot/loader.conf
console="comconsole"

17. edit /etc/ttys and turn off all ttyv's except 0 and 1
also turn on ttyd0, change type to vt100:
ttyd0 "/usr/libexec/getty std.9600" vt100 on secure

kill -1 1

on console:
vi /etc/remote
(rename port to jail4 depending on where and which digi plugged into)
test serial console

18. install linux_base:
cd /usr/ports/emulators/linux_base
make install clean (2450: 7min, supermicro: 2mins)

reibuild rpmdb cause we had probs installing aacapps
cd /compat/linux/bin
./rpm --initdb
./rpm --rebuilddb

19. install aacapps-4.1-0.i386.rpm
scp backup2:/mnt/data4/build/freebsd/aacapps-4.1-0.i386.rpm /tmp/.
/compat/linux/bin/rpm -ivh /tmp/aacapps-4.1-0.i386.rpm
cd /dev
sh MAKEDEV aac0

test out /compat/linux/usr/sbin/aaccli

20. edit root's path and login script:
vi /root/.cshrc
add to path: /usr/local/jail/bin /compat/linux/usr/sbin

Change alias entries (add G):
alias la ls -aG
alias lf ls -FAG
alias ll ls -lAG
alias ls ls -AG
alias mbm mb mount
alias mbu mb umount

and alter the prompt, set the following:
set prompt = "`/bin/hostname -s` %/# "

at the bottom of the file add:
set sshtty=`who am i|awk '{print $2}'`
/usr/sbin/rtprio 3 -`ps auxwJ | grep $sshtty | awk '{print $2}'`

set shortty=`who am i | awk '{print $2}' | sed -E 's/.*(..)$/\1/'`
foreach x (`ps cauxJ | grep sh | grep $shortty | awk '{print $2}'`)
/usr/sbin/rtprio 2 -$x
end

21. install rsync from ports
cd /usr/ports/net/rsync
make install clean

22. create & populate binaries/scripts dirs
mkdir -p /usr/local/jail/bin
mkdir -p /usr/local/jail/rc.d
scp backup2:"/mnt/data4/bin/freebsd/*" /usr/local/jail/bin
cd /usr/local/jail/rc.d/
touch quad1
touch quad2
touch quad3
touch quad4
touch safe1
touch safe2
touch safe3
touch safe4
chmod +x *
cd /usr/local/jail/bin
ln -s /usr/local/jail/rc.d/quad1 quad1
ln -s /usr/local/jail/rc.d/quad2 quad2
ln -s /usr/local/jail/rc.d/quad3 quad3
ln -s /usr/local/jail/rc.d/quad4 quad4
ln -s /usr/local/jail/rc.d/safe1 safe1
ln -s /usr/local/jail/rc.d/safe2 safe2
ln -s /usr/local/jail/rc.d/safe3 safe3
ln -s /usr/local/jail/rc.d/safe4 safe4
rehash

23. configure inetd to respond to mrtg load queries
cat >> /etc/inetd.conf
load stream tcp nowait user /usr/local/jail/bin/load.pl load.pl

cat >> /etc/services
load 12384/tcp

kill -HUP `cat /var/run/inetd.pid`

24. configure load mrtg, on mail
vi /usr/local/etc/mrtg/mrtg1.cfg
(add new entry to file following existing format)

25. configure bb on mail:
vi /usr/home/bb/bbsrc/bb1.9e-btf/etc/bb-hosts
10.1.4.104 jail4.johncompanies.com # ssh

su bb
cd /usr/home/bb/bbsrc/bb1.9e-btf
./runbb.sh stop
./runbb.sh start
exit

26. install bb client
adduser -group 1984 -shell /bin/csh -uid 1984 bb
cd /usr/home/bb
scp backup2:/mnt/data4/build/bb/bb-freebsd.tar .
tar xvf bb-freebsd.tar

cat > /home/bb/bbc1.9e-btf/etc/bb-hosts with something like:
10.1.4.105 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh
10.1.4.118 jail18.johncompanies.com # ssh

vi /home/bb/bbc1.9e-btf/ext/openfiles and change:
MACHINE="jail18,johncompanies,com" # HAS TO BE IN A,B,C FORM

cd /usr/home/bb/bbc1.9e-btf/etc
./bbchkcfg.sh
./bbchkhosts.sh (ignore ssh errors)
cd ../..
chown -R bb .
su bb
cd bbc1.9e-btf/
./runbb.sh start
more BBOUT (look for errors)
exit

cat > /usr/local/etc/rc.d/bb.sh
su - bb -c "cd /home/bb/bbc1.9e-btf; ./runbb.sh start"

chmod +x /usr/local/etc/rc.d/bb.sh

27. remove reserve space, enable softupdates (probably already set, so not necessary)
cd
umount /mnt/data1
umount /mnt/data2
tunefs -m 0 /mnt/data1
tunefs -m 0 /mnt/data2

mount -a

28. configure ntp
cat > /etc/ntp.conf
server 10.1.4.105

/usr/sbin/ntpd -p /var/run/ntpd.pid
ntpq -p
(confirm it’s able to reach our time server)

29. mrtg switch graphs

31. fwd and reverse lookups on ns1c
vi johncompanies.com
rr johncompanies.com
vi internal.johncompanies.com
rndc reload johncompanies.com IN private
(edit the PTR too)

32. create all /dev/vn and /dev/pty files in /dev
cat > /tmp/runme.sh
#!/bin/sh
cd /dev
for i in 1 2 3 4 5 6 7 ; do sh MAKEDEV pty$i ; done
for i in 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 ; do sh MAKEDEV vn$i ; done

NOTE: there are only 4 lines in the above output - however the lines may be wrapped on your terminal - make sure that there are only four lines in the script you make.

chmod +x /tmp/runme.sh
/tmp/runme.sh

rm /tmp/runme.sh

ls /dev/vn*|wc -l (make sure the output is a high number – larger than 128)
ls /dev/pty*|wc -l (make sure the output is exactly 256)

33. populate /etc/rc.conf with IPs and NFS settings
vi /etc/rc.conf

kern_securelevel_enable="NO"
portmap_enable="NO"
sendmail_enable="NO"
usbd_enable="NO"

xntpd_enable="YES"
nfs_client_enable="YES"
nfs_reserved_port_only="YES"
sshd_enable="YES"
inetd_enable="YES"
inetd_flags="-wW -a 10.1.4.104"

ifconfig_xl0="inet 10.1.4.104 netmask 255.255.255.0"
ifconfig_fxp0="inet 69.55.228.101 netmask 255.255.255.0"
defaultrouter="69.55.228.1"
ifconfig_fxp0_alias0="inet 69.55.2xx.xx netmask 255.255.255.0"

static_routes="t1 office"
route_t1="-net 10.1.5 10.1.4.2"
route_office="-net 10.1.6 10.1.4.2"

34. make sure sysctls are set and preserved after reboot
cat >> /etc/sysctl.conf
kern.consmute=0
jail.sysvipc_allowed=1
kern.ipc.shm_use_phys=1
kern.ipc.shmall=65535
kern.ipc.shmmax=134217728
net.inet.tcp.syncookies=0
kern.maxfiles=32768
kern.fallback_elf_brand=3
kern.maxprocperuid=4000
jail.max_procs_per_jail: 1026

35. reboot

36. create the jail template
vnconfig -T -S 1g -Z -s labels -c /dev/vn1 /mnt/data1/jail
disklabel -r -w vn1 auto
newfs /dev/vn1c
cd /usr/src
mkdir -p /mnt/data1/jail-DIR
mount /dev/vn1c /mnt/data1/jail-DIR
make world DESTDIR=/mnt/data1/jail-DIR (2450: 45mins, supermicro: 19mins)
cd etc
make distribution DESTDIR=/mnt/data1/jail-DIR -DNO_MAKEDEV_RUN
cd /mnt/data1/jail-DIR/dev
sh MAKEDEV jail
cd /mnt/data1/jail-DIR
ln -sf dev/null kernel

jail /mnt/data1/jail-DIR testhostname 192.168.11.100 /bin/sh
csh
touch /etc/fstab
cat > /etc/rc.conf
portmap_enable="NO"
network_interfaces=""
hostname="newsystem"
kern_securelevel_enable="NO"
sendmail_enable="YES"
sshd_enable="YES"

cat >> /etc/resolv.conf
nameserver 69.55.225.225
nameserver 69.55.230.3

vi /etc/crontab
remove the adjkerntz lines
comment out periodic’s and put this line above them:
# DO NOT UNCOMMENT THESE

rm -rf /etc/periodic/daily/400.status-disks

mv /bin/df /bin/df_

cat > /bin/df
#!/bin/sh
/bin/df_ $* .

chmod +x /bin/df

cat > /sbin/mount
echo `df | tail -1 | awk '{print $1 " on " $6 " (ufs, local)"}'`

check /tmp for crap

vi /etc/periodic/security/100.chksetuid
replace: MP=`mount -t ufs | grep -v " nosuid" | awk '{ print $3 }' | sort`
with: MP='/'

mkdir -p /usr/compat/linux/dev

adduser (Add account for user)

put user in wheel group
vi /etc/group

vi /usr/home/user/.profile (and add to the file):
TERM=vt100; export TERM

cd /etc
vipw -d .
root:$1$krszPxhk$xkCepSnz3mIikT3vCtJCt0:0:0::0:0:Charlie &:/root:/bin/csh
user:$1$Mx9p5Npk$QdMU6c8YQqp2FW2M3irEh/:1001:1001::0:0:User &:/home/user:/bin/sh

tzsetup

newaliases

chflags schg /dev/*mem

cd /dev
rm console
ln -s null console

exit
exit

cd /usr/ports
make -DNOCLEANDEPENDS clean (2450: 47mins , supermicro: 17mins)
rm -fr /usr/ports/distfiles/*
cp -r /usr/ports /mnt/data1/jail-DIR/usr

cd /mnt/data1/jail-DIR/dev
rm kmem
mknod kmem c 2 1 root:kmem
chmod 640 kmem
rm mem
mknod mem c 2 0 root:kmem
chmod 640 mem

rm /mnt/data1/jail-DIR/root/.history

sh
for i in 1 2 3 4 5 6 7 ; do sh MAKEDEV pty$i ; done
exit

cd /mnt/data1/jail-DIR/usr/compat/linux/dev
mknod null c 2 2
mknod random c 2 3

cd
mkdir /usr/local/jail/template/
dump -0a -f /usr/local/jail/template/411template /dev/vn1

umount /dev/vn1c
vnconfig -u /dev/vn1

rm /mnt/data1/jail
rm -fr /mnt/data1/jail-DIR

37. setup backups
cat > /usr/local/jail/bin/backup.config
#!/bin/sh
backupdir=/mnt/data2/jail18_rsync

## ENTRY /etc
## ENTRY /usr/local/etc
## ENTRY /usr/local/jail
## ENTRY /root/logs

on backup2:
setup backup dirs:
mkdir -p /mnt/data2/jail18_rsync/0_today
mkdir -p /mnt/data2/jail18_rsync/1_yesterday
mkdir -p /mnt/data2/jail18_rsync/2_two_day

add the system to /mnt/data1/bin/rsync_houseclean

on mail:
vi /usr/local/www/mgmt/cgi/backupgraph.pl
(add hostname)

38. mkdir /root/logs

39. edit sshd_config for security
vi /etc/ssh/sshd_config
ListenAddress 69.55.228.101
ListenAddress 10.1.4.104

kill -1 `cat /var/run/sshd.pid`

40. add crontab entries
crontab -e
5 0 * * * /usr/local/jail/bin/backup
1 0 1 * * /usr/local/jail/bin/ipfwreset
0 18 * * * /usr/local/jail/bin/ipfwbackup
4,9,14,19,24,29,34,39,44,49,55,59 * * * * /usr/local/jail/bin/trafstats
55 10,23 * * * /usr/local/jail/bin/trafficwatch.pl

41. Reboot notify script
cat > /usr/local/etc/rc.d/notify.sh
echo "`/bin/hostname` rebooted" | /usr/bin/mail reboot@johncompanies.com

chmod +x /usr/local/etc/rc.d/notify.sh

42. copy jailmake from prev system
scp user@10.1.4.118:/usr/local/jail/bin/jailmake /usr/local/jail/bin
rehash
NOTE: remove df altering code from jailmake since we put the correct df in the template, and make sure path to template file is right

43. add to templates via mgmt system

44. add to server/cabinet map. On mail:
vi /usr/local/www/mgmt/html/cabinetmap.html

45. add an outside blocking rule to the firewall, so this machine can only be reached from inside the firewall. Follow example already in firewall jail17 is:

00117 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 or 69.55.238.150 } to 69.55.228.2
00117 deny ip from any to 69.55.228.2

jail4 would be 00104...
ipfw add 00118 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 or 69.55.238.150 } to 69.55.228.140
ipfw add 00118 deny ip from any to 69.55.228.140

46. select customers for probe map
</pre>

= FreeBSD 6.1 =

Last updated 2006-05-09

<pre>All time extimates below assume disks aren’t scrubbing

1. make sure bios is setup for bios console redirect
Supermicro:
Console redirection:
Com port addr: on-board COM A
Baud: 38400
Console type: vt100
Flow control: none
Console connection: direct
Continue cr after post: off

2450:
Make sure running bios A09
Console Redirection: VT100/VT220

2. assuming mirrors (or at least disks) created (if not, refer to this), boot to disk 1 of 5.4
skip kernel config (enter)
custom install

partition ->
move cursor to amrd0, hit space (takes you to partition map screen)
a for entire disk
q to quit and save
standard mbr (no boot manager)
space to unselect aacd0
cursor over aacd1
space
a for entire disk
q to quit and save
none (leave untouched)

cursor over aacd0
space
(takes you into part. Screen again) q to exit
none
Make sure both are checked and tab to ok

Label ->
Make sure aacd0 is highlighted
a to start with defaults

john likes:
/1g
/var 256m

Glenn likes:
/128
/var 256
/usr 3g
/tmp 256

We do:
delete everything
/ 128M
swap 2G
/var 256M
/tmp 256M
/usr 3G
/mnt/data1 remaining space

Make sure to toggle S for soft updates on all (should look like UFS2+S Y under the Newfs column)

move cursor to aacd1
swap 2G
/mnt/data2 remaining space

q to save and exit

distributions ->
developer
yes to install ports
exit

media ->
cd

commit ->
yes
(2450: 14mins, supermicro: 12mins)

yes to "visit general config" ->
Set root pwd

Add user ‘user’ member group is wheel, set password

Set tz

Networking->interfaces->Fxp0 ->
No IPV6
dhcp=no
Set hostname & domain
Enable sshd

exit...
exit install ->
yes

take the cd out and let the machine reboot
when it comes back up, enter junk when it asks for key seed

3. double check the date/time

4. edit /etc/make.conf (only add the console speed line if this is a supermicro capable of outputting BIOS redirect at that speed – 2450’s can only do 9600)
echo "WITHOUT_X11=yes \
KERNCONF=jail19 \
BOOT_COMCONSOLE_SPEED=38400" >> /etc/make.conf

5. add console="comconsole" to /boot/loader.conf
echo "console=""comconsole""" >> /boot/loader.conf

6. turn off all ttyv's except 0 and 1 in /etc/ttys
also turn on ttyd0, change type to vt100:
vi /etc/ttys
ttyd0 "/usr/libexec/getty std.9600" vt100 on secure

kill -1 1

on console server:
vi /etc/remote
(rename port to jail18 depending on where and which digi plugged into)
test serial console

7. install cvsup
cd /usr/ports/net/cvsup-without-gui
make install clean; rehash
(stay close for gettext options, 2450: 27mins, supermicro: 17mins)

8. get latest sources for this release:
cd /usr/src
echo "*default host=cvsup4.freebsd.org\
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_6_1\
*default delete use-rel-suffix\
*default compress\
src-all" > sup

cvsup sup
(2450, ~12mins, supermicro, 27mins)

9. populate hosts
echo "10.1.4.3 backup2" >> /etc/hosts

10. put key in authorized_keys on backup2
cd
ssh-keygen -t dsa -b 1024
(default location, leave password blank)
scp /root/.ssh/id_dsa.pub user@backup2:/tmp/jailkey

cat /root/.ssh/id_dsa.pub | ssh backup2 'cat - >> /root/.ssh/authorized_keys'

confirm that you can ssh to backup2 without getting a login prompt

ssh backup2

11. configure new kernel. Get config from similar machine or there may be a master copy somewhere under /mnt/data4/build/freebsd (name the kernel config the same as the jail, ex jail18):
cd /usr/src/sys/i386/conf
scp backup2:/mnt/data4/build/freebsd/kern_config-6.1 ./jail19

edit the kernel config and change ident to be the name of the jail:
vi jail14
ident jail14

edit /sys/conf/newvers.sh to add –jc1 to the end of the BRANCH string (RELEASE-jc1)
vi /sys/conf/newvers.sh

12. bring over patches from backup2
The various patches are in /mnt/data4/build/freebsd/patches on backup2. There are dirs for each version. Not all dirs are populated, but patches for later versions work on older ones unless there is a new patch in the older dir.
cd /usr/src
scp backup2:"/mnt/data4/build/freebsd/patches/6.1/*" .
jail_proc_patch 100% |*************************************************| 2593 00:00
restore-patch 100% |*************************************************| 3295 00:00
54ps-jail-patch 100% |*************************************************| 2602 00:00
jail_rtprio_patch 100% |*************************************************| 301 00:00
udp-patch 100% |*************************************************| 594 00:00

Apply patches:
patch < 54ps-jail-patch
patch < jail_proc_patch
patch < restore-patch
patch < jail_rtprio_patch
patch < udp-patch

13. build, install kernel and world
cd /usr/src
make buildworld buildkernel installkernel
(2450: 1:56min, supermicro::59mins)
make installworld
(2450: 3min, supermicro: 1min)
mergemaster -i
delete /var/tmp/temproot
delete bsnmpd
delete temporary ./etc/hosts
delete temporary ./etc/motd
delete /var/tmp/temproot

14. reboot. Confirm new kernel is loaded

uname –a

15. update ports:
cd /usr/ports
echo "*default host=cvsup4.FreeBSD.org\
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_6_1\
*default delete use-rel-suffix\
*default compress\
ports-all tag=." > sup

cvsup sup
(2450: 18mins, supermicro: 19mins)

18. (only applies if adaptec card installed)
install linux_base:
cd /usr/ports/emulators/linux_base
make install clean
(2450: 7min, supermicro: 3mins)

reibuild rpmdb cause we had probs installing aacapps
cd /compat/linux/bin
./rpm --initdb
./rpm --rebuilddb

install aacapps-4.1-0.i386.rpm
scp backup2:/mnt/data4/build/freebsd/aacapps-4.1-0.i386.rpm /tmp/.
/compat/linux/bin/rpm -ivh --excludepath=/dev /tmp/aacapps-4.1-0.i386.rpm

test out;
/compat/linux/usr/sbin/aaccli

20. edit root's path and login script:
vi /root/.cshrc
add to path:
/usr/local/jail/bin
(if adaptec card installed, also add /compat/linux/usr/sbin)

Change alias entries (add G):
alias la ls -aG
alias lf ls -FAG
alias ll ls -lAG
alias ls ls -AG
alias mbm mb mount
alias mbu mb umount

and alter the prompt, set the following:
set prompt = "`/bin/hostname -s` %/# "

at the bottom of the file add:
set sshtty=`who am i|awk '{print $2}'`
/usr/sbin/rtprio 3 -`psj | grep $sshtty | awk '{print $2}'`

set shortty=`who am i | awk '{print $2}' | sed -E 's/.*(..)$/\1/'`
foreach x (`psj | grep sh | grep $shortty | awk '{print $2}'`)
/usr/sbin/rtprio 2 -$x
end

21. install rsync from ports
cd /usr/ports/net/rsync
make install clean

choose default options

21. install perl from ports
PROB NOT NECESSARY – INSTALLED WITH LINUX I THINK
cd /usr/ports/lang/perl5.8/
make install clean; rehash
(supermicro: 5min)

22. create & populate binaries/scripts dirs
mkdir -p /usr/local/jail/bin
mkdir -p /usr/local/jail/rc.d
mkdir /mnt/data1
mkdir /mnt/data2
scp backup2:"/mnt/data4/bin/freebsd6.x/*" /usr/local/jail/bin
cd /usr/local/jail/rc.d/
touch quad1
touch quad2
touch quad3
touch quad4
touch safe1
touch safe2
touch safe3
touch safe4
chmod +x *
cd /usr/local/jail/bin
ln -s /usr/local/jail/rc.d/quad1 quad1
ln -s /usr/local/jail/rc.d/quad2 quad2
ln -s /usr/local/jail/rc.d/quad3 quad3
ln -s /usr/local/jail/rc.d/quad4 quad4
ln -s /usr/local/jail/rc.d/safe1 safe1
ln -s /usr/local/jail/rc.d/safe2 safe2
ln -s /usr/local/jail/rc.d/safe3 safe3
ln -s /usr/local/jail/rc.d/safe4 safe4

rehash

23. configure inetd to respond to mrtg load queries
echo "load stream tcp nowait user /usr/local/jail/bin/load.pl load.pl" >> /etc/inetd.conf

echo "load 12384/tcp" >> /etc/services

26. install bb client
adduser
cd /usr/home/bb
scp backup2:/mnt/data4/build/bb/bb-freebsd.tar .
tar xvf bb-freebsd.tar

edit /home/bb/bbc1.9e-btf/etc/bb-hosts with something like:
echo "10.1.4.105 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh \
10.1.4.103 jail3.johncompanies.com # ssh" > /home/bb/bbc1.9e-btf/etc/bb-hosts

vi /home/bb/bbc1.9e-btf/ext/openfiles
MACHINE="jail19,johncompanies,com" # HAS TO BE IN A,B,C FORM

cd /usr/home/bb/bbc1.9e-btf/etc
./bbchkcfg.sh (y to questions)
./bbchkhosts.sh (ignore ssh errors)
cd ../..
chown -R bb .
su bb
cd
cd bbc1.9e-btf/src
make; make install
cd ..
./runbb.sh start
more BBOUT
(look for errors)
exit

echo 'su - bb -c "cd /home/bb/bbc1.9e-btf; ./runbb.sh start"' > /usr/local/etc/rc.d/bb.sh
chmod +x /usr/local/etc/rc.d/bb.sh

modify bb-msgtab to look for raid failures

24. configure load mrtg, on mail
vi /usr/local/www/mgmt/mrtg/mrtg1.cfg
(add new entry to file following existing format)

25. configure bb on mail:
vi /usr/home/bb/bbsrc/bb1.9e-btf/etc/bb-hosts
10.1.4.119 jail19.johncompanies.com # ssh

su bb
cd
bbsrc/bb/runbb.sh restart ; exit

27. remove reserve space, enable softupdates (probably already set, so not necessary)
NOT APPLICABLE IF USING GVINUM
cd
umount /mnt/data1
umount /mnt/data2
tunefs -m 0 /mnt/data1
tunefs -m 0 /mnt/data2
mount -a

28. configure ntp
echo "server 10.1.4.105" > /etc/ntp.conf

/usr/sbin/ntpd -p /var/run/ntpd.pid
sleep 2; ntpq -p
(confirm it’s able to reach our time server)

29. mrtg switch graphs

31. fwd and reverse lookups on ns1c
vr johncompanies.com
vi internal.johncompanies.com
rndc reload johncompanies.com IN private
(edit the PTR too)

33. populate /etc/rc.conf with IPs and NFS settings
vi /etc/rc.conf

kern_securelevel_enable="NO"
portmap_enable="NO"
sendmail_enable="NO"
usbd_enable="YES"

xntpd_enable="YES"
nfs_client_enable="YES"
nfs_reserved_port_only="YES"
sshd_enable="YES"
inetd_enable="YES"
inetd_flags="-wW -a 10.1.4.119"
devfs_system_ruleset="devfsrules_show_all"

ifconfig_xl0="inet 10.1.4.118 netmask 255.255.255.0"
ifconfig_fxp0="inet 69.55.228.101 netmask 255.255.255.0"
defaultrouter="69.55.228.1"
ifconfig_fxp0_alias0="inet 69.55.2xx.xx netmask 255.255.255.0"

static_routes="t1 office"
route_t1="-net 10.1.5 10.1.4.2"
route_office="-net 10.1.6 10.1.4.2"
gvinum_enable="YES"
fsck_y_enable="YES"
background_fsck="NO"

34. make sure sysctls are set and preserved after reboot
echo "kern.consmute=0\
kern.ipc.shm_use_phys=1\
kern.ipc.shmall=65535\
kern.ipc.shmmax=134217728\
net.inet.tcp.syncookies=0\
kern.maxfiles=32768\
kern.fallback_elf_brand=3\
kern.maxprocperuid=4000\
security.jail.sysvipc_allowed=1\
security.jail.max_procs_per_jail: 1026\
security.jail.allow_raw_sockets=1\
security.jail.socket_unixiproute_only=1\
security.jail.chflags_allowed=0" >> /etc/sysctl.conf

35. mount procfs
echo "proc /proc procfs rw 0 0" >> /etc/fstab

36. enable noatime option
NOT APPLICABLE IF RUNNING GVINUM
data1 and data2 should look something like:
/dev/amrd0s1g /mnt/data1 ufs rw,noatime 2 2

36. populate devfs ruleset
scp backup2:/mnt/data4/build/freebsd/devfs.rules /etc

35. reboot
Check rules:
devfs rule showsets
devfs rule -s 3 show

36. create gvinum volumes

Make a g partition:

bsdlabel -e /dev/aacd0s1

given:
# /dev/aacd0s1:
8 partitions:
# size offset fstype [fsize bsize bps/cpg]
a: 262144 0 4.2BSD 2048 16384 16392
b: 4194304 262144 swap
c: 143363997 0 unused 0 0 # "raw" part, don't edit
d: 524288 4456448 4.2BSD 2048 16384 32776
e: 524288 4980736 4.2BSD 2048 16384 32776
f: 6291456 5505024 4.2BSD 2048 16384 28552

new offset = 6291456 + 5505024 = 11796480
new size is size for 'c' partition minus the new start from above
143363997 - 11796480 = 131567517
So:
g: 131567517 11796480 unused 0 0

For a 73G drive (after OS), we can fit 31 2G volumes so:

echo 'drive data1 device /dev/aacd0s1g' > /tmp/cgv
sh
for f in 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31;\
do echo "volume v$f" >> /tmp/cgv; echo 'plex org concat' >> /tmp/cgv;\
echo 'sd length 2g drive data1' >> /tmp/cgv; done; exit

gvinum create /tmp/cgv

For a 146G drive (-4G for swap), we can fit 66 2G volumes so:

echo 'drive data2 device /dev/aacd1s1g' > /tmp/cgv
sh
for f in 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97;\
do echo "volume v$f" >> /tmp/cgv; echo 'plex org concat' >> /tmp/cgv;\
echo 'sd length 2g drive data2' >> /tmp/cgv; done; exit

gvinum create /tmp/cgv

For 3rd 73G drive (after 2G swap), we can fit 33 2G volumes so:
Label should be:
# /dev/aacd2s1:
8 partitions:
# size offset fstype [fsize bsize bps/cpg]
c: 143299737 0 unused 0 0 # "raw" part, don't edit
g: 143299721 16 unused 0 0

echo 'drive data3 device /dev/aacd2s1g' > /tmp/cgv
sh
for f in 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131;\
do echo "volume v$f" >> /tmp/cgv; echo 'plex org concat' >> /tmp/cgv;\
echo 'sd length 2g drive data3' >> /tmp/cgv; done

gvinum create /tmp/cgv

For a 2nd 73G drive (after 2G swap), we can fit 33 2G volumes so:

echo 'drive data2 device /dev/aacd1s1g' > /tmp/cgv
sh
for f in 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64;\
do echo "volume v$f" >> /tmp/cgv; echo 'plex org concat' >> /tmp/cgv;\
echo 'sd length 2g drive data2' >> /tmp/cgv; done; exit

gvinum create /tmp/cgv

36. create the jail template

mkdir /mnt/jail
newfs /dev/gvinum/v1
mount /dev/gvinum/v1 /mnt/jail

cd /usr/src
make clean
rm -fr /usr/obj/
make world DESTDIR=/mnt/jail
(2450: 2:28mins, supermicro: 55mins)
cd etc
make distribution DESTDIR=/mnt/jail
mount_devfs devfs /mnt/jail/dev
devfs -m /mnt/jail/dev rule -s 3 applyset
cd /mnt/jail
ln -sf dev/null kernel

jail /mnt/jail testhostname 192.168.11.100 /bin/sh
csh
touch /etc/fstab
echo 'network_interfaces=""\
hostname="newsystem"\
kern_securelevel_enable="NO"\
sendmail_enable="YES"\
sshd_enable="YES"' > /etc/rc.conf

echo "nameserver 69.55.225.225\
nameserver 69.55.230.3" >> /etc/resolv.conf

vi /etc/crontab
remove the adjkerntz lines
comment out periodic’s and put this line above them:
# DO NOT UNCOMMENT THESE

rm -rf /etc/periodic/daily/400.status-disks

check /tmp for crap

vi /etc/periodic/security/100.chksetuid
replace: MP=`mount -t ufs | grep -v " nosuid" | awk '{ print $3 }' | sort`
with: MP='/' (use single quotes)

mkdir -p /usr/compat/linux/dev

adduser (Add account for user, make sure in wheel group)

user/root passwd: 8ico2987

Set root password

vi /usr/home/user/.profile (and add to the file):
TERM=vt100; export TERM

tzsetup

newaliases

#cd /dev
#rm console
#ln -s null console

exit
exit

cd /usr/ports
make -DNOCLEANDEPENDS clean
(2450: 15mins , supermicro: 29mins)
rm -fr /usr/ports/distfiles/*
cp -r /usr/ports /mnt/jail/usr (2450: 2:00 mins , supermicro: 15mins)

rm /mnt/jail/root/.history

cd
mkdir /usr/local/jail/template/
dump -0a -f /usr/local/jail/template/61template /dev/gvinum/v1

umount /mnt/jail/dev
umount /dev/gvinum/v1

rm -fr /mnt/jail

37. setup backups
echo '#\!/bin/sh\
backupdir=/mnt/data3jail3_rsync\
\
## ENTRY /etc\
## ENTRY /usr/local/etc\
## ENTRY /usr/local/jail\
## ENTRY /root/logs' > /usr/local/jail/bin/backup.config

on backup2:
setup backup dirs:
mkdir -p /mnt/data3/jail3/0

add the system to
vi /mnt/data4/bin/snapshot_rotate

on mail:
vi /usr/local/www/mgmt/cgi/backupgraph.pl
(add hostname)

38. mkdir /root/logs

39. edit sshd_config for security
vi /etc/ssh/sshd_config
ListenAddress 69.55.228.101
ListenAddress 10.1.4.118

kill -1 `cat /var/run/sshd.pid`

40. add crontab entries
crontab -e
5 0 * * * /usr/local/jail/bin/backup
1 0 1 * * /usr/local/jail/bin/ipfwreset
0 18 * * * /usr/local/jail/bin/ipfwbackup
4,9,14,19,24,29,34,39,44,49,55,59 * * * * /usr/local/jail/bin/trafstats
55 10,23 * * * /usr/local/jail/bin/trafficwatch.pl

41. Reboot notify script
ln -s /usr/local/jail/bin/notify.sh /usr/local/etc/rc.d/notify.sh

42. copy jailmake from prev system
scp user@10.1.4.119:/usr/local/jail/bin/jailmake /usr/local/jail/bin
rehash
NOTE: remove df altering code from jailmake since we put the correct df in the template, and make sure path to template file is right

43. add to management db (on mail and devweb) jc.ref_machines and jc.ref_templates

uname -r
5.4-RELEASE-p2-jc2

insert into ref_machines values (null,'jail19','jail19.johncompanies.com',0,'l');
select machine_id from ref_machines where host='jail19';
+------------+
| machine_id |
+------------+
| 35 |
+------------+
insert into ref_templates values ('',' 6.2-RELEASE-jc1',5,'FreeBSD 6.2',0);

44. add to server/cabinet map. On mail:
vi /usr/local/www/mgmt/html/cabinetmap.html

45. add an outside blocking rule to the firewall, so this machine can only be reached from inside the firewall. Follow example already in firewall jail17 is:

00119 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 or 69.55.238.150 } to 69.55.228.200
00119 deny ip from any to 69.55.228.200

jail19 would be 00119...
ipfw add 00119 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 or 69.55.238.150 } to 69.55.228.200
ipfw add 00119 deny ip from any to 69.55.228.200

46. select customers for probe map

47. install raid monitor

cd /usr/ports/sysutils/asr-utils
make install clean

48. make gv start on boot

scp backup2:/mnt/data4/build/freebsd/gvinum /etc/rc.d/gvinum

gconcat label -v somelabel /dev/gvinum/a /dev/gvinum/b
bsdlabel -r -w /dev/concat/somelabel
newfs /dev/concat/somelabela
mount /dev/concat/somelabel /mount/point

umount /dev/concat/somelabel
gconcat stop somelabel
gconcat label -v /dev/gvinum/a /dev/gvinum/b /dev/gvinum/c /dev/gvinum/d
growfs /dev/concat/somelabel
mount /dev/concat/somelabel /mount/point

volume f
plex org concat
sd length 30449m drive data1

where f is

D data1 State: up /dev/aacd0s1g A: 30449/64241 MB (47%)

gvinum rm -r f

so i setup 2 machines with 6.1. A had 2 logical drives and gv's created across both. B had 1 logical drive and gv's across the 1 drive.
the labeling for the gv's was v1-vN (till i ran out of space)
when i moved aac1 from A to B, the gv's on A's aac1, took precedence over the similarly labeled gv's on B's aac0. in other words. B's aac0 used to have v1-v30. A's aac1 had vn6-vn30. when A's aac1 was moved to B, v1-v5 were linked to B's aac0, and v6-v30 linked to A's aac1
i relabeled B's aac0 gv's to something different (not v1-vN) and tried again and they all showed up.
i tried the experiment again this time moving A's aac0 to B. at that point nothing worked. i moved A's aac0 back to A and renamed the device (data1) to something else, then tried the test again. still no beans.
it's hazy what i did after that cause the machine's weren't cooperating and i was trying to rename the device but basically i think i need to repeat the test and see if i can re-define the device, and probably also give it a unique name and i bet it would have worked.
also interesting- A's aac1 contained data about A's aac0 which showed up when i put aac1 into B
skeeter: as for the gv stuff, it definitely sounds like using some sort of serial numbering scheme would be the way to go if you want to be able to move disks around....
that overlap is still an issue (aac1's device was 'data2') when i had A's aac0 in B nothing worked both disks gv devices were called data1
skeeter: I suppose you could serialize those names as well...</pre>

= FreeBSD 6.2 =

Last updated 2007-10-15

<pre>All time extimates below assume disks aren’t scrubbing. Setup instructions below are for LSI card:

1. make sure bios is setup for bios console redirect
Supermicro:
Console redirection:
Com port addr: on-board COM A
Baud: 38400
Console type: vt100
Flow control: none
Console connection: direct
Continue cr after post: on

2450:
Make sure running bios A09
Console Redirection: VT100/VT220

2. assuming mirrors (or at least disks) created (if not, refer to this), boot to disk 1 of 5.4
skip kernel config (enter)
custom install

partition ->
move cursor to amrd0, hit space (takes you to partition map screen)
a for entire disk
q to quit and save
standard mbr (no boot manager)
space to unselect amrd0
cursor over amrd1
space
a for entire disk
q to quit and save
none (leave untouched)

cursor over amrd0
space
(takes you into part. Screen again) q to exit
none
Make sure both are checked and tab to ok

Label ->
Make sure amrd0 is highlighted
a to start with defaults

john likes:
/1g
/var 256m

Glenn likes:
/128
/var 256
/usr 3g
/tmp 256

We do:
delete everything
/ 128M
swap 2G (for 2950 make it 4G)
/var 256M
/tmp 256M
/usr 3.5G (3584M)
/mnt/data1 remaining space (no need to newfs)

Make sure to toggle S for soft updates on all (should look like UFS2+S Y under the Newfs column)

move cursor to amrd1
swap 4G
/mnt/data2 remaining space (no need to newfs)

q to save and exit

distributions ->
developer
yes to install ports
exit

media ->
cd

commit ->
yes
(2450: 14mins, supermicro: 12mins)

yes to "visit general config" ->
Set root pwd

Add user ‘user’ member group is wheel, set password

Set tz

Networking->interfaces->Fxp0 ->
No IPV6
dhcp=no
Set hostname & domain
Enable sshd

exit...
exit install ->
yes

take the cd out and let the machine reboot

3. double check the date/time

4. edit /etc/make.conf (only add the console speed line if this is a supermicro capable of outputting BIOS redirect at that speed – 2450’s can only do 9600)
echo "WITHOUT_X11=yes \
KERNCONF=jail7 \
BOOT_COMCONSOLE_SPEED=115200" >> /etc/make.conf

5. add console="comconsole" to /boot/loader.conf
echo "console=""comconsole""" >> /boot/loader.conf

6. turn off all ttyv's except 0 and 1 in /etc/ttys
also turn on ttyd0, change type to vt100:
vi /etc/ttys
ttyd0 "/usr/libexec/getty std.9600" vt100 on secure

kill -1 1

on console server:
vi /etc/remote
(rename port to jail18 depending on where and which digi plugged into)
test serial console

7. populate hosts
echo "10.1.4.3 backup2" >> /etc/hosts

8. put key in authorized_keys on backup2
cd
ssh-keygen -t dsa -b 1024
(default location, leave password blank)

cat /root/.ssh/id_dsa.pub | ssh backup2 'cat - >> /root/.ssh/authorized_keys'

confirm that you can ssh to backup2 without getting a login prompt

ssh backup2

9. create & populate binaries/scripts dirs
mkdir -p /usr/local/jail/bin
mkdir -p /usr/local/jail/rc.d
mkdir -p /usr/local/jail/template/
mkdir /mnt/data1
mkdir /mnt/data2
scp backup2:"/mnt/data4/bin/freebsd6.x/*" /usr/local/jail/bin
cd /usr/local/jail/rc.d/
touch quad1
touch quad2
touch quad3
touch quad4
touch safe1
touch safe2
touch safe3
touch safe4
chmod +x *
cd /usr/local/jail/bin
ln -s /usr/local/jail/rc.d/quad1 quad1
ln -s /usr/local/jail/rc.d/quad2 quad2
ln -s /usr/local/jail/rc.d/quad3 quad3
ln -s /usr/local/jail/rc.d/quad4 quad4
ln -s /usr/local/jail/rc.d/safe1 safe1
ln -s /usr/local/jail/rc.d/safe2 safe2
ln -s /usr/local/jail/rc.d/safe3 safe3
ln -s /usr/local/jail/rc.d/safe4 safe4

rehash

10. edit root's path and login script:
vi /root/.cshrc
add to path:
/usr/local/jail/bin
(if adaptec card installed, also add /compat/linux/usr/sbin)

Change alias entries (add G):
alias la ls -aG
alias lf ls -FAG
alias ll ls -lAG
alias ls ls -AG
alias mbm mb mount
alias mbu mb umount

and alter the prompt, set the following:
set prompt = "`/bin/hostname -s` %/# "

at the bottom of the file add:
set sshtty=`who am i|awk '{print $2}'`
/usr/sbin/rtprio 3 -`psj | grep $sshtty | awk '{print $2}'`

set shortty=`who am i | awk '{print $2}' | sed -E 's/.*(..)$/\1/'`
foreach x (`psj | grep sh | grep $shortty | awk '{print $2}'`)
/usr/sbin/rtprio 2 -$x
end

To load the new file:
source /root/.cshrc

11. install cvsup
cd /usr/ports/net/cvsup-without-gui
make install clean; rehash; mail -s 'cvs installed' dave.boodman@vtext.com < /dev/null

(stay close for gettext options, 2450: 27mins, supermicro: 17mins, 2950: 22mins)

12. get latest sources for this release:
cd /usr/src
echo "*default host=cvsup4.freebsd.org\
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_6_2\
*default delete use-rel-suffix\
*default compress\
src-all" > sup

cvsup sup ; mail -s 'cvs sup done' dave.boodman@vtext.com < /dev/null

(2450, ~12mins, supermicro, 27mins, 2950: 7mins)

13. configure new kernel. Get config from similar machine or there may be a master copy somewhere under /mnt/data4/build/freebsd (name the kernel config the same as the jail, ex jail18):
cd /usr/src/sys/i386/conf
scp backup2:/mnt/data4/build/freebsd/kern_config-6.2 ./jail7

or for PAE

scp backup2:/mnt/data4/build/freebsd/kern_config-6.2-PAE ./jail7

edit the kernel config and change ident to be the name of the jail:
vi jail7
ident jail7

edit /sys/conf/newvers.sh to add –jc1 to the end of the BRANCH string (RELEASE-jc1)
vi /sys/conf/newvers.sh

14. bring over patches from backup2

The various patches are in /mnt/data4/build/freebsd/patches on backup2. There are dirs for each version. Not all dirs are populated, but patches for later versions work on older ones unless there is a new patch in the older dir.

cd /usr/src
scp backup2:"/mnt/data4/build/freebsd/patches/6.x/*" .

Apply patches:
patch -l < jls-patch

Apply these only to 2950 with PAE:
patch -p0 < mfi-patch
patch -p0 < gvinum-staticcompile-patch
patch -p0 < gvinum-bin-patch

15. build, install kernel and world
cd /usr/src
make buildworld buildkernel installkernel; mail -s 'kernel build done' dave.boodman@vtext.com < /dev/null
(2450: 1:56min, supermicro: 59mins, 2950: 38mins)
make installworld
(2450: 3min, supermicro: 1min, 2950: :34)
mergemaster -i
delete /var/tmp/temproot
delete bsnmpd
delete temporary ./etc/hosts
delete temporary ./etc/motd
delete /var/tmp/temproot

16. populate devfs ruleset
scp backup2:/mnt/data4/build/freebsd/devfs.rules /etc

17. populate /etc/rc.conf with IPs and NFS settings
vi /etc/rc.conf

kern_securelevel_enable="NO"
portmap_enable="NO"
sendmail_enable="NO"
usbd_enable="YES"

xntpd_enable="YES"
nfs_client_enable="YES"
nfs_reserved_port_only="YES"
sshd_enable="YES"
inetd_enable="YES"
inetd_flags="-wW -a 10.1.4.119"
devfs_system_ruleset="devfsrules_show_all"

ifconfig_xl0="inet 10.1.4.118 netmask 255.255.255.0"
ifconfig_fxp0="inet 69.55.228.101 netmask 255.255.255.0"
defaultrouter="69.55.228.1"
ifconfig_fxp0_alias0="inet 69.55.2xx.xx netmask 255.255.255.0"

static_routes="t1 office"
route_t1="-net 10.1.5 10.1.4.2"
route_office="-net 10.1.6 10.1.4.2"
gvinum_enable="YES"
fsck_y_enable="YES"
background_fsck="NO"

18. make sure sysctls are set and preserved after reboot
echo "kern.consmute=0\
kern.ipc.shm_use_phys=1\
kern.ipc.shmall=65535\
kern.ipc.shmmax=134217728\
net.inet.tcp.syncookies=0\
kern.maxfiles=32768\
kern.fallback_elf_brand=3\
kern.maxprocperuid=4000\
security.jail.sysvipc_allowed=1\
security.jail.max_procs_per_jail: 1026\
security.jail.allow_raw_sockets=1\
security.jail.socket_unixiproute_only=1\
security.jail.chflags_allowed=0\
dev.amr.0.allow_volume_configure=1\
compat.linux.osrelease=2.6.12" >> /etc/sysctl.conf

19. mount procfs
echo "proc /proc procfs rw 0 0" >> /etc/fstab

For Dell 2950:
echo "linprocfs /usr/compat/linux/proc linprocfs rw 0 0" >> /etc/fstab
echo "linsysfs /usr/compat/linux/sys linsysfs rw 0 0" >> /etc/fstab

20. reboot. Confirm new kernel is loaded

uname –a

Check devfs rules:
devfs rule showsets
devfs rule -s 3 show

21. update ports:
cd /usr/ports
echo "*default host=cvsup4.FreeBSD.org\
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_6_2\
*default delete use-rel-suffix\
*default compress\
ports-all tag=." > sup

cvsup sup; mail -s 'cvs sup ports done' dave.boodman@vtext.com < /dev/null

(2450: 18mins, supermicro: 19mins; 2950: 24mins)

22. Install raid mgmt tool
install linux_base:
cd /usr/ports/emulators/linux_base-fc4
make install clean
(2450: 7min, supermicro: 3mins, 2950: 14mins)

(for LSI)
cd /usr/ports/sysutils/linux-megamgr
make install clean

cd /usr/ports/sysutils/megarc
make install clean

(for Perc5/i)
cd /usr/ports/sysutils/linux-megacli
make install clean

Test:
rehash; megacli ldinfo lall a0

(for adaptec)
This didn’t work: reibuild rpmdb cause we had probs installing aacapps
cd /compat/linux/bin
./rpm --initdb
./rpm --rebuilddb

Install of linux-base lead to broken rpm on 6.2 so:
install aacapps-4.1-0.i386.rpm
scp backup2:/mnt/data4/build/freebsd/aacapps-4.1-0.i386.rpm /tmp/.
/compat/linux/bin/rpm -ivh --excludepath=/dev /tmp/aacapps-4.1-0.i386.rpm
scp user@10.1.4.107:/compat/linux/usr/sbin/aaccli /compat/linux/usr/sbin/aaccli

test out;
/compat/linux/usr/sbin/aaccli

rpm didn’t even install on latest so just scp’d over aaccli and it worked

23. install rsync from ports
cd /usr/ports/net/rsync
make install clean

choose default options

24. install perl from ports
Not necessary if linux_base is installed

cd /usr/ports/lang/perl5.8/
make install clean; rehash
(supermicro: 5min)

25. configure inetd to respond to mrtg load queries
echo "load stream tcp nowait user /usr/local/jail/bin/load.pl load.pl" >> /etc/inetd.conf

echo "load 12384/tcp" >> /etc/services

26. install bb client
adduser
Username: bb
Full name: bb
Uid (Leave empty for default): 1984
Login group [bb]:
Login group is bb. Invite bb into other groups? []:
Login class [default]:
Shell (sh csh tcsh nologin) [sh]:
Home directory [/home/bb]:
Use password-based authentication? [yes]:
Use an empty password? (yes/no) [no]:
Use a random password? (yes/no) [no]: yes
Lock out the account after creation? [no]:
Username : bb
Password : <random>
Full Name : bb
Uid : 1984
Class :
Groups : bb
Home : /home/bb
Shell : /bin/sh
Locked : no
OK? (yes/no): yes

cd /usr/home/bb
scp backup2:/mnt/data4/build/bb/bb-freebsd.tar .
tar xvf bb-freebsd.tar

edit /home/bb/bbc1.9e-btf/etc/bb-hosts with something like:
echo "10.1.4.5 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh \
10.1.4.107 jail7.johncompanies.com # ssh" > /home/bb/bbc1.9e-btf/etc/bb-hosts

vi /home/bb/bbc1.9e-btf/ext/openfiles
MACHINE="jail19,johncompanies,com" # HAS TO BE IN A,B,C FORM

cd /usr/home/bb/bbc1.9e-btf/etc
./bbchkcfg.sh
(y to questions)
./bbchkhosts.sh
(ignore ssh errors)
cd ../..
chown -R bb .
su bb
cd
cd bbc1.9e-btf/src
make; make install
cd ..
./runbb.sh start
more BBOUT
(look for errors)
exit

echo 'su - bb -c "cd /home/bb/bbc1.9e-btf; ./runbb.sh start"' > /usr/local/etc/rc.d/bb.sh
chmod +x /usr/local/etc/rc.d/bb.sh

27. configure load mrtg, on mail
vi /usr/local/www/mgmt/mrtg/mrtg1.cfg
(add new entry to file following existing format)

28. configure bb on mail:
vi /usr/home/bb/bbsrc/bb1.9e-btf/etc/bb-hosts
10.1.4.119 jail19.johncompanies.com # ssh

su bb
cd
bbsrc/bb/runbb.sh restart ; exit

29. remove reserve space, enable softupdates (probably already set, so not necessary)
NOT APPLICABLE IF USING GVINUM
cd
umount /mnt/data1
umount /mnt/data2
tunefs -m 0 /mnt/data1
tunefs -m 0 /mnt/data2
mount -a

30. configure ntp
echo "server 10.1.4.5" > /etc/ntp.conf

/usr/sbin/ntpd -p /var/run/ntpd.pid
sleep 2; ntpq -p
(confirm it’s able to reach our time server)

31. mrtg switch graphs

32. fwd and reverse lookups on ns1c
vr johncompanies.com
vi internal.johncompanies.com
rndc reload johncompanies.com IN private
(edit the PTR too)

33. enable noatime option
NOT APPLICABLE IF RUNNING GVINUM
data1 and data2 should look something like:
/dev/amrd0s1g /mnt/data1 ufs rw,noatime 2 2

reboot

34. create gvinum volumes

Make a g partition:

bsdlabel -e /dev/amrd0s1

given:
# /dev/aacd0s1:
8 partitions:
# size offset fstype [fsize bsize bps/cpg]
a: 262144 0 4.2BSD 2048 16384 16392
b: 4194304 262144 swap
c: 143363997 0 unused 0 0 # "raw" part, don't edit
d: 524288 4456448 4.2BSD 2048 16384 32776
e: 524288 4980736 4.2BSD 2048 16384 32776
f: 6291456 5505024 4.2BSD 2048 16384 28552

new offset = 6291456 + 5505024 = 11796480
new size is size for 'c' partition minus the new start from above
143363997 - 11796480 = 131567517
So:
g: 131567517 11796480 unused 0 0

bsdlabel -e /dev/amrd1s1
change d to g

For a 73G drive (after OS), we can fit 31 2G volumes so:

echo 'drive data1 device /dev/aacd0s1g' > /tmp/cgv

or

echo '#\!/bin/sh\
i="1"\
while [ $i -le 31 ]\
do\
echo "volume v$i" >> /tmp/cgv;\
echo 'plex org concat' >> /tmp/cgv;\
echo 'sd length 2g drive data1' >> /tmp/cgv;\
i=`expr $i + 1`\
done' > /tmp/mkcgv
sh /tmp/mkcgv
gvinum create /tmp/cgv

For a 146G drive (-4G for swap), we can fit 66 2G volumes so:

echo 'drive data2 device /dev/amrd1s1g' > /tmp/cgv
sh
for f in 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97;\
do echo "volume v$f" >> /tmp/cgv; echo 'plex org concat' >> /tmp/cgv;\
echo 'sd length 2g drive data2' >> /tmp/cgv; done; exit

gvinum create /tmp/cgv

For 3rd 73G drive (after 2G swap), we can fit 33 2G volumes so:
Label should be:
# /dev/aacd2s1:
8 partitions:
# size offset fstype [fsize bsize bps/cpg]
c: 143299737 0 unused 0 0 # "raw" part, don't edit
g: 143299721 16 unused 0 0

/dev/aacd1s1d
echo 'drive data3 device /dev/aacd2s1g' > /tmp/cgv
sh
for f in 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131;\
do echo "volume v$f" >> /tmp/cgv; echo 'plex org concat' >> /tmp/cgv;\
echo 'sd length 2g drive data3' >> /tmp/cgv; done

gvinum create /tmp/cgv

For a 2nd 73G drive (after 2G swap), we can fit 33 2G volumes so:

echo 'drive data2 device /dev/aacd1s1g' > /tmp/cgv
sh
for f in 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64;\
do echo "volume v$f" >> /tmp/cgv; echo 'plex org concat' >> /tmp/cgv;\
echo 'sd length 2g drive data2' >> /tmp/cgv; done; exit

gvinum create /tmp/cgv

For a 2nd 73G drive (after 4G swap), we can fit 32 2G volumes so:

echo 'drive data2 device /dev/aacd1s1g' > /tmp/cgv
sh
for f in 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63;\
do echo "volume v$f" >> /tmp/cgv; echo 'plex org concat' >> /tmp/cgv;\
echo 'sd length 2g drive data2' >> /tmp/cgv; done; exit

gvinum create /tmp/cgv

For a 3nd 73G drive, we can fit 34 2G volumes so:

echo 'drive data3 device /dev/mfid2s1g' > /tmp/cgv
sh
for f in 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163;\
do echo "volume v$f" >> /tmp/cgv; echo 'plex org concat' >> /tmp/cgv;\
echo 'sd length 2g drive data3' >> /tmp/cgv; done; exit

gvinum create /tmp/cgv

2950:
bsdlabel -e /dev/mfid0s1
bsdlabel -e /dev/mfid1s1

For 1st drive (146G)
echo 'drive data1 device /dev/mfid0s1g' > /tmp/cgv
sh
for f in 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63;\
do echo "volume v$f" >> /tmp/cgv; echo 'plex org concat' >> /tmp/cgv;\
echo 'sd length 2g drive data1' >> /tmp/cgv; done; exit

Usually there’s 2040m leftover so create one more smaller vol:
echo 'volume v64\
plex org concat\
sd length 2040m drive data1' >> /tmp/cgv

gvinum create /tmp/cgv

For 2nd drive (146G)
echo 'drive data2 device /dev/mfid1s1g' > /tmp/cgv
sh
for f in 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130;\
do echo "volume v$f" >> /tmp/cgv; echo 'plex org concat' >> /tmp/cgv;\
echo 'sd length 2g drive data2' >> /tmp/cgv; done; exit
gvinum create /tmp/cgv

35. create the jail template

mkdir /mnt/jail
newfs /dev/gvinum/v1
mount /dev/gvinum/v1 /mnt/jail

cd /usr/src
make clean
rm -fr /usr/obj/
make buildworld
(2450: 2:28mins, supermicro: 53mins, 2950: 36min)
make world DESTDIR=/mnt/jail
(2450: 2:28mins, supermicro: 55mins, 2950: )
cd etc
make distribution DESTDIR=/mnt/jail
mount_devfs devfs /mnt/jail/dev
devfs -m /mnt/jail/dev rule -s 3 applyset
cd /mnt/jail
ln -sf dev/null kernel

jail /mnt/jail testhostname 192.168.11.100 /bin/sh
csh
touch /etc/fstab
echo 'network_interfaces=""\
kern_securelevel_enable="NO"\
sendmail_enable="YES"\
sshd_enable="YES"' > /etc/rc.conf

echo "nameserver 69.55.225.225\
nameserver 69.55.230.3" >> /etc/resolv.conf

vi /etc/crontab
remove the adjkerntz lines
comment out periodic’s and put this line above them:
# DO NOT UNCOMMENT THESE

rm -rf /etc/periodic/daily/400.status-disks

check /tmp for crap

vi /etc/periodic/security/100.chksetuid
replace: MP=`mount -t ufs | grep -v " nosuid" | awk '{ print $3 }' | sort`
with: MP='/' (use single quotes)

mkdir -p /usr/compat/linux/dev

adduser (Add account for user, make sure in wheel group)

user/root passwd: 8ico2987

Set root password

vi /usr/home/user/.profile (and add to the file):
TERM=vt100; export TERM

tzsetup

newaliases

#cd /dev
#rm console
#ln -s null console

vi /etc/syslog.conf (comment out console and move to /var/log/messages):
#*.err;kern.warning;auth.notice;mail.crit /dev/console *.err;kern.warning;auth.notice;mail.crit /var/log/messages

exit
exit

cd /usr/ports
make -DNOCLEANDEPENDS clean
(2450: 15mins , supermicro: 29mins, 2950: 24mins)
rm -fr /usr/ports/distfiles/*
cp -r /usr/ports /mnt/jail/usr (2450: 2:00 mins , supermicro: 15mins, 2950: 3mins)

rm /mnt/jail/root/.history

cd
dump -0a -f /usr/local/jail/template/61template /dev/gvinum/v1

umount /mnt/jail/dev
umount /dev/gvinum/v1

rm -fr /mnt/jail

36. setup backups
echo '#\!/bin/sh\
backupdir=/mnt/data4/jail7\
\
## ENTRY /etc\
## ENTRY /usr/local/etc\
## ENTRY /usr/local/jai7\
## ENTRY /root/logs' > /usr/local/jail/bin/backup.config

on backup2:
setup backup dirs:
mkdir -p /mnt/data2/jail7/0

add the system to
vi /mnt/data4/bin/snapshot_rotate

on mail:
vi /usr/local/www/mgmt/cgi/backupgraph.pl
(add hostname)

Edit /usr/local/jail/bin/backup to use the right drives

37. mkdir /root/logs

38. edit sshd_config for security
vi /etc/ssh/sshd_config
ListenAddress 69.55.226.128
ListenAddress 10.1.4.114

kill -1 `cat /var/run/sshd.pid`

39. add crontab entries
crontab -e
5 0 * * * /usr/local/jail/bin/backup
1 0 1 * * /usr/local/jail/bin/ipfwreset
0 18 * * * /usr/local/jail/bin/ipfwbackup
4,9,14,19,24,29,34,39,44,49,55,59 * * * * /usr/local/jail/bin/trafstats
55 10,23 * * * /usr/local/jail/bin/trafficwatch.pl

On 2950 with Perc5/i also add:
*/5 * * * * /usr/local/jail/bin/perc5iraidchk

40. Reboot notify script
ln -s /usr/local/jail/bin/notify.sh /usr/local/etc/rc.d/notify.sh

41. add to management db (on mail and devweb) jc.ref_machines and jc.ref_templates

uname -r
6.2-RELEASE-jc1

insert into ref_machines values (null,'jail19','jail19.johncompanies.com',0,'l');
select machine_id from ref_machines where host='jail19';
+------------+
| machine_id |
+------------+
| 35 |
+------------+
insert into ref_templates values ('',' 6.2-RELEASE-jc1',15,'FreeBSD 6.2',0);

42. add to server/cabinet map. On mail:
vi /usr/local/www/mgmt/html/cabinetmap.html

43. add an outside blocking rule to the firewall, so this machine can only be reached from inside the firewall. Follow example already in firewall jail17 is:

00117 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 or 69.55.238.150 } to 69.55.228.200
00117 deny ip from any to 69.55.228.200

jail19 would be 00119...
ipfw add 00107 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 or 69.55.238.150 } to 69.55.230.108
ipfw add 00107 deny ip from any to 69.55.220.108

44. select customers for probe map

45. install raid monitor (LSI 320 only):

scp backup2:/d4/build/megaraid/MegaMonitor1.02.tgz /tmp
pkg_add MegaMonitor1.02.tgz

edit line in /usr/local/etc/rc.d/megamonitor.sh to look like:
/usr/sbin/MegaCtrl -start -log /var/log/messages -disChkCon -SMART9999 > /megamonitor.out

comment out:
localhost: /var/log/messages : : : **Monitor** :
In /usr/home/bb/bbc1.9e-btf/etc/bb-msgstab

cd /usr/ports/sysutils/megarc
make install clean

46. make gv start on boot

scp backup2:/mnt/data4/build/freebsd/gvinum /etc/rc.d/gvinum

If they start stale:

echo '#\!/bin/sh\
i="1"\
while [ $i -le 64 ]\
do\
gvinum start v$i\
i=`expr $i + 1`\
done' > /tmp/startgv

sh /tmp/startgv

47. patch jail against starting jails with rtprio

mv /usr/sbin/jail /usr/sbin/jail_
echo '#\!/bin/sh\
/usr/sbin/rtprio -t /usr/sbin/jail_ $*' > /usr/sbin/jail
chmod +x /usr/sbin/jail

48. make sure mail works
If there are map errors:
cd /etc/mail; make maps</pre>

= FreeBSD 6.2 -> 6.3 =

Last updated 2008-08-07

<pre>

1. remove old src
cd /usr
rm –fr src/*

2. get latest sources for this release:

cd /usr/src
echo "*default host=cvsup4.freebsd.org\
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_6_3\
*default delete use-rel-suffix\
*default compress\
src-all" > sup

cvsup sup ; pagedave
(20min)

3. configure new kernel. Get config from similar machine or there may be a master copy somewhere under /mnt/data4/build/freebsd (name the kernel config the same as the jail, ex jail18):
cd /usr/src/sys/i386/conf
scp backup2:/mnt/data4/build/freebsd/kern_config-6.2 ./jail7

or for PAE

scp backup2:/mnt/data4/build/freebsd/kern_config-6.2-PAE ./jail7

edit the kernel config and change ident to be the name of the jail:
vi jail7
ident jail7

edit /sys/conf/newvers.sh to add –jc1 to the end of the BRANCH string (RELEASE-jc1)
vi /sys/conf/newvers.sh

3. bring over patches from backup2

The various patches are in /mnt/data4/build/freebsd/patches on backup2. There are dirs for each version. Not all dirs are populated, but patches for later versions work on older ones unless there is a new patch in the older dir.

cd /usr/src
scp backup2:"/mnt/data4/build/freebsd/patches/6.x/*" .

Apply patches:
patch -l < jls-patch

Apply these only to 2950 with PAE:
patch -p0 < gvinum-staticcompile-patch
patch -p0 < gvinum-bin-patch

4. build, install kernel
cd /usr/src
make buildkernel; pagedave
(2min)

cd /boot
mv kernel.old kernel.old.old
(optional move old kernel out of the way)

cd /usr/src
make installkernel

cd /boot
mv kernel.old kernel.6.2

5. take ½ of mirror and test (boot up) in new hardware

6. improved loader configs:

/boot/loader.conf:
console="comconsole,vidconsole"
boot_serial="YES"
boot_multicons="YES"

7. build userland
cd /usr/src
make buildworld; pagedave
(33mins)
make installworld

mergemaster –i
reboot
8. patch jail against starting jails with rtprio

mv /usr/sbin/jail /usr/sbin/jail_
echo '#\!/bin/sh\
/usr/sbin/rtprio -t /usr/sbin/jail_ $*' > /usr/sbin/jail
chmod +x /usr/sbin/jail
</pre>

= FreeBSD 7.0 =

Last updated 2008-04-30

<pre>

All time extimates below assume disks aren’t scrubbing. Setup instructions below are for LSI card:

1. make sure bios is setup for bios console redirect
Supermicro:
Console redirection:
Com port addr: on-board COM A
Baud: 38400
Console type: vt100
Flow control: none
Console connection: direct
Continue cr after post: on

2450:
Make sure running bios A09
Console Redirection: VT100/VT220

2. assuming mirrors (or at least disks) created (if not, refer to this), boot to disk 1 of 5.4
skip kernel config (enter)
custom install

partition ->
move cursor to amrd0, hit space (takes you to partition map screen)
a for entire disk
q to quit and save
standard mbr (no boot manager)
space to unselect amrd0
cursor over amrd1
space
a for entire disk
q to quit and save
none (leave untouched)

cursor over amrd0
space
(takes you into part. Screen again) q to exit
none
Make sure both are checked and tab to ok

Label ->
Make sure mfid0 is highlighted

/ 256M
swap 2G (for 2950 make it 4G)
/var 256M (4.6G)
/tmp 256M
/usr 4G
/mnt/data1 remaining space (no need to newfs)

Make sure to toggle S for soft updates on all (should look like UFS2+S Y under the Newfs column)

move cursor to mfid1
swap 8G
/mnt/data2 remaining space (no need to newfs)

q to save and exit

distributions ->
developer
yes to install ports
exit

media ->
cd

commit ->
yes
(2450: 14mins, supermicro: 12mins)

yes to "visit general config" ->
Set root pwd

Add user ‘user’ member group is wheel, set password

Set tz

Networking->interfaces->Fxp0 ->
No IPV6
dhcp=no
Set hostname & domain
Enable sshd

exit...
exit install ->
yes

take the cd out and let the machine reboot

3. double check the date/time

4. edit /etc/make.conf (only add the console speed line if this is a supermicro capable of outputting BIOS redirect at that speed – 2450’s can only do 9600)
echo "WITHOUT_X11=yes \
KERNCONF=jail2 \
BOOT_COMCONSOLE_SPEED=115200" >> /etc/make.conf

5. add settings to /boot/loader.conf and /boot.config

echo "-Dh" >> /boot.config

echo 'console="comconsole,vidconsole" \
boot_multicons="YES" \
boot_serial="YES" \
mfi_linux_load="YES" \
comconsole_speed="115200"' >> /boot/loader.conf

(leave out the speed and mfi lines for 2450s)

6. turn off all ttyv's except 0 and 1 in /etc/ttys
also turn on ttyd0, change type to vt100:
vi /etc/ttys
ttyd0 "/usr/libexec/getty std.9600" vt100 on secure

kill -1 1

on console server:
vi /etc/remote
(rename port to jail18 depending on where and which digi plugged into)
test serial console

7. populate hosts
echo "10.1.4.3 backup2" >> /etc/hosts
echo "10.1.4.8 backup1" >> /etc/hosts

8. put key in authorized_keys on backup2
cd
ssh-keygen -t dsa -b 1024
(default location, leave password blank)

cat /root/.ssh/id_dsa.pub | ssh backup2 'cat - >> /root/.ssh/authorized_keys'
cat /root/.ssh/id_dsa.pub | ssh backup1 'cat - >> /root/.ssh/authorized_keys'

confirm that you can ssh to backup2 and backup1 without getting a login prompt

ssh backup2 hostname

ssh backup1 hostname

9. create & populate binaries/scripts dirs
mkdir -p /usr/local/jail/bin
mkdir -p /usr/local/jail/rc.d
mkdir -p /usr/local/jail/template/
mkdir /mnt/data1
mkdir /mnt/data2
scp backup2:"/mnt/data4/bin/freebsd7.x/*" /usr/local/jail/bin
cd /usr/local/jail/rc.d/
touch quad1
touch quad2
touch quad3
touch quad4
touch safe1
touch safe2
touch safe3
touch safe4
chmod +x *
cd /usr/local/jail/bin
ln -s /usr/local/jail/rc.d/quad1 quad1
ln -s /usr/local/jail/rc.d/quad2 quad2
ln -s /usr/local/jail/rc.d/quad3 quad3
ln -s /usr/local/jail/rc.d/quad4 quad4
ln -s /usr/local/jail/rc.d/safe1 safe1
ln -s /usr/local/jail/rc.d/safe2 safe2
ln -s /usr/local/jail/rc.d/safe3 safe3
ln -s /usr/local/jail/rc.d/safe4 safe4
ln -s /usr/local/jail/bin/jailmake_zfs jailmake
--OR--
ln -s /usr/local/jail/bin/jailmake_geom jailmake

ln -s /usr/local/jail/bin/js_zfs js
--OR--
ln -s /usr/local/jail/bin/js_geom js

rehash

10. edit root's path and login script:
vi /root/.cshrc
add to path:
/usr/local/jail/bin
(if adaptec card installed, also add /compat/linux/usr/sbin)

Change alias entries (add G):
alias la ls -aG
alias lf ls -FAG
alias ll ls -lAG
alias ls ls -AG
alias mbm mb mount
alias mbu mb umount

and alter the prompt, set the following:
set prompt = "`/bin/hostname -s` %/# "

at the bottom of the file add:
set sshtty=`who am i|awk '{print $2}'`
/usr/sbin/rtprio 3 -`psj | grep $sshtty | awk '{print $2}'`

set shortty=`who am i | awk '{print $2}' | sed -E 's/.*(..)$/\1/'`
foreach x (`psj | grep sh | grep $shortty | awk '{print $2}'`)
/usr/sbin/rtprio 2 -$x
end

To load the new file:
source /root/.cshrc

11. install cvsup
cd /usr/ports/net/cvsup-without-gui
make install clean; rehash; mail -s 'cvs installed' dave.boodman@vtext.com < /dev/null

(stay close for gettext options, 2450: 27mins, supermicro: 17mins, 2950: 22mins)

12. get latest sources for this release:
cd /usr/src
echo "*default host=cvsup4.freebsd.org\
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_7\
*default delete use-rel-suffix\
*default compress\
src-all" > sup

-OR-

echo "*default host=cvsup4.freebsd.org\
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_7_1\
*default delete use-rel-suffix\
*default compress\
src-all" > sup

cvsup sup ; mail -s 'cvs sup done' dave.boodman@vtext.com < /dev/null

(2450, ~12mins, supermicro, 27mins, 2950: 7mins)

13. configure new kernel.
for i386:
cd /usr/src/sys/i386/conf
scp backup2:/mnt/data4/build/freebsd/kern_config-7.0-PAE ./mx2

-or-
for amd64:
cd /usr/src/sys/amd64/conf
scp backup2:/mnt/data4/build/freebsd/kern_config-7.0-amd64 ./jail2

------

edit the kernel config and change ident to be the name of the jail:
vi jail2
ident jail2

edit /sys/conf/newvers.sh to add –jc2 to the end of the BRANCH string (RELEASE-jc2)
vi /sys/conf/newvers.sh

notes: http://www.FreeBSD.org/doc/en_US.ISO8859-1/books/handbook/kernelconfig-config.html

14. bring over patches from backup2

The various patches are in /mnt/data4/build/freebsd/patches on backup2. There are dirs for each version. Not all dirs are populated, but patches for later versions work on older ones unless there is a new patch in the older dir.

cd /usr/src
scp backup2:"/mnt/data4/build/freebsd/patches/7.x/*" .

Apply patches:
patch -l < jls-patch

15. build, install kernel and world
cd /usr/src
make buildworld buildkernel installkernel; mail -s 'kernel build done' dave.boodman@vtext.com < /dev/null
(2450: 1:56min, supermicro: 59mins, 2950: 38mins)
make installworld
(2450: 3min, supermicro: 1min, 2950: :34)
mergemaster -i
delete /var/tmp/temproot
delete bsnmpd
delete temporary ./etc/hosts
delete temporary ./etc/motd
delete /var/tmp/temproot

cd /usr/src/sys/modules/zfs
make
make install

16. populate devfs ruleset
scp backup2:/mnt/data4/build/freebsd/devfs.rules /etc

17. populate /etc/rc.conf with IPs and NFS settings
vi /etc/rc.conf

kern_securelevel_enable="NO"
portmap_enable="NO"
sendmail_enable="NO"
usbd_enable="YES"

xntpd_enable="YES"
nfs_client_enable="YES"
nfs_reserved_port_only="YES"
sshd_enable="YES"
inetd_enable="YES"
inetd_flags="-wW -a 10.1.4.102"
devfs_system_ruleset="devfsrules_show_all"

hostname="jail2.johncompanies.com"
ifconfig_xl0="inet 10.1.4.102 netmask 255.255.255.0"
ifconfig_fxp0="inet 69.55.228.53 netmask 255.255.255.0"
defaultrouter="69.55.228.1"
#ifconfig_fxp0_alias0="inet 69.55.2xx.xx netmask 255.255.255.0"

static_routes="t1 office"
route_t1="-net 10.1.5 10.1.4.2"
route_office="-net 10.1.6 10.1.4.2"
zfs_enable="YES"

18. make sure sysctls are set and preserved after reboot
echo "kern.consmute=0\
kern.ipc.shm_use_phys=1\
kern.ipc.shmall=65535\
kern.ipc.shmmax=134217728\
net.inet.tcp.syncookies=0\
kern.maxfiles=32768\
kern.fallback_elf_brand=3\
kern.maxprocperuid=4000\
security.jail.sysvipc_allowed=1\
security.jail.allow_raw_sockets=1\
security.jail.socket_unixiproute_only=1\
security.jail.chflags_allowed=0\
compat.linux.osrelease=2.6.12\
kern.maxvnodes=400000" >> /etc/sysctl.conf

19. mount procfs
echo "proc /proc procfs rw 0 0" >> /etc/fstab

For Dell 2950/2450:
echo "linprocfs /usr/compat/linux/proc linprocfs rw 0 0" >> /etc/fstab

For Dell 2950:
echo "linsysfs /usr/compat/linux/sys linsysfs rw 0 0" >> /etc/fstab

20. reboot. Confirm new kernel is loaded

uname –a

Check devfs rules:
devfs rule showsets
devfs rule -s 3 show

21. update ports:
cd /usr/ports
echo "*default host=cvsup4.FreeBSD.org\
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_7_0\
*default delete use-rel-suffix\
*default compress\
ports-all tag=." > sup

cvsup sup; mail -s 'cvs sup ports done' dave.boodman@vtext.com < /dev/null

(2450: 18mins, supermicro: 19mins; 2950: 24mins)

22. Install raid mgmt tool

For LSI based cards:

install linux_base:
cd /usr/ports/emulators/linux_base-fc4
make install clean
(2450: 7min, supermicro: 3mins, 2950: 14mins)

(for LSI)
cd /usr/ports/sysutils/linux-megamgr
make install clean

cd /usr/ports/sysutils/megarc
make install clean

(for Perc5/i)
cd /usr/ports/sysutils/linux-megacli
make install clean

Test:
rehash; megacli ldinfo lall a0

For adaptec:
On jail18:
scp /compat/linux/usr/sbin/aaccli user@10.1.4.102:~

mv ~user/aaccli /compat/linux/usr/sbin/aaccli

test out;
/compat/linux/usr/sbin/aaccli

For 3w9690:
scp backup2:/d4/build/3ware/tw_cli-freebsd-x86_64-9.5.0.1.tgz /usr/local/sbin
cd /usr/local/sbin xzvf tw_cli-freebsd-x86_64-9.5.0.1.tgz
rm tw_cli-freebsd-x86_64-9.5.0.1.tgz

test out;
./tw_cli /c0 show allstatus

23. install rsync from ports
cd /usr/ports/net/rsync
make install clean

choose default options

24. install perl from ports
Not necessary if linux_base is installed

cd /usr/ports/lang/perl5.8/
make install clean; rehash
(supermicro: 5min)

25. configure inetd to respond to mrtg load queries
echo "load stream tcp nowait user /usr/local/jail/bin/load.pl load.pl" >> /etc/inetd.conf

echo "load 12384/tcp" >> /etc/services

26. install bb client
adduser
Username: bb
Full name: bb
Uid (Leave empty for default): 1984
Login group [bb]:
Login group is bb. Invite bb into other groups? []:
Login class [default]:
Shell (sh csh tcsh nologin) [sh]:
Home directory [/home/bb]:
Use password-based authentication? [yes]:
Use an empty password? (yes/no) [no]:
Use a random password? (yes/no) [no]: yes
Lock out the account after creation? [no]:
Username : bb
Password : <random>
Full Name : bb
Uid : 1984
Class :
Groups : bb
Home : /home/bb
Shell : /bin/sh
Locked : no
OK? (yes/no): yes

cd /usr/home/bb
scp backup2:/mnt/data4/build/bb/bb-freebsd.tar .
tar xf bb-freebsd.tar

edit /home/bb/bbc1.9e-btf/etc/bb-hosts with something like:
echo "10.1.4.5 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh \
10.1.4.108 jail2.johncompanies.com # ssh" > /home/bb/bbc1.9e-btf/etc/bb-hosts

vi /home/bb/bbc1.9e-btf/ext/openfiles
MACHINE="jail2,johncompanies,com" # HAS TO BE IN A,B,C FORM

cd /usr/home/bb/bbc1.9e-btf/etc
./bbchkcfg.sh
(y to questions)
./bbchkhosts.sh
(ignore ssh errors)
cd ../..
chown -R bb .
su bb
cd
cd bbc1.9e-btf/src
make; make install
cd ..
./runbb.sh start
more BBOUT
(look for errors)
exit

echo 'su - bb -c "cd /home/bb/bbc1.9e-btf; ./runbb.sh start"' > /usr/local/etc/rc.d/bb.sh
chmod +x /usr/local/etc/rc.d/bb.sh

NOTE: to get bb working on amd, had to copy over bin dir from linux dist

27. configure load mrtg, on mail
vi /usr/local/www/mgmt/mrtg/mrtg1.cfg
(add new entry to file following existing format)

28. configure bb on mail:
vi /usr/home/bb/bbsrc/bb1.9e-btf/etc/bb-hosts
10.1.4.108 jail2.johncompanies.com # ssh

su bb
cd
bbsrc/bb/runbb.sh restart ; exit

29. remove reserve space, enable softupdates (probably already set, so not necessary)
NOT APPLICABLE IF USING GVINUM
cd
umount /mnt/data1
umount /mnt/data2
tunefs -m 0 /mnt/data1
tunefs -m 0 /mnt/data2
mount -a

30. configure ntp
echo "server 10.1.4.5" > /etc/ntp.conf

/usr/sbin/ntpd -p /var/run/ntpd.pid
sleep 2; ntpq -p
(confirm it’s able to reach our time server)

31. mrtg switch graphs

32. fwd and reverse lookups on ns1c
vr johncompanies.com
vi internal.johncompanies.com
rndc reload johncompanies.com IN private
(edit the PTR too)

33. create zpools

Make a g partition:

bsdlabel -e /dev/mfid0s1

given:
# /dev/aacd0s1:
8 partitions:
# size offset fstype [fsize bsize bps/cpg]
a: 262144 0 4.2BSD 2048 16384 16392
b: 4194304 262144 swap
c: 143363997 0 unused 0 0 # "raw" part, don't edit
d: 524288 4456448 4.2BSD 2048 16384 32776
e: 524288 4980736 4.2BSD 2048 16384 32776
f: 6291456 5505024 4.2BSD 2048 16384 28552

new offset = 6291456 + 5505024 = 11796480
new size is size for 'c' partition minus the new start from above
143363997 - 11796480 = 131567517
So:
g: 131567517 11796480 unused 0 0

bsdlabel -e /dev/mfid1s1
change d to g

zpool create pool1 mfid0s1g
zpool create pool2 mfid1s1g
zfs set atime=off pool1
zfs set atime=off pool2

echo ' vfs.zfs.prefetch_disable="1" \
vfs.zfs.arc_min=16777216 \
vfs.zfs.arc_max=33554432 \
vm.kmem_size_max="1G" # for i386\
vm.kmem_size="1G" # for i386\
vm.kmem_size_max="1.5G" # for amd64\
vm.kmem_size="1.5G" # for amd64\
vfs.zfs.zil_disable="1" ' >> /boot/loader.conf

35. create the jail template

zfs create -o mountpoint=/mnt/data1/jail -o quota=4G pool1/jail

cd /usr/ports/sysutils/jailutils
make install clean

cd /usr/src
make world DESTDIR=/mnt/data1/jail
(2450: 2:28mins, supermicro: 55mins, 2950: 1h )
cd etc
make distribution DESTDIR=/mnt/data1/jail
mount -t devfs devfs /mnt/data1/jail/dev
devfs -m /mnt/data1/jail/dev rule -s 3 applyset
cd /mnt/data1/jail
ln -sf dev/null kernel
cp /usr/local/sbin/jkill /mnt/data1/jail/sbin

jail /mnt/data1/jail testhostname 192.168.11.100 /bin/sh
csh
touch /etc/fstab
echo 'network_interfaces=""\
hostname="newsystem"\
kern_securelevel_enable="NO"\
sendmail_enable="YES"\
sshd_enable="YES"' > /etc/rc.conf

echo "nameserver 69.55.225.225\
nameserver 69.55.230.3" >> /etc/resolv.conf

vi /etc/crontab
remove the adjkerntz lines
comment out periodic’s and put this line above them:
# IF YOU UNCOMMENT THESE, PLEASE ADJUST THEIR RUN TIME

rm -rf /etc/periodic/daily/400.status-disks

check /tmp for crap

vi /etc/periodic/security/100.chksetuid
replace: MP=`mount -t ufs | grep -v " nosuid" | awk '{ print $3 }' | sort`
with: MP='/' (use single quotes)

mkdir -p /usr/compat/linux/dev

adduser (Add account for user, make sure in wheel group)

user/root passwd: 8ico2987

Set root password

vi /usr/home/user/.profile (and add to the file):
TERM=vt100; export TERM

tzsetup

newaliases

rm /sbin/halt /sbin/reboot
ln /sbin/jkill /sbin/halt
ln /sbin/jkill /sbin/reboot

#cd /dev
#rm console
#ln -s null console

vi /etc/syslog.conf (comment out console and move to /var/log/messages):
#*.err;kern.warning;auth.notice;mail.crit /dev/console *.err;kern.warning;auth.notice;mail.crit /var/log/messages

cd /libexec
mv ld-elf32.so.1 ld-elf32.so.1-orig
ln ld-elf.so.1 ld-elf32.so.1

exit
exit

cd /usr/ports
make -DNOCLEANDEPENDS clean
(2450: 15mins , supermicro: 29mins, 2950: 24mins)
rm -fr /usr/ports/distfiles/*
cp -r /usr/ports /mnt/data1/jail/usr (2450: 2:00 mins , supermicro: 15mins, 2950: 3mins)

rm /mnt/data1/jail/root/.history

cd
zfs snapshot pool1/jail@now
zfs send pool1/jail@now | cat > /usr/local/jail/template/70template
zfs destroy pool1/jail@now

zfs destroy pool1/jail

rmdir /mnt/data1/jail
f

36. setup backups
echo '#\!/bin/sh\
backupdir=/data/jail2\
server=backup1\
\
## ENTRY /etc\
## ENTRY /usr/local/etc\
## ENTRY /usr/local/jail\
## ENTRY /root/logs' > /usr/local/jail/bin/backup.config

on backup1:
setup backup dirs:
mkdir -p /data/jail2/0

add the system to
vi /usr/local/sbin/snapshot_rotate

on mail:
vi /usr/local/www/mgmt/cgi/backupgraph.pl
(add hostname)

Edit /usr/local/jail/bin/backup to use the right drives

37. mkdir /root/logs

38. edit sshd_config for security
vi /etc/ssh/sshd_config
ListenAddress 69.55.238.233
ListenAddress 10.1.4.108

kill -1 `cat /var/run/sshd.pid`

39. add crontab entries
crontab -e
5 0 * * * /usr/local/jail/bin/backup.zfs
1 0 1 * * /usr/local/jail/bin/ipfwreset
0 18 * * * /usr/local/jail/bin/ipfwbackup
4,9,14,19,24,29,34,39,44,49,55,59 * * * * /usr/local/jail/bin/trafstats

On 2950 with Perc5/i also add:
*/5 * * * * /usr/local/jail/bin/perc5iraidchk

40. Reboot notify script
ln -s /usr/local/jail/bin/notify.sh /usr/local/etc/rc.d/notify.sh

41. add to management db (on mail and devweb) jc.ref_machines and jc.ref_templates

uname -r
6.2-RELEASE-jc1

insert into ref_machines values (null,'mx2','mx2.johncompanies.com',0,'m');
select machine_id from ref_machines where host='mx2';
+------------+
| machine_id |
+------------+
| 35 |
+------------+
insert into ref_templates values ('',' 7.1-RELEASE-jc2',44,'FreeBSD 7.1',0);

42. add to server/cabinet map. On mail:
vi /usr/local/www/mgmt/html/cabinetmap.html

43. add an outside blocking rule to the firewall, so this machine can only be reached from inside the firewall. Follow example already in firewall jail17 is:

00117 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 or 69.55.238.150 } to 69.55.228.200
00117 deny ip from any to 69.55.228.200

jail19 would be 00119...
ipfw add 00107 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 or 69.55.238.150 } to 69.55.230.108
ipfw add 00107 deny ip from any to 69.55.220.108

44. select customers for probe map

47. patch jail against starting jails with rtprio

mv /usr/sbin/jail /usr/sbin/jail_
echo '#\!/bin/sh\
/usr/sbin/rtprio -t /usr/sbin/jail_ $*' > /usr/sbin/jail
chmod +x /usr/sbin/jail

48. make sure mail works
If there are map errors:
cd /etc/mail; make maps

49. move and pare down generic kernel
mv kernel.old/ kernel.GENERIC

/ too full, remove old kernel modules except for:

Id Refs Address Size Name
1 14 0xffffffff80100000 ac6c08 kernel
2 1 0xffffffff80bc8000 1128 mfi_linux.ko
3 4 0xffffffff80bca000 39138 linux.ko
4 1 0xffffffffb48f8000 358c linprocfs.ko
5 1 0xffffffffb48fc000 9d3 linsysfs.ko
6 1 0xffffffffb490c000 80ee8 zfs.ko

50. ntp doesn’t seem to start from rc so,

echo '/usr/sbin/ntpd -c /etc/ntp.conf -p /var/run/ntpd.pid -f /var/db/ntpd.drift' > /usr/local/etc/rc.d/ntp.sh
chmod 0700 /usr/local/etc/rc.d/ntp.sh
</pre>

= FreeBSD 7.1 =

Last updated 2009-02-16

<pre>All time extimates below assume disks aren’t scrubbing. Setup instructions below are for LSI card:

1. make sure bios is setup for bios console redirect
Supermicro:
Console redirection:
Com port addr: on-board COM A
Baud: 38400
Console type: vt100
Flow control: none
Console connection: direct
Continue cr after post: on

2450:
Make sure running bios A09
Console Redirection: VT100/VT220

2. assuming mirrors (or at least disks) created (if not, refer to this), boot to disk 1 of 5.4
skip kernel config (enter)
custom install

partition ->
move cursor to amrd0, hit space (takes you to partition map screen)
a for entire disk
q to quit and save

NOTE
For gvin probs made 2 slices, 1st 8960M
Freebsd boot mgr

standard mbr (no boot manager)
space to unselect amrd0
cursor over amrd1
space
a for entire disk
q to quit and save
none (leave untouched)

cursor over amrd0
space
(takes you into part. Screen again) q to exit
none
Make sure both are checked and tab to ok

Label ->
Make sure mfid0 is highlighted

/ 256M
swap 2G (for 2950 make it 4G)
/var 256M (4.6G)
/tmp 256M
/usr 4G
/mnt/data1 remaining space (no need to newfs)

Make sure to toggle S for soft updates on all (should look like UFS2+S Y under the Newfs column)

move cursor to mfid1
swap 8G
/mnt/data2 remaining space (no need to newfs)

q to save and exit

distributions ->
developer
yes to install ports
exit

media ->
cd

commit ->
yes
(2450: 14mins, supermicro: 12mins)

yes to "visit general config" ->
Set root pwd

Add user ‘user’ member group is wheel, set password

Set tz

Networking->interfaces->Fxp0 ->
No IPV6
dhcp=no
Set hostname & domain
Enable sshd

exit...
exit install ->
yes

take the cd out and let the machine reboot

3. double check the date/time

4. edit /etc/make.conf (only add the console speed line if this is a supermicro capable of outputting BIOS redirect at that speed – 2450’s can only do 9600)
echo "WITHOUT_X11=yes \
KERNCONF=jail2 \
BOOT_COMCONSOLE_SPEED=115200" >> /etc/make.conf

5. add settings to /boot/loader.conf and /boot.config

echo "-Dh" >> /boot.config

echo 'console="comconsole,vidconsole" \
boot_multicons="YES" \
boot_serial="YES" \
mfi_linux_load="YES" \
comconsole_speed="115200"' >> /boot/loader.conf

(leave out the speed and mfi lines for 2450s)

6. turn off all ttyv's except 0 and 1 in /etc/ttys
also turn on ttyd0, change type to vt100:
vi /etc/ttys

ttyv2 "/usr/libexec/getty Pc" cons25 off secure
ttyv3 "/usr/libexec/getty Pc" cons25 off secure
ttyv4 "/usr/libexec/getty Pc" cons25 off secure
ttyv5 "/usr/libexec/getty Pc" cons25 off secure
ttyv6 "/usr/libexec/getty Pc" cons25 off secure
ttyv7 "/usr/libexec/getty Pc" cons25 off secure

ttyd0 "/usr/libexec/getty std.9600" vt100 on secure

kill -1 1

on console server:
vi /etc/remote
(rename port to jail18 depending on where and which digi plugged into)
test serial console

7. populate hosts
echo "10.1.4.3 backup2" >> /etc/hosts
echo "10.1.4.8 backup1" >> /etc/hosts

8. put key in authorized_keys on backup2
cd
ssh-keygen -t dsa -b 1024
(default location, leave password blank)

cat /root/.ssh/id_dsa.pub | ssh backup2 'cat - >> /root/.ssh/authorized_keys'
cat /root/.ssh/id_dsa.pub | ssh backup1 'cat - >> /root/.ssh/authorized_keys'

confirm that you can ssh to backup2 and backup1 without getting a login prompt

ssh backup2 hostname

ssh backup1 hostname

9. create & populate binaries/scripts dirs
mkdir -p /usr/local/jail/bin
mkdir -p /usr/local/jail/rc.d
mkdir -p /usr/local/jail/template/
mkdir /mnt/data1
mkdir /mnt/data2
scp backup2:"/mnt/data4/bin/freebsd7.x/*" /usr/local/jail/bin
cd /usr/local/jail/rc.d/
touch quad1
touch quad2
touch quad3
touch quad4
touch safe1
touch safe2
touch safe3
touch safe4
chmod +x *
cd /usr/local/jail/bin
ln -s /usr/local/jail/rc.d/quad1 quad1
ln -s /usr/local/jail/rc.d/quad2 quad2
ln -s /usr/local/jail/rc.d/quad3 quad3
ln -s /usr/local/jail/rc.d/quad4 quad4
ln -s /usr/local/jail/rc.d/safe1 safe1
ln -s /usr/local/jail/rc.d/safe2 safe2
ln -s /usr/local/jail/rc.d/safe3 safe3
ln -s /usr/local/jail/rc.d/safe4 safe4
ln -s /usr/local/jail/bin/jailmake_geom jailmake
ln -s /usr/local/jail/bin/js_geom js
ln -s /usr/local/jail/bin/canceljail_geom canceljail
ln -s /usr/local/jail/bin/jailmakeempty_geom jailmakeempty

ln -s /usr/local/jail/bin/jailmake_md jailmake
ln -s /usr/local/jail/bin/js_md js
ln -s /usr/local/jail/bin/canceljail_md canceljail
ln -s /usr/local/jail/bin/jailmakeempty_md jailmakeempty
ln -s /usr/local/jail/bin/postboot_md postboot

rehash

10. edit root's path and login script:
vi /root/.cshrc

Change alias entries (add G):
alias la ls -aG
alias lf ls -FAG
alias ll ls -lAG
alias ls ls -AG
alias mbm mb mount
alias mbu mb umount

add to path:
/usr/local/jail/bin
(if adaptec card installed, also add /compat/linux/usr/sbin)

and alter the prompt, set the following:
set prompt = "`/bin/hostname -s` %/# "

at the bottom of the file add:
set sshtty=`who am i|awk '{print $2}'`
/usr/sbin/rtprio 3 -`psj | grep $sshtty | awk '{print $2}'`

set shortty=`who am i | awk '{print $2}' | sed -E 's/.*(..)$/\1/'`
foreach x (`psj | grep sh | grep $shortty | awk '{print $2}'`)
/usr/sbin/rtprio 2 -$x
end

To load the new file:
source /root/.cshrc

11. install cvsup
cd /usr/ports/net/cvsup-without-gui
make install clean; rehash; mail -s 'cvs installed' dave.boodman@vtext.com < /dev/null

(stay close for gettext options, 2450: 27mins, supermicro: 17mins, 2950: 22mins)

12. get latest sources for this release:
cd /usr/src
echo "*default host=cvsup4.freebsd.org\
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_7_1\
*default delete use-rel-suffix\
*default compress\
src-all" > sup

-OR-

echo "*default host=cvsup4.freebsd.org\
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_7\
*default delete use-rel-suffix\
*default compress\
src-all" > sup

(stable)

cvsup sup ; mail -s 'cvs sup done' dave.boodman@vtext.com < /dev/null

(2450, ~12mins, supermicro, 27mins, 2950: 7mins)

13. configure new kernel.
for i386:
cd /usr/src/sys/i386/conf
scp backup2:/mnt/data4/build/freebsd/kern_config-7.0-PAE ./mx2

-or-

for amd64:
cd /usr/src/sys/amd64/conf
scp backup2:/mnt/data4/build/freebsd/kern_config-7.1-amd64 ./jail2

-------

edit the kernel config and change ident to be the name of the jail:
vi jail2
ident jail2

edit /sys/conf/newvers.sh to add –jc2 to the end of the BRANCH string (RELEASE-jc2)
vi /sys/conf/newvers.sh

notes: http://www.FreeBSD.org/doc/en_US.ISO8859-1/books/handbook/kernelconfig-config.html

14. bring over patches from backup2

The various patches are in /mnt/data4/build/freebsd/patches on backup2. There are dirs for each version. Not all dirs are populated, but patches for later versions work on older ones unless there is a new patch in the older dir.

cd /usr/src
scp backup2:"/mnt/data4/build/freebsd/patches/7.x/*" .

Apply patches:
patch -l < jls-patch

15. build, install kernel and world

cd /boot

mv kernel kernel.GENERIC
cd kernel.GENERIC
mkdir hold
mv mfi_linux.ko hold/
mv linux.ko hold/
mv linprocfs.ko hold/
mv linsysfs.ko hold/
mv geom_vinum.ko hold/
mv geom_concat.ko hold/
rm *.ko
rm *.symbols
mv hold/* .
rmdir hold/

cd /usr/src
make buildkernel installkernel

make buildworld ; mail -s 'kernel build done' dave.boodman@vtext.com < /dev/null
(2450: 1:56min, supermicro: 59mins, 2950: 38mins)
make installworld
(2450: 3min, supermicro: 1min, 2950: :34)
mergemaster -i
delete /var/tmp/temproot
delete bsnmpd
delete temporary ./etc/hosts
delete temporary ./etc/motd
delete /var/tmp/temproot

cd /sys/modules/geom/geom_vinum
make
make install

16. populate devfs ruleset
scp backup2:/mnt/data4/build/freebsd/devfs.rules /etc

17. populate /etc/rc.conf with IPs and NFS settings
vi /etc/rc.conf

kern_securelevel_enable="NO"
portmap_enable="NO"
sendmail_enable="NO"
usbd_enable="YES"

xntpd_enable="YES"
nfs_client_enable="YES"
nfs_reserved_port_only="YES"
inetd_enable="YES"
inetd_flags="-wW -a 10.1.4.102"
devfs_system_ruleset="devfsrules_show_all"

#ifconfig_xl0="inet 10.1.4.102 netmask 255.255.255.0"
#ifconfig_fxp0="inet 69.55.228.53 netmask 255.255.255.0"
#ifconfig_fxp0_alias0="inet 69.55.2xx.xx netmask 255.255.255.0"

static_routes="t1 office"
route_t1="-net 10.1.5 10.1.4.2"
route_office="-net 10.1.6 10.1.4.2"
gvinum_enable="YES"
fsck_y_enable="YES"
background_fsck="NO"
rc_mfi_raid_tty_log="YES"

18. make sure sysctls are set and preserved after reboot
echo "kern.consmute=0\
kern.ipc.shm_use_phys=1\
kern.ipc.shmall=131070\
kern.ipc.shmmax=134217728\
net.inet.tcp.syncookies=0\
kern.maxfiles=32768\
kern.fallback_elf_brand=3\
kern.maxprocperuid=4000\
security.jail.sysvipc_allowed=1\
security.jail.allow_raw_sockets=1\
security.jail.socket_unixiproute_only=1\
security.jail.chflags_allowed=0\
dev.amr.0.allow_volume_configure=1\
compat.linux.osrelease=2.6.12" >> /etc/sysctl.conf

19. mount procfs
echo "proc /proc procfs rw 0 0" >> /etc/fstab

For Dell 2950/2450:
echo "linprocfs /usr/compat/linux/proc linprocfs rw 0 0" >> /etc/fstab

For Dell 2950:
echo "linsysfs /usr/compat/linux/sys linsysfs rw 0 0" >> /etc/fstab

mkdir -p /usr/compat/linux/proc
mkdir -p /usr/compat/linux/sys

20. reboot. Confirm new kernel is loaded

uname -a

Check devfs rules:
devfs rule showsets
devfs rule -s 3 show

21. update ports:
cd /usr/ports
echo "*default host=cvsup4.FreeBSD.org\
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_7_1\
*default delete use-rel-suffix\
*default compress\
ports-all tag=." > sup

cvsup sup; mail -s 'cvs sup ports done' dave.boodman@vtext.com < /dev/null

(2450: 18mins, supermicro: 19mins; 2950: 24mins)

22. Install raid mgmt tool

For LSI based cards:

install linux_base:
cd /usr/ports/emulators/linux_base-fc4
make install clean
(2450: 7min, supermicro: 3mins, 2950: 14mins)

(for LSI)
cd /usr/ports/sysutils/linux-megamgr
make install clean

cd /usr/ports/sysutils/megarc
make install clean

(for Perc5/i, 6/i)
cd /usr/ports/sysutils/linux-megacli2
make install clean

Test:
rehash; megacli ldinfo lall a0

For adaptec:
On jail18:
scp /compat/linux/usr/sbin/aaccli user@10.1.4.102:~

mv ~user/aaccli /compat/linux/usr/sbin/aaccli

test out;
/compat/linux/usr/sbin/aaccli

For 3w9690:
scp backup2:/d4/build/3ware/tw_cli-freebsd-x86_64-9.5.0.1.tgz /usr/local/sbin
cd /usr/local/sbin xzvf tw_cli-freebsd-x86_64-9.5.0.1.tgz
rm tw_cli-freebsd-x86_64-9.5.0.1.tgz

test out;
./tw_cli /c0 show allstatus

23. install rsync from ports
cd /usr/ports/net/rsync
make install clean

choose default options

25. configure inetd to respond to mrtg load queries
echo "load stream tcp nowait user /usr/local/jail/bin/load.pl load.pl" >> /etc/inetd.conf

echo "load 12384/tcp" >> /etc/services

26. install bb client
adduser
Username: bb
Full name: bb
Uid (Leave empty for default): 1984
Login group [bb]:
Login group is bb. Invite bb into other groups? []:
Login class [default]:
Shell (sh csh tcsh nologin) [sh]:
Home directory [/home/bb]:
Use password-based authentication? [yes]:
Use an empty password? (yes/no) [no]:
Use a random password? (yes/no) [no]: yes
Lock out the account after creation? [no]:
Username : bb
Password : <random>
Full Name : bb
Uid : 1984
Class :
Groups : bb
Home : /home/bb
Shell : /bin/sh
Locked : no
OK? (yes/no): yes

cd /usr/home/bb
scp backup2:/mnt/data4/build/bb/bb-freebsd_linuxcompat.tgz .
tar xzf bb-freebsd_linuxcompat.tgz

edit /home/bb/bbc1.9e-btf/etc/bb-hosts with something like:
echo "10.1.4.5 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh \
10.1.4.102 jail2.johncompanies.com # ssh" > /home/bb/bbc1.9e-btf/etc/bb-hosts

vi /home/bb/bbc1.9e-btf/ext/openfiles
MACHINE="jail2,johncompanies,com" # HAS TO BE IN A,B,C FORM

cd /usr/home/bb/bbc1.9e-btf/etc
./bbchkcfg.sh
(y to questions)
./bbchkhosts.sh
(ignore ssh errors)
cd ../..
chown -R bb .
su bb
cd
cd bbc1.9e-btf
./runbb.sh start
more BBOUT
(look for errors)
exit

echo 'su - bb -c "cd /home/bb/bbc1.9e-btf; ./runbb.sh start"' > /usr/local/etc/rc.d/bb.sh
chmod +x /usr/local/etc/rc.d/bb.sh

NOTE: to get bb working on amd, had to copy over bin dir from linux dist

27. configure load mrtg, on mail
vi /usr/local/www/mgmt/mrtg/mrtg1.cfg
(add new entry to file following existing format)

28. configure bb on mail:
vi /usr/home/bb/bbsrc/bb1.9e-btf/etc/bb-hosts
10.1.4.102 jail2.johncompanies.com # ssh

su bb
cd
bbsrc/bb/runbb.sh restart ; exit

29. remove reserve space, enable softupdates (probably already set, so not necessary)
NOT APPLICABLE IF USING GVINUM
cd
umount /mnt/data1
umount /mnt/data2
tunefs -m 0 /mnt/data1
tunefs -m 0 /mnt/data2
mount -a

30. configure ntp
echo "server 10.1.4.5" > /etc/ntp.conf

/usr/sbin/ntpd -p /var/run/ntpd.pid
sleep 2; ntpq -p
(confirm it’s able to reach our time server)

32. fwd and reverse lookups on ns1c
vr johncompanies.com
vi internal.johncompanies.com
rndc reload johncompanies.com IN private
(edit the PTR too)

33. create gvinum volumes

Make a g partition:

2950:
bsdlabel -e /dev/mfid0s1
bsdlabel -e /dev/mfid1s1
bsdlabel -e /dev/mfid2s1

given:
# /dev/aacd0s1:
8 partitions:
# size offset fstype [fsize bsize bps/cpg]
a: 262144 0 4.2BSD 2048 16384 16392
b: 4194304 262144 swap
c: 143363997 0 unused 0 0 # "raw" part, don't edit
d: 524288 4456448 4.2BSD 2048 16384 32776
e: 524288 4980736 4.2BSD 2048 16384 32776
f: 6291456 5505024 4.2BSD 2048 16384 28552

new offset = 6291456 + 5505024 = 11796480
new size is size for 'c' partition minus the new start from above
143363997 - 11796480 = 131567517
So:
g: 131567517 11796480 unused 0 0

bsdlabel -e /dev/amrd1s1
change d to g

For a 146G drive (after OS and 4G swap), we can fit 127 1G volumes so:

echo 'drive data1 device /dev/mfid0s1g' > /tmp/cgv

echo '#\!/bin/sh\
i="1"\
while [ $i -le 127 ]\
do\
echo "volume v$i" >> /tmp/cgv;\
echo 'plex org concat' >> /tmp/cgv;\
echo 'sd length 1g drive data1' >> /tmp/cgv;\
i=`expr $i + 1`\
done' > /tmp/mkcgv
sh /tmp/mkcgv
gvinum create /tmp/cgv

volume 1
plex org concat
sd length 3g drive data1
volume 2
plex org concat
sd length 5g drive data1
volume 3
plex org concat
sd length 8g drive data1
volume 4
plex org concat
sd length 6g drive data1
volume 5
plex org concat
sd length 10g drive data1

for f in 1 2 3 4 5; do bsdlabel -rw /dev/gvinum/$f; done
for f in 1 2 3 4 5; do newfs /dev/gvinum/${f}a; done
for f in 1 2 3 4 5; do mkdir /mnt/data1/$f; mount /dev/gvinum/${f}a /mnt/data1/$f;done
for f in 1 2 3 4 5; do dd if=/dev/zero of=/mnt/data1/$f/this_is_$f bs=1m count=$f; done
for f in 1 2 3 4 5; do ll -h /mnt/data1/$f/; done

For a 300G drive (after OS and 4G swap), we can fit 273 1G volumes so:

echo 'drive data2 device /dev/mfid1s1g' > /tmp/cgv

echo '#\!/bin/sh\
i="128"\
while [ $i -le 401 ]\
do\
echo "volume v$i" >> /tmp/cgv;\
echo 'plex org concat' >> /tmp/cgv;\
echo 'sd length 1g drive data2' >> /tmp/cgv;\
i=`expr $i + 1`\
done' > /tmp/mkcgv
sh /tmp/mkcgv
gvinum create /tmp/cgv

For a 300G drive (after OS and 4G swap), we can fit 273 1G volumes so:

echo 'drive data3 device /dev/mfid2s1g' > /tmp/cgv

echo '#\!/bin/sh\
i="402"\
while [ $i -le 675 ]\
do\
echo "volume v$i" >> /tmp/cgv;\
echo 'plex org concat' >> /tmp/cgv;\
echo 'sd length 1g drive data3' >> /tmp/cgv;\
i=`expr $i + 1`\
done' > /tmp/mkcgv
sh /tmp/mkcgv
gvinum create /tmp/cgv

Sometimes there’s 2040m leftover so create one more smaller vol:
echo 'volume v64\
plex org concat\
sd length 2040m drive data1' >> /tmp/cgv

To delete:

echo '#\!/bin/sh\
i="1"\
while [ $i -le 127 ]\
do\
echo "gvinum rm -r v$i" >> /tmp/dgv;\
i=`expr $i + 1`\
done' > /tmp/mkdgv
sh /tmp/mkdgv
sh /tmp/dgv

dd if=/dev/zero of=/dev/sdb seek=285474816 bs=512
dd if=/dev/zero of=/dev/mfid0s2 bs=1m count=1000 (do this to all drives)

35. create the jail template

touch /mnt/data1/jail
mdconfig -a -t vnode -s 1g -f /mnt/data1/jail -u 0
bsdlabel -rw md0
newfs4x /dev/md0c
mkdir /mnt/data1/jail-DIR
mount /dev/md0c /mnt/data1/jail-DIR

cd /usr/ports/sysutils/jailutils
make install clean

cd /usr/src
make world DESTDIR=/mnt/data1/jail-DIR
(2450: 2:28mins, supermicro: 55mins, 2950: 1h )
cd etc
make distribution DESTDIR=/mnt/data1/jail-DIR
mount -t devfs devfs /mnt/data1/jail-DIR/dev
devfs -m /mnt/data1/jail-DIR/dev rule -s 3 applyset
cd /mnt/data1/jail-DIR
ln -sf dev/null kernel
cp /usr/local/sbin/jkill /mnt/data1/jail-DIR/sbin

jail /mnt/data1/jail-DIR testhostname 192.168.11.100 /bin/sh
csh
touch /etc/fstab
echo 'network_interfaces=""\
hostname="newsystem"\
kern_securelevel_enable="NO"\
sendmail_enable="YES"\
sshd_enable="YES"' > /etc/rc.conf

echo "nameserver 69.55.225.225\
nameserver 69.55.230.3" >> /etc/resolv.conf

vi /etc/crontab
remove the adjkerntz lines
comment out periodic’s and put this line above them:
# IF YOU UNCOMMENT THESE, PLEASE ADJUST THEIR RUN TIME

rm -rf /etc/periodic/daily/400.status-disks

check /tmp for crap

vi /etc/periodic/security/100.chksetuid
replace: MP=`mount -t ufs | grep -v " nosuid" | awk '{ print $3 }' | sort`
with: MP='/' (use single quotes)

mkdir -p /usr/compat/linux/dev

adduser (Add account for user, make sure in wheel group)

user/root passwd: 8ico2987

Set root password

vi /usr/home/user/.profile (and add to the file):
TERM=vt100; export TERM

tzsetup

newaliases

rm /sbin/halt /sbin/reboot
ln /sbin/jkill /sbin/halt
ln /sbin/jkill /sbin/reboot

#cd /dev
#rm console
#ln -s null console

vi /etc/syslog.conf (comment out console and move to /var/log/messages):
#*.err;kern.warning;auth.notice;mail.crit /dev/console *.err;kern.warning;auth.notice;mail.crit /var/log/messages

exit
exit

cd libexec
mv ld-elf32.so.1 ld-elf32.so.1-orig
ln ld-elf.so.1 ld-elf32.so.1

may have to run:
chflags noschg ld-elf32.so.1
chflags noschg ld-elf.so.1
then
chflags schg ld-elf.so.1
chflags schg ld-elf32.so.1

cd /usr/ports
make -DNOCLEANDEPENDS clean
(2450: 15mins , supermicro: 29mins, 2950: 18mins)
rm -fr /usr/ports/distfiles/*
cp -r /usr/ports /mnt/data1/jail-DIR/usr (2450: 2:00 mins , supermicro: 15mins, 2950: 3mins)

rm /mnt/data1/jail-DIR/root/.history

cd
dump -0a -f /usr/local/jail/template/71template /dev/md0c
umount /mnt/data1/jail-DIR/dev
umount /dev/md0c
rmdir /mnt/data1/jail-DIR

36. setup backups
echo '#\!/bin/sh\
backupdir=/data/jail2\
server=backup1\
\
## ENTRY /etc\
## ENTRY /usr/local/etc\
## ENTRY /usr/local/jail\
## ENTRY /root/logs' > /usr/local/jail/bin/backup.config

on backup1:
setup backup dirs:
ssh backup1 mkdir -p /data/jail2/0

add the system to
vi /usr/local/sbin/snapshot_rotate

on mail:
vi /usr/local/www/mgmt/cgi/backupgraph.pl
(add hostname)

Edit /usr/local/jail/bin/backup.xxx to use the right drives and copy to /usr/local/jail/bin/backup

37. mkdir /root/logs

38. edit sshd_config for security
vi /etc/ssh/sshd_config
ListenAddress 69.55.228.53
ListenAddress 10.1.4.102

kill -1 `cat /var/run/sshd.pid`

39. add crontab entries
crontab -e
5 0 * * * /usr/local/jail/bin/backup.md
1 0 1 * * /usr/local/jail/bin/ipfwreset
0 18 * * * /usr/local/jail/bin/ipfwbackup
4,9,14,19,24,29,34,39,44,49,55,59 * * * * /usr/local/jail/bin/trafstats

On 2950 with Perc5/i also add:
*/5 * * * * /usr/local/jail/bin/perc5iraidchk

40. Reboot notify script
ln -s /usr/local/jail/bin/notify.sh /usr/local/etc/rc.d/notify.sh

41. add to management db (on mail and devweb) jc.ref_machines and jc.ref_templates

uname -r
6.2-RELEASE-jc1

insert into ref_machines values (null,'mx2','mx2.johncompanies.com',0,'m');
select machine_id from ref_machines where host='mx2';
+------------+
| machine_id |
+------------+
| 35 |
+------------+
insert into ref_templates values ('',' 7.1-RELEASE-jc2',44,'FreeBSD 7.1',0);

42. add to server/cabinet map. On mail:
vi /usr/local/www/mgmt/html/cabinetmap.html

43. add an outside blocking rule to the firewall, so this machine can only be reached from inside the firewall. Follow example already in firewall jail17 is:

00117 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 or 69.55.238.150 } to 69.55.228.200
00117 deny ip from any to 69.55.228.200

jail19 would be 00119...
ipfw add 00107 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 or 69.55.238.150 } to 69.55.230.108
ipfw add 00107 deny ip from any to 69.55.220.108

44. select customers for probe map

46. make gv start on boot

scp backup2:/mnt/data4/build/freebsd/gvinum /etc/rc.d/gvinum

If they start stale:

echo '#\!/bin/sh\
i="1"\
while [ $i -le 127 ]\
do\
gvinum start v$i\
i=`expr $i + 1`\
done' > /tmp/startgv

sh /tmp/startgv

47. patch jail against starting jails with rtprio

mv /usr/sbin/jail /usr/sbin/jail_
echo '#\!/bin/sh\
/usr/sbin/rtprio -t /usr/sbin/jail_ $*' > /usr/sbin/jail
chmod +x /usr/sbin/jail

48. make sure mail works
If there are map errors:
cd /etc/mail; make maps

49. move and pare down generic kernel
mv kernel.old/ kernel.GENERIC

/ too full, remove old kernel modules except for:

Id Refs Address Size Name
1 14 0xffffffff80100000 ac6c08 kernel
2 1 0xffffffff80bc8000 1128 mfi_linux.ko
3 4 0xffffffff80bca000 39138 linux.ko
4 1 0xffffffffb48f8000 358c linprocfs.ko
5 1 0xffffffffb48fc000 9d3 linsysfs.ko
6 1 0xffffffffb490c000 80ee8 zfs.ko

50. ntp doesn’t seem to start from rc so,

echo '/usr/sbin/ntpd -c /etc/ntp.conf -p /var/run/ntpd.pid -f /var/db/ntpd.drift' > /usr/local/etc/rc.d/ntp.sh
chmod 0700 /usr/local/etc/rc.d/ntp.sh</pre>

= FreeBSD 7.2 =

Last updated 2009-06-18

<pre>All time extimates below assume disks aren’t scrubbing. Setup instructions below are for LSI card:

1. make sure bios is setup for bios console redirect
2950:
Console redirection:
LCD string..
Date to GMT

2. assuming mirrors (or at least disks) created (if not, refer to this), boot to disk 1 of 7.2
skip kernel config (enter)
custom install

partition ->
move cursor to mfid0, hit space (takes you to partition map screen)
a for entire disk
q to quit and save

standard mbr (no boot manager)
space to unselect mfid0
cursor over mfid1
space
a for entire disk
q to quit and save
none (leave untouched)

cursor over mfid0
space
(takes you into part. Screen again) q to exit
none
Make sure both are checked and tab to ok

Label ->
Make sure mfid0 is highlighted

/ 256M
swap 2G (for 2950 make it 4G)
/var 256M (4.6G)
/tmp 256M
/usr 4G
/mnt/data1 remaining space (no need to newfs)

Make sure to toggle S for soft updates on all (should look like UFS2+S Y under the Newfs column)

move cursor to mfid1
swap 8G (or 4G if there’s a 3rd drive)
/mnt/data2 remaining space (no need to newfs)

q to save and exit

distributions ->
developer
yes to install ports
exit

media ->
cd

commit ->
yes
(2450: 14mins, supermicro: 12mins)

yes to "visit general config" ->
Set root pwd

Add user ‘user’ member group is wheel, set password

Set tz

Networking->interfaces->Fxp0 ->
No IPV6
dhcp=no
Set hostname & domain
Enable sshd

exit...
exit install ->
yes

take the cd out and let the machine reboot

3. double check the date/time

4. edit /etc/make.conf
echo "WITHOUT_X11=yes \
KERNCONF=jail8 \
BOOT_COMCONSOLE_SPEED=115200" >> /etc/make.conf

5. add settings to /boot/loader.conf and /boot.config

echo "-Dh" >> /boot.config

echo 'console="comconsole,vidconsole" \
boot_multicons="YES" \
boot_serial="YES" \
mfi_linux_load="YES" \
comconsole_speed="115200"' >> /boot/loader.conf

6. turn off all ttyv's except 0 and 1 in /etc/ttys
also turn on ttyd0, change type to vt100:
vi /etc/ttys

ttyv2 "/usr/libexec/getty Pc" cons25 off secure
ttyv3 "/usr/libexec/getty Pc" cons25 off secure
ttyv4 "/usr/libexec/getty Pc" cons25 off secure
ttyv5 "/usr/libexec/getty Pc" cons25 off secure
ttyv6 "/usr/libexec/getty Pc" cons25 off secure
ttyv7 "/usr/libexec/getty Pc" cons25 off secure

ttyd0 "/usr/libexec/getty std.9600" vt100 on secure

kill -1 1

on console server:
vi /etc/remote
(rename port to jail18 depending on where and which digi plugged into)
test serial console

7. populate hosts
echo "10.1.4.3 backup2" >> /etc/hosts
echo "10.1.4.8 backup1" >> /etc/hosts

8. put key in authorized_keys on backup2
cd
ssh-keygen -t dsa -b 1024
(default location, leave password blank)

cat /root/.ssh/id_dsa.pub | ssh backup2 'cat - >> /root/.ssh/authorized_keys'
cat /root/.ssh/id_dsa.pub | ssh backup1 'cat - >> /root/.ssh/authorized_keys'

confirm that you can ssh to backup2 and backup1 without getting a login prompt

ssh backup2 hostname

ssh backup1 hostname

9. create & populate binaries/scripts dirs
mkdir -p /usr/local/jail/bin
mkdir -p /usr/local/jail/rc.d
mkdir -p /usr/local/jail/template/
mkdir /mnt/data1
mkdir /mnt/data2
scp backup2:"/mnt/data4/bin/freebsd7.x/*" /usr/local/jail/bin
cd /usr/local/jail/rc.d/
touch quad1
touch deprecated
chmod +x *
cd /usr/local/jail/bin
ln -s /usr/local/jail/rc.d/quad1 quad1
ln -s /usr/local/jail/bin/jailmake_md jailmake
ln -s /usr/local/jail/bin/js_md js
ln -s /usr/local/jail/bin/canceljail_md canceljail
ln -s /usr/local/jail/bin/jailmakeempty_md jailmakeempty
ln -s /usr/local/jail/bin/postboot_md postboot

rehash

10. edit root's path and login script:
vi /root/.cshrc

Change alias entries (add G):
alias la ls -aG
alias lf ls -FAG
alias ll ls -lAG
alias ls ls -AG
alias mbm mb mount
alias mbu mb umount

add to path:
/usr/local/jail/bin
(if adaptec card installed, also add /compat/linux/usr/sbin)

and alter the prompt, set the following:
set prompt = "`/bin/hostname -s` %/# "

at the bottom of the file add:
set sshtty=`who am i|awk '{print $2}'`
/usr/sbin/rtprio 3 -`psj | grep $sshtty | awk '{print $2}'`

set shortty=`who am i | awk '{print $2}' | sed -E 's/.*(..)$/\1/'`
foreach x (`psj | grep sh | grep $shortty | awk '{print $2}'`)
/usr/sbin/rtprio 2 -$x
end

To load the new file:
source /root/.cshrc

11. install cvsup
cd /usr/ports/net/cvsup-without-gui
make install clean; rehash; mail -s 'cvs installed' dave.boodman@vtext.com < /dev/null

(stay close for gettext options, 2450: 27mins, supermicro: 17mins, 2950: 22mins)

12. get latest sources for this release:
cd /usr/src
echo "*default host=cvsup4.freebsd.org\
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_7_2\
*default delete use-rel-suffix\
*default compress\
src-all" > sup

-OR-

echo "*default host=cvsup4.freebsd.org\
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_7\
*default delete use-rel-suffix\
*default compress\
src-all" > sup

(stable)

cvsup sup ; mail -s 'cvs sup done' dave.boodman@vtext.com < /dev/null

(2450, ~12mins, supermicro, 27mins, 2950: 7mins)

13. configure new kernel.

cd /usr/src/sys/amd64/conf
scp backup2:/mnt/data4/build/freebsd/kern_config-7.1-amd64 ./jail8

edit the kernel config and change ident to be the name of the jail:
vi jail8
ident jail8

edit /sys/conf/newvers.sh to add –jc2 to the end of the BRANCH string (RELEASE-jc2)
vi /sys/conf/newvers.sh

notes: http://www.FreeBSD.org/doc/en_US.ISO8859-1/books/handbook/kernelconfig-config.html

14. bring over patches from backup2

The various patches are in /mnt/data4/build/freebsd/patches on backup2. There are dirs for each version. Not all dirs are populated, but patches for later versions work on older ones unless there is a new patch in the older dir.

cd /usr/src
scp backup2:"/mnt/data4/build/freebsd/patches/7.2/*" .

Apply patches:
patch -l < jls-patch

15. build, install kernel and world

cd /boot

mv kernel kernel.GENERIC
cd kernel.GENERIC
mkdir hold
mv mfi_linux.ko hold/
mv linux.ko hold/
mv linprocfs.ko hold/
mv linsysfs.ko hold/
mv geom_vinum.ko hold/
mv geom_concat.ko hold/
rm *.ko
rm *.symbols
mv hold/* .
rmdir hold/

cd /usr/src
make buildkernel installkernel

make buildworld ; mail -s 'buildworld done' dave.boodman@vtext.com < /dev/null
(2450: 1:56min, supermicro: 59mins, 2950: 38mins)
make installworld
(2450: 3min, supermicro: 1min, 2950: :34)
mergemaster -i
delete /var/tmp/temproot
delete bsnmpd
delete temporary ./etc/hosts
delete temporary ./etc/motd
delete /var/tmp/temproot

cd /sys/modules/geom/geom_vinum
make
make install

16. populate devfs ruleset
scp backup2:/mnt/data4/build/freebsd/devfs.rules /etc

17. populate /etc/rc.conf with IPs and NFS settings
vi /etc/rc.conf

kern_securelevel_enable="NO"
portmap_enable="NO"
sendmail_enable="NO"
usbd_enable="YES"

xntpd_enable="YES"
nfs_client_enable="YES"
nfs_reserved_port_only="YES"
inetd_enable="YES"
inetd_flags="-wW -a 10.1.4.102"
devfs_system_ruleset="devfsrules_show_all"

#ifconfig_xl0="inet 10.1.4.102 netmask 255.255.255.0"
#ifconfig_fxp0="inet 69.55.228.53 netmask 255.255.255.0"
#ifconfig_fxp0_alias0="inet 69.55.2xx.xx netmask 255.255.255.0"

static_routes="t1 office"
route_t1="-net 10.1.5 10.1.4.2"
route_office="-net 10.1.6 10.1.4.2"
gvinum_enable="YES"
fsck_y_enable="YES"
background_fsck="NO"
rc_mfi_raid_tty_log="YES"

18. make sure sysctls are set and preserved after reboot
echo "kern.consmute=0\
kern.ipc.shm_use_phys=1\
kern.ipc.shmall=131070\
kern.ipc.shmmax=134217728\
net.inet.tcp.syncookies=0\
kern.maxfiles=32768\
kern.fallback_elf_brand=3\
kern.maxprocperuid=4000\
security.jail.sysvipc_allowed=1\
security.jail.allow_raw_sockets=1\
security.jail.socket_unixiproute_only=1\
security.jail.chflags_allowed=0\
dev.amr.0.allow_volume_configure=1\
compat.linux.osrelease=2.6.12\
vm.pmap.shpgperproc=300" >> /etc/sysctl.conf

19. mount procfs
echo "proc /proc procfs rw 0 0" >> /etc/fstab

For Dell 2950/2450:
echo "linprocfs /usr/compat/linux/proc linprocfs rw 0 0" >> /etc/fstab

For Dell 2950:
echo "linsysfs /usr/compat/linux/sys linsysfs rw 0 0" >> /etc/fstab

mkdir -p /usr/compat/linux/proc
mkdir -p /usr/compat/linux/sys

19. enable noatime option
NOT APPLICABLE IF RUNNING GVINUM
data1 and data2 should look something like:
/dev/amrd0s1g /mnt/data1 ufs rw,noatime 2 2

20. reboot. Confirm new kernel is loaded

uname -a

Check devfs rules:
devfs rule showsets
devfs rule -s 3 show

21. update ports:
cd /usr/ports
echo "*default host=cvsup4.FreeBSD.org\
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_7_2\
*default delete use-rel-suffix\
*default compress\
ports-all tag=." > sup

cvsup sup; mail -s 'cvs sup ports done' dave.boodman@vtext.com < /dev/null

(2450: 18mins, supermicro: 19mins; 2950: 24mins)

22. Install raid mgmt tool

For LSI based cards:

install linux_base:
cd /usr/ports/emulators/linux_base-fc4
make install clean
(2450: 7min, supermicro: 3mins, 2950: 14mins)

(for LSI)
cd /usr/ports/sysutils/linux-megamgr
make install clean

cd /usr/ports/sysutils/megarc
make install clean

(for Perc5/i, 6/i)
cd /usr/ports/sysutils/linux-megacli2
make install clean

Test:
rehash; megacli ldinfo lall a0

For adaptec:
On jail18:
scp /compat/linux/usr/sbin/aaccli user@10.1.4.102:~

mv ~user/aaccli /compat/linux/usr/sbin/aaccli

test out;
/compat/linux/usr/sbin/aaccli

For 3w9690:
scp backup2:/d4/build/3ware/tw_cli-freebsd-x86_64-9.5.0.1.tgz /usr/local/sbin
cd /usr/local/sbin xzvf tw_cli-freebsd-x86_64-9.5.0.1.tgz
rm tw_cli-freebsd-x86_64-9.5.0.1.tgz

test out;
./tw_cli /c0 show allstatus

23. install rsync from ports
cd /usr/ports/net/rsync
make install clean

choose default options

25. configure inetd to respond to mrtg load queries
echo "load stream tcp nowait user /usr/local/jail/bin/load.pl load.pl" >> /etc/inetd.conf

echo "load 12384/tcp" >> /etc/services

26. install bb client
adduser
Username: bb
Full name: bb
Uid (Leave empty for default): 1984
Login group [bb]:
Login group is bb. Invite bb into other groups? []:
Login class [default]:
Shell (sh csh tcsh nologin) [sh]:
Home directory [/home/bb]:
Use password-based authentication? [yes]:
Use an empty password? (yes/no) [no]:
Use a random password? (yes/no) [no]: yes
Lock out the account after creation? [no]:
Username : bb
Password : <random>
Full Name : bb
Uid : 1984
Class :
Groups : bb
Home : /home/bb
Shell : /bin/sh
Locked : no
OK? (yes/no): yes

cd /usr/home/bb
scp backup2:/mnt/data4/build/bb/bb-freebsd_linuxcompat.tgz .
tar xzf bb-freebsd_linuxcompat.tgz

edit /home/bb/bbc1.9e-btf/etc/bb-hosts with something like:
echo "10.1.4.5 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh \
10.1.4.108 jail8.johncompanies.com # ssh" > /home/bb/bbc1.9e-btf/etc/bb-hosts

vi /home/bb/bbc1.9e-btf/ext/openfiles
MACHINE="jail8,johncompanies,com" # HAS TO BE IN A,B,C FORM

cd /usr/home/bb/bbc1.9e-btf/etc
./bbchkcfg.sh
(y to questions)
./bbchkhosts.sh
(ignore ssh errors)
cd ../..
chown -R bb .
su bb
cd
cd bbc1.9e-btf
./runbb.sh start
more BBOUT
(look for errors)
exit

echo 'su - bb -c "cd /home/bb/bbc1.9e-btf; ./runbb.sh start"' > /usr/local/etc/rc.d/bb.sh
chmod +x /usr/local/etc/rc.d/bb.sh

NOTE: to get bb working on amd, had to copy over bin dir from linux dist

27. configure load mrtg, on mail
vi /usr/local/www/mgmt/mrtg/mrtg1.cfg
(add new entry to file following existing format)

28. configure bb on mail:
vi /usr/home/bb/bbsrc/bb1.9e-btf/etc/bb-hosts
10.1.4.102 jail2.johncompanies.com # ssh

su bb
cd
bbsrc/bb/runbb.sh restart ; exit

29. remove reserve space, enable softupdates (probably already set, so not necessary)
NOT APPLICABLE IF USING GVINUM
cd
umount /mnt/data1
umount /mnt/data2
tunefs -m 0 /mnt/data1
tunefs -m 0 /mnt/data2
mount -a

30. configure ntp
echo "server 10.1.4.5" > /etc/ntp.conf

/usr/sbin/ntpd -p /var/run/ntpd.pid
sleep 2; ntpq -p
(confirm it’s able to reach our time server)

32. fwd and reverse lookups on ns1c
vr johncompanies.com
vi internal.johncompanies.com
rndc reload johncompanies.com IN private
(edit the PTR too)

35. create the jail template

touch /mnt/data1/jail
mdconfig -a -t vnode -s 1g -f /mnt/data1/jail -u 0
bsdlabel -rw md0
newfs4x -i 4096 /dev/md0c
mkdir /mnt/data1/jail-DIR
mount /dev/md0c /mnt/data1/jail-DIR

cd /usr/ports/sysutils/jailutils
make install clean

cd /usr/src
make world DESTDIR=/mnt/data1/jail-DIR; pagedave
(2450: 2:28mins, supermicro: 55mins, 2950: 1h )
cd etc
make distribution DESTDIR=/mnt/data1/jail-DIR
mount -t devfs devfs /mnt/data1/jail-DIR/dev
devfs -m /mnt/data1/jail-DIR/dev rule -s 3 applyset
cd /mnt/data1/jail-DIR
ln -sf dev/null kernel
cp /usr/local/sbin/jkill /mnt/data1/jail-DIR/sbin

jail /mnt/data1/jail-DIR testhostname 192.168.11.100 /bin/sh
csh
touch /etc/fstab
echo 'network_interfaces=""\
hostname="newsystem"\
kern_securelevel_enable="NO"\
sendmail_enable="YES"\
sshd_enable="YES"' > /etc/rc.conf

echo "nameserver 69.55.225.225\
nameserver 69.55.230.3" >> /etc/resolv.conf

vi /etc/crontab
remove the adjkerntz lines
comment out periodic’s and put this line above them:
# IF YOU UNCOMMENT THESE, PLEASE ADJUST THEIR RUN TIME

rm -rf /etc/periodic/daily/400.status-disks

check /tmp for crap

vi /etc/periodic/security/100.chksetuid
replace: MP=`mount -t ufs | grep -v " nosuid" | awk '{ print $3 }' | sort`
with: MP='/' (use single quotes)

mkdir -p /usr/compat/linux/dev

adduser (Add account for user, make sure in wheel group)

Username: user
Full name: user
Uid (Leave empty for default):
Login group [user]:
Login group is user. Invite user into other groups? []: wheel
Login class [default]:
Shell (sh csh tcsh nologin) [sh]:
Home directory [/home/user]:
Home directory permissions (Leave empty for default):
Use password-based authentication? [yes]:
Use an empty password? (yes/no) [no]:
Use a random password? (yes/no) [no]: y
Lock out the account after creation? [no]:
Username : user
Password : <random>
Full Name : user
Uid : 1001
Class :
Groups : user
Home : /home/user
Home Mode :
Shell : /bin/sh
Locked : no
OK? (yes/no): y
adduser: INFO: Successfully added (user) to the user database.
adduser: INFO: Password for (user) is: 901gmYjO
Add another user? (yes/no): n
Goodbye!

vi /usr/home/user/.profile (and add to the file):
TERM=vt100; export TERM

tzsetup

newaliases

rm /sbin/halt /sbin/reboot
ln /sbin/jkill /sbin/halt
ln /sbin/jkill /sbin/reboot

#cd /dev
#rm console
#ln -s null console

vi /etc/syslog.conf (comment out console and move to /var/log/messages):
#*.err;kern.warning;auth.notice;mail.crit /dev/console *.err;kern.warning;auth.notice;mail.crit /var/log/messages

exit
exit

cd libexec
chflags noschg ld-elf32.so.1
chflags noschg ld-elf.so.1

mv ld-elf32.so.1 ld-elf32.so.1-orig
ln ld-elf.so.1 ld-elf32.so.1

chflags schg ld-elf.so.1
chflags schg ld-elf32.so.1

cd /usr/ports
make -DNOCLEANDEPENDS clean
(2450: 15mins , supermicro: 29mins, 2950: 18mins)
rm -fr /usr/ports/distfiles/*
cp -r /usr/ports /mnt/data1/jail-DIR/usr (2450: 2:00 mins , supermicro: 15mins, 2950: 3mins)

rm /mnt/data1/jail-DIR/root/.history

cd
umount /mnt/data1/jail-DIR/dev
dump -0a -f /usr/local/jail/template/72template /dev/md0c
umount /dev/md0c
rmdir /mnt/data1/jail-DIR

36. setup backups
echo '#\!/bin/sh\
backupdir=/data/jail2\
server=backup1\
\
## ENTRY /etc\
## ENTRY /usr/local/etc\
## ENTRY /usr/local/jail\
## ENTRY /root/logs' > /usr/local/jail/bin/backup.config

on backup1:
setup backup dirs:
ssh backup1 mkdir -p /data/jail2/0

add the system to
vi /usr/local/sbin/snapshot_rotate

on mail:
vi /usr/local/www/mgmt/cgi/backupgraph.pl
(add hostname)

Edit /usr/local/jail/bin/backup.xxx to use the right drives and copy to /usr/local/jail/bin/backup

37. mkdir /root/logs

38. edit sshd_config for security
vi /etc/ssh/sshd_config
ListenAddress 69.55.228.53
ListenAddress 10.1.4.102

kill -1 `cat /var/run/sshd.pid`

39. add crontab entries
crontab -e
5 0 * * * /usr/local/jail/bin/backup.md
1 0 1 * * /usr/local/jail/bin/ipfwreset
0 18 * * * /usr/local/jail/bin/ipfwbackup
4,9,14,19,24,29,34,39,44,49,55,59 * * * * /usr/local/jail/bin/trafstats

On 2950 with Perc5/i also add:
*/5 * * * * /usr/local/jail/bin/perc5iraidchk

40. Reboot notify script
ln -s /usr/local/jail/bin/notify.sh /usr/local/etc/rc.d/notify.sh

41. add to management db (on mail and devweb) jc.ref_machines and jc.ref_templates

uname -r
6.2-RELEASE-jc1

insert into ref_machines values (null,'mx2','mx2.johncompanies.com',0,'m');
select machine_id from ref_machines where host='mx2';
+------------+
| machine_id |
+------------+
| 35 |
+------------+
insert into ref_templates values ('',' 7.1-RELEASE-jc2',44,'FreeBSD 7.1',0);

42. add to server/cabinet map. On mail:
vi /usr/local/www/mgmt/html/cabinetmap.html

43. add an outside blocking rule to the firewall, so this machine can only be reached from inside the firewall. Follow example already in firewall jail17 is:

00117 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 or 69.55.238.150 } to 69.55.228.200
00117 deny ip from any to 69.55.228.200

jail19 would be 00119...
ipfw add 00107 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 or 69.55.238.150 } to 69.55.230.108
ipfw add 00107 deny ip from any to 69.55.220.108

44. select customers for probe map

47. patch jail against starting jails with rtprio

mv /usr/sbin/jail /usr/sbin/jail_
echo '#\!/bin/sh\
/usr/sbin/rtprio -t /usr/sbin/jail_ $*' > /usr/sbin/jail
chmod +x /usr/sbin/jail

48. make sure mail works
If there are map errors:
cd /etc/mail; make maps

50. ntp doesn’t seem to start from rc so,

echo '/usr/sbin/ntpd -c /etc/ntp.conf -p /var/run/ntpd.pid -f /var/db/ntpd.drift' > /usr/local/etc/rc.d/ntp.sh
chmod 0700 /usr/local/etc/rc.d/ntp.sh

51. recover space on /usr

rm -fr /usr/obj

52. setup fuse

cd /usr/ports/sysutils/fusefs-kmod/
make install

vi /etc/rc.conf
fusefs_enable="YES"

sysctl vfs.usermount=1

cd /usr/ports/sysutils/fusefs-sshfs
make install

sshfs 1005@usw-s009.rsync.net: /mnt/data1/69.55.234.68-col00001-DIR/mnt</pre>

= Stress testing a 6.x jail =

<pre>mkdir /mnt/jail
newfs /dev/gvinum/v1
mount /dev/gvinum/v1 /mnt/jail
cd /mnt/jail
restore -r -f /path/to/61stress .
cd ..
umount /mnt/jail

sh
for f in 1 2 3 4 5 6 7 8 9 10; do mkdir /mnt/data1/$f; done

gconcat label v3-v5 /dev/gvinum/v3 /dev/gvinum/v4 /dev/gvinum/v5
bsdlabel -r -w /dev/concat/v3-v5
newfs /dev/concat/v3-v5a
mount /dev/concat/v3-v5a /mnt/data1/1
bsdlabel -r -w /dev/gvinum/v10
newfs /dev/gvinum/v10a
mount /dev/gvinum/v10a /mnt/data1/2
gconcat label v23-v25 /dev/gvinum/v23 /dev/gvinum/v24 /dev/gvinum/v25
bsdlabel -r -w /dev/concat/v23-v25
newfs /dev/concat/v23-v25a
mount /dev/concat/v23-v25a /mnt/data1/3
bsdlabel -r -w /dev/gvinum/v50
newfs /dev/gvinum/v50a
mount /dev/gvinum/v50a /mnt/data1/4
gconcat label v63-v65 /dev/gvinum/v63 /dev/gvinum/v64 /dev/gvinum/v65
bsdlabel -r -w /dev/concat/v63-v65
newfs /dev/concat/v63-v65a
mount /dev/concat/v63-v65a /mnt/data1/5
bsdlabel -r -w /dev/gvinum/v70
newfs /dev/gvinum/v70a
mount /dev/gvinum/v70a /mnt/data1/6
gconcat label v83-v85 /dev/gvinum/v83 /dev/gvinum/v84 /dev/gvinum/v85
bsdlabel -r -w /dev/concat/v83-v85
newfs /dev/concat/v83-v85a
mount /dev/concat/v83-v85a /mnt/data1/7
bsdlabel -r -w /dev/gvinum/v100
newfs /dev/gvinum/v100a
mount /dev/gvinum//v100a /mnt/data1/8
gconcat label v123-v125 /dev/gvinum/v123 /dev/gvinum/v124 /dev/gvinum/v125
bsdlabel -r -w /dev/concat/v123-v125
mount /dev/concat/v123-v125a /mnt/data1/9
newfs /dev/concat/v123-v125a
mount /dev/concat/v123-v125a /mnt/data1/9
bsdlabel -r -w /dev/gvinum/v130
newfs /dev/gvinum/v130a
mount /dev/gvinum//v130a /mnt/data1/10

sh
for f in 1 2 3 4 5 6 7 8 9 10; do cd /mnt/data1/$f; dump -0a -f - /dev/gvinum/v1| restore -r -f - ; done

sh
for f in 1 2 3 4 5 6 7 8 9 10; do mount_devfs devfs /mnt/data1/$f/dev; devfs -m /mnt/data1/$f/dev rule -s 3 applyset; done

sh
for f in 15 16 17 18 19 20 21 22 23 24; do ifconfig bce0 alias 10.1.6.$f netmask 255.255.255.255; done

jail /mnt/data1/1 testhostname1 10.1.6.15 /bin/sh /etc/rc 2> /dev/null
jail /mnt/data1/2 testhostname2 10.1.6.16 /bin/sh /etc/rc 2> /dev/null
jail /mnt/data1/3 testhostname3 10.1.6.17 /bin/sh /etc/rc 2> /dev/null
jail /mnt/data1/4 testhostname4 10.1.6.18 /bin/sh /etc/rc 2> /dev/null
jail /mnt/data1/5 testhostname5 10.1.6.19 /bin/sh /etc/rc 2> /dev/null
jail /mnt/data1/6 testhostname6 10.1.6.20 /bin/sh /etc/rc 2> /dev/null
jail /mnt/data1/7 testhostname7 10.1.6.21 /bin/sh /etc/rc 2> /dev/null
jail /mnt/data1/8 testhostname8 10.1.6.22 /bin/sh /etc/rc 2> /dev/null
jail /mnt/data1/9 testhostname9 10.1.6.23 /bin/sh /etc/rc 2> /dev/null
jail /mnt/data1/10 testhostname10 10.1.6.24 /bin/sh /etc/rc 2> /dev/null

systat -vmstat</pre>

= Stress testing zfs =

<pre>zfs receive pool1/jail2 < /usr/local/jail/template/template
zfs set mountpoint=/mnt/data1/jail2 pool1/jail2
zfs set quota=4G pool1/jail2
zfs destroy pool1/jail2@now</pre>
install apps that will run when jail started

<pre>cp -r /usr/src /mnt/data1/jail2/usr
rm -fr /mnt/data1/jail2/usr/obj

mount -t devfs devfs /mnt/data1/jail2/dev; devfs -m /mnt/data1/jail2/dev rule -s 3 applyset;

jail /mnt/data1/jail2 stress-test 69.55.234.86 /bin/sh
csh
cd /usr/ports/benchmarks/bonnie
make install clean
cd /usr/ports/sysutils/stress
make install clean
cd /usr/ports/net/rsync
make install clean
cd /usr/ports/lang/perl5.8
make install clean

cd /usr/local/etc/rc.d/

cat >> boot.sh
sleep 30 && sh /usr/local/etc/rc.d/buildworld &
sleep 30 && sh /usr/local/etc/rc.d/portindex &
sleep 30 && sh /usr/local/etc/rc.d/stress &
sleep 30 && sh /usr/local/etc/rc.d/bonnie &

cat > buildworld
#!/bin/sh
while (true); do cd /usr/src; make buildworld; done

cat > portindex
while (true); do cd /usr/ports/; make index; done

cat > stress
/usr/local/bin/stress -c 1 -i 1 -m 1 -d 1 --vm-bytes 768M --hdd-bytes 128M

cat > bonnie
#!/bin/sh
while (true); do /usr/local/bin/bonnie -s 2g; done

chmod +x *

exit
exit

cd
zfs snapshot pool1/jail2@now

sh
for f in 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105; do zfs send pool1/jail2@now | zfs receive pool1/jail$f;
zfs set quota=10G pool1/jail$f;
zfs set mountpoint=/mnt/data1/jail$f pool1/jail$f;
done

for f in 106 107 108 114 115 116 117 118 119 120 121 122 123 125 126 127 128 129 130 131; do zfs send pool1/jail2@now | zfs receive pool2/jail$f
zfs set quota=10G pool2/jail$f;
zfs set mountpoint=/mnt/data2/jail$f pool2/jail$f;
done

for f in 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101; do chmod 0 /mnt/data1/jail$f/usr/local/etc/rc.d/bonnie; done
for f in 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105; do chmod 0 /mnt/data1/jail$f/usr/local/etc/rc.d/stress; done

for f in 115 116 117 118 119 120 121 122 123 125 126 127; do chmod 0 /mnt/data2/jail$f/usr/local/etc/rc.d/bonnie; done
for f in 119 120 121 122 123 125 126 127 128 129 130 131; do chmod 0 /mnt/data2/jail$f/usr/local/etc/rc.d/stress; done

for f in 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105; do mount -t devfs devfs /mnt/data1/jail$f/dev; devfs -m /mnt/data1/jail$f/dev rule -s 3 applyset; done

for f in 106 107 108 114 115 116 117 118 119 120 121 122 123 125 126 127 128 129 130 131; do mount -t devfs devfs /mnt/data2/jail$f/dev; devfs -m /mnt/data2/jail$f/dev rule -s 3 applyset; done

for f in 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105;
do jail /mnt/data1/jail$f testhostname$f 69.55.234.$f /bin/sh /etc/rc 2> /dev/null; done

for f in 106 107 108 114 115 116 117 118 119 120 121 122 123 125 126 127 128 129 130 131;
do jail /mnt/data2/jail$f testhostname$f 69.55.234.$f /bin/sh /etc/rc 2> /dev/null; done

systat -vmstat
sysctl vfs.numvnodes

for f in 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105; do umount /mnt/data1/jail$f/dev; zfs destroy -r pool1/jail$f; done

for f in 106 107 108 114 115 116 117 118 119 120 121 122 123 125 126 127 128 129 130 131; do umount /mnt/data2/jail$f/dev; zfs destroy -r pool2/jail$f; done

for f in 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105; do touch /mnt/data1/jail$f;
mdconfig -a -t vnode -s 10g -f /mnt/data1/jail$f -u $f;
bsdlabel -r -w md$f auto;
newfs -O 1 /dev/md${f}a;
mkdir /mnt/data1/jail$f-DIR;
mount /dev/md${f}a /mnt/data1/jail$f-DIR;
rsync -aSH /mnt/data2/jail2/ /mnt/data1/jail$f-DIR/;
done

for f in 106 107 108 114 115 116 117 118 119 120 121 122 123 125 126 127 128 129 130 131; do touch /mnt/data2/jail$f;
mdconfig -a -t vnode -s 10g -f /mnt/data2/jail$f -u $f;
bsdlabel -r -w md$f auto;
newfs -O 1 /dev/md${f}a;
mkdir /mnt/data2/jail$f-DIR;
mount /dev/md${f}a /mnt/data2/jail$f-DIR;
rsync -aSH /mnt/data1/jail86-DIR/ /mnt/data2/jail$f-DIR/;
done

for f in 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105;
do chmod 0700 /mnt/data1/jail$f-DIR/usr/local/etc/rc.d/bonnie; done
for f in 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105;
do chmod 0700 /mnt/data1/jail$f-DIR/usr/local/etc/rc.d/stress; done

for f in 106 107 108 114 115 116 117 118 119 120 121 122 123 125 126 127 128 129 130 131;
do chmod 0700 /mnt/data2/jail$f-DIR/usr/local/etc/rc.d/bonnie; done
for f in 106 107 108 114 115 116 117 118 119 120 121 122 123 125 126 127 128 129 130 131;
do chmod 0700 /mnt/data2/jail$f-DIR/usr/local/etc/rc.d/stress; done

for f in 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101; do chmod 0 /mnt/data1/jail$f-DIR/usr/local/etc/rc.d/bonnie; done
for f in 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105; do chmod 0 /mnt/data1/jail$f-DIR/usr/local/etc/rc.d/stress; done

for f in 115 116 117 118 119 120 121 122 123 125 126 127; do chmod 0 /mnt/data2/jail$f-DIR/usr/local/etc/rc.d/bonnie; done
for f in 119 120 121 122 123 125 126 127 128 129 130 131; do chmod 0 /mnt/data2/jail$f-DIR/usr/local/etc/rc.d/stress; done

for f in 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105; do mount -t devfs devfs /mnt/data1/jail$f-DIR/dev; devfs -m /mnt/data1/jail$f-DIR/dev rule -s 3 applyset; done

for f in 106 107 108 114 115 116 117 118 119 120 121 122 123 125 126 127 128 129 130 131; do mount -t devfs devfs /mnt/data2/jail$f-DIR/dev; devfs -m /mnt/data2/jail$f-DIR/dev rule -s 3 applyset; done

for f in 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105;
do jail /mnt/data1/jail$f-DIR testhostname$f 69.55.234.$f /bin/sh /etc/rc 2> /dev/null; done

for f in 106 107 108 114 115 116 117 118 119 120 121 122 123 125 126 127 128 129 130 131;
do jail /mnt/data2/jail$f-DIR testhostname$f 69.55.234.$f /bin/sh /etc/rc 2> /dev/null; done

for f in 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105;
do cp /mnt/data1/boot.sh /mnt/data1/jail$f-DIR/usr/local/etc/rc.d/;
cp /mnt/data1/stress /mnt/data1/jail$f-DIR/usr/local/etc/rc.d/;
cp /mnt/data1/bonnie /mnt/data1/jail$f-DIR/usr/local/etc/rc.d/;
done

for f in 106 107 108 114 115 116 117 118 119 120 121 122 123 125 126 127 128 129 130 131;
do cp /mnt/data1/boot.sh /mnt/data2/jail$f-DIR/usr/local/etc/rc.d/;
cp /mnt/data1/stress /mnt/data2/jail$f-DIR/usr/local/etc/rc.d/;
cp /mnt/data1/bonnie /mnt/data2/jail$f-DIR/usr/local/etc/rc.d/;
done

for f in 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105; do umount /mnt/data1/jail$f-DIR/dev; umount /mnt/data1/jail$f-DIR; mdconfig -d -u $f; done

for f in 106 107 108 114 115 116 117 118 119 120 121 122 123 125 126 127 128 129 130 131; do umount /mnt/data2/jail$f-DIR/dev; umount /mnt/data2/jail$f-DIR; mdconfig -d -u $f; done

for f in 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105; do
mkdir /mnt/data1/jail$f-DIR/;
cd /mnt/data1/jail$f-DIR/;
dump -0a -f - /dev/md0a | restore -r -f -;
done

for f in 106 107 108 114 115 116 117 118 119 120 121 122 123 125 126 127 128 129 130 131; do mkdir /mnt/data2/jail$f-DIR/;
cd /mnt/data2/jail$f-DIR/;
dump -0a -f - /dev/md0a | restore -r -f -;
done

for f in 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105; do chmod 0700 /mnt/data1/jail$f-DIR/usr/local/etc/rc.d/*; done

for f in 106 107 108 114 115 116 117 118 119 120 121 122 123 125 126 127 128 129 130 131; do chmod 0700 /mnt/data2/jail$f-DIR/usr/local/etc/rc.d/*; done

--vm-bytes 300M</pre>

= Reloading FreeBSD (while preserving customer data) =

<pre>2. assuming 2 mirrors, boot to disk 1 of 4.11
skip kernel config (enter)
custom install
(skip partition)
label
move cursor to aacd0, hit space (takes you to partition map screen)
given this as the pre-existing partition map:
Filesystem 1K-blocks Used Avail Capacity Mounted on
/dev/aacd0s1a 128990 36888 81784 31% /
/dev/aacd0s1h 30499106 4 28059174 0% /mnt/data1
/dev/aacd1s1e 69526550 4 63964422 0% /mnt/data2
/dev/aacd0s1f 257998 10 237350 0% /tmp
/dev/aacd0s1g 3096462 852082 1996664 30% /usr
/dev/aacd0s1e 257998 1198 236162 1% /var
procfs 4 4 0 100% /proc

relabel and all mount points according to prev settings

or, delete all partitions except for data parts, re-create as necessary

re-toggle newfs on all mount points except for /mnt/data1 - make sure under the Newfs column the data1 (and data2) partiotions look like: UFS+S Y

make sure under the Newfs column the data1 (and data2) partiotions look like
UFS+S N

space to unselect aacd0
cursor over aacd1
space

Space to select both drives and tab to ok

Continue installation as per normal
distributions
developer
yes to install ports
exit
media
cd
commit
yes
(2450: 16mins, supermicro: 11mins)
yes to "visit general config"
Set root pwd
Add user ‘user’ member group is wheel, set password
Set tz
Networking->interfaces->Fxp0
No IPV6
dhcp=yes
Set hostname & domain

exit...
exit install
yes
take the cd out and let the machine reboot</pre>

Jail Server Install

2013-09-25T17:46:29Z

24.165.90.109: /* populate /etc/rc.conf with IPs and service settings */

= FreeBSD 9.x =

== Assumptions ==

Setup instructions below assume this is DELL 2950 with an LSI-based SAS RAID card.

Server is at castle, connected to pub, private, serial and DRAC

Assuming OS loading done via IPKVM with ISO mounted via USB

Assumes at 4 drives, 2 mirrors

== Configure server BIOS ==

setup console redirect, speed 115200

set LCD string to name of server "jail8"

set date to GMT

go into RAID bios and setup mirrors

== Setup DRAC ==
[[DRAC/RMM#DRAC_setup|DRAC setup]]

== Install OS (sysinstall) ==

boot to bootonly disk for AMD version of FreeBSD, i.e. <tt>FreeBSD-8.3-RELEASE-amd64-bootonly.iso</tt>

when the install menu appears, choose <tt>custom install</tt>

=== partition menu ===
move cursor to mfid0, hit space (takes you to partition map screen). If there is only 1 mirror, there will be no option to select a specifit drive: mfid0 will be selected.

type 'a' to use entire disk<br>
type 'q' to quit and save<br>
choose 'freebsd standard mbr'

space to '''un'''select mfid0<br>
cursor down to mfid1<br>
hit space

type 'a' to use entire disk<br>
type 'q' to quit and save<br>
choose 'none' for boot mgr (leave untouched)

cursor over mfid0<br>
space<br>
(takes you into part. Screen again) 'q' to exit<br>
none for boot mgr<br>

Make sure both drives (mfid0 and mfid1) are checked and tab to ok

=== Label menu ===

Make sure mfid0 is highlighted at the top of the screen, setup the following partitions

/ 512M<br>
swap 6G<br>
/var 1G <br>
/tmp 256M<br>
/usr 8G<br>
/mnt/data1 remaining space

All partitions except / should be setup for soft updates. If not, type 's' to enable for soft updates on all except for / (should look like UFS2+S Y under the Newfs column)

move cursor to mfid1 at the top of the screen<br>
swap 8G (or 4G if there’s a 3rd drive)<br>
/mnt/data2 remaining space <br>
'q' to save and exit

=== distributions ===
Choose the following distribudions

* developer (ok to install ports)
* custom -> lib32
exit

=== media ===
if you are installing via a cd, no need to enter this menu or change anything. Otherwise, choose ftp to install via ftp. You will be prompted to setup networking. You will need to choose a nic (typically bce0 or bce1). Say no to DHCP and IPv6. Hopefully the public nic cable was installed in bce0 so start with that nic and provide the hostname, (public) IP, netmask, gateway and DNS. When configured, it should start pinging. If it doesn't, have the NOC swap cables. Select any FTP server, usually Main or ftp4.

=== commit ===

this usually takes 12mins<br>
during the process you may need to select a new ftp mirror, this is not a problem.<br>
at the conclution of the install you will be prompted to enter the root password (2x) and returned to the configuration menu.

=== add user ===

'''Add''' user 'user'. Defaults for everything is fine, just remember to enter 'wheel' in the member group field.
Do set the password.

=== Setup timezone ===

PT

=== Networking ===
page down to the bottom and enable '[X]' sshd

If you installed via cd, you will need to visit:<br>
interfaces->bce0-><br>
No IPV6<br>
dhcp=no<br>
Set hostname, IP, DNS, gateway<br>
(i.e. setup the nic as indicated above)

Exit the install and if you installed via CD, take it out and let the machine reboot

== Configure OS, kernel, userland, jail ==

=== double check the date/time ===

date

=== populate /etc/resolv.conf ===

echo "search johncompanies.com \
nameserver 69.55.225.225 \
nameserver 69.55.230.3 \
nameserver 69.55.229.3" > /etc/resolv.conf

=== edit /etc/make.conf ===
echo "WITHOUT_X11=yes \
KERNCONF=jail4 \
BOOT_COMCONSOLE_SPEED=115200" >> /etc/make.conf

=== setup bootloader for console, etc ===

add settings to /boot/loader.conf and /boot.config:

echo "-Dh" >> /boot.config

echo 'console="comconsole,vidconsole" \
boot_multicons="YES" \
boot_serial="YES" \
mfi_linux_load="YES" \
comconsole_speed="115200"' >> /boot/loader.conf

=== enable login via serial console ===
turn off all ttyv's except 0 and 1 in /etc/ttys and turn on ttyd0, change type to vt100:

vi /etc/ttys

The changed lines should look like:

ttyv2 "/usr/libexec/getty Pc" cons25 off secure
ttyv3 "/usr/libexec/getty Pc" cons25 off secure
ttyv4 "/usr/libexec/getty Pc" cons25 off secure
ttyv5 "/usr/libexec/getty Pc" cons25 off secure
ttyv6 "/usr/libexec/getty Pc" cons25 off secure
ttyv7 "/usr/libexec/getty Pc" cons25 off secure
# Serial terminals
# The 'dialup' keyword identifies dialin lines to login, fingerd etc.
ttyu0 "/usr/libexec/getty std.9600" vt100 on secure

Restart init
kill -1 1

At this point you should have a login on console.

To configure serial console access, login to the console server as root and run:

# vi /etc/remote

following examples there, rename port to server's hostname, depending on where and which digi box this server is plugged into. Make sure to get speed right too: 115200

=== populate hosts ===

If server is at castle:
echo "10.1.4.3 backup2" >> /etc/hosts
echo "10.1.4.8 backup1" >> /etc/hosts

If server is at i2b:
echo "69.55.230.10 backup2" >> /etc/hosts
echo "10.1.2.3 backup3" >> /etc/hosts
echo "69.55.230.11 backup1" >> /etc/hosts

=== create ssh key, upload to backup servers ===
cd
ssh-keygen -t dsa -b 1024
(default location, leave password blank)

If server is at castle:
cat /root/.ssh/id_dsa.pub | ssh backup2 'cat - >> /root/.ssh/authorized_keys'
cat /root/.ssh/id_dsa.pub | ssh backup1 'cat - >> /root/.ssh/authorized_keys'

If server is at i2b:
cat /root/.ssh/id_dsa.pub | ssh backup3 'cat - >> /root/.ssh/authorized_keys'
cat /root/.ssh/id_dsa.pub | ssh backup2 'cat - >> /root/.ssh/authorized_keys'
cat /root/.ssh/id_dsa.pub | ssh backup1 'cat - >> /root/.ssh/authorized_keys'

confirm that you can ssh to backup2 and backup1 without getting a login prompt

ssh backup2 hostname
ssh backup1 hostname
ssh backup3 hostname

=== create & populate binaries/scripts dirs ===
<pre>mkdir -p /usr/local/jail/bin
mkdir -p /usr/local/jail/rc.d
mkdir -p /usr/local/jail/template/
mkdir /mnt/data1
mkdir /mnt/data2
scp backup2:"/mnt/data4/bin/freebsd9.x/*" /usr/local/jail/bin
cd /usr/local/jail/rc.d/
touch quad1
touch deprecated
chmod +x *
cd /usr/local/jail/bin
ln -s /usr/local/jail/rc.d/quad1 quad1
ln -s /usr/local/jail/bin/jailmake_md jailmake
ln -s /usr/local/jail/bin/js_md js
ln -s /usr/local/jail/bin/canceljail_md canceljail
ln -s /usr/local/jail/bin/jailmakeempty_md jailmakeempty
ln -s /usr/local/jail/bin/postboot_md postboot
ln -s /usr/local/jail/bin/preboot_md preboot
ln -s /usr/local/jail/bin/startjail_md startjail
ln -s /usr/local/jail/bin/stopjail_md stopjail

rehash</pre>

=== edit root's path and login script ===

vi /root/.cshrc

Change alias entries (add G):

<pre>alias la ls -aG
alias lf ls -FAG
alias ll ls -lAG
alias ls ls -AG
alias mbm mb mount
alias mbu mb umount
alias cjb cd /usr/local/jail/bin
alias cd1 cd /mnt/data1
alias cd2 cd /mnt/data2
alias cd3 cd /mnt/data3
alias jtop jtop lj
alias j jobs</pre>

add to path be careful to leave a space after bin and make sure the wrapping isn't broken:
/usr/local/jail/bin

alter the prompt, set the following:
set prompt = "$user@`/bin/hostname -s` %/# "

at the bottom of the file add:
<pre>set sshtty=`who am i|awk '{print $2}'`
/usr/sbin/rtprio 3 -`psj | grep $sshtty | awk '{print $2}'`

set shortty=`who am i | awk '{print $2}' | sed -E 's/.*(..)$/\1/'`
foreach x (`psj | grep sh | grep $shortty | awk '{print $2}'`)
/usr/sbin/rtprio 2 -$x
end</pre>

Make the new settings active in current shell:
source /root/.cshrc

=== update ports ===

portsnap fetch
portsnap extract

To update later on:
portsnap fetch
portsnap update

=== install svn ===

<pre>
setenv PACKAGESITE "ftp://ftp4.freebsd.org/pub/FreeBSD/ports/amd64/packages-9-current/Latest/"
pkg_add subversion
</pre>

=== get latest sources for this release ===

<pre>
cd /usr

# mv src/ src.orig
tar cvzf src.orig.tgz src
rm -fr src/*
svn checkout svn://svn.FreeBSD.org/base/stable/9 /usr/src
</pre>

To update:
make update SVN_UPDATE=yes

=== configure new kernel ===

Pull down the kernel config we are using for this distribution. In this case we use an 8.2 kernel config on 8.3, which is valid. The local file should be the same name as host- <tt>jail4</tt> in this example

<pre>cd /usr/src/sys/amd64/conf
scp backup2:/mnt/data4/build/freebsd/kern_config-9.1-amd64 ./jail4</pre>

edit the kernel config and change ident to be the name of the jail:
vi jail4
ident jail4

Optional, edit <tt>/sys/conf/newvers.sh</tt> to add <tt>–jc2</tt> to the end of the BRANCH string (RELEASE-jc2)
vi /sys/conf/newvers.sh

notes on kernel configuring: http://www.FreeBSD.org/doc/en_US.ISO8859-1/books/handbook/kernelconfig-config.html

=== install patches ===

We don't have any patches right now. Refer to older FreeBSD version build docs on how that is/was done.

=== build, install kernel and world ===

Rename current generic kernel so it will always be available to boot from. Save room by removing non-needed kernel modules:

<pre>cd /boot
mv kernel kernel.GENERIC
cd kernel.GENERIC
mkdir hold
mv mfi_linux.ko hold/
mv linux.ko hold/
mv linprocfs.ko hold/
mv linsysfs.ko hold/
mv geom_vinum.ko hold/
mv geom_concat.ko hold/
mv zfs.* hold/
mv opensolaris* hold/

rm *.ko
rm *.symbols
mv hold/* .
rmdir hold/

Note on -DWITHOUT_CLANG: try to do it without including that directive, it may work for you.

cd /usr/src
make buildworld KERNCONF=jail4; mail -s 'buildworld done' dave.boodman@vtext.com < /dev/null

make -DWITHOUT_CLANG buildworld KERNCONF=jail4; mail -s 'buildworld done' dave.boodman@vtext.com < /dev/null
</pre>
~4.5hr

cd /usr/src
make buildkernel installkernel

mergemaster -p
You will be prompted to merge, replace or ignore files changed by the src update. In most cases you can '''d'''elete the temp (new) files.

make -DWITHOUT_CLANG installworld
~34min

ONLY if this will be a zfs system (not currently used in 8.x):
<pre>cd /sys/modules/zfs
make
make install
cd /sys/modules/opensolaris
make
make install</pre>

=== populate devfs ruleset ===
scp backup2:/mnt/data4/build/freebsd/devfs.rules.8x /etc/devfs.rules

=== populate /etc/rc.conf with IPs and service settings ===

<pre>vi /etc/rc.conf

kern_securelevel_enable="NO"
portmap_enable="NO"
sendmail_enable="NO"
usbd_enable="YES"

nfs_client_enable="YES"
nfs_reserved_port_only="YES"
inetd_enable="YES"
inetd_flags="-wW -a 10.1.4.XXX"
devfs_system_ruleset="devfsrules_show_all"

ifconfig_bce1="inet 10.1.4.XXX netmask 255.255.255.0"
ifconfig_bce0="inet 69.55.2XX.XXX''' netmask 255.255.255.0"
#ifconfig_bce0_alias0="inet 69.55.2XX.XXX netmask 255.255.255.0"

fsck_y_enable="YES"
background_fsck="NO"
#rc_mfi_raid_tty_log="YES"
#zfs_enable="YES"</pre>

Modify IPs, hostname, gateway for this box.

=== make sure sysctls are set and preserved upon boot ===

<pre>echo "kern.consmute=0\
kern.ipc.shm_use_phys=1\
kern.ipc.shmall=131070\
kern.ipc.shmmax=134217728\
net.inet.tcp.syncookies=0\
kern.maxfiles=32768\
kern.fallback_elf_brand=3\
kern.maxprocperuid=4000\
security.jail.sysvipc_allowed=1\
security.jail.allow_raw_sockets=1\
security.jail.socket_unixiproute_only=1\
security.jail.chflags_allowed=0\
dev.amr.0.allow_volume_configure=1\
compat.linux.osrelease=2.6.12\
vm.pmap.shpgperproc=500\
security.bsd.unprivileged_read_msgbuf=0\
kern.maxvnodes=400000" >> /etc/sysctl.conf</pre>

Tuning note: watch vfs.numvnodes while the server is live to get guidance on where to set maxvnodes

=== mount procfs ===
echo "proc /proc procfs rw 0 0" >> /etc/fstab
echo "linprocfs /usr/compat/linux/proc linprocfs rw 0 0" >> /etc/fstab
echo "linsysfs /usr/compat/linux/sys linsysfs rw 0 0" >> /etc/fstab
mkdir -p /usr/compat/linux/proc
mkdir -p /usr/compat/linux/sys

=== enable noatime option ===

grep data /etc/fstab

data1 and data2 should look something like (add ',noatime' after 'rw'):

/dev/mfid0s1g /mnt/data1 ufs rw,noatime 2 2
/dev/mfid1s1d /mnt/data2 ufs rw,noatime 2 2

=== reboot. Confirm new kernel is loaded, devfs in place ===

uname -a

Check devfs rules
devfs rule showsets
devfs rule -s 3 show

Should see:
<pre># devfs rule showsets
1
2
3
4

# devfs rule -s 3 show
100 include 1
207 path pts* unhide
217 path fd unhide
218 path fd/* unhide
300 path ttyp* unhide
301 path ttyq* unhide
302 path ttyr* unhide
303 path ttys* unhide
304 path ttyP* unhide
305 path ttyQ* unhide
306 path ttyR* unhide
307 path ttyS* unhide
400 path null unhide
500 path zero unhide
600 path random unhide
610 path urandom unhide
700 path mem unhide
710 path kmem unhide
810 path mdctl unhide
900 path stdin unhide
910 path stdout unhide
920 path stderr unhide</pre>

=== Install raid mgmt tool ===

==== Perc5/i, 6/i ====
Pull over cli from previous system (jail9)
scp root@10.1.4.109:"/usr/local/sbin/mega*" /usr/local/sbin/
scp root@10.1.4.109:/usr/local/libexec/MegaCli /usr/local/libexec/MegaCli

These are linux-based tools. This will require linux base...which you might install via:

pkg_add -r linux_base

Test:
rehash; megacli ldinfo lall a0
or
megarc -ldInfo -a0 -Lall
(2850)

however, linux does seem to be installed already so we don't need to pkg_add or port install or rsync anything over from a current system.

==== 2850 PERC 4e/Di- no linux ====
<pre>cd /usr/ports/distfiles/
fetch http://backup01.best-hosting.ru/pub/FreeBSD/ports/distfiles/dr_freebsd_1.51.zip
cd /usr/ports/sysutils/megarc
make install clean
megarc -dispCfg -a0</pre>

=== install rsync from ports ===
cd /usr/ports/net/rsync
make install clean

choose default options

=== configure inetd to respond to mrtg load queries ===
echo "load stream tcp nowait user /usr/local/jail/bin/load.pl load.pl" >> /etc/inetd.conf
echo "load 12384/tcp" >> /etc/services

=== install perl ===
cd /usr/ports/lang/perl5.12
make install clean

choose defaults

=== install bb client ===

Compiling from source on AMD64 will not work. So, we use a linux-compiled version and rely on linux compat.

adduser

Output/response:

<pre>Username: bb
Full name: bb
Uid (Leave empty for default): 1984
Login group [bb]:
Login group is bb. Invite bb into other groups? []:
Login class [default]:
Shell (sh csh tcsh nologin) [sh]:
Home directory [/home/bb]:
Use password-based authentication? [yes]:
Use an empty password? (yes/no) [no]:
Use a random password? (yes/no) [no]: yes
Lock out the account after creation? [no]:
Username : bb
Password : <random>
Full Name : bb
Uid : 1984
Class :
Groups : bb
Home : /home/bb
Shell : /bin/sh
Locked : no
OK? (yes/no): yes</pre>

cd /usr/home/bb
scp backup2:/mnt/data4/build/bb/bb-freebsd_linuxcompat.tgz .
tar xzf bb-freebsd_linuxcompat.tgz

edit /home/bb/bbc1.9e-btf/etc/bb-hosts with something like:

echo "10.1.4.5 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh \
10.1.4.'''104 jail4'''.johncompanies.com # ssh" > /home/bb/bbc1.9e-btf/etc/bb-hosts

Edit for machine name and private IP.

if this machine is at i2b:
echo "69.55.230.2 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh \
69.55.228.104 jail4.johncompanies.com # ssh" > /home/bb/bbc1.9e-btf/etc/bb-hosts

<pre>vi /home/bb/bbc1.9e-btf/ext/openfiles

MACHINE="jail4,johncompanies,com" # HAS TO BE IN A,B,C FORM</pre>
Edit for machine name.

<pre>cd /usr/home/bb/bbc1.9e-btf/etc
./bbchkcfg.sh
(y to questions)
./bbchkhosts.sh
(ignore ssh errors)
cd ../..
chown -R bb .
su bb
cd
cd bbc1.9e-btf
./runbb.sh start
more BBOUT
(look for errors)
exit</pre>

Put in script to start bb @ boot:
echo 'su - bb -c "cd /home/bb/bbc1.9e-btf; ./runbb.sh start"' > /usr/local/etc/rc.d/bb.sh
chmod +x /usr/local/etc/rc.d/bb.sh

=== remove reserve space ===
<pre>cd
umount /mnt/data1
umount /mnt/data2
tunefs -m 0 /mnt/data1
tunefs -m 0 /mnt/data2
mount -a</pre>

=== setup rdate ===
cd /usr/ports/sysutils/rdate
make install clean

crontab -e
0 0 * * * /usr/local/sbin/rdate -s utcnist.colorado.edu

/usr/local/sbin/rdate -s utcnist.colorado.edu

We used to use ntpd, however it listens on jail IPs which is a security risk. So we stopped. Here's the old instructions:
install new ntp from ports
/usr/ports/net/ntp
make install clean

<pre>echo "server 10.1.4.5" > /etc/ntp.conf
/usr/sbin/ntpd -p /var/run/ntpd.pid
sleep 2; ntpq -p</pre>(confirm it’s able to reach our time server)

=== make a data partition ===

IF you didn't format the data partition during sysinstall:

Create a g partition on 2nd mirror – bsdlabel no longer works (below shows d partition made with sysinstall):

<pre>jail8 /usr/home/bb# gpart show
=> 63 285474735 mfid0 MBR (136G)
63 285458922 1 freebsd [active] (136G)
285458985 15813 - free - (7.7M)

=> 0 285458922 mfid0s1 BSD (136G)
0 524288 1 freebsd-ufs (256M)
524288 12582912 2 freebsd-swap (6.0G)
13107200 524288 4 freebsd-ufs (256M)
13631488 524288 5 freebsd-ufs (256M)
14155776 8388608 6 freebsd-ufs (4.0G)
22544384 262914538 7 freebsd-ufs (125G)

=> 63 584843175 mfid1 MBR (279G)
63 584830197 1 freebsd [active] (279G)
584830260 12978 - free - (6.3M)

=> 0 584830197 mfid1s1 BSD (279G)
0 16777216 2 freebsd-swap (8.0G)
16777216 568052981 4 freebsd-ufs (271G)

jail8 /usr/home/bb# gpart show mfid1s1
=> 0 584830197 mfid1s1 BSD (279G)
0 16777216 2 freebsd-swap (8.0G)
16777216 568052981 4 freebsd-ufs (271G)

# gpart list mfid1s1

Geom name: mfid1s1
fwheads: 255
fwsectors: 63
last: 584830196
first: 0
entries: 8
scheme: BSD
Providers:
1. Name: mfid1s1b
Mediasize: 8589934592 (8.0G)
Sectorsize: 512
Mode: r1w1e0
rawtype: 1
length: 8589934592
offset: 0
type: freebsd-swap
index: 2
end: 16777215
start: 0
2. Name: mfid1s1d
Mediasize: 290843126272 (271G)
Sectorsize: 512
Mode: r0w0e0
rawtype: 7
length: 290843126272
offset: 8589934592
type: freebsd-ufs
index: 4
end: 584830196
start: 16777216
Consumers:
1. Name: mfid1s1
Mediasize: 299433060864 (279G)
Sectorsize: 512
Mode: r1w1e1

# gpart delete -i 4 mfid1s1
mfid1s1d deleted
jail8 /usr/home/bb# gpart list mfid1s1
Geom name: mfid1s1
fwheads: 255
fwsectors: 63
last: 584830196
first: 0
entries: 8
scheme: BSD
Providers:
1. Name: mfid1s1b
Mediasize: 8589934592 (8.0G)
Sectorsize: 512
Mode: r1w1e0
rawtype: 1
length: 8589934592
offset: 0
type: freebsd-swap
index: 2
end: 16777215
start: 0
Consumers:
1. Name: mfid1s1
Mediasize: 299433060864 (279G)
Sectorsize: 512
Mode: r1w1e1

# gpart show mfid1s1
=> 0 584830197 mfid1s1 BSD (279G)
0 16777216 2 freebsd-swap (8.0G)
16777216 568052981 - free - (271G)

# gpart add -t freebsd-ufs -i 7 mfid1s1
mfid1s1g added

# gpart show mfid1s1
=> 0 584830197 mfid1s1 BSD (279G)
0 16777216 2 freebsd-swap (8.0G)
16777216 568052981 7 freebsd-ufs (271G)</pre>

Here's how we USED to do it with bsdlabel:

bsdlabel -e /dev/mfid0s1

given:
<pre># /dev/aacd0s1:
8 partitions:
# size offset fstype [fsize bsize bps/cpg]
a: 262144 0 4.2BSD 2048 16384 16392
b: 4194304 262144 swap
c: 143363997 0 unused 0 0 # "raw" part, don't edit
d: 524288 4456448 4.2BSD 2048 16384 32776
e: 524288 4980736 4.2BSD 2048 16384 32776
f: 6291456 5505024 4.2BSD 2048 16384 28552</pre>

new offset = 6291456 + 5505024 = 11796480<br>
new size is size for 'c' partition minus the new start from above<br>
143363997 - 11796480 = 131567517

So:<br>
g: 131567517 11796480 unused 0 0

=== create the jail template ===

cd /usr/ports/sysutils/jailutils
make install clean

Create an md device to hold the jail:
<pre>touch /mnt/data1/jail-template20g
mdconfig -a -t vnode -s 20g -f /mnt/data1/jail-template20g -u 0
newfs -O 1 /dev/md0
mkdir /mnt/data1/jail-DIR
mount /dev/md0 /mnt/data1/jail-DIR</pre>

Build world into the jail:
<pre>cd /usr/src
make world DESTDIR=/mnt/data1/jail-DIR; taskdone</pre>
~4.5 hr

Make /etc into the jail, mount dev, copy in jkill:
<pre>cd etc
make distribution DESTDIR=/mnt/data1/jail-DIR
mount -t devfs devfs /mnt/data1/jail-DIR/dev
devfs -m /mnt/data1/jail-DIR/dev rule -s 3 applyset
cd /mnt/data1/jail-DIR
ln -sf dev/null kernel
cp /usr/local/sbin/jkill /mnt/data1/jail-DIR/sbin</pre>

Enter into jail to do configuration:
jail /mnt/data1/jail-DIR testhostname 192.168.11.100 /bin/sh
csh

Create fstab:
<pre>touch /etc/fstab
echo 'network_interfaces=""\
hostname="newsystem"\
kern_securelevel_enable="NO"\
sendmail_enable="YES"\
sshd_enable="YES"' > /etc/rc.conf

echo "nameserver 69.55.225.225\
nameserver 69.55.230.3" >> /etc/resolv.conf</pre>

Edit crontab:
vi /etc/crontab
remove the adjkerntz lines
comment out periodic’s and put this line above them:
# IF YOU UNCOMMENT THESE, PLEASE ADJUST THEIR RUN TIME

rm -rf /etc/periodic/daily/400.status-disks

check and remove any crap in /tmp

vi /etc/periodic/security/100.chksetuid
replace: <tt>MP=`mount -t ufs | grep -v " nosuid" | awk '{ print $3 }' | sort`</tt><br>
with: <tt>MP='/'</tt> (use single quotes)

mkdir -p /usr/compat/linux/dev

Add account for user. Output/response:
<pre>adduser

Username: user
Full name: user
Uid (Leave empty for default):
Login group [user]:
Login group is user. Invite user into other groups? []: wheel
Login class [default]:
Shell (sh csh tcsh nologin) [sh]:
Home directory [/home/user]:
Home directory permissions (Leave empty for default):
Use password-based authentication? [yes]:
Use an empty password? (yes/no) [no]:
Use a random password? (yes/no) [no]: y
Lock out the account after creation? [no]:
Username : user
Password : <random>
Full Name : user
Uid : 1001
Class :
Groups : user
Home : /home/user
Home Mode :
Shell : /bin/sh
Locked : no
OK? (yes/no): y
adduser: INFO: Successfully added (user) to the user database.
adduser: INFO: Password for (user) is: 901gmYjO
Add another user? (yes/no): n
Goodbye!</pre>

set TERM:
vi /usr/home/user/.profile
TERM=vt100; export TERM

Set time zone to PT:
tzsetup

Reload aliases:
newaliases

Replace reboot/halt:
rm /sbin/halt /sbin/reboot
ln /sbin/jkill /sbin/halt
ln /sbin/jkill /sbin/reboot

Redirect console output, comment out console and move to /var/log/messages:
vi /etc/syslog.conf
#*.err;kern.warning;auth.notice;mail.crit /dev/console
*.err;kern.warning;auth.notice;mail.crit /var/log/messages

exit
exit

Lib32 compat library:
cd libexec
chflags noschg ld-elf32.so.1
chflags noschg ld-elf.so.1
mv ld-elf32.so.1 ld-elf32.so.1-orig
ln ld-elf.so.1 ld-elf32.so.1
chflags schg ld-elf.so.1
chflags schg ld-elf32.so.1

Replace traceroute:
mv /mnt/data1/jail-DIR/usr/sbin/traceroute /mnt/data1/jail-DIR/usr/sbin/_traceroute
echo '#\!/bin/sh\
/usr/sbin/_traceroute -i bce0 $1' >> /mnt/data1/jail-DIR/usr/sbin/traceroute
chmod +x /mnt/data1/jail-DIR/usr/sbin/traceroute

Modify 'bce0' to reflect whichever nic is public on this hardware.

Clean out ports before copying into the jail (~30mins):
cd /usr/ports
make -DNOCLEANDEPENDS clean
rm -fr /usr/ports/distfiles/*
cp -r /usr/ports /mnt/data1/jail-DIR/usr

rm /mnt/data1/jail-DIR/root/.history

Umount the jail and dump it:
<pre>cd /
umount /mnt/data1/jail-DIR/dev
dump -0a -f /usr/local/jail/template/template /dev/md0
umount /dev/md0
rmdir /mnt/data1/jail-DIR
mdconfig -d -u 0</pre>

=== setup backups ===
<pre>echo '#\!/bin/sh\
backupdir=/data/jail3\
server=backup1\
\
## ENTRY /etc\
## ENTRY /usr/local/etc\
## ENTRY /usr/local/jail\
## ENTRY /root/logs' > /usr/local/jail/bin/backup.config</pre>

Edit to reflect backup server and jail hostname

On backup server, setup backup dirs:
ssh backup1 mkdir -p /data/jail3/0

On backup server, setup backup dirs:
backup1# vi /usr/local/sbin/snapshot_rotate

Edit /usr/local/jail/bin/backup.xxx to use the right drives and copy to /usr/local/jail/bin/backup
vi /usr/local/jail/bin/backup.md
adjust df so it includes all relevant drives, currently <tt>df > /etc/df.bak</tt> is fine. Also, make sure the binary/script source is pulling from the right dir on backup2:
/usr/local/bin/rsync -a backup2:/mnt/data4/bin/freebsd8.x/ /usr/local/jail/bin/

cp /usr/local/jail/bin/backup.md /usr/local/jail/bin/backup

=== create /root/logs ===
mkdir /root/logs

=== edit sshd_config ===
vi /etc/ssh/sshd_config
ListenAddress 69.55.'''229.7'''
ListenAddress 10.1.4.'''103'''

Adjust to pub/private IP for jail.

Restart sshd:
kill -1 `cat /var/run/sshd.pid`

=== add crontab entries ===
<pre>crontab -e
5 0 * * * /usr/local/jail/bin/backup
1 0 1 * * /usr/local/jail/bin/ipfwreset
0 18 * * * /usr/local/jail/bin/ipfwbackup
4,9,14,19,24,29,34,39,44,49,55,59 * * * * /usr/local/jail/bin/trafstats
0 0,6,12,18 * * * /usr/local/jail/bin/sync_jail_names</pre>

On 2950:
*/5 * * * * /usr/local/jail/bin/perc5iraidchk

On 3ware-based servers:
0 0 * * * /usr/local/jail/bin/3wraidchk

=== Reboot notify script ===
ln -s /usr/local/jail/bin/notify.sh /usr/local/etc/rc.d/notify.sh

=== patch jail against starting jails with rtprio ===

<pre>mv /usr/sbin/jail /usr/sbin/jail_
echo '#\!/bin/sh\
/usr/sbin/rtprio -t /usr/sbin/jail_ $*' > /usr/sbin/jail
chmod +x /usr/sbin/jail</pre>

=== make sure mail works ===
If there are map errors:
cd /etc/mail; make maps

=== recover space on /usr (optional) ===

rm -fr /usr/obj

=== wrapper for jps ===

mv /usr/local/sbin/jps /usr/local/sbin/jps_

=== wrapper for jls ===

mv /usr/sbin/jls /usr/sbin/jls_

=== wrapper for jexec ===

mv /usr/sbin/jexec /usr/sbin/jexec_

=== install jtop ===

cd /usr/ports/sysutils/jtop
make install clean

=== block jails from reaching private net ===
echo 'ipfw add 1 deny ip from 69.55.224.0/20 to 10.1.4.0/24' > /usr/local/etc/rc.d/ipfw.sh
chmod 0700 /usr/local/etc/rc.d/ipfw.sh

== add to management infrastructure ==

=== mail ===

==== add to management db ====

tables: jc.ref_machines and jc.ref_templates

on jail run:
uname -r

Which shows something like:
8.3-RELEASE-p2

Insert into db:

<pre>insert into ref_machines values (null,'jail3','mx3.johncompanies.com',0,'f8');
select machine_id from ref_machines where host='jail3';
+------------+
| machine_id |
+------------+
| 35 |
+------------+
insert into ref_templates values ('','8.3-RELEASE-jc2',35,'FreeBSD 8.3',1);</pre>

==== add to bb server ====
vi /usr/home/bb/bbsrc/bb1.9i-btf/etc/bb-hosts
10.1.4.'''109 jail9'''.johncompanies.com # ssh

In the case of an i2b server, use real ip:
69.55.'''229.7 jail3'''.johncompanies.com # ssh

<pre>su bb
cd
bbsrc/bb/runbb.sh restart ; exit</pre>

==== Update backupgraph ====
vi /usr/local/www/mgmt/cgi/backupgraph.pl
(add hostname)

==== Update load mrtg ====
vi /usr/local/www/mgmt/mrtg/mrtg1.cfg
(add new entry to file following existing format)

=== ns1c ===
fwd and reverse lookups:
vr johncompanies.com

vi internal.johncompanies.com
rndc reload johncompanies.com IN private

ptr 69.55.227.x

=== wiki ===
add to server/cabinet map

=== firewall ===
add an outside blocking rule to the firewall, so this machine can only be reached from inside the firewall. Follow example already in firewall jail17 is:

00117 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 or 69.55.238.150 } to 69.55.228.200
00117 deny ip from any to 69.55.228.200

jail19 would be 00119...
ipfw add 00109 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 } to 69.55.232.3
ipfw add 00109 deny ip from any to 69.55.232.3

=== select some customers for castle probe map ===

= FreeBSD 8.x =

== Assumptions ==

Setup instructions below assume this is DELL 2950 with an LSI-based SAS RAID card.

Server is at castle, connected to pub, private, serial and DRAC

Assuming OS loading done via IPKVM with ISO mounted via USB

Assumes at 4 drives, 2 mirrors

== Configure server BIOS ==

setup console redirect, speed 115200

set LCD string to name of server "jail8"

set date to GMT

go into RAID bios and setup mirrors

configure DRAC: TODO

== Install OS (sysinstall) ==

boot to bootonly disk for AMD version of FreeBSD, i.e. <tt>FreeBSD-8.3-RELEASE-amd64-bootonly.iso</tt>

when the install menu appears, choose <tt>custom install</tt>

=== partition menu ===
move cursor to mfid0, hit space (takes you to partition map screen). If there is only 1 mirror, there will be no option to select a specifit drive: mfid0 will be selected.

type 'a' to use entire disk<br>
type 'q' to quit and save<br>
choose 'freebsd standard mbr'

space to '''un'''select mfid0<br>
cursor down to mfid1<br>
hit space

type 'a' to use entire disk<br>
type 'q' to quit and save<br>
choose 'none' for boot mgr (leave untouched)

cursor over mfid0<br>
space<br>
(takes you into part. Screen again) 'q' to exit<br>
none for boot mgr<br>

Make sure both drives (mfid0 and mfid1) are checked and tab to ok

=== Label menu ===

Make sure mfid0 is highlighted at the top of the screen, setup the following partitions

/ 512M<br>
swap 6G<br>
/var 256M <br>
/tmp 256M<br>
/usr 5G<br>
/mnt/data1 remaining space

All partitions except / should be setup for soft updates. If not, type 's' to enable for soft updates on all except for / (should look like UFS2+S Y under the Newfs column)

move cursor to mfid1 at the top of the screen<br>
swap 8G (or 4G if there’s a 3rd drive)<br>
/mnt/data2 remaining space <br>
'q' to save and exit

=== distributions ===
Choose the following distribudions

* developer (ok to install ports)
* custom -> lib32
exit

=== media ===
if you are installing via a cd, no need to enter this menu or change anything. Otherwise, choose ftp to install via ftp. You will be prompted to setup networking. You will need to choose a nic (typically bce0 or bce1). Say no to DHCP and IPv6. Hopefully the public nic cable was installed in bce0 so start with that nic and provide the hostname, (public) IP, netmask, gateway and DNS. When configured, it should start pinging. If it doesn't, have the NOC swap cables. Select any FTP server, usually Main or ftp4.

=== commit ===

this usually takes 12mins<br>
during the process you may need to select a new ftp mirror, this is not a problem.<br>
at the conclution of the install you will be prompted to enter the root password (2x) and returned to the configuration menu.

=== add user ===

'''Add''' user 'user'. Defaults for everything is fine, just remember to enter 'wheel' in the member group field.
Do set the password.

=== Setup timezone ===

PT

=== Networking ===
page down to the bottom and enable '[X]' sshd

If you installed via cd, you will need to visit:<br>
interfaces->bce0-><br>
No IPV6<br>
dhcp=no<br>
Set hostname, IP, DNS, gateway<br>
(i.e. setup the nic as indicated above)

Exit the install and if you installed via CD, take it out and let the machine reboot

== Configure OS, kernel, userland, jail ==

=== double check the date/time ===

=== populate /etc/resolv.conf ===

echo "nameserver 69.55.225.225\
nameserver 69.55.230.3
nameserver 69.55.229.3" > /etc/resolv.conf

=== edit /etc/make.conf ===
echo "WITHOUT_X11=yes \
KERNCONF=jail3 \
BOOT_COMCONSOLE_SPEED=115200" >> /etc/make.conf

=== setup bootloader for console, etc ===

add settings to /boot/loader.conf and /boot.config:

echo "-Dh" >> /boot.config

echo 'console="comconsole,vidconsole" \
boot_multicons="YES" \
boot_serial="YES" \
mfi_linux_load="YES" \
comconsole_speed="115200"' >> /boot/loader.conf

=== enable login via serial console ===
turn off all ttyv's except 0 and 1 in /etc/ttys and turn on ttyd0, change type to vt100:

vi /etc/ttys

The changed lines should look like:

ttyv2 "/usr/libexec/getty Pc" cons25 off secure
ttyv3 "/usr/libexec/getty Pc" cons25 off secure
ttyv4 "/usr/libexec/getty Pc" cons25 off secure
ttyv5 "/usr/libexec/getty Pc" cons25 off secure
ttyv6 "/usr/libexec/getty Pc" cons25 off secure
ttyv7 "/usr/libexec/getty Pc" cons25 off secure
# Serial terminals
# The 'dialup' keyword identifies dialin lines to login, fingerd etc.
ttyu0 "/usr/libexec/getty std.9600" vt100 on secure

Restart init
kill -1 1

At this point you should have a login on console.

To configure serial console access, login to the console server as root and run:

# vi /etc/remote

following examples there, rename port to server's hostname, depending on where and which digi box this server is plugged into. Make sure to get speed right too: 115200

=== populate hosts ===

If server is at castle:
echo "10.1.4.3 backup2" >> /etc/hosts
echo "10.1.4.8 backup1" >> /etc/hosts

If server is at i2b:
echo "69.55.230.10 backup2" >> /etc/hosts
echo "10.1.2.3 backup3" >> /etc/hosts
echo "69.55.230.11 backup1" >> /etc/hosts

=== create ssh key, upload to backup servers ===
cd
ssh-keygen -t dsa -b 1024
(default location, leave password blank)

If server is at castle:
cat /root/.ssh/id_dsa.pub | ssh backup2 'cat - >> /root/.ssh/authorized_keys'
cat /root/.ssh/id_dsa.pub | ssh backup1 'cat - >> /root/.ssh/authorized_keys'

If server is at i2b:
cat /root/.ssh/id_dsa.pub | ssh backup3 'cat - >> /root/.ssh/authorized_keys'
cat /root/.ssh/id_dsa.pub | ssh backup2 'cat - >> /root/.ssh/authorized_keys'
cat /root/.ssh/id_dsa.pub | ssh backup1 'cat - >> /root/.ssh/authorized_keys'

confirm that you can ssh to backup2 and backup1 without getting a login prompt

ssh backup2 hostname
ssh backup1 hostname
ssh backup3 hostname

=== create & populate binaries/scripts dirs ===
<pre>mkdir -p /usr/local/jail/bin
mkdir -p /usr/local/jail/rc.d
mkdir -p /usr/local/jail/template/
mkdir /mnt/data1
mkdir /mnt/data2
scp backup2:"/mnt/data4/bin/freebsd8.x/*" /usr/local/jail/bin
cd /usr/local/jail/rc.d/
touch quad1
touch deprecated
chmod +x *
cd /usr/local/jail/bin
ln -s /usr/local/jail/rc.d/quad1 quad1
ln -s /usr/local/jail/bin/jailmake_md jailmake
ln -s /usr/local/jail/bin/js_md js
ln -s /usr/local/jail/bin/canceljail_md canceljail
ln -s /usr/local/jail/bin/jailmakeempty_md jailmakeempty
ln -s /usr/local/jail/bin/postboot_md postboot
ln -s /usr/local/jail/bin/preboot_md preboot
ln -s /usr/local/jail/bin/startjail_md startjail
ln -s /usr/local/jail/bin/stopjail_md stopjail

rehash</pre>

=== edit root's path and login script ===

vi /root/.cshrc

Change alias entries (add G):

<pre>alias la ls -aG
alias lf ls -FAG
alias ll ls -lAG
alias ls ls -AG
alias mbm mb mount
alias mbu mb umount
alias cjb cd /usr/local/jail/bin
alias cd1 cd /mnt/data1
alias cd2 cd /mnt/data2
alias cd3 cd /mnt/data3
alias jtop jtop lj
alias j jobs</pre>

add to path be careful to leave a space after bin and make sure the wrapping isn't broken:
/usr/local/jail/bin

alter the prompt, set the following:
set prompt = "`/bin/hostname -s` %/# "

at the bottom of the file add:
<pre>set sshtty=`who am i|awk '{print $2}'`
/usr/sbin/rtprio 3 -`psj | grep $sshtty | awk '{print $2}'`

set shortty=`who am i | awk '{print $2}' | sed -E 's/.*(..)$/\1/'`
foreach x (`psj | grep sh | grep $shortty | awk '{print $2}'`)
/usr/sbin/rtprio 2 -$x
end</pre>

Make the new settings active in current shell:
source /root/.cshrc

=== install cvsup ===

<pre>cd /usr/ports/net/cvsup-without-gui
make install clean; rehash; mail -s 'cvs installed' support@johncompanies.com < /dev/null</pre>
stand by for gettext options (use defaults). this process takes approx 22mins- hence the email/page notice above.

=== get latest sources for this release ===

<pre>cd /usr/src
echo "*default host=cvsup4.freebsd.org\
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_8_3\
*default delete use-rel-suffix\
*default compress\
src-all" > sup</pre>

If you need to run stable (cause release is broken or some other reason) make the sup file look like:

<pre>echo "*default host=cvsup4.freebsd.org\
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_8\
*default delete use-rel-suffix\
*default compress\
src-all" > sup</pre>

cvsup sup ; mail -s 'cvs sup done' support@johncompanies.com < /dev/null

time varies, 10-20mins

=== configure new kernel ===

Pull down the kernel config we are using for this distribution. In this case we use an 8.2 kernel config on 8.3, which is valid. The local file should be the same name as host- <tt>jail3</tt> in this example

<pre>cd /usr/src/sys/amd64/conf
scp backup2:/mnt/data4/build/freebsd/kern_config-8.2-amd64 ./jail3</pre>

edit the kernel config and change ident to be the name of the jail:
vi jail3
ident jail3

Optional, edit <tt>/sys/conf/newvers.sh</tt> to add <tt>–jc2</tt> to the end of the BRANCH string (RELEASE-jc2)
vi /sys/conf/newvers.sh

notes on kernel configuring: http://www.FreeBSD.org/doc/en_US.ISO8859-1/books/handbook/kernelconfig-config.html

=== install patches ===

The various patches are in /mnt/data4/build/freebsd/patches on backup2. There are dirs for each version. Not all dirs are populated, but patches for later versions work on older ones unless there is a new patch in the older dir.

there are no patches we use for 8.x, but here would be the commands:

cd /usr/src
scp backup2:"/mnt/data4/build/freebsd/patches/8.0/*" .

Apply patches, i.e. the <tt>jls-patch</tt>:
patch -l < jls-patch

=== build, install kernel and world ===

Rename current generic kernel so it will always be available to boot from. Save room by removing non-needed kernel modules:

<pre>cd /boot
mv kernel kernel.GENERIC
cd kernel.GENERIC
mkdir hold
mv mfi_linux.ko hold/
mv linux.ko hold/
mv linprocfs.ko hold/
mv linsysfs.ko hold/
mv geom_vinum.ko hold/
mv geom_concat.ko hold/
mv zfs.* hold/
mv opensolaris* hold/

rm *.ko
rm *.symbols
mv hold/* .
rmdir hold/

cd /usr/src
make buildkernel installkernel

make buildworld ; mail -s 'buildworld done' dave.boodman@vtext.com < /dev/null</pre>
~38mins

make installworld
~34min

mergemaster -i
You will be prompted to merge, replace or ignore files changed by the src update. In most cases you can '''d'''elete the temp (new) files.

ONLY if this will be a zfs system (not currently used in 8.x):
<pre>cd /sys/modules/zfs
make
make install
cd /sys/modules/opensolaris
make
make install</pre>

=== populate devfs ruleset ===
scp backup2:/mnt/data4/build/freebsd/devfs.rules.8x /etc/devfs.rules

=== populate /etc/rc.conf with IPs and service settings ===

<pre>vi /etc/rc.conf

kern_securelevel_enable="NO"
portmap_enable="NO"
sendmail_enable="NO"
usbd_enable="YES"

nfs_client_enable="YES"
nfs_reserved_port_only="YES"
inetd_enable="YES"
inetd_flags="-wW -a 10.1.4.'''103'''"
devfs_system_ruleset="devfsrules_show_all"

ifconfig_bce1="inet 10.1.4.'''103''' netmask 255.255.255.0"
ifconfig_bce0="inet 69.55.'''229.7''' netmask 255.255.255.0"
#ifconfig_bce0_alias0="inet 69.55.2xx.xx netmask 255.255.255.0"

fsck_y_enable="YES"
background_fsck="NO"
#rc_mfi_raid_tty_log="YES"
#zfs_enable="YES"</pre>

Modify IPs, hostname, gateway for this box.

=== make sure sysctls are set and preserved upon boot ===

<pre>echo "kern.consmute=0\
kern.ipc.shm_use_phys=1\
kern.ipc.shmall=131070\
kern.ipc.shmmax=134217728\
net.inet.tcp.syncookies=0\
kern.maxfiles=32768\
kern.fallback_elf_brand=3\
kern.maxprocperuid=4000\
security.jail.sysvipc_allowed=1\
security.jail.allow_raw_sockets=1\
security.jail.socket_unixiproute_only=1\
security.jail.chflags_allowed=0\
dev.amr.0.allow_volume_configure=1\
compat.linux.osrelease=2.6.12\
vm.pmap.shpgperproc=500\
security.bsd.unprivileged_read_msgbuf=0\
kern.maxvnodes=400000" >> /etc/sysctl.conf</pre>

Tuning note: watch vfs.numvnodes while the server is live to get guidance on where to set maxvnodes

=== mount procfs ===
echo "proc /proc procfs rw 0 0" >> /etc/fstab
echo "linprocfs /usr/compat/linux/proc linprocfs rw 0 0" >> /etc/fstab
echo "linsysfs /usr/compat/linux/sys linsysfs rw 0 0" >> /etc/fstab
mkdir -p /usr/compat/linux/proc
mkdir -p /usr/compat/linux/sys

=== enable noatime option ===
data1 and data2 should look something like (add ',noatime' after 'rw'):
/dev/mfid0s1g /mnt/data1 ufs rw,noatime 2 2
/dev/mfid1s1d /mnt/data2 ufs rw,noatime 2 2

=== reboot. Confirm new kernel is loaded, devfs in place ===

uname -a

Check devfs rules
devfs rule showsets
devfs rule -s 3 show

Should see:
<pre># devfs rule showsets
1
2
3
4

# devfs rule -s 3 show
100 include 1
207 path pts* unhide
217 path fd unhide
218 path fd/* unhide
300 path ttyp* unhide
301 path ttyq* unhide
302 path ttyr* unhide
303 path ttys* unhide
304 path ttyP* unhide
305 path ttyQ* unhide
306 path ttyR* unhide
307 path ttyS* unhide
400 path null unhide
500 path zero unhide
600 path random unhide
610 path urandom unhide
700 path mem unhide
710 path kmem unhide
810 path mdctl unhide
900 path stdin unhide
910 path stdout unhide
920 path stderr unhide</pre>

=== update ports ===

<pre>cd /usr/ports
echo "*default host=cvsup4.FreeBSD.org\
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_8_3\
*default delete use-rel-suffix\
*default compress\
ports-all tag=." > sup

cvsup sup; mail -s 'cvs sup ports done' support@johncompanies.com < /dev/null</pre>
~24mins

=== Install raid mgmt tool ===

==== Perc5/i, 6/i ====
Pull over cli from previous system (jail9)
scp /usr/local/sbin/mega* root@10.1.4.109:/usr/local/sbin/
scp /usr/local/libexec/MegaCli root@10.1.4.109:/usr/local/libexec/MegaCli
These are linux-based tools. This will require linux base which we would normally install from ports, but since it's failing lately, we just pull in the compat libraries from another system:

rsync -aSHv --exclude=proc --exclude=sys 10.1.4.109:/usr/compat/linux/ /usr/compat/linux/

Test:
rehash; megacli ldinfo lall a0

DEPRECATED: Assuming it worked, here's how we ''used to'' install linux_base:
----
cd /usr/ports/emulators/linux_base-fc4
make install clean
(didnt succeed due to libtool requirement)

cd /usr/ports/distfiles
fetch http://www.lsi.com/DistributionSystem/AssetDocument/support/downloads/megaraid/miscellaneous/linux/2.00.15_Linux_MegaCLI.zip
cd /usr/ports/sysutils/linux-megacli
make install clean
(also failed due to libtool)
----

==== 2850 PERC 4e/Di- no linux ====
<pre>cd /usr/ports/distfiles/
fetch http://backup01.best-hosting.ru/pub/FreeBSD/ports/distfiles/dr_freebsd_1.51.zip
cd /usr/ports/sysutils/megarc
make install clean
megarc -dispCfg -a0</pre>

=== install rsync from ports ===
cd /usr/ports/net/rsync
make install clean

choose default options

=== configure inetd to respond to mrtg load queries ===
echo "load stream tcp nowait user /usr/local/jail/bin/load.pl load.pl" >> /etc/inetd.conf
echo "load 12384/tcp" >> /etc/services

=== install perl ===
cd /usr/ports/lang/perl5.12
make install clean

choose defaults

=== install bb client ===

Compiling from source on AMD64 will not work. So, we use a linux-compiled version and rely on linux compat. Per [[#Perc5.2Fi.2C_6.2Fi|above]], linux compat won't install on 8.x - libtool 2.4 need. So, instead we copy(ed) over linux:
rsync -aSHv --exclude=proc --exclude=sys 10.1.4.108:/usr/compat/linux/ /usr/compat/linux/

adduser

Output/response:

<pre>Username: bb
Full name: bb
Uid (Leave empty for default): 1984
Login group [bb]:
Login group is bb. Invite bb into other groups? []:
Login class [default]:
Shell (sh csh tcsh nologin) [sh]:
Home directory [/home/bb]:
Use password-based authentication? [yes]:
Use an empty password? (yes/no) [no]:
Use a random password? (yes/no) [no]: yes
Lock out the account after creation? [no]:
Username : bb
Password : <random>
Full Name : bb
Uid : 1984
Class :
Groups : bb
Home : /home/bb
Shell : /bin/sh
Locked : no
OK? (yes/no): yes</pre>

cd /usr/home/bb
scp backup2:/mnt/data4/build/bb/bb-freebsd_linuxcompat.tgz .
tar xzf bb-freebsd_linuxcompat.tgz

edit /home/bb/bbc1.9e-btf/etc/bb-hosts with something like:

echo "10.1.4.5 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh \
10.1.4.'''103 jail3'''.johncompanies.com # ssh" > /home/bb/bbc1.9e-btf/etc/bb-hosts

Edit for machine name and private IP.

if this machine is at i2b:
echo "69.55.230.2 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh \
69.55.229.7 jail3.johncompanies.com # ssh" > /home/bb/bbc1.9e-btf/etc/bb-hosts

<pre>vi /home/bb/bbc1.9e-btf/ext/openfiles

MACHINE="jail3,johncompanies,com" # HAS TO BE IN A,B,C FORM</pre>
Edit for machine name.

<pre>cd /usr/home/bb/bbc1.9e-btf/etc
./bbchkcfg.sh
(y to questions)
./bbchkhosts.sh
(ignore ssh errors)
cd ../..
chown -R bb .
su bb
cd
cd bbc1.9e-btf
./runbb.sh start
more BBOUT
(look for errors)
exit</pre>

Put in script to start bb @ boot:
echo 'su - bb -c "cd /home/bb/bbc1.9e-btf; ./runbb.sh start"' > /usr/local/etc/rc.d/bb.sh
chmod +x /usr/local/etc/rc.d/bb.sh

=== remove reserve space ===
<pre>cd
umount /mnt/data1
umount /mnt/data2
tunefs -m 0 /mnt/data1
tunefs -m 0 /mnt/data2
mount -a</pre>

=== setup rdate ===
cd /usr/ports/sysutils/rdate
make install clean

crontab -e
0 0 * * * /usr/local/sbin/rdate -s utcnist.colorado.edu

/usr/local/sbin/rdate -s utcnist.colorado.edu

We used to use ntpd, however it listens on jail IPs which is a security risk. So we stopped. Here's the old instructions:
install new ntp from ports
/usr/ports/net/ntp
make install clean

<pre>echo "server 10.1.4.5" > /etc/ntp.conf
/usr/sbin/ntpd -p /var/run/ntpd.pid
sleep 2; ntpq -p</pre>(confirm it’s able to reach our time server)

=== make a data partition ===

IF you didn't format the data partition during sysinstall:

Create a g partition on 2nd mirror – bsdlabel no longer works (below shows d partition made with sysinstall):

<pre>jail8 /usr/home/bb# gpart show
=> 63 285474735 mfid0 MBR (136G)
63 285458922 1 freebsd [active] (136G)
285458985 15813 - free - (7.7M)

=> 0 285458922 mfid0s1 BSD (136G)
0 524288 1 freebsd-ufs (256M)
524288 12582912 2 freebsd-swap (6.0G)
13107200 524288 4 freebsd-ufs (256M)
13631488 524288 5 freebsd-ufs (256M)
14155776 8388608 6 freebsd-ufs (4.0G)
22544384 262914538 7 freebsd-ufs (125G)

=> 63 584843175 mfid1 MBR (279G)
63 584830197 1 freebsd [active] (279G)
584830260 12978 - free - (6.3M)

=> 0 584830197 mfid1s1 BSD (279G)
0 16777216 2 freebsd-swap (8.0G)
16777216 568052981 4 freebsd-ufs (271G)

jail8 /usr/home/bb# gpart show mfid1s1
=> 0 584830197 mfid1s1 BSD (279G)
0 16777216 2 freebsd-swap (8.0G)
16777216 568052981 4 freebsd-ufs (271G)

# gpart list mfid1s1

Geom name: mfid1s1
fwheads: 255
fwsectors: 63
last: 584830196
first: 0
entries: 8
scheme: BSD
Providers:
1. Name: mfid1s1b
Mediasize: 8589934592 (8.0G)
Sectorsize: 512
Mode: r1w1e0
rawtype: 1
length: 8589934592
offset: 0
type: freebsd-swap
index: 2
end: 16777215
start: 0
2. Name: mfid1s1d
Mediasize: 290843126272 (271G)
Sectorsize: 512
Mode: r0w0e0
rawtype: 7
length: 290843126272
offset: 8589934592
type: freebsd-ufs
index: 4
end: 584830196
start: 16777216
Consumers:
1. Name: mfid1s1
Mediasize: 299433060864 (279G)
Sectorsize: 512
Mode: r1w1e1

# gpart delete -i 4 mfid1s1
mfid1s1d deleted
jail8 /usr/home/bb# gpart list mfid1s1
Geom name: mfid1s1
fwheads: 255
fwsectors: 63
last: 584830196
first: 0
entries: 8
scheme: BSD
Providers:
1. Name: mfid1s1b
Mediasize: 8589934592 (8.0G)
Sectorsize: 512
Mode: r1w1e0
rawtype: 1
length: 8589934592
offset: 0
type: freebsd-swap
index: 2
end: 16777215
start: 0
Consumers:
1. Name: mfid1s1
Mediasize: 299433060864 (279G)
Sectorsize: 512
Mode: r1w1e1

# gpart show mfid1s1
=> 0 584830197 mfid1s1 BSD (279G)
0 16777216 2 freebsd-swap (8.0G)
16777216 568052981 - free - (271G)

# gpart add -t freebsd-ufs -i 7 mfid1s1
mfid1s1g added

# gpart show mfid1s1
=> 0 584830197 mfid1s1 BSD (279G)
0 16777216 2 freebsd-swap (8.0G)
16777216 568052981 7 freebsd-ufs (271G)</pre>

Here's how we USED to do it with bsdlabel:

bsdlabel -e /dev/mfid0s1

given:
<pre># /dev/aacd0s1:
8 partitions:
# size offset fstype [fsize bsize bps/cpg]
a: 262144 0 4.2BSD 2048 16384 16392
b: 4194304 262144 swap
c: 143363997 0 unused 0 0 # "raw" part, don't edit
d: 524288 4456448 4.2BSD 2048 16384 32776
e: 524288 4980736 4.2BSD 2048 16384 32776
f: 6291456 5505024 4.2BSD 2048 16384 28552</pre>

new offset = 6291456 + 5505024 = 11796480<br>
new size is size for 'c' partition minus the new start from above<br>
143363997 - 11796480 = 131567517

So:<br>
g: 131567517 11796480 unused 0 0

=== create the jail template ===

cd /usr/ports/sysutils/jailutils
make install clean

Create an md device to hold the jail:
<pre>touch /mnt/data1/jail-template20g
mdconfig -a -t vnode -s 20g -f /mnt/data1/jail-template20g -u 0
newfs -O 1 /dev/md0
mkdir /mnt/data1/jail-DIR
mount /dev/md0 /mnt/data1/jail-DIR</pre>

Build world into the jail:
<pre>cd /usr/src
make world DESTDIR=/mnt/data1/jail-DIR; taskdone</pre>
~1hr

Make /etc into the jail, mount dev, copy in jkill:
<pre>cd etc
make distribution DESTDIR=/mnt/data1/jail-DIR
mount -t devfs devfs /mnt/data1/jail-DIR/dev
devfs -m /mnt/data1/jail-DIR/dev rule -s 3 applyset
cd /mnt/data1/jail-DIR
ln -sf dev/null kernel
cp /usr/local/sbin/jkill /mnt/data1/jail-DIR/sbin</pre>

Enter into jail to do configuration:
jail /mnt/data1/jail-DIR testhostname 192.168.11.100 /bin/sh
csh

Create fstab:
<pre>touch /etc/fstab
echo 'network_interfaces=""\
hostname="newsystem"\
kern_securelevel_enable="NO"\
sendmail_enable="YES"\
sshd_enable="YES"' > /etc/rc.conf

echo "nameserver 69.55.225.225\
nameserver 69.55.230.3" >> /etc/resolv.conf</pre>

Edit crontab:
vi /etc/crontab
remove the adjkerntz lines
comment out periodic’s and put this line above them:
# IF YOU UNCOMMENT THESE, PLEASE ADJUST THEIR RUN TIME

rm -rf /etc/periodic/daily/400.status-disks

check and remove any crap in /tmp

vi /etc/periodic/security/100.chksetuid
replace: <tt>MP=`mount -t ufs | grep -v " nosuid" | awk '{ print $3 }' | sort`</tt><br>
with: <tt>MP='/'</tt> (use single quotes)

mkdir -p /usr/compat/linux/dev

Add account for user. Output/response:
<pre>adduser

Username: user
Full name: user
Uid (Leave empty for default):
Login group [user]:
Login group is user. Invite user into other groups? []: wheel
Login class [default]:
Shell (sh csh tcsh nologin) [sh]:
Home directory [/home/user]:
Home directory permissions (Leave empty for default):
Use password-based authentication? [yes]:
Use an empty password? (yes/no) [no]:
Use a random password? (yes/no) [no]: y
Lock out the account after creation? [no]:
Username : user
Password : <random>
Full Name : user
Uid : 1001
Class :
Groups : user
Home : /home/user
Home Mode :
Shell : /bin/sh
Locked : no
OK? (yes/no): y
adduser: INFO: Successfully added (user) to the user database.
adduser: INFO: Password for (user) is: 901gmYjO
Add another user? (yes/no): n
Goodbye!</pre>

set TERM:
vi /usr/home/user/.profile
TERM=vt100; export TERM

Set time zone to PT:
tzsetup

Reload aliases:
newaliases

Replace reboot/halt:
rm /sbin/halt /sbin/reboot
ln /sbin/jkill /sbin/halt
ln /sbin/jkill /sbin/reboot

Redirect console output, comment out console and move to /var/log/messages:
vi /etc/syslog.conf
#*.err;kern.warning;auth.notice;mail.crit /dev/console
*.err;kern.warning;auth.notice;mail.crit /var/log/messages

exit
exit

Lib32 compat library:
cd libexec
chflags noschg ld-elf32.so.1
chflags noschg ld-elf.so.1
mv ld-elf32.so.1 ld-elf32.so.1-orig
ln ld-elf.so.1 ld-elf32.so.1
chflags schg ld-elf.so.1
chflags schg ld-elf32.so.1

Replace traceroute:
mv /mnt/data1/jail-DIR/usr/sbin/traceroute /mnt/data1/jail-DIR/usr/sbin/_traceroute
echo '#\!/bin/sh\
/usr/sbin/_traceroute -i bce0 $1' >> /mnt/data1/jail-DIR/usr/sbin/traceroute
chmod +x /mnt/data1/jail-DIR/usr/sbin/traceroute

Modify 'bce0' to reflect whichever nic is public on this hardware.

Clean out ports before copying into the jail (~30mins):
cd /usr/ports
make -DNOCLEANDEPENDS clean
rm -fr /usr/ports/distfiles/*
cp -r /usr/ports /mnt/data1/jail-DIR/usr

rm /mnt/data1/jail-DIR/root/.history

Umount the jail and dump it:
<pre>cd /
umount /mnt/data1/jail-DIR/dev
dump -0a -f /usr/local/jail/template/template /dev/md0
umount /dev/md0
rmdir /mnt/data1/jail-DIR
mdconfig -d -u 0</pre>

=== setup backups ===
<pre>echo '#\!/bin/sh\
backupdir=/data/jail3\
server=backup1\
\
## ENTRY /etc\
## ENTRY /usr/local/etc\
## ENTRY /usr/local/jail\
## ENTRY /root/logs' > /usr/local/jail/bin/backup.config</pre>

Edit to reflect backup server and jail hostname

On backup server, setup backup dirs:
ssh backup1 mkdir -p /data/jail3/0

On backup server, setup backup dirs:
backup1# vi /usr/local/sbin/snapshot_rotate

Edit /usr/local/jail/bin/backup.xxx to use the right drives and copy to /usr/local/jail/bin/backup
vi /usr/local/jail/bin/backup.md
adjust df so it includes all relevant drives, currently <tt>df > /etc/df.bak</tt> is fine. Also, make sure the binary/script source is pulling from the right dir on backup2:
/usr/local/bin/rsync -a backup2:/mnt/data4/bin/freebsd8.x/ /usr/local/jail/bin/

cp /usr/local/jail/bin/backup.md /usr/local/jail/bin/backup

=== create /root/logs ===
mkdir /root/logs

=== edit sshd_config ===
vi /etc/ssh/sshd_config
ListenAddress 69.55.229.7
ListenAddress 10.1.4.103

'''''Adjust to pub/private IP for jail.'''
''
Restart sshd:
kill -1 `cat /var/run/sshd.pid`

=== add crontab entries ===
<pre>crontab -e
5 0 * * * /usr/local/jail/bin/backup
1 0 1 * * /usr/local/jail/bin/ipfwreset
0 18 * * * /usr/local/jail/bin/ipfwbackup
4,9,14,19,24,29,34,39,44,49,55,59 * * * * /usr/local/jail/bin/trafstats
0 0,6,12,18 * * * /usr/local/jail/bin/sync_jail_names</pre>

On 2950:
*/5 * * * * /usr/local/jail/bin/perc5iraidchk

On 3ware-based servers:
0 0 * * * /usr/local/jail/bin/3wraidchk

=== Reboot notify script ===
ln -s /usr/local/jail/bin/notify.sh /usr/local/etc/rc.d/notify.sh

=== patch jail against starting jails with rtprio ===

<pre>mv /usr/sbin/jail /usr/sbin/jail_
echo '#\!/bin/sh\
/usr/sbin/rtprio -t /usr/sbin/jail_ $*' > /usr/sbin/jail
chmod +x /usr/sbin/jail</pre>

=== make sure mail works ===
If there are map errors:
cd /etc/mail; make maps

=== recover space on /usr (optional) ===

rm -fr /usr/obj

=== wrapper for jps ===

mv /usr/local/sbin/jps /usr/local/sbin/jps_

=== wrapper for jls ===

mv /usr/sbin/jls /usr/sbin/jls_

=== wrapper for jexec ===

mv /usr/sbin/jexec /usr/sbin/jexec_

=== install jtop ===

cd /usr/ports/sysutils/jtop
make install clean

=== block jails from reaching private net ===
echo 'ipfw add 1 deny ip from 69.55.224.0/20 to 10.1.4.0/24' > /usr/local/etc/rc.d/ipfw.sh
chmod 0700 /usr/local/etc/rc.d/ipfw.sh

== add to management infrastructure ==

=== mail ===

==== add to management db ====

tables: jc.ref_machines and jc.ref_templates

on jail run:
uname -r

Which shows something like:
8.3-RELEASE-p2

Insert into db:

<pre>insert into ref_machines values (null,'jail3','mx3.johncompanies.com',0,'f8');
select machine_id from ref_machines where host='jail3';
+------------+
| machine_id |
+------------+
| 35 |
+------------+
insert into ref_templates values ('','8.3-RELEASE-jc2',35,'FreeBSD 8.3',1);</pre>

==== add to bb server ====
vi /usr/home/bb/bbsrc/bb1.9i-btf/etc/bb-hosts
10.1.4.109 jail9.johncompanies.com # ssh

In the case of an i2b server, use real ip:
69.55.229.7 jail3.johncompanies.com # ssh

<pre>su bb
cd
bbsrc/bb/runbb.sh restart ; exit</pre>

==== Update backupgraph ====
vi /usr/local/www/mgmt/cgi/backupgraph.pl
(add hostname)

==== Update load mrtg ====
vi /usr/local/www/mgmt/mrtg/mrtg1.cfg
(add new entry to file following existing format)

=== ns1c ===
fwd and reverse lookups:
vr johncompanies.com

vi internal.johncompanies.com
rndc reload johncompanies.com IN private

ptr 69.55.227.x

=== wiki ===
add to server/cabinet map

=== firewall ===
add an outside blocking rule to the firewall, so this machine can only be reached from inside the firewall. Follow example already in firewall jail17 is:

00117 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 or 69.55.238.150 } to 69.55.228.200
00117 deny ip from any to 69.55.228.200

jail19 would be 00119...
ipfw add 00109 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 } to 69.55.232.3
ipfw add 00109 deny ip from any to 69.55.232.3

=== select some customers for castle probe map ===

= FreeBSD 4.11 =

<pre>
4.11
Last updated 2006-1-26

All time extimates below assume disks aren’t scrubbing

1. make sure bios is setup for bios console redirect
Supermicro:
Console redirection:
Com port addr: on-board COM A
Baud: 38400
Console type: vt100
Flow control: none
Console connection: direct
Continue cr after post: off

2450:
Make sure running bios A09
Console Redirection: VT100/VT220

2. assuming mirrors (or at least disks) created (if not, refer to this), boot to disk 1 of 4.11
skip kernel config (enter)
custom install
partition
move cursor to aacd0, hit space (takes you to partition map screen)
a for entire disk
q to quit and save
standard mbr (no boot manager)
space to unselect aacd0
cursor over aacd1
space
a for entire disk
q to quit and save
none (leave untouched)

cursor over aacd0
space
(takes you into part. Screen again) q to exit
none
Make sure both are checked and tab to ok

label
Make sure aacd0 is highlighted
a to start with defaults

john likes:
/1g
/var 256m

Glenn likes:
/128
/var 256
/usr 3g
/tmp 256

We do:
delete everything
/ 128M
swap 4G
/var 256M
/tmp 256M
/usr 3G
/mnt/data1 remaining space

Make sure to toggle S for soft updates on all (should look like UFS+S Y under the Newfs column)
Set the noatime option

move cursor to aacd1
swap 4G
/mnt/data2 remaining space
Set the noatime option

q to save and exit

distributions
developer
yes to install ports
exit
media
cd
commit
yes
(2450: 16mins, supermicro: 11mins)
yes to "visit general config"
Set root pwd
Add user ‘user’ member group is wheel, set password
Set tz
Networking->interfaces->Fxp0
No IPV6
dhcp=yes
Set hostname & domain

Startup services:
Disble usbd

exit...
exit install
yes
take the cd out and let the machine reboot

3. put some temp settings in /etc/rc.conf:
usbd_enable="NO"
sendmail_enable="NO"

4. reboot

5. double check the date/time

6. edit /etc/make.conf (only add the console speed line if this is a supermicro capable of outputting BIOS redirect at that speed – 2450’s can only do 9600)
cat >> /etc/make.conf
WITHOUT_X11=yes
KERNCONF=jail18
BOOT_COMCONSOLE_SPEED=38400

7. install cvsup
cd /usr/ports/net/cvsup-without-gui
make install clean (stay close for gettext options, 2450: 21mins, supermicro: 14mins)
rehash

8. get latest sources for this release:
cd /usr/src
cat > sup
*default host=cvsup4.freebsd.org
*default base=/usr
*default prefix=/usr
*default release=cvs tag=RELENG_4_11
*default delete use-rel-suffix
*default compress
src-all

cvsup sup (2450, 4.10: 13mins, supermicro, 4.11: 11mins)

9. populate hosts
cat >> /etc/hosts
10.1.4.3 backup2

10. put key in authorized_keys on backup2
ssh-keygen -t dsa -b 1024 (default location, leave password blank)
scp /root/.ssh/id_dsa.pub user@backup2:/tmp/jail18pub
on backup2:
cat /tmp/jail18pub >> /root/.ssh/authorized_keys

confirm that you can ssh to backup2 without getting a login prompt

11. configure new kernel. Get config from similar machine or there may be a master copy somewhere under /mnt/data4/build (name the kernel config the same as the jail, ex jail4):
cd /usr/src/sys/i386/conf
scp backup2:/mnt/data4/build/freebsd/kern_config-4.11 ./jail18

edit the kernel config and change ident to be the name of the jail:
ident jail4

IMPORTANT CUSTOMIZATION:
for machines with >4G RAM, add to the config:
options PAE

for supermicro mobo’s with broadcom nics, add to the config:
device bge # Broadcom BCM570x (``Tigon III'')

for machines where lots of postgres might be running, change SHMMAXPGS:
options SHMMAXPGS=40960

edit /sys/conf/newvers.sh to add –jc2 to the end of the BRANCH string (RELEASE-p9-jc2)

12. bring over patches from backup2
The various patches are in /mnt/data4/build/freebsd/patches on backup2. There are dirs for each version. Not all dirs are populated, but patches for later versions work on older ones unless there is a new patch in the older dir.
cd /usr/src
scp backup2:"/mnt/data4/build/freebsd/patches/4.11/*" .
jail_proc_patch 100% |*************************************************| 2593 00:00
restore-patch 100% |*************************************************| 3295 00:00
411ps-jail-patch 100% |*************************************************| 2602 00:00
jail_rtprio_patch 100% |*************************************************| 301 00:00
udp-patch 100% |*************************************************| 594 00:00

Apply patches:
patch < 411ps-jail-patch
patch < jail_proc_patch
patch < restore-patch
patch < jail_rtprio_patch
patch < udp-patch

13. build, install kernel and world
make buildworld buildkernel installkernel (2450: 48min, supermicro: 20mins)
(Any compile errors can be looked up in /usr/include/sys/signal.h, other errors, do a rm -R /usr/obj/*)
make installworld (2450: 2min, supermicro: 1mins)
mergemaster -i
(answer no to most of it)

14. reboot. Confirm new kernel is loaded (uname -a)

15. update ports:
cd /usr/ports
cat > sup
*default host=cvsup4.FreeBSD.org
*default base=/usr
*default prefix=/usr
*default release=cvs tag=RELENG_4
*default delete use-rel-suffix
*default compress
ports-all tag=.

cvsup sup (2450: 26mins, supermicro: 26mins)

16. add console="comconsole" to /boot/loader.conf
cat >> /boot/loader.conf
console="comconsole"

17. edit /etc/ttys and turn off all ttyv's except 0 and 1
also turn on ttyd0, change type to vt100:
ttyd0 "/usr/libexec/getty std.9600" vt100 on secure

kill -1 1

on console:
vi /etc/remote
(rename port to jail4 depending on where and which digi plugged into)
test serial console

18. install linux_base:
cd /usr/ports/emulators/linux_base
make install clean (2450: 7min, supermicro: 2mins)

reibuild rpmdb cause we had probs installing aacapps
cd /compat/linux/bin
./rpm --initdb
./rpm --rebuilddb

19. install aacapps-4.1-0.i386.rpm
scp backup2:/mnt/data4/build/freebsd/aacapps-4.1-0.i386.rpm /tmp/.
/compat/linux/bin/rpm -ivh /tmp/aacapps-4.1-0.i386.rpm
cd /dev
sh MAKEDEV aac0

test out /compat/linux/usr/sbin/aaccli

20. edit root's path and login script:
vi /root/.cshrc
add to path: /usr/local/jail/bin /compat/linux/usr/sbin

Change alias entries (add G):
alias la ls -aG
alias lf ls -FAG
alias ll ls -lAG
alias ls ls -AG
alias mbm mb mount
alias mbu mb umount

and alter the prompt, set the following:
set prompt = "`/bin/hostname -s` %/# "

at the bottom of the file add:
set sshtty=`who am i|awk '{print $2}'`
/usr/sbin/rtprio 3 -`ps auxwJ | grep $sshtty | awk '{print $2}'`

set shortty=`who am i | awk '{print $2}' | sed -E 's/.*(..)$/\1/'`
foreach x (`ps cauxJ | grep sh | grep $shortty | awk '{print $2}'`)
/usr/sbin/rtprio 2 -$x
end

21. install rsync from ports
cd /usr/ports/net/rsync
make install clean

22. create & populate binaries/scripts dirs
mkdir -p /usr/local/jail/bin
mkdir -p /usr/local/jail/rc.d
scp backup2:"/mnt/data4/bin/freebsd/*" /usr/local/jail/bin
cd /usr/local/jail/rc.d/
touch quad1
touch quad2
touch quad3
touch quad4
touch safe1
touch safe2
touch safe3
touch safe4
chmod +x *
cd /usr/local/jail/bin
ln -s /usr/local/jail/rc.d/quad1 quad1
ln -s /usr/local/jail/rc.d/quad2 quad2
ln -s /usr/local/jail/rc.d/quad3 quad3
ln -s /usr/local/jail/rc.d/quad4 quad4
ln -s /usr/local/jail/rc.d/safe1 safe1
ln -s /usr/local/jail/rc.d/safe2 safe2
ln -s /usr/local/jail/rc.d/safe3 safe3
ln -s /usr/local/jail/rc.d/safe4 safe4
rehash

23. configure inetd to respond to mrtg load queries
cat >> /etc/inetd.conf
load stream tcp nowait user /usr/local/jail/bin/load.pl load.pl

cat >> /etc/services
load 12384/tcp

kill -HUP `cat /var/run/inetd.pid`

24. configure load mrtg, on mail
vi /usr/local/etc/mrtg/mrtg1.cfg
(add new entry to file following existing format)

25. configure bb on mail:
vi /usr/home/bb/bbsrc/bb1.9e-btf/etc/bb-hosts
10.1.4.104 jail4.johncompanies.com # ssh

su bb
cd /usr/home/bb/bbsrc/bb1.9e-btf
./runbb.sh stop
./runbb.sh start
exit

26. install bb client
adduser -group 1984 -shell /bin/csh -uid 1984 bb
cd /usr/home/bb
scp backup2:/mnt/data4/build/bb/bb-freebsd.tar .
tar xvf bb-freebsd.tar

cat > /home/bb/bbc1.9e-btf/etc/bb-hosts with something like:
10.1.4.105 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh
10.1.4.118 jail18.johncompanies.com # ssh

vi /home/bb/bbc1.9e-btf/ext/openfiles and change:
MACHINE="jail18,johncompanies,com" # HAS TO BE IN A,B,C FORM

cd /usr/home/bb/bbc1.9e-btf/etc
./bbchkcfg.sh
./bbchkhosts.sh (ignore ssh errors)
cd ../..
chown -R bb .
su bb
cd bbc1.9e-btf/
./runbb.sh start
more BBOUT (look for errors)
exit

cat > /usr/local/etc/rc.d/bb.sh
su - bb -c "cd /home/bb/bbc1.9e-btf; ./runbb.sh start"

chmod +x /usr/local/etc/rc.d/bb.sh

27. remove reserve space, enable softupdates (probably already set, so not necessary)
cd
umount /mnt/data1
umount /mnt/data2
tunefs -m 0 /mnt/data1
tunefs -m 0 /mnt/data2

mount -a

28. configure ntp
cat > /etc/ntp.conf
server 10.1.4.105

/usr/sbin/ntpd -p /var/run/ntpd.pid
ntpq -p
(confirm it’s able to reach our time server)

29. mrtg switch graphs

31. fwd and reverse lookups on ns1c
vi johncompanies.com
rr johncompanies.com
vi internal.johncompanies.com
rndc reload johncompanies.com IN private
(edit the PTR too)

32. create all /dev/vn and /dev/pty files in /dev
cat > /tmp/runme.sh
#!/bin/sh
cd /dev
for i in 1 2 3 4 5 6 7 ; do sh MAKEDEV pty$i ; done
for i in 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 ; do sh MAKEDEV vn$i ; done

NOTE: there are only 4 lines in the above output - however the lines may be wrapped on your terminal - make sure that there are only four lines in the script you make.

chmod +x /tmp/runme.sh
/tmp/runme.sh

rm /tmp/runme.sh

ls /dev/vn*|wc -l (make sure the output is a high number – larger than 128)
ls /dev/pty*|wc -l (make sure the output is exactly 256)

33. populate /etc/rc.conf with IPs and NFS settings
vi /etc/rc.conf

kern_securelevel_enable="NO"
portmap_enable="NO"
sendmail_enable="NO"
usbd_enable="NO"

xntpd_enable="YES"
nfs_client_enable="YES"
nfs_reserved_port_only="YES"
sshd_enable="YES"
inetd_enable="YES"
inetd_flags="-wW -a 10.1.4.104"

ifconfig_xl0="inet 10.1.4.104 netmask 255.255.255.0"
ifconfig_fxp0="inet 69.55.228.101 netmask 255.255.255.0"
defaultrouter="69.55.228.1"
ifconfig_fxp0_alias0="inet 69.55.2xx.xx netmask 255.255.255.0"

static_routes="t1 office"
route_t1="-net 10.1.5 10.1.4.2"
route_office="-net 10.1.6 10.1.4.2"

34. make sure sysctls are set and preserved after reboot
cat >> /etc/sysctl.conf
kern.consmute=0
jail.sysvipc_allowed=1
kern.ipc.shm_use_phys=1
kern.ipc.shmall=65535
kern.ipc.shmmax=134217728
net.inet.tcp.syncookies=0
kern.maxfiles=32768
kern.fallback_elf_brand=3
kern.maxprocperuid=4000
jail.max_procs_per_jail: 1026

35. reboot

36. create the jail template
vnconfig -T -S 1g -Z -s labels -c /dev/vn1 /mnt/data1/jail
disklabel -r -w vn1 auto
newfs /dev/vn1c
cd /usr/src
mkdir -p /mnt/data1/jail-DIR
mount /dev/vn1c /mnt/data1/jail-DIR
make world DESTDIR=/mnt/data1/jail-DIR (2450: 45mins, supermicro: 19mins)
cd etc
make distribution DESTDIR=/mnt/data1/jail-DIR -DNO_MAKEDEV_RUN
cd /mnt/data1/jail-DIR/dev
sh MAKEDEV jail
cd /mnt/data1/jail-DIR
ln -sf dev/null kernel

jail /mnt/data1/jail-DIR testhostname 192.168.11.100 /bin/sh
csh
touch /etc/fstab
cat > /etc/rc.conf
portmap_enable="NO"
network_interfaces=""
hostname="newsystem"
kern_securelevel_enable="NO"
sendmail_enable="YES"
sshd_enable="YES"

cat >> /etc/resolv.conf
nameserver 69.55.225.225
nameserver 69.55.230.3

vi /etc/crontab
remove the adjkerntz lines
comment out periodic’s and put this line above them:
# DO NOT UNCOMMENT THESE

rm -rf /etc/periodic/daily/400.status-disks

mv /bin/df /bin/df_

cat > /bin/df
#!/bin/sh
/bin/df_ $* .

chmod +x /bin/df

cat > /sbin/mount
echo `df | tail -1 | awk '{print $1 " on " $6 " (ufs, local)"}'`

check /tmp for crap

vi /etc/periodic/security/100.chksetuid
replace: MP=`mount -t ufs | grep -v " nosuid" | awk '{ print $3 }' | sort`
with: MP='/'

mkdir -p /usr/compat/linux/dev

adduser (Add account for user)

put user in wheel group
vi /etc/group

vi /usr/home/user/.profile (and add to the file):
TERM=vt100; export TERM

cd /etc
vipw -d .
root:$1$krszPxhk$xkCepSnz3mIikT3vCtJCt0:0:0::0:0:Charlie &:/root:/bin/csh
user:$1$Mx9p5Npk$QdMU6c8YQqp2FW2M3irEh/:1001:1001::0:0:User &:/home/user:/bin/sh

tzsetup

newaliases

chflags schg /dev/*mem

cd /dev
rm console
ln -s null console

exit
exit

cd /usr/ports
make -DNOCLEANDEPENDS clean (2450: 47mins , supermicro: 17mins)
rm -fr /usr/ports/distfiles/*
cp -r /usr/ports /mnt/data1/jail-DIR/usr

cd /mnt/data1/jail-DIR/dev
rm kmem
mknod kmem c 2 1 root:kmem
chmod 640 kmem
rm mem
mknod mem c 2 0 root:kmem
chmod 640 mem

rm /mnt/data1/jail-DIR/root/.history

sh
for i in 1 2 3 4 5 6 7 ; do sh MAKEDEV pty$i ; done
exit

cd /mnt/data1/jail-DIR/usr/compat/linux/dev
mknod null c 2 2
mknod random c 2 3

cd
mkdir /usr/local/jail/template/
dump -0a -f /usr/local/jail/template/411template /dev/vn1

umount /dev/vn1c
vnconfig -u /dev/vn1

rm /mnt/data1/jail
rm -fr /mnt/data1/jail-DIR

37. setup backups
cat > /usr/local/jail/bin/backup.config
#!/bin/sh
backupdir=/mnt/data2/jail18_rsync

## ENTRY /etc
## ENTRY /usr/local/etc
## ENTRY /usr/local/jail
## ENTRY /root/logs

on backup2:
setup backup dirs:
mkdir -p /mnt/data2/jail18_rsync/0_today
mkdir -p /mnt/data2/jail18_rsync/1_yesterday
mkdir -p /mnt/data2/jail18_rsync/2_two_day

add the system to /mnt/data1/bin/rsync_houseclean

on mail:
vi /usr/local/www/mgmt/cgi/backupgraph.pl
(add hostname)

38. mkdir /root/logs

39. edit sshd_config for security
vi /etc/ssh/sshd_config
ListenAddress 69.55.228.101
ListenAddress 10.1.4.104

kill -1 `cat /var/run/sshd.pid`

40. add crontab entries
crontab -e
5 0 * * * /usr/local/jail/bin/backup
1 0 1 * * /usr/local/jail/bin/ipfwreset
0 18 * * * /usr/local/jail/bin/ipfwbackup
4,9,14,19,24,29,34,39,44,49,55,59 * * * * /usr/local/jail/bin/trafstats
55 10,23 * * * /usr/local/jail/bin/trafficwatch.pl

41. Reboot notify script
cat > /usr/local/etc/rc.d/notify.sh
echo "`/bin/hostname` rebooted" | /usr/bin/mail reboot@johncompanies.com

chmod +x /usr/local/etc/rc.d/notify.sh

42. copy jailmake from prev system
scp user@10.1.4.118:/usr/local/jail/bin/jailmake /usr/local/jail/bin
rehash
NOTE: remove df altering code from jailmake since we put the correct df in the template, and make sure path to template file is right

43. add to templates via mgmt system

44. add to server/cabinet map. On mail:
vi /usr/local/www/mgmt/html/cabinetmap.html

45. add an outside blocking rule to the firewall, so this machine can only be reached from inside the firewall. Follow example already in firewall jail17 is:

00117 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 or 69.55.238.150 } to 69.55.228.2
00117 deny ip from any to 69.55.228.2

jail4 would be 00104...
ipfw add 00118 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 or 69.55.238.150 } to 69.55.228.140
ipfw add 00118 deny ip from any to 69.55.228.140

46. select customers for probe map
</pre>

= FreeBSD 6.1 =

Last updated 2006-05-09

<pre>All time extimates below assume disks aren’t scrubbing

1. make sure bios is setup for bios console redirect
Supermicro:
Console redirection:
Com port addr: on-board COM A
Baud: 38400
Console type: vt100
Flow control: none
Console connection: direct
Continue cr after post: off

2450:
Make sure running bios A09
Console Redirection: VT100/VT220

2. assuming mirrors (or at least disks) created (if not, refer to this), boot to disk 1 of 5.4
skip kernel config (enter)
custom install

partition ->
move cursor to amrd0, hit space (takes you to partition map screen)
a for entire disk
q to quit and save
standard mbr (no boot manager)
space to unselect aacd0
cursor over aacd1
space
a for entire disk
q to quit and save
none (leave untouched)

cursor over aacd0
space
(takes you into part. Screen again) q to exit
none
Make sure both are checked and tab to ok

Label ->
Make sure aacd0 is highlighted
a to start with defaults

john likes:
/1g
/var 256m

Glenn likes:
/128
/var 256
/usr 3g
/tmp 256

We do:
delete everything
/ 128M
swap 2G
/var 256M
/tmp 256M
/usr 3G
/mnt/data1 remaining space

Make sure to toggle S for soft updates on all (should look like UFS2+S Y under the Newfs column)

move cursor to aacd1
swap 2G
/mnt/data2 remaining space

q to save and exit

distributions ->
developer
yes to install ports
exit

media ->
cd

commit ->
yes
(2450: 14mins, supermicro: 12mins)

yes to "visit general config" ->
Set root pwd

Add user ‘user’ member group is wheel, set password

Set tz

Networking->interfaces->Fxp0 ->
No IPV6
dhcp=no
Set hostname & domain
Enable sshd

exit...
exit install ->
yes

take the cd out and let the machine reboot
when it comes back up, enter junk when it asks for key seed

3. double check the date/time

4. edit /etc/make.conf (only add the console speed line if this is a supermicro capable of outputting BIOS redirect at that speed – 2450’s can only do 9600)
echo "WITHOUT_X11=yes \
KERNCONF=jail19 \
BOOT_COMCONSOLE_SPEED=38400" >> /etc/make.conf

5. add console="comconsole" to /boot/loader.conf
echo "console=""comconsole""" >> /boot/loader.conf

6. turn off all ttyv's except 0 and 1 in /etc/ttys
also turn on ttyd0, change type to vt100:
vi /etc/ttys
ttyd0 "/usr/libexec/getty std.9600" vt100 on secure

kill -1 1

on console server:
vi /etc/remote
(rename port to jail18 depending on where and which digi plugged into)
test serial console

7. install cvsup
cd /usr/ports/net/cvsup-without-gui
make install clean; rehash
(stay close for gettext options, 2450: 27mins, supermicro: 17mins)

8. get latest sources for this release:
cd /usr/src
echo "*default host=cvsup4.freebsd.org\
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_6_1\
*default delete use-rel-suffix\
*default compress\
src-all" > sup

cvsup sup
(2450, ~12mins, supermicro, 27mins)

9. populate hosts
echo "10.1.4.3 backup2" >> /etc/hosts

10. put key in authorized_keys on backup2
cd
ssh-keygen -t dsa -b 1024
(default location, leave password blank)
scp /root/.ssh/id_dsa.pub user@backup2:/tmp/jailkey

cat /root/.ssh/id_dsa.pub | ssh backup2 'cat - >> /root/.ssh/authorized_keys'

confirm that you can ssh to backup2 without getting a login prompt

ssh backup2

11. configure new kernel. Get config from similar machine or there may be a master copy somewhere under /mnt/data4/build/freebsd (name the kernel config the same as the jail, ex jail18):
cd /usr/src/sys/i386/conf
scp backup2:/mnt/data4/build/freebsd/kern_config-6.1 ./jail19

edit the kernel config and change ident to be the name of the jail:
vi jail14
ident jail14

edit /sys/conf/newvers.sh to add –jc1 to the end of the BRANCH string (RELEASE-jc1)
vi /sys/conf/newvers.sh

12. bring over patches from backup2
The various patches are in /mnt/data4/build/freebsd/patches on backup2. There are dirs for each version. Not all dirs are populated, but patches for later versions work on older ones unless there is a new patch in the older dir.
cd /usr/src
scp backup2:"/mnt/data4/build/freebsd/patches/6.1/*" .
jail_proc_patch 100% |*************************************************| 2593 00:00
restore-patch 100% |*************************************************| 3295 00:00
54ps-jail-patch 100% |*************************************************| 2602 00:00
jail_rtprio_patch 100% |*************************************************| 301 00:00
udp-patch 100% |*************************************************| 594 00:00

Apply patches:
patch < 54ps-jail-patch
patch < jail_proc_patch
patch < restore-patch
patch < jail_rtprio_patch
patch < udp-patch

13. build, install kernel and world
cd /usr/src
make buildworld buildkernel installkernel
(2450: 1:56min, supermicro::59mins)
make installworld
(2450: 3min, supermicro: 1min)
mergemaster -i
delete /var/tmp/temproot
delete bsnmpd
delete temporary ./etc/hosts
delete temporary ./etc/motd
delete /var/tmp/temproot

14. reboot. Confirm new kernel is loaded

uname –a

15. update ports:
cd /usr/ports
echo "*default host=cvsup4.FreeBSD.org\
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_6_1\
*default delete use-rel-suffix\
*default compress\
ports-all tag=." > sup

cvsup sup
(2450: 18mins, supermicro: 19mins)

18. (only applies if adaptec card installed)
install linux_base:
cd /usr/ports/emulators/linux_base
make install clean
(2450: 7min, supermicro: 3mins)

reibuild rpmdb cause we had probs installing aacapps
cd /compat/linux/bin
./rpm --initdb
./rpm --rebuilddb

install aacapps-4.1-0.i386.rpm
scp backup2:/mnt/data4/build/freebsd/aacapps-4.1-0.i386.rpm /tmp/.
/compat/linux/bin/rpm -ivh --excludepath=/dev /tmp/aacapps-4.1-0.i386.rpm

test out;
/compat/linux/usr/sbin/aaccli

20. edit root's path and login script:
vi /root/.cshrc
add to path:
/usr/local/jail/bin
(if adaptec card installed, also add /compat/linux/usr/sbin)

Change alias entries (add G):
alias la ls -aG
alias lf ls -FAG
alias ll ls -lAG
alias ls ls -AG
alias mbm mb mount
alias mbu mb umount

and alter the prompt, set the following:
set prompt = "`/bin/hostname -s` %/# "

at the bottom of the file add:
set sshtty=`who am i|awk '{print $2}'`
/usr/sbin/rtprio 3 -`psj | grep $sshtty | awk '{print $2}'`

set shortty=`who am i | awk '{print $2}' | sed -E 's/.*(..)$/\1/'`
foreach x (`psj | grep sh | grep $shortty | awk '{print $2}'`)
/usr/sbin/rtprio 2 -$x
end

21. install rsync from ports
cd /usr/ports/net/rsync
make install clean

choose default options

21. install perl from ports
PROB NOT NECESSARY – INSTALLED WITH LINUX I THINK
cd /usr/ports/lang/perl5.8/
make install clean; rehash
(supermicro: 5min)

22. create & populate binaries/scripts dirs
mkdir -p /usr/local/jail/bin
mkdir -p /usr/local/jail/rc.d
mkdir /mnt/data1
mkdir /mnt/data2
scp backup2:"/mnt/data4/bin/freebsd6.x/*" /usr/local/jail/bin
cd /usr/local/jail/rc.d/
touch quad1
touch quad2
touch quad3
touch quad4
touch safe1
touch safe2
touch safe3
touch safe4
chmod +x *
cd /usr/local/jail/bin
ln -s /usr/local/jail/rc.d/quad1 quad1
ln -s /usr/local/jail/rc.d/quad2 quad2
ln -s /usr/local/jail/rc.d/quad3 quad3
ln -s /usr/local/jail/rc.d/quad4 quad4
ln -s /usr/local/jail/rc.d/safe1 safe1
ln -s /usr/local/jail/rc.d/safe2 safe2
ln -s /usr/local/jail/rc.d/safe3 safe3
ln -s /usr/local/jail/rc.d/safe4 safe4

rehash

23. configure inetd to respond to mrtg load queries
echo "load stream tcp nowait user /usr/local/jail/bin/load.pl load.pl" >> /etc/inetd.conf

echo "load 12384/tcp" >> /etc/services

26. install bb client
adduser
cd /usr/home/bb
scp backup2:/mnt/data4/build/bb/bb-freebsd.tar .
tar xvf bb-freebsd.tar

edit /home/bb/bbc1.9e-btf/etc/bb-hosts with something like:
echo "10.1.4.105 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh \
10.1.4.103 jail3.johncompanies.com # ssh" > /home/bb/bbc1.9e-btf/etc/bb-hosts

vi /home/bb/bbc1.9e-btf/ext/openfiles
MACHINE="jail19,johncompanies,com" # HAS TO BE IN A,B,C FORM

cd /usr/home/bb/bbc1.9e-btf/etc
./bbchkcfg.sh (y to questions)
./bbchkhosts.sh (ignore ssh errors)
cd ../..
chown -R bb .
su bb
cd
cd bbc1.9e-btf/src
make; make install
cd ..
./runbb.sh start
more BBOUT
(look for errors)
exit

echo 'su - bb -c "cd /home/bb/bbc1.9e-btf; ./runbb.sh start"' > /usr/local/etc/rc.d/bb.sh
chmod +x /usr/local/etc/rc.d/bb.sh

modify bb-msgtab to look for raid failures

24. configure load mrtg, on mail
vi /usr/local/www/mgmt/mrtg/mrtg1.cfg
(add new entry to file following existing format)

25. configure bb on mail:
vi /usr/home/bb/bbsrc/bb1.9e-btf/etc/bb-hosts
10.1.4.119 jail19.johncompanies.com # ssh

su bb
cd
bbsrc/bb/runbb.sh restart ; exit

27. remove reserve space, enable softupdates (probably already set, so not necessary)
NOT APPLICABLE IF USING GVINUM
cd
umount /mnt/data1
umount /mnt/data2
tunefs -m 0 /mnt/data1
tunefs -m 0 /mnt/data2
mount -a

28. configure ntp
echo "server 10.1.4.105" > /etc/ntp.conf

/usr/sbin/ntpd -p /var/run/ntpd.pid
sleep 2; ntpq -p
(confirm it’s able to reach our time server)

29. mrtg switch graphs

31. fwd and reverse lookups on ns1c
vr johncompanies.com
vi internal.johncompanies.com
rndc reload johncompanies.com IN private
(edit the PTR too)

33. populate /etc/rc.conf with IPs and NFS settings
vi /etc/rc.conf

kern_securelevel_enable="NO"
portmap_enable="NO"
sendmail_enable="NO"
usbd_enable="YES"

xntpd_enable="YES"
nfs_client_enable="YES"
nfs_reserved_port_only="YES"
sshd_enable="YES"
inetd_enable="YES"
inetd_flags="-wW -a 10.1.4.119"
devfs_system_ruleset="devfsrules_show_all"

ifconfig_xl0="inet 10.1.4.118 netmask 255.255.255.0"
ifconfig_fxp0="inet 69.55.228.101 netmask 255.255.255.0"
defaultrouter="69.55.228.1"
ifconfig_fxp0_alias0="inet 69.55.2xx.xx netmask 255.255.255.0"

static_routes="t1 office"
route_t1="-net 10.1.5 10.1.4.2"
route_office="-net 10.1.6 10.1.4.2"
gvinum_enable="YES"
fsck_y_enable="YES"
background_fsck="NO"

34. make sure sysctls are set and preserved after reboot
echo "kern.consmute=0\
kern.ipc.shm_use_phys=1\
kern.ipc.shmall=65535\
kern.ipc.shmmax=134217728\
net.inet.tcp.syncookies=0\
kern.maxfiles=32768\
kern.fallback_elf_brand=3\
kern.maxprocperuid=4000\
security.jail.sysvipc_allowed=1\
security.jail.max_procs_per_jail: 1026\
security.jail.allow_raw_sockets=1\
security.jail.socket_unixiproute_only=1\
security.jail.chflags_allowed=0" >> /etc/sysctl.conf

35. mount procfs
echo "proc /proc procfs rw 0 0" >> /etc/fstab

36. enable noatime option
NOT APPLICABLE IF RUNNING GVINUM
data1 and data2 should look something like:
/dev/amrd0s1g /mnt/data1 ufs rw,noatime 2 2

36. populate devfs ruleset
scp backup2:/mnt/data4/build/freebsd/devfs.rules /etc

35. reboot
Check rules:
devfs rule showsets
devfs rule -s 3 show

36. create gvinum volumes

Make a g partition:

bsdlabel -e /dev/aacd0s1

given:
# /dev/aacd0s1:
8 partitions:
# size offset fstype [fsize bsize bps/cpg]
a: 262144 0 4.2BSD 2048 16384 16392
b: 4194304 262144 swap
c: 143363997 0 unused 0 0 # "raw" part, don't edit
d: 524288 4456448 4.2BSD 2048 16384 32776
e: 524288 4980736 4.2BSD 2048 16384 32776
f: 6291456 5505024 4.2BSD 2048 16384 28552

new offset = 6291456 + 5505024 = 11796480
new size is size for 'c' partition minus the new start from above
143363997 - 11796480 = 131567517
So:
g: 131567517 11796480 unused 0 0

For a 73G drive (after OS), we can fit 31 2G volumes so:

echo 'drive data1 device /dev/aacd0s1g' > /tmp/cgv
sh
for f in 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31;\
do echo "volume v$f" >> /tmp/cgv; echo 'plex org concat' >> /tmp/cgv;\
echo 'sd length 2g drive data1' >> /tmp/cgv; done; exit

gvinum create /tmp/cgv

For a 146G drive (-4G for swap), we can fit 66 2G volumes so:

echo 'drive data2 device /dev/aacd1s1g' > /tmp/cgv
sh
for f in 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97;\
do echo "volume v$f" >> /tmp/cgv; echo 'plex org concat' >> /tmp/cgv;\
echo 'sd length 2g drive data2' >> /tmp/cgv; done; exit

gvinum create /tmp/cgv

For 3rd 73G drive (after 2G swap), we can fit 33 2G volumes so:
Label should be:
# /dev/aacd2s1:
8 partitions:
# size offset fstype [fsize bsize bps/cpg]
c: 143299737 0 unused 0 0 # "raw" part, don't edit
g: 143299721 16 unused 0 0

echo 'drive data3 device /dev/aacd2s1g' > /tmp/cgv
sh
for f in 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131;\
do echo "volume v$f" >> /tmp/cgv; echo 'plex org concat' >> /tmp/cgv;\
echo 'sd length 2g drive data3' >> /tmp/cgv; done

gvinum create /tmp/cgv

For a 2nd 73G drive (after 2G swap), we can fit 33 2G volumes so:

echo 'drive data2 device /dev/aacd1s1g' > /tmp/cgv
sh
for f in 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64;\
do echo "volume v$f" >> /tmp/cgv; echo 'plex org concat' >> /tmp/cgv;\
echo 'sd length 2g drive data2' >> /tmp/cgv; done; exit

gvinum create /tmp/cgv

36. create the jail template

mkdir /mnt/jail
newfs /dev/gvinum/v1
mount /dev/gvinum/v1 /mnt/jail

cd /usr/src
make clean
rm -fr /usr/obj/
make world DESTDIR=/mnt/jail
(2450: 2:28mins, supermicro: 55mins)
cd etc
make distribution DESTDIR=/mnt/jail
mount_devfs devfs /mnt/jail/dev
devfs -m /mnt/jail/dev rule -s 3 applyset
cd /mnt/jail
ln -sf dev/null kernel

jail /mnt/jail testhostname 192.168.11.100 /bin/sh
csh
touch /etc/fstab
echo 'network_interfaces=""\
hostname="newsystem"\
kern_securelevel_enable="NO"\
sendmail_enable="YES"\
sshd_enable="YES"' > /etc/rc.conf

echo "nameserver 69.55.225.225\
nameserver 69.55.230.3" >> /etc/resolv.conf

vi /etc/crontab
remove the adjkerntz lines
comment out periodic’s and put this line above them:
# DO NOT UNCOMMENT THESE

rm -rf /etc/periodic/daily/400.status-disks

check /tmp for crap

vi /etc/periodic/security/100.chksetuid
replace: MP=`mount -t ufs | grep -v " nosuid" | awk '{ print $3 }' | sort`
with: MP='/' (use single quotes)

mkdir -p /usr/compat/linux/dev

adduser (Add account for user, make sure in wheel group)

user/root passwd: 8ico2987

Set root password

vi /usr/home/user/.profile (and add to the file):
TERM=vt100; export TERM

tzsetup

newaliases

#cd /dev
#rm console
#ln -s null console

exit
exit

cd /usr/ports
make -DNOCLEANDEPENDS clean
(2450: 15mins , supermicro: 29mins)
rm -fr /usr/ports/distfiles/*
cp -r /usr/ports /mnt/jail/usr (2450: 2:00 mins , supermicro: 15mins)

rm /mnt/jail/root/.history

cd
mkdir /usr/local/jail/template/
dump -0a -f /usr/local/jail/template/61template /dev/gvinum/v1

umount /mnt/jail/dev
umount /dev/gvinum/v1

rm -fr /mnt/jail

37. setup backups
echo '#\!/bin/sh\
backupdir=/mnt/data3jail3_rsync\
\
## ENTRY /etc\
## ENTRY /usr/local/etc\
## ENTRY /usr/local/jail\
## ENTRY /root/logs' > /usr/local/jail/bin/backup.config

on backup2:
setup backup dirs:
mkdir -p /mnt/data3/jail3/0

add the system to
vi /mnt/data4/bin/snapshot_rotate

on mail:
vi /usr/local/www/mgmt/cgi/backupgraph.pl
(add hostname)

38. mkdir /root/logs

39. edit sshd_config for security
vi /etc/ssh/sshd_config
ListenAddress 69.55.228.101
ListenAddress 10.1.4.118

kill -1 `cat /var/run/sshd.pid`

40. add crontab entries
crontab -e
5 0 * * * /usr/local/jail/bin/backup
1 0 1 * * /usr/local/jail/bin/ipfwreset
0 18 * * * /usr/local/jail/bin/ipfwbackup
4,9,14,19,24,29,34,39,44,49,55,59 * * * * /usr/local/jail/bin/trafstats
55 10,23 * * * /usr/local/jail/bin/trafficwatch.pl

41. Reboot notify script
ln -s /usr/local/jail/bin/notify.sh /usr/local/etc/rc.d/notify.sh

42. copy jailmake from prev system
scp user@10.1.4.119:/usr/local/jail/bin/jailmake /usr/local/jail/bin
rehash
NOTE: remove df altering code from jailmake since we put the correct df in the template, and make sure path to template file is right

43. add to management db (on mail and devweb) jc.ref_machines and jc.ref_templates

uname -r
5.4-RELEASE-p2-jc2

insert into ref_machines values (null,'jail19','jail19.johncompanies.com',0,'l');
select machine_id from ref_machines where host='jail19';
+------------+
| machine_id |
+------------+
| 35 |
+------------+
insert into ref_templates values ('',' 6.2-RELEASE-jc1',5,'FreeBSD 6.2',0);

44. add to server/cabinet map. On mail:
vi /usr/local/www/mgmt/html/cabinetmap.html

45. add an outside blocking rule to the firewall, so this machine can only be reached from inside the firewall. Follow example already in firewall jail17 is:

00119 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 or 69.55.238.150 } to 69.55.228.200
00119 deny ip from any to 69.55.228.200

jail19 would be 00119...
ipfw add 00119 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 or 69.55.238.150 } to 69.55.228.200
ipfw add 00119 deny ip from any to 69.55.228.200

46. select customers for probe map

47. install raid monitor

cd /usr/ports/sysutils/asr-utils
make install clean

48. make gv start on boot

scp backup2:/mnt/data4/build/freebsd/gvinum /etc/rc.d/gvinum

gconcat label -v somelabel /dev/gvinum/a /dev/gvinum/b
bsdlabel -r -w /dev/concat/somelabel
newfs /dev/concat/somelabela
mount /dev/concat/somelabel /mount/point

umount /dev/concat/somelabel
gconcat stop somelabel
gconcat label -v /dev/gvinum/a /dev/gvinum/b /dev/gvinum/c /dev/gvinum/d
growfs /dev/concat/somelabel
mount /dev/concat/somelabel /mount/point

volume f
plex org concat
sd length 30449m drive data1

where f is

D data1 State: up /dev/aacd0s1g A: 30449/64241 MB (47%)

gvinum rm -r f

so i setup 2 machines with 6.1. A had 2 logical drives and gv's created across both. B had 1 logical drive and gv's across the 1 drive.
the labeling for the gv's was v1-vN (till i ran out of space)
when i moved aac1 from A to B, the gv's on A's aac1, took precedence over the similarly labeled gv's on B's aac0. in other words. B's aac0 used to have v1-v30. A's aac1 had vn6-vn30. when A's aac1 was moved to B, v1-v5 were linked to B's aac0, and v6-v30 linked to A's aac1
i relabeled B's aac0 gv's to something different (not v1-vN) and tried again and they all showed up.
i tried the experiment again this time moving A's aac0 to B. at that point nothing worked. i moved A's aac0 back to A and renamed the device (data1) to something else, then tried the test again. still no beans.
it's hazy what i did after that cause the machine's weren't cooperating and i was trying to rename the device but basically i think i need to repeat the test and see if i can re-define the device, and probably also give it a unique name and i bet it would have worked.
also interesting- A's aac1 contained data about A's aac0 which showed up when i put aac1 into B
skeeter: as for the gv stuff, it definitely sounds like using some sort of serial numbering scheme would be the way to go if you want to be able to move disks around....
that overlap is still an issue (aac1's device was 'data2') when i had A's aac0 in B nothing worked both disks gv devices were called data1
skeeter: I suppose you could serialize those names as well...</pre>

= FreeBSD 6.2 =

Last updated 2007-10-15

<pre>All time extimates below assume disks aren’t scrubbing. Setup instructions below are for LSI card:

1. make sure bios is setup for bios console redirect
Supermicro:
Console redirection:
Com port addr: on-board COM A
Baud: 38400
Console type: vt100
Flow control: none
Console connection: direct
Continue cr after post: on

2450:
Make sure running bios A09
Console Redirection: VT100/VT220

2. assuming mirrors (or at least disks) created (if not, refer to this), boot to disk 1 of 5.4
skip kernel config (enter)
custom install

partition ->
move cursor to amrd0, hit space (takes you to partition map screen)
a for entire disk
q to quit and save
standard mbr (no boot manager)
space to unselect amrd0
cursor over amrd1
space
a for entire disk
q to quit and save
none (leave untouched)

cursor over amrd0
space
(takes you into part. Screen again) q to exit
none
Make sure both are checked and tab to ok

Label ->
Make sure amrd0 is highlighted
a to start with defaults

john likes:
/1g
/var 256m

Glenn likes:
/128
/var 256
/usr 3g
/tmp 256

We do:
delete everything
/ 128M
swap 2G (for 2950 make it 4G)
/var 256M
/tmp 256M
/usr 3.5G (3584M)
/mnt/data1 remaining space (no need to newfs)

Make sure to toggle S for soft updates on all (should look like UFS2+S Y under the Newfs column)

move cursor to amrd1
swap 4G
/mnt/data2 remaining space (no need to newfs)

q to save and exit

distributions ->
developer
yes to install ports
exit

media ->
cd

commit ->
yes
(2450: 14mins, supermicro: 12mins)

yes to "visit general config" ->
Set root pwd

Add user ‘user’ member group is wheel, set password

Set tz

Networking->interfaces->Fxp0 ->
No IPV6
dhcp=no
Set hostname & domain
Enable sshd

exit...
exit install ->
yes

take the cd out and let the machine reboot

3. double check the date/time

4. edit /etc/make.conf (only add the console speed line if this is a supermicro capable of outputting BIOS redirect at that speed – 2450’s can only do 9600)
echo "WITHOUT_X11=yes \
KERNCONF=jail7 \
BOOT_COMCONSOLE_SPEED=115200" >> /etc/make.conf

5. add console="comconsole" to /boot/loader.conf
echo "console=""comconsole""" >> /boot/loader.conf

6. turn off all ttyv's except 0 and 1 in /etc/ttys
also turn on ttyd0, change type to vt100:
vi /etc/ttys
ttyd0 "/usr/libexec/getty std.9600" vt100 on secure

kill -1 1

on console server:
vi /etc/remote
(rename port to jail18 depending on where and which digi plugged into)
test serial console

7. populate hosts
echo "10.1.4.3 backup2" >> /etc/hosts

8. put key in authorized_keys on backup2
cd
ssh-keygen -t dsa -b 1024
(default location, leave password blank)

cat /root/.ssh/id_dsa.pub | ssh backup2 'cat - >> /root/.ssh/authorized_keys'

confirm that you can ssh to backup2 without getting a login prompt

ssh backup2

9. create & populate binaries/scripts dirs
mkdir -p /usr/local/jail/bin
mkdir -p /usr/local/jail/rc.d
mkdir -p /usr/local/jail/template/
mkdir /mnt/data1
mkdir /mnt/data2
scp backup2:"/mnt/data4/bin/freebsd6.x/*" /usr/local/jail/bin
cd /usr/local/jail/rc.d/
touch quad1
touch quad2
touch quad3
touch quad4
touch safe1
touch safe2
touch safe3
touch safe4
chmod +x *
cd /usr/local/jail/bin
ln -s /usr/local/jail/rc.d/quad1 quad1
ln -s /usr/local/jail/rc.d/quad2 quad2
ln -s /usr/local/jail/rc.d/quad3 quad3
ln -s /usr/local/jail/rc.d/quad4 quad4
ln -s /usr/local/jail/rc.d/safe1 safe1
ln -s /usr/local/jail/rc.d/safe2 safe2
ln -s /usr/local/jail/rc.d/safe3 safe3
ln -s /usr/local/jail/rc.d/safe4 safe4

rehash

10. edit root's path and login script:
vi /root/.cshrc
add to path:
/usr/local/jail/bin
(if adaptec card installed, also add /compat/linux/usr/sbin)

Change alias entries (add G):
alias la ls -aG
alias lf ls -FAG
alias ll ls -lAG
alias ls ls -AG
alias mbm mb mount
alias mbu mb umount

and alter the prompt, set the following:
set prompt = "`/bin/hostname -s` %/# "

at the bottom of the file add:
set sshtty=`who am i|awk '{print $2}'`
/usr/sbin/rtprio 3 -`psj | grep $sshtty | awk '{print $2}'`

set shortty=`who am i | awk '{print $2}' | sed -E 's/.*(..)$/\1/'`
foreach x (`psj | grep sh | grep $shortty | awk '{print $2}'`)
/usr/sbin/rtprio 2 -$x
end

To load the new file:
source /root/.cshrc

11. install cvsup
cd /usr/ports/net/cvsup-without-gui
make install clean; rehash; mail -s 'cvs installed' dave.boodman@vtext.com < /dev/null

(stay close for gettext options, 2450: 27mins, supermicro: 17mins, 2950: 22mins)

12. get latest sources for this release:
cd /usr/src
echo "*default host=cvsup4.freebsd.org\
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_6_2\
*default delete use-rel-suffix\
*default compress\
src-all" > sup

cvsup sup ; mail -s 'cvs sup done' dave.boodman@vtext.com < /dev/null

(2450, ~12mins, supermicro, 27mins, 2950: 7mins)

13. configure new kernel. Get config from similar machine or there may be a master copy somewhere under /mnt/data4/build/freebsd (name the kernel config the same as the jail, ex jail18):
cd /usr/src/sys/i386/conf
scp backup2:/mnt/data4/build/freebsd/kern_config-6.2 ./jail7

or for PAE

scp backup2:/mnt/data4/build/freebsd/kern_config-6.2-PAE ./jail7

edit the kernel config and change ident to be the name of the jail:
vi jail7
ident jail7

edit /sys/conf/newvers.sh to add –jc1 to the end of the BRANCH string (RELEASE-jc1)
vi /sys/conf/newvers.sh

14. bring over patches from backup2

The various patches are in /mnt/data4/build/freebsd/patches on backup2. There are dirs for each version. Not all dirs are populated, but patches for later versions work on older ones unless there is a new patch in the older dir.

cd /usr/src
scp backup2:"/mnt/data4/build/freebsd/patches/6.x/*" .

Apply patches:
patch -l < jls-patch

Apply these only to 2950 with PAE:
patch -p0 < mfi-patch
patch -p0 < gvinum-staticcompile-patch
patch -p0 < gvinum-bin-patch

15. build, install kernel and world
cd /usr/src
make buildworld buildkernel installkernel; mail -s 'kernel build done' dave.boodman@vtext.com < /dev/null
(2450: 1:56min, supermicro: 59mins, 2950: 38mins)
make installworld
(2450: 3min, supermicro: 1min, 2950: :34)
mergemaster -i
delete /var/tmp/temproot
delete bsnmpd
delete temporary ./etc/hosts
delete temporary ./etc/motd
delete /var/tmp/temproot

16. populate devfs ruleset
scp backup2:/mnt/data4/build/freebsd/devfs.rules /etc

17. populate /etc/rc.conf with IPs and NFS settings
vi /etc/rc.conf

kern_securelevel_enable="NO"
portmap_enable="NO"
sendmail_enable="NO"
usbd_enable="YES"

xntpd_enable="YES"
nfs_client_enable="YES"
nfs_reserved_port_only="YES"
sshd_enable="YES"
inetd_enable="YES"
inetd_flags="-wW -a 10.1.4.119"
devfs_system_ruleset="devfsrules_show_all"

ifconfig_xl0="inet 10.1.4.118 netmask 255.255.255.0"
ifconfig_fxp0="inet 69.55.228.101 netmask 255.255.255.0"
defaultrouter="69.55.228.1"
ifconfig_fxp0_alias0="inet 69.55.2xx.xx netmask 255.255.255.0"

static_routes="t1 office"
route_t1="-net 10.1.5 10.1.4.2"
route_office="-net 10.1.6 10.1.4.2"
gvinum_enable="YES"
fsck_y_enable="YES"
background_fsck="NO"

18. make sure sysctls are set and preserved after reboot
echo "kern.consmute=0\
kern.ipc.shm_use_phys=1\
kern.ipc.shmall=65535\
kern.ipc.shmmax=134217728\
net.inet.tcp.syncookies=0\
kern.maxfiles=32768\
kern.fallback_elf_brand=3\
kern.maxprocperuid=4000\
security.jail.sysvipc_allowed=1\
security.jail.max_procs_per_jail: 1026\
security.jail.allow_raw_sockets=1\
security.jail.socket_unixiproute_only=1\
security.jail.chflags_allowed=0\
dev.amr.0.allow_volume_configure=1\
compat.linux.osrelease=2.6.12" >> /etc/sysctl.conf

19. mount procfs
echo "proc /proc procfs rw 0 0" >> /etc/fstab

For Dell 2950:
echo "linprocfs /usr/compat/linux/proc linprocfs rw 0 0" >> /etc/fstab
echo "linsysfs /usr/compat/linux/sys linsysfs rw 0 0" >> /etc/fstab

20. reboot. Confirm new kernel is loaded

uname –a

Check devfs rules:
devfs rule showsets
devfs rule -s 3 show

21. update ports:
cd /usr/ports
echo "*default host=cvsup4.FreeBSD.org\
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_6_2\
*default delete use-rel-suffix\
*default compress\
ports-all tag=." > sup

cvsup sup; mail -s 'cvs sup ports done' dave.boodman@vtext.com < /dev/null

(2450: 18mins, supermicro: 19mins; 2950: 24mins)

22. Install raid mgmt tool
install linux_base:
cd /usr/ports/emulators/linux_base-fc4
make install clean
(2450: 7min, supermicro: 3mins, 2950: 14mins)

(for LSI)
cd /usr/ports/sysutils/linux-megamgr
make install clean

cd /usr/ports/sysutils/megarc
make install clean

(for Perc5/i)
cd /usr/ports/sysutils/linux-megacli
make install clean

Test:
rehash; megacli ldinfo lall a0

(for adaptec)
This didn’t work: reibuild rpmdb cause we had probs installing aacapps
cd /compat/linux/bin
./rpm --initdb
./rpm --rebuilddb

Install of linux-base lead to broken rpm on 6.2 so:
install aacapps-4.1-0.i386.rpm
scp backup2:/mnt/data4/build/freebsd/aacapps-4.1-0.i386.rpm /tmp/.
/compat/linux/bin/rpm -ivh --excludepath=/dev /tmp/aacapps-4.1-0.i386.rpm
scp user@10.1.4.107:/compat/linux/usr/sbin/aaccli /compat/linux/usr/sbin/aaccli

test out;
/compat/linux/usr/sbin/aaccli

rpm didn’t even install on latest so just scp’d over aaccli and it worked

23. install rsync from ports
cd /usr/ports/net/rsync
make install clean

choose default options

24. install perl from ports
Not necessary if linux_base is installed

cd /usr/ports/lang/perl5.8/
make install clean; rehash
(supermicro: 5min)

25. configure inetd to respond to mrtg load queries
echo "load stream tcp nowait user /usr/local/jail/bin/load.pl load.pl" >> /etc/inetd.conf

echo "load 12384/tcp" >> /etc/services

26. install bb client
adduser
Username: bb
Full name: bb
Uid (Leave empty for default): 1984
Login group [bb]:
Login group is bb. Invite bb into other groups? []:
Login class [default]:
Shell (sh csh tcsh nologin) [sh]:
Home directory [/home/bb]:
Use password-based authentication? [yes]:
Use an empty password? (yes/no) [no]:
Use a random password? (yes/no) [no]: yes
Lock out the account after creation? [no]:
Username : bb
Password : <random>
Full Name : bb
Uid : 1984
Class :
Groups : bb
Home : /home/bb
Shell : /bin/sh
Locked : no
OK? (yes/no): yes

cd /usr/home/bb
scp backup2:/mnt/data4/build/bb/bb-freebsd.tar .
tar xvf bb-freebsd.tar

edit /home/bb/bbc1.9e-btf/etc/bb-hosts with something like:
echo "10.1.4.5 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh \
10.1.4.107 jail7.johncompanies.com # ssh" > /home/bb/bbc1.9e-btf/etc/bb-hosts

vi /home/bb/bbc1.9e-btf/ext/openfiles
MACHINE="jail19,johncompanies,com" # HAS TO BE IN A,B,C FORM

cd /usr/home/bb/bbc1.9e-btf/etc
./bbchkcfg.sh
(y to questions)
./bbchkhosts.sh
(ignore ssh errors)
cd ../..
chown -R bb .
su bb
cd
cd bbc1.9e-btf/src
make; make install
cd ..
./runbb.sh start
more BBOUT
(look for errors)
exit

echo 'su - bb -c "cd /home/bb/bbc1.9e-btf; ./runbb.sh start"' > /usr/local/etc/rc.d/bb.sh
chmod +x /usr/local/etc/rc.d/bb.sh

27. configure load mrtg, on mail
vi /usr/local/www/mgmt/mrtg/mrtg1.cfg
(add new entry to file following existing format)

28. configure bb on mail:
vi /usr/home/bb/bbsrc/bb1.9e-btf/etc/bb-hosts
10.1.4.119 jail19.johncompanies.com # ssh

su bb
cd
bbsrc/bb/runbb.sh restart ; exit

29. remove reserve space, enable softupdates (probably already set, so not necessary)
NOT APPLICABLE IF USING GVINUM
cd
umount /mnt/data1
umount /mnt/data2
tunefs -m 0 /mnt/data1
tunefs -m 0 /mnt/data2
mount -a

30. configure ntp
echo "server 10.1.4.5" > /etc/ntp.conf

/usr/sbin/ntpd -p /var/run/ntpd.pid
sleep 2; ntpq -p
(confirm it’s able to reach our time server)

31. mrtg switch graphs

32. fwd and reverse lookups on ns1c
vr johncompanies.com
vi internal.johncompanies.com
rndc reload johncompanies.com IN private
(edit the PTR too)

33. enable noatime option
NOT APPLICABLE IF RUNNING GVINUM
data1 and data2 should look something like:
/dev/amrd0s1g /mnt/data1 ufs rw,noatime 2 2

reboot

34. create gvinum volumes

Make a g partition:

bsdlabel -e /dev/amrd0s1

given:
# /dev/aacd0s1:
8 partitions:
# size offset fstype [fsize bsize bps/cpg]
a: 262144 0 4.2BSD 2048 16384 16392
b: 4194304 262144 swap
c: 143363997 0 unused 0 0 # "raw" part, don't edit
d: 524288 4456448 4.2BSD 2048 16384 32776
e: 524288 4980736 4.2BSD 2048 16384 32776
f: 6291456 5505024 4.2BSD 2048 16384 28552

new offset = 6291456 + 5505024 = 11796480
new size is size for 'c' partition minus the new start from above
143363997 - 11796480 = 131567517
So:
g: 131567517 11796480 unused 0 0

bsdlabel -e /dev/amrd1s1
change d to g

For a 73G drive (after OS), we can fit 31 2G volumes so:

echo 'drive data1 device /dev/aacd0s1g' > /tmp/cgv

or

echo '#\!/bin/sh\
i="1"\
while [ $i -le 31 ]\
do\
echo "volume v$i" >> /tmp/cgv;\
echo 'plex org concat' >> /tmp/cgv;\
echo 'sd length 2g drive data1' >> /tmp/cgv;\
i=`expr $i + 1`\
done' > /tmp/mkcgv
sh /tmp/mkcgv
gvinum create /tmp/cgv

For a 146G drive (-4G for swap), we can fit 66 2G volumes so:

echo 'drive data2 device /dev/amrd1s1g' > /tmp/cgv
sh
for f in 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97;\
do echo "volume v$f" >> /tmp/cgv; echo 'plex org concat' >> /tmp/cgv;\
echo 'sd length 2g drive data2' >> /tmp/cgv; done; exit

gvinum create /tmp/cgv

For 3rd 73G drive (after 2G swap), we can fit 33 2G volumes so:
Label should be:
# /dev/aacd2s1:
8 partitions:
# size offset fstype [fsize bsize bps/cpg]
c: 143299737 0 unused 0 0 # "raw" part, don't edit
g: 143299721 16 unused 0 0

/dev/aacd1s1d
echo 'drive data3 device /dev/aacd2s1g' > /tmp/cgv
sh
for f in 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131;\
do echo "volume v$f" >> /tmp/cgv; echo 'plex org concat' >> /tmp/cgv;\
echo 'sd length 2g drive data3' >> /tmp/cgv; done

gvinum create /tmp/cgv

For a 2nd 73G drive (after 2G swap), we can fit 33 2G volumes so:

echo 'drive data2 device /dev/aacd1s1g' > /tmp/cgv
sh
for f in 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64;\
do echo "volume v$f" >> /tmp/cgv; echo 'plex org concat' >> /tmp/cgv;\
echo 'sd length 2g drive data2' >> /tmp/cgv; done; exit

gvinum create /tmp/cgv

For a 2nd 73G drive (after 4G swap), we can fit 32 2G volumes so:

echo 'drive data2 device /dev/aacd1s1g' > /tmp/cgv
sh
for f in 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63;\
do echo "volume v$f" >> /tmp/cgv; echo 'plex org concat' >> /tmp/cgv;\
echo 'sd length 2g drive data2' >> /tmp/cgv; done; exit

gvinum create /tmp/cgv

For a 3nd 73G drive, we can fit 34 2G volumes so:

echo 'drive data3 device /dev/mfid2s1g' > /tmp/cgv
sh
for f in 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163;\
do echo "volume v$f" >> /tmp/cgv; echo 'plex org concat' >> /tmp/cgv;\
echo 'sd length 2g drive data3' >> /tmp/cgv; done; exit

gvinum create /tmp/cgv

2950:
bsdlabel -e /dev/mfid0s1
bsdlabel -e /dev/mfid1s1

For 1st drive (146G)
echo 'drive data1 device /dev/mfid0s1g' > /tmp/cgv
sh
for f in 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63;\
do echo "volume v$f" >> /tmp/cgv; echo 'plex org concat' >> /tmp/cgv;\
echo 'sd length 2g drive data1' >> /tmp/cgv; done; exit

Usually there’s 2040m leftover so create one more smaller vol:
echo 'volume v64\
plex org concat\
sd length 2040m drive data1' >> /tmp/cgv

gvinum create /tmp/cgv

For 2nd drive (146G)
echo 'drive data2 device /dev/mfid1s1g' > /tmp/cgv
sh
for f in 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130;\
do echo "volume v$f" >> /tmp/cgv; echo 'plex org concat' >> /tmp/cgv;\
echo 'sd length 2g drive data2' >> /tmp/cgv; done; exit
gvinum create /tmp/cgv

35. create the jail template

mkdir /mnt/jail
newfs /dev/gvinum/v1
mount /dev/gvinum/v1 /mnt/jail

cd /usr/src
make clean
rm -fr /usr/obj/
make buildworld
(2450: 2:28mins, supermicro: 53mins, 2950: 36min)
make world DESTDIR=/mnt/jail
(2450: 2:28mins, supermicro: 55mins, 2950: )
cd etc
make distribution DESTDIR=/mnt/jail
mount_devfs devfs /mnt/jail/dev
devfs -m /mnt/jail/dev rule -s 3 applyset
cd /mnt/jail
ln -sf dev/null kernel

jail /mnt/jail testhostname 192.168.11.100 /bin/sh
csh
touch /etc/fstab
echo 'network_interfaces=""\
kern_securelevel_enable="NO"\
sendmail_enable="YES"\
sshd_enable="YES"' > /etc/rc.conf

echo "nameserver 69.55.225.225\
nameserver 69.55.230.3" >> /etc/resolv.conf

vi /etc/crontab
remove the adjkerntz lines
comment out periodic’s and put this line above them:
# DO NOT UNCOMMENT THESE

rm -rf /etc/periodic/daily/400.status-disks

check /tmp for crap

vi /etc/periodic/security/100.chksetuid
replace: MP=`mount -t ufs | grep -v " nosuid" | awk '{ print $3 }' | sort`
with: MP='/' (use single quotes)

mkdir -p /usr/compat/linux/dev

adduser (Add account for user, make sure in wheel group)

user/root passwd: 8ico2987

Set root password

vi /usr/home/user/.profile (and add to the file):
TERM=vt100; export TERM

tzsetup

newaliases

#cd /dev
#rm console
#ln -s null console

vi /etc/syslog.conf (comment out console and move to /var/log/messages):
#*.err;kern.warning;auth.notice;mail.crit /dev/console *.err;kern.warning;auth.notice;mail.crit /var/log/messages

exit
exit

cd /usr/ports
make -DNOCLEANDEPENDS clean
(2450: 15mins , supermicro: 29mins, 2950: 24mins)
rm -fr /usr/ports/distfiles/*
cp -r /usr/ports /mnt/jail/usr (2450: 2:00 mins , supermicro: 15mins, 2950: 3mins)

rm /mnt/jail/root/.history

cd
dump -0a -f /usr/local/jail/template/61template /dev/gvinum/v1

umount /mnt/jail/dev
umount /dev/gvinum/v1

rm -fr /mnt/jail

36. setup backups
echo '#\!/bin/sh\
backupdir=/mnt/data4/jail7\
\
## ENTRY /etc\
## ENTRY /usr/local/etc\
## ENTRY /usr/local/jai7\
## ENTRY /root/logs' > /usr/local/jail/bin/backup.config

on backup2:
setup backup dirs:
mkdir -p /mnt/data2/jail7/0

add the system to
vi /mnt/data4/bin/snapshot_rotate

on mail:
vi /usr/local/www/mgmt/cgi/backupgraph.pl
(add hostname)

Edit /usr/local/jail/bin/backup to use the right drives

37. mkdir /root/logs

38. edit sshd_config for security
vi /etc/ssh/sshd_config
ListenAddress 69.55.226.128
ListenAddress 10.1.4.114

kill -1 `cat /var/run/sshd.pid`

39. add crontab entries
crontab -e
5 0 * * * /usr/local/jail/bin/backup
1 0 1 * * /usr/local/jail/bin/ipfwreset
0 18 * * * /usr/local/jail/bin/ipfwbackup
4,9,14,19,24,29,34,39,44,49,55,59 * * * * /usr/local/jail/bin/trafstats
55 10,23 * * * /usr/local/jail/bin/trafficwatch.pl

On 2950 with Perc5/i also add:
*/5 * * * * /usr/local/jail/bin/perc5iraidchk

40. Reboot notify script
ln -s /usr/local/jail/bin/notify.sh /usr/local/etc/rc.d/notify.sh

41. add to management db (on mail and devweb) jc.ref_machines and jc.ref_templates

uname -r
6.2-RELEASE-jc1

insert into ref_machines values (null,'jail19','jail19.johncompanies.com',0,'l');
select machine_id from ref_machines where host='jail19';
+------------+
| machine_id |
+------------+
| 35 |
+------------+
insert into ref_templates values ('',' 6.2-RELEASE-jc1',15,'FreeBSD 6.2',0);

42. add to server/cabinet map. On mail:
vi /usr/local/www/mgmt/html/cabinetmap.html

43. add an outside blocking rule to the firewall, so this machine can only be reached from inside the firewall. Follow example already in firewall jail17 is:

00117 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 or 69.55.238.150 } to 69.55.228.200
00117 deny ip from any to 69.55.228.200

jail19 would be 00119...
ipfw add 00107 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 or 69.55.238.150 } to 69.55.230.108
ipfw add 00107 deny ip from any to 69.55.220.108

44. select customers for probe map

45. install raid monitor (LSI 320 only):

scp backup2:/d4/build/megaraid/MegaMonitor1.02.tgz /tmp
pkg_add MegaMonitor1.02.tgz

edit line in /usr/local/etc/rc.d/megamonitor.sh to look like:
/usr/sbin/MegaCtrl -start -log /var/log/messages -disChkCon -SMART9999 > /megamonitor.out

comment out:
localhost: /var/log/messages : : : **Monitor** :
In /usr/home/bb/bbc1.9e-btf/etc/bb-msgstab

cd /usr/ports/sysutils/megarc
make install clean

46. make gv start on boot

scp backup2:/mnt/data4/build/freebsd/gvinum /etc/rc.d/gvinum

If they start stale:

echo '#\!/bin/sh\
i="1"\
while [ $i -le 64 ]\
do\
gvinum start v$i\
i=`expr $i + 1`\
done' > /tmp/startgv

sh /tmp/startgv

47. patch jail against starting jails with rtprio

mv /usr/sbin/jail /usr/sbin/jail_
echo '#\!/bin/sh\
/usr/sbin/rtprio -t /usr/sbin/jail_ $*' > /usr/sbin/jail
chmod +x /usr/sbin/jail

48. make sure mail works
If there are map errors:
cd /etc/mail; make maps</pre>

= FreeBSD 6.2 -> 6.3 =

Last updated 2008-08-07

<pre>

1. remove old src
cd /usr
rm –fr src/*

2. get latest sources for this release:

cd /usr/src
echo "*default host=cvsup4.freebsd.org\
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_6_3\
*default delete use-rel-suffix\
*default compress\
src-all" > sup

cvsup sup ; pagedave
(20min)

3. configure new kernel. Get config from similar machine or there may be a master copy somewhere under /mnt/data4/build/freebsd (name the kernel config the same as the jail, ex jail18):
cd /usr/src/sys/i386/conf
scp backup2:/mnt/data4/build/freebsd/kern_config-6.2 ./jail7

or for PAE

scp backup2:/mnt/data4/build/freebsd/kern_config-6.2-PAE ./jail7

edit the kernel config and change ident to be the name of the jail:
vi jail7
ident jail7

edit /sys/conf/newvers.sh to add –jc1 to the end of the BRANCH string (RELEASE-jc1)
vi /sys/conf/newvers.sh

3. bring over patches from backup2

The various patches are in /mnt/data4/build/freebsd/patches on backup2. There are dirs for each version. Not all dirs are populated, but patches for later versions work on older ones unless there is a new patch in the older dir.

cd /usr/src
scp backup2:"/mnt/data4/build/freebsd/patches/6.x/*" .

Apply patches:
patch -l < jls-patch

Apply these only to 2950 with PAE:
patch -p0 < gvinum-staticcompile-patch
patch -p0 < gvinum-bin-patch

4. build, install kernel
cd /usr/src
make buildkernel; pagedave
(2min)

cd /boot
mv kernel.old kernel.old.old
(optional move old kernel out of the way)

cd /usr/src
make installkernel

cd /boot
mv kernel.old kernel.6.2

5. take ½ of mirror and test (boot up) in new hardware

6. improved loader configs:

/boot/loader.conf:
console="comconsole,vidconsole"
boot_serial="YES"
boot_multicons="YES"

7. build userland
cd /usr/src
make buildworld; pagedave
(33mins)
make installworld

mergemaster –i
reboot
8. patch jail against starting jails with rtprio

mv /usr/sbin/jail /usr/sbin/jail_
echo '#\!/bin/sh\
/usr/sbin/rtprio -t /usr/sbin/jail_ $*' > /usr/sbin/jail
chmod +x /usr/sbin/jail
</pre>

= FreeBSD 7.0 =

Last updated 2008-04-30

<pre>

All time extimates below assume disks aren’t scrubbing. Setup instructions below are for LSI card:

1. make sure bios is setup for bios console redirect
Supermicro:
Console redirection:
Com port addr: on-board COM A
Baud: 38400
Console type: vt100
Flow control: none
Console connection: direct
Continue cr after post: on

2450:
Make sure running bios A09
Console Redirection: VT100/VT220

2. assuming mirrors (or at least disks) created (if not, refer to this), boot to disk 1 of 5.4
skip kernel config (enter)
custom install

partition ->
move cursor to amrd0, hit space (takes you to partition map screen)
a for entire disk
q to quit and save
standard mbr (no boot manager)
space to unselect amrd0
cursor over amrd1
space
a for entire disk
q to quit and save
none (leave untouched)

cursor over amrd0
space
(takes you into part. Screen again) q to exit
none
Make sure both are checked and tab to ok

Label ->
Make sure mfid0 is highlighted

/ 256M
swap 2G (for 2950 make it 4G)
/var 256M (4.6G)
/tmp 256M
/usr 4G
/mnt/data1 remaining space (no need to newfs)

Make sure to toggle S for soft updates on all (should look like UFS2+S Y under the Newfs column)

move cursor to mfid1
swap 8G
/mnt/data2 remaining space (no need to newfs)

q to save and exit

distributions ->
developer
yes to install ports
exit

media ->
cd

commit ->
yes
(2450: 14mins, supermicro: 12mins)

yes to "visit general config" ->
Set root pwd

Add user ‘user’ member group is wheel, set password

Set tz

Networking->interfaces->Fxp0 ->
No IPV6
dhcp=no
Set hostname & domain
Enable sshd

exit...
exit install ->
yes

take the cd out and let the machine reboot

3. double check the date/time

4. edit /etc/make.conf (only add the console speed line if this is a supermicro capable of outputting BIOS redirect at that speed – 2450’s can only do 9600)
echo "WITHOUT_X11=yes \
KERNCONF=jail2 \
BOOT_COMCONSOLE_SPEED=115200" >> /etc/make.conf

5. add settings to /boot/loader.conf and /boot.config

echo "-Dh" >> /boot.config

echo 'console="comconsole,vidconsole" \
boot_multicons="YES" \
boot_serial="YES" \
mfi_linux_load="YES" \
comconsole_speed="115200"' >> /boot/loader.conf

(leave out the speed and mfi lines for 2450s)

6. turn off all ttyv's except 0 and 1 in /etc/ttys
also turn on ttyd0, change type to vt100:
vi /etc/ttys
ttyd0 "/usr/libexec/getty std.9600" vt100 on secure

kill -1 1

on console server:
vi /etc/remote
(rename port to jail18 depending on where and which digi plugged into)
test serial console

7. populate hosts
echo "10.1.4.3 backup2" >> /etc/hosts
echo "10.1.4.8 backup1" >> /etc/hosts

8. put key in authorized_keys on backup2
cd
ssh-keygen -t dsa -b 1024
(default location, leave password blank)

cat /root/.ssh/id_dsa.pub | ssh backup2 'cat - >> /root/.ssh/authorized_keys'
cat /root/.ssh/id_dsa.pub | ssh backup1 'cat - >> /root/.ssh/authorized_keys'

confirm that you can ssh to backup2 and backup1 without getting a login prompt

ssh backup2 hostname

ssh backup1 hostname

9. create & populate binaries/scripts dirs
mkdir -p /usr/local/jail/bin
mkdir -p /usr/local/jail/rc.d
mkdir -p /usr/local/jail/template/
mkdir /mnt/data1
mkdir /mnt/data2
scp backup2:"/mnt/data4/bin/freebsd7.x/*" /usr/local/jail/bin
cd /usr/local/jail/rc.d/
touch quad1
touch quad2
touch quad3
touch quad4
touch safe1
touch safe2
touch safe3
touch safe4
chmod +x *
cd /usr/local/jail/bin
ln -s /usr/local/jail/rc.d/quad1 quad1
ln -s /usr/local/jail/rc.d/quad2 quad2
ln -s /usr/local/jail/rc.d/quad3 quad3
ln -s /usr/local/jail/rc.d/quad4 quad4
ln -s /usr/local/jail/rc.d/safe1 safe1
ln -s /usr/local/jail/rc.d/safe2 safe2
ln -s /usr/local/jail/rc.d/safe3 safe3
ln -s /usr/local/jail/rc.d/safe4 safe4
ln -s /usr/local/jail/bin/jailmake_zfs jailmake
--OR--
ln -s /usr/local/jail/bin/jailmake_geom jailmake

ln -s /usr/local/jail/bin/js_zfs js
--OR--
ln -s /usr/local/jail/bin/js_geom js

rehash

10. edit root's path and login script:
vi /root/.cshrc
add to path:
/usr/local/jail/bin
(if adaptec card installed, also add /compat/linux/usr/sbin)

Change alias entries (add G):
alias la ls -aG
alias lf ls -FAG
alias ll ls -lAG
alias ls ls -AG
alias mbm mb mount
alias mbu mb umount

and alter the prompt, set the following:
set prompt = "`/bin/hostname -s` %/# "

at the bottom of the file add:
set sshtty=`who am i|awk '{print $2}'`
/usr/sbin/rtprio 3 -`psj | grep $sshtty | awk '{print $2}'`

set shortty=`who am i | awk '{print $2}' | sed -E 's/.*(..)$/\1/'`
foreach x (`psj | grep sh | grep $shortty | awk '{print $2}'`)
/usr/sbin/rtprio 2 -$x
end

To load the new file:
source /root/.cshrc

11. install cvsup
cd /usr/ports/net/cvsup-without-gui
make install clean; rehash; mail -s 'cvs installed' dave.boodman@vtext.com < /dev/null

(stay close for gettext options, 2450: 27mins, supermicro: 17mins, 2950: 22mins)

12. get latest sources for this release:
cd /usr/src
echo "*default host=cvsup4.freebsd.org\
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_7\
*default delete use-rel-suffix\
*default compress\
src-all" > sup

-OR-

echo "*default host=cvsup4.freebsd.org\
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_7_1\
*default delete use-rel-suffix\
*default compress\
src-all" > sup

cvsup sup ; mail -s 'cvs sup done' dave.boodman@vtext.com < /dev/null

(2450, ~12mins, supermicro, 27mins, 2950: 7mins)

13. configure new kernel.
for i386:
cd /usr/src/sys/i386/conf
scp backup2:/mnt/data4/build/freebsd/kern_config-7.0-PAE ./mx2

-or-
for amd64:
cd /usr/src/sys/amd64/conf
scp backup2:/mnt/data4/build/freebsd/kern_config-7.0-amd64 ./jail2

------

edit the kernel config and change ident to be the name of the jail:
vi jail2
ident jail2

edit /sys/conf/newvers.sh to add –jc2 to the end of the BRANCH string (RELEASE-jc2)
vi /sys/conf/newvers.sh

notes: http://www.FreeBSD.org/doc/en_US.ISO8859-1/books/handbook/kernelconfig-config.html

14. bring over patches from backup2

The various patches are in /mnt/data4/build/freebsd/patches on backup2. There are dirs for each version. Not all dirs are populated, but patches for later versions work on older ones unless there is a new patch in the older dir.

cd /usr/src
scp backup2:"/mnt/data4/build/freebsd/patches/7.x/*" .

Apply patches:
patch -l < jls-patch

15. build, install kernel and world
cd /usr/src
make buildworld buildkernel installkernel; mail -s 'kernel build done' dave.boodman@vtext.com < /dev/null
(2450: 1:56min, supermicro: 59mins, 2950: 38mins)
make installworld
(2450: 3min, supermicro: 1min, 2950: :34)
mergemaster -i
delete /var/tmp/temproot
delete bsnmpd
delete temporary ./etc/hosts
delete temporary ./etc/motd
delete /var/tmp/temproot

cd /usr/src/sys/modules/zfs
make
make install

16. populate devfs ruleset
scp backup2:/mnt/data4/build/freebsd/devfs.rules /etc

17. populate /etc/rc.conf with IPs and NFS settings
vi /etc/rc.conf

kern_securelevel_enable="NO"
portmap_enable="NO"
sendmail_enable="NO"
usbd_enable="YES"

xntpd_enable="YES"
nfs_client_enable="YES"
nfs_reserved_port_only="YES"
sshd_enable="YES"
inetd_enable="YES"
inetd_flags="-wW -a 10.1.4.102"
devfs_system_ruleset="devfsrules_show_all"

hostname="jail2.johncompanies.com"
ifconfig_xl0="inet 10.1.4.102 netmask 255.255.255.0"
ifconfig_fxp0="inet 69.55.228.53 netmask 255.255.255.0"
defaultrouter="69.55.228.1"
#ifconfig_fxp0_alias0="inet 69.55.2xx.xx netmask 255.255.255.0"

static_routes="t1 office"
route_t1="-net 10.1.5 10.1.4.2"
route_office="-net 10.1.6 10.1.4.2"
zfs_enable="YES"

18. make sure sysctls are set and preserved after reboot
echo "kern.consmute=0\
kern.ipc.shm_use_phys=1\
kern.ipc.shmall=65535\
kern.ipc.shmmax=134217728\
net.inet.tcp.syncookies=0\
kern.maxfiles=32768\
kern.fallback_elf_brand=3\
kern.maxprocperuid=4000\
security.jail.sysvipc_allowed=1\
security.jail.allow_raw_sockets=1\
security.jail.socket_unixiproute_only=1\
security.jail.chflags_allowed=0\
compat.linux.osrelease=2.6.12\
kern.maxvnodes=400000" >> /etc/sysctl.conf

19. mount procfs
echo "proc /proc procfs rw 0 0" >> /etc/fstab

For Dell 2950/2450:
echo "linprocfs /usr/compat/linux/proc linprocfs rw 0 0" >> /etc/fstab

For Dell 2950:
echo "linsysfs /usr/compat/linux/sys linsysfs rw 0 0" >> /etc/fstab

20. reboot. Confirm new kernel is loaded

uname –a

Check devfs rules:
devfs rule showsets
devfs rule -s 3 show

21. update ports:
cd /usr/ports
echo "*default host=cvsup4.FreeBSD.org\
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_7_0\
*default delete use-rel-suffix\
*default compress\
ports-all tag=." > sup

cvsup sup; mail -s 'cvs sup ports done' dave.boodman@vtext.com < /dev/null

(2450: 18mins, supermicro: 19mins; 2950: 24mins)

22. Install raid mgmt tool

For LSI based cards:

install linux_base:
cd /usr/ports/emulators/linux_base-fc4
make install clean
(2450: 7min, supermicro: 3mins, 2950: 14mins)

(for LSI)
cd /usr/ports/sysutils/linux-megamgr
make install clean

cd /usr/ports/sysutils/megarc
make install clean

(for Perc5/i)
cd /usr/ports/sysutils/linux-megacli
make install clean

Test:
rehash; megacli ldinfo lall a0

For adaptec:
On jail18:
scp /compat/linux/usr/sbin/aaccli user@10.1.4.102:~

mv ~user/aaccli /compat/linux/usr/sbin/aaccli

test out;
/compat/linux/usr/sbin/aaccli

For 3w9690:
scp backup2:/d4/build/3ware/tw_cli-freebsd-x86_64-9.5.0.1.tgz /usr/local/sbin
cd /usr/local/sbin xzvf tw_cli-freebsd-x86_64-9.5.0.1.tgz
rm tw_cli-freebsd-x86_64-9.5.0.1.tgz

test out;
./tw_cli /c0 show allstatus

23. install rsync from ports
cd /usr/ports/net/rsync
make install clean

choose default options

24. install perl from ports
Not necessary if linux_base is installed

cd /usr/ports/lang/perl5.8/
make install clean; rehash
(supermicro: 5min)

25. configure inetd to respond to mrtg load queries
echo "load stream tcp nowait user /usr/local/jail/bin/load.pl load.pl" >> /etc/inetd.conf

echo "load 12384/tcp" >> /etc/services

26. install bb client
adduser
Username: bb
Full name: bb
Uid (Leave empty for default): 1984
Login group [bb]:
Login group is bb. Invite bb into other groups? []:
Login class [default]:
Shell (sh csh tcsh nologin) [sh]:
Home directory [/home/bb]:
Use password-based authentication? [yes]:
Use an empty password? (yes/no) [no]:
Use a random password? (yes/no) [no]: yes
Lock out the account after creation? [no]:
Username : bb
Password : <random>
Full Name : bb
Uid : 1984
Class :
Groups : bb
Home : /home/bb
Shell : /bin/sh
Locked : no
OK? (yes/no): yes

cd /usr/home/bb
scp backup2:/mnt/data4/build/bb/bb-freebsd.tar .
tar xf bb-freebsd.tar

edit /home/bb/bbc1.9e-btf/etc/bb-hosts with something like:
echo "10.1.4.5 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh \
10.1.4.108 jail2.johncompanies.com # ssh" > /home/bb/bbc1.9e-btf/etc/bb-hosts

vi /home/bb/bbc1.9e-btf/ext/openfiles
MACHINE="jail2,johncompanies,com" # HAS TO BE IN A,B,C FORM

cd /usr/home/bb/bbc1.9e-btf/etc
./bbchkcfg.sh
(y to questions)
./bbchkhosts.sh
(ignore ssh errors)
cd ../..
chown -R bb .
su bb
cd
cd bbc1.9e-btf/src
make; make install
cd ..
./runbb.sh start
more BBOUT
(look for errors)
exit

echo 'su - bb -c "cd /home/bb/bbc1.9e-btf; ./runbb.sh start"' > /usr/local/etc/rc.d/bb.sh
chmod +x /usr/local/etc/rc.d/bb.sh

NOTE: to get bb working on amd, had to copy over bin dir from linux dist

27. configure load mrtg, on mail
vi /usr/local/www/mgmt/mrtg/mrtg1.cfg
(add new entry to file following existing format)

28. configure bb on mail:
vi /usr/home/bb/bbsrc/bb1.9e-btf/etc/bb-hosts
10.1.4.108 jail2.johncompanies.com # ssh

su bb
cd
bbsrc/bb/runbb.sh restart ; exit

29. remove reserve space, enable softupdates (probably already set, so not necessary)
NOT APPLICABLE IF USING GVINUM
cd
umount /mnt/data1
umount /mnt/data2
tunefs -m 0 /mnt/data1
tunefs -m 0 /mnt/data2
mount -a

30. configure ntp
echo "server 10.1.4.5" > /etc/ntp.conf

/usr/sbin/ntpd -p /var/run/ntpd.pid
sleep 2; ntpq -p
(confirm it’s able to reach our time server)

31. mrtg switch graphs

32. fwd and reverse lookups on ns1c
vr johncompanies.com
vi internal.johncompanies.com
rndc reload johncompanies.com IN private
(edit the PTR too)

33. create zpools

Make a g partition:

bsdlabel -e /dev/mfid0s1

given:
# /dev/aacd0s1:
8 partitions:
# size offset fstype [fsize bsize bps/cpg]
a: 262144 0 4.2BSD 2048 16384 16392
b: 4194304 262144 swap
c: 143363997 0 unused 0 0 # "raw" part, don't edit
d: 524288 4456448 4.2BSD 2048 16384 32776
e: 524288 4980736 4.2BSD 2048 16384 32776
f: 6291456 5505024 4.2BSD 2048 16384 28552

new offset = 6291456 + 5505024 = 11796480
new size is size for 'c' partition minus the new start from above
143363997 - 11796480 = 131567517
So:
g: 131567517 11796480 unused 0 0

bsdlabel -e /dev/mfid1s1
change d to g

zpool create pool1 mfid0s1g
zpool create pool2 mfid1s1g
zfs set atime=off pool1
zfs set atime=off pool2

echo ' vfs.zfs.prefetch_disable="1" \
vfs.zfs.arc_min=16777216 \
vfs.zfs.arc_max=33554432 \
vm.kmem_size_max="1G" # for i386\
vm.kmem_size="1G" # for i386\
vm.kmem_size_max="1.5G" # for amd64\
vm.kmem_size="1.5G" # for amd64\
vfs.zfs.zil_disable="1" ' >> /boot/loader.conf

35. create the jail template

zfs create -o mountpoint=/mnt/data1/jail -o quota=4G pool1/jail

cd /usr/ports/sysutils/jailutils
make install clean

cd /usr/src
make world DESTDIR=/mnt/data1/jail
(2450: 2:28mins, supermicro: 55mins, 2950: 1h )
cd etc
make distribution DESTDIR=/mnt/data1/jail
mount -t devfs devfs /mnt/data1/jail/dev
devfs -m /mnt/data1/jail/dev rule -s 3 applyset
cd /mnt/data1/jail
ln -sf dev/null kernel
cp /usr/local/sbin/jkill /mnt/data1/jail/sbin

jail /mnt/data1/jail testhostname 192.168.11.100 /bin/sh
csh
touch /etc/fstab
echo 'network_interfaces=""\
hostname="newsystem"\
kern_securelevel_enable="NO"\
sendmail_enable="YES"\
sshd_enable="YES"' > /etc/rc.conf

echo "nameserver 69.55.225.225\
nameserver 69.55.230.3" >> /etc/resolv.conf

vi /etc/crontab
remove the adjkerntz lines
comment out periodic’s and put this line above them:
# IF YOU UNCOMMENT THESE, PLEASE ADJUST THEIR RUN TIME

rm -rf /etc/periodic/daily/400.status-disks

check /tmp for crap

vi /etc/periodic/security/100.chksetuid
replace: MP=`mount -t ufs | grep -v " nosuid" | awk '{ print $3 }' | sort`
with: MP='/' (use single quotes)

mkdir -p /usr/compat/linux/dev

adduser (Add account for user, make sure in wheel group)

user/root passwd: 8ico2987

Set root password

vi /usr/home/user/.profile (and add to the file):
TERM=vt100; export TERM

tzsetup

newaliases

rm /sbin/halt /sbin/reboot
ln /sbin/jkill /sbin/halt
ln /sbin/jkill /sbin/reboot

#cd /dev
#rm console
#ln -s null console

vi /etc/syslog.conf (comment out console and move to /var/log/messages):
#*.err;kern.warning;auth.notice;mail.crit /dev/console *.err;kern.warning;auth.notice;mail.crit /var/log/messages

cd /libexec
mv ld-elf32.so.1 ld-elf32.so.1-orig
ln ld-elf.so.1 ld-elf32.so.1

exit
exit

cd /usr/ports
make -DNOCLEANDEPENDS clean
(2450: 15mins , supermicro: 29mins, 2950: 24mins)
rm -fr /usr/ports/distfiles/*
cp -r /usr/ports /mnt/data1/jail/usr (2450: 2:00 mins , supermicro: 15mins, 2950: 3mins)

rm /mnt/data1/jail/root/.history

cd
zfs snapshot pool1/jail@now
zfs send pool1/jail@now | cat > /usr/local/jail/template/70template
zfs destroy pool1/jail@now

zfs destroy pool1/jail

rmdir /mnt/data1/jail
f

36. setup backups
echo '#\!/bin/sh\
backupdir=/data/jail2\
server=backup1\
\
## ENTRY /etc\
## ENTRY /usr/local/etc\
## ENTRY /usr/local/jail\
## ENTRY /root/logs' > /usr/local/jail/bin/backup.config

on backup1:
setup backup dirs:
mkdir -p /data/jail2/0

add the system to
vi /usr/local/sbin/snapshot_rotate

on mail:
vi /usr/local/www/mgmt/cgi/backupgraph.pl
(add hostname)

Edit /usr/local/jail/bin/backup to use the right drives

37. mkdir /root/logs

38. edit sshd_config for security
vi /etc/ssh/sshd_config
ListenAddress 69.55.238.233
ListenAddress 10.1.4.108

kill -1 `cat /var/run/sshd.pid`

39. add crontab entries
crontab -e
5 0 * * * /usr/local/jail/bin/backup.zfs
1 0 1 * * /usr/local/jail/bin/ipfwreset
0 18 * * * /usr/local/jail/bin/ipfwbackup
4,9,14,19,24,29,34,39,44,49,55,59 * * * * /usr/local/jail/bin/trafstats

On 2950 with Perc5/i also add:
*/5 * * * * /usr/local/jail/bin/perc5iraidchk

40. Reboot notify script
ln -s /usr/local/jail/bin/notify.sh /usr/local/etc/rc.d/notify.sh

41. add to management db (on mail and devweb) jc.ref_machines and jc.ref_templates

uname -r
6.2-RELEASE-jc1

insert into ref_machines values (null,'mx2','mx2.johncompanies.com',0,'m');
select machine_id from ref_machines where host='mx2';
+------------+
| machine_id |
+------------+
| 35 |
+------------+
insert into ref_templates values ('',' 7.1-RELEASE-jc2',44,'FreeBSD 7.1',0);

42. add to server/cabinet map. On mail:
vi /usr/local/www/mgmt/html/cabinetmap.html

43. add an outside blocking rule to the firewall, so this machine can only be reached from inside the firewall. Follow example already in firewall jail17 is:

00117 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 or 69.55.238.150 } to 69.55.228.200
00117 deny ip from any to 69.55.228.200

jail19 would be 00119...
ipfw add 00107 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 or 69.55.238.150 } to 69.55.230.108
ipfw add 00107 deny ip from any to 69.55.220.108

44. select customers for probe map

47. patch jail against starting jails with rtprio

mv /usr/sbin/jail /usr/sbin/jail_
echo '#\!/bin/sh\
/usr/sbin/rtprio -t /usr/sbin/jail_ $*' > /usr/sbin/jail
chmod +x /usr/sbin/jail

48. make sure mail works
If there are map errors:
cd /etc/mail; make maps

49. move and pare down generic kernel
mv kernel.old/ kernel.GENERIC

/ too full, remove old kernel modules except for:

Id Refs Address Size Name
1 14 0xffffffff80100000 ac6c08 kernel
2 1 0xffffffff80bc8000 1128 mfi_linux.ko
3 4 0xffffffff80bca000 39138 linux.ko
4 1 0xffffffffb48f8000 358c linprocfs.ko
5 1 0xffffffffb48fc000 9d3 linsysfs.ko
6 1 0xffffffffb490c000 80ee8 zfs.ko

50. ntp doesn’t seem to start from rc so,

echo '/usr/sbin/ntpd -c /etc/ntp.conf -p /var/run/ntpd.pid -f /var/db/ntpd.drift' > /usr/local/etc/rc.d/ntp.sh
chmod 0700 /usr/local/etc/rc.d/ntp.sh
</pre>

= FreeBSD 7.1 =

Last updated 2009-02-16

<pre>All time extimates below assume disks aren’t scrubbing. Setup instructions below are for LSI card:

1. make sure bios is setup for bios console redirect
Supermicro:
Console redirection:
Com port addr: on-board COM A
Baud: 38400
Console type: vt100
Flow control: none
Console connection: direct
Continue cr after post: on

2450:
Make sure running bios A09
Console Redirection: VT100/VT220

2. assuming mirrors (or at least disks) created (if not, refer to this), boot to disk 1 of 5.4
skip kernel config (enter)
custom install

partition ->
move cursor to amrd0, hit space (takes you to partition map screen)
a for entire disk
q to quit and save

NOTE
For gvin probs made 2 slices, 1st 8960M
Freebsd boot mgr

standard mbr (no boot manager)
space to unselect amrd0
cursor over amrd1
space
a for entire disk
q to quit and save
none (leave untouched)

cursor over amrd0
space
(takes you into part. Screen again) q to exit
none
Make sure both are checked and tab to ok

Label ->
Make sure mfid0 is highlighted

/ 256M
swap 2G (for 2950 make it 4G)
/var 256M (4.6G)
/tmp 256M
/usr 4G
/mnt/data1 remaining space (no need to newfs)

Make sure to toggle S for soft updates on all (should look like UFS2+S Y under the Newfs column)

move cursor to mfid1
swap 8G
/mnt/data2 remaining space (no need to newfs)

q to save and exit

distributions ->
developer
yes to install ports
exit

media ->
cd

commit ->
yes
(2450: 14mins, supermicro: 12mins)

yes to "visit general config" ->
Set root pwd

Add user ‘user’ member group is wheel, set password

Set tz

Networking->interfaces->Fxp0 ->
No IPV6
dhcp=no
Set hostname & domain
Enable sshd

exit...
exit install ->
yes

take the cd out and let the machine reboot

3. double check the date/time

4. edit /etc/make.conf (only add the console speed line if this is a supermicro capable of outputting BIOS redirect at that speed – 2450’s can only do 9600)
echo "WITHOUT_X11=yes \
KERNCONF=jail2 \
BOOT_COMCONSOLE_SPEED=115200" >> /etc/make.conf

5. add settings to /boot/loader.conf and /boot.config

echo "-Dh" >> /boot.config

echo 'console="comconsole,vidconsole" \
boot_multicons="YES" \
boot_serial="YES" \
mfi_linux_load="YES" \
comconsole_speed="115200"' >> /boot/loader.conf

(leave out the speed and mfi lines for 2450s)

6. turn off all ttyv's except 0 and 1 in /etc/ttys
also turn on ttyd0, change type to vt100:
vi /etc/ttys

ttyv2 "/usr/libexec/getty Pc" cons25 off secure
ttyv3 "/usr/libexec/getty Pc" cons25 off secure
ttyv4 "/usr/libexec/getty Pc" cons25 off secure
ttyv5 "/usr/libexec/getty Pc" cons25 off secure
ttyv6 "/usr/libexec/getty Pc" cons25 off secure
ttyv7 "/usr/libexec/getty Pc" cons25 off secure

ttyd0 "/usr/libexec/getty std.9600" vt100 on secure

kill -1 1

on console server:
vi /etc/remote
(rename port to jail18 depending on where and which digi plugged into)
test serial console

7. populate hosts
echo "10.1.4.3 backup2" >> /etc/hosts
echo "10.1.4.8 backup1" >> /etc/hosts

8. put key in authorized_keys on backup2
cd
ssh-keygen -t dsa -b 1024
(default location, leave password blank)

cat /root/.ssh/id_dsa.pub | ssh backup2 'cat - >> /root/.ssh/authorized_keys'
cat /root/.ssh/id_dsa.pub | ssh backup1 'cat - >> /root/.ssh/authorized_keys'

confirm that you can ssh to backup2 and backup1 without getting a login prompt

ssh backup2 hostname

ssh backup1 hostname

9. create & populate binaries/scripts dirs
mkdir -p /usr/local/jail/bin
mkdir -p /usr/local/jail/rc.d
mkdir -p /usr/local/jail/template/
mkdir /mnt/data1
mkdir /mnt/data2
scp backup2:"/mnt/data4/bin/freebsd7.x/*" /usr/local/jail/bin
cd /usr/local/jail/rc.d/
touch quad1
touch quad2
touch quad3
touch quad4
touch safe1
touch safe2
touch safe3
touch safe4
chmod +x *
cd /usr/local/jail/bin
ln -s /usr/local/jail/rc.d/quad1 quad1
ln -s /usr/local/jail/rc.d/quad2 quad2
ln -s /usr/local/jail/rc.d/quad3 quad3
ln -s /usr/local/jail/rc.d/quad4 quad4
ln -s /usr/local/jail/rc.d/safe1 safe1
ln -s /usr/local/jail/rc.d/safe2 safe2
ln -s /usr/local/jail/rc.d/safe3 safe3
ln -s /usr/local/jail/rc.d/safe4 safe4
ln -s /usr/local/jail/bin/jailmake_geom jailmake
ln -s /usr/local/jail/bin/js_geom js
ln -s /usr/local/jail/bin/canceljail_geom canceljail
ln -s /usr/local/jail/bin/jailmakeempty_geom jailmakeempty

ln -s /usr/local/jail/bin/jailmake_md jailmake
ln -s /usr/local/jail/bin/js_md js
ln -s /usr/local/jail/bin/canceljail_md canceljail
ln -s /usr/local/jail/bin/jailmakeempty_md jailmakeempty
ln -s /usr/local/jail/bin/postboot_md postboot

rehash

10. edit root's path and login script:
vi /root/.cshrc

Change alias entries (add G):
alias la ls -aG
alias lf ls -FAG
alias ll ls -lAG
alias ls ls -AG
alias mbm mb mount
alias mbu mb umount

add to path:
/usr/local/jail/bin
(if adaptec card installed, also add /compat/linux/usr/sbin)

and alter the prompt, set the following:
set prompt = "`/bin/hostname -s` %/# "

at the bottom of the file add:
set sshtty=`who am i|awk '{print $2}'`
/usr/sbin/rtprio 3 -`psj | grep $sshtty | awk '{print $2}'`

set shortty=`who am i | awk '{print $2}' | sed -E 's/.*(..)$/\1/'`
foreach x (`psj | grep sh | grep $shortty | awk '{print $2}'`)
/usr/sbin/rtprio 2 -$x
end

To load the new file:
source /root/.cshrc

11. install cvsup
cd /usr/ports/net/cvsup-without-gui
make install clean; rehash; mail -s 'cvs installed' dave.boodman@vtext.com < /dev/null

(stay close for gettext options, 2450: 27mins, supermicro: 17mins, 2950: 22mins)

12. get latest sources for this release:
cd /usr/src
echo "*default host=cvsup4.freebsd.org\
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_7_1\
*default delete use-rel-suffix\
*default compress\
src-all" > sup

-OR-

echo "*default host=cvsup4.freebsd.org\
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_7\
*default delete use-rel-suffix\
*default compress\
src-all" > sup

(stable)

cvsup sup ; mail -s 'cvs sup done' dave.boodman@vtext.com < /dev/null

(2450, ~12mins, supermicro, 27mins, 2950: 7mins)

13. configure new kernel.
for i386:
cd /usr/src/sys/i386/conf
scp backup2:/mnt/data4/build/freebsd/kern_config-7.0-PAE ./mx2

-or-

for amd64:
cd /usr/src/sys/amd64/conf
scp backup2:/mnt/data4/build/freebsd/kern_config-7.1-amd64 ./jail2

-------

edit the kernel config and change ident to be the name of the jail:
vi jail2
ident jail2

edit /sys/conf/newvers.sh to add –jc2 to the end of the BRANCH string (RELEASE-jc2)
vi /sys/conf/newvers.sh

notes: http://www.FreeBSD.org/doc/en_US.ISO8859-1/books/handbook/kernelconfig-config.html

14. bring over patches from backup2

The various patches are in /mnt/data4/build/freebsd/patches on backup2. There are dirs for each version. Not all dirs are populated, but patches for later versions work on older ones unless there is a new patch in the older dir.

cd /usr/src
scp backup2:"/mnt/data4/build/freebsd/patches/7.x/*" .

Apply patches:
patch -l < jls-patch

15. build, install kernel and world

cd /boot

mv kernel kernel.GENERIC
cd kernel.GENERIC
mkdir hold
mv mfi_linux.ko hold/
mv linux.ko hold/
mv linprocfs.ko hold/
mv linsysfs.ko hold/
mv geom_vinum.ko hold/
mv geom_concat.ko hold/
rm *.ko
rm *.symbols
mv hold/* .
rmdir hold/

cd /usr/src
make buildkernel installkernel

make buildworld ; mail -s 'kernel build done' dave.boodman@vtext.com < /dev/null
(2450: 1:56min, supermicro: 59mins, 2950: 38mins)
make installworld
(2450: 3min, supermicro: 1min, 2950: :34)
mergemaster -i
delete /var/tmp/temproot
delete bsnmpd
delete temporary ./etc/hosts
delete temporary ./etc/motd
delete /var/tmp/temproot

cd /sys/modules/geom/geom_vinum
make
make install

16. populate devfs ruleset
scp backup2:/mnt/data4/build/freebsd/devfs.rules /etc

17. populate /etc/rc.conf with IPs and NFS settings
vi /etc/rc.conf

kern_securelevel_enable="NO"
portmap_enable="NO"
sendmail_enable="NO"
usbd_enable="YES"

xntpd_enable="YES"
nfs_client_enable="YES"
nfs_reserved_port_only="YES"
inetd_enable="YES"
inetd_flags="-wW -a 10.1.4.102"
devfs_system_ruleset="devfsrules_show_all"

#ifconfig_xl0="inet 10.1.4.102 netmask 255.255.255.0"
#ifconfig_fxp0="inet 69.55.228.53 netmask 255.255.255.0"
#ifconfig_fxp0_alias0="inet 69.55.2xx.xx netmask 255.255.255.0"

static_routes="t1 office"
route_t1="-net 10.1.5 10.1.4.2"
route_office="-net 10.1.6 10.1.4.2"
gvinum_enable="YES"
fsck_y_enable="YES"
background_fsck="NO"
rc_mfi_raid_tty_log="YES"

18. make sure sysctls are set and preserved after reboot
echo "kern.consmute=0\
kern.ipc.shm_use_phys=1\
kern.ipc.shmall=131070\
kern.ipc.shmmax=134217728\
net.inet.tcp.syncookies=0\
kern.maxfiles=32768\
kern.fallback_elf_brand=3\
kern.maxprocperuid=4000\
security.jail.sysvipc_allowed=1\
security.jail.allow_raw_sockets=1\
security.jail.socket_unixiproute_only=1\
security.jail.chflags_allowed=0\
dev.amr.0.allow_volume_configure=1\
compat.linux.osrelease=2.6.12" >> /etc/sysctl.conf

19. mount procfs
echo "proc /proc procfs rw 0 0" >> /etc/fstab

For Dell 2950/2450:
echo "linprocfs /usr/compat/linux/proc linprocfs rw 0 0" >> /etc/fstab

For Dell 2950:
echo "linsysfs /usr/compat/linux/sys linsysfs rw 0 0" >> /etc/fstab

mkdir -p /usr/compat/linux/proc
mkdir -p /usr/compat/linux/sys

20. reboot. Confirm new kernel is loaded

uname -a

Check devfs rules:
devfs rule showsets
devfs rule -s 3 show

21. update ports:
cd /usr/ports
echo "*default host=cvsup4.FreeBSD.org\
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_7_1\
*default delete use-rel-suffix\
*default compress\
ports-all tag=." > sup

cvsup sup; mail -s 'cvs sup ports done' dave.boodman@vtext.com < /dev/null

(2450: 18mins, supermicro: 19mins; 2950: 24mins)

22. Install raid mgmt tool

For LSI based cards:

install linux_base:
cd /usr/ports/emulators/linux_base-fc4
make install clean
(2450: 7min, supermicro: 3mins, 2950: 14mins)

(for LSI)
cd /usr/ports/sysutils/linux-megamgr
make install clean

cd /usr/ports/sysutils/megarc
make install clean

(for Perc5/i, 6/i)
cd /usr/ports/sysutils/linux-megacli2
make install clean

Test:
rehash; megacli ldinfo lall a0

For adaptec:
On jail18:
scp /compat/linux/usr/sbin/aaccli user@10.1.4.102:~

mv ~user/aaccli /compat/linux/usr/sbin/aaccli

test out;
/compat/linux/usr/sbin/aaccli

For 3w9690:
scp backup2:/d4/build/3ware/tw_cli-freebsd-x86_64-9.5.0.1.tgz /usr/local/sbin
cd /usr/local/sbin xzvf tw_cli-freebsd-x86_64-9.5.0.1.tgz
rm tw_cli-freebsd-x86_64-9.5.0.1.tgz

test out;
./tw_cli /c0 show allstatus

23. install rsync from ports
cd /usr/ports/net/rsync
make install clean

choose default options

25. configure inetd to respond to mrtg load queries
echo "load stream tcp nowait user /usr/local/jail/bin/load.pl load.pl" >> /etc/inetd.conf

echo "load 12384/tcp" >> /etc/services

26. install bb client
adduser
Username: bb
Full name: bb
Uid (Leave empty for default): 1984
Login group [bb]:
Login group is bb. Invite bb into other groups? []:
Login class [default]:
Shell (sh csh tcsh nologin) [sh]:
Home directory [/home/bb]:
Use password-based authentication? [yes]:
Use an empty password? (yes/no) [no]:
Use a random password? (yes/no) [no]: yes
Lock out the account after creation? [no]:
Username : bb
Password : <random>
Full Name : bb
Uid : 1984
Class :
Groups : bb
Home : /home/bb
Shell : /bin/sh
Locked : no
OK? (yes/no): yes

cd /usr/home/bb
scp backup2:/mnt/data4/build/bb/bb-freebsd_linuxcompat.tgz .
tar xzf bb-freebsd_linuxcompat.tgz

edit /home/bb/bbc1.9e-btf/etc/bb-hosts with something like:
echo "10.1.4.5 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh \
10.1.4.102 jail2.johncompanies.com # ssh" > /home/bb/bbc1.9e-btf/etc/bb-hosts

vi /home/bb/bbc1.9e-btf/ext/openfiles
MACHINE="jail2,johncompanies,com" # HAS TO BE IN A,B,C FORM

cd /usr/home/bb/bbc1.9e-btf/etc
./bbchkcfg.sh
(y to questions)
./bbchkhosts.sh
(ignore ssh errors)
cd ../..
chown -R bb .
su bb
cd
cd bbc1.9e-btf
./runbb.sh start
more BBOUT
(look for errors)
exit

echo 'su - bb -c "cd /home/bb/bbc1.9e-btf; ./runbb.sh start"' > /usr/local/etc/rc.d/bb.sh
chmod +x /usr/local/etc/rc.d/bb.sh

NOTE: to get bb working on amd, had to copy over bin dir from linux dist

27. configure load mrtg, on mail
vi /usr/local/www/mgmt/mrtg/mrtg1.cfg
(add new entry to file following existing format)

28. configure bb on mail:
vi /usr/home/bb/bbsrc/bb1.9e-btf/etc/bb-hosts
10.1.4.102 jail2.johncompanies.com # ssh

su bb
cd
bbsrc/bb/runbb.sh restart ; exit

29. remove reserve space, enable softupdates (probably already set, so not necessary)
NOT APPLICABLE IF USING GVINUM
cd
umount /mnt/data1
umount /mnt/data2
tunefs -m 0 /mnt/data1
tunefs -m 0 /mnt/data2
mount -a

30. configure ntp
echo "server 10.1.4.5" > /etc/ntp.conf

/usr/sbin/ntpd -p /var/run/ntpd.pid
sleep 2; ntpq -p
(confirm it’s able to reach our time server)

32. fwd and reverse lookups on ns1c
vr johncompanies.com
vi internal.johncompanies.com
rndc reload johncompanies.com IN private
(edit the PTR too)

33. create gvinum volumes

Make a g partition:

2950:
bsdlabel -e /dev/mfid0s1
bsdlabel -e /dev/mfid1s1
bsdlabel -e /dev/mfid2s1

given:
# /dev/aacd0s1:
8 partitions:
# size offset fstype [fsize bsize bps/cpg]
a: 262144 0 4.2BSD 2048 16384 16392
b: 4194304 262144 swap
c: 143363997 0 unused 0 0 # "raw" part, don't edit
d: 524288 4456448 4.2BSD 2048 16384 32776
e: 524288 4980736 4.2BSD 2048 16384 32776
f: 6291456 5505024 4.2BSD 2048 16384 28552

new offset = 6291456 + 5505024 = 11796480
new size is size for 'c' partition minus the new start from above
143363997 - 11796480 = 131567517
So:
g: 131567517 11796480 unused 0 0

bsdlabel -e /dev/amrd1s1
change d to g

For a 146G drive (after OS and 4G swap), we can fit 127 1G volumes so:

echo 'drive data1 device /dev/mfid0s1g' > /tmp/cgv

echo '#\!/bin/sh\
i="1"\
while [ $i -le 127 ]\
do\
echo "volume v$i" >> /tmp/cgv;\
echo 'plex org concat' >> /tmp/cgv;\
echo 'sd length 1g drive data1' >> /tmp/cgv;\
i=`expr $i + 1`\
done' > /tmp/mkcgv
sh /tmp/mkcgv
gvinum create /tmp/cgv

volume 1
plex org concat
sd length 3g drive data1
volume 2
plex org concat
sd length 5g drive data1
volume 3
plex org concat
sd length 8g drive data1
volume 4
plex org concat
sd length 6g drive data1
volume 5
plex org concat
sd length 10g drive data1

for f in 1 2 3 4 5; do bsdlabel -rw /dev/gvinum/$f; done
for f in 1 2 3 4 5; do newfs /dev/gvinum/${f}a; done
for f in 1 2 3 4 5; do mkdir /mnt/data1/$f; mount /dev/gvinum/${f}a /mnt/data1/$f;done
for f in 1 2 3 4 5; do dd if=/dev/zero of=/mnt/data1/$f/this_is_$f bs=1m count=$f; done
for f in 1 2 3 4 5; do ll -h /mnt/data1/$f/; done

For a 300G drive (after OS and 4G swap), we can fit 273 1G volumes so:

echo 'drive data2 device /dev/mfid1s1g' > /tmp/cgv

echo '#\!/bin/sh\
i="128"\
while [ $i -le 401 ]\
do\
echo "volume v$i" >> /tmp/cgv;\
echo 'plex org concat' >> /tmp/cgv;\
echo 'sd length 1g drive data2' >> /tmp/cgv;\
i=`expr $i + 1`\
done' > /tmp/mkcgv
sh /tmp/mkcgv
gvinum create /tmp/cgv

For a 300G drive (after OS and 4G swap), we can fit 273 1G volumes so:

echo 'drive data3 device /dev/mfid2s1g' > /tmp/cgv

echo '#\!/bin/sh\
i="402"\
while [ $i -le 675 ]\
do\
echo "volume v$i" >> /tmp/cgv;\
echo 'plex org concat' >> /tmp/cgv;\
echo 'sd length 1g drive data3' >> /tmp/cgv;\
i=`expr $i + 1`\
done' > /tmp/mkcgv
sh /tmp/mkcgv
gvinum create /tmp/cgv

Sometimes there’s 2040m leftover so create one more smaller vol:
echo 'volume v64\
plex org concat\
sd length 2040m drive data1' >> /tmp/cgv

To delete:

echo '#\!/bin/sh\
i="1"\
while [ $i -le 127 ]\
do\
echo "gvinum rm -r v$i" >> /tmp/dgv;\
i=`expr $i + 1`\
done' > /tmp/mkdgv
sh /tmp/mkdgv
sh /tmp/dgv

dd if=/dev/zero of=/dev/sdb seek=285474816 bs=512
dd if=/dev/zero of=/dev/mfid0s2 bs=1m count=1000 (do this to all drives)

35. create the jail template

touch /mnt/data1/jail
mdconfig -a -t vnode -s 1g -f /mnt/data1/jail -u 0
bsdlabel -rw md0
newfs4x /dev/md0c
mkdir /mnt/data1/jail-DIR
mount /dev/md0c /mnt/data1/jail-DIR

cd /usr/ports/sysutils/jailutils
make install clean

cd /usr/src
make world DESTDIR=/mnt/data1/jail-DIR
(2450: 2:28mins, supermicro: 55mins, 2950: 1h )
cd etc
make distribution DESTDIR=/mnt/data1/jail-DIR
mount -t devfs devfs /mnt/data1/jail-DIR/dev
devfs -m /mnt/data1/jail-DIR/dev rule -s 3 applyset
cd /mnt/data1/jail-DIR
ln -sf dev/null kernel
cp /usr/local/sbin/jkill /mnt/data1/jail-DIR/sbin

jail /mnt/data1/jail-DIR testhostname 192.168.11.100 /bin/sh
csh
touch /etc/fstab
echo 'network_interfaces=""\
hostname="newsystem"\
kern_securelevel_enable="NO"\
sendmail_enable="YES"\
sshd_enable="YES"' > /etc/rc.conf

echo "nameserver 69.55.225.225\
nameserver 69.55.230.3" >> /etc/resolv.conf

vi /etc/crontab
remove the adjkerntz lines
comment out periodic’s and put this line above them:
# IF YOU UNCOMMENT THESE, PLEASE ADJUST THEIR RUN TIME

rm -rf /etc/periodic/daily/400.status-disks

check /tmp for crap

vi /etc/periodic/security/100.chksetuid
replace: MP=`mount -t ufs | grep -v " nosuid" | awk '{ print $3 }' | sort`
with: MP='/' (use single quotes)

mkdir -p /usr/compat/linux/dev

adduser (Add account for user, make sure in wheel group)

user/root passwd: 8ico2987

Set root password

vi /usr/home/user/.profile (and add to the file):
TERM=vt100; export TERM

tzsetup

newaliases

rm /sbin/halt /sbin/reboot
ln /sbin/jkill /sbin/halt
ln /sbin/jkill /sbin/reboot

#cd /dev
#rm console
#ln -s null console

vi /etc/syslog.conf (comment out console and move to /var/log/messages):
#*.err;kern.warning;auth.notice;mail.crit /dev/console *.err;kern.warning;auth.notice;mail.crit /var/log/messages

exit
exit

cd libexec
mv ld-elf32.so.1 ld-elf32.so.1-orig
ln ld-elf.so.1 ld-elf32.so.1

may have to run:
chflags noschg ld-elf32.so.1
chflags noschg ld-elf.so.1
then
chflags schg ld-elf.so.1
chflags schg ld-elf32.so.1

cd /usr/ports
make -DNOCLEANDEPENDS clean
(2450: 15mins , supermicro: 29mins, 2950: 18mins)
rm -fr /usr/ports/distfiles/*
cp -r /usr/ports /mnt/data1/jail-DIR/usr (2450: 2:00 mins , supermicro: 15mins, 2950: 3mins)

rm /mnt/data1/jail-DIR/root/.history

cd
dump -0a -f /usr/local/jail/template/71template /dev/md0c
umount /mnt/data1/jail-DIR/dev
umount /dev/md0c
rmdir /mnt/data1/jail-DIR

36. setup backups
echo '#\!/bin/sh\
backupdir=/data/jail2\
server=backup1\
\
## ENTRY /etc\
## ENTRY /usr/local/etc\
## ENTRY /usr/local/jail\
## ENTRY /root/logs' > /usr/local/jail/bin/backup.config

on backup1:
setup backup dirs:
ssh backup1 mkdir -p /data/jail2/0

add the system to
vi /usr/local/sbin/snapshot_rotate

on mail:
vi /usr/local/www/mgmt/cgi/backupgraph.pl
(add hostname)

Edit /usr/local/jail/bin/backup.xxx to use the right drives and copy to /usr/local/jail/bin/backup

37. mkdir /root/logs

38. edit sshd_config for security
vi /etc/ssh/sshd_config
ListenAddress 69.55.228.53
ListenAddress 10.1.4.102

kill -1 `cat /var/run/sshd.pid`

39. add crontab entries
crontab -e
5 0 * * * /usr/local/jail/bin/backup.md
1 0 1 * * /usr/local/jail/bin/ipfwreset
0 18 * * * /usr/local/jail/bin/ipfwbackup
4,9,14,19,24,29,34,39,44,49,55,59 * * * * /usr/local/jail/bin/trafstats

On 2950 with Perc5/i also add:
*/5 * * * * /usr/local/jail/bin/perc5iraidchk

40. Reboot notify script
ln -s /usr/local/jail/bin/notify.sh /usr/local/etc/rc.d/notify.sh

41. add to management db (on mail and devweb) jc.ref_machines and jc.ref_templates

uname -r
6.2-RELEASE-jc1

insert into ref_machines values (null,'mx2','mx2.johncompanies.com',0,'m');
select machine_id from ref_machines where host='mx2';
+------------+
| machine_id |
+------------+
| 35 |
+------------+
insert into ref_templates values ('',' 7.1-RELEASE-jc2',44,'FreeBSD 7.1',0);

42. add to server/cabinet map. On mail:
vi /usr/local/www/mgmt/html/cabinetmap.html

43. add an outside blocking rule to the firewall, so this machine can only be reached from inside the firewall. Follow example already in firewall jail17 is:

00117 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 or 69.55.238.150 } to 69.55.228.200
00117 deny ip from any to 69.55.228.200

jail19 would be 00119...
ipfw add 00107 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 or 69.55.238.150 } to 69.55.230.108
ipfw add 00107 deny ip from any to 69.55.220.108

44. select customers for probe map

46. make gv start on boot

scp backup2:/mnt/data4/build/freebsd/gvinum /etc/rc.d/gvinum

If they start stale:

echo '#\!/bin/sh\
i="1"\
while [ $i -le 127 ]\
do\
gvinum start v$i\
i=`expr $i + 1`\
done' > /tmp/startgv

sh /tmp/startgv

47. patch jail against starting jails with rtprio

mv /usr/sbin/jail /usr/sbin/jail_
echo '#\!/bin/sh\
/usr/sbin/rtprio -t /usr/sbin/jail_ $*' > /usr/sbin/jail
chmod +x /usr/sbin/jail

48. make sure mail works
If there are map errors:
cd /etc/mail; make maps

49. move and pare down generic kernel
mv kernel.old/ kernel.GENERIC

/ too full, remove old kernel modules except for:

Id Refs Address Size Name
1 14 0xffffffff80100000 ac6c08 kernel
2 1 0xffffffff80bc8000 1128 mfi_linux.ko
3 4 0xffffffff80bca000 39138 linux.ko
4 1 0xffffffffb48f8000 358c linprocfs.ko
5 1 0xffffffffb48fc000 9d3 linsysfs.ko
6 1 0xffffffffb490c000 80ee8 zfs.ko

50. ntp doesn’t seem to start from rc so,

echo '/usr/sbin/ntpd -c /etc/ntp.conf -p /var/run/ntpd.pid -f /var/db/ntpd.drift' > /usr/local/etc/rc.d/ntp.sh
chmod 0700 /usr/local/etc/rc.d/ntp.sh</pre>

= FreeBSD 7.2 =

Last updated 2009-06-18

<pre>All time extimates below assume disks aren’t scrubbing. Setup instructions below are for LSI card:

1. make sure bios is setup for bios console redirect
2950:
Console redirection:
LCD string..
Date to GMT

2. assuming mirrors (or at least disks) created (if not, refer to this), boot to disk 1 of 7.2
skip kernel config (enter)
custom install

partition ->
move cursor to mfid0, hit space (takes you to partition map screen)
a for entire disk
q to quit and save

standard mbr (no boot manager)
space to unselect mfid0
cursor over mfid1
space
a for entire disk
q to quit and save
none (leave untouched)

cursor over mfid0
space
(takes you into part. Screen again) q to exit
none
Make sure both are checked and tab to ok

Label ->
Make sure mfid0 is highlighted

/ 256M
swap 2G (for 2950 make it 4G)
/var 256M (4.6G)
/tmp 256M
/usr 4G
/mnt/data1 remaining space (no need to newfs)

Make sure to toggle S for soft updates on all (should look like UFS2+S Y under the Newfs column)

move cursor to mfid1
swap 8G (or 4G if there’s a 3rd drive)
/mnt/data2 remaining space (no need to newfs)

q to save and exit

distributions ->
developer
yes to install ports
exit

media ->
cd

commit ->
yes
(2450: 14mins, supermicro: 12mins)

yes to "visit general config" ->
Set root pwd

Add user ‘user’ member group is wheel, set password

Set tz

Networking->interfaces->Fxp0 ->
No IPV6
dhcp=no
Set hostname & domain
Enable sshd

exit...
exit install ->
yes

take the cd out and let the machine reboot

3. double check the date/time

4. edit /etc/make.conf
echo "WITHOUT_X11=yes \
KERNCONF=jail8 \
BOOT_COMCONSOLE_SPEED=115200" >> /etc/make.conf

5. add settings to /boot/loader.conf and /boot.config

echo "-Dh" >> /boot.config

echo 'console="comconsole,vidconsole" \
boot_multicons="YES" \
boot_serial="YES" \
mfi_linux_load="YES" \
comconsole_speed="115200"' >> /boot/loader.conf

6. turn off all ttyv's except 0 and 1 in /etc/ttys
also turn on ttyd0, change type to vt100:
vi /etc/ttys

ttyv2 "/usr/libexec/getty Pc" cons25 off secure
ttyv3 "/usr/libexec/getty Pc" cons25 off secure
ttyv4 "/usr/libexec/getty Pc" cons25 off secure
ttyv5 "/usr/libexec/getty Pc" cons25 off secure
ttyv6 "/usr/libexec/getty Pc" cons25 off secure
ttyv7 "/usr/libexec/getty Pc" cons25 off secure

ttyd0 "/usr/libexec/getty std.9600" vt100 on secure

kill -1 1

on console server:
vi /etc/remote
(rename port to jail18 depending on where and which digi plugged into)
test serial console

7. populate hosts
echo "10.1.4.3 backup2" >> /etc/hosts
echo "10.1.4.8 backup1" >> /etc/hosts

8. put key in authorized_keys on backup2
cd
ssh-keygen -t dsa -b 1024
(default location, leave password blank)

cat /root/.ssh/id_dsa.pub | ssh backup2 'cat - >> /root/.ssh/authorized_keys'
cat /root/.ssh/id_dsa.pub | ssh backup1 'cat - >> /root/.ssh/authorized_keys'

confirm that you can ssh to backup2 and backup1 without getting a login prompt

ssh backup2 hostname

ssh backup1 hostname

9. create & populate binaries/scripts dirs
mkdir -p /usr/local/jail/bin
mkdir -p /usr/local/jail/rc.d
mkdir -p /usr/local/jail/template/
mkdir /mnt/data1
mkdir /mnt/data2
scp backup2:"/mnt/data4/bin/freebsd7.x/*" /usr/local/jail/bin
cd /usr/local/jail/rc.d/
touch quad1
touch deprecated
chmod +x *
cd /usr/local/jail/bin
ln -s /usr/local/jail/rc.d/quad1 quad1
ln -s /usr/local/jail/bin/jailmake_md jailmake
ln -s /usr/local/jail/bin/js_md js
ln -s /usr/local/jail/bin/canceljail_md canceljail
ln -s /usr/local/jail/bin/jailmakeempty_md jailmakeempty
ln -s /usr/local/jail/bin/postboot_md postboot

rehash

10. edit root's path and login script:
vi /root/.cshrc

Change alias entries (add G):
alias la ls -aG
alias lf ls -FAG
alias ll ls -lAG
alias ls ls -AG
alias mbm mb mount
alias mbu mb umount

add to path:
/usr/local/jail/bin
(if adaptec card installed, also add /compat/linux/usr/sbin)

and alter the prompt, set the following:
set prompt = "`/bin/hostname -s` %/# "

at the bottom of the file add:
set sshtty=`who am i|awk '{print $2}'`
/usr/sbin/rtprio 3 -`psj | grep $sshtty | awk '{print $2}'`

set shortty=`who am i | awk '{print $2}' | sed -E 's/.*(..)$/\1/'`
foreach x (`psj | grep sh | grep $shortty | awk '{print $2}'`)
/usr/sbin/rtprio 2 -$x
end

To load the new file:
source /root/.cshrc

11. install cvsup
cd /usr/ports/net/cvsup-without-gui
make install clean; rehash; mail -s 'cvs installed' dave.boodman@vtext.com < /dev/null

(stay close for gettext options, 2450: 27mins, supermicro: 17mins, 2950: 22mins)

12. get latest sources for this release:
cd /usr/src
echo "*default host=cvsup4.freebsd.org\
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_7_2\
*default delete use-rel-suffix\
*default compress\
src-all" > sup

-OR-

echo "*default host=cvsup4.freebsd.org\
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_7\
*default delete use-rel-suffix\
*default compress\
src-all" > sup

(stable)

cvsup sup ; mail -s 'cvs sup done' dave.boodman@vtext.com < /dev/null

(2450, ~12mins, supermicro, 27mins, 2950: 7mins)

13. configure new kernel.

cd /usr/src/sys/amd64/conf
scp backup2:/mnt/data4/build/freebsd/kern_config-7.1-amd64 ./jail8

edit the kernel config and change ident to be the name of the jail:
vi jail8
ident jail8

edit /sys/conf/newvers.sh to add –jc2 to the end of the BRANCH string (RELEASE-jc2)
vi /sys/conf/newvers.sh

notes: http://www.FreeBSD.org/doc/en_US.ISO8859-1/books/handbook/kernelconfig-config.html

14. bring over patches from backup2

The various patches are in /mnt/data4/build/freebsd/patches on backup2. There are dirs for each version. Not all dirs are populated, but patches for later versions work on older ones unless there is a new patch in the older dir.

cd /usr/src
scp backup2:"/mnt/data4/build/freebsd/patches/7.2/*" .

Apply patches:
patch -l < jls-patch

15. build, install kernel and world

cd /boot

mv kernel kernel.GENERIC
cd kernel.GENERIC
mkdir hold
mv mfi_linux.ko hold/
mv linux.ko hold/
mv linprocfs.ko hold/
mv linsysfs.ko hold/
mv geom_vinum.ko hold/
mv geom_concat.ko hold/
rm *.ko
rm *.symbols
mv hold/* .
rmdir hold/

cd /usr/src
make buildkernel installkernel

make buildworld ; mail -s 'buildworld done' dave.boodman@vtext.com < /dev/null
(2450: 1:56min, supermicro: 59mins, 2950: 38mins)
make installworld
(2450: 3min, supermicro: 1min, 2950: :34)
mergemaster -i
delete /var/tmp/temproot
delete bsnmpd
delete temporary ./etc/hosts
delete temporary ./etc/motd
delete /var/tmp/temproot

cd /sys/modules/geom/geom_vinum
make
make install

16. populate devfs ruleset
scp backup2:/mnt/data4/build/freebsd/devfs.rules /etc

17. populate /etc/rc.conf with IPs and NFS settings
vi /etc/rc.conf

kern_securelevel_enable="NO"
portmap_enable="NO"
sendmail_enable="NO"
usbd_enable="YES"

xntpd_enable="YES"
nfs_client_enable="YES"
nfs_reserved_port_only="YES"
inetd_enable="YES"
inetd_flags="-wW -a 10.1.4.102"
devfs_system_ruleset="devfsrules_show_all"

#ifconfig_xl0="inet 10.1.4.102 netmask 255.255.255.0"
#ifconfig_fxp0="inet 69.55.228.53 netmask 255.255.255.0"
#ifconfig_fxp0_alias0="inet 69.55.2xx.xx netmask 255.255.255.0"

static_routes="t1 office"
route_t1="-net 10.1.5 10.1.4.2"
route_office="-net 10.1.6 10.1.4.2"
gvinum_enable="YES"
fsck_y_enable="YES"
background_fsck="NO"
rc_mfi_raid_tty_log="YES"

18. make sure sysctls are set and preserved after reboot
echo "kern.consmute=0\
kern.ipc.shm_use_phys=1\
kern.ipc.shmall=131070\
kern.ipc.shmmax=134217728\
net.inet.tcp.syncookies=0\
kern.maxfiles=32768\
kern.fallback_elf_brand=3\
kern.maxprocperuid=4000\
security.jail.sysvipc_allowed=1\
security.jail.allow_raw_sockets=1\
security.jail.socket_unixiproute_only=1\
security.jail.chflags_allowed=0\
dev.amr.0.allow_volume_configure=1\
compat.linux.osrelease=2.6.12\
vm.pmap.shpgperproc=300" >> /etc/sysctl.conf

19. mount procfs
echo "proc /proc procfs rw 0 0" >> /etc/fstab

For Dell 2950/2450:
echo "linprocfs /usr/compat/linux/proc linprocfs rw 0 0" >> /etc/fstab

For Dell 2950:
echo "linsysfs /usr/compat/linux/sys linsysfs rw 0 0" >> /etc/fstab

mkdir -p /usr/compat/linux/proc
mkdir -p /usr/compat/linux/sys

19. enable noatime option
NOT APPLICABLE IF RUNNING GVINUM
data1 and data2 should look something like:
/dev/amrd0s1g /mnt/data1 ufs rw,noatime 2 2

20. reboot. Confirm new kernel is loaded

uname -a

Check devfs rules:
devfs rule showsets
devfs rule -s 3 show

21. update ports:
cd /usr/ports
echo "*default host=cvsup4.FreeBSD.org\
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_7_2\
*default delete use-rel-suffix\
*default compress\
ports-all tag=." > sup

cvsup sup; mail -s 'cvs sup ports done' dave.boodman@vtext.com < /dev/null

(2450: 18mins, supermicro: 19mins; 2950: 24mins)

22. Install raid mgmt tool

For LSI based cards:

install linux_base:
cd /usr/ports/emulators/linux_base-fc4
make install clean
(2450: 7min, supermicro: 3mins, 2950: 14mins)

(for LSI)
cd /usr/ports/sysutils/linux-megamgr
make install clean

cd /usr/ports/sysutils/megarc
make install clean

(for Perc5/i, 6/i)
cd /usr/ports/sysutils/linux-megacli2
make install clean

Test:
rehash; megacli ldinfo lall a0

For adaptec:
On jail18:
scp /compat/linux/usr/sbin/aaccli user@10.1.4.102:~

mv ~user/aaccli /compat/linux/usr/sbin/aaccli

test out;
/compat/linux/usr/sbin/aaccli

For 3w9690:
scp backup2:/d4/build/3ware/tw_cli-freebsd-x86_64-9.5.0.1.tgz /usr/local/sbin
cd /usr/local/sbin xzvf tw_cli-freebsd-x86_64-9.5.0.1.tgz
rm tw_cli-freebsd-x86_64-9.5.0.1.tgz

test out;
./tw_cli /c0 show allstatus

23. install rsync from ports
cd /usr/ports/net/rsync
make install clean

choose default options

25. configure inetd to respond to mrtg load queries
echo "load stream tcp nowait user /usr/local/jail/bin/load.pl load.pl" >> /etc/inetd.conf

echo "load 12384/tcp" >> /etc/services

26. install bb client
adduser
Username: bb
Full name: bb
Uid (Leave empty for default): 1984
Login group [bb]:
Login group is bb. Invite bb into other groups? []:
Login class [default]:
Shell (sh csh tcsh nologin) [sh]:
Home directory [/home/bb]:
Use password-based authentication? [yes]:
Use an empty password? (yes/no) [no]:
Use a random password? (yes/no) [no]: yes
Lock out the account after creation? [no]:
Username : bb
Password : <random>
Full Name : bb
Uid : 1984
Class :
Groups : bb
Home : /home/bb
Shell : /bin/sh
Locked : no
OK? (yes/no): yes

cd /usr/home/bb
scp backup2:/mnt/data4/build/bb/bb-freebsd_linuxcompat.tgz .
tar xzf bb-freebsd_linuxcompat.tgz

edit /home/bb/bbc1.9e-btf/etc/bb-hosts with something like:
echo "10.1.4.5 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh \
10.1.4.108 jail8.johncompanies.com # ssh" > /home/bb/bbc1.9e-btf/etc/bb-hosts

vi /home/bb/bbc1.9e-btf/ext/openfiles
MACHINE="jail8,johncompanies,com" # HAS TO BE IN A,B,C FORM

cd /usr/home/bb/bbc1.9e-btf/etc
./bbchkcfg.sh
(y to questions)
./bbchkhosts.sh
(ignore ssh errors)
cd ../..
chown -R bb .
su bb
cd
cd bbc1.9e-btf
./runbb.sh start
more BBOUT
(look for errors)
exit

echo 'su - bb -c "cd /home/bb/bbc1.9e-btf; ./runbb.sh start"' > /usr/local/etc/rc.d/bb.sh
chmod +x /usr/local/etc/rc.d/bb.sh

NOTE: to get bb working on amd, had to copy over bin dir from linux dist

27. configure load mrtg, on mail
vi /usr/local/www/mgmt/mrtg/mrtg1.cfg
(add new entry to file following existing format)

28. configure bb on mail:
vi /usr/home/bb/bbsrc/bb1.9e-btf/etc/bb-hosts
10.1.4.102 jail2.johncompanies.com # ssh

su bb
cd
bbsrc/bb/runbb.sh restart ; exit

29. remove reserve space, enable softupdates (probably already set, so not necessary)
NOT APPLICABLE IF USING GVINUM
cd
umount /mnt/data1
umount /mnt/data2
tunefs -m 0 /mnt/data1
tunefs -m 0 /mnt/data2
mount -a

30. configure ntp
echo "server 10.1.4.5" > /etc/ntp.conf

/usr/sbin/ntpd -p /var/run/ntpd.pid
sleep 2; ntpq -p
(confirm it’s able to reach our time server)

32. fwd and reverse lookups on ns1c
vr johncompanies.com
vi internal.johncompanies.com
rndc reload johncompanies.com IN private
(edit the PTR too)

35. create the jail template

touch /mnt/data1/jail
mdconfig -a -t vnode -s 1g -f /mnt/data1/jail -u 0
bsdlabel -rw md0
newfs4x -i 4096 /dev/md0c
mkdir /mnt/data1/jail-DIR
mount /dev/md0c /mnt/data1/jail-DIR

cd /usr/ports/sysutils/jailutils
make install clean

cd /usr/src
make world DESTDIR=/mnt/data1/jail-DIR; pagedave
(2450: 2:28mins, supermicro: 55mins, 2950: 1h )
cd etc
make distribution DESTDIR=/mnt/data1/jail-DIR
mount -t devfs devfs /mnt/data1/jail-DIR/dev
devfs -m /mnt/data1/jail-DIR/dev rule -s 3 applyset
cd /mnt/data1/jail-DIR
ln -sf dev/null kernel
cp /usr/local/sbin/jkill /mnt/data1/jail-DIR/sbin

jail /mnt/data1/jail-DIR testhostname 192.168.11.100 /bin/sh
csh
touch /etc/fstab
echo 'network_interfaces=""\
hostname="newsystem"\
kern_securelevel_enable="NO"\
sendmail_enable="YES"\
sshd_enable="YES"' > /etc/rc.conf

echo "nameserver 69.55.225.225\
nameserver 69.55.230.3" >> /etc/resolv.conf

vi /etc/crontab
remove the adjkerntz lines
comment out periodic’s and put this line above them:
# IF YOU UNCOMMENT THESE, PLEASE ADJUST THEIR RUN TIME

rm -rf /etc/periodic/daily/400.status-disks

check /tmp for crap

vi /etc/periodic/security/100.chksetuid
replace: MP=`mount -t ufs | grep -v " nosuid" | awk '{ print $3 }' | sort`
with: MP='/' (use single quotes)

mkdir -p /usr/compat/linux/dev

adduser (Add account for user, make sure in wheel group)

Username: user
Full name: user
Uid (Leave empty for default):
Login group [user]:
Login group is user. Invite user into other groups? []: wheel
Login class [default]:
Shell (sh csh tcsh nologin) [sh]:
Home directory [/home/user]:
Home directory permissions (Leave empty for default):
Use password-based authentication? [yes]:
Use an empty password? (yes/no) [no]:
Use a random password? (yes/no) [no]: y
Lock out the account after creation? [no]:
Username : user
Password : <random>
Full Name : user
Uid : 1001
Class :
Groups : user
Home : /home/user
Home Mode :
Shell : /bin/sh
Locked : no
OK? (yes/no): y
adduser: INFO: Successfully added (user) to the user database.
adduser: INFO: Password for (user) is: 901gmYjO
Add another user? (yes/no): n
Goodbye!

vi /usr/home/user/.profile (and add to the file):
TERM=vt100; export TERM

tzsetup

newaliases

rm /sbin/halt /sbin/reboot
ln /sbin/jkill /sbin/halt
ln /sbin/jkill /sbin/reboot

#cd /dev
#rm console
#ln -s null console

vi /etc/syslog.conf (comment out console and move to /var/log/messages):
#*.err;kern.warning;auth.notice;mail.crit /dev/console *.err;kern.warning;auth.notice;mail.crit /var/log/messages

exit
exit

cd libexec
chflags noschg ld-elf32.so.1
chflags noschg ld-elf.so.1

mv ld-elf32.so.1 ld-elf32.so.1-orig
ln ld-elf.so.1 ld-elf32.so.1

chflags schg ld-elf.so.1
chflags schg ld-elf32.so.1

cd /usr/ports
make -DNOCLEANDEPENDS clean
(2450: 15mins , supermicro: 29mins, 2950: 18mins)
rm -fr /usr/ports/distfiles/*
cp -r /usr/ports /mnt/data1/jail-DIR/usr (2450: 2:00 mins , supermicro: 15mins, 2950: 3mins)

rm /mnt/data1/jail-DIR/root/.history

cd
umount /mnt/data1/jail-DIR/dev
dump -0a -f /usr/local/jail/template/72template /dev/md0c
umount /dev/md0c
rmdir /mnt/data1/jail-DIR

36. setup backups
echo '#\!/bin/sh\
backupdir=/data/jail2\
server=backup1\
\
## ENTRY /etc\
## ENTRY /usr/local/etc\
## ENTRY /usr/local/jail\
## ENTRY /root/logs' > /usr/local/jail/bin/backup.config

on backup1:
setup backup dirs:
ssh backup1 mkdir -p /data/jail2/0

add the system to
vi /usr/local/sbin/snapshot_rotate

on mail:
vi /usr/local/www/mgmt/cgi/backupgraph.pl
(add hostname)

Edit /usr/local/jail/bin/backup.xxx to use the right drives and copy to /usr/local/jail/bin/backup

37. mkdir /root/logs

38. edit sshd_config for security
vi /etc/ssh/sshd_config
ListenAddress 69.55.228.53
ListenAddress 10.1.4.102

kill -1 `cat /var/run/sshd.pid`

39. add crontab entries
crontab -e
5 0 * * * /usr/local/jail/bin/backup.md
1 0 1 * * /usr/local/jail/bin/ipfwreset
0 18 * * * /usr/local/jail/bin/ipfwbackup
4,9,14,19,24,29,34,39,44,49,55,59 * * * * /usr/local/jail/bin/trafstats

On 2950 with Perc5/i also add:
*/5 * * * * /usr/local/jail/bin/perc5iraidchk

40. Reboot notify script
ln -s /usr/local/jail/bin/notify.sh /usr/local/etc/rc.d/notify.sh

41. add to management db (on mail and devweb) jc.ref_machines and jc.ref_templates

uname -r
6.2-RELEASE-jc1

insert into ref_machines values (null,'mx2','mx2.johncompanies.com',0,'m');
select machine_id from ref_machines where host='mx2';
+------------+
| machine_id |
+------------+
| 35 |
+------------+
insert into ref_templates values ('',' 7.1-RELEASE-jc2',44,'FreeBSD 7.1',0);

42. add to server/cabinet map. On mail:
vi /usr/local/www/mgmt/html/cabinetmap.html

43. add an outside blocking rule to the firewall, so this machine can only be reached from inside the firewall. Follow example already in firewall jail17 is:

00117 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 or 69.55.238.150 } to 69.55.228.200
00117 deny ip from any to 69.55.228.200

jail19 would be 00119...
ipfw add 00107 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 or 69.55.238.150 } to 69.55.230.108
ipfw add 00107 deny ip from any to 69.55.220.108

44. select customers for probe map

47. patch jail against starting jails with rtprio

mv /usr/sbin/jail /usr/sbin/jail_
echo '#\!/bin/sh\
/usr/sbin/rtprio -t /usr/sbin/jail_ $*' > /usr/sbin/jail
chmod +x /usr/sbin/jail

48. make sure mail works
If there are map errors:
cd /etc/mail; make maps

50. ntp doesn’t seem to start from rc so,

echo '/usr/sbin/ntpd -c /etc/ntp.conf -p /var/run/ntpd.pid -f /var/db/ntpd.drift' > /usr/local/etc/rc.d/ntp.sh
chmod 0700 /usr/local/etc/rc.d/ntp.sh

51. recover space on /usr

rm -fr /usr/obj

52. setup fuse

cd /usr/ports/sysutils/fusefs-kmod/
make install

vi /etc/rc.conf
fusefs_enable="YES"

sysctl vfs.usermount=1

cd /usr/ports/sysutils/fusefs-sshfs
make install

sshfs 1005@usw-s009.rsync.net: /mnt/data1/69.55.234.68-col00001-DIR/mnt</pre>

= Stress testing a 6.x jail =

<pre>mkdir /mnt/jail
newfs /dev/gvinum/v1
mount /dev/gvinum/v1 /mnt/jail
cd /mnt/jail
restore -r -f /path/to/61stress .
cd ..
umount /mnt/jail

sh
for f in 1 2 3 4 5 6 7 8 9 10; do mkdir /mnt/data1/$f; done

gconcat label v3-v5 /dev/gvinum/v3 /dev/gvinum/v4 /dev/gvinum/v5
bsdlabel -r -w /dev/concat/v3-v5
newfs /dev/concat/v3-v5a
mount /dev/concat/v3-v5a /mnt/data1/1
bsdlabel -r -w /dev/gvinum/v10
newfs /dev/gvinum/v10a
mount /dev/gvinum/v10a /mnt/data1/2
gconcat label v23-v25 /dev/gvinum/v23 /dev/gvinum/v24 /dev/gvinum/v25
bsdlabel -r -w /dev/concat/v23-v25
newfs /dev/concat/v23-v25a
mount /dev/concat/v23-v25a /mnt/data1/3
bsdlabel -r -w /dev/gvinum/v50
newfs /dev/gvinum/v50a
mount /dev/gvinum/v50a /mnt/data1/4
gconcat label v63-v65 /dev/gvinum/v63 /dev/gvinum/v64 /dev/gvinum/v65
bsdlabel -r -w /dev/concat/v63-v65
newfs /dev/concat/v63-v65a
mount /dev/concat/v63-v65a /mnt/data1/5
bsdlabel -r -w /dev/gvinum/v70
newfs /dev/gvinum/v70a
mount /dev/gvinum/v70a /mnt/data1/6
gconcat label v83-v85 /dev/gvinum/v83 /dev/gvinum/v84 /dev/gvinum/v85
bsdlabel -r -w /dev/concat/v83-v85
newfs /dev/concat/v83-v85a
mount /dev/concat/v83-v85a /mnt/data1/7
bsdlabel -r -w /dev/gvinum/v100
newfs /dev/gvinum/v100a
mount /dev/gvinum//v100a /mnt/data1/8
gconcat label v123-v125 /dev/gvinum/v123 /dev/gvinum/v124 /dev/gvinum/v125
bsdlabel -r -w /dev/concat/v123-v125
mount /dev/concat/v123-v125a /mnt/data1/9
newfs /dev/concat/v123-v125a
mount /dev/concat/v123-v125a /mnt/data1/9
bsdlabel -r -w /dev/gvinum/v130
newfs /dev/gvinum/v130a
mount /dev/gvinum//v130a /mnt/data1/10

sh
for f in 1 2 3 4 5 6 7 8 9 10; do cd /mnt/data1/$f; dump -0a -f - /dev/gvinum/v1| restore -r -f - ; done

sh
for f in 1 2 3 4 5 6 7 8 9 10; do mount_devfs devfs /mnt/data1/$f/dev; devfs -m /mnt/data1/$f/dev rule -s 3 applyset; done

sh
for f in 15 16 17 18 19 20 21 22 23 24; do ifconfig bce0 alias 10.1.6.$f netmask 255.255.255.255; done

jail /mnt/data1/1 testhostname1 10.1.6.15 /bin/sh /etc/rc 2> /dev/null
jail /mnt/data1/2 testhostname2 10.1.6.16 /bin/sh /etc/rc 2> /dev/null
jail /mnt/data1/3 testhostname3 10.1.6.17 /bin/sh /etc/rc 2> /dev/null
jail /mnt/data1/4 testhostname4 10.1.6.18 /bin/sh /etc/rc 2> /dev/null
jail /mnt/data1/5 testhostname5 10.1.6.19 /bin/sh /etc/rc 2> /dev/null
jail /mnt/data1/6 testhostname6 10.1.6.20 /bin/sh /etc/rc 2> /dev/null
jail /mnt/data1/7 testhostname7 10.1.6.21 /bin/sh /etc/rc 2> /dev/null
jail /mnt/data1/8 testhostname8 10.1.6.22 /bin/sh /etc/rc 2> /dev/null
jail /mnt/data1/9 testhostname9 10.1.6.23 /bin/sh /etc/rc 2> /dev/null
jail /mnt/data1/10 testhostname10 10.1.6.24 /bin/sh /etc/rc 2> /dev/null

systat -vmstat</pre>

= Stress testing zfs =

<pre>zfs receive pool1/jail2 < /usr/local/jail/template/template
zfs set mountpoint=/mnt/data1/jail2 pool1/jail2
zfs set quota=4G pool1/jail2
zfs destroy pool1/jail2@now</pre>
install apps that will run when jail started

<pre>cp -r /usr/src /mnt/data1/jail2/usr
rm -fr /mnt/data1/jail2/usr/obj

mount -t devfs devfs /mnt/data1/jail2/dev; devfs -m /mnt/data1/jail2/dev rule -s 3 applyset;

jail /mnt/data1/jail2 stress-test 69.55.234.86 /bin/sh
csh
cd /usr/ports/benchmarks/bonnie
make install clean
cd /usr/ports/sysutils/stress
make install clean
cd /usr/ports/net/rsync
make install clean
cd /usr/ports/lang/perl5.8
make install clean

cd /usr/local/etc/rc.d/

cat >> boot.sh
sleep 30 && sh /usr/local/etc/rc.d/buildworld &
sleep 30 && sh /usr/local/etc/rc.d/portindex &
sleep 30 && sh /usr/local/etc/rc.d/stress &
sleep 30 && sh /usr/local/etc/rc.d/bonnie &

cat > buildworld
#!/bin/sh
while (true); do cd /usr/src; make buildworld; done

cat > portindex
while (true); do cd /usr/ports/; make index; done

cat > stress
/usr/local/bin/stress -c 1 -i 1 -m 1 -d 1 --vm-bytes 768M --hdd-bytes 128M

cat > bonnie
#!/bin/sh
while (true); do /usr/local/bin/bonnie -s 2g; done

chmod +x *

exit
exit

cd
zfs snapshot pool1/jail2@now

sh
for f in 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105; do zfs send pool1/jail2@now | zfs receive pool1/jail$f;
zfs set quota=10G pool1/jail$f;
zfs set mountpoint=/mnt/data1/jail$f pool1/jail$f;
done

for f in 106 107 108 114 115 116 117 118 119 120 121 122 123 125 126 127 128 129 130 131; do zfs send pool1/jail2@now | zfs receive pool2/jail$f
zfs set quota=10G pool2/jail$f;
zfs set mountpoint=/mnt/data2/jail$f pool2/jail$f;
done

for f in 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101; do chmod 0 /mnt/data1/jail$f/usr/local/etc/rc.d/bonnie; done
for f in 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105; do chmod 0 /mnt/data1/jail$f/usr/local/etc/rc.d/stress; done

for f in 115 116 117 118 119 120 121 122 123 125 126 127; do chmod 0 /mnt/data2/jail$f/usr/local/etc/rc.d/bonnie; done
for f in 119 120 121 122 123 125 126 127 128 129 130 131; do chmod 0 /mnt/data2/jail$f/usr/local/etc/rc.d/stress; done

for f in 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105; do mount -t devfs devfs /mnt/data1/jail$f/dev; devfs -m /mnt/data1/jail$f/dev rule -s 3 applyset; done

for f in 106 107 108 114 115 116 117 118 119 120 121 122 123 125 126 127 128 129 130 131; do mount -t devfs devfs /mnt/data2/jail$f/dev; devfs -m /mnt/data2/jail$f/dev rule -s 3 applyset; done

for f in 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105;
do jail /mnt/data1/jail$f testhostname$f 69.55.234.$f /bin/sh /etc/rc 2> /dev/null; done

for f in 106 107 108 114 115 116 117 118 119 120 121 122 123 125 126 127 128 129 130 131;
do jail /mnt/data2/jail$f testhostname$f 69.55.234.$f /bin/sh /etc/rc 2> /dev/null; done

systat -vmstat
sysctl vfs.numvnodes

for f in 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105; do umount /mnt/data1/jail$f/dev; zfs destroy -r pool1/jail$f; done

for f in 106 107 108 114 115 116 117 118 119 120 121 122 123 125 126 127 128 129 130 131; do umount /mnt/data2/jail$f/dev; zfs destroy -r pool2/jail$f; done

for f in 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105; do touch /mnt/data1/jail$f;
mdconfig -a -t vnode -s 10g -f /mnt/data1/jail$f -u $f;
bsdlabel -r -w md$f auto;
newfs -O 1 /dev/md${f}a;
mkdir /mnt/data1/jail$f-DIR;
mount /dev/md${f}a /mnt/data1/jail$f-DIR;
rsync -aSH /mnt/data2/jail2/ /mnt/data1/jail$f-DIR/;
done

for f in 106 107 108 114 115 116 117 118 119 120 121 122 123 125 126 127 128 129 130 131; do touch /mnt/data2/jail$f;
mdconfig -a -t vnode -s 10g -f /mnt/data2/jail$f -u $f;
bsdlabel -r -w md$f auto;
newfs -O 1 /dev/md${f}a;
mkdir /mnt/data2/jail$f-DIR;
mount /dev/md${f}a /mnt/data2/jail$f-DIR;
rsync -aSH /mnt/data1/jail86-DIR/ /mnt/data2/jail$f-DIR/;
done

for f in 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105;
do chmod 0700 /mnt/data1/jail$f-DIR/usr/local/etc/rc.d/bonnie; done
for f in 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105;
do chmod 0700 /mnt/data1/jail$f-DIR/usr/local/etc/rc.d/stress; done

for f in 106 107 108 114 115 116 117 118 119 120 121 122 123 125 126 127 128 129 130 131;
do chmod 0700 /mnt/data2/jail$f-DIR/usr/local/etc/rc.d/bonnie; done
for f in 106 107 108 114 115 116 117 118 119 120 121 122 123 125 126 127 128 129 130 131;
do chmod 0700 /mnt/data2/jail$f-DIR/usr/local/etc/rc.d/stress; done

for f in 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101; do chmod 0 /mnt/data1/jail$f-DIR/usr/local/etc/rc.d/bonnie; done
for f in 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105; do chmod 0 /mnt/data1/jail$f-DIR/usr/local/etc/rc.d/stress; done

for f in 115 116 117 118 119 120 121 122 123 125 126 127; do chmod 0 /mnt/data2/jail$f-DIR/usr/local/etc/rc.d/bonnie; done
for f in 119 120 121 122 123 125 126 127 128 129 130 131; do chmod 0 /mnt/data2/jail$f-DIR/usr/local/etc/rc.d/stress; done

for f in 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105; do mount -t devfs devfs /mnt/data1/jail$f-DIR/dev; devfs -m /mnt/data1/jail$f-DIR/dev rule -s 3 applyset; done

for f in 106 107 108 114 115 116 117 118 119 120 121 122 123 125 126 127 128 129 130 131; do mount -t devfs devfs /mnt/data2/jail$f-DIR/dev; devfs -m /mnt/data2/jail$f-DIR/dev rule -s 3 applyset; done

for f in 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105;
do jail /mnt/data1/jail$f-DIR testhostname$f 69.55.234.$f /bin/sh /etc/rc 2> /dev/null; done

for f in 106 107 108 114 115 116 117 118 119 120 121 122 123 125 126 127 128 129 130 131;
do jail /mnt/data2/jail$f-DIR testhostname$f 69.55.234.$f /bin/sh /etc/rc 2> /dev/null; done

for f in 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105;
do cp /mnt/data1/boot.sh /mnt/data1/jail$f-DIR/usr/local/etc/rc.d/;
cp /mnt/data1/stress /mnt/data1/jail$f-DIR/usr/local/etc/rc.d/;
cp /mnt/data1/bonnie /mnt/data1/jail$f-DIR/usr/local/etc/rc.d/;
done

for f in 106 107 108 114 115 116 117 118 119 120 121 122 123 125 126 127 128 129 130 131;
do cp /mnt/data1/boot.sh /mnt/data2/jail$f-DIR/usr/local/etc/rc.d/;
cp /mnt/data1/stress /mnt/data2/jail$f-DIR/usr/local/etc/rc.d/;
cp /mnt/data1/bonnie /mnt/data2/jail$f-DIR/usr/local/etc/rc.d/;
done

for f in 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105; do umount /mnt/data1/jail$f-DIR/dev; umount /mnt/data1/jail$f-DIR; mdconfig -d -u $f; done

for f in 106 107 108 114 115 116 117 118 119 120 121 122 123 125 126 127 128 129 130 131; do umount /mnt/data2/jail$f-DIR/dev; umount /mnt/data2/jail$f-DIR; mdconfig -d -u $f; done

for f in 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105; do
mkdir /mnt/data1/jail$f-DIR/;
cd /mnt/data1/jail$f-DIR/;
dump -0a -f - /dev/md0a | restore -r -f -;
done

for f in 106 107 108 114 115 116 117 118 119 120 121 122 123 125 126 127 128 129 130 131; do mkdir /mnt/data2/jail$f-DIR/;
cd /mnt/data2/jail$f-DIR/;
dump -0a -f - /dev/md0a | restore -r -f -;
done

for f in 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105; do chmod 0700 /mnt/data1/jail$f-DIR/usr/local/etc/rc.d/*; done

for f in 106 107 108 114 115 116 117 118 119 120 121 122 123 125 126 127 128 129 130 131; do chmod 0700 /mnt/data2/jail$f-DIR/usr/local/etc/rc.d/*; done

--vm-bytes 300M</pre>

= Reloading FreeBSD (while preserving customer data) =

<pre>2. assuming 2 mirrors, boot to disk 1 of 4.11
skip kernel config (enter)
custom install
(skip partition)
label
move cursor to aacd0, hit space (takes you to partition map screen)
given this as the pre-existing partition map:
Filesystem 1K-blocks Used Avail Capacity Mounted on
/dev/aacd0s1a 128990 36888 81784 31% /
/dev/aacd0s1h 30499106 4 28059174 0% /mnt/data1
/dev/aacd1s1e 69526550 4 63964422 0% /mnt/data2
/dev/aacd0s1f 257998 10 237350 0% /tmp
/dev/aacd0s1g 3096462 852082 1996664 30% /usr
/dev/aacd0s1e 257998 1198 236162 1% /var
procfs 4 4 0 100% /proc

relabel and all mount points according to prev settings

or, delete all partitions except for data parts, re-create as necessary

re-toggle newfs on all mount points except for /mnt/data1 - make sure under the Newfs column the data1 (and data2) partiotions look like: UFS+S Y

make sure under the Newfs column the data1 (and data2) partiotions look like
UFS+S N

space to unselect aacd0
cursor over aacd1
space

Space to select both drives and tab to ok

Continue installation as per normal
distributions
developer
yes to install ports
exit
media
cd
commit
yes
(2450: 16mins, supermicro: 11mins)
yes to "visit general config"
Set root pwd
Add user ‘user’ member group is wheel, set password
Set tz
Networking->interfaces->Fxp0
No IPV6
dhcp=yes
Set hostname & domain

exit...
exit install
yes
take the cd out and let the machine reboot</pre>