Editing
RAID Cards
Jump to navigation
Jump to search
Warning:
You are not logged in. Your IP address will be publicly visible if you make any edits. If you
log in
or
create an account
, your edits will be attributed to your username, along with other benefits.
Anti-spam check. Do
not
fill this in!
= Adaptec = == CLI notes == On all of our 2450 servers and a handful of old FreeBSD boxes, we are running on adaptec RAID cards. The CLI program is <tt>aaccli</tt> on FreeBSD and <tt>afacli</tt> on linux Once you're in the CLI, open the container - either `open aac0` on fbsd or `open afa0` on linux - everything else is the same regardless of the OS. Here are the most common things we ever do: <pre>AAC0> container list /full Executing: container list /full=TRUE Num Total Oth Chunk Scsi Partition Creation System Label Type Size Ctr Size Usage B:ID:L Offset:Size State RO Lk Task Done% Ent Date Time Files ----- ------ ------ --- ------ ------- ------ ------------- ------- -- -- ------- ------ --- ------ -------- ------ 0 Mirror 33.9GB Open 0:01:0 64.0KB:33.9GB Normal 0 071002 05:39:32 /dev/aacd0 mirror0 0:00:0 64.0KB:33.9GB Normal 1 071002 05:39:32 1 Mirror 33.9GB Open 0:02:0 64.0KB:33.9GB Normal 0 071002 05:39:50 /dev/aacd1 mirror1 0:03:0 64.0KB:33.9GB Normal 1 071002 05:39:50 AAC0> disk list /full Executing: disk list /full=TRUE B:ID:L Device Type Removable media Vendor-ID Product-ID Rev Blocks Bytes/Block Usage Shared Rate ------ -------------- --------------- --------- ---------------- ----- --------- ----------- ---------------- ------ ---- 0:00:0 Disk N FUJITSU MAJ3364MC 3702 71390320 512 Initialized NO 160 0:01:0 Disk N FUJITSU MAJ3364MC 3702 71390320 512 Initialized NO 160 0:02:0 Disk N FUJITSU MAJ3364MC 3702 71390320 512 Initialized NO 160 0:03:0 Disk N FUJITSU MAJ3364MC 3702 71390320 512 Initialized NO 160 AAC0> AAC0> disk show smart Executing: disk show smart Smart Method of Enable Capable Informational Exception Performance Error B:ID:L Device Exceptions(MRIE) Control Enabled Count ------ ------- ---------------- --------- ----------- ------ 0:00:0 Y 6 Y N 0 0:01:0 Y 6 Y N 0 0:02:0 Y 6 Y N 0 0:03:0 Y 6 Y N 0 0:06:0 N AAC0> task list Executing: task list Controller Tasks TaskId Function Done% Container State Specific1 Specific2 ------ -------- ------- --------- ----- --------- --------- No tasks currently running on controller AAC0> controller details Executing: controller details Controller Information ---------------------- Remote Computer: S Device Name: S Controller Type: PERC 3/Si Access Mode: READ-WRITE Controller Serial Number: Last Six Digits = 8C01D0 Number of Buses: 1 Devices per Bus: 15 Controller CPU: i960 R series Controller CPU Speed: 100 Mhz Controller Memory: 64 Mbytes Battery State: Not Present Component Revisions ------------------- CLI: 1.0-0 (Build #5263) API: 1.0-0 (Build #5263) Miniport Driver: 2.7-1 (Build #3571) Controller Software: 2.7-1 (Build #3571) Controller BIOS: 2.7-1 (Build #3571) Controller Firmware: (Build #3571) AAC0> disk show smart AAC0> task list</pre> type exit to leave it. All the mirrors you are getting are healthy - so even if a drive fails you should be ok. If an unexplained system crash occurs, after restarting all the systems you should check the mirrors with `container list /f` to see if one has degraded (one disk has died). Once in a while a mirror degrading will cause the system to crash - but not usually - usually it is just silent. == Creating/removing mirror == === via CLI === First, you have to make sure the new disk(s) is/are wiped clean – '''a disk with containers from another system will crash the system when inserted'''. You can do this initializing the drive(s) in another system. Get into the CLI, rescan the controller: AAC0> controller rescan to get it to find the newly inserted disks <pre>AAC0> container list Executing: container list Num Total Oth Chunk Scsi Partition Label Type Size Ctr Size Usage B:ID:L Offset:Size ----- ------ ------ --- ------ ------- ------ ------------- 0 Mirror 68.3GB Open 0:00:0 64.0KB:68.3GB /dev/sda 0:01:0 64.0KB:68.3GB AAC0> disk list Executing: disk list B:ID:L Device Type Blocks Bytes/Block Usage Shared Rate ------ -------------- --------- ----------- ---------------- ------ ---- 0:00:0 Disk 143374738 512 Initialized NO 160 0:01:0 Disk 143374738 512 Initialized NO 160 0:02:0 Disk 143374738 512 Initialized NO 160 0:03:0 Disk 143374738 512 Initialized NO 160</pre> if the disks aren’t initialized: <pre>AAC0> disk initialize (0,2,0) AAC0> disk initialize (0,3,0) AAC0> disk show space Executing: disk show space Scsi B:ID:L Usage Size ----------- ---------- ------------- 0:00:0 Container 64.0KB:68.3GB 0:00:0 Free 68.3GB:7.50KB 0:01:0 Container 64.0KB:68.3GB 0:01:0 Free 68.3GB:7.50KB </pre> Finally, create the mirror: <pre>AAC0> container create volume /label=MIRROR1 ((0,2,0)) Executing: container create volume /label="MIRROR1" (BUS=0,ID=2,LUN=0) Container 1 created at /dev/sdb AAC0> container list Executing: container list Num Total Oth Chunk Scsi Partition Label Type Size Ctr Size Usage B:ID:L Offset:Size ----- ------ ------ --- ------ ------- ------ ------------- 0 Mirror 68.3GB Open 0:00:0 64.0KB:68.3GB /dev/sda 0:01:0 64.0KB:68.3GB 1 Volume 8.47GB Valid 0:02:0 64.0KB:8.47GB /dev/sdb MIRROR1 Create volume on 1 of to disks to be in mirror (fdisk, format), Then to mirror: AAC0> container create mirror 1 (0,3,0) Executing: container create mirror 1 (BUS=0,ID=3,LUN=0) AAC0> container list Executing: container list Num Total Oth Chunk Scsi Partition Label Type Size Ctr Size Usage B:ID:L Offset:Size ----- ------ ------ --- ------ ------- ------ ------------- 0 Mirror 68.3GB Open 0:00:0 64.0KB:68.3GB /dev/sda 0:01:0 64.0KB:68.3GB 1 Mirror 8.47GB Valid 0:02:0 64.0KB:8.47GB /dev/sdb MIRROR1 0:03:0 64.0KB:8.47GB AAC0> task list Executing: task list Controller Tasks TaskId Function Done% Container State Specific1 Specific2 ------ -------- ------- --------- ----- --------- --------- 100 Bld/Vfy 1.3% 1 RUN 00000000 00000000</pre> ---- To remove: MAKE SURE DRIVE NOT MOUNTED FIRST! Example: delete 2nd mirror installed in slots 2-3 <pre>AAC0> container delete 1 Executing: container delete 1 AAC0> container list Executing: container list Num Total Oth Chunk Scsi Partition Label Type Size Ctr Size Usage B:ID:L Offset:Size ----- ------ ------ --- ------ ------- ------ ------------- 0 Mirror 68.3GB Open 0:00:0 64.0KB:68.3GB /dev/sda 0:01:0 64.0KB:68.3GB AAC0> enclosure prepare slot 0 2 Executing: enclosure prepare slot 0 2 AAC0> enclosure prepare slot 0 3 Executing: enclosure prepare slot 0 3</pre> You may now remove drives ---- To create mirrors from a volume (see next section where we add drives as volume) <pre>CLI > open aac0 AAC0> disk list Executing: disk list C:ID:L Device Type Blocks Bytes/Block Usage Shared Rate ------ -------------- --------- ----------- ---------------- ------ ---- 0:00:0 Disk 143374738 512 Initialized NO 160 0:01:0 Disk 143374738 512 Initialized NO 40 0:02:0 Disk 286749488 512 Initialized NO 320 0:03:0 Disk 286749488 512 Initialized NO 320 AAC0> container list Executing: container list Num Total Oth Stripe Scsi Partition Label Type Size Ctr Size Usage C:ID:L Offset:Size ----- ------ ------ --- ------ ------- ------ ------------- 0 Volume 68.3GB Open 0:00:0 64.0KB:68.3GB /dev/aacd0 MIRROR0 1 Volume 136GB Open 0:02:0 64.0KB: 136GB /dev/aacd1 MIRROR1 AAC0> container create mirror 1 (0,3,0) Executing: container create mirror 1 (BUS=0,ID=3,LUN=0) AAC0> container list Executing: container list Num Total Oth Stripe Scsi Partition Label Type Size Ctr Size Usage C:ID:L Offset:Size ----- ------ ------ --- ------ ------- ------ ------------- 0 Volume 68.3GB Open 0:00:0 64.0KB:68.3GB /dev/aacd0 MIRROR0 1 Mirror 136GB Open 0:02:0 64.0KB: 136GB /dev/aacd1 MIRROR1 0:03:0 64.0KB: 136GB AAC0> task list Executing: task list Controller Tasks TaskId Function Done% Container State Specific1 Specific2 ------ -------- ------- --------- ----- --------- --------- 100 Bld/Vfy 6.5% 1 RUN 00000000 00000000</pre> If this happens: <pre>AAC0> container create mirror 1 (0:3:0) Executing: container create mirror 1 (BUS=0,ID=3,LUN=0) Command Error: <The controller was not able to mirror the specified container.> AAC0> disk initialize /always (0,3,0) Executing: disk initialize /always=TRUE (BUS=0,ID=3,LUN=0)</pre> Then: <pre>AAC0> container create mirror 1 (0:3:0) Executing: container create mirror 1 (BUS=0,ID=3,LUN=0)</pre> In the case of a drive that won't remirror due to a Mirror Failover Container 0 no failover assigned: container set failover 0 (0,0,0) where 0,0,0 is the new empty drive === via BIOS configuration utility === Press Ctrl-A to enter the configuration utility while the Adaptec BIOS is booting. <pre>Container configuration utility-> Initialize drives->(ins to select all drives you wish to use- DO NOT CHOOSE DRIVES IN USE IN EXISTING ARRAY) Create container-> (Select the newly-initialized drives) Container type: RAID1 Container label: MIRROR0 (confirm correct size) Read caching: Y Write caching: enable when protected (done) </pre> Repeat the process if you're building another mirror. Call it MIRROR1 Procedure for setting two, ½ mirrors (two volumes) which you can later convert to two mirrors: <pre>Create array-> (Select 1st drive) (enter) Container type: Volume Container label: MIRROR0 (confirm correct size) Read caching: Y Write caching: enable always (answer y to both warnings) (done) Create array-> (Select 3rd drive) Container type: Volume Container label: MIRROR1 (confirm correct size) Read caching: Y Write caching: enable always (answer y to both warnings) (done) (exit util)</pre> ---- To remove a drive: Go to the "Manage Arrays" screen, move the cursor over the mirror you want to delete and press the delete key. == Remove a live drive == Remove, via CLI: <pre>AAC0> container list Executing: container list Num Total Oth Chunk Scsi Partition Label Type Size Ctr Size Usage B:ID:L Offset:Size ----- ------ ------ --- ------ ------- ------ ------------- 0 Mirror 68.3GB Open 0:00:0 64.0KB:68.3GB /dev/aacd0 0:01:0 64.0KB:68.3GB AAC0> container unmirror 0 AAC0> container list Executing: container list Num Total Oth Stripe Scsi Partition Label Type Size Ctr Size Usage C:ID:L Offset:Size ----- ------ ------ --- ------ ------- ------ ------------- 0 Volume 68.3GB Open 0:00:0 64.0KB:68.3GB /dev/aacd0 MIRROR0 AAC0> enclosure prepare slot 0 1</pre> (remove the drive) ---- To replace drive: put in new drive, run <tt>controller rescan</tt>, if necessary <pre>AAC0> container create mirror 0 1 AAC0> task list (scrubbing) </pre> Note: if you unmirror an unhealthy mirror, it will leave the healthy one there, otherwise it will remove (whichever is listed as) the 2nd drive in the mirror. == Replace a dead ("missing") drive == <pre>AAC0> container list Executing: container list Num Total Oth Chunk Scsi Partition Label Type Size Ctr Size Usage B:ID:L Offset:Size ----- ------ ------ --- ------ ------- ------ ------------- 0 Mirror 68.3GB Open 0:00:0 64.0KB:68.3GB /dev/aacd0 0:01:0 64.0KB:68.3GB 1 Mirror 68.3GB Open 0:02:0 64.0KB:68.3GB /dev/aacd1 MIRROR1 ?:??:? - Missing – (put in new drive)</pre> No tasks showing up after 2min? Rescan: <pre>AAC0> controller rescan AAC0> task list Executing: task list Controller Tasks TaskId Function Done% Container State Specific1 Specific2 ------ -------- ------- --------- ----- --------- --------- 100 Rebuild 0.0% 1 RUN 00000000 00000000 AAC0></pre> To replace failed drive, you can prepare slot and put in new (replacement) drives on dells and it should start scrubbing. In case it doesn’t start scrubbing (there is a delay, so wait 2min), AAC0> container set failover (scsi id of new disk) Or if that fails AAC0> container unmirror 1 That will demote it to a volume, Then container create, etc… == Enabling/disabling write cache == Write cache should ideally only be enabled when there is a backup battery present. Write cache speeds up I/O performance by allowing the RAID card to write data to a buffer before committing it to disk. This data lives in RAM. If power is lost and there is no backup battery, that data is lost. However, you can force write cache to be on regardless of whether a battery is present. Turning write cache on: <pre>AFA0> container show cache 1 Executing: container show cache 1 Global Container Read Cache Size : 0 Global Container Write Cache Size : 51380224 Read Cache Setting : ENABLE Write Cache Setting : DISABLE Write Cache Status : Inactive, battery not present AFA0> container set cache /unprotected=1 /write_cache_enable=1 0 Executing: container set cache /unprotected=TRUE /write_cache_enable=TRUE 0 AFA0> container set cache /unprotected=1 /write_cache_enable=1 1 Executing: container set cache /unprotected=TRUE /write_cache_enable=TRUE 1 AFA0> container show cache 0 Executing: container show cache 0 Global Container Read Cache Size : 483328 Global Container Write Cache Size : 51380224 Read Cache Setting : ENABLE Write Cache Setting : ENABLE ALWAYS Write Cache Status : Active, not protected, battery not present AFA0> container show cache 1 Executing: container show cache 1 Global Container Read Cache Size : 483328 Global Container Write Cache Size : 51380224 Read Cache Setting : ENABLE Write Cache Setting : ENABLE ALWAYS Write Cache Status : Active, not protected, battery not present</pre> Shutting off write cache: <pre>FASTCMD> open afa0 Executing: open "afa0" AFA0> container set cache 0 /write_cache_enable=0 Too many parameters AFA0> container set cache /write_cache_enable=0 0 Executing: container set cache /write_cache_enable=FALSE 0 AFA0> container set cache /write_cache_enable=0 1 Executing: container set cache /write_cache_enable=FALSE 1 AFA0> container show cache 0 Executing: container show cache 0 Global Container Read Cache Size : 483328 Global Container Write Cache Size : 51380224 Read Cache Setting : ENABLE Write Cache Setting : DISABLE Write Cache Status : Inactive, battery not present AFA0> container show cache 1 Executing: container show cache 1 Global Container Read Cache Size : 483328 Global Container Write Cache Size : 51380224 Read Cache Setting : ENABLE Write Cache Setting : DISABLE Write Cache Status : Inactive, battery not present AFA0> exit Executing: exit</pre> = LSI/MegaRaid = == Notes == Unlike the Adaptec RAID cards, the LSI cards store two copies of the configuration meta-data. Once copy is in NVRAM on the cards itself, the other copy is on the disks. If there is configuration data in NVRAM it will be used, even if the data on the disks is different. What all this means is that if a card ever needs to be swapped out, you have to make sure the NVRAM is cleared before installing the new card. The clear config option in the menus doesn’t work. In order to clear the NVRAM choose new config and then don’t create any arrays. Then save the new config. Once the config is cleared you can install the card in an existing system. Once installed, you have to go to the view/add config menu and make sure the arrays are there, then save the config. Another thing to be aware of is that once you create an array, there’s no way to delete it without doing an entirely new config. You can get around this by taking note of the current config, then do a new config with only the arrays you want, and save the config. DO NOT initialize the arrays after saving the new config or you’ll lose all the data on that array. I did something weird where I cleared the config, then re-created the config, all without rebooting, and when I exited after re-creating, the system was fine (despite giving lots of errors after the config was cleared). Technical support<br> 8:30am to 5:30pm EST<br> 800-633-4545 == CLI notes == There are 2 shell-based options to manage the RAID card: <tt>megamgr</tt> and <tt>megarc</tt> megamgr is a curses-based program which closely mimics the BIOS config utility. To run it, you must be in the directory where it's installed: cd /usr/local/sbin/; megamgr When working in megamgr and entering an ESC, you'll notice nothing happens immediately. After pressing the ESC key, you must press another key before the ESC takes effect. So for instance, after pressing ESC press the up or down arrow (which is inocuous since it won't cause any undesired changes). There is also a CLI-based program called <tt>megarc</tt> == Common CLI commands == All commands must be run from /usr/local/sbin/ Info/status: megarc -ldInfo -a0 -Lall take drive 3 offline: megarc "-physOff -a0 pd[0:3]" Rebuild drive 3: megarc "-doRbld -RbldArray[0:3] -a0 -ShowProg" View rebuild progress: megarc -showRbld -a0 == Creating/removing mirror == === via CLI === This will add a new mirror from drives located in the 2nd and 3rd slots: <pre>cd /usr/local/sbin/; megarc -addCfg -a0 -R1[0:2,0:3] WB CIO echo "scsi add-single-device 0 0 1 0" > /proc/scsi/scsi</pre> To delete that mirror: <pre>cd /usr/local/sbin/; megarc -delLD -a0 -L1 echo "scsi remove-single-device 0 0 1 0" > /proc/scsi/scsi</pre> This will add a new mirror from drives located in the 4th and 5th slots: <pre>cd /usr/local/sbin/; megarc -addCfg -a0 -R1[0:4,0:5] WT CIO echo "scsi add-single-device 0 0 2 0" > /proc/scsi/scsi</pre> To delete that mirror: <pre>cd /usr/local/sbin/; megarc -delLD -a0 -L2 echo "scsi remove-single-device 0 0 2 0" > /proc/scsi/scsi</pre> None of the above is data destructive. You can remove a mirror and re-create it and all the data will still be there...as long as you don't initialize the drive. === via BIOS configuration utility or Megamgr === Totally new configuration: <pre>Configure, New configuration (space 2x to select 2 drives) Enter to end selection F10 to configure Space to select Span-1 F10 to configure RAID=1 StripeSize=64k Write policy=WRBACK Read policy=NORMAL Cache Policy=Cached IO ESC, accept</pre> NOTE: when adding an additional drive, choose view/add not New Configuration – otherwise you'll wipe out the existing mirror/config. Adding to existing configuration: <pre>Configure->View/Add Configuration-> ([space] on the 2 drives)->[enter] F10 [space] Span-1 F10 Advanced->Cache Policy=CachedIO Accept Save Exit </pre> If this was added to a running machine, to get the device to show up (in linux) run: echo "scsi add-single-device 0 0 1 0" > /proc/scsi/scsi Make sure SCSI transfer rate is 320M under objects, channel, channel 1 Go to objects, adapter Make sure alarm is enabled ---- To remove a mirror: Objects->Logical Drive->(move cursor to drive)->F5 If on a running server (assuming you unmounted it first): echo "scsi remove-single-device 0 0 1 0" > /proc/scsi/scsi ---- Create mirror from volume: #Remove drive (megarc -delLD -a0 -L1) #Then re-add config (I did it with megamgr) #Then fail the new drive and rebuild it ---- This will add a new '''VOLUME''' from drive located in the 2nd slot: <pre>Configure->View/Add Configuration-> ([space] on the #2 drive)->[enter] F10 [space] Span-1 F10 Advanced->Cache Policy=CachedIO Accept Initialize->Logical Drive 2->[space]->F10 Save Exit</pre> If on a running linux system, to add the device: echo "scsi add-single-device 0 0 1 0" > /proc/scsi/scsi == Move drives from one LSI card to another == MAKE SURE ALL DRIVES ARE REMOVED BEFORE COMPLETING!! On new card: <pre>Configure, New configuration F10 Save Power down Insert drives Reboot</pre> DO NOT INITIALIZE DRIVES! Alternate (if the drives are still in when you clear config) You can re-create mirrors without destroying data as long as created the same and drives are not initialized. <pre>Configure, Clear configuration Easy configuration (create as normal) DON’T INITIALIZE Reboot</pre> == Converting broken mirror to volume back to mirror == Assumes drive #1 is dead/out # convert #0 to volume. New config, #0 = volume, #2,#3 = R1 # put cleaned drive into slot #1. All drives out- put drive in #1, new conf, #1 volume, init, delete LD ## THIS DOESN’T WORK: new config, #0,#1 = R1, #2,#3 = R1. After reboot end up in grub shell – RAID card confused about which is boot drive ## new config, #0,#1 = R1, #2,#3 = R1. Power down. Remove #1. Power up then insert #1 = mirror 0 rebuilds == Move 2 mirrors from 2 different machines into 1 == Premise: Orig machine has 2 R1’s, taking mirror #1 from other machine and replacing mirror #1 on target machine # power off target machine, remove #1 mirror drives. Unseat #0 mirror # power up. Clear config. New config. Reboot to confirm # power down. Reinsert mirror #0 and mirror #1 from other machine # power up. Vew/add config, select DISK config, save reboot # will see original #0 mirror and #1 mirror from other machine = Perc 4/5/6 = == Notes == Perc cards are installed in all our modern Dell servers (2850/2950) in various flavors. All are based on LSI architecture and therefore use similar commands/utilities. The BIOS config utility is different however. Perc5 (and 6?) cards use the <tt>megacli</tt> utility, whereas Perc4 (2850's) use the <tt>megarc</tt> utility. It's possible to take a drive from a live mirror on a 5/i card and move it to replace a dead mirror on a 6/i card, the 6/i card will recognize the newly-inserted drive as a spare and rebuild using the remaining good drive as the master. == Common CLI commands (megacli) == Reference:<br> http://tools.rapidsoft.de/perc/<br> http://linux.dell.com/files/whitepapers/solaris/Managing_PERC6_0714.pdf View rebuild progress for drive 1: megacli -PDRbld -ProgDsply -PhysDrv[1:1] -aALL View rebuild ETA for drive 1: megacli -PDRbld -ShowProg -PhysDrv[1:1] -aALL View rebuild rate megacli -AdpGetProp RebuildRate -Aall View info on logical drives and physical drives megacli -LdPdInfo -aALL Set drive 1 offline megacli -PDOffline -PhysDrv[1:1] –aAll Set drive 1 online megacli -PDOnline -PhysDrv[1:1] –aAll View info on logical drives megacli -LDInfo -Lall -aAll Setup a new raid1 config from drives 0,1 (works on 2950/Perc5/i/linux) megacli -CfgLdAdd -r1[1:0,1:1] WB ADRA Cached -a0 Show adapter info megacli adapallinfo a0 view logical drive config info megacli cfgdsply a0 view physical drive info for all drives megacli pdlist a0 When you create a raid0 mirror you know it’s raid0 per the following in ldinfo: RAID Level: Primary-0, Secondary-0, RAID Level Qualifier-0 For raid1 it’s: RAID Level: Primary-1, Secondary-0, RAID Level Qualifier-0 Remove LD1 megacli cfglddel L1 a0 Init a disk megacli ldinit start L1 a0 Check consistency (jail7 freezes when trying to perform): megacli -LDCC -Start -L0 -a0 Watch consistency check in realtime: megacli -LDCC -ProgDsply -L0 -a0 See consistency check progress snapshot: megacli -LDCC -ShowProg -L0 -a0 List adapter serial number and firmware level megacli -AdpAllInfo -aALL Testing Battery megacli -AdpBbuCmd -aAll Enclosure info megacli -EncInfo -aALL Controller log megacli -FwTermLog -Dsply -aALL RAID array status megacli -LDInfo -Lall -aALL Patrol Read - The megacli tool has a 'patrol read' function which provides for regular media scanning. To show the current settings, megacli -AdpPR -Info -aALL To perform a manual scan, megacli -AdpPR -Start -aALL The progress can be monitored using the 'Info' option. megacli -AdpPR -Info -aALL Battery Monitoring megacli -AdpBbuCmd -aALL Logical Device Consistency Check megacli -LDCC -Start -LALL -aALL megacli -LDCC -ShowProg -LALL -aALL Power Supply Monitoring - The power supply failures are logged through the event log. megacli -AdpEventLog -GetEvents -f /tmp/event.log -a1 This writes a set of event entries into the file. RAID 0 to RAID 1 conversion for system disks Delete the second raid0 : megacli -CfgLdDel L1 -a0 Migrate to raid1 : megacli -LDRecon start r1 [Add PhysDrv[32:1]] L0 -a0 Check progression : megacli -LDRecon ShowProg L0 -a0 Check the logical drive information and note that it still displays as a 1-disk RAID0 but it also displays the reconfiguration in progress under “Ongoing Progresses” field. The VD configuration information changes after the Raid Level Migration has completed successfully. megacli -LDInfo -Lall -aALL Converting from RAID-5/Spare to RAID-6 megacli -PDHSP -Rmv -PhysDrv [32:5] -a0 megacli -CfgLdDel L1 -a0 megacli -CfgLdAdd -r6[32:2,32:3,32:4,32:5] -a0 == Creating/removing mirror == === Perc5: via CLI === This will add a new mirror from drives located in the 2nd and 3rd slots: megacli cfgldadd r1[1:2,1:3] wb adra cached a0 megacli ldinfo lall a0 (confirm it’s there, look for bg init, make take a few mins to show up) <pre>Adapter 0 -- Virtual Drive Information: Virtual Disk: 0 Name: RAID Level: Primary-1, Secondary-0, RAID Level Qualifier-0 Size:139392MB State: Optimal Stripe Size: 64kB Number Of Drives:2 Span Depth:1 Default Cache Policy: WriteBack ReadAdaptive Direct Current Cache Policy: WriteThrough ReadAdaptive Direct Access Policy: Read/Write Disk Cache Policy: Disk's Default Virtual Disk: 1 Name: RAID Level: Primary-1, Secondary-0, RAID Level Qualifier-0 Size:428672MB State: Optimal Stripe Size: 64kB Number Of Drives:2 Span Depth:1 Default Cache Policy: WriteBack ReadAdaptive Direct Current Cache Policy: WriteThrough ReadAdaptive Direct Access Policy: Read/Write Disk Cache Policy: Disk's Default Ongoing Progresses: Background Initialization: Completed 1%, Taken 0 min.</pre> If you have DRAC virtual media enabled, the scsi devices look like: <pre>cat /proc/scsi/scsi Attached devices: Host: scsi0 Channel: 00 Id: 08 Lun: 00 Vendor: DP Model: BACKPLANE Rev: 1.00 Type: Enclosure ANSI SCSI revision: 05 Host: scsi0 Channel: 02 Id: 00 Lun: 00 Vendor: DELL Model: PERC 5/i Rev: 1.00 Type: Direct-Access ANSI SCSI revision: 05 Host: scsi1 Channel: 00 Id: 00 Lun: 00 Vendor: Dell Model: Virtual CDROM Rev: 123 Type: CD-ROM ANSI SCSI revision: 02 Host: scsi2 Channel: 00 Id: 00 Lun: 00 Vendor: Dell Model: Virtual Floppy Rev: 123 Type: Direct-Access ANSI SCSI revision: 02</pre> And the new drive will show up as /dev/sdc So, turn off virtual media first, to do this: <pre>cat /proc/scsi/scsi Attached devices: Host: scsi0 Channel: 00 Id: 08 Lun: 00 Vendor: DP Model: BACKPLANE Rev: 1.00 Type: Enclosure ANSI SCSI revision: 05 Host: scsi0 Channel: 02 Id: 00 Lun: 00 Vendor: DELL Model: PERC 5/i Rev: 1.00 Type: Direct-Access ANSI SCSI revision: 05 Host: scsi0 Channel: 02 Id: 01 Lun: 00 Vendor: DELL Model: PERC 5/i Rev: 1.00 Type: Direct-Access ANSI SCSI revision: 05 Host: scsi1 Channel: 00 Id: 00 Lun: 00 Vendor: Dell Model: Virtual CDROM Rev: 123 Type: CD-ROM ANSI SCSI revision: 02 Host: scsi2 Channel: 00 Id: 00 Lun: 00 Vendor: Dell Model: Virtual Floppy Rev: 123 Type: Direct-Access ANSI SCSI revision: 02 [root@virt18 ~]# echo "scsi remove-single-device 1 0 0 0" > /proc/scsi/scsi [root@virt18 ~]# cat /proc/scsi/scsi Attached devices: Host: scsi0 Channel: 00 Id: 08 Lun: 00 Vendor: DP Model: BACKPLANE Rev: 1.00 Type: Enclosure ANSI SCSI revision: 05 Host: scsi0 Channel: 02 Id: 00 Lun: 00 Vendor: DELL Model: PERC 5/i Rev: 1.00 Type: Direct-Access ANSI SCSI revision: 05 Host: scsi0 Channel: 02 Id: 01 Lun: 00 Vendor: DELL Model: PERC 5/i Rev: 1.00 Type: Direct-Access ANSI SCSI revision: 05 Host: scsi2 Channel: 00 Id: 00 Lun: 00 Vendor: Dell Model: Virtual Floppy Rev: 123 Type: Direct-Access ANSI SCSI revision: 02 [root@virt18 ~]# echo "scsi remove-single-device 2 0 0 0" > /proc/scsi/scsi [root@virt18 ~]# cat /proc/scsi/scsi Attached devices: Host: scsi0 Channel: 00 Id: 08 Lun: 00 Vendor: DP Model: BACKPLANE Rev: 1.00 Type: Enclosure ANSI SCSI revision: 05 Host: scsi0 Channel: 02 Id: 00 Lun: 00 Vendor: DELL Model: PERC 5/i Rev: 1.00 Type: Direct-Access ANSI SCSI revision: 05 Host: scsi0 Channel: 02 Id: 01 Lun: 00 Vendor: DELL Model: PERC 5/i Rev: 1.00 Type: Direct-Access ANSI SCSI revision: 05 [root@virt18 ~]# echo "scsi add-single-device 0 2 2 0" > /proc/scsi/scsi Vendor: DELL Model: PERC 5/i Rev: 1.00 Type: Direct-Access ANSI SCSI revision: 05 SCSI device sdc: 142082048 512-byte hdwr sectors (72746 MB) sdc: asking for cache data failed sdc: assuming drive cache: write through SCSI device sdc: 142082048 512-byte hdwr sectors (72746 MB) sdc: asking for cache data failed sdc: assuming drive cache: write through Attached scsi disk sdc at scsi0, channel 2, id 2, lun 0 [root@virt18 ~]# cat /proc/scsi/scsi Attached devices: Host: scsi0 Channel: 00 Id: 08 Lun: 00 Vendor: DP Model: BACKPLANE Rev: 1.00 Type: Enclosure ANSI SCSI revision: 05 Host: scsi0 Channel: 02 Id: 00 Lun: 00 Vendor: DELL Model: PERC 5/i Rev: 1.00 Type: Direct-Access ANSI SCSI revision: 05 Host: scsi0 Channel: 02 Id: 01 Lun: 00 Vendor: DELL Model: PERC 5/i Rev: 1.00 Type: Direct-Access ANSI SCSI revision: 05 Host: scsi0 Channel: 02 Id: 02 Lun: 00 Vendor: DELL Model: PERC 5/i Rev: 1.00 Type: Direct-Access ANSI SCSI revision: 05 [root@virt18 ~]# fdisk -l Disk /dev/sda: 146.1 GB, 146163105792 bytes 255 heads, 63 sectors/track, 17769 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Device Boot Start End Blocks Id System /dev/sda1 * 1 392 3148708+ 83 Linux /dev/sda2 393 914 4192965 82 Linux swap /dev/sda3 915 17769 135387787+ 83 Linux Disk /dev/sdb: 146.1 GB, 146163105792 bytes 255 heads, 63 sectors/track, 17769 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Device Boot Start End Blocks Id System /dev/sdb1 * 1 522 4192933+ 82 Linux swap /dev/sdb2 523 17769 138536527+ 83 Linux Disk /dev/sdc: 72.7 GB, 72746008576 bytes 255 heads, 63 sectors/track, 8844 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Disk /dev/sdc doesn't contain a valid partition table [root@virt18 ~]# echo "scsi add-single-device 0 0 3 0" > /proc/scsi/scsi [root@virt18 ~]# echo "scsi add-single-device 1 0 0 0" > /proc/scsi/scsi Vendor: Dell Model: Virtual CDROM Rev: 123 Type: CD-ROM ANSI SCSI revision: 02 sr0: scsi-1 drive [root@virt18 ~]# echo "scsi add-single-device 2 0 0 0" > /proc/scsi/scsi Vendor: Dell Model: Virtual Floppy Rev: 123 Type: Direct-Access ANSI SCSI revision: 02 Attached scsi removable disk sdd at scsi2, channel 0, id 0, lun 0 [root@virt18 ~]# cat /proc/scsi/scsi Attached devices: Host: scsi0 Channel: 00 Id: 08 Lun: 00 Vendor: DP Model: BACKPLANE Rev: 1.00 Type: Enclosure ANSI SCSI revision: 05 Host: scsi0 Channel: 02 Id: 00 Lun: 00 Vendor: DELL Model: PERC 5/i Rev: 1.00 Type: Direct-Access ANSI SCSI revision: 05 Host: scsi0 Channel: 02 Id: 01 Lun: 00 Vendor: DELL Model: PERC 5/i Rev: 1.00 Type: Direct-Access ANSI SCSI revision: 05 Host: scsi0 Channel: 02 Id: 02 Lun: 00 Vendor: DELL Model: PERC 5/i Rev: 1.00 Type: Direct-Access ANSI SCSI revision: 05 Host: scsi1 Channel: 00 Id: 00 Lun: 00 Vendor: Dell Model: Virtual CDROM Rev: 123 Type: CD-ROM ANSI SCSI revision: 02 Host: scsi2 Channel: 00 Id: 00 Lun: 00 Vendor: Dell Model: Virtual Floppy Rev: 123 Type: Direct-Access ANSI SCSI revision: 02</pre> Then we have: <pre>cat /proc/scsi/scsi Attached devices: Host: scsi0 Channel: 00 Id: 08 Lun: 00 Vendor: DP Model: BACKPLANE Rev: 1.00 Type: Enclosure ANSI SCSI revision: 05 Host: scsi0 Channel: 02 Id: 00 Lun: 00 Vendor: DELL Model: PERC 5/i Rev: 1.00 Type: Direct-Access ANSI SCSI revision: 05</pre> And we do: echo "scsi add-single-device 0 2 1 0" > /proc/scsi/scsi And it’d show up as /dev/sdb To watch the bg init: megacli ldbi progdsply l1 a0 ---- To delete that mirror: megacli cfglddel L1 a0 echo "scsi remove-single-device 0 2 1 0" > /proc/scsi/scsi None of the above is data destructive On a Dell5/i (2950) on linux: <pre>[root@virt18 ~]# cat /proc/scsi/scsi Attached devices: Host: scsi0 Channel: 00 Id: 08 Lun: 00 Vendor: DP Model: BACKPLANE Rev: 1.00 Type: Enclosure ANSI SCSI revision: 05 Host: scsi0 Channel: 02 Id: 00 Lun: 00 Vendor: DELL Model: PERC 5/i Rev: 1.00 Type: Direct-Access ANSI SCSI revision: 05 Host: scsi0 Channel: 02 Id: 01 Lun: 00 Vendor: DELL Model: PERC 5/i Rev: 1.00 Type: Direct-Access ANSI SCSI revision: 05 Host: scsi1 Channel: 00 Id: 00 Lun: 00 Vendor: Dell Model: Virtual CDROM Rev: 123 Type: CD-ROM ANSI SCSI revision: 02 Host: scsi2 Channel: 00 Id: 00 Lun: 00 Vendor: Dell Model: Virtual Floppy Rev: 123 Type: Direct-Access ANSI SCSI revision: 02 [root@virt18 ~]# echo "scsi remove-single-device 2 2 0" > /proc/scsi/scsi</pre> And to add that drive back in: <pre>[root@virt18 ~]# echo "scsi add-single-device 0 2 2 0" > /proc/scsi/scsi [root@virt18 ~]# cat /proc/scsi/scsi Attached devices: Host: scsi0 Channel: 00 Id: 08 Lun: 00 Vendor: DP Model: BACKPLANE Rev: 1.00 Type: Enclosure ANSI SCSI revision: 05 Host: scsi0 Channel: 02 Id: 00 Lun: 00 Vendor: DELL Model: PERC 5/i Rev: 1.00 Type: Direct-Access ANSI SCSI revision: 05 Host: scsi0 Channel: 02 Id: 01 Lun: 00 Vendor: DELL Model: PERC 5/i Rev: 1.00 Type: Direct-Access ANSI SCSI revision: 05 Host: scsi1 Channel: 00 Id: 00 Lun: 00 Vendor: Dell Model: Virtual CDROM Rev: 123 Type: CD-ROM ANSI SCSI revision: 02 Host: scsi2 Channel: 00 Id: 00 Lun: 00 Vendor: Dell Model: Virtual Floppy Rev: 123 Type: Direct-Access ANSI SCSI revision: 02 Host: scsi0 Channel: 02 Id: 02 Lun: 00 Vendor: DELL Model: PERC 5/i Rev: 1.00 Type: Direct-Access ANSI SCSI revision: 05</pre> === Perc4: via CLI === Make sure (to allow addcfg): dev.amr.0.allow_volume_configure: 1 List drives <pre>megarc -LogPhysInfo -a0 Finding Devices On Each MegaRAID Adapter... Scanning Ha 0, Chnl 1 Target 15 Adapter No 0 : Total Logical Drive(s)- 1 Logical drive 0: RaidLevel 1 Physical Drive Information Channel 0 139900MB drive scsi ID 0 CoerSZ: 286515200(Sectors) 139900(MB) RawSZ: 286749487(Sectors) 139900MB drive scsi ID 1 CoerSZ: 286515200(Sectors) 139900(MB) RawSZ: 286749487(Sectors) 69880MB drive scsi ID 2 CoerSZ: 143114240(Sectors) 69880(MB) RawSZ: 143374649(Sectors) 69880MB drive scsi ID 3 CoerSZ: 143114240(Sectors) 69880(MB) RawSZ: 143374649(Sectors)</pre> Before: <pre>megarc -ldInfo -a0 -lall Finding Devices On Each MegaRAID Adapter... Scanning Ha 0, Chnl 1 Target 15 *******Information Of Logical Drive 0******* Logical Drive : 0( Adapter: 0 ): Status: OPTIMAL --------------------------------------------------- SpanDepth :01 RaidLevel: 1 RdAhead : Adaptive Cache: CachedIo StripSz :064KB Stripes : 2 WrPolicy: WriteBack Logical Drive 0 : SpanLevel_0 Disks Chnl Target StartBlock Blocks Physical Target Status ---- ------ ---------- ------ ---------------------- 0 00 0x00000000 0x1113e000 ONLINE 0 01 0x00000000 0x1113e000 ONLINE</pre> After:<br> This will add a new mirror from drives located in the 2nd and 3rd slots (make sure in sh): <pre>sh megarc -addCfg -a0 -R1[0:2,0:3] WB RAA CIO -strpsz64 megacli ldinfo lall a0 *******Information Of Logical Drive 0******* Logical Drive : 0( Adapter: 0 ): Status: OPTIMAL --------------------------------------------------- SpanDepth :01 RaidLevel: 1 RdAhead : Adaptive Cache: CachedIo StripSz :064KB Stripes : 2 WrPolicy: WriteBack Logical Drive 0 : SpanLevel_0 Disks Chnl Target StartBlock Blocks Physical Target Status ---- ------ ---------- ------ ---------------------- 0 00 0x00000000 0x1113e000 ONLINE 0 01 0x00000000 0x1113e000 ONLINE *******Information Of Logical Drive 1******* Logical Drive : 1( Adapter: 0 ): Status: OPTIMAL --------------------------------------------------- SpanDepth :01 RaidLevel: 1 RdAhead : Adaptive Cache: CachedIo StripSz :064KB Stripes : 2 WrPolicy: WriteBack Logical Drive 1 : SpanLevel_0 Disks Chnl Target StartBlock Blocks Physical Target Status ---- ------ ---------- ------ ---------------------- 0 02 0x00000000 0x0887c000 ONLINE 0 03 0x00000000 0x0887c000 ONLINE</pre> Initialize the drive: <pre>megarc -initLd -a0 -l1 ********************************************************************** Monitoring Initialization on Adapter No 0 ********************************************************************** LD00: LD01:000% LD02: LD03: LD04: LD05: LD06: LD07: LD08: LD09: LD10: LD11: LD12: LD13: LD14: LD15: LD16: LD17: LD18: LD19: LD20: LD21: LD22: LD23: LD24: LD25: LD26: LD27: LD28: LD29: LD30: LD31: LD32: LD33: LD34: LD35: LD36: LD37: LD38: LD39: ********************************************************************** Monitoring Initialization on Adapter No 0 ********************************************************************** LD00: LD01:100% LD02: LD03: LD04: LD05: LD06: LD07: LD08: LD09: LD10: LD11: LD12: LD13: LD14: LD15: LD16: LD17: LD18: LD19: LD20: LD21: LD22: LD23: LD24: LD25: LD26: LD27: LD28: LD29: LD30: LD31: LD32: LD33: LD34: LD35: LD36: LD37: LD38: LD39:</pre> (back to shell, continues in bg) to see: megarc -ViewBICCProg -a0 Can't delete that mirror: <pre>megarc -DelLd -a0 -l1 ********************************************************************** MEGARC MegaRAID Configuration Utility(FreeBSD)-1.04(03-02-2005) By LSI Logic Corp.,USA ********************************************************************** [Note: For SATA-2, 4 and 6 channel controllers, please specify Ch=0 Id=0..15 for specifying physical drive(Ch=channel, Id=Target)] Type ? as command line arg for help Finding Devices On Each MegaRAID Adapter... Scanning Ha 0, Chnl 1 Target 15 Failed to delete logical drive 1 on adapter 0</pre> Check consistency: <pre>megarc -chkonLd -a0 -l0 ********************************************************************** Monitoring Check Consistency on Adapter No 0 ********************************************************************** LD00:000% LD01: LD02: LD03: LD04: LD05: LD06: LD07: LD08: LD09: LD10: LD11: LD12: LD13: LD14: LD15: LD16: LD17: LD18: LD19: LD20: LD21: LD22: LD23: LD24: LD25: LD26: LD27: LD28: LD29: LD30: LD31: LD32: LD33: LD34: LD35: LD36: LD37: LD38: LD39:</pre> (spits out new screen every few sec) To watch again after cancelling: megarc -ViewBICCProg -a0 === via BIOS configuration utility === = 3ware = == Notes == To replace a failed mirror with the drive that failed, boot up with only the failed drive (to be safe, make sure the other drive boots fine), delete the mirror on the bad drive. Boot up again with both drives and select the failed (now empty/available drive) and the remaining degraded mirror and select to rebuild. Hit F8 on the way out to save/rebuild. *adding mirror(s) to a new/empty machine (with no os) *adding mirror(s) to a running machine *moving mirror(s) to a new/empty machine (with no os) *moving mirror(s) to a running machine *replacing dead/degraded mirror on running machine (please include pointers on which spare drives can be used as replacements*) *silencing alarms (a nod to castle staff) *please include documentation for both CLI and BIOS apps. where appropriate and where differences exist. *include instructions up to the point where the partition is mounted and usable (or recognized by os in the case of new install) *please include (separate) instructions for clearing off used drive so it can be reused in above procedures. Tech support: 800 840-6055 == CLI (9xxx) == === Replacing a failed drive === <pre> tw_cli /c0 show all Port Status Unit Size Blocks Serial --------------------------------------------------------------- p0 OK u0 1.82 TB 3907029168 WD-WCAVY0647904 p1 OK u0 1.82 TB 3907029168 WD-WCAVY0608298 p2 OK u1 1.82 TB 3907029168 WD-WCAVY0629856 p3 OK u1 1.82 TB 3907029168 WD-WCAVY0627316 p4 OK - 1.82 TB 3907029168 WD-WCAVY0564054 </pre> then you should run: <pre> tw_cli /c0 rescan tw_cli /c0 show all Port Status Unit Size Blocks Serial --------------------------------------------------------------- p0 OK u0 1.82 TB 3907029168 WD-WCAVY0647904 p1 OK u0 1.82 TB 3907029168 WD-WCAVY0608298 p2 OK u1 1.82 TB 3907029168 WD-WCAVY0629856 p3 OK u1 1.82 TB 3907029168 WD-WCAVY0627316 p4 DEGRADED u1 1.82 TB 3907029168 WD-WCAVY0564054 </pre> and the array should show that it is rebuilding BUT, if you attempt to rebuild with a command like that, and you get: Error: The following drive(s) cannot be used [4]. you need to remove it, then rescan, then rebuild: tw_cli /c0/p4 remove tw_cli /c0 rescan tw_cli /c0/u0 start rebuild disk=4 == Rebuild speed == Do the following to slow down a rebuild speed so as to improve performance (1=fast, 5=slow) tw_cli /c0 set rebuild=5 turn off storsave protection (set back to protect) tw_cli /c0/u0 set storsave=perform turn off write cache tw_cli /c0/u0 set cache=off == Hot Spare == To add a hot spare drive tw_cli /c0 add type=spare disk=XX == CLI (8xxx) == Reference guide: ftp://ftp.rackable.com/public/Technical%20Support/Pdf%20files/3Ware/7000_8000/CLI-UserGuide.pdf === Replacing a failed drive === If you don't remove the drive from a RAID-1 array before you replace it, the new drive comes up as a separate JBOD device. To fix this do <pre> tw_cli maint deleteunit c0 u1 tw_cli maint remove c0 u0 p1 (may not be needed) tw_cli maint rescan c0 tw_cli /c0 show all tw_cli maint rebuild c0 u0 p1 tw_cli /c0 show all </pre> After replacing the 2 dead drives, we see: <pre> tw_cli info c0 Controller: c0 ------------- Driver: 1.50.01.002 Model: 7500-8 FW: FE7X 1.05.00.068 BIOS: BE7X 1.08.00.048 Monitor: ME7X 1.01.00.040 Serial #: F11605A3180172 PCB: Rev3 PCHIP: 1.30-33 ACHIP: 3.20 # of units: 3 Unit 0: JBOD 186.31 GB ( 390721968 blocks): OK Unit 1: RAID 5 465.77 GB ( 976790016 blocks): DEGRADED Unit 5: RAID 5 698.65 GB ( 1465185024 blocks): DEGRADED # of ports: 8 Port 0: WDC WD2000JB-00KFA0 WD-WCAMT1451690 186.31 GB (390721968 blocks): OK(unit 0) Port 1: WDC WD2500JB-00GVC0 WD-WCAL78219488 232.88 GB (488397168 blocks): OK(unit 1) Port 2: WDC WD2000 0.00 MB (0 blocks): OK(NO UNIT) Port 3: WDC WD2500JB-00GVC0 WD-WMAL73882417 232.88 GB (488397168 blocks): OK(unit 1) Port 4: WDC WD2000 0.00 MB (0 blocks): OK(NO UNIT) Port 5: WDC WD2500JB-00GVA0 WD-WMAL71338097 232.88 GB (488397168 blocks): OK(unit 5) Port 6: WDC WD2500JB-32EVA0 WD-WMAEH1301595 232.88 GB (488397168 blocks): OK(unit 5) Port 7: WDC WD2500JB-00GVC0 WD-WCAL78165566 232.88 GB (488397168 blocks): OK(unit 5) </pre> which is to say there are 2 drives which don't belong to any units and don't have any size. They need to be removed from the controller: tw_cli maint remove c0 p2 tw_cli maint remove c0 p4 although it seems like we should be able to add them back in with tw_cli maint add c0 p2 jbod tw_cli maint add c0 p2 spare it doesn't work, and there's no rescan command, so we resort to rebooting. We won't be able to boot all the way up however, cause the 2 drives will turn into JBOD's and throw off the device ordering: <pre>twed0: <Unit 0, JBOD, Normal> on twe0 twed0: 190782MB (390721968 sectors) twed1: <Unit 1, RAID5, Degraded> on twe0 twed1: 476948MB (976790016 sectors) twed2: <Unit 2, JBOD, Normal> on twe0 twed2: 239372MB (490234752 sectors) twed3: <Unit 4, JBOD, Normal> on twe0 twed3: 239372MB (490234752 sectors) twed4: <Unit 5, RAID5, Degraded> on twe0 twed4: 715422MB (1465185024 sectors) twed5: <Unit 0, RAID5, Normal> on twe1 twed5: 715422MB (1465185024 sectors) twed6: <Unit 4, RAID5, Normal> on twe1 twed6: 715422MB (1465185024 sectors) </pre> So whereas twed2 used to be a RAID5 device, it got pushed down (to twed4?) by the JBOD standalone drive. So we could edit the fstab and stop the OS from trying to mount the other devices. After the rebuild (or during) we could reboot and the next time it comes up the devices will fall back to their regular ordering. We usually opt to, over serial console, enter into single user mode (which is automatic with the failed mounts), and do the remirroring: <pre> mount /dev/twed0s1g /usr (tw_cli is in /usr we we need to mount it manually) tw_cli info c0 Controller: c0 ------------- Driver: 1.50.01.002 Model: 7500-8 FW: FE7X 1.05.00.068 BIOS: BE7X 1.08.00.048 Monitor: ME7X 1.01.00.040 Serial #: F11605A3180172 PCB: Rev3 PCHIP: 1.30-33 ACHIP: 3.20 # of units: 5 Unit 0: JBOD 186.31 GB ( 390721968 blocks): OK Unit 1: RAID 5 465.77 GB ( 976790016 blocks): DEGRADED Unit 2: JBOD 233.76 GB ( 490234752 blocks): OK Unit 4: JBOD 233.76 GB ( 490234752 blocks): OK Unit 5: RAID 5 698.65 GB ( 1465185024 blocks): DEGRADED # of ports: 8 Port 0: WDC WD2000JB-00KFA0 WD-WCAMT1451690 186.31 GB (390721968 blocks): OK(unit 0) Port 1: WDC WD2500JB-00GVC0 WD-WCAL78219488 232.88 GB (488397168 blocks): OK(unit 1) Port 2: WDC WD2500SB-01RFA0 WD-WMANK3040813 233.76 GB (490234752 blocks): OK(unit 2) Port 3: WDC WD2500JB-00GVC0 WD-WMAL73882417 232.88 GB (488397168 blocks): OK(unit 1) Port 4: WDC WD2500SB-01RFA0 WD-WMANK3356318 233.76 GB (490234752 blocks): OK(unit 4) Port 5: WDC WD2500JB-00GVA0 WD-WMAL71338097 232.88 GB (488397168 blocks): OK(unit 5) Port 6: WDC WD2500JB-32EVA0 WD-WMAEH1301595 232.88 GB (488397168 blocks): OK(unit 5) Port 7: WDC WD2500JB-00GVC0 WD-WCAL78165566 232.88 GB (488397168 blocks): OK(unit 5) </pre> 2 new drives show up as JBOD's, which is fine, we can allocate into a mirror, but first we must delete the JBOD unit: BE VERY CAREFUL about this, double check that the unit you are deleting has 1 member and it's member is on the port which contains the new drive. <pre> tw_cli maint deleteunit c0 u2 Deleting unit /ct0/u2 ...wed2: detached Done. tw_cli maint deleteunit c0 u4 Deleting unit /ct0/u4 ...wed3: detached Done. tw_cli info c0 Controller: c0 ------------- Driver: 1.50.01.002 Model: 7500-8 FW: FE7X 1.05.00.068 BIOS: BE7X 1.08.00.048 Monitor: ME7X 1.01.00.040 Serial #: F11605A3180172 PCB: Rev3 PCHIP: 1.30-33 ACHIP: 3.20 # of units: 3 Unit 0: JBOD 186.31 GB ( 390721968 blocks): OK Unit 1: RAID 5 465.77 GB ( 976790016 blocks): DEGRADED Unit 5: RAID 5 698.65 GB ( 1465185024 blocks): DEGRADED # of ports: 8 Port 0: WDC WD2000JB-00KFA0 WD-WCAMT1451690 186.31 GB (390721968 blocks): OK(unit 0) Port 1: WDC WD2500JB-00GVC0 WD-WCAL78219488 232.88 GB (488397168 blocks): OK(unit 1) Port 2: WDC WD2500SB-01RFA0 WD-WMANK3040813 233.76 GB (490234752 blocks): OK(NO UNIT) Port 3: WDC WD2500JB-00GVC0 WD-WMAL73882417 232.88 GB (488397168 blocks): OK(unit 1) Port 4: WDC WD2500SB-01RFA0 WD-WMANK3356318 233.76 GB (490234752 blocks): OK(NO UNIT) Port 5: WDC WD2500JB-00GVA0 WD-WMAL71338097 232.88 GB (488397168 blocks): OK(unit 5) Port 6: WDC WD2500JB-32EVA0 WD-WMAEH1301595 232.88 GB (488397168 blocks): OK(unit 5) Port 7: WDC WD2500JB-00GVC0 WD-WCAL78165566 232.88 GB (488397168 blocks): OK(unit 5) </pre> Now the 2 drives show up as full size and available. We can rebuild: <pre> tw_cli maint rebuild c0 u1 p2 Rebuild started twon unit /c0/u1 AEN: <twed1: rebuild started> tw_cli maint rebuild c0 u5 p4 Rebuild started ton unit /c0/u5 AEN: <twed1: rebuild started> tw_cli info c0 Controller: c0 ------------- Driver: 1.50.01.002 Model: 7500-8 FW: FE7X 1.05.00.068 BIOS: BE7X 1.08.00.048 Monitor: ME7X 1.01.00.040 Serial #: F11605A3180172 PCB: Rev3 PCHIP: 1.30-33 ACHIP: 3.20 # of units: 3 Unit 0: JBOD 186.31 GB ( 390721968 blocks): OK Unit 1: RAID 5 465.77 GB ( 976790016 blocks): REBUILDING (0%) Unit 5: RAID 5 698.65 GB ( 1465185024 blocks): REBUILDING (0%) # of ports: 8 Port 0: WDC WD2000JB-00KFA0 WD-WCAMT1451690 186.31 GB (390721968 blocks): OK(unit 0) Port 1: WDC WD2500JB-00GVC0 WD-WCAL78219488 232.88 GB (488397168 blocks): OK(unit 1) Port 2: WDC WD2500SB-01RFA0 WD-WMANK3040813 233.76 GB (490234752 blocks): OK(unit 1) Port 3: WDC WD2500JB-00GVC0 WD-WMAL73882417 232.88 GB (488397168 blocks): OK(unit 1) Port 4: WDC WD2500SB-01RFA0 WD-WMANK3356318 233.76 GB (490234752 blocks): OK(unit 5) Port 5: WDC WD2500JB-00GVA0 WD-WMAL71338097 232.88 GB (488397168 blocks): OK(unit 5) Port 6: WDC WD2500JB-32EVA0 WD-WMAEH1301595 232.88 GB (488397168 blocks): OK(unit 5) Port 7: WDC WD2500JB-00GVC0 WD-WCAL78165566 232.88 GB (488397168 blocks): OK(unit 5) </pre> And now we can reboot, the rebuild will continue backgrounded, and the devices will come up in correct order: <pre>twed0: <Unit 0, JBOD, Normal> on twe0 twed0: 190782MB (390721968 sectors) twed1: <Unit 1, RAID5, Rebuilding> on twe0 twed1: 476948MB (976790016 sectors) twed2: <Unit 4, RAID5, Rebuilding> on twe0 twed2: 715422MB (1465185024 sectors) twed3: <Unit 0, RAID5, Normal> on twe1 twed3: 715422MB (1465185024 sectors) twed4: <Unit 4, RAID5, Normal> on twe1 twed4: 715422MB (1465185024 sectors)</pre> = Areca = Downloads http://www.areca.com.tw/support/main.htm CLI Manual http://www.areca.us/support/download/RaidCards/Documents/Manual_Spec/CLIManual.zip == CLI == <pre> cli64 rsf info view state of RAID cli64 vsf info view state of rebuild cli64 disk info view all disks cli64 event info view error events cli64 disk identify drv=xx turn on lights on a disk cli64 vsf check vol=1 careful (may start a long test)??? </pre> To replace a drive and start a rebuild you must make sure the disk is removed from the RAID array and then insert the new disk and make it a hot spare. <pre> cli64 disk info cli64 disk fail drv=XX only if disk is still in the array ------- physically replace the drive cli64 rsf createhs drv=XX start the rebuild </pre> View status of rebuild. <pre> cli64 vsf info </pre> == Updating FirmWare == <pre> cd /tmp wget http://www.areca.us/support/download/RaidCards/BIOS_Firmware/ARC1160.zip cli64 sys updatefw path=/tmp/ARC1160/149-20101202/ARC1160FIRM.BIN </pre>
Summary:
Please note that all contributions to JCWiki may be edited, altered, or removed by other contributors. If you do not want your writing to be edited mercilessly, then do not submit it here.
You are also promising us that you wrote this yourself, or copied it from a public domain or similar free resource (see
JCWiki:Copyrights
for details).
Do not submit copyrighted work without permission!
Cancel
Editing help
(opens in new window)
Navigation menu
Personal tools
Not logged in
Talk
Contributions
Create account
Log in
Namespaces
Page
Discussion
English
Views
Read
Edit
View history
More
Search
Navigation
Main page
Recent changes
Random page
Help about MediaWiki
Tools
What links here
Related changes
Special pages
Page information