View Disks -> Wrong Serial shown (Unknown or from controller)

[willytenner]

I am using FreeNAS 8.0.2 64-Bit on a HP ProLiant? DL380 G5 server with 8 GB RAM and a Smart Array P400 controller with 8 internal SAS ports. Every single data disk is configured as a RAID0 disk, only the system disk /dev/da0 consists of two disks configured as RAID1. So the data disks were shown as /dev/da1, /dev/da2, and so on.

Bug: In the "Storage->Volumes->View All Volumes->Active Volumes->View Disks" disk listing, the wrong serial number is shown, when the disks are behind a HP Smart Array controller. The shown serial is always the serial from the controller or shown as Unknown. See attached snapshot.

To obtain the correct serial of disks behind a HP Smart Array Controller or a Highpoint Rocket Raid Controller as well, I attach a small code snippet script, which shows the correct serial. It makes use of the smartctl and the camcontrol utility.

=====================================================================

#!/bin/bash
# set -x

parse_serial_number()
{
        disk=$1
        awk -v dev=$disk '{
                if ($1 ~ "^scbus"){
                        controller=$3
                }
                if ( $NF ~ ","dev"\\\)" || $NF ~ "\\\("dev"," ){
			ctrl_type=$1" "$2
                        if (ctrl_type=="<HPT DISK"){
                                print "smartctl -i /dev/hptrr -d hpt,"$(NF-1)+1"/"$(NF-3)+1" | grep \"Serial number:\" | awk \"{print \\\$NF}\""
                        } else if (ctrl_type=="<COMPAQ RAID"){
                                print "smartctl -i /dev/"controller" -d cciss,"$(NF-3)" | grep \"Serial number:\" | awk \"{print \\\$NF}\""
                        } else {
                                print "camcontrol inquiry /dev/"dev" -S"
                        }
                }
        }'

}

get_serial_number()
{
        disk=$1
        camcontrol devlist -v | parse_serial_number $disk
}

eval "$(get_serial_number $1)"

=====================================================================

[william]

Please give it a try with the nightly release, post r12162
(Image should be available tomorrow morning)

[noobsauce80]

Will,

I see you are working on this issue. I am having an issue with 3ware controllers. I believe it can be fixed by adding another condition for when 3ware controllers are detected. Please see thread ​http://forums.freenas.org/showthread.php?8426-SMART-on-3ware-controller for info I have provided.

If you would like more info feel free to send a PM or try IRC. If you would like temporary access to the server via teamviewer I can provide that too. I think the fix is very simple but with my limited BASH experience alot of the commands are leaving me with only a confused look on my face :)

[willytenner]

I tried FreeNAS-8.3.1-ALPHA-r12166-x64 yesterday. Disks behind a HP Smart Array Controller now show the correct serial number when they are configured as JBOD (RAID-0 with 1 member). Thanks a lot.

Another nice thing will be to pre-fill the Description field with the vendor string and the product string (separated with a space). This information can be also extracted from the smartctl output (exactly like the serial number). William, do you think it's possible for you to implement this too?

But I just see the old output from 8.0.2: There the Description field was pre-filled with the volumes membership of the disk. I think this information is shown in the Volume Status of the FreeNAS GUI now. Vendor and Product string is a better option for this field.

Kind regards,
willy

[william]

I believe so, but hardly for 8.3, we will see... We are already a bit behind of schedule.

Can you open another ticket with that request please?

Thanks for the help.

[noobsauce80]

The fix for the 3ware controller is 1/2 working. Here's the issue:

24 port controller with drive in physical ports 0-7 and 12-19. These are being mapped as u0 through u15(which also correspond to da0 through da15). Because physical ports are skipped the serial numbers are currently weird using build 12174.

da0 through da7 are working. The serial numbers in the GUI do match correctly for da0 through 7. However because 4 physical ports are skipped everything gets messed up. da8 through da11 are showing no serial number and da12 through da15 are showing serial numbers, but they are incorrect. The GUI shows da12 serial for what is actually da8, da13 shows da9 and so on.

The fix needs to be changed slightly to allow or skipped devices. If you look at my last post(which was probably posted at the same time William was working on it) it will explain how I think we can identify skipped physical ports. The uXX will tell us how many ports are actually in use. The daXX and uXX always match as long as only 1 3ware controller is in use. The post I made just before William committed the changes may help clarify everything. It can be found at ​http://forums.freenas.org/showthread.php?8426-SMART-on-3ware-controller&p=35402&viewfull=1#post35402. Please take note that uXX, pXX, and daXX are not always equal.

Strangely, I just tried moving all of the hard drives to physical ports p0 through p15 and da8 through da11 are still blank but da12 through da15 are displayed and accurate. That puzzles me a little. I expected that everything would work perfectly.

Thanks for the hard work. This is likely an easy fix since it 1/2 works.

[noobsauce80]

Another addition:

With the disks in physical ports 0 through 15 SMART started with no issues and ix-smartd properly identifed all disks. When disk are installed in physical ports 4-11 and 16-23 SMART only recognizes ports 4 to 11. All disks previously were detected by ix-smartd when in ports 0 through 15. I verified all of this by examining /usr/local/etc/smartd.conf.

[william]

Well, I dont own a 3ware, so a patch would be very welcome.

[noobsauce80]

I don't have alot of experience with programming for Unix. I'm not even completely sure what language it is. Most of my knowledge is self-taught and from high school years ago. I do enjoy the challenge of reading up and learning from looking at other code and experimenting.

Right now, I think the easiest way to fix this issue is to ignore the physical ports and simply set uXX to daXX. The controller seems to make those 2 match so long as there isn't any other daXX devices that are mapped first. Luckily the USB key seems to always be assigned last, so it won't interfere. It may not work for everyone's setup, but for configurations that have only 1 3ware controller it should work just fine. Simply put, obtain the serial for /c0/u0 and then assume that serial is for da0. Just keep counting up as long as there are more uXX devices.

As for SMART, I'm really at a loss to explain how the ix-smartd worked great with all 16 drives in the first 16 physical ports but didn't work when any physical ports were skipped. Unfortunately the hard disk run quite warm when all shoved together so we have had to space them out to keep them all below 40C.

William,
can you provide the files you changed that originally added support for SMART and serial numbers? I don't know what files to look at. I don't know anyone personally that has dealt with SVN repositories, diffs, etc. My programming experience is mostly limited to small projects that didn't have/need repositories. I'll definitely give it my best shot if you can provide the files. :) I'll definitely enjoy playing around and I have lots of time to learn, not to mention the desire to get more involved with development of FreeNAS if I can get the hang of it. I've already compiled from the trunk before so I have a small handle on what I need to do.

[noobsauce80]

Let me show you this...

If you do a # camcontrol devlist output I'll see the list of disks as:

<AMCC 9650SE-24M DISK 4.10>        at scbus3 target 0 lun 0 (pass0,da0)
<AMCC 9650SE-24M DISK 4.10>        at scbus3 target 1 lun 0 (pass1,da1)
<AMCC 9650SE-24M DISK 4.10>        at scbus3 target 2 lun 0 (pass2,da2)
<AMCC 9650SE-24M DISK 4.10>        at scbus3 target 3 lun 0 (pass3,da3)
<AMCC 9650SE-24M DISK 4.10>        at scbus3 target 4 lun 0 (pass4,da4)
<AMCC 9650SE-24M DISK 4.10>        at scbus3 target 5 lun 0 (pass5,da5)
<AMCC 9650SE-24M DISK 4.10>        at scbus3 target 6 lun 0 (pass6,da6)
<AMCC 9650SE-24M DISK 4.10>        at scbus3 target 7 lun 0 (pass7,da7)
<AMCC 9650SE-24M DISK 4.10>        at scbus3 target 8 lun 0 (pass8,da8)
<AMCC 9650SE-24M DISK 4.10>        at scbus3 target 9 lun 0 (pass9,da9)
<AMCC 9650SE-24M DISK 4.10>        at scbus3 target 10 lun 0 (pass10,da10)
<AMCC 9650SE-24M DISK 4.10>        at scbus3 target 11 lun 0 (pass11,da11)
<AMCC 9650SE-24M DISK 4.10>        at scbus3 target 12 lun 0 (pass12,da12)
<AMCC 9650SE-24M DISK 4.10>        at scbus3 target 13 lun 0 (pass13,da13)
<AMCC 9650SE-24M DISK 4.10>        at scbus3 target 14 lun 0 (pass14,da14)
<AMCC 9650SE-24M DISK 4.10>        at scbus3 target 15 lun 0 (pass15,da15)
<Corsair Voyager 1100>             at scbus11 target 0 lun 0 (pass16,da16)

So now we know that my 3ware controller has 16 disks attached and they are da0 through da15.

Note: the output from # camcontrol devlist output never change despite where the disks are physically connected.. so this is a good way to determine which daXX's are for the 3ware controller.

So now if you do a # tw_cli /c0 show you will get:

Unit  UnitType  Status         %RCmpl  %V/I/M  Stripe  Size(GB)  Cache  AVrfy
------------------------------------------------------------------------------
u0    JBOD      OK             -       -       -       1397.27   Ri     OFF
u1    JBOD      OK             -       -       -       1397.27   Ri     OFF
u2    JBOD      OK             -       -       -       1397.27   Ri     OFF
u3    JBOD      OK             -       -       -       1397.26   Ri     OFF
u4    JBOD      OK             -       -       -       1397.27   Ri     OFF
u5    JBOD      OK             -       -       -       1397.27   Ri     OFF
u6    JBOD      OK             -       -       -       1397.27   Ri     OFF
u7    JBOD      OK             -       -       -       1397.27   Ri     OFF
u8    JBOD      OK             -       -       -       931.513   Ri     OFF
u9    JBOD      OK             -       -       -       931.513   Ri     OFF
u10   JBOD      OK             -       -       -       931.513   Ri     OFF
u11   JBOD      OK             -       -       -       931.513   Ri     OFF
u12   JBOD      OK             -       -       -       931.512   Ri     OFF
u13   JBOD      OK             -       -       -       931.513   Ri     OFF
u14   JBOD      OK             -       -       -       931.513   Ri     OFF
u15   JBOD      OK             -       -       -       931.513   Ri     OFF

VPort Status         Unit Size      Type  Phy Encl-Slot    Model
------------------------------------------------------------------------------
p4    OK             u0   1.36 TB   SATA  4   -            ST31500341AS
p5    OK             u1   1.36 TB   SATA  5   -            ST31500341AS
p6    OK             u2   1.36 TB   SATA  6   -            ST31500341AS
p7    OK             u3   1.36 TB   SATA  7   -            ST31500341AS
p8    OK             u4   1.36 TB   SATA  8   -            ST31500341AS
p9    OK             u5   1.36 TB   SATA  9   -            ST31500341AS
p10   OK             u6   1.36 TB   SATA  10  -            ST31500341AS
p11   OK             u7   1.36 TB   SATA  11  -            ST31500341AS
p16   OK             u8   931.51 GB SATA  16  -            ST31000528AS
p17   OK             u9   931.51 GB SATA  17  -            ST31000340AS
p18   OK             u10  931.51 GB SATA  18  -            ST31000340AS
p19   OK             u11  931.51 GB SATA  19  -            ST31000340AS
p20   OK             u12  931.51 GB SATA  20  -            ST31000528AS
p21   OK             u13  931.51 GB SATA  21  -            ST31000340AS
p22   OK             u14  931.51 GB SATA  22  -            ST31000340AS
p23   OK             u15  931.51 GB SATA  23  -            ST31000340AS

The uYY are mapped in numerical order to the daXX. So we can take the first daXX and match it to the first uYY, then the second daXX to the second uYY and so on you'll have a table comparing daXX to uYY. We can ignore the physical ports(pXX), or also include them in our internal table by taking the uYY and comparing them to pZZ. Effectively we now know:

da   u    p     serial
======================
0    0    4     ?????
1    1    5     ?????
2    2    6     ?????
3    3    7     ?????
4    4    8     ?????
5    5    9     ?????
6    6    10    ?????
7    7    11    ?????
8    8    16    ?????
9    9    17    ?????
10   10   18    ?????
11   11   19    ?????
12   12   20    ?????
13   13   21    ?????
14   14   22    ?????
15   15   23    ?????

Then you can get the serials for the daXX by running # tw_cli /c0/uYY show serial and you get:
/c0/u0 serial number = 9VS0NXRE000000000000

Now have a daXX, the associated uYY and the associated serial number for it. It's too bad I don't know my way around better. I can get the thought process but I don't know how to actually do it :(

Now all of this falls apart if we have 2 controllers because I'm not sure how the # camcontrol devlist output will look. I assume it will be a different lun or scbus#, but since I only have 1 controller I can't actually tell you for sure.

As for the SMARTD I'm not to sure how ix-smartd works internally to determine which disks are which and whatnot, but since we have the table comparing daXX, uYY, pZZ and serials(of course, the serials are pointless for SMARTd commands) it should be easy to get ix-smartd to create the applicable conf file. I can load up some smart info on any disk by running # smartctl -a -d 3ware,YY /dev/twa0 If I use the default values from the FreeNAS GUI I'd expect the conf file for the above configurations to look like:

/dev/twa0 -d 3ware,4 -n never -W 0,0,0 -m myemail@email.com
/dev/twa0 -d 3ware,5 -n never -W 0,0,0 -m myemail@email.com
/dev/twa0 -d 3ware,6 -n never -W 0,0,0 -m myemail@email.com
/dev/twa0 -d 3ware,7 -n never -W 0,0,0 -m myemail@email.com
/dev/twa0 -d 3ware,8 -n never -W 0,0,0 -m myemail@email.com
/dev/twa0 -d 3ware,9 -n never -W 0,0,0 -m myemail@email.com
/dev/twa0 -d 3ware,10 -n never -W 0,0,0 -m myemail@email.com
/dev/twa0 -d 3ware,11 -n never -W 0,0,0 -m myemail@email.com
/dev/twa0 -d 3ware,16 -n never -W 0,0,0 -m myemail@email.com
/dev/twa0 -d 3ware,17 -n never -W 0,0,0 -m myemail@email.com
/dev/twa0 -d 3ware,18 -n never -W 0,0,0 -m myemail@email.com
/dev/twa0 -d 3ware,19 -n never -W 0,0,0 -m myemail@email.com
/dev/twa0 -d 3ware,20 -n never -W 0,0,0 -m myemail@email.com
/dev/twa0 -d 3ware,21 -n never -W 0,0,0 -m myemail@email.com
/dev/twa0 -d 3ware,22 -n never -W 0,0,0 -m myemail@email.com
/dev/twa0 -d 3ware,23 -n never -W 0,0,0 -m myemail@email.com

Please note that for all of the situations where uYY is used in a shell command you can use 2 digits for numbers 0 through 9. That is you can use '04' instead of '4' if that makes anything easier.

[william]

I really appreciate your input, it is very useful.

However it can't do anything without a testing scenario. I am not _that_ good programmer to write code to something I can't test, it just won't work, it will be an endless ping-pong.

So either someone could give me access to a box like that or provide a starting patch.

[william]

Ok, noobsauce80 was kind and provided access to his freenas...

3ware support should have been fixed in r12204

Thanks