mfi and Dell PERC 6/i

[Danilo Baio]

Hi guys,

I have a DELL PERC 6/i controller and I can't find the problem.
The system was running and lost disk access with this messages on console:
"mfi0: COMMAND 0xffffff80005d1770 TIMEOUT AFTER 6178 SECONDS"
...
Loading Image...


I can't log in, after a reboot, nothing on the logs.

only shows this on boot:

Aug 24 23:04:58 bazinga kernel: mfi0: 2806 (335999491s/0x0020/DEAD) - Fatal
firmware error: Line 156 in ../../raid/1078int.c

What can I do to find the problem ?
This controller is really supported by mfi ?

Thanks for the help.

bazinga# mfiutil show volumes
mfi0 Volumes:
Id Size Level Stripe State Cache Name
mfid0 ( 465G) RAID-1 64K OPTIMAL Disabled <VD-SO>
mfid1 ( 272G) RAID-10 64K OPTIMAL Disabled <VD-VIRTUAL>
bazinga# mfiutil show drives
mfi0 Physical Drives:
( 137G) ONLINE <SEAGATE ST9146852SS HT62 serial=3TB1EFZY> SAS enclosure 1,
slot 0
( 137G) ONLINE <SEAGATE ST9146852SS HT62 serial=3TB1DT9Y> SAS enclosure 1,
slot 1
( 137G) ONLINE <SEAGATE ST9146852SS HT62 serial=3TB1EFW6> SAS enclosure 1,
slot 2
( 137G) ONLINE <SEAGATE ST9146852SS HT62 serial=3TB1DZ7E> SAS enclosure 1,
slot 3
( 466G) ONLINE <SEAGATE ST9500430SS DS62 serial=9SP1F2WT> SAS enclosure 1,
slot 4
( 466G) ONLINE <SEAGATE ST9500430SS DS62 serial=9SP1F77F> SAS enclosure 1,
slot 5


bazinga# mfiutil show firmware
mfi0 Firmware Package Version: 6.2.0-0013
mfi0 Firmware Images:
Name Version Date Time Status
APP 1.22.02-0612 Mar 30 2009 14:41:22 active
BIOS 2.04.00 active
BCON 1.1-46-e_15-Rel Mar 2 2008 14:06:08 active
CTLR 1.02-015B Jan 27 2009 12:02:58 active
PCLI 01.00-023:#%00006 Nov 25 2008 17:21:50 active
BTBL 1.00.00.01-0011 Nov 27 2007 18:29:20 active

I am running 8.1-RELEASE amd64 with DELL R610.

This is all messages from boot that shows "mfi0"

Aug 24 23:04:58 bazinga kernel: mfi0: <Dell PERC 6> port 0xfc00-0xfcff mem
0xdf180000-0xdf1bffff,0xdf1c0000-0xdf1fffff irq 16 at device 0.0 on pci3
Aug 24 23:04:58 bazinga kernel: mfi0: Megaraid SAS driver Ver 3.00
Aug 24 23:04:58 bazinga kernel: mfi0: 2708 (335880663s/0x0020/info) -
Shutdown command received from host
Aug 24 23:04:58 bazinga kernel: mfi0: 2709 (boot + 3s/0x0020/info) -
Firmware initialization started (PCI ID 0060/1000/1f0c/1028)
Aug 24 23:04:58 bazinga kernel: mfi0: 2710 (boot + 3s/0x0020/info) -
Firmware version 1.22.02-0612
Aug 24 23:04:58 bazinga kernel: mfi0: 2711 (boot + 3s/0x0020/info) -
Firmware initialization started (PCI ID 0060/1000/1f0c/1028)
Aug 24 23:04:58 bazinga kernel: mfi0: 2712 (boot + 3s/0x0020/info) -
Firmware version 1.22.02-0612
Aug 24 23:04:58 bazinga kernel: mfi0: 2713 (boot + 23s/0x0008/info) -
Battery Present
Aug 24 23:04:58 bazinga kernel: mfi0: 2714 (boot + 23s/0x0020/info) -
Controller hardware revision ID (0x0)
Aug 24 23:04:58 bazinga kernel: mfi0: 2715 (boot + 23s/0x0020/info) -
Package version 6.2.0-0013
Aug 24 23:04:58 bazinga kernel: mfi0: 2716 (boot + 23s/0x0020/info) - Board
Revision
Aug 24 23:04:58 bazinga kernel: mfi0: 2717 (boot + 49s/0x0002/info) -
Unexpected sense: Encl PD 20 Path 5a4ba0b032c08400, CDB: 12 00 00 00 04 00,
Sense: 6/29/00
Aug 24 23:04:58 bazinga kernel: mfi0: 2718 (boot + 50s/0x0004/info) -
Enclosure PD 20(c None/p0) communication restored
Aug 24 23:04:58 bazinga kernel: mfi0: 2719 (boot + 50s/0x0004/info) -
Enclosure PD 20(c None/p0) element (SES code 0x17) status changed
Aug 24 23:04:58 bazinga kernel: mfi0: 2720 (boot + 50s/0x0004/info) -
Enclosure PD 20(c None/p0) element (SES code 0x17) status changed
Aug 24 23:04:58 bazinga kernel: mfi0: 2721 (boot + 50s/0x0004/info) -
Enclosure PD 20(c None/p0) element (SES code 0x17) status changed
Aug 24 23:04:58 bazinga kernel: mfi0: 2722 (boot + 50s/0x0004/info) -
Enclosure PD 20(c None/p0) element (SES code 0x17) status changed
Aug 24 23:04:58 bazinga kernel: mfi0: 2723 (boot + 50s/0x0004/info) -
Enclosure PD 20(c None/p0) element (SES code 0x17) status changed
Aug 24 23:04:58 bazinga kernel: mfi0: 2724 (boot + 50s/0x0004/info) -
Enclosure PD 20(c None/p0) element (SES code 0x17) status changed
Aug 24 23:04:58 bazinga kernel: mfi0: 2725 (boot + 50s/0x0002/info) -
Inserted: Encl PD 20
Aug 24 23:04:58 bazinga kernel: mfi0: 2726 (boot + 50s/0x0002/info) -
Inserted: PD 20(c None/p0) Info: enclPd=20, scsiType=d, portMap=09,
sasAddr=5a4ba0b032c08400,0000000000000000
Aug 24 23:04:58 bazinga kernel: mfi0: 2727 (boot + 50s/0x0002/info) -
Inserted: PD 00(e0x20/s0)
Aug 24 23:04:58 bazinga kernel: mfi0: 2728 (boot + 50s/0x0002/info) -
Inserted: PD 00(e0x20/s0) Info: enclPd=20, scsiType=0, portMap=00,
sasAddr=5000c500231e5111,0000000000000000
Aug 24 23:04:58 bazinga kernel: mfi0: 2729 (boot + 50s/0x0002/info) -
Inserted: PD 01(e0x20/s1)
Aug 24 23:04:58 bazinga kernel: mfi0: 2730 (boot + 50s/0x0002/info) -
Inserted: PD 01(e0x20/s1) Info: enclPd=20, scsiType=0, portMap=01,
sasAddr=5000c500231e6e4d,0000000000000000
Aug 24 23:04:58 bazinga kernel: mfi0: 2731 (boot + 50s/0x0002/info) -
Inserted: PD 02(e0x20/s2)
Aug 24 23:04:58 bazinga kernel: mfi0: 2732 (boot + 50s/0x0002/info) -
Inserted: PD 02(e0x20/s2) Info: enclPd=20, scsiType=0, portMap=02,
sasAddr=5000c500231e5239,0000000000000000
Aug 24 23:04:58 bazinga kernel: mfi0: 2733 (boot + 50s/0x0002/info) -
Inserted: PD 03(e0x20/s3)
Aug 24 23:04:58 bazinga kernel: mfi0: 2734 (boot + 50s/0x0002/info) -
Inserted: PD 03(e0x20/s3) Info: enclPd=20, scsiType=0, portMap=03,
sasAddr=5000c500231e0b2d,0000000000000000
Aug 24 23:04:58 bazinga kernel: mfi0: 2735 (boot + 50s/0x0002/info) -
Inserted: PD 04(e0x20/s4)
Aug 24 23:04:58 bazinga kernel: mfi0: 2736 (boot + 50s/0x0002/info) -
Inserted: PD 04(e0x20/s4) Info: enclPd=20, scsiType=0, portMap=04,
sasAddr=5000c50020c23955,0000000000000000
Aug 24 23:04:58 bazinga kernel: mfi0: 2737 (boot + 50s/0x0002/info) -
Inserted: PD 05(e0x20/s5)
Aug 24 23:04:58 bazinga kernel: mfi0: 2738 (boot + 50s/0x0002/info) -
Inserted: PD 05(e0x20/s5) Info: enclPd=20, scsiType=0, portMap=05,
sasAddr=5000c50020c244c9,0000000000000000
Aug 24 23:04:58 bazinga kernel: mfi0: 2739 (335880741s/0x0020/info) - Time
established as 08/23/10 12:12:21; (51 seconds since power on)
Aug 24 23:04:58 bazinga kernel: mfi0: 2740 (335880778s/0x0008/info) -
Battery temperature is normal
Aug 24 23:04:58 bazinga kernel: mfi0: 2741 (335880778s/0x0008/info) -
Battery started charging
Aug 24 23:04:58 bazinga kernel: mfi0: 2742 (335880778s/0x0008/info) -
Current capacity of the battery is above threshold
Aug 24 23:04:58 bazinga kernel: mfi0: 2743 (335891243s/0x0008/info) -
Battery charge complete
Aug 24 23:04:58 bazinga kernel: mfi0: 2744 (335891851s/0x0020/info) - Patrol
Read started
Aug 24 23:04:58 bazinga kernel: mfi0: 2805 (335899090s/0x0020/info) - Patrol
Read complete
Aug 24 23:04:58 bazinga kernel: mfi0: 2806 (335999491s/0x0020/DEAD) - Fatal
firmware error: Line 156 in ../../raid/1078int.c
Aug 24 23:04:58 bazinga kernel:
Aug 24 23:04:58 bazinga kernel: mfi0: 2807 (boot + 3s/0x0020/info) -
Firmware initialization started (PCI ID 0060/1000/1f0c/1028)
Aug 24 23:04:58 bazinga kernel: mfi0: 2808 (boot + 3s/0x0020/info) -
Firmware version 1.22.02-0612
Aug 24 23:04:58 bazinga kernel: mfi0: 2809 (boot + 3s/0x0020/info) -
Firmware initialization started (PCI ID 0060/1000/1f0c/1028)
Aug 24 23:04:58 bazinga kernel: mfi0: 2810 (boot + 3s/0x0020/info) -
Firmware version 1.22.02-0612
Aug 24 23:04:58 bazinga kernel: mfi0: 2811 (boot + 23s/0x0008/info) -
Battery Present
Aug 24 23:04:58 bazinga kernel: mfi0: 2812 (boot + 23s/0x0020/info) -
Controller hardware revision ID (0x0)
Aug 24 23:04:58 bazinga kernel: mfi0: 2813 (boot + 23s/0x0020/info) -
Package version 6.2.0-0013
Aug 24 23:04:58 bazinga kernel: mfi0: 2814 (boot + 23s/0x0020/info) - Board
Revision
Aug 24 23:04:58 bazinga kernel: mfi0: 2815 (boot + 62s/0x0002/WARN) - PD
110(e0x00/s16) Path 5a4ba0b032c08400 reset (Type 03)
Aug 24 23:04:58 bazinga kernel: mfi0: 2816 (boot + 66s/0x0004/info) -
Enclosure PD 20(c None/p0) communication restored
Aug 24 23:04:58 bazinga kernel: mfi0: 2817 (boot + 67s/0x0002/info) -
Inserted: Encl PD 20
Aug 24 23:04:58 bazinga kernel: mfi0: 2818 (boot + 67s/0x0002/info) -
Inserted: PD 20(c None/p0) Info: enclPd=20, scsiType=d, portMap=09,
sasAddr=5a4ba0b032c08400,0000000000000000
Aug 24 23:04:58 bazinga kernel: mfi0: 2819 (boot + 67s/0x0002/info) -
Inserted: PD 00(e0x20/s0)
Aug 24 23:04:58 bazinga kernel: mfi0: 2820 (boot + 67s/0x0002/info) -
Inserted: PD 00(e0x20/s0) Info: enclPd=20, scsiType=0, portMap=00,
sasAddr=5000c500231e5111,0000000000000000
Aug 24 23:04:58 bazinga kernel: mfi0: 2821 (boot + 67s/0x0002/info) -
Inserted: PD 01(e0x20/s1)
Aug 24 23:04:58 bazinga kernel: mfi0: 2822 (boot + 67s/0x0002/info) -
Inserted: PD 01(e0x20/s1) Info: enclPd=20, scsiType=0, portMap=01,
sasAddr=5000c500231e6e4d,0000000000000000
Aug 24 23:04:58 bazinga kernel: mfi0: 2823 (boot + 67s/0x0002/info) -
Inserted: PD 02(e0x20/s2)
Aug 24 23:04:58 bazinga kernel: mfi0: 2824 (boot + 67s/0x0002/info) -
Inserted: PD 02(e0x20/s2) Info: enclPd=20, scsiType=0, portMap=02,
sasAddr=5000c500231e5239,0000000000000000
Aug 24 23:04:58 bazinga kernel: mfi0: 2825 (boot + 67s/0x0002/info) -
Inserted: PD 03(e0x20/s3)
Aug 24 23:04:58 bazinga kernel: mfi0: 2826 (boot + 67s/0x0002/info) -
Inserted: PD 03(e0x20/s3) Info: enclPd=20, scsiType=0, portMap=03,
sasAddr=5000c500231e0b2d,0000000000000000
Aug 24 23:04:58 bazinga kernel: mfi0: 2827 (boot + 67s/0x0002/info) -
Inserted: PD 04(e0x20/s4)
Aug 24 23:04:58 bazinga kernel: mfi0: 2828 (boot + 67s/0x0002/info) -
Inserted: PD 04(e0x20/s4) Info: enclPd=20, scsiType=0, portMap=04,
sasAddr=5000c50020c23955,0000000000000000
Aug 24 23:04:58 bazinga kernel: mfi0: 2829 (boot + 67s/0x0002/info) -
Inserted: PD 05(e0x20/s5)
Aug 24 23:04:58 bazinga kernel: mfi0: 2830 (boot + 67s/0x0002/info) -
Inserted: PD 05(e0x20/s5) Info: enclPd=20, scsiType=0, portMap=05,
sasAddr=5000c50020c244c9,0000000000000000
Aug 24 23:04:58 bazinga kernel: mfi0: 2831 (336006245s/0x0020/info) - Time
established as 08/24/10 23:04:05; (68 seconds since power on)
Aug 24 23:04:58 bazinga kernel: mfi0: 2832 (336006265s/0x0008/info) -
Battery temperature is normal

[Scott Long]

Post by Danilo Baio
Hi guys,
I have a DELL PERC 6/i controller and I can't find the problem.
"mfi0: COMMAND 0xffffff80005d1770 TIMEOUT AFTER 6178 SECONDS"
...
http://img806.imageshack.us/img806/2300/errorr6102.png
I can't log in, after a reboot, nothing on the logs.
Aug 24 23:04:58 bazinga kernel: mfi0: 2806 (335999491s/0x0020/DEAD) - Fatal
firmware error: Line 156 in ../../raid/1078int.c

The firmware on the controller crashed. The best I can suggest is to look for newer firmware (mfiutil can flash firmware) and to call LSI or Dell tech-support and report the problem. In the past, there have been bugs with patrol reads causing crashes under heavy load, so you might also look at disabling that option.

Scott

[Danilo Baio]

Post by Scott Long

Post by Danilo Baio
Hi guys,
I have a DELL PERC 6/i controller and I can't find the problem.
The system was running and lost disk access with this messages on
"mfi0: COMMAND 0xffffff80005d1770 TIMEOUT AFTER 6178 SECONDS"
...
http://img806.imageshack.us/img806/2300/errorr6102.png
I can't log in, after a reboot, nothing on the logs.
Aug 24 23:04:58 bazinga kernel: mfi0: 2806 (335999491s/0x0020/DEAD) -

Fatal

Post by Danilo Baio
firmware error: Line 156 in ../../raid/1078int.c

The firmware on the controller crashed. The best I can suggest is to look
for newer firmware (mfiutil can flash firmware) and to call LSI or Dell
tech-support and report the problem. In the past, there have been bugs with
patrol reads causing crashes under heavy load, so you might also look at
disabling that option.
Scott

Intersting, patrol read is automatic and before crash show on the logs that
patrol read has started.

I disabled this feature, rebooted the server and didn't show that firmware
error...

I will test for some days.

Thank you Scott.

[Ian FREISLICH]

Post by Danilo Baio

Post by Scott Long
The firmware on the controller crashed. The best I can suggest is to look
for newer firmware (mfiutil can flash firmware) and to call LSI or Dell
tech-support and report the problem. In the past, there have been bugs with
patrol reads causing crashes under heavy load, so you might also look at
disabling that option.

Dell will not be interested unless the adapter is running the most
recent firmware.

Post by Danilo Baio
Intersting, patrol read is automatic and before crash show on the logs that
patrol read has started.
I disabled this feature, rebooted the server and didn't show that firmware
error...
I will test for some days.

Dell, (maybe) Scott and I recomend that you ensure you're on the
latest firmware:

[firewall2] ~ # mfiutil -u0 show adapter
mfi0 Adapter:
Product Name: PERC 6/i Adapter
Serial Number: 1122334455667788
Firmware: 6.2.0-0013
RAID Levels: JBOD, RAID0, RAID1, RAID5, RAID6, RAID10, RAID50
Battery Backup: present
NVRAM: 32K
Onboard Memory: 256M
Minimum Stripe: 8K
Maximum Stripe: 1M

I'm sure there are bugs on this firmware too, but reading the fixes
that were made between the version that came on the adapter and
this version were Truely Frightening.

It's trivial to update the firmware with the mfiutl program.

Ian

[Danilo Baio]

Post by Ian FREISLICH

Post by Danilo Baio

Post by Scott Long
The firmware on the controller crashed. The best I can suggest is to

look

Post by Danilo Baio

Post by Scott Long
for newer firmware (mfiutil can flash firmware) and to call LSI or Dell
tech-support and report the problem. In the past, there have been bugs

with

Post by Danilo Baio

Post by Scott Long
patrol reads causing crashes under heavy load, so you might also look

at

Post by Danilo Baio

Post by Scott Long
disabling that option.

Dell will not be interested unless the adapter is running the most
recent firmware.

Post by Danilo Baio
Intersting, patrol read is automatic and before crash show on the logs

that

Post by Danilo Baio
patrol read has started.
I disabled this feature, rebooted the server and didn't show that

firmware

Post by Danilo Baio
error...
I will test for some days.

Dell, (maybe) Scott and I recomend that you ensure you're on the
[firewall2] ~ # mfiutil -u0 show adapter
Product Name: PERC 6/i Adapter
Serial Number: 1122334455667788
Firmware: 6.2.0-0013
RAID Levels: JBOD, RAID0, RAID1, RAID5, RAID6, RAID10, RAID50
Battery Backup: present
NVRAM: 32K
Onboard Memory: 256M
Minimum Stripe: 8K
Maximum Stripe: 1M
I'm sure there are bugs on this firmware too, but reading the fixes
that were made between the version that came on the adapter and
this version were Truely Frightening.
It's trivial to update the firmware with the mfiutl program.
Ian
--
Ian Freislich

Hi Ian,

It´s already running the last one.

bazinga# mfiutil -u0 show adapter
mfi0 Adapter:
Product Name: PERC 6/i Integrated
Serial Number: 1122334455667788
Firmware: 6.2.0-0013
RAID Levels: JBOD, RAID0, RAID1, RAID5, RAID6, RAID10, RAID50
Battery Backup: present
NVRAM: 32K
Onboard Memory: 256M
Minimum Stripe: 8K
Maximum Stripe: 1M


bazinga# mfiutil -u0 show firmware
mfi0 Firmware Package Version: 6.2.0-0013
mfi0 Firmware Images:
Name Version Date Time Status
APP 1.22.02-0612 Mar 30 2009 14:41:22 active
BIOS 2.04.00 active
BCON 1.1-46-e_15-Rel Mar 2 2008 14:06:08 active
CTLR 1.02-015B Jan 27 2009 12:02:58 active
PCLI 01.00-023:#%00006 Nov 25 2008 17:21:50 active
BTBL 1.00.00.01-0011 Nov 27 2007 18:29:20 active
bazinga#


It is the last one that i've found on dell website:
http://ftp.us.dell.com/SAS-RAID/R216023.txt

This is a firmware release for the Dell PERC 6/i Adapter.

Component Current Version Previous Version

Package 6.2.0-0013 6.1.1-0047
Firmware 1.22.02-0612 1.21.02-0528
Bootblock 1.00.00.01-0011 1.00.00.01-0011
Ctrl-R 1.02.015B 1.02.014B



Thank you...

[Danilo Baio]

Post by Danilo Baio

Post by Scott Long

Post by Danilo Baio

Post by Scott Long
The firmware on the controller crashed. The best I can suggest is to

look

Post by Danilo Baio

Post by Scott Long
for newer firmware (mfiutil can flash firmware) and to call LSI or

Dell

Post by Danilo Baio

Post by Scott Long
tech-support and report the problem. In the past, there have been

bugs with

Post by Danilo Baio

Post by Scott Long
patrol reads causing crashes under heavy load, so you might also look

at

Post by Danilo Baio

Post by Scott Long
disabling that option.

Dell will not be interested unless the adapter is running the most
recent firmware.

Post by Danilo Baio
Intersting, patrol read is automatic and before crash show on the logs

that

Post by Danilo Baio
patrol read has started.
I disabled this feature, rebooted the server and didn't show that

firmware

Post by Danilo Baio
error...
I will test for some days.

Dell, (maybe) Scott and I recomend that you ensure you're on the
[firewall2] ~ # mfiutil -u0 show adapter
Product Name: PERC 6/i Adapter
Serial Number: 1122334455667788
Firmware: 6.2.0-0013
RAID Levels: JBOD, RAID0, RAID1, RAID5, RAID6, RAID10, RAID50
Battery Backup: present
NVRAM: 32K
Onboard Memory: 256M
Minimum Stripe: 8K
Maximum Stripe: 1M
I'm sure there are bugs on this firmware too, but reading the fixes
that were made between the version that came on the adapter and
this version were Truely Frightening.
It's trivial to update the firmware with the mfiutl program.
Ian
--
Ian Freislich

Hi Ian,
It´s already running the last one.
bazinga# mfiutil -u0 show adapter
Product Name: PERC 6/i Integrated
Serial Number: 1122334455667788
Firmware: 6.2.0-0013
RAID Levels: JBOD, RAID0, RAID1, RAID5, RAID6, RAID10, RAID50
Battery Backup: present
NVRAM: 32K
Onboard Memory: 256M
Minimum Stripe: 8K
Maximum Stripe: 1M
bazinga# mfiutil -u0 show firmware
mfi0 Firmware Package Version: 6.2.0-0013
Name Version Date Time Status
APP 1.22.02-0612 Mar 30 2009 14:41:22 active
BIOS 2.04.00 active
BCON 1.1-46-e_15-Rel Mar 2 2008 14:06:08 active
CTLR 1.02-015B Jan 27 2009 12:02:58 active
PCLI 01.00-023:#%00006 Nov 25 2008 17:21:50 active
BTBL 1.00.00.01-0011 Nov 27 2007 18:29:20 active
bazinga#
http://ftp.us.dell.com/SAS-RAID/R216023.txt
This is a firmware release for the Dell PERC 6/i Adapter.
Component Current Version Previous Version
Package 6.2.0-0013 6.1.1-0047
Firmware 1.22.02-0612 1.21.02-0528
Bootblock 1.00.00.01-0011 1.00.00.01-0011
Ctrl-R 1.02.015B 1.02.014B

Hi guys,

Solved my problem.

The support DELL replaced the battery and the problem persisted, so they
replaced all the controller for a new one and solved the problem.

Today this controller is running on a production/stress server with no
problem, uptime 31 days.
FreeBSD 8.1
Raid 1 (2 disks sata 500) and Raid 10 (four disks sas 15k)

Thank you all.