By Nabil Nawaz, Senior Principal Consultant, Viscosity NA
Sys Admins and Exadata DBA’s probably have run the SunDiag scripts from Oracle Support to gather diagnostic information for diagnosing physical and flash disk failures/issues. One thing to remember is that the version of the SunDiag script that is on the system may not be the latest so please download the latest one from the following Support note: Oracle Exadata Diagnostic Information required for Disk Failures [ID 761868.1]
The output file is compressed in bz2 format and furthermore it is not easy to parse and read the all of the files. There is however a better script that can be run to do similar diagnostic against the Flash Cards/Disks on the Storage Cells and the output is all captured in a single text logfile that is easy to read. The script is called CheckDBM.sh and was written by Bob Clare, Oracle Exadata Support. The script can be run from any node (DB or Storage) individually, but must be run as ‘root’, and can also be run in parallel on multiple nodes via dcli (much as with sundiag, as shown in Note 761868.1). Here’s how I typically run it:
$ dcli -g ~/all_group -l root -t -x CheckDBM.sh | tee /tmp/CheckDBM-All.log
where ~ corresponds to /root (assuming dcli group files have been copied to same, else use full /opt/… patch to group files as in sundiag note), and of course other group files could be substituted, e.g. cell_group, w/pipe to tee gathering all results in a single local log (so you don’t have to go gather them individually from each node).
Sample output for one of the Exadata Storage Cells below.
Target cells: [‘exa01-priv’, ‘exa02-priv’, ‘exa03-priv’, ‘exa04-priv’, ‘exacell01-priv’, ‘exacell02-priv’, ‘exacell03-priv’, ‘exacell04-priv’, ‘exacell05-priv’, ‘exacell06-priv’, ‘exacell07-priv’]
exa01-priv:
exa01-priv: CheckDBM.sh (Version 2.5) started on: Tuesday, 05/07/13 at: 23:18:17 CDT …
exa01-priv: ================================ System ================================
exa01-priv: Node ‘exa01’ local time: 23:18:17 up 17 days
exa01-priv: MACHINETYPE: 1 X2-2 Half rack
exa01-priv: ——————————– OS ——————————–
exa01-priv: Linux 2.6.18-194.3.1.0.4.el5 x86_64
exa01-priv: ==> /etc/enterprise-release <==
exa01-priv: Enterprise Linux Enterprise Linux Server release 5.5 (Carthage)
exa01-priv:
exa01-priv: ==> /etc/redhat-release <==
exa01-priv: Red Hat Enterprise Linux Server release 5.5 (Tikanga)
exa01-priv: ——————————– DMI ——————————–
exa01-priv: BIOS Information
exa01-priv: Vendor: American Megatrends Inc.
exa01-priv: Version: 08060108
exa01-priv: Release Date: 12/27/2010
exa01-priv: System Information
exa01-priv: Manufacturer: SUN MICROSYSTEMS
exa01-priv: Product Name: SUN FIRE X4170 M2 SERVER
exa01-priv: Serial Number: 1111ABC123
exa01-priv: ——————————– IPMI/ILOM ——————————–
exa01-priv: ipmitool version 1.8.10.3
exa01-priv: system_description = SUN FIRE X4170 M2 SERVER, ILOM v3.0.14.11.b, r62978
exa01-priv: system_identifier = Exadata Database Machine X2-2 AK00014603
exa01-priv: product_serial_number = 111ABC123EFG
exa01-priv: ——————————– Exadata Image History ——————————–
exa01-priv: Version : 11.2.2.3.5.110815
exa01-priv: Image activation date : 2011-08-30 06:18:15 -0500
exa01-priv: Imaging mode : fresh
exa01-priv: Imaging status : success
exa01-priv:
exa01-priv: ================================ MultiDisk (SW RAID) Info ================================
exa01-priv: ================================ MegaRAID ================================
exa01-priv: MegaCLI SAS RAID Management Tool Ver 8.00.23 May 17, 2010
exa01-priv: ——————————– BBU Status ——————————–
exa01-priv: Absolute state of charge: 0 %
exa01-priv: Absolute State of charge: 83 %
exa01-priv: Average Time to full: Battery is not being charged
exa01-priv: Battery FRU: N/A
exa01-priv: Battery Pack Missing : No
exa01-priv: Battery Replacement required : No
exa01-priv: Battery state:
exa01-priv: BatteryType: iBBU08
exa01-priv: Charging current: 0 mA
exa01-priv: Charging Status : None
exa01-priv: Charging Terminated : No
exa01-priv: Cycle Count: 0
exa01-priv: Date of Manufacture: 02/23, 2013
exa01-priv: Design Capacity: 1500 mAh
exa01-priv: Design Voltage: 4100 mV
exa01-priv: Discharging : No
exa01-priv: Full Charge Capacity: 1317 mAh
exa01-priv: Fully Charged : No
exa01-priv: Fully Discharged : No
exa01-priv: Learn Cycle Active : No
exa01-priv: Learn Cycle Requested : No
exa01-priv: Learn Cycle Status : OK
exa01-priv: Learn Cycle Timeout : No
exa01-priv: Max Error: 0 %
exa01-priv: Over Charged : No
exa01-priv: Over Temperature : No
exa01-priv: Relative State of Charge: 95 %
exa01-priv: Remaining Capacity: 1248 mAh
exa01-priv: Remaining Capacity Alarm: 0 mAh
exa01-priv: Remaining Capacity Alarm: No
exa01-priv: Remaining Capacity Low : No
exa01-priv: Run time to empty: Battery is not being discharged
exa01-priv: Temperature: 44 C
exa01-priv: Temperature : OK
exa01-priv: Voltage: 4040 mV
exa01-priv: Voltage : OK
exa01-priv: ——————————– Disk Status ——————————–
exa01-priv: Adapter: 0
exa01-priv: Product Name: LSI MegaRAID SAS 9261-8i
exa01-priv: BBU: Present
exa01-priv: Number of DISK GROUPS: 1
exa01-priv:
exa01-priv: DISK GROUP: 0
exa01-priv: Virtual Drive: 0 (Target Id: 0)
exa01-priv: RAID Level : Primary-5, Secondary-0
exa01-priv: Size : 557.75 GB
exa01-priv: State : Optimal
exa01-priv: Number Of Drives : 3
exa01-priv: Default Cache Policy: WriteBack, ReadAheadNone
exa01-priv: Current Cache Policy: WriteBack, ReadAheadNone
exa01-priv: Disk Cache Policy : Disabled
exa01-priv: Slot Number: 0
exa01-priv: Media Error Count: 0
exa01-priv: Other Error Count: 0
exa01-priv: Predictive Failure Count: 0
exa01-priv: Firmware state: Online, Spun Up
exa01-priv: Inquiry Data: SEAGATE ST930003SSUN300G0B701109735H0L
exa01-priv: Foreign State: None
exa01-priv:
exa01-priv: Slot Number: 1
exa01-priv: Media Error Count: 0
exa01-priv: Other Error Count: 0
exa01-priv: Predictive Failure Count: 0
exa01-priv: Firmware state: Online, Spun Up
exa01-priv: Inquiry Data: SEAGATE ST930003SSUN300G0B701110738G7Q
exa01-priv: Foreign State: None
exa01-priv:
exa01-priv: Slot Number: 2
exa01-priv: Media Error Count: 0
exa01-priv: Other Error Count: 0
exa01-priv: Predictive Failure Count: 0
exa01-priv: Firmware state: Online, Spun Up
exa01-priv: Inquiry Data: SEAGATE ST930003SSUN300G0B7011097348K5
exa01-priv: Foreign State: None
exa01-priv:
exa01-priv: ================================ Flash Disks ================================
exa01-priv: ——————————– Physical ——————————–
exa01-priv: Name FW Ver LUN Status Serial#
exa01-priv: FLASH_1_0 D20Y 1_0 normal 111ABC123E
exa01-priv: FLASH_1_1 D20Y 1_1 normal 111ABC123E
exa01-priv: FLASH_1_2 D20Y 1_2 normal 111ABC123E
exa01-priv: FLASH_1_3 D20Y 1_3 normal 111ABC123E
exa01-priv: FLASH_2_0 D20Y 2_0 normal 111ABC123E
exa01-priv: FLASH_2_1 D20Y 2_1 normal 111ABC123E
exa01-priv: FLASH_2_2 D20Y 2_2 normal 111ABC123E
exa01-priv: FLASH_2_3 D20Y 2_3 normal 111ABC123E
exa01-priv: FLASH_4_0 D20Y 4_0 normal 111ABC123E
exa01-priv: FLASH_4_1 D20Y 4_1 normal 111ABC123E
exa01-priv: FLASH_4_2 D20Y 4_2 normal 111ABC123E
exa01-priv: FLASH_4_3 D20Y 4_3 normal 111ABC123E
exa01-priv: FLASH_5_0 D20Y 5_0 normal 111ABC123E
exa01-priv: FLASH_5_1 D20Y 5_1 normal 111ABC123E
exa01-priv: FLASH_5_2 D20Y 5_2 normal 111ABC123E
exa01-priv: FLASH_5_3 D20Y 5_3 normal 111ABC123E
exa01-priv: ——————————– LUNs ——————————–
exa01-priv: Name Cell Disk Status Phys Disk
exa01-priv: 1_0 FD_00_exa01-priv normal FLASH_1_0
exa01-priv: 1_1 FD_01_exa01-priv normal FLASH_1_1
exa01-priv: 1_2 FD_02_exa01-priv normal FLASH_1_2
exa01-priv: 1_3 FD_03_exa01-priv normal FLASH_1_3
exa01-priv: 2_0 FD_04_exa01-priv normal FLASH_2_0
exa01-priv: 2_1 FD_05_exa01-priv normal FLASH_2_1
exa01-priv: 2_2 FD_06_exa01-priv normal FLASH_2_2
exa01-priv: 2_3 FD_07_exa01-priv normal FLASH_2_3
exa01-priv: 4_0 FD_08_exa01-priv normal FLASH_4_0
exa01-priv: 4_1 FD_09_exa01-priv normal FLASH_4_1
exa01-priv: 4_2 FD_10_exa01-priv normal FLASH_4_2
exa01-priv: 4_3 FD_11_exa01-priv normal FLASH_4_3
exa01-priv: 5_0 FD_12_exa01-priv normal FLASH_5_0
exa01-priv: 5_1 FD_13_exa01-priv normal FLASH_5_1
exa01-priv: 5_2 FD_14_exa01-priv normal FLASH_5_2
exa01-priv: 5_3 FD_15_exa01-priv normal FLASH_5_3
exa01-priv: ——————————– Cell ——————————–
exa01-priv: Name LUN Status Size
exa01-priv: FD_00_exa01-priv 1_0 normal 22.875G
exa01-priv: FD_01_exa01-priv 1_1 normal 22.875G
exa01-priv: FD_02_exa01-priv 1_2 normal 22.875G
exa01-priv: FD_03_exa01-priv 1_3 normal 22.875G
exa01-priv: FD_04_exa01-priv 2_0 normal 22.875G
exa01-priv: FD_05_exa01-priv 2_1 normal 22.875G
exa01-priv: FD_06_exa01-priv 2_2 normal 22.875G
exa01-priv: FD_07_exa01-priv 2_3 normal 22.875G
exa01-priv: FD_08_exa01-priv 4_0 normal 22.875G
exa01-priv: FD_09_exa01-priv 4_1 normal 22.875G
exa01-priv: FD_10_exa01-priv 4_2 normal 22.875G
exa01-priv: FD_11_exa01-priv 4_3 normal 22.875G
exa01-priv: FD_12_exa01-priv 5_0 normal 22.875G
exa01-priv: FD_13_exa01-priv 5_1 normal 22.875G
exa01-priv: FD_14_exa01-priv 5_2 normal 22.875G
exa01-priv: FD_15_exa01-priv 5_3 normal 22.875G
exa01-priv: ——————————– Grid ——————————–
exa01-priv: Name Status Size ASM Mode, DeActOut
exa01-priv: ================================ Flash Cache ================================
exa01-priv: Name Status Size Effective
exa01-priv: exa01-priv_FLASHCACHE normal 365.25G 365.25G
…
exa01-priv: ================================ End Of Report ================================
Comments are closed