incorrect fan status showing when running plugin from command line

[Vani2468]

We are using check_hpasm plugin to check HP proliant hardware Monitoring, but when we run plugin from command line we are getting critical message even though in ILO there is no issue.

OMD[testworker@XXXX]:$ /omd/plugins/check_hpasm --hostname testhost --community test --perfdata -t 600 --eventrange=1d/12h
CRITICAL - fan 3 (system) needs attention, fan 4 (system) needs attention, fan 5 (system) needs attention, fan 6 (system) needs attention, fan 7 (system) needs attention, System: 'proliant dl360 gen10', S/N: 'test', ROM: 'U32 v3.30' | pc_1=55;800;800 pc_2=28;800;800 fan_1=0% fan_2=0% fan_3=50% fan_4=50% fan_5=50% fan_6=50% fan_7=50% temp_1_ambient=21;42;42 temp_2_cpu=40;70;70 temp_6_memory=28;90;90 temp_12_system=35;60;60 temp_14_system=26;60;60 temp_15_ambient=24;60;60 temp_16_system=32;115;115 temp_18_system=27;115;115 temp_19_system=29;115;115 temp_22_system=38;100;100 temp_23_system=67;110;110 temp_24_system=39;90;90 temp_25_system=57;100;100 temp_26_system=40;85;85 temp_28_ioBoard=40;100;100 temp_29_system=31;90;90 temp_30_ioBoard=63;100;100 temp_31_ioBoard=37;90;90 temp_32_ioBoard=75;100;100 temp_33_ioBoard=33;90;90 temp_38_system=35;75;75 temp_43_powerSupply=23;100;100 temp_44_powerSupply=28;75;75
OMD[vieworker@XXXX]:$

Kindly advice how to fix the issue to get the OK state in the output.

[codeautopilot]

Potential solution

The issue seems to stem from a mismatch between the SNMP data interpretation in the plugin and the actual fan status as reported by the ILO interface. The solution involves ensuring that the SNMP data retrieved by the plugin accurately reflects the fan status and that the logic used to interpret this data aligns with the ILO's reporting. This can be achieved by verifying the SNMP OIDs, adjusting the logic for status interpretation, and ensuring that redundancy and custom thresholds are correctly handled.

What is causing this bug?

The bug is likely caused by one or more of the following issues:

1. SNMP OID Mismatch: The SNMP OIDs used in the plugin might not match those used by the specific server model or firmware version, leading to incorrect data retrieval.

2. Incorrect Status Mapping: The mapping of SNMP values to status strings might not accurately reflect all possible values returned by the SNMP query, causing incorrect status reporting.

3. Redundancy Handling: The logic for handling fan redundancy might be misinterpreting the redundancy status, leading to false positives in fan status.

4. Custom Thresholds: Incorrect custom thresholds might cause the plugin to report a CRITICAL status even when the hardware is functioning correctly.

5. Message Handling Logic: The logic for determining when to add CRITICAL messages might be too broad, leading to inaccurate status reporting.

Code

To address these issues, consider the following code adjustments:

1. Verify SNMP OIDs: Ensure that the SNMP OIDs used in the FanSubsystem/SNMP.pm file match those used by the specific server model and firmware version.

2. Adjust Status Mapping: Update the status mapping logic in the FanSubsystem.pm file to accurately reflect all possible SNMP values.

3. Review Redundancy Handling: Check the logic for handling fan redundancy in the check_hpasm.pl script and ensure it accurately reflects the server's configuration.

4. Verify Custom Thresholds: Ensure that any custom thresholds set for fan speeds or temperatures are appropriate for the specific server model.

5. Improve Message Specificity: Refine the logic for adding CRITICAL messages in the check_hpasm.pl script to provide more specific information about fan issues.

Example code snippet for adjusting status mapping:

# Example adjustment in FanSubsystem.pm
sub check {
    my $self = shift;
    # Adjust the logic to accurately reflect fan status
    if ($self->{cpqHeFltTolFanSpeed} eq 'high' && $self->{expected_condition}) {
        # Adjust condition to match expected behavior
        $self->set_status('OK');
    } elsif ($self->{cpqHeFltTolFanSpeed} eq 'failed') {
        $self->set_status('CRITICAL');
    }
    # Additional conditions as needed
}

How to replicate the bug

To replicate the bug, follow these steps:

1. Run the check_hpasm plugin from the command line with the specified arguments:

   /omd/plugins/check_hpasm --hostname testhost --community test --perfdata -t 600 --eventrange=1d/12h

2. Observe the output for a CRITICAL message regarding fan status, even though the ILO interface reports no issues.

3. Verify the SNMP data retrieved by the plugin and compare it with the ILO's fan status to identify discrepancies.

By following these steps, you should be able to replicate the issue and test the effectiveness of the proposed solution.

Click here to create a Pull Request with the proposed solution

Files used for this task:

Changes on plugins-scripts/HP/Proliant/Component/FanSubsystem.pm

Based on the provided code from the FanSubsystem.pm file, here are some observations and potential areas that could be causing the incorrect fan status reporting:

1. Fan Status Interpretation:

   • The check method in the HP::Proliant::Component::FanSubsystem::Fan::FTol class interprets the fan status based on several conditions. It checks for conditions like failed, degraded, and others to determine if a fan is in a CRITICAL or WARNING state.
   • The logic for determining if a fan is CRITICAL or WARNING might not align with the actual status reported by ILO. For example, the condition if ($self->{cpqHeFltTolFanSpeed} eq 'high') triggers a CRITICAL message, which might not be appropriate if high speed is normal under certain conditions.

2. Redundancy Handling:

   • The code checks for fan redundancy and sets the status to notRedundant if certain conditions are met. This logic might be misinterpreting the redundancy status, especially if the ILO reports redundancy differently.

3. Fan Presence and Condition:

   • The code checks if a fan is present, failed, or absent and assigns statuses accordingly. If the SNMP data or ILO reports these states differently, it could lead to incorrect status reporting.

4. Performance Data:

   • The performance data is added based on the fan speed and type. If the fan speed is misinterpreted (e.g., due to incorrect parsing of SNMP data), it could lead to incorrect performance data and status messages.

5. Overall Health Check:

   • The overall_check method is called to determine the overall health of the fans. If this method's logic is flawed or if it doesn't align with ILO's health reporting, it could lead to discrepancies.

Recommendations:

• Verify SNMP Data: Ensure that the SNMP data being retrieved accurately reflects the fan status as reported by ILO. This might involve checking the SNMP OIDs and their values.
• Align Logic with ILO: Review the conditions used to determine fan status and ensure they align with how ILO reports fan health. This might involve adjusting thresholds or conditions.
• Debugging and Logging: Add additional logging to capture the raw SNMP data and the interpreted status to help identify where the discrepancy occurs.
• Consult Documentation: Review HP's documentation for SNMP MIBs related to fan status to ensure the plugin's logic aligns with the expected data structure and values.

This analysis focuses on the logic within this file. Further investigation might be needed in other parts of the codebase, especially in how SNMP data is retrieved and processed.

Changes on plugins-scripts/HP/Proliant/Component/FanSubsystem/SNMP.pm

Based on the provided code, the HP::Proliant::Component::FanSubsystem::SNMP module is responsible for retrieving and interpreting SNMP data related to the fan subsystem of HP Proliant servers. The issue reported is that the command-line plugin is showing a critical status for fans, even though the ILO (Integrated Lights-Out) interface does not report any issues.

Here are some potential causes for the discrepancy:

1. SNMP OID Mismatch: The SNMP OIDs used in the script might not match the actual OIDs used by the specific server model or firmware version. This could lead to incorrect data being retrieved.

2. Incorrect Status Mapping: The mapping of SNMP values to status strings (e.g., 'ok', 'degraded', 'failed') might not be accurate or comprehensive for all possible values returned by the SNMP query.

3. Data Processing Logic: The logic that processes the SNMP data and determines the overall status might not be correctly handling certain conditions. For example, the overall_check method checks both sysstatus and cpustatus, but there might be edge cases not accounted for.

4. Redundancy Handling: The script includes logic for handling redundant fans, but if redundancy is not correctly detected or handled, it might incorrectly report a critical status.

5. Case Sensitivity: The script uses bitwise OR with lc (lowercase conversion) on status values, which seems unnecessary and could potentially cause issues if not handled correctly.

6. Incomplete Data: If the SNMP walk does not return all expected data, the script might default to a critical status due to missing information.

To address these potential issues, consider the following steps:

• Verify that the SNMP OIDs used in the script match those used by the specific server model and firmware version.
• Ensure that the status mapping accurately reflects all possible SNMP values.
• Review the logic in the overall_check method to ensure it correctly interprets the combined status of system and CPU fans.
• Check the handling of redundant fans to ensure it accurately reflects the server's configuration.
• Remove or review the use of bitwise OR with lc on status values to ensure it does not introduce errors.
• Ensure that the SNMP walk retrieves all necessary data and handle cases where data might be missing.

These steps should help identify and resolve the discrepancy between the command-line plugin output and the ILO interface.

Changes on plugins-scripts/check_hpasm.pl

Based on the provided source code of the check_hpasm.pl script, here is an analysis and potential causes for the bug reported:

Analysis

1. Plugin Initialization and Argument Parsing:

   • The script initializes a Nagios::MiniPlugin object and sets up various arguments for SNMP communication and other configurations.
   • It includes options for handling fan redundancy and custom thresholds, which might be relevant to the fan status issue.

2. Timeout Handling:

   • The script sets a timeout for the plugin execution, which is handled by an alarm signal. If the script times out, it exits with an UNKNOWN status.

3. Server Object Initialization:

   • An HP::Server object is created, which seems to handle the core logic for interacting with the server hardware.
   • The options passed to this object include ignore_fan_redundancy, which might affect how fan statuses are interpreted.

4. Message Handling:

   • The script uses check_messages to determine the overall status. If no issues are detected, it adds an OK message.
   • If there are issues, it adds a CRITICAL message with 'wrong device', which might not be the most descriptive for fan issues.

5. Output and Exit:

   • The script constructs a message based on the collected messages and exits with the appropriate Nagios status code.

Potential Causes for the Bug

1. Fan Redundancy Option:

   • The ignore_fan_redundancy option might be incorrectly set or interpreted, leading to false positives in fan status.

2. Custom Thresholds:

   • If custom thresholds are set incorrectly, they might cause the script to report a CRITICAL status even when the hardware is functioning correctly.

3. Message Handling Logic:

   • The logic for determining when to add CRITICAL messages might be too broad or not specific enough to accurately reflect the fan status.

4. Integration with HP::Server:

   • The interaction between the check_hpasm.pl script and the HP::Server object might have issues, particularly in how fan data is retrieved and interpreted.

Recommendations

• Review Fan Redundancy Handling: Check if the ignore_fan_redundancy option is being used correctly and whether it should be enabled or disabled based on the hardware configuration.
• Verify Custom Thresholds: Ensure that any custom thresholds set for fan speeds or temperatures are appropriate for the specific server model.
• Improve Message Specificity: Consider refining the logic for adding CRITICAL messages to provide more specific information about fan issues.
• Debug SNMP Data Retrieval: Investigate how the HP::Server object retrieves and processes SNMP data related to fan status to ensure accuracy.

This analysis focuses on the check_hpasm.pl script and its potential role in the reported bug. Further investigation into related files, such as those handling SNMP data retrieval and fan status interpretation, may be necessary.

This comment was generated by AI. Information provided may be incorrect.

Current plan usage: 8%

Have feedback or need help?
Documentation
[email protected]

[lausser]

Run check_hpasm-vv to see what theilo board reports viia snmp.
Use the latest code.