VMware KB (2007236): Storage device performance deteriorated

Posted on by

Storage device performance deteriorated (2007236)

Details

This message is received when the latency on the device is higher than the average latency. The device latency increased which generated this event. This occurs when either the latency ratio to the last time the log was updated is 30 or if the ratio doubled since the last log.

Some of the possible reasons why the device latency increased are:

  • Changes made on the target
  • Disk or media failures
  • Overload conditions on the device
  • Failover

For more/related information, see Information regarding vSphere storage device latency performance metrics (2001676).

Impact

The storage device performance declines.

Example

This is an example of the message received when this occurs:

Device naa.5000c5000b36354b performance has deteriorated. I/O latency increased from average value of 1832 microseconds to 19403 microsecond

Note: The message shows microseconds which can be converted to milliseconds: 19403 microseconds = 19.403 milliseconds.

Solution

If you observe that this latency is too high for a consistent period of time, this indicates that there is a concern about storage performance and you must check the logs on the storage array for any indication of a failure. If failures are logged on the storage array side, corrective action should be taken. Contact your storage vendor for information regarding checking logs on the array.

Also check if these messages are generated when there were any scheduled tasks, such as backups, replications, etc., as these can also cause intermittent performance hits.

If the message is generated because of an overload condition, attempt to reduce the load on the affected storage device. If the storage device is overloaded, reduce the load on the storage device.

If running a LUN replication tool, pause the task from the storage end and attempt a storage vMotion to a different datastore. This should help improve the I/O operations.

For more/related information on diagnosing overload/performance issues, see Using esxtop to identify storage performance issues (1008205).

Leave a comment