EqualLogics Latency Update

After spending some time wading through a lot of stats and figures and checking the configuration on all the switches we have partly resolved the problem at our most badly affected site.
This always seemed to have extra issues as the problems seemed sporadic and didn't follow the same pattern we saw at other sites.
We traced to the problem to two ports that were connected in to the second newer EqualLogic unit. These were showing a number of errors on the switch end. Shutting down these two ports stopped all the latency emails we were getting and now we are just getting the normal pattern were a number of servers all email about latency at the same time rather than random emails from different servers.

I have yet to visit the site and ascertain whether it is a problem with the switch, controller or the patch lead.
But it just goes to show that if you start at one end and work your way through the path, make sure every setting is what it should be and above all never assume anything, something will turn up.
I would recommend reading up on any configuration guides for iSCSI and specifically EqualLogics and any switches that you use, we found one particular good one related to Cisco 3750 switches from Dell that listed all the settings you need to configure a switch correctly, QOS was something we had not previously configured.

For a 3750 the following should work

1. Clear the current configuration.

switch>enable

switch#write erase

switch#reload

2. Set up the out-of-band management interface.

switch>enable

switch#config

switch(config)#int fastethernet 0

switch(config-if)#ip address 192.168.x.y 255.255.255.0

switch(config-if)#no shut

switch(config-if)#exit

switch(config)#ip default-gateway 192.168.x.y

3. Set up the ports.

switch(config)#int range gig 1/0/1-48, gig 2/0/1-48

switch(config-if-range)#flowcontrol receive on

switch(config-if-range)#spanning-tree portfast

switch(config-if-range)#no storm-control unicast level

switch(config-if-range)#no shut

switch(config-if-range)#exit

switch(config)#system mtu jumbo 9198

4. Configure QOS and optimize buffers for EQL iSCSI use.

switch(config)#mls qos queue-set output 1 buffers 4 88 4 4

switch(config)#mls qos queue-set output 1 threshold 1 100 100 100 400

switch(config)#mls qos queue-set output 1 threshold 2 3200 100 10 3200

switch(config)#mls qos queue-set output 1 threshold 3 100 100 100 400

switch(config)#mls qos queue-set output 1 threshold 4 100 100 100 400

5. Configure telnet access.

switch(config)#enable password 0 yourpassword

switch(config)#line vty 0

switch(config-line)#password 0 yourpassword

switch(config-line)#exit

switch(config)#exit

6. Save the configuration and activate it.

switch#copy run start

switch#reload

If you have stacked switches the MTU is set on the stack not each switch.

We are still struggling with the 11 second period when the EqualLogic stops processing iSCSI traffic which Dell don't seem that interested in doing anything about or even acknowledging. It definitely seems to be something to do with the controllers, but at least one part is now working a lot better.