Very slow server and CONSTANT disk activity.

I have one ESX3i 3.5.0;110271 server where all the VMs and thier files are on an iSCSI SAN.  The only data on the local drives is VMware itself.   The VMs run quite slowly but are stable.  However, attempting to attach with Virtual Center or perform just about any management task (talking to ESX itself) is nightmarishly slow and usually times out.   I noticed when looking at the physical box that the HD activity lights are pegs - the drives seem to be running constantly.  I don't see this on my other ESX hosts.   But I'm rather baffled on tracking down why it is so terribly slow.   Then every one in awhile VI starts working then the server blinks out again as disconnected.

I'm able to connect to the box (it takes a long time) from VIMA's resxtop which shows a relatively quite box except this entry seems to stand out -

ID    GID NAME          NWLD   %USED    %RUN    %SYS   %WAIT    %RDY   %IDLE

334    334 hostd.1392      19   14.41   13.47    3.58 1900.00    6.86    0.00

resxtop show idle CPU time (~173%) and no memory overcommit or swapping.

Logs captured by VIMA look like:

vi-admin@vima fob-red.mormail.com$ tail hostd.log

Task Completed : haTask-ha-host-vim.DiagnosticManager.browse-1014920

Task Created : haTask-ha-host-vim.DiagnosticManager.browse-1014922

Task Completed : haTask-ha-host-vim.DiagnosticManager.browse-1014922

Hw info file: /etc/vmware/hostd/hwInfo.xml

Config target info loaded

Task Created : haTask-ha-host-vim.DiagnosticManager.browse-1014933

Task Completed : haTask-ha-host-vim.DiagnosticManager.browse-1014933

Task Created : haTask-ha-host-vim.DiagnosticManager.browse-1014936

Task Completed : haTask-ha-host-vim.DiagnosticManager.browse-1014936

Task Created : haTask-ha-host-vim.DiagnosticManager.browse-1014938

$ tail messages.log

FEB 26 08:08:44 Hostd: Task Created : haTask-ha-host-vim.DiagnosticManager.browse-1014933

FEB 26 08:08:44 Hostd: Task Completed : haTask-ha-host-vim.DiagnosticManager.browse-1014933

FEB 26 08:08:48 Hostd: Task Created : haTask-ha-host-vim.DiagnosticManager.browse-1014936

FEB 26 08:08:46 Hostd: Task Completed : haTask-ha-host-vim.DiagnosticManager.browse-1014936

FEB 26 08:08:50 Hostd: Task Created : haTask-ha-host-vim.DiagnosticManager.browse-1014938

FEB 26 08:08:52 Hostd: Task Completed : haTask-ha-host-vim.DiagnosticManager.browse-1014938

FEB 26 08:09:58 crond[1361]: USER root pid 59267725 cmd /sbin/decodeSel.sh #Every 10 minutes, translate the latest IPMI SEL data

FEB 26 08:11:30 Hostd: Task Created : haTask-ha-host-vim.DiagnosticManager.browse-1014948

FEB 26 08:11:32 Hostd: Task Completed : haTask-ha-host-vim.DiagnosticManager.browse-1014948

FEB 26 08:11:34 Hostd: Task Created : haTask-ha-host-vim.DiagnosticManager.browse-1014949

$ tail vpxa.log

Guest HeartbeatStatus Changed

Last stats polling used ms

heartbeating 192.168.1.50 ...Count 1111657

heartbeating 192.168.1.50 ...Count 1111658

heartbeating 192.168.1.50 ...Count 1111659

Received callback in WaitForUpdatesDone

Applying updates from 267442 to 267443 (at 267442)

Last stats polling used ms

Received callback in WaitForUpdatesDone

Applying updates from 267443 to 267444 (at 267443)