ESX Cluster Drops Network Connectivity

Hello Guys,

im in really big trouble!

I have a cluster of 3 ESX Server with HA and DRS.

So far all works good. But sind last Thursday i have a mega problem.

Ramdomly one ESX Server looses a vmnic connection.

Mostly the vmnic witch handes the traffic internal of the VMs (more traffice).

The other VMnic vor Vmotion or External connection still work.

Now im moving non stop the VMs arround (witch work after direcly, so the vms are ok) when the error occour.

In the log i see this:

Jul 21 11:58:31 esx03 vmkernel: 5:16:14:24.813 cpu1:1025)WARNING: LinNet: 2035: Watchdog timeout for device vmnic0

Jul 21 11:58:31 esx03 vmkernel: 5:16:14:24.813 cpu1:1025)<6>vmnic0: Got tx_timeout. irq: 00000000

Jul 21 11:58:31 esx03 vmkernel: 5:16:14:24.813 cpu1:1025)<6>vmnic0: Ring at 3d542800: get 3d542ad0 put 3d542ad0

Jul 21 11:58:31 esx03 vmkernel: 5:16:14:24.813 cpu1:1025)<6>vmnic0: Dumping tx registers

Jul 21 11:58:31 esx03 vmkernel: 5:16:14:24.813 cpu1:1025)<6>  0: 00000000 00000060 00000003 0006000d 00000000 00000000 00000000 00000000

Jul 21 11:58:31 esx03 vmkernel: 5:16:14:24.813 cpu1:1025)<6> 20: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000

Jul 21 11:58:31 esx03 vmkernel: 5:16:14:24.813 cpu1:1025)<6> 40: 0420e20e 0000a855 00002e20 00000000 00000000 00000000 00000000 00000000

Jul 21 11:58:31 esx03 vmkernel: 5:16:14:24.813 cpu1:1025)<6> 60: 00000000 00000000 00000000 0000ffff 0000ffff 0000ffff 0000ffff 00000000

Jul 21 11:58:31 esx03 vmkernel: 5:16:14:24.813 cpu1:1025)<6> 80: 003b0f3c 40040001 00000000 007f0080 0000061c 00000001 00080000 00007fc6

Jul 21 11:58:31 esx03 vmkernel: 5:16:14:24.813 cpu1:1025)<6> a0: 0014050f 00000016 7581e000 0000f6c3 00000001 00000000 00000000 00000000

Jul 21 11:58:31 esx03 vmkernel: 5:16:14:24.813 cpu1:1025)<6> c0: 10000002 00000001 00000001 00000001 00000001 00000001 00000001 00000001

Jul 21 11:58:31 esx03 vmkernel: 5:16:14:24.813 cpu1:1025)<6> e0: 00000001 00000001 00000001 00000001 00000001 00000001 00000001 00000001

Jul 21 11:58:31 esx03 vmkernel: 5:16:14:24.813 cpu1:1025)<6>100: 40f42800 40f42000 007f003f 00008000 00010032 00000000 00000019 40f42ad0

Jul 21 11:58:31 esx03 vmkernel: 5:16:14:24.813 cpu1:1025)<6>120: 40f42340 051d4bc0 a000ffc9 4025c1b0 8000061c 40f42adc 40f422a0 0fe08000

Jul 21 11:58:31 esx03 vmkernel: 5:16:14:24.813 cpu1:1025)<6>140: 00304120 80c02600 00000000 00000000 00000004 00000000 00000000 00000000

Jul 21 11:58:31 esx03 vmkernel: 5:16:14:24.813 cpu1:1025)<6>160: 00000000 00000000 00000000 00000000 01ff0080 0000c000 00000000 00000000

Jul 21 11:58:31 esx03 vmkernel: 5:16:14:24.813 cpu1:1025)<6>180: 00000006 00000008 0294796d 00008103 0000004a 00003c00 00000080 0000bd83

Jul 21 11:58:31 esx03 vmkernel: 5:16:14:24.813 cpu1:1025)<6>1a0: 00000006 00000008 0294796d 00008103 0000004a 00003c00 00000080 0000bd83

Jul 21 11:58:31 esx03 vmkernel: 5:16:14:24.813 cpu1:1025)<6>1c0: 00000006 00000008 0294796d 00008103 0000004a 00003c00 00000080 0000bd83

Jul 21 11:58:31 esx03 vmkernel: 5:16:14:24.813 cpu1:1025)<6>1e0: 00000006 00000008 0294796d 00008103 0000004a 00003c00 00000080 0000bd83

Jul 21 11:58:31 esx03 vmkernel: 5:16:14:24.813 cpu1:1025)<6>200: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000

Jul 21 11:58:31 esx03 vmkernel: 5:16:14:24.813 cpu1:1025)<6>220: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000

Jul 21 11:58:31 esx03 vmkernel: 5:16:14:24.813 cpu1:1025)<6>240: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000

Jul 21 11:58:31 esx03 vmkernel: 5:16:14:24.813 cpu1:1025)<6>260: 00000000 00000000 fe027001 00000100 00000011 000000a3 fe027011 000001a3

Jul 21 11:58:31 esx03 vmkernel: 5:16:14:24.813 cpu1:1025)<6>280: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000

Jul 21 11:58:31 esx03 vmkernel: 5:16:14:24.813 cpu1:1025)<6>2a0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000

Jul 21 11:58:31 esx03 vmkernel: 5:16:14:24.813 cpu1:1025)<6>2c0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 0000012d

Jul 21 11:58:31 esx03 vmkernel: 5:16:14:24.813 cpu1:1025)<6>2e0: 00000000 00000000 00000000 00000000 00000000 00000001 00000001 00000001

Jul 21 11:58:31 esx03 vmkernel: 5:16:14:24.813 cpu1:1025)<6>300: 80212000 00000000 00000000 00000000 00000000 00002000 00000000 00000000

Jul 21 11:58:31 esx03 vmkernel: 5:16:14:24.813 cpu1:1025)<6>320: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000

Jul 21 11:58:31 esx03 vmkernel: 5:16:14:24.813 cpu1:1025)<6>340: 00000000 00000000 00000000 00000000 00000000 00000020 3a7af9e3 00000000

Jul 21 11:58:31 esx03 vmkernel: 5:16:14:24.813 cpu1:1025)<6>360: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000

Jul 21 11:58:31 esx03 vmkernel: 5:16:14:24.813 cpu1:1025)<6>380: 00000000 00000000 00000000 00000000 00000000 00000000 00000002 00000000

Jul 21 11:58:31 esx03 vmkernel: 5:16:14:24.813 cpu1:1025)<6>3a0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000

Jul 21 11:58:31 esx03 vmkernel: 5:16:14:24.813 cpu1:1025)<6>3c0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000

Jul 21 11:58:31 esx03 vmkernel: 5:16:14:24.813 cpu1:1025)<6>3e0: 06255300 00701365 00000000 00000000 00000000 00000000 00000000 00000000

Jul 21 11:58:31 esx03 vmkernel: 5:16:14:24.813 cpu1:1025)<6>400: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000

Jul 21 11:58:31 esx03 vmkernel: 5:16:14:24.813 cpu1:1025)<6>420: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000

Jul 21 11:58:31 esx03 vmkernel: 5:16:14:24.813 cpu1:1025)<6>440: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000

Jul 21 11:58:31 esx03 vmkernel: 5:16:14:24.813 cpu1:1025)<6>460: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000

Jul 21 11:58:31 esx03 vmkernel: 5:16:14:24.813 cpu1:1025)<6>480: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000

Jul 21 11:58:31 esx03 vmkernel: 5:16:14:24.813 cpu1:1025)<6>4a0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000

Jul 21 11:58:31 esx03 vmkernel: 5:16:14:24.813 cpu1:1025)<6>4c0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000

Jul 21 11:58:31 esx03 vmkernel: 5:16:14:24.813 cpu1:1025)<6>4e0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000

Jul 21 11:58:31 esx03 vmkernel: 5:16:14:24.813 cpu1:1025)<6>500: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000

Jul 21 11:58:31 esx03 vmkernel: 5:16:14:24.813 cpu1:1025)<6>520: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000

Jul 21 11:58:31 esx03 vmkernel: 5:16:14:24.813 cpu1:1025)<6>540: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000

Jul 21 11:58:31 esx03 vmkernel: 5:16:14:24.813 cpu1:1025)<6>560: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000

Jul 21 11:58:31 esx03 vmkernel: 5:16:14:24.813 cpu1:1025)<6>580: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000

Jul 21 11:58:31 esx03 vmkernel: 5:16:14:24.813 cpu1:1025)<6>5a0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000

Jul 21 11:58:31 esx03 vmkernel: 5:16:14:24.813 cpu1:1025)<6>5c0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000

Jul 21 11:58:31 esx03 vmkernel: 5:16:14:24.813 cpu1:1025)<6>5e0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000

Jul 21 11:58:31 esx03 vmkernel: 5:16:14:24.813 cpu1:1025)<6>600: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000

Jul 21 11:58:31 esx03 vmkernel: 5:16:14:24.813 cpu1:1025)<6>vmnic0: Dumping tx ring

Jul 21 11:58:31 esx03 vmkernel: 5:16:14:24.813 cpu1:1025)<6>000: 00000003 fe24402e 00000000 // 00000004 0c58fb88 00000000 // 00000003 96b12580 00000000 // 00000003 5d5db900 00000000

Jul 21 11:58:31 esx03 vmkernel: 5:16:14:24.813 cpu1:1025)<6>004: 00000003 93333f00 20000124 // 00000000 402edb98 00000000 // 00000003 fe24482e 00000000 // 00000004 0c58f8c8 00000000

Jul 21 11:58:31 esx03 vmkernel: 5:16:14:24.813 cpu1:1025)<6>008: 00000003 dc0efdc0 00000000 // 00000003 965ec040 00000000 // 00000003 52bbef00 20000124 // 00000000 403d5e90 00000000

Jul 21 11:58:31 esx03 vmkernel: 5:16:14:24.813 cpu1:1025)<6>00c: 00000002 91a4502e 00000000 // 00000004 0c58f558 00000000 // 00000003 7772d980 00000000 // 00000001 cab3a900 00000000

Jul 21 11:58:31 esx03 vmkernel: 5:16:14:24.813 cpu1:1025)<6>010: 00000002 7dec1740 20000124 // 00000000 d7a4d9e0 00000000 // 00000002 91a4582e 00000000 // 00000004 0c58f818 00000000

Jul 21 11:58:31 esx03 vmkernel: 5:16:14:24.813 cpu1:1025)<6>014: 00000003 bf437780 00000000 // 00000003 3ddf3a40 00000000 // 00000003 4fb18040 20000124 // 00000000 4020f8c0 00000000

Jul 21 11:58:31 esx03 vmkernel: 5:16:14:24.813 cpu1:1025)<6>018: 00000003 0776602e 00000000 // 00000004 0c58f348 00000000 // 00000003 c84abf00 00000000 // 00000003 8584ec80 00000000

Jul 21 11:58:31 esx03 vmkernel: 5:16:14:24.813 cpu1:1025)<6>01c: 00000003 a59c1900 20000124 // 00000000 4025f898 00000000 // 00000003 0776682e 00000000 // 00000004 0c58fad8 00000000

Jul 21 11:58:31 esx03 vmkernel: 5:16:14:24.813 cpu1:1025)<6>020: 00000002 91e86540 20000090 // 00000000 d7a929a0 00000000 // 00000003 45fa702e 00000000 // 00000004 0c58f138 00000000

Jul 21 11:58:31 esx03 vmkernel: 5:16:14:24.813 cpu1:1025)<6>024: 00000003 eb2d1540 00000000 // 00000003 2c558040 00000000 // 00000002 9bb2f6c0 20000124 // 00000000 d7a13628 00000000

Jul 21 11:58:31 esx03 vmkernel: 5:16:14:24.813 cpu1:1025)<6>028: 00000003 45fa782e 00000000 // 00000004 0c58f608 00000000 // 00000003 2c5587c0 00000000 // 00000003 52bbed40 00000000

Jul 21 11:58:31 esx03 vmkernel: 5:16:14:24.813 cpu1:1025)<6>02c: 00000004 051d4b40 20000124 // 00000000 4034e2c0 2000059e // 00000000 40304d48 00000000 // 00000002 a6be202e 20000530

Jul 21 11:58:31 esx03 vmkernel: 5:16:14:24.813 cpu1:1025)<6>030: 00000000 402d7378 00000000 // 00000002 a6be282e 2000032a // 00000000 d7aa9558 00000000 // 00000002 c416302e 00000000

Jul 21 11:58:31 esx03 vmkernel: 5:16:14:24.813 cpu1:1025)<6>034: 00000004 0c58fce8 00000000 // 00000003 c458fdc0 00000000 // 00000003 2c558540 00000000 // 00000002 f92dea80 20000124

Jul 21 11:58:31 esx03 vmkernel: 5:16:14:24.813 cpu1:1025)<6>038: 00000000 d7a8c070 00000000 // 00000002 c416382e 00000000 // 00000004 0c58fc38 00000000 // 00000003 57b33f00 00000000

Jul 21 11:58:31 esx03 vmkernel: 5:16:14:24.814 cpu1:1025)<6>03c: 00000003 9828cd40 00000000 // 00000003 02815f00 00000000 // 00000003 8d467d40 2000016e // 00000000 d7acad98 00000000

VMnic0 is accualy the NIC for my internal Network.

What can be the problem? Is there any solution i can do temp. till i have fix it?

with many thx.

Stefan