Am I hosed? Ran out of physical space in an ESX host's storage partition: Guest now locked up...

Many of you would know the scenario:

You goto work to fix a few things over the weekend whilst noone is there. 

At my work, I have a host that everyone uses that is soon to be transitioned to a new environment.  Backup in the meanwhile is a manual affair due to authentication and access controls, and as a result I generally do the backup on the weekends. Today however, I did it a bit different: This weekend I gzipped and movied some of the old backups to another host (normally I just create and run).

This time I thought I'd clean up the old stuff too.  (o, what great luck I had coming!)

Once everything had copied off I deleted the original backup files (mainly db dumps) which freed up about 3 GB space. Almost done and time to go home for dinner! Yay! Oh, no wait a minute, I forgot to archive the /var/www directory. So I start that (100MB) before thinking again OK, time to log out and go home.

But, waaaaBAM! My ssh session to the Guest drops, and it no longer repsonds.

Hmmm. OK, time to login to VirtualCentre to see the guest's has an exclamation mark beside it, and the terminal console locked. This is the error showing in VC's interface:

"There is no more space to write the redo log of linuxhost.vmdk. You may be able to continue this session by freeing disk space on the relevant partition, and clicking retry"

I open up a session to the host and see I have no more space left on the partition. Worse, there seems nothing unimportant left to delete, so cannot free up space. (Can anything extend an ESX 3.5 guest partition hot with a locked host?)

How it happened:

This Linux Guest was originally built on VMware server 1.05.  It was upgrded to 1.06 and then ESX 3.5.

As you probably guess, the console in the guest machine was busy tarring files at the time the phsycial filesystem ran out of space.  The space actually utilised on the guest is very small (6GB total) but the vmdk is configured at 250GB (growable).  The physical partition is only 126GB in size however.  On the previous host htis was OK as we could never see ourselves using that much space within 5 years, and by then we'd have more storage.  Thankfully, this partition holds no other Guests so nothing else is locked up, however neither is there anything I can move off (?), so perhaps some guru out there might lend some advice?

• How can I recover this machine without corrupting the guest (or host) filesystems?

• Can I stop/reset the guest?  The archive operation can be cleaned up later but I am worried about corrupting the filesystem (and really don't want to have to rebuild the OS if I can avoid it)

Sorry to be so dumb we thought we were smart moving to ESX... but didn't do our homework. This little host ran fine for years under VMware Server with nice compact files (even if it did have 250GB worth of 'potential'). ESX seems to gobble up space like it's starving for the stuff!  (see the filesizes listed below)

Oh well here I go, I'm hitting the VI guide now- need to fix this before morning

Any help lent would be greatly appreciated!

Here is a file listing, and the vmx's... if it might help:

     et.vmx                      1.26KB
                                    et.vmdk                       400b
                                    et.vmsd                       848b
                                    et.vmxf                        260b
                                    et-SnapShot18.vmsn    2GB
                                    et-e73ff811.vswp          2GB
                                    et-flat.vmdk              125GB
                                    et-000001.vmdk           246b
                                    et-000001-delta.vmdk   2GB

(BTW; 'ET' is the name of the Guest)

This is what et.vmx looks like:

     ethernet0.generatedAddressOffset = "0"
                                    tools.syncTime = "FALSE"
                                    checkpoint.vmState = ""
                                    autostart = "poweron"
                                    Ethernet0.startConnected = "true"
                                    scsi0:0.deviceType = "scsi-hardDisk"
                                    extendedConfigFile = "et.vmxf"
                                    virtualHW.productCompatibility = "hosted"
                                    tools.upgrade.policy = "manual"
                                    sched.cpu.max = "unlimited"
                                    sched.cpu.units = "mhz"
                                    sched.cpu.shares = "normal"
                                    guestOSAltName = "Other Linux (32-bit)"
                                    sched.swap.derivedName = "/vmfs/volumes/488b4452-2c83ebb5-3f7f-0017085d2386/ET/ET-e73ff811.vswp"
                                    ethernet0.networkName = "VM Network"
                                    sched.cpu.min = "0"
                                    sched.mem.shares = "normal"

and this is the contents of et.vmxf, jic it helps any (I am having trouble understanding it)

     <?xml version="1.0"?>
                                    <Foundry>
                                    <VM>
                                    <VMId type="string">52 55 2c 9b 4c e2 d4 01-79 2d f2 9d c1 c1 2f 22</VMId>
                                    <ClientMetaData>
                                    <clientMetaDataAttributes/>
                                    <HistoryEventList/></ClientMetaData>
                                    <vmxPathName type="string">ET.vmx</vmxPathName></VM></Foundry>

Interestingly I also found these files in the root of the datastore, where this Guest is housed:

     .fbb.sf    736KB
                                    .fdc.sf      61MB
                                    .pbc.sf   249MB
                                    .sbc.sf   255MB
                                    .vh.sf        4MB