VMware vCenter Server Appliance with embedded database fails due to full database partition

vCenter Server Appliance with an embedded vPostgres database may stop running with out any warning. After logging into the web user interface (web UI) of the appliance you may notice that, in the Storage Usage on the Summary tab, the database statistics are showing 100%. Attempting to start the vCenter Server services at this point will result in the service running for a brief moment before is automatically shuts down again. This occurs because a disk partition on the vPostgres database contains no free space.

VMware doesn't have any recommended preventative maintenance tasks at this time but it doesn't hurt to establish your own in order to prevent this sort of issue in the first place. There is a retention policy for the database and we will look at why that may not be adequate for some installations.

The following information was compiled from VMware knowledge base articles KB2058273 and KB2092127

Getting vCenter Running Again

Let's first see about getting the vCenter appliance running again. In order to perform maintenance on your vCenter Appliance, you will need to open an SSH session with the appliance. We typically use PuTTY since it is free and reliable. Login using the root account and password. This is likely the same password you use to login to the appliance's web UI.

Perform a check to verify that the vPostgres database is full using the following command.

In the example above the /storage/db is at 20% utilization, which is good. When this reaches 100%, the system will automatically shutdown the database until there is enough space available to operate. With vCenter appliance v5.5 U2 the log files often get too large and too fast so that the normal retention time doesn't get rid of them  

If your situation is such that you are seeing 100% utilization on the database storage volume then you can follow the remaining steps to clear out some log files to free up available space.

1. Run this command to stop the VMware VirtualCenter Server service:
   service vmware-vpxd stop
2. Run this command to stop the embedded vCenter Server database service:
   service vmware-vpostgres stop
3. Run this command to change to the Postgres log directory:
   cd /storage/db/vpostgres/pg_log/
4. Run this command to view a current list of log files:
   ls
5. Run this command to remove the older logs:
   rm postgressql-YYYY-MM-DD_xxxxxx.log
   
   You may also use a wild card operator to remove multiple log files at once for a given date as noted in the following example to delete all logs for May of 2015.
   rm postgressql-2015-05-*.log
   
   Repeat the deletion process until you have the most recent one or two days of logs left.
6. Run this command to view the current status of your storage space:
   df -h
   
   If your database storage is now showing less than 50%, you should be ready to restart your vCenter services.
7. Run this command to restart your vCenter database services:
   service vmware-vpostgres start
8. Run this command to restart your vCenter server services:
   service vmware-vpxd start

 At this time your vCenter appliance should be running.

vCenter Configuration Change

VMware has noted that with vCenter v5.5 Update 2 changes were made to database driver which has caused the database to grow at a faster rate than the default allotted storage can retain with the default retention policy. VMware knowledge base article KB2092117 notes a change to the configuration file for vCenter to help reduce the log file size.

We hope that this information is helpful. Please let us know how this has helped you or if you have additional questions. As always Firestone Technical Resources, Inc. is here to help with your computer support issues - "Providing personal service for your impersonal technology."