Slow Backups

For a while now I’ve been trying to resolve some slow backup jobs we’ve been having while using Symantec Backup Exec 12.5.   

Jobs which previously ran at over 1,000MB per min were now running at under half that rate.   The jobs were using a Dell PowerVault PV124T library with an LTO3 drive, so rates should be much higher than a few hundred MB per min?

Debugging the Problem

With no errors shown on either the backup server, or the server being backed up (for example a standard Windows Storage Server 2003 machine), everything looked fine, however backups weren’t completed within their allocated time slots?

Other jobs (especially a large Exchange store backup) were showing high throughput, so this ruled out a potential tape drive/SCSI problem.   Still, I made sure all drivers, Tape, SCSI controller, Autoloader,etc. were up to date, along with all firmware.   Still the same…

Applying SP4 to Backup Exec made no difference either (though still good practice, so worth doing).

The Solution

Clutching at straws, I tried a search for Backup Exec, along with our Anti Virus software “Trend Micro”.   From previous experience, I’ve found AV and Backup software (even from the same vendor!) doesn’t always play nicely together.    This was when I found the following article:

http://esupport.trendmicro.com/Pages/Slow-BackUps-with-.aspx

We’re using Backup Exec 12.5 with Trend Micro OfficeScan 12.5.

Kinda hits the nail on the head doesn’t it!

Now the first hurdle was that the path given in the registry didn’t exist:

HKEY_LOCAL_MACHINE\SOFTWARE\TrendMicro\Aegis\RmSysEventDebugFlags

So I did a search for RmSysEventDebugFlags..

As this machine was a 64bit install, the key appeared under:

HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\TrendMicro\AEGIS\RmSysEventDebugFlags

Now the article suggested stopping the TMBMSRV service, making the registry and rebooting each machine.   The first problem here is that there isn’t a TMBMSRV service (maybe this is for the worry free version of Trend Micro).    The second problem is that I didn’t like the idea of rebooting all of our production servers!

So instead, I clicked on the OfficeScan icon, and selected unload Officescan.   This shuts down all OfficeScan services and temporarily disables the AV protection.    I made the registry change and started the AV protection again.

After running a test job, the rates were back up where they should be once again – if anything faster than ever due to all the driver/firmware updates!

I started seeing several warnings on different PCs from the Trend Micro Web Reputation service this morning.   All warnings pointed to http://public.bay.livefilestore.com as the source.

My first thought was that it was a new virus outbreak that had infected several PCs so I got stuck in and started to investigate.

All virus scans on the local PCs were coming back clean, so something else must be going on?

Eventually I tracked it down to Windows Live Messenger as I discovered that Livefilestore.com is a Microsoft site used for file storage in the cloud.  

When you have the Windows Live Messenger main window open – you’ll notice that towards the bottom of the window is the “What’s new” section.   This section has constantly changing updates from your contacts.

What I discovered was that the virus warnings were appearing just as certain “what’s new” updates cycled through.   It looks like Trend is blocking livefilestore.com which is used to store photos which are then previewed through messenger.

Until Microsoft and Trend sort this issue, you can click on the icon of a spanner next to the What’s new” heading and deselect updates or block updates from specific people.

Alternatively you can just close the main messenger window as this stops the warnings too.

I’ve seen this happen a couple of times now.   A working Hyper-V server, when restarted, “looses” virtual machines!   No errors, and no trace of the VM! Scary!

Turns out, it’s because the anti-virus software is scanning the virtual machines as they’re loaded and gets in the way.

Here’s a good article describing the problem and which folders you need to exclude from the AV scan:

http://social.technet.microsoft.com/Forums/en-US/winserverhyperv/thread/e81d0f59-18b5-4214-94ea-6cde883fdd4e