Kaseya Community

BU/DR - How fast are your backups taking to complete?

  • I wanted to get a sense for how quickly the full Volume back ups are taking for folks here. When you look at your logs and take the total backed up and divide by the time to complete a full Volume back up, what types of rates do you see for your machines?

    On some machines, the averageis more like 5 MB/s to 6 MB/s. We do have some machines that havelower averages, in the 2 MB/s to 4 MB/s range, which poses a problem when back up windows are considered.

    Does anyone here see substantially faster back up times with their back up jobs? The concern is if you have a customer with 30 machines who want BU/DR, and it's time to do full back ups, you may not have enough of a back up window to complete the full back ups.

    Has anyone run into this? If you do see faster back up times, what type of configuration do you have that got you to this speed?

    Thanks in advance.

    Vince


    Legacy Forum Name: BU/DR - How fast are your backups taking to complete?,
    Legacy Posted By Username: vplaza
  • vplaza wrote:
    I wanted to get a sense for how quickly the full Volume back ups are taking for folks here. When you look at your logs and take the total backed up and divide by the time to complete a full Volume back up, what types of rates do you see for your machines?

    On some machines, the averageis more like 5 MB/s to 6 MB/s. We do have some machines that havelower averages, in the 2 MB/s to 4 MB/s range, which poses a problem when back up windows are considered.

    Does anyone here see substantially faster back up times with their back up jobs? The concern is if you have a customer with 30 machines who want BU/DR, and it's time to do full back ups, you may not have enough of a back up window to complete the full back ups.

    Has anyone run into this? If you do see faster back up times, what type of configuration do you have that got you to this speed?

    Thanks in advance.

    Vince


    Hi Vince,

    You definitly are having a problem. We see speeds from 12MB (Megabytes) per seconds to 70MB/sec. 12MB would be for 100MB NICs on slower hardware. 70MB/sec would be on Quality Class server hardware with RAID etc.


    Legacy Forum Name: Backup and Disaster Recovery,
    Legacy Posted By Username: far182
  • Weird. Not sure what is causing this, then. That 5MB/s to 6MB/s are when we back up servers with SCSI RAID1 and RAID5, over 100MB/s Ethernet to a server with a DAS (fiber channel) using SATA drives in RAID5. All good hardware.

    Hmm, any ideas on what to look at?


    Legacy Forum Name: Backup and Disaster Recovery,
    Legacy Posted By Username: vplaza
  • far182 wrote:
    Hi Vince,

    You definitly are having a problem. We see speeds from 12MB (Megabytes) per seconds to 70MB/sec. 12MB would be for 100MB NICs on slower hardware. 70MB/sec would be on Quality Class server hardware with RAID etc.




    Farzon,

    What compression are you using for the back ups? "Normal"?

    Thanks.

    Vince


    Legacy Forum Name: Backup and Disaster Recovery,
    Legacy Posted By Username: vplaza
  • Yes - Normal.

    We have found many things that cause performance issues. I suggest using a utility called IOMETER. This program will let you test disk subsystems speed locally and over the network. It will help you identify bottlenecks.

    What I can tell you is that it's most likely not BUDR. It's hardware, software, configuration, or a combination.


    Legacy Forum Name: Backup and Disaster Recovery,
    Legacy Posted By Username: far182
  • Thanks. The many different combinations of hardware, network, etc. on both the machine being backed up and the image location sure inserts a lot of variability.

    I'm testing backing up a simple Sony VAIO laptop to a NAS device and the performance is not at all exciting. Nearly two hours to back up a 17G image file. I know the laptop isn't a screamer, but I was hoping for better performance. And I'm sure the NAS device lends its own issues to the mix.

    Can you describe a configuration that you have that results in back up rates over 10 MB/s? I would love to try and emulate it as closely as possible to see if we can get near that quick of a rate.


    Legacy Forum Name: Backup and Disaster Recovery,
    Legacy Posted By Username: vplaza
  • Do this, do a test backup from one server to the other WITHOUT the DAS. I suspect the DAS.

    Legacy Forum Name: Backup and Disaster Recovery,
    Legacy Posted By Username: far182
  • I can definitely give that a try.

    Legacy Forum Name: Backup and Disaster Recovery,
    Legacy Posted By Username: vplaza
  • far182 wrote:
    Do this, do a test backup from one server to the other WITHOUT the DAS. I suspect the DAS.


    Farzon,

    You were correct. I only have one night of full backups to compare, but when I switched the image location to go to a server with a SCSI RAID 5 drive instead of to a server with a DAS (Dell AX100 with fiber channel and SATA drives in RAID 5), the back up time dropped.

    The full backup on one serverlast night ran at 9.23 MB/s instead of 6.39 MB/s that the pervious full backup took going to the server with DAS.

    On another server, it ran at 11.0 MB/s instead of 5.03 MB/s that the previous full backup took going to the server with DAS.

    I suppose the SATA drives on the DAS really can't outperform the SCSI drives connected internally to a server. Ineresting.

    So, another question, when you do BU/DR for your customers, what kind of hardware do you send your image files to? I know it depends on a number of factors, such as number of machines and amount of data to back up, but do you standardize on a storage server for images?

    Thanks for your help!

    Vince


    Legacy Forum Name: Backup and Disaster Recovery,
    Legacy Posted By Username: vplaza
  • vplaza wrote:
    far182 wrote:
    Do this, do a test backup from one server to the other WITHOUT the DAS. I suspect the DAS.


    Farzon,

    You were correct. I only have one night of full backups to compare, but when I switched the image location to go to a server with a SCSI RAID 5 drive instead of to a server with a DAS (Dell AX100 with fiber channel and SATA drives in RAID 5), the back up time dropped.

    The full backup on one serverlast night ran at 9.23 MB/s instead of 6.39 MB/s that the pervious full backup took going to the server with DAS.

    On another server, it ran at 11.0 MB/s instead of 5.03 MB/s that the previous full backup took going to the server with DAS.

    I suppose the SATA drives on the DAS really can't outperform the SCSI drives connected internally to a server. Ineresting.

    So, another question, when you do BU/DR for your customers, what kind of hardware do you send your image files to? I know it depends on a number of factors, such as number of machines and amount of data to back up, but do you standardize on a storage server for images?

    Thanks for your help!

    Vince


    No problem at all. Glad I can be of help.

    It might not be the DAS exactly, I wouldn't shelve it. We can get 20mb/sec backups to USB drives even. It has A LOT todo with the drivers/settings of things. For example: is the switch Gigabit? Are the addapters Gigabit? What kind of switch/adapters? Are you using large frames? DAS controller driver/settings. Etc etc.

    Our solution wouldn't work at 6mb/sec. Most of our clients just have too many servers and the backups would never get done. I freakout when we drop below 12mb/sec.

    We use a lot of different hardware. - it's depending on the client. We like to use NAS servers for storage.


    Legacy Forum Name: Backup and Disaster Recovery,
    Legacy Posted By Username: far182
  • far182 wrote:
    We like to use NAS servers for storage.


    Any particular make and model that you've found worked well for any specific environment?

    I've been testing a couple of different brands but have not seen stellar back up rates to them.


    Legacy Forum Name: Backup and Disaster Recovery,
    Legacy Posted By Username: vplaza
  • Vince,

    far182 is right, there are many variables here to consider: number of drives in a RAID volume,drive speeds,NIC types, frame type, type of server hosting the DAS (isn't AX100 technically a SAN? It handles I/O processing internally and can be accessed by multiple hosts), how busy the storage system is at the time of the backup (AX100 might be utilized by multiple servers). Also RAID 5 is not the best choice for write-intensive activity since parity has to be calculated and written in addition to data (lots of overhead I/O).

    Realistically, as far as the storage system is concerned, with no concurrent activity 11mb/s should not be a bottleneck for any modern SATA hard disk, let alonean arraylike an AX100. Your bottleneck is probably the network or the server hosting the SAN.

    -Ed


    Legacy Forum Name: Backup and Disaster Recovery,
    Legacy Posted By Username: bellcpa
  • bellcpa wrote:
    Vince,

    far182 is right, there are many variables here to consider: number of drives in a RAID volume,drive speeds,NIC types, frame type, type of server hosting the DAS (isn't AX100 technically a SAN? It handles I/O processing internally and can be accessed by multiple hosts), how busy the storage system is at the time of the backup (AX100 might be utilized by multiple servers). Also RAID 5 is not the best choice for write-intensive activity since parity has to be calculated and written in addition to data (lots of overhead I/O).

    Realistically, as far as the storage system is concerned, with no concurrent activity 11mb/s should not be a bottleneck for any modern SATA hard disk, let alonean arraylike an AX100. Your bottleneck is probably the network or the server hosting the SAN.

    -Ed



    Thanks, Ed. We're using the AX100 in DAS mode instead of SAN mode, so it is actually connected to a PE 1750 and all it does is host the terabytes of storage. All this is on a 100M LAN, not Gig, so I see that as an opportunity to improve. But, if we are able to see an increase in performance simply by going to another server with internal SCSI RAID 5(albeit, a 2850 with more porcessing power than the 1750) instead of the DAS, I'm thinking the LAN, at this point, isn't my biggest bottleneck, but rather, the storage server and the disk subsystem of that server.

    Vince





    Legacy Forum Name: Backup and Disaster Recovery,
    Legacy Posted By Username: vplaza
  • Vince,

    Besides bandwidth,a network bottleneck could arise due to differences in network adapters and their settings. 12.5mb/s is a maximum raw amount of data a 100mbit network adapter can transfer (100/8). This is not counting overhead (frame/packet headers, protocol traffic, etc) and concurrent network traffic.

    In my experience, most network adapters rarely utilize their true advertised speed so I would have to say you are very fortunate to see even a sustained 11mb/s transfer rate with a 100M adapter. It might in fact be due to some form of data compression. Of course run some performance counters on all systems involved to be absolutely sure before swapping equipment.




    Legacy Forum Name: Backup and Disaster Recovery,
    Legacy Posted By Username: bellcpa