DSM-G600, DNS-3xx and NSA-220 Hack Forum

Unfortunately no one can be told what fun_plug is - you have to see it for yourself.

You are not logged in.

Announcement

#1 2008-12-31 16:05:40

gabe216
New member
Registered: 2008-12-31
Posts: 2

How's rsync performance on large (50GB) directory tree?

I'm planning a to buy a DND323 to have an incremental rsync based backup for my archive partitions on my PC - it's about 50GB each in 15K directories and 100K files.

Is it feasible to let rsync loose on one 50GB directory tree? I never tried this and I'm concerned the 64MB RAM will be enough. Please let me know if you tried to sync a comparable size of dir structure from PC to DNS323.

-- Gabe

Last edited by gabe216 (2008-12-31 16:06:11)

Offline

 

#2 2008-12-31 18:43:48

mazzer
Member
Registered: 2008-02-11
Posts: 13

Re: How's rsync performance on large (50GB) directory tree?

I rsync about 800 GB every night from a Mac (client) to a DNS-323 (daemon). I split it into four separate rsync jobs (run via cron on the Mac): sizes of each are about 80 GB, 200 GB, 300 GB, and 250 GB. The initial sync is a beast, but incremental syncs work great and are reasonably fast (about 3-4 minutes at the maximum). Never run into memory trouble, though I probably only have about 10K directories and 50K files in my 800 GB. I would definitely recommend splitting it into separate jobs using rsync [modules].

Offline

 

#3 2008-12-31 20:52:08

gabe216
New member
Registered: 2008-12-31
Posts: 2

Re: How's rsync performance on large (50GB) directory tree?

I can confirm the memory does not seem to be the bottleneck. The dry-run switch (-n) shows how much memory the backup will require on the server. In my 40G example with 132,000 files the memory was just 3.66M with the new 3.0.4 rsync on both ends (3.0.0 made good progress as I read):

sent 3.66M bytes received 411.81K bytes 28.59K bytes/sec
total size is 40.03G speedup is 9827.05 (DRY RUN)

Offline

 

Board footer

Powered by PunBB
© Copyright 2002–2010 PunBB