DSM-G600, DNS-3xx and NSA-220 Hack Forum

Your goals and the goals given the article are quite different.  Your goals are easier to achieve.

I'm not sure why the algorithm would be to connect to older/uppermost.  There is nothing that says the uppermost is anything different from the bottommost ones or one on the same level for that matter.  Much easier to start a search building up a checksums for files, and on the matches do the linking.  So you are just linking to the first file with the same checksum (of course filtering by size first so that you don't have to read every file every time).  I might worry about the low amount of RAM on the DNS-323 since the best way to do this would be in memory in a Perl hash, and with a large amount of files that might exceed what the DNS-323 can do, and any kind of pushing it out to a file might be slow/complicated.

Even though I some times duplicate pictures and put them in different directories to categorize them (don't trust Picasa's database) I don't know that I would ever have enough duplicate files to even justify writing the Perl script.

Except for not being easy to see what is connected, hard links would be better anyway, and no application would know.  They can't go across file systems, but this kind of operation probably shouldn't be across file systems anyways.
Of course you really would want to do this only for "data" disks so I'm not sure if would really matter to most applications if they are soft links.  Of course all of this is on the DNS-323 side if you are using SMB to export, since to the OS that mounts that drive there are no links, it just looks like a copy of the file to it.  If NFS is used for the export I guess the mounting OS would see the links.

I'm trying remember if an rsync to another drive will maintain those hard links properly.  I think it does, you wouldn't want it make copies of the file on the backup drive.

pnin wrote:

Wow! Thanks for all the input, still digesting it. B)

Chriso, that explanation pretty much sums up what I've been doing by hand and Windows hasn't complained once.

The warnings about links are about applications running on the DNS-323 not on Windows.

Windows networks (using SMB protocol) know nothing about links, so as far as Windows is concerned the links on the DNS-323 are just files (that is how SMB shows them to Windows), the magic to trace the link back and give the real file data is all done on the DNS-323 side.

When you do this manually how do you determine, which files to link?

I just played with it  (in Perl) and have decided it is too dangerous for my tastes.

The reason I call it dangerous is because of something I thought about when I ran my Perl program.
If you copy A to B, run the program, then either A or B is going to be linked to the other.
Now you decide to change B, both A and B will be changed.  So heavens forbid if you like to copy things and then change them later, and intended A to be a backup copy.  I noticed this because I had copied some program code from one directory to another and then changed some of the files.  Well for the files I didn't change before running this program, if I changed them in the future I would be in a real surprise if I ever wanted the original copy back, because I will have been editing the original and the "copy" at the same time.

You have to be real sure of what files you are going to run something like this over.

Also a few more things.  Symbolic links are not going to cut it, because of what happens on a delete.
Link X to real file Y
From Windows delete Y.  On Windows side both X and Y disappear.  On the DNS-323 side X exists and points to nothing.  They both disappear on Windows side because if SMB can't trace the real file it doesn't make X appear on the Windows side.  If instead of deleting Y you delete X from Windows then only X disappears from both Windows and the DNS-323.

This is certainly not what you want.  You want hard links for sure.
With a hard link if you delete X or Y, the other will remain.

On the subject of performance and capabilities (and don't think that a "regular (C)" program is going to be more efficient, I'm and expert in both C and Perl and I can tell you in this application Perl is going to be equal or more efficient to the C program, both in memory and the speed).

The test directories have about 86,000 files and 14.4 GB in size.
The DNS-323 processed these in about 20 minutes.  The speed is mostly determined by the total size of the files, but also a bit by how many since opening a file is slower operation then say reading X blocks of data.

The program consumed about 25% of the memory on the DNS-323.  I have found that the DNS-323 does not do swap well if its memory is maxed out.  You probably need to stay under 90%.  The amount of memory consumed is the hash (md5 16 bytes) and the file name and size, times the number of unique files (duplicates you process, but do not store, you ignore soft links, and directories).  So a rough guess for the max files (depends on length of path) is 300,000 files, unless a file is used to store the hash data instead of keeping it in memory (if a file is used to store the data processing will be MUCH slower).

Well that is my take for what it is worth.

Last edited by chriso (2010-10-01 10:27:26)

The inode is not sufficient, when you go to do the linking you can't do a link from inode to new file name.  You have to link from file name to file name.  So you have to save the file path of the "principal copy".  I did keep track of the inode too (didn't mention it above, but it is small in comparison to the file name).  You really want it for optimizing.  For instance if you run across X with inode 7777, and save its path(and read for its MD5 hash) just in case you have to link to it, and then you encounter Y with a inode of 7777.  You know that you don't have to even read the file for its MD5 hash because you know it is just another link of the same file.

On the performance front, the speed I mentioned would only be for the first time through the data.
You would could store the date/time you when through the data and only work on modified files.  But you still need a copy of all the "principal copy" files data (MD5 hash, file path, ...) (and you should store this in the data file too) in memory just like on the first pass.

So the performance would be MUCH faster on passes through the data after the first pass.  Memory requirements are about the same though.

I should mention also the original poster was using this for Windows side data, so I don't think he would want "read-only" for the directories in question.

Last edited by chriso (2010-10-01 23:59:56)