DSM-G600, DNS-3xx and NSA-220 Hack Forum

I have yet to see a gigabit network transfer data at anything remotely approaching that 900mbps - mind you I'm not saying it's impossible - the best I can do is around 300 mbps.

Also where did you get that 400 mbps figure for hard drive transfers from - a lot of disk manufavturers don't quote sustained transfer speeds which will actually vary depending on which part of the disk you're reading/writing - outer cylinders have mode data passing under the head in any give time frame than inner.

Anyway - to answer your qestion - you can get close to 30 MB/sec (I've seen as much as 35 MB/s reported) using jumbo frame, the throughput seems to be limited by processor resources - it does take a certain amount of oomph to calculate the CRCs etc. involved in tcp/ip transfers

Ì have tested the CPU load on a DNS-323 and it rarely goes oves 35% when copying with samba and I have the jumbo frames on.

The speed on a 430 MBytes file I am getting is 10MB/s while writing and 18MB/s on reading the RAID1. It would have been under 4 MB/s on a 10 Mbit network.

The same samba test against FC8 server with exactly the same disks and also in RAID1 on the same network yields 54 MB/s write and the same speed for read operations.

You can check with cat /proc/loadavg in telnetpromt gives you cpuload over 1, 5 and 15 minutes.

I'm aware that the reported cpu loads do not back my statement on the lack lustre network performance of the device, but, I'd love to hear some one else provide a better explanation.

Here is my reasoning - and yes it is a conclusion based on obervation and reasoning.

On a 100mbps LAN on which I can routinely achieve 98mbps transfers, my DNS-323 cannot deliver greater than approximately 70~80 mbps, switch to gigabit and prior to the introduction of jumbo frame, my best speeds were 150~160 mbps, on a LAN known to deliver 300+ mbps performance.

What we can see here is that the limitation is internal to the DNS-323, since all other components involved in the transfer are known to achieve speeds at a minimum 25% greater than that observed with the DNS-323.

Now introduce fw 1.05 and jumbo frame and with gigabit+jumbo we record a dramatic increase to ~270 mbps, again, less than the external components are know to be capabale of delivering, so again, we can assume safely, that the limitation is internal to the DNS-323.

What does jumbo frame do - how does it achieve greater throughput - by reducing the overhead in the tcp/ip stack, the framing, the addressing and the crc calculations required to assemble the packets.

Now - where is all of this work being done?  As far as I'm aware, there is no ToE (tcp offload engine) in the DNS-323, so it's being done by the system processor - a reduction in the amout of work required to transfer the data and the end result is more data can be transferred in the same period of time.

As my old math teacher used to say QED - quod erat demonstrandum

Feel free to shoot me down, but, please if you do, show me the mistake in my reasoning

Interesting - but my theory is not that the NIC limits the performance, but rather the processor.

And just for the sake of discussion - even if the nic limited the performance then depending on where that limit was you could still see a difference in between read & write operations.

Purely for the sake of discussion - lets assume the nic limits at 150 mbps - and I can write at 100 and read at 200 - any tests of write speed would be limited to 100 mbps, whilst any test of read speed would be limited to 150 mbps - I'm pretty certain in other posts on this subject I have pointed out that the transfer speed is as fast as the slowest link - it is now a matter of determining where that link lies.

Lets go back to my theory.

How are the disks read and written - PIO ? DMA ?  Is it not the same processor still doing the work?
And the RAID1 overhead comes from where - a RAID controller?  This is software RAID - do we not once again have the same processor doing duplicated writes?

Using NASTester 0.4 for convenience

To/from the DNS-323...

100mbps - write - 7.39 MB/sec, read - 8.39 MB/sec
GbE - write - 14.82 MB/sec, read - 15.87 MB/sec
GbE+Jumbo - write - 17.31 MB/sec, read - 21.72 MB/sec

And as a reference - to/from my desktop (Windows XP Pro and running the same make & model of disk as the DNS-323)

GbE - write - 24.46 MB/sec, read - 40.59 MB/sec

First - we can see that the network itself and the node running NASTester is capable of writes of at least 24 MB/sec and reads of 40 MB/sec - so it is safe to assume that any lower limitation that exists is in the DNS-323.

Second - we know that the NIC is capable of transferring data at least up to 21 MB/sec (I have actually recorded over 30 MB/sec)
Third - we know that the disk subsystem is capable of transferring data at least up to 17.31 MB/sec in either direction

How do you account for the drop in performance from 17.31 MB/sec to 14.82 MB/sec (on a write) and from 21.72 MB/sec to 15.87 (on a read)?

What have we changed - the amount of work required to assemble the data packets in the tcp/ip stack - what is doing this work - the processor.

No - that would take more know how than I possess

bspvette86 - I'm not certain if I understand you correctly, but it seems that you're saying a read off of the disk (writing to /dev/null - which is a none existent device, effectively the bit bucket) delivers 43~45 MB/sec.

I have that correct it would eliminate the disk interface (or I/O subsystem) as the bottleneck which is skydreamer suggestion.

fordem wrote:

bspvette86 - I'm not certain if I understand you correctly, but it seems that you're saying a read off of the disk (writing to /dev/null - which is a none existent device, effectively the bit bucket) delivers 43~45 MB/sec.

I have that correct it would eliminate the disk interface (or I/O subsystem) as the bottleneck which is skydreamer suggestion.

Fordem,
That is what the results indicate.  Perhaps you should run some similar tests on yours with some big files...

To me, hdparm does not reflect the results of reading "real" files from the disks.

Regards,
BV

I know well the Seagate disks used in my tests and they are able to produce 90MB/s sustained rate which drops to 60 MBytes/s closer to the center. There is a lot of variables affecting the disk susbsystem performance indeed but having Linux Software RAID on both the FC8 and DNS-323 I feel confident that the hdparm read test produces results which are OK to compare.

I quite do not understand why would the write speed drop so significantly and knowing that the CPU load is rather low I would assume that the bottleneck is lying between the CPU and the disk interface. It does not help that I cannot read hdparm -I the interface status, it may reveal more information- perhaps the DNS-323 turns off caching on the disks, who knows?

I had some very peculiar experience with RAID1 on ICH8 under windows that resulted in 30% RAID speed drop when the RAID1 volume cache was on. This was unlikely the disk cache itself, which in contrast boosted the RAID performance.
I am not sure if this is a bug in Intel's Fake RAID driver or some sort of interaction between the volume cache and the hard drive cache. And along this line of thought there might be a similar issue with the DNS-323.

I agree with your point that the jumbo frames do provide significant speed boost but am not sure why should they contribute to a higher processor load? The CPU should need less interrupts to dispatch/receive the packet as all the overhead information is calculated less frequently and the dataflow handling is not as demandig as it is in the case of smaller packets.

Eventhough the SATA disks are equipped with a 32 MByte buffer I do not think that it is sufficient to hold the full track and my guess is that because of this inefficient data handling by the DNS-323 the disks do not manage to pump the data out in one go as the head flies over the track hence the sudden speed drop. In comparison the read operations are cached in advance using speculative read hence there is less missed turns of the disk platters. I assume that the data is defragmented here.

Let's take this as a brainstorming session and we might figure out what is not working well and perhaps even fix it? I am still behind my theory that it is the disk subsystem rather than network interface that contributes to the speed issues.

Would you have any idea why is the hdparm not producing any results on the physical disks in DNS-323?

Well this post is getting results very close to the speeds that we have seen on the network- so it looks more like an inefficient disk I/O subsystem:

http://dns323.kood.org/forum/p20185-Tod … tml#p20185

bspvette 86:

I did an internal copy of a 220MB file from /mnt/HD_a2 to /mnt/HD_b2 in 9.21 seconds.  That is a transfer rate of roughly 24MB/s.


# # time cp /mnt/HD_b2/tmp/test1 .

real    0m 9.21s
user    0m 0.21s
sys     0m 7.62s


Regards,
BSPvette

Last edited by skydreamer (2008-09-08 10:59:18)

The small ram size is also a problem. if you manage to lower the ram usage by disabling not needed services, you can improve the performance.

In my book, the DNS-323 is CPU bound based on the fact that  A load average > 1 = CPU bound.  When I fire off an FTP copy of a very large file, the load average is > 1.  Network by itself is not the issue, disk by itself is not the issue, but when you add it all up, the CPU is choking.

Regards,
BSPvette