You are Here:
Download Linux Lite 5.4 RC1 today - See Release Announcements



[ SOLVED ] 1.6TB Data - Sharp Decline in file copying speeds between USB drives

Author (Read 3331 times)

0 Members and 1 Guest are viewing this topic.

 

m654321

  • Gold Level Poster
  • *******
  • 891
    Posts
  • Country: gb
  • Reputation: 86
  • Linux Lite Member, 'Advocate' & Donator

  • Linux Lite: 4.6 64bit

  • CPU: Intel Pentium [email protected] (2cores) on an Asus X71Q

  • MEMORY: 4Gb

  • VIDEO CARD: Intel GM45 Express Chipset

  • Kernel: 4.x
Why do file copying speeds decline significantly over time, when transferring large amounts of data, following an initial burst?

Let me explain my recent experience.
I have 1.6TB of video recordings (i.e. 3000+ files classified under different subject folders) on a 4TB USB Seagate drive (ntfs formatted).

I'm now backing these up onto a newly formatted 4TB USB Samsung drive (ext4 formatted). File copying speeds, as indicated by the File Operations notification, were initially 112MB/s (4 hours shown to complete). Three hours later, this had slowed to 35MB/s (9 hours to complete); 6 hours later to 31 MB/s, etc...  I should add that I'm using the GUI method of copying/pasting files and not any command-line method.

Is there any action one can take to maintain the initially high file copying speeds when copying large amount of data? Or, is the only way to maintain high speeds to laboriously copy smaller chunks of data at a time, rather than copying the whole lot (1.6TB) in one go as I have done.  Would a command-line method be a more speed-efficient way to go?

Many thanks for your advice on this one
Mike

PS. Apologies if this post has come in bits, rather than all at once - I had problems posting it due to the 403 error forbidden message - I discovered the reason appeared to be the copying and pasting my post from LibreOffice Writer.  The forbidden message didn't pop-up if I typed the post directly into the forum.
Last Edit: June 21, 2017, 12:08:16 AM by m654321
64bit OS (32-bit on Samsung netbook) installed in Legacy mode on MBR-formatted SSDs (except pi which uses a micro SDHC card):
2017 - Raspberry pi 3B (4cores) ~ [email protected] - LibreElec, used for upgrading our Samsung TV (excellent for the task)  
2012 - Lenovo G580 2689 (2cores; 4threads] ~ [email protected] - LL3.8/Win8.1 dual-boot (LL working smoothly)
2011 - Samsung NP-N145 Plus (1core; 2threads) ~ Intel Atom [email protected] - LL 3.8 32-bit (64-bit too 'laggy')
2008 - Asus X71Q (2cores) ~ Intel [email protected] - LL4.6/Win8.1 dual-boot, LL works fine with kernel 4.15
2007 - Dell Latitude D630 (2cores) ~ Intel [email protected] - LL4.6, works well with kernel 4.4; 4.15 doesn't work
 


Re: 1.6TB Data - Sharp Decline in file copying speeds between USB drives
« Reply #1 on: June 01, 2017, 09:25:50 AM »
 

gold_finger

  • Documentation Writer
  • Platinum Level Poster
  • *****
  • 1094
    Posts
  • Country: us
  • Reputation: 325
  • Linux Lite Member

  • Linux Lite: 3.2 64bit

  • CPU: Intel Core2 Duo E7500 3.0GHz

  • MEMORY: 4Gb

  • VIDEO CARD: Intel 4 Series Integrated Graphics
Don't know answer to your problem.  Did some searching -- saw several posts about same issue, most with no definitive answers.  Found one post that might be of help though.  Have a look at recommendation to add grub parameter for USBs here:  https://forums.linuxmint.com/viewtopic.php?t=199196#p1291555

Try Linux Beginner Search Engine for answers to Linux questions.
 

Re: 1.6TB Data - Sharp Decline in file copying speeds between USB drives
« Reply #2 on: June 01, 2017, 06:26:01 PM »
 

ralphy

  • PayPal Supporter
  • Gold Level Poster
  • *****
  • 741
    Posts
  • Country: us
  • Reputation: 115
  • Linux Lite Developer

  • Linux Lite: 3.8 64bit

I think you're seeing the effects of drive head seek latency when copying. Also, your drive will likely use its cache for smaller amount of data until it is finally written to the drive. This gives the impression that transfers are faster at the beginning as you have pointe out. As the cache fills up and the head seek latency increases depending on the number of files, their sizes and the final destination on the drive, you'll certainly see exactly what you described.


Alternatively, you can try rsync to monitor your transfer speed. For example:


Code: [Select]
rsync -PSauv sourcedirectory/* /dev/sdXX/destdirectory/

If you actually see a drop in transfer speed, as opposed to files that are just big in size, then your disk is I/O-bound hence more noticeable slow downs (often seen when writing heavily to a flash drive or an external USB device).
https://unlockforus.com

Sorry for seeming stupid and preferring Linux - I just don't know any better.

 

Re: 1.6TB Data - Sharp Decline in file copying speeds between USB drives
« Reply #3 on: June 03, 2017, 02:00:25 AM »
 

m654321

  • Gold Level Poster
  • *******
  • 891
    Posts
  • Country: gb
  • Reputation: 86
  • Linux Lite Member, 'Advocate' & Donator

  • Linux Lite: 4.6 64bit

  • CPU: Intel Pentium [email protected] (2cores) on an Asus X71Q

  • MEMORY: 4Gb

  • VIDEO CARD: Intel GM45 Express Chipset

  • Kernel: 4.x
Code: [Select]
rsync -PSauv sourcedirectory/* /dev/sdXX/destdirectory/

Ralphy, many thanks for the explanation & code for monitory source & destination drive speeds.
Using the command line you've given me above, how would I substitute the following information...

Source directory: 'home/mike/Downloads' on /dev/sda
Destination directory: 'TVprogrammes1' on /dev/sdb2 

I notice 'rsync' is already installed by default in the LL3.4 package - can it also be used to transfer data.  If so, is it superior to the GUI copy/paste method? I suppose the problem with the cache bottle-necking is the same whether a GUI or command-line method is used to copy large amounts of data ...

Mike
Last Edit: June 03, 2017, 09:53:32 AM by m654321
64bit OS (32-bit on Samsung netbook) installed in Legacy mode on MBR-formatted SSDs (except pi which uses a micro SDHC card):
2017 - Raspberry pi 3B (4cores) ~ [email protected] - LibreElec, used for upgrading our Samsung TV (excellent for the task)  
2012 - Lenovo G580 2689 (2cores; 4threads] ~ [email protected] - LL3.8/Win8.1 dual-boot (LL working smoothly)
2011 - Samsung NP-N145 Plus (1core; 2threads) ~ Intel Atom [email protected] - LL 3.8 32-bit (64-bit too 'laggy')
2008 - Asus X71Q (2cores) ~ Intel [email protected] - LL4.6/Win8.1 dual-boot, LL works fine with kernel 4.15
2007 - Dell Latitude D630 (2cores) ~ Intel T[email protected] - LL4.6, works well with kernel 4.4; 4.15 doesn't work
 

Re: 1.6TB Data - Sharp Decline in file copying speeds between USB drives
« Reply #4 on: June 03, 2017, 10:06:10 AM »
 

m654321

  • Gold Level Poster
  • *******
  • 891
    Posts
  • Country: gb
  • Reputation: 86
  • Linux Lite Member, 'Advocate' & Donator

  • Linux Lite: 4.6 64bit

  • CPU: Intel Pentium [email protected] (2cores) on an Asus X71Q

  • MEMORY: 4Gb

  • VIDEO CARD: Intel GM45 Express Chipset

  • Kernel: 4.x
I tried this ...
Code: [Select]
[email protected]:~$ rsync -PSauv Downloads/* /dev/sdb2/TV-30-05-17/
sending incremental file list
rsync: ERROR: cannot stat destination "/dev/sdb2/TV-30-05-17/": Not a directory (20)
rsync error: errors selecting input/output files, dirs (code 3) at main.c(652) [Receiver=3.1.1]
[email protected]:~$

@ralphy
What I'm trying to do with the above is to get the stats for copying video files from home/mike/Downloads on my laptop (located on partition dev/sda6) to the directory named 'TV-30-05-17' on an external USB drive (located at dev/sdb2/). As you can see it didn't work ... thanks again ralphy for any advice.

Last Edit: June 03, 2017, 10:12:51 AM by m654321
64bit OS (32-bit on Samsung netbook) installed in Legacy mode on MBR-formatted SSDs (except pi which uses a micro SDHC card):
2017 - Raspberry pi 3B (4cores) ~ [email protected] - LibreElec, used for upgrading our Samsung TV (excellent for the task)  
2012 - Lenovo G580 2689 (2cores; 4threads] ~ [email protected] - LL3.8/Win8.1 dual-boot (LL working smoothly)
2011 - Samsung NP-N145 Plus (1core; 2threads) ~ Intel Atom [email protected] - LL 3.8 32-bit (64-bit too 'laggy')
2008 - Asus X71Q (2cores) ~ Intel [email protected] - LL4.6/Win8.1 dual-boot, LL works fine with kernel 4.15
2007 - Dell Latitude D630 (2cores) ~ Intel [email protected] - LL4.6, works well with kernel 4.4; 4.15 doesn't work
 

Re: 1.6TB Data - Sharp Decline in file copying speeds between USB drives
« Reply #5 on: June 14, 2017, 04:57:24 AM »
 

gold_finger

  • Documentation Writer
  • Platinum Level Poster
  • *****
  • 1094
    Posts
  • Country: us
  • Reputation: 325
  • Linux Lite Member

  • Linux Lite: 3.2 64bit

  • CPU: Intel Core2 Duo E7500 3.0GHz

  • MEMORY: 4Gb

  • VIDEO CARD: Intel 4 Series Integrated Graphics
/dev/sdb2 is partition designation for the USB drive partition, but is not the mount point for it in the file system; therefore won't work with command.


Plug-in USB drive and mount it again.  If your system auto-mounts and opens USB sticks/external drives when you plug them in, (which is default behavior in LL unless you changed it), you don't need to do anything more to mount it.  Then open a terminal and run this command to see what the "MOUNTPOINT" is for that 2nd partition on your USB drive:
Code: [Select]
lsblk

Mount point will likely be something like /media/mike/usb-drive-partition.  Use the mount point shown by lsblk when running the rsync command.


Example:
Code: [Select]
rsync -PSauv ~/Downloads/* /media/mike/usb-drive-partition/TV-30-05-17

NOTE:  If you have other things in addition to video files in your Downloads folder those will also get copied to the TV-30-05-17 directory with this command.
Last Edit: June 14, 2017, 05:04:34 AM by gold_finger
Try Linux Beginner Search Engine for answers to Linux questions.
 

Re: 1.6TB Data - Sharp Decline in file copying speeds between USB drives
« Reply #6 on: June 20, 2017, 04:22:08 AM »
 

m654321

  • Gold Level Poster
  • *******
  • 891
    Posts
  • Country: gb
  • Reputation: 86
  • Linux Lite Member, 'Advocate' & Donator

  • Linux Lite: 4.6 64bit

  • CPU: Intel Pentium [email protected] (2cores) on an Asus X71Q

  • MEMORY: 4Gb

  • VIDEO CARD: Intel GM45 Express Chipset

  • Kernel: 4.x
I’ve made an interesting discovery regarding the significant decline in copying/pasting speed, when transferring large amounts of data - the copying speed issue now appears to be solved...

As mentioned in the first post of this thread, I was copying 1.6TB of video files (3000+ files @ average size ~ 500MB) from a 4TB USB Seagate drive (ntfs format) to a 4TB USB Samsung drive (ext4 format). Speeds plummeted from an initial high of around 120-130 MB/s to less than 30 MB/s, resulting in a very significant increase in the time for data transfer to reach completion.

Investigation 1.
First, I wanted to check whether copying from files from a drive with an ntfs format to one with an ext4 format was the culprit.  I had a second 4TB USB Seagate drive (also ntfs formatted). I found no slowdown in speed when copying between USB nfts drives (e.g. speeds remained at 110 to 130 MB/s) throughout the 4 hours it took to copy the 1.6TB.  However, following this observation, I was left unsure whether this was a disk formatting issue or a difference in drive make/model influencing the outcome, i.e. behaviour of Seagate vs. Samsung – so, roll on to Investigation 2 below...

Investigation 2.
I then changed the format of the Samsung drive, from ext4 to nfts. If the copy/paste slowdown I observed initially is due to a difference in drive format between ‘donor’ and ‘receiver’ drive, then there should be no slowdown when copying from Seagate (ntfs) to Samsung (ntfs) – and that’s exactly what I subsequently observed.   Speeds remained high (110 to 130 MB/s) taking around 4 hours to transfer 1.6TB of data, as I observed with the previous Seagate (ntfs) to Seagate (ntfs) copying/pasting.

Conclusion.
The observations above indicate that the speed of copying/pasting a large amount of data, directly from one USB drive to another, is influenced significantly by the drive format.  To maintain high copying/pasting speeds throughout, both USB drives should have the same format, i.e. ntfs in the above case (or ext4 drives, observation of the 6 July 2017).   Conversely, a very significant reduction in data transfer speed will occur if the drives have different formats, i.e. in the above case with a ‘donor’ drive = ntfs, and  ‘recipient’ drive =  ext4.

In view of the above discovery, I’ll therefore now mark this thread as solved  :)

Regards
Mike
Last Edit: July 17, 2017, 04:31:08 AM by m654321
64bit OS (32-bit on Samsung netbook) installed in Legacy mode on MBR-formatted SSDs (except pi which uses a micro SDHC card):
2017 - Raspberry pi 3B (4cores) ~ [email protected] - LibreElec, used for upgrading our Samsung TV (excellent for the task)  
2012 - Lenovo G580 2689 (2cores; 4threads] ~ [email protected] - LL3.8/Win8.1 dual-boot (LL working smoothly)
2011 - Samsung NP-N145 Plus (1core; 2threads) ~ Intel Atom [email protected] - LL 3.8 32-bit (64-bit too 'laggy')
2008 - Asus X71Q (2cores) ~ Intel [email protected] - LL4.6/Win8.1 dual-boot, LL works fine with kernel 4.15
2007 - Dell Latitude D630 (2cores) ~ Intel [email protected] - LL4.6, works well with kernel 4.4; 4.15 doesn't work
 

Re: 1.6TB Data - Sharp Decline in file copying speeds between USB drives
« Reply #7 on: July 06, 2017, 11:23:37 PM »
 

m654321

  • Gold Level Poster
  • *******
  • 891
    Posts
  • Country: gb
  • Reputation: 86
  • Linux Lite Member, 'Advocate' & Donator

  • Linux Lite: 4.6 64bit

  • CPU: Intel Pentium [email protected] (2cores) on an Asus X71Q

  • MEMORY: 4Gb

  • VIDEO CARD: Intel GM45 Express Chipset

  • Kernel: 4.x
See last post - update added on 6 July 2017 (in blue font) under 'Conclusion'
Last Edit: July 07, 2017, 08:58:51 AM by m654321
64bit OS (32-bit on Samsung netbook) installed in Legacy mode on MBR-formatted SSDs (except pi which uses a micro SDHC card):
2017 - Raspberry pi 3B (4cores) ~ [email protected] - LibreElec, used for upgrading our Samsung TV (excellent for the task)  
2012 - Lenovo G580 2689 (2cores; 4threads] ~ [email protected] - LL3.8/Win8.1 dual-boot (LL working smoothly)
2011 - Samsung NP-N145 Plus (1core; 2threads) ~ Intel Atom [email protected] - LL 3.8 32-bit (64-bit too 'laggy')
2008 - Asus X71Q (2cores) ~ Intel [email protected] - LL4.6/Win8.1 dual-boot, LL works fine with kernel 4.15
2007 - Dell Latitude D630 (2cores) ~ Intel [email protected] - LL4.6, works well with kernel 4.4; 4.15 doesn't work
 


Tags:
 


Download Linux Lite 5.4 RC1 today - See Release Announcements