[SOLVED] GCFUSE upload performance

DBabo · 04-17-2021, 11:21 PM

hello,
the fuse-mounted gcp bucket upload speeds are abysmal - ~400KB/s. My 40GB archives will take forever to upload.

Question:
is this a normal upload speed and how can it be addressed?

thank you.

pan64 · 04-18-2021, 02:51 AM

definitely 400 KB/s is too low. But we cannot diagnose this problem for you without details.

DBabo · 04-18-2021, 12:11 PM

Quote:

Originally Posted by pan64

definitely 400 KB/s is too low. But we cannot diagnose this problem for you without details.

setup :
mount the media_storageXX bucket and keep the fuse in foregraound to observe logs.
cp files into the bucket:

Code:

gcsfuse --foreground  -o allow_other -o noexec --gid 1000 --key-file /home/az/Downloads/fuse-311002-5299a6547756.json --uid 1000  -o rw -o nosuid -o nodev media_storageXX /home/az/Pictures/media_storageXX 
2021/04/18 13:18:26.644817 Using mount point: /home/az/Pictures/media_storageXX
2021/04/18 13:18:26.644891 Opening GCS connection...
2021/04/18 13:18:26.646233 Set up root directory for bucket media_storageXX
2021/04/18 13:18:26.646252 OpenBucket("media_storageXX", "")
2021/04/18 13:18:29.169732 Mounting file system "media_storageXX"...
2021/04/18 13:18:29.176732 File system has been successfully mounted.

<cat irrelevant logs>

2021/04/18 13:20:37.927330 Not retrying StatObject("20160911_151739.jpg/") after error of type *gcs.NotFoundError ("gcs.NotFoundError: googleapi: Error 404: No such object: media_storageXX/20160911_151739.jpg/, notFound"): &gcs.NotFoundError{Err:(*googleapi.Error)(0xc0000a66c0)}

2021/04/18 13:21:45.643602 Not retrying StatObject("20160911_151739.jpg/") after error of type *gcs.NotFoundError ("gcs.NotFoundError: googleapi: Error 404: No such object: media_storageXX/20160911_151739.jpg/, notFound"): &gcs.NotFoundError{Err:(*googleapi.Error)(0xc000480de0)}
2021/04/18 13:21:46.230419 Not retrying StatObject("20160914_173925.jpg/") after error of type *gcs.NotFoundError ("gcs.NotFoundError: googleapi: Error 404: No such object: media_storageXX/20160914_173925.jpg/, notFound"): &gcs.NotFoundError{Err:(*googleapi.Error)(0xc00010e180)}

while :

Code:

cp -v ./Test/* ./media_storage_family_2017/
'./Test/20160911_151739.jpg' -> './media_storageXX/20160911_151739.jpg'
'./Test/20160914_173925.jpg' -> './media_storageXX/20160914_173925.jpg'

DBabo · 04-18-2021, 12:37 PM

Code:

gcsfuse --foreground  --limit-bytes-per-sec -1 --limit-ops-per-sec -1 -o allow_other -o noexec --gid 1000 --key-file /home/az/Downloads/fuse-311002-5299a6547756.json --uid 1000  -o rw -o nosuid -o nodev media_storageXX /home/az/Pictures/media_storageXX 
2021/04/18 13:33:33.358192 Using mount point: /home/az/Pictures/media_storageXX
2021/04/18 13:33:33.358264 Opening GCS connection...
2021/04/18 13:33:33.359123 Set up root directory for bucket media_storageXX
2021/04/18 13:33:33.359141 OpenBucket("media_storageXX", "")
2021/04/18 13:33:35.270322 Mounting file system "media_storageXX"...
2021/04/18 13:33:35.275216 File system has been successfully mounted.

doesn't seemed to make any significant difference on the upload performance.

Code:

date && cp -v ./100Mb/* ./media_storage_family_2017/ && date
Sun Apr 18 01:40:15 PM EDT 2021
...
Sun Apr 18 01:55:08 PM EDT 2021

Roughly 100Mb/15 minutes ~113 KBs

without "--limit-bytes-per-sec -1 --limit-ops-per-sec -1":
Sun Apr 18 02:02:12 PM EDT 2021
Sun Apr 18 02:18:16 PM EDT 2021

~16 minutes.

DBabo · 04-18-2021, 01:34 PM

Perfstats:
TLDR:
------------------------------------------------------------------------------
Write Throughput
------------------------------------------------------------------------------
Copied 5 1 MiB file(s) for a total transfer size of 5 MiB.
Write throughput: 772.94 Kibit/s. {around 100KBS}
Parallelism strategy: None

From testmy.net the upload speed using Firefox is 1.1Mbps (140KBs), but using speetest.net 2.2Mbps (0.2MBps ~ 275KBs).

I should be happy to get the upload with gcp this fast... (facepalm).

Code:

==============================================================================
                              DIAGNOSTIC RESULTS                              
==============================================================================

------------------------------------------------------------------------------
                                   Latency                                    
------------------------------------------------------------------------------
Operation       Size  Trials  Mean (ms)  Std Dev (ms)  Median (ms)  90th % (ms)
=========  =========  ======  =========  ============  ===========  ===========
   Delete        0 B       5       76.3          11.2         72.6         90.0 
   Delete      1 KiB       5       73.3          16.2         65.9         90.1 
   Delete    100 KiB       5       76.6           8.5         74.5         86.5 
   Delete      1 MiB       5       79.5           7.5         76.9         87.9 
 Download        0 B       5      143.0           5.2        140.9        149.1 
 Download      1 KiB       5      180.7          24.7        166.1        211.2 
 Download    100 KiB       5      258.2          26.7        244.7        289.0 
 Download      1 MiB       5     1160.8         488.2        823.1       1760.2 
 Metadata        0 B       5      367.8         616.2         60.4        986.0 
 Metadata      1 KiB       5       70.6          11.9         62.0         85.6 
 Metadata    100 KiB       5       67.2           8.8         72.1         75.6 
 Metadata      1 MiB       5       63.0           4.0         62.7         67.5 
   Upload        0 B       5      174.6          35.4        158.9        211.8 
   Upload      1 KiB       5      163.5          18.0        161.3        182.9 
   Upload    100 KiB       5     1607.6         440.2       1584.7       2112.2 
   Upload      1 MiB       5    12302.2        3532.2      12734.9      16152.1 

------------------------------------------------------------------------------
                               Write Throughput                               
------------------------------------------------------------------------------
Copied 5 1 MiB file(s) for a total transfer size of 5 MiB.
Write throughput: 772.94 Kibit/s.
Parallelism strategy: None

------------------------------------------------------------------------------
                               Read Throughput                                
------------------------------------------------------------------------------
Copied 5 1 MiB file(s) for a total transfer size of 5 MiB.
Read throughput: 9.24 Mibit/s.
Parallelism strategy: None

------------------------------------------------------------------------------
                              System Information                              
------------------------------------------------------------------------------
IP Address: 
  192.168.1.22
Temporary Directory: 
  /tmp
Bucket URI: 
  gs://media_storageXXX/
gsutil Version: 
  4.60
boto Version: 
  2.49.0
Measurement time: 
 2021-04-18 02:23:40 PM 
Running on GCE: 
  False
Bucket location: 
  US-EAST1
Bucket storage class: 
  STANDARD
Google Server: 
  None
Google Server IP Addresses: 
  172.217.9.240
  172.217.11.16
  142.250.80.16
  172.217.165.144
  172.217.3.112
  172.217.12.208
  172.217.12.176
  142.250.64.112
  172.217.11.48
  172.217.7.16
  172.217.10.16
  172.217.6.208
  172.217.10.240
  172.217.12.144
  172.217.6.240
  142.250.64.80
Google Server Hostnames: 
  lga34s11-in-f16.1e100.net
  lga25s60-in-f16.1e100.net
  lga34s33-in-f16.1e100.net
  lga25s70-in-f16.1e100.net
  lga34s18-in-f16.1e100.net
  lga25s63-in-f16.1e100.net
  lga25s62-in-f16.1e100.net
  lga34s31-in-f16.1e100.net
  lga25s61-in-f16.1e100.net
  lga25s56-in-f16.1e100.net
  lga34s12-in-f16.1e100.net
  lga25s54-in-f208.1e100.net
  lga25s59-in-f16.1e100.net
  lga34s19-in-f16.1e100.net
  lga25s55-in-f240.1e100.net
  lga34s30-in-f16.1e100.net
Google DNS thinks your IP is: 
  XXX.XXX.XXX.XXX
CPU Count: 
  4
CPU Load Average: 
  [0.91, 0.77, 0.69]
Total Memory: 
  7.5 GiB
Free Memory: 
  3.34 GiB
TCP segments sent during test:
  85121
TCP segments received during test:
  66195
TCP segments retransmit during test:
  891
Disk Counter Deltas:
    disk    reads   writes   rbytes   wbytes    rtime    wtime 
   dm-0       35      412   155648  1703936      518      356 
   dm-1        0        0        0        0        0        0 
   dm-2      171     2701 102912000 14049280     1309    16005 
   sda1        0        0        0        0        0        0 
   sda2        0        0        0        0        0        0 
   sda3        0        0        0        0        0        0 
   sda4        0        0        0        0        0        0 
   sda5        0        0        0        0        0        0 
   sda6        0        0        0        0        0        0 
   sda7      201      989 103067648 15413248     1912     4358 
  zram0       12        0    49152        0        0        0 
TCP /proc values:
    rmem_default = 262144
   rmem_max = 4194304
   wmem_default = 262144
   wmem_max = 1048576
   tcp_timestamps = 1
   tcp_sack = 1
   tcp_window_scaling = 1
Boto HTTPS Enabled: 
  True
Requests routed through proxy: 
  False
Latency of the DNS lookup for Google Storage server (ms): 
  1.7
Latencies connecting to Google Storage server IPs (ms):
  142.250.64.80 = 431.6

------------------------------------------------------------------------------
                          In-Process HTTP Statistics                          
------------------------------------------------------------------------------
Total HTTP requests made: 94
HTTP 5xx errors: 0
HTTP connections broken: 0
Availability: 100%

Output file written to '/root/output.json'.