What do I need to add some more of to my system???

taylorkh · 08-06-2022, 08:17 AM

Perhaps it is the North Carolina summer weather. Temps in the low to mid 90 Fs, humidity in the low 120%s. I took the dogs for a walk this morning at 78 F and was soaked with sweat by time I got back. So I am a bit irritable and impatient

Following that amusing editorial comment, here is the issue...

I am copying about 30 GB of data from my workstation to one of my servers.

The workstation is a Dell T3620 with an i7-6700 4 core CPU and 32 GB of RAM. The files reside on a FAST M.2 PCIe drive.

The "server" is a low end Dell T20, Dual core Pentium, 4 GB RAM and some slow, low energy mechanical drives. It is used just for data backup, not typical server stuff.

Both boxes are running CentOS 7. The system monitor utility is showing a transfer rate of 40 - 50 MiB/s whichis typical. Not fast but not the problem.

The problem comes in on the workstation. While these files are in the process of being transferred the overall performance of the workstation slows almost to a stop. Navigating the file system with gnome-commander or the caja file manager (even not on the drive in use), working within a virtual machine such as typing this message in Firefox on a CentOS 7 VM on VMWare Workstation, moving from window to window, workspace to workspace etc.

CPU usage is shown as minimal - a few % at most, memory usage is also minimal. SOMETHING is being used up. I had been playing with a docking station hooked to a Raspberry Pi 3. It had a couple of OLD 1 TB drives. Due to horsepower limitations on the Pi I could get a network throughput of only about 10 MiB/s. Even this small through put would bring the workstation to its knees.

I know that all data flowing onto the network goes through the CPU. I always thought this somewhat dumb. Many years ago a server design called I2O was presented as a potential fix. The CPU would tell the SMART controller "fetch this file and put it on the wire to the destination of..." Make it so and let me know when you are done! I thought that was briliant but it never took off.

This phenomenon has probably always occurred. Perhaps I am getting more sensitive in old age and miserable weather

I am looking for some ideas. Perhaps there is something going amiss which I need to investigate.

TIA,

Ken

wpeckham · 08-06-2022, 11:38 AM

You failed to mention some of the most critical pieces of this picture:
1. What SOFTWARE are you using to do the transfer?
2. What is the NETWORKING like between them? (Hardware, protocol, number of eth hops).

I suspect that we might find a solution for you in the software, but knowing the rest will help avoid giving you bad advice.

shortarcflyer · 08-06-2022, 11:49 AM

Running this code in terminal and pasting it up would give us more info on your machine to help you better.

Code:

inxi -Fxxxrz

taylorkh · 08-06-2022, 12:12 PM

Thanks wpeckham,

I guess that might help complet the picture

The server is exposing the file system with NFS. nfs-server provided by CentOS.

The NFS export is mapped to the file system on the workstation thusly:

sudo mount taylor14:/media/data14.2a /data/_servers/data14.2

I use bash scripts to mount and unmount the servers when I need to access them. I have found that if I shut down a server without unmounting the NFS shares - all sorts of strang things happen with Caja.

As to the actual transfer, I am using gnome-commander to copy the files from the physical drive in the workstation to the mount point created above. I have found this to be a little quicker than doing an ssh connect and copy. Both machines are on a home LAN so not too much of a security concern. Copying over ssh of course reaquires encryption/decryption overhead. The file system on the server is encrypted with dmcrypt/LUKS but while that might slow down writing to disk I do not think the workstaion would know or care.

Both machines have static IP addresses and are conneced by gigabit Ethernet. The server and 2 others are plugged into a Netgear hub which connects to another hub to which the workstation is plugged. As these are dumb hubs I do not think they count as :hops". The IP address never changes en-route.

I do keep the workstation busy. 3 or 4 virtual machines running at any time. Typically not all of them are working hard on anything at the same time. Just sort of my version of Qubes OS

Still, system-monitor, top, free do not show me challenging my resources.

And I might add... The video card is an NVIDIA Quadro K620 driving two monitors. Driver is from elRepo. I do not recall how much memory but it was said to be a decent card when I purchased it a few years ago with the computer.

Perhaps a dual socket 12 core Xeon, 128 GB RAM replacement (or not!)

Thanks again,

Ken

computersavvy · 08-06-2022, 01:14 PM

There are a lot of bottlenecks in what you describe and almost all are io related.
Reading from a drive on the pc is the first bottleneck. It physically takes time to read data from a drive, regardless of whether it is an HDD or SSD.
Once the data is read it must be sent by network to the server.
Then the server must receive the data and the biggest bottleneck occurs with writing to the spinning disks there.

You can usually see the actual numbers and their relationship using the command 'iostat' on the server and on the workstation over time after starting this type transfer and repeating it at intervals while the transfer is occurring.

This is what I saw on my system while using rsync to copy a 50GB vm image from a raid array (md127) to a single disk (sdc), both internal on my PC and both connected SATA III (600mbps). The data is displayed at 4 second intervals. Note that a large percentage of cpu time is in iowait even though the machine is running other tasks at near 10% cpu load constantly.

Code:

$ iostat -t -c -d 4 /dev/md127 /dev/sdc > iostat.log

08/06/2022 12:54:28 PM
avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           2.32    8.36    3.37   12.61    0.00   73.34

Device             tps    kB_read/s    kB_wrtn/s    kB_dscd/s    kB_read    kB_wrtn    kB_dscd
md127           370.50    120832.00       116.00         0.00     483328        464          0
sdc             140.50         1.00     88070.00         0.00          4     352280          0


08/06/2022 12:54:32 PM
avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           2.87    8.30    4.83   13.33    0.00   70.67

Device             tps    kB_read/s    kB_wrtn/s    kB_dscd/s    kB_read    kB_wrtn    kB_dscd
md127           481.00    166912.00       123.00         0.00     667648        492          0
sdc             304.50         0.00    183096.00         0.00          0     732384          0


08/06/2022 12:54:36 PM
avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           3.05    8.33    5.11   10.39    0.00   73.12

Device             tps    kB_read/s    kB_wrtn/s    kB_dscd/s    kB_read    kB_wrtn    kB_dscd
md127           566.25    195072.00       183.00         0.00     780288        732          0
sdc             314.75         0.00    187262.00         0.00          0     749048          0


08/06/2022 12:54:40 PM
avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           2.13    8.32    4.40   12.75    0.00   72.40

Device             tps    kB_read/s    kB_wrtn/s    kB_dscd/s    kB_read    kB_wrtn    kB_dscd
md127           457.50    148481.00        74.00         0.00     593924        296          0
sdc             314.50         1.00    182116.00         0.00          4     728464          0


08/06/2022 12:54:44 PM
avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           2.82    8.33    4.41   13.15    0.00   71.29

Device             tps    kB_read/s    kB_wrtn/s    kB_dscd/s    kB_read    kB_wrtn    kB_dscd
md127           466.75    144587.00      6468.00         0.00     578348      25872          0
sdc             337.75         0.00    181428.00         0.00          0     725712          0

This is the result after the transfer ended.

Code:

08/06/2022 12:56:00 PM
avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           2.09    8.34    0.73    0.21    0.00   88.63

Device             tps    kB_read/s    kB_wrtn/s    kB_dscd/s    kB_read    kB_wrtn    kB_dscd
md127            15.75         0.00       165.00         0.00          0        660          0
sdc               0.00         0.00         0.00         0.00          0          0          0


08/06/2022 12:56:04 PM
avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           2.18    8.29    0.85    0.02    0.00   88.65

Device             tps    kB_read/s    kB_wrtn/s    kB_dscd/s    kB_read    kB_wrtn    kB_dscd
md127             3.24         0.00        12.97         0.00          0         52          0
sdc               0.00         0.00         0.00         0.00          0          0          0

michaelk · 08-06-2022, 01:16 PM

Maybe your nfs configuration needs a bit of tuning?

https://www.redhat.com/sysadmin/using-nfsstat-nfsiostat
http://nfs.sourceforge.net/nfs-howto/ar01s05.html

taylorkh · 08-06-2022, 02:16 PM

Thanks michaelk,

That is some good info on nfs. I can see how that might improve the throughput to the server. However, that is not the concern. Whatever phenomenon is occurring it is hosing up performance of things on the workstation which should be unrelated to the file transfer.

The PCIe SSD is rated at 2500 MB/s read. I just plugged in an admittedly not to fast USB mechanical drive. Caja (Nautilus in MATE) transferred data at ~120 MiB/s. Much faster than the network transfer. Yet I experienced NO slowdowns of anything else. The workstation was as peppy as usual. Caja would use about 50% of one core and not much memory.

gnome-commander seemed to move the files to the USB drive even faster although it does not have a built in speed monitor. It would occasionally jump from 0 to 75% of one code and then drop back down after a second or so. Again, NO slowdowns anywhere else.

Perhaps I need to bring the server back up and connect with ssh via gnome-commander and transfer some files that way. It might be slower that nfs but if it does not interfere with the workstation otherwise...

I may also use a VM to do the transfer and see if VMWare give any clues. If it does I will definitely run the nfs diagnostics.

Of course at the rate solid state memory is moving... before long I will be able to replace my 80 TB server farm with a handful of MicroSD cards. The vendor I usually buy such stuff from has 1GB MicroSD cards for $120 US, That is scary

I told the vendor they need a storage case for the cards which is at least as big as a football. A terabyte of important data on something smaller than a thumbnail... TOO easy to physically loose the bloody thing.

Ken

taylorkh · 08-06-2022, 05:56 PM

Thanks shortarcflyer,

Sorry I overshot your post. Here is the info as requested.

Code:

[ken@taylor20 ~]$ inxi -Fxxxrz
System:
  Kernel: 3.10.0-1160.71.1.el7.x86_64 arch: x86_64 bits: 64 compiler: gcc
    v: 4.8.5 Desktop: MATE v: 1.16.2 info: mate-panel wm: marco v: 1.16.1 vt: 1
    dm: LightDM v: 1.25.0 Distro: CentOS Linux release 7.9.2009 (Core)
Machine:
  Type: Desktop System: Dell product: Precision Tower 3620 v: N/A
    serial: <superuser required> Chassis: type: 3 serial: <superuser required>
  Mobo: Dell model: 09WH54 v: A01 serial: <superuser required>
    UEFI-[Legacy]: Dell v: 2.7.3 date: 01/31/2018
CPU:
  Info: quad core model: Intel Core i7-6700 bits: 64 type: MT MCP
    smt: enabled arch: Skylake-S rev: 3 cache: L1: 256 KiB L2: 1024 KiB
    L3: 8 MiB
  Speed (MHz): avg: 891 high: 900 min/max: 800/4000 cores: 1: 900 2: 900
    3: 900 4: 900 5: 900 6: 873 7: 898 8: 861 bogomips: 54528
  Flags: avx avx2 ht lm nx pae sse sse2 sse3 sse4_1 sse4_2 ssse3 vmx
Graphics:
  Device-1: Intel HD Graphics 530 vendor: Dell driver: i915 v: kernel
    arch: Gen9 ports: active: none
    empty: DP-1, DP-2, DP-3, HDMI-A-1, HDMI-A-2, HDMI-A-3 bus-ID: 00:02.0
    chip-ID: 8086:1912 class-ID: 0380
  Device-2: NVIDIA GM107GL [Quadro K620] driver: nvidia v: 510.60.02
    arch: Maxwell pcie: speed: 2.5 GT/s lanes: 16 bus-ID: 01:00.0
    chip-ID: 10de:13bb class-ID: 0300
  Display: x11 server: X.Org v: 1.20.4 compositor: marco v: 1.16.1 driver:
    X: loaded: nvidia gpu: i915,nvidia display-ID: :0 screens: 1
  Screen-1: 0 s-res: 3120x1920 s-dpi: 96 s-size: 826x508mm (32.52x20.00")
    s-diag: 970mm (38.18")
  Monitor-1: DP-1 pos: top-right res: 1200x1920 hz: 60 dpi: 94
    size: 324x518mm (12.76x20.39") diag: 611mm (24.05") modes: N/A
  Monitor-2: DVI-I-1 pos: primary,bottom-l res: 1920x1200 hz: 60 dpi: 94
    size: 518x324mm (20.39x12.76") diag: 611mm (24.05") modes: N/A

Thanks computersavvy,

I will experiment with iostat and post what I find.

Ken
  OpenGL: renderer: Quadro K620/PCIe/SSE2 v: 4.6.0 NVIDIA 510.60.02
    direct render: Yes
Audio:
  Device-1: NVIDIA GM107 High Definition Audio [GeForce 940MX]
    driver: snd_hda_intel v: kernel pcie: speed: 5 GT/s lanes: 16
    bus-ID: 01:00.1 chip-ID: 10de:0fbc class-ID: 0403
  Device-2: C-Media CMI8738/CMI8768 PCI Audio driver: snd_cmipci v: kernel
    bus-ID: 07:00.0 chip-ID: 13f6:0111 class-ID: 0401
  Sound Server-1: ALSA v: k3.10.0-1160.71.1.el7.x86_64 running: yes
  Sound Server-2: JACK v: N/A running: no
  Sound Server-3: PulseAudio v: 10.0 running: yes
Network:
  Device-1: Intel Ethernet I219-LM vendor: Dell driver: e1000e v: 3.2.6-k
    port: N/A bus-ID: 00:1f.6 chip-ID: 8086:15b7 class-ID: 0200
  IF: enp0s31f6 state: up speed: 1000 Mbps duplex: full mac: <filter>
  IF-ID-1: vmnet1 state: unknown speed: N/A duplex: N/A mac: <filter>
  IF-ID-2: vmnet8 state: unknown speed: N/A duplex: N/A mac: <filter>
Drives:
  Local Storage: total: 5.2 TiB used: 1.08 TiB (20.7%)
  ID-1: /dev/nvme0n1 model: BPXP size: 447.13 GiB speed: 31.6 Gb/s lanes: 4
    type: SSD serial: <filter> rev: ECFM12.3 temp: 37 C scheme: MBR
  ID-2: /dev/nvme1n1 model: BPX size: 223.57 GiB speed: 31.6 Gb/s lanes: 4
    type: SSD serial: <filter> rev: E7FM03.6 temp: 61 C scheme: MBR
  ID-3: /dev/sda vendor: Western Digital model: WD40EZRZ-00GXCB0
    size: 3.64 TiB speed: 6.0 Gb/s type: HDD rpm: 5400 serial: <filter>
    rev: 0A80 temp: 33 C scheme: GPT
  ID-4: /dev/sdb vendor: HGST (Hitachi) model: HTS721010A9E630
    size: 931.51 GiB speed: 6.0 Gb/s type: HDD rpm: 7200 serial: <filter>
    rev: A3W0 temp: 32 C scheme: GPT
Partition:
  ID-1: / size: 11.69 GiB used: 7.63 GiB (65.3%) fs: ext4 dev: /dev/dm-0
    mapped: luks-d19388d9-b77f-4431-aa85-5de7b73c9e22
  ID-2: /boot size: 508.7 MiB used: 436.5 MiB (85.8%) fs: xfs
    dev: /dev/nvme0n1p1
  ID-3: /home size: 3.81 GiB used: 257 MiB (6.6%) fs: ext4 dev: /dev/dm-1
    mapped: luks-ae552387-4c78-4a70-8ded-e5f5cec34e67
  ID-4: /var size: 5.99 GiB used: 844.1 MiB (13.8%) fs: xfs dev: /dev/dm-2
    mapped: luks-514fb788-8706-41bb-ab08-97f0fa1cbf3b
Swap:
  Alert: No swap data was found.
Sensors:
  System Temperatures: cpu: 29.8 C mobo: 27.8 C gpu: nvidia temp: 49 C
  Fan Speeds (RPM): N/A gpu: nvidia fan: 36%
Repos:
  Packages: N/A note: see --pkg
  No active yum repos in: /etc/yum.conf
  No active yum repos in: /etc/yum.repos.d/CentOS-Base.repo
  No active yum repos in: /etc/yum.repos.d/CentOS-CR.repo
  No active yum repos in: /etc/yum.repos.d/CentOS-Debuginfo.repo
  No active yum repos in: /etc/yum.repos.d/CentOS-Media.repo
  No active yum repos in: /etc/yum.repos.d/CentOS-Sources.repo
  No active yum repos in: /etc/yum.repos.d/CentOS-Vault.repo
  No active yum repos in: /etc/yum.repos.d/CentOS-fasttrack.repo
  No active yum repos in: /etc/yum.repos.d/CentOS-x86_64-kernel.repo
  No active yum repos in: /etc/yum.repos.d/elrepo.repo
  No active yum repos in: /etc/yum.repos.d/epel-testing.repo
  Active yum repos in: /etc/yum.repos.d/epel.repo
    1: epel ~ https://mirrors.fedoraproject.org/metalink?repo=epel-7&arch=$basearch
  No active yum repos in: /etc/yum.repos.d/nux-dextop.repo
  No active yum repos in: /etc/yum.repos.d/rpmfusion-free-updates-testing.repo
  Active yum repos in: /etc/yum.repos.d/rpmfusion-free-updates.repo
    1: rpmfusion-free-updates ~ http://mirrors.rpmfusion.org/mirrorlist?repo=free-el-updates-released-7&arch=$basearch
  No active yum repos in: /etc/yum.repos.d/rpmfusion-nonfree-updates-testing.repo
  No active yum repos in: /etc/yum.repos.d/rpmfusion-nonfree-updates.repo
  Active yum repos in: /etc/yum.repos.d/scootersoftware.repo
    1: scootersoftware ~ https://www.scootersoftware.com/bcompare4
  No active yum repos in: /etc/yum.repos.d/wine32.repo
Info:
  Processes: 367 Uptime: 6d 3h 39m wakeups: 115 Memory: 31.16 GiB
  used: 2.32 GiB (7.4%) Init: systemd v: 219 target: graphical (5)
  default: graphical Compilers: gcc: 4.8.5 Shell: Bash v: 4.2.46
  running-in: mate-terminal inxi: 3.3.19

wpeckham · 08-06-2022, 07:26 PM

NFS can be MASSIVELY convenient, but may not be optimal for your purpose. I love it for file work, but not for mass migration or large file transfer. There are known performance issues, even with the latest versions.

I would do that transfer using rsync. It is very performant, can be throttled at need, and makes very different use of the IO buffers, I cannot be sure those buffers are implicated here, and you certainly have enough buffer memory on the workstation side if I understand things correctly. Best of all it can be restarted if it gets interrupted.

What t is NOT very efficient for is mass copying of a single large binary file (it works best on text files of various kinds).
If your machines each had a single unused 100bt port I would run a CAT5/5e/6 jumper between and configure those interfaces on a single shared unique non-routable subnet and do the transfer over that, thus d4ecoupliong the IO from other interface buffers and the production network. Just my $0.02 USD.

taylorkh · 08-06-2022, 08:02 PM

Yes, I should use more efficient tools. However, I like to see what I am grabbing and where I am shoving it. The best program Microsloth ever came out with was Winfile (File Manager) the two panel program that resembled the old Norton Commander. The 16 bit version on Win 7 etc was crap. The 32 bit version from Win ND (which I used) was great. It handled permissions etc. very well. When I found it was not on XP I copied it from an NT and burned it to a CD. Whenever I got a new XP machine I immediately copied Winfile to it.

The Windoze explored methodology - grab some files and HOPE you are dumping them in the right place was sort of kamikaze. I know several EXCELLENT Windows admins who accidentally dropped big collections of rubbish on C:\ - hours of work to clean up.

Perhaps it is the I/O buffers which are getting used up and preventing other I/O activity on the workstaion.