LinuxQuestions.org
Welcome to the most active Linux Forum on the web.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Networking
User Name
Password
Linux - Networking This forum is for any issue related to networks or networking.
Routing, network cards, OSI, etc. Anything is fair game.

Notices


Reply
  Search this Thread
Old 11-03-2014, 08:56 AM   #1
Bernd_K
LQ Newbie
 
Registered: Oct 2014
Posts: 4

Rep: Reputation: Disabled
How can I make a process read and write _simultaneously_ over NFS ?


Hi all.

When a process is running, "iostat -nkx 1" looks like this:

Code:
Device:rkB_nor/s    wkB_nor/s rkB_dir/s wkB_dir/s    rkB_svr/s    wkB_svr/s 

pr2     10232.00     10234.50      0.00      0.00     16384.00     59935.75 
pr2     57235.25     57223.15      0.00      0.00     60416.00         0.00 
pr2     56915.50     56932.60      0.00      0.00     61440.00         0.00 
pr2     67707.06     67702.07      0.00      0.00     54272.00         0.00
pr2     54517.38     54557.34      0.00      0.00     59392.00         0.00 
pr2     54597.31     54557.34      0.00      0.00     62464.00         0.00 
pr2     65770.36     65775.69      0.00      0.00     53734.65         0.00 
pr2     54517.38     54514.49      0.00      0.00     61440.00         0.00 
pr2     60912.38     60909.88      0.00      0.00     56320.00         0.00 
pr2     54517.38     54557.34      0.00      0.00     61440.00         0.00 
pr2     54597.31     54557.34      0.00      0.00     61440.00         0.00 
pr2     36371.56     36371.56      0.00      0.00     26624.00         0.00 
pr2     54517.38     54557.34      0.00      0.00     52224.00         0.00 
pr2     54597.31     54557.34      0.00      0.00     53248.00         0.00 
pr2     55716.44     55693.96      0.00      0.00     58368.00         0.00 
pr2     59473.50     59533.45      0.00      0.00     60416.00         0.00 
pr2     49149.69     49090.33      0.00      0.00     59817.82         0.00 
pr2     54517.38     54557.34      0.00      0.00     38912.00         0.00 
pr2     20304.12     20279.54      0.00      0.00     23552.00     71680.00 
pr2       639.50       694.07      0.00      0.00         0.00    105611.52 
pr2         0.00         0.00      0.00      0.00         0.00    114688.00 
pr2         0.00         0.00      0.00      0.00         0.00    115712.00 
pr2         0.00         0.00      0.00      0.00         0.00    114688.00 
pr2         0.00         0.00      0.00      0.00         0.00    114688.00 
pr2         0.00         0.00      0.00      0.00         0.00    115712.00 
pr2         0.00         0.00      0.00      0.00         0.00    114688.00 
pr2       949.75       885.45      0.00      0.00         0.00    108889.39 
pr2     44844.94     44889.90      0.00      0.00     46080.00       757.18 
pr2     60033.06     60048.05      0.00      0.00     56538.69         0.00 
pr2     55816.36     55803.87      0.00      0.00     61440.00         0.00 
pr2     72003.70     71991.21      0.00      0.00     48128.00         0.00 
pr2     54597.31     54557.34      0.00      0.00     59520.54         0.00 
pr2     54517.38     54557.34      0.00      0.00     61440.00         0.00 
pr2     54597.31     54557.34      0.00      0.00     61440.00         0.00 
pr2     56114.54     56131.86      0.00      0.00     60831.68         0.00 
pr2     54677.25     54687.24      0.00      0.00     61440.00         0.00 
pr2     58354.38     58374.36      0.00      0.00     61212.00         0.00 
pr2     66667.88     66660.38      0.00      0.00     53248.00         0.00 
pr2     54597.31     54557.34      0.00      0.00     61440.00         0.00 
pr2     65483.80     65541.26      0.00      0.00     54272.00         0.00 
pr2     55009.49     54997.00      0.00      0.00     60388.00         0.00 
pr2     55529.08     55549.46      0.00      0.00     61440.00         0.00 
pr2     60352.81     60327.44      0.00      0.00     57344.00         0.00
pr2     51879.44     51904.81      0.00      0.00     50176.00     14336.00 
pr2         0.00         0.00      0.00      0.00         0.00    114566.34 
pr2      1119.12      1088.76      0.00      0.00         0.00    104447.83 
pr2         0.00         0.00      0.00      0.00         0.00    115712.00 
pr2         0.00         0.00      0.00      0.00         0.00    114688.00 
pr2         0.00         0.00      0.00      0.00         0.00    114688.00 
pr2         0.00         0.00      0.00      0.00         0.00    115712.00 
pr2         0.00         0.00      0.00      0.00         0.00    114688.00 
pr2         0.00         0.00      0.00      0.00         0.00    114688.00 
pr2      2717.88      2700.39      0.00      0.00     13312.00     52264.20 
pr2     36371.56     36371.56      0.00      0.00     25600.00         0.00 
pr2     25722.46     25727.41      0.00      0.00     21291.09         0.00 
pr2     14948.31     14933.32      0.00      0.00     25600.00         0.00 
pr2     20943.62     21008.57      0.00      0.00     19456.00         0.00
i.e. there is either NFS-read or NFS-write, but never read and write at the same time.
Or, in other words, the process stops reading from RAID when it starts writing to RAID, and resumes reading when it finished writing.
I can influence the length of the "write-only period" with the dirty_xxx parameters under /proc/sys/vm, but this does not change the overall performance, since the length of the "read only period" changes accordingly.

--

My question is: how can I convince the NFS-I/O to read and write at the same time? Is there a parameter I could set accordingly?
I am never writing to any file being read at the same time, so there is no risk in simultaneous read/write.


--

Some additional information:

It is a seismic processing environment, 13 nodes dual Xeon E5-2667 v2, Linux version 2.6.18-371.el5 .
22 Disks, 40TB total, RAID controller LS8271-8I, Level 5 and 6 (says sysman)
The above is the "bulk-data" storage, the one which has critical impact on runtime if it does not constantly read/write as much as possible.

Thanks and Cheers
Bernd
 
Old 11-03-2014, 09:45 AM   #2
suicidaleggroll
LQ Guru
 
Registered: Nov 2010
Location: Colorado
Distribution: OpenSUSE, CentOS
Posts: 5,573

Rep: Reputation: 2142Reputation: 2142Reputation: 2142Reputation: 2142Reputation: 2142Reputation: 2142Reputation: 2142Reputation: 2142Reputation: 2142Reputation: 2142Reputation: 2142
I'm not sure, but it's not an issue on the NFS mounts on my system:
Code:
Filesystem:               rkB_nor/s    wkB_nor/s    rkB_dir/s    wkB_dir/s    rkB_svr/s    wkB_svr/s     ops/s    rops/s    wops/s
server:/home/                  0.00         0.00         0.00         0.00         0.00         0.00      0.00      0.00      0.00
server:/home/                  0.00         0.00         0.00         0.00         0.00         0.00      0.00      0.00      0.00
server:/home/                  0.00         0.00         0.00         0.00         0.00         0.00      1.00      0.00      0.00
server:/home/              76800.00         0.00         0.00         0.00     75776.00         0.00     90.00     74.00      0.00
server:/home/             308224.00         0.00         0.00         0.00    308224.00         0.00    314.00    301.00      0.00
server:/home/             278528.00         0.00         0.00         0.00    278528.00         0.00    270.00    272.00      0.00
server:/home/             311296.00         0.00         0.00         0.00    311296.00         0.00    300.00    304.00      0.00
server:/home/             290816.00         0.00         0.00         0.00    290816.00         0.00    290.00    284.00      0.00
server:/home/             291840.00    185432.25         0.00         0.00    291840.00        88.25    307.00    285.00      1.00
server:/home/             316416.00   1808385.85         0.00         0.00    308314.09         1.85    319.00    303.00      1.00
server:/home/             314368.00   1811546.55         0.00         0.00    314368.00        90.55    319.00    307.00      2.00
server:/home/             332800.00   1757227.28         0.00         0.00    324698.55        43.28    349.00    319.00      1.00
server:/home/             316416.00   1875024.22         0.00         0.00    269312.00      1031.07    267.00    263.00      2.00
server:/home/             280576.00   1819249.89         0.00         0.00    280576.00      1649.89    289.00    274.00      4.00
server:/home/             253952.00    987400.51         0.00         0.00    253952.00     70920.51    338.00    248.00     72.00
server:/home/             211968.00         0.00         0.00         0.00    211968.00    230400.00    455.00    207.00    225.00
server:/home/             221184.00         0.00         0.00         0.00    221184.00    235520.00    464.00    216.00    230.00
server:/home/             224256.00         0.00         0.00         0.00    224256.00    241664.00    461.00    219.00    236.00
server:/home/             218112.00         0.00         0.00         0.00    218112.00    238592.00    474.00    213.00    233.00
server:/home/             205824.00         0.00         0.00         0.00    206848.00    221184.00    436.00    202.00    216.00
server:/home/             216064.00         0.00         0.00         0.00    215040.00    229376.00    440.00    210.00    224.00
server:/home/             208896.00         0.00         0.00         0.00    208896.00    229376.00    447.00    204.00    224.00
server:/home/             186368.00         0.00         0.00         0.00    186368.00    233472.00    443.00    182.00    228.00
server:/home/              98304.00         0.00         0.00         0.00     98304.00    106496.00    216.00     96.00    104.00
server:/home/             219136.00         0.00         0.00         0.00    220160.00    253952.00    486.00    215.00    248.00
server:/home/             209920.00         0.00         0.00         0.00    208896.00    229376.00    459.00    204.00    224.00
server:/home/             208896.00         0.00         0.00         0.00    208896.00    225280.00    437.00    204.00    220.00
server:/home/             215040.00         0.00         0.00         0.00    215040.00    229376.00    436.00    210.00    224.00
server:/home/             204800.00         0.00         0.00         0.00    204800.00    229376.00    448.00    200.00    224.00
server:/home/             156672.00         0.00         0.00         0.00    156672.00    165888.00    335.00    153.00    162.00
server:/home/             197632.00         0.00         0.00         0.00    197632.00    212992.00    413.00    193.00    208.00
server:/home/             223232.00         0.00         0.00         0.00    223232.00    239616.00    448.00    218.00    234.00
server:/home/             208896.00         0.00         0.00         0.00    208896.00    277504.00    539.00    204.00    271.00
server:/home/             202772.28         0.00         0.00         0.00    202772.28    271714.85    530.69    198.02    265.35
server:/home/             143360.00         0.00         0.00         0.00    145408.00    189440.00    385.00    142.00    185.00
server:/home/             150528.00         0.00         0.00         0.00    148480.00    168960.00    307.00    145.00    165.00
server:/home/             207872.00         0.00         0.00         0.00    207872.00    275456.00    541.00    203.00    269.00
server:/home/             208896.00         0.00         0.00         0.00    208896.00    276480.00    539.00    204.00    270.00
server:/home/             208896.00         0.00         0.00         0.00    209920.00    277504.00    541.00    205.00    271.00
server:/home/             130048.00         0.00         0.00         0.00    130048.00    165888.00    322.00    127.00    162.00
server:/home/             140288.00         0.00         0.00         0.00    139264.00    285696.00    485.00    136.00    279.00
server:/home/             221184.00         0.00         0.00         0.00    222208.00    284672.00    572.00    217.00    278.00
server:/home/             228352.00         0.00         0.00         0.00    228352.00    244736.00    472.00    224.00    239.00
server:/home/             233472.00         0.00         0.00         0.00    232448.00    280576.00    556.00    226.00    274.00
server:/home/             218112.00         0.00         0.00         0.00    218112.00    234496.00    465.00    213.00    229.00
server:/home/             226304.00         0.00         0.00         0.00    226304.00    256000.00    503.00    221.00    250.00
server:/home/             192512.00         0.00         0.00         0.00    192512.00    215040.00    405.00    188.00    210.00
server:/home/             219136.00         0.00         0.00         0.00    219136.00    233472.00    466.00    214.00    228.00
server:/home/             212992.00         0.00         0.00         0.00    212992.00    230400.00    436.00    208.00    225.00
server:/home/             150528.00         0.00         0.00         0.00    150528.00    276480.00    440.00    147.00    270.00
server:/home/                  0.00         0.00         0.00         0.00         0.00    331776.00    418.00      0.00    324.00
server:/home/                  0.00         0.00         0.00         0.00         0.00    334848.00    394.00      0.00    327.00
server:/home/                  0.41         0.00         0.00         0.00         0.13    329728.00    426.00      1.00    322.00
server:/home/             112641.23         2.59         0.00         0.00    104973.54    283650.59    408.00    107.00    280.00
server:/home/             285696.00    275719.09         0.00         0.00      2494.98    336896.00    350.00      6.00    329.00
server:/home/               2048.00         0.00         0.00         0.00         0.00    325650.45    325.00      1.00    319.00
server:/home/               2048.00         0.00         0.00         0.00         0.00     28672.00     30.00      0.00     28.00
server:/home/               2048.00         0.00         0.00         0.00         0.00         0.00      1.00      0.00      0.00
server:/home/               2048.00         0.00         0.00         0.00         0.00         0.00      1.00      0.00      0.00
server:/home/               1024.00         0.00         0.00         0.00         0.00         0.00      0.00      0.00      0.00
Is it possible there's a network issue at play? Half duplex vs full duplex, etc.

Last edited by suicidaleggroll; 11-03-2014 at 09:49 AM.
 
Old 11-04-2014, 01:58 AM   #3
Bernd_K
LQ Newbie
 
Registered: Oct 2014
Posts: 4

Original Poster
Rep: Reputation: Disabled
Hi suicidaleggroll.

Thank you very much for your reply.

Good to see that simultaneous read/write is possible over NFS, now it comes down to finding the flaw in our setting.

And wow!, great I/O performance. Wish I would have that, would reduce my runtimes a lot.

---

I checked the eth0 settings using ethtool:

For the RAID:

ethtool eth0
Settings for eth0:
Supported ports: [ FIBRE ]
Supported link modes: 10000baseT/Full
Supported pause frame use: No
Supports auto-negotiation: No
Advertised link modes: 10000baseT/Full
Advertised pause frame use: No
Advertised auto-negotiation: No
Speed: 10000Mb/s
Duplex: Full
Port: Direct Attach Copper
PHYAD: 0
Transceiver: external
Auto-negotiation: off
Supports Wake-on: d
Wake-on: d
Current message level: 0x00000007 (7)
drv probe link
Link detected: yes


For one of my nodes:

ethtool eth0
Settings for eth0:
Supported ports: [ TP ]
Supported link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Full
Supports auto-negotiation: Yes
Advertised link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Full
Advertised auto-negotiation: Yes
Speed: 1000Mb/s
Duplex: Full
Port: Twisted Pair
PHYAD: 1
Transceiver: internal
Auto-negotiation: on
Supports Wake-on: pumbg
Wake-on: g
Current message level: 0x00000007 (7)
Link detected: yes

So duplex is Full on both sides, but Auto-negotiation is On for the Node and Off for the RAID. Is this a problem?

Cheers,
Bernd
 
Old 11-04-2014, 11:40 AM   #4
suicidaleggroll
LQ Guru
 
Registered: Nov 2010
Location: Colorado
Distribution: OpenSUSE, CentOS
Posts: 5,573

Rep: Reputation: 2142Reputation: 2142Reputation: 2142Reputation: 2142Reputation: 2142Reputation: 2142Reputation: 2142Reputation: 2142Reputation: 2142Reputation: 2142Reputation: 2142
Quote:
Originally Posted by Bernd_K View Post
And wow!, great I/O performance. Wish I would have that, would reduce my runtimes a lot.
It's a Netgear XS712T, regular Cat6A cabling, and the servers either have built in 10GbE NICs (Supermicro X9SRH-7TF or X9DRT-PT (2027PR-DTTR)), or they have 10GbE add-on cards (Intel X540T2). Single-transfers generally top out around 220 MB/s, but if I run 4-5 of them simultaneously it'll saturate the 10GbE.

Quote:
Originally Posted by Bernd_K View Post
So duplex is Full on both sides, but Auto-negotiation is On for the Node and Off for the RAID. Is this a problem?
I remember reading an article about that not too long ago saying that it could cause problems, but I can't seem to find it right now. What happens if you set them both to auto?

edit: here it is:
https://www.appliedtrust.com/resourc...mance-problems

And a nice summary table:
https://d39z25yyporgxc.cloudfront.ne...formance_3.gif

Last edited by suicidaleggroll; 11-04-2014 at 11:42 AM.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
read write priority samba nfs unsolved Linux - Server 1 09-11-2009 09:08 AM
Read and write to a co-process in bash Bambi535 Linux - Software 3 11-25-2008 12:58 PM
USB HardDrive mounts read-only. Is there anyway to make it read-write? Fritz_Katz Linux - Hardware 7 09-13-2007 06:50 PM
NFS -- no read write ihopeto Linux - Networking 0 01-18-2006 10:31 PM
NFS non root read write access matthewkeen Linux - Networking 8 09-09-2004 10:52 PM

LinuxQuestions.org > Forums > Linux Forums > Linux - Networking

All times are GMT -5. The time now is 08:35 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration