Swap Partition Ruining PC Performance?

H_TeXMeX_H · 12-08-2011, 08:38 AM

I missed that.

Some observations:

You are using a lot of apps that are leaky and eat up a lot of RAM. I would never run all of those at once. From what I see above, I think some part of XFCE may be leaking as well as Firefox (update to latest version). Maybe try another desktop environment other than XFCE.

FredJones · 12-08-2011, 08:59 AM

Regarding the RAM settings, here are the outputs that I know about, to see what's happening. If you could take a minute and confirm that everything matches. These are all taken within a few seconds of each other:

htop. What I find odd is that the bars for Mem shouldn't be so far across. 2566 is 78% of 3271. There are 30 bars and so 78% of those is 23. In the attached PNG you can see that they are ALL lit up. Here is the text version:

Code:

  1  [||                        4.6%]     Tasks: 114, 56 kthr; 1 running
  2  [||                        2.6%]     Load average: 0.06 0.03 0.05 
  3  [|                         1.3%]     Uptime: 1 day, 19:42:14
  4  [||                        4.0%]
  Mem[|||||||||||||||||||2566/3271MB]
  Swp[|                   153/9091MB]

  PID USER      PRI  NI  VIRT   RES   SHR S CPU% MEM%   TIME+  Command
 1459 fred       20   0 1190M  737M 28224 S  0.0 22.5 49:41.26 firefox
 1420 fred       20   0  752M  343M 27160 S  0.0 10.5 13:06.78 /usr/lib/thunderb
  961 mysql      20   0  307M  167M  3144 S  0.0  5.1 15:40.78 /usr/bin/mysqld -
 4962 fred       20   0  369M  154M  7764 S  0.0  4.7  3:28.80 java -Xmx192M -ja
 1310 fred       20   0  134M  121M  6508 S  0.0  3.7  2:43.24 /usr/lib/xfce4/pa
 1635 fred       20   0  233M 99024 22416 S  0.0  3.0  1:16.90 /usr/lib/chromium
 1581 fred       20   0  396M 70236 27368 S  0.0  2.1  8:21.43 /usr/lib/chromium
 5833 fred       20   0  471M 58808 17188 S  0.0  1.8 30:51.45 /usr/lib/firefox/
10672 fred       20   0  233M 57940 32432 S  0.0  1.7  0:52.50 /usr/lib/libreoff
 1642 fred       25   5  198M 53700 15712 S  0.0  1.6  2:03.73 /usr/lib/chromium
 6389 fred       20   0 94824 52236  4504 S  0.0  1.6  7:05.66 skype
 7785 http       20   0 69784 50564  2176 S  0.0  1.5 18:45.72 /usr/sbin/httpd -
 1072 http       20   0 67064 50268  2176 S  0.0  1.5 21:18.12 /usr/sbin/httpd -
 1219 root       20   0  134M 49952 12092 S  2.0  1.5 37:08.20 /usr/bin/X -nolis
F1Help  F2Setup F3SearchF4FilterF5Tree  F6SortByF7Nice -F8Nice +F9Kill  F10Quit

Here is top:

Code:

top - 16:48:13 up 1 day, 19:42,  4 users,  load average: 0.05, 0.03, 0.05
Tasks: 170 total,   1 running, 169 sleeping,   0 stopped,   0 zombie
Cpu(s):  0.8%us,  0.8%sy,  0.0%ni, 98.3%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:   3349920k total,  2919088k used,   430832k free,     4576k buffers
Swap:  9310012k total,   156856k used,  9153156k free,   287868k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND            
 1219 root      20   0  134m  48m  11m S    4  1.5  37:08.93 X                  
 6478 fred      20   0 70820  16m 9836 S    2  0.5   0:33.20 Terminal           
15442 fred      20   0  4916 1748 1156 S    1  0.1   0:00.51 htop               
 1699 fred      20   0  201m  33m  16m S    1  1.0  24:16.41 chromium           
 5833 fred      20   0  471m  57m  16m S    1  1.8  30:51.57 plugin-containe    
 1459 fred      20   0 1190m 737m  27m S    0 22.5  49:41.33 firefox            
 1581 fred      20   0  396m  68m  26m S    0  2.1   8:21.50 chromium           
 6389 fred      20   0 94824  51m 4504 S    0  1.6   7:05.70 skype              
    1 root      20   0  1972  520  492 S    0  0.0   0:01.96 init               
    2 root      20   0     0    0    0 S    0  0.0   0:00.01 kthreadd           
    3 root      20   0     0    0    0 S    0  0.0   0:03.57 ksoftirqd/0        
    5 root      20   0     0    0    0 S    0  0.0   0:00.00 kworker/u:0        
    6 root      RT   0     0    0    0 S    0  0.0   0:00.00 migration/0        
    7 root      RT   0     0    0    0 S    0  0.0   0:00.46 watchdog/0         
    8 root      RT   0     0    0    0 S    0  0.0   0:00.00 migration/1        
   10 root      20   0     0    0    0 S    0  0.0   0:04.62 ksoftirqd/1        
   12 root      RT   0     0    0    0 S    0  0.0   0:00.42 watchdog/1

free and /etc/rc.conf

Code:

[fred@arch ~]$ free
             total       used       free     shared    buffers     cached
Mem:       3349920    2919964     429956          0       4596     288204
-/+ buffers/cache:    2627164     722756
Swap:      9310012     156856    9153156
[fred@arch ~]$ free -m
             total       used       free     shared    buffers     cached
Mem:          3271       2851        419          0          4        281
-/+ buffers/cache:       2565        705
Swap:         9091        153       8938
[fred@arch ~]$ more /etc/rc.conf
#
# /etc/rc.conf - Main Configuration for Arch Linux
#

# -----------------------------------------------------------------------
# LOCALIZATION
# -----------------------------------------------------------------------
#
# LOCALE: available languages can be listed with the 'locale -a' command
# DAEMON_LOCALE: If set to 'yes', use $LOCALE as the locale during daemon
# startup and during the boot process. If set to 'no', the C locale is used.
# HARDWARECLOCK: set to "UTC" or "localtime", any other value will result
#   in the hardware clock being left untouched (useful for virtualization)
#   Note: Using "localtime" is discouraged.
# TIMEZONE: timezones are found in /usr/share/zoneinfo
# KEYMAP: keymaps are found in /usr/share/kbd/keymaps
# CONSOLEFONT: found in /usr/share/kbd/consolefonts (only needed for non-US)
# CONSOLEMAP: found in /usr/share/kbd/consoletrans
# USECOLOR: use ANSI color sequences in startup messages
# VERBOSE: Verbose level (from 1 to 8). man 3 syslog for level info
#
LOCALE="en_US.UTF-8"
DAEMON_LOCALE="no"
HARDWARECLOCK="UTC"
TIMEZONE="Asia/Jerusalem"
KEYMAP="us"
CONSOLEFONT=
CONSOLEMAP=
USECOLOR="yes"
VERBOSE="3"

# -----------------------------------------------------------------------
# HARDWARE
# -----------------------------------------------------------------------
#
# MODULES: Modules to load at boot-up. Blacklisting is no longer supported.
#   Replace every !module by an entry as on the following line in a file in
#   /etc/modprobe.d:
#     blacklist module
#   See "man modprobe.conf" for details.
#
MODULES=(vboxdrv)

# Udev settle timeout (default to 30)
UDEV_TIMEOUT=30

# Scan for FakeRAID (dmraid) Volumes at startup
USEDMRAID="no"

# Scan for BTRFS volumes at startup
USEBTRFS="no"

# Scan for LVM volume groups at startup, required if you use LVM
USELVM="no"

# -----------------------------------------------------------------------
# NETWORKING
# -----------------------------------------------------------------------
#
# HOSTNAME: Hostname of machine. Should also be put in /etc/hosts
#
HOSTNAME="arch"

# Use 'ip addr' or 'ls /sys/class/net/' to see all available interfaces.
#
# Wired network setup
#   - interface: name of device (required)
#   - address: IP address (leave blank for DHCP)
#   - netmask: subnet mask (ignored for DHCP)
#   - gateway: default route (ignored for DHCP)
# 
# Static IP example
# interface=eth0
# address=192.168.0.2
# netmask=255.255.255.0
# gateway=192.168.0.1
#
# DHCP example
# interface=eth0
# address=
# netmask=
# gateway=

interface=eth0
address=
netmask=
gateway=

# Setting this to "yes" will skip network shutdown.
# This is required if your root device is on NFS.
NETWORK_PERSIST="no"

# Enable these netcfg profiles at boot-up. These are useful if you happen to
# need more advanced network features than the simple network service
# supports, such as multiple network configurations (ie, laptop users)
#   - set to 'menu' to present a menu during boot-up (dialog package required)
#   - prefix an entry with a ! to disable it
#
# Network profiles are found in /etc/network.d
#
# This requires the netcfg package
#
#NETWORKS=(main)

# -----------------------------------------------------------------------
# DAEMONS
# -----------------------------------------------------------------------
#
# Daemons to start at boot-up (in this order)
#   - prefix a daemon with a ! to disable it
#   - prefix a daemon with a @ to start it up in the background
#
# If something other takes care of your hardware clock (ntpd, dual-boot...)
# you should disable 'hwclock' here.
#
DAEMONS=(!syslog-ng @dbus @network @ntpd @netfs @crond @httpd @mysqld @oss @zramswap)

Here is ps_mem. It may not be 100% clear, but at least the relative numbers tell something:

Code:

[fred@arch ~]$ sudo ps_mem
Password: 
 Private  +   Shared  =  RAM used	Program 

  4.0 KiB +  21.5 KiB =  25.5 KiB	ck-launch-session
  4.0 KiB +  21.5 KiB =  25.5 KiB	xinit
  4.0 KiB +  22.0 KiB =  26.0 KiB	dbus-launch
 24.0 KiB +  29.5 KiB =  53.5 KiB	rtkit-daemon
 48.0 KiB +  12.5 KiB =  60.5 KiB	init
  4.0 KiB +  64.0 KiB =  68.0 KiB	login
  4.0 KiB +  76.5 KiB =  80.5 KiB	dconf-service
  4.0 KiB +  77.5 KiB =  81.5 KiB	sh
  4.0 KiB +  78.5 KiB =  82.5 KiB	startx
  4.0 KiB +  80.0 KiB =  84.0 KiB	mysqld_safe
 72.0 KiB +  15.0 KiB =  87.0 KiB	dhcpcd
 24.0 KiB +  70.5 KiB =  94.5 KiB	libreoffice
  4.0 KiB +  94.5 KiB =  98.5 KiB	gconf-helper
  4.0 KiB +  96.0 KiB = 100.0 KiB	at-spi-bus-launcher
 20.0 KiB +  80.0 KiB = 100.0 KiB	agetty (5)
 44.0 KiB +  75.5 KiB = 119.5 KiB	gvfsd-dnssd
104.0 KiB +  18.5 KiB = 122.5 KiB	gnome-pty-helper
112.0 KiB +  12.5 KiB = 124.5 KiB	crond
112.0 KiB +  19.0 KiB = 131.0 KiB	gpg-agent
 84.0 KiB +  60.5 KiB = 144.5 KiB	gvfsd-metadata
160.0 KiB +  64.0 KiB = 224.0 KiB	gvfsd
144.0 KiB +  80.5 KiB = 224.5 KiB	gvfs-fuse-daemon
180.0 KiB + 110.5 KiB = 290.5 KiB	gvfsd-network
140.0 KiB + 156.5 KiB = 296.5 KiB	oosplash.bin
224.0 KiB + 101.5 KiB = 325.5 KiB	at-spi2-registryd
260.0 KiB +  69.5 KiB = 329.5 KiB	xfconfd
284.0 KiB +  47.0 KiB = 331.0 KiB	ntpd
264.0 KiB +  71.5 KiB = 335.5 KiB	jeditbg
352.0 KiB +  39.0 KiB = 391.0 KiB	top
272.0 KiB + 141.0 KiB = 413.0 KiB	xfsettingsd
296.0 KiB + 170.0 KiB = 466.0 KiB	console-kit-daemon
380.0 KiB +  95.5 KiB = 475.5 KiB	plugplay.exe
416.0 KiB +  62.5 KiB = 478.5 KiB	gconfd-2
396.0 KiB + 113.5 KiB = 509.5 KiB	gvfsd-trash
272.0 KiB + 253.0 KiB = 525.0 KiB	udevd (3)
388.0 KiB + 144.0 KiB = 532.0 KiB	upowerd
  4.5 MiB + -3994.0 KiB = 626.0 KiB	winedevice.exe
452.0 KiB + 182.5 KiB = 634.5 KiB	pulseaudio
456.0 KiB + 181.0 KiB = 637.0 KiB	polkit-gnome-authentication-agent-1
604.0 KiB +  41.0 KiB = 645.0 KiB	sudo
556.0 KiB + 116.5 KiB = 672.5 KiB	services.exe
500.0 KiB + 205.0 KiB = 705.0 KiB	xfce4-settings-helper
624.0 KiB +  99.0 KiB = 723.0 KiB	gvfs-gdu-volume-monitor
544.0 KiB + 267.0 KiB = 811.0 KiB	panel-6-systray
712.0 KiB + 137.0 KiB = 849.0 KiB	gnome-keyring-daemon
808.0 KiB +  44.0 KiB = 852.0 KiB	htop
  4.7 MiB + -3862.5 KiB = 961.5 KiB	udisks-daemon (2)
688.0 KiB + 329.0 KiB =   1.0 MiB	xfce4-power-manager (2)
  4.7 MiB + -3784.0 KiB =   1.0 MiB	nm-applet
856.0 KiB + 184.0 KiB =   1.0 MiB	gnome-settings-daemon
  1.0 MiB + 106.5 KiB =   1.1 MiB	polkitd [updated]
928.0 KiB + 354.5 KiB =   1.3 MiB	panel-14-dateti
  1.2 MiB + 519.0 KiB =   1.7 MiB	bash (4)
  1.8 MiB + 259.0 KiB =   2.0 MiB	xfce4-session
  1.7 MiB + 374.5 KiB =   2.0 MiB	dbus-daemon (3)
  1.5 MiB + 653.5 KiB =   2.1 MiB	xfdesktop
  1.7 MiB + 548.0 KiB =   2.3 MiB	explorer.exe
  2.7 MiB +  20.5 KiB =   2.7 MiB	wineserver
  2.4 MiB + 819.0 KiB =   3.2 MiB	panel-9-xfce4-n
  2.9 MiB + 535.5 KiB =   3.4 MiB	xfce4-xkb-plugin
  5.1 MiB + 481.5 KiB =   5.5 MiB	panel-5-xfce4-c
  5.1 MiB + 689.0 KiB =   5.8 MiB	xfwm4
115.6 MiB + -110262.0 KiB =   8.0 MiB	xfce4-netload-plugin
  7.7 MiB + 867.5 KiB =   8.6 MiB	Terminal
  8.0 MiB + 847.0 KiB =   8.8 MiB	hamster-service
 13.0 MiB + -3075.0 KiB =  10.0 MiB	filezilla
 13.8 MiB + -2704.5 KiB =  11.1 MiB	hamster-tray
 16.4 MiB + -2810.0 KiB =  13.7 MiB	Thunar
 13.6 MiB + 826.5 KiB =  14.4 MiB	xfce4-panel
 14.5 MiB + 836.5 KiB =  15.4 MiB	SQLyogCommunity
 47.0 MiB + -16073.0 KiB =  31.3 MiB	skype
167.1 MiB + -135059.5 KiB =  35.2 MiB	mysqld
 38.5 MiB + 581.5 KiB =  39.0 MiB	Xorg
148.0 MiB + -108600.5 KiB =  42.0 MiB	java
 49.3 MiB + -7141.0 KiB =  42.3 MiB	soffice.bin
 43.7 MiB +   5.4 MiB =  49.2 MiB	plugin-container
399.7 MiB + -281018.5 KiB = 125.3 MiB	httpd (12)
217.6 MiB + -29451.0 KiB = 188.8 MiB	chromium (11)
330.7 MiB + -96333.5 KiB = 236.6 MiB	thunderbird-bin
732.5 MiB + -83329.5 KiB = 651.2 MiB	firefox
---------------------------------
                          1.5 GiB
=================================

 Private  +   Shared  =  RAM used	Program

Even if Firefox is a hog (as Thunderbird also is sometimes) it's 700 or 800 M shouldn't cause all the headaches I have. Oddly, I turned off my Xfce panel CPU monitor because it was occasionally coming to the top of htop as using a lot of RAM. No idea why.

FredJones · 12-08-2011, 09:03 AM

Quote:

Originally Posted by H_TeXMeX_H

You are using a lot of apps that are leaky and eat up a lot of RAM. I would never run all of those at once. From what I see above, I think some part of XFCE may be leaking as well as Firefox (update to latest version). Maybe try another desktop environment other than XFCE.

I need to run the apps to work.

I thought 4G (even 3.5) was a lot and it could handle all this. Maybe I am wrong. Your point about Xfce is interesting and pre-empts my previous post where I mentioned the CPU monitor (we posted simultaneously it seems) and I suppose that to try another DE is actually a very good idea. I could try and see what happens.

H_TeXMeX_H · 12-08-2011, 09:24 AM

Firefox is the main issue I think. Maybe try removing plugins or extensions that you don't need.

H_TeXMeX_H · 12-12-2011, 12:29 PM

Here's another thing to try:

Code:

echo 1 > /proc/sys/vm/dirty_background_ratio

I have noticed that when the disk / page cache gets full, the system starts to become unresponsive with short hangs. Using the above setting it helps a lot.

I'll experiment with some options to try and solve the issue. Probably a definitive solution is more RAM.

Also see:
http://www.westnet.com/~gsmith/conte...ux-pdflush.htm
http://kmaiti.blogspot.com/2011/09/c...cache-and.html

FredJones · 12-28-2011, 08:23 AM

I got a 500 G USB and put on it a swap partition. I tried adding it in /etc/fstab but I failed. So now I have no swap at all, temporarily.

I rebooted and when I just launch a few apps on my fresh Xfce desktop, I hear the disk drive grinding already. I opened up a terminal and htop said 400/3271MB and then 720/3271MB when all 6 apps were launched.

And this is without Firefox.

My point is that the disk is grinding and there's LOTS of free RAM. That's how it apears to me anyhow.

If I open Thunar and browse to /srv/http and then click the little icon to show all the directories in the left sidebar, I hear the disk grind for 3 seconds before it finishes and shows the directories. There are 44 directories plus 69 files in this /srv/http . Seems bizarre that it should grind, and this with 1450/3271MB Mem in htop.

This grinding has nothing to do with RAM nor swap, as far as I can see.

I ran

echo 1 > /proc/sys/vm/dirty_background_ratio

and then played with Thunar and it still grinds. So on the one hand, this seems unrelated. This setting is a RAM paging issue and I am testing disk read I guess. But perhaps the above demonstrates that just reading from the disk is NOT working as it should. Because the physical RAM is not even 1/3 in use.

Now I figured out how to manually add the swap partition so I have

Code:

$ free -m
             total       used       free     shared    buffers     cached
Mem:          3271       2444        827          0         45        644
-/+ buffers/cache:       1753       1517
Swap:         4999         51       4948

An hour after I took the above "free -m" I restarted Firefox now and the disk started to grind. I ran htop after a second or two and it said Mem 1600/3271MB and Swp 58/4999MB. I didn't time it but the grinding noise lasted like 15 seconds. Then I restarted it again, right away, before doing anything and it restarted (with 7 open tabs to load) in 5 seconds with no grinding.

I am confused and quite discouraged. Unless I am misunderstanding something, this is not a RAM problem, because I am seeing slowness and grinding with NO swap being used or with SWAP on a separate physical machine.

Here is another experiment, a somewhat scientific one. I made a bash script:

Code:

#!/bin/bash
rm -f lines.txt
date
for i in {1..5000000}
do
   echo "Line Number $i" >> lines.txt
done
echo "finished" >> lines.txt
date
grep "finished" lines.txt
date
ls lines.txt -l

This is what I see when I run it:

Code:

$ ./lines.sh
Wed Dec 28 16:11:19 IST 2011
Wed Dec 28 16:14:05 IST 2011
finished
Wed Dec 28 16:14:05 IST 2011
-rw-r--r-- 1 kirk kirk 98888905 Dec 28 16:14 lines.txt

Close to 3 minutes to make a 94 MB file. This is with me doing very little else on the machine during that time. Can anyone else also run this and see?

I unfortunately don't have any other ideas for clear tests to run.

I have only two ideas left:

1. Use PAE Kernel, but this will only gain me 0.5 G physical RAM and so I don't see much reason to think it will fix my other problems
2. Reinstall using 64 bit Kernel, perhaps on a different drive. Truth is this is already a replacement drive, but I used Clonezilla to copy from the old one and maybe there is something wrong with the setup or partitions. Don't know what, but maybe.

I could try a brand new install on a separate disk and then have both disks in the same PC and see if it helps...

H_TeXMeX_H · 12-28-2011, 08:46 AM

Maybe there are a lot of fragmented files. If you want, I wrote an experimental defrag script that you can try, it is on my site.
http://htexmexh.my3gb.com/linux/scripts.html

FredJones · 12-28-2011, 10:33 AM

Doesn't seem like it found much:

Code:

$ sudo ./localfinddefrag
Password: 
find: `/home/kirk/#swap.txt#': No such file or directory
find: `/home/kirk/.gvfs': Permission denied
Ouput written to /home/kirk/.defrag/output
Sorted output written to /home/kirk/.defrag/sorted

Most fragmented files
1|/home/kirk/.wine/drive_c/windows/winsxs/x86_microsoft.vc90.crt_1fc8b3b9a1e18e3b_9.0.30729.4148_none_deadbeef/msvcr90.dll
1|/home/kirk/.wine/drive_c/windows/winsxs/x86_microsoft.windows.gdiplus_6595b64144ccf1df_1.0.6000.16386_none_deadbeef/gdiplus.dll
1|/home/kirk/.wine/.update-timestamp
1|/home/kirk/.wine/userdef.reg
1|/home/kirk/.Xauthority
1|/home/kirk/.xchat2/keybindings.conf
1|/home/kirk/.xchat2/scrollback/FreeNode/seank.txt
1|/home/kirk/.xchat2/xchat.conf
1|/home/kirk/.xinitrc
1|/home/kirk/xx.sql

I do use one Windows app in Wine: SQLyog. It's the one app I haven't found a good replacement for, yet, in native Linux.

TobiSGD · 12-28-2011, 11:45 AM

Quote:

Originally Posted by FredJones

This is what I see when I run it:

Code:

$ ./lines.sh
Wed Dec 28 16:11:19 IST 2011
Wed Dec 28 16:14:05 IST 2011
finished
Wed Dec 28 16:14:05 IST 2011
-rw-r--r-- 1 kirk kirk 98888905 Dec 28 16:14 lines.txt

Close to 3 minutes to make a 94 MB file. This is with me doing very little else on the machine during that time. Can anyone else also run this and see?

Tried that script:

Code:

Wed Dec 28 18:41:34 CET 2011
Wed Dec 28 18:43:20 CET 2011
finished
Wed Dec 28 18:43:20 CET 2011
-rw-r--r-- 1 tobi users 98888905 Dec 28 18:43 lines.txt

This is on a six core machine with Slackware 13.37 64 bit on 8GB of RAM, the script was run on a RAID 0-array.

H_TeXMeX_H · 12-28-2011, 12:33 PM

It takes longer than 3 minutes on my netbook. I didn't wait for it, because it was taking too long.

johnsfine · 12-28-2011, 01:02 PM

I tried that on an old (hardware and software) Centos system that I happened to have a putty window open on and that was otherwise idle. It took 129 seconds. But then I realized my home directory there is an NFS mount, which makes exactly this kind of file I/O much slower than local (even though the LAN is 1Gb and the file server super fast RAID and heavily cached with ram and lightly loaded.)

So I switched to a local directory and tried it. That took 103 seconds.

I was curious how much of that time was disk I/O, so I switched to a directory on a tmpfs (with enough free ram that this test wouldn't cause tmpfs to use any SWAP) and the test took 88 seconds.

I don't really understand my middle result. I expect write behind caching to be very limited when using NFS, so the extra time there is reasonable for lots of waits for short messages across the LAN. But I expected writes to a local disk file (on a system with lots of free ram) would just go into "write behind caching" and be nearly as fast as ram. The actual disk writes should occur after the script thinks it is done.

FredJones · 12-28-2011, 01:13 PM

Quote:

Originally Posted by H_TeXMeX_H

Maybe there are a lot of fragmented files. If you want, I wrote an experimental defrag script that you can try, it is on my site.
http://htexmexh.my3gb.com/linux/scripts.html

I looked at .defrag/output now. Lines 194 through 1108 are in ~/.mozilla/firefox/

Then 1111 through 1901 are ~/.wine/drive_c/

2162 - 9835 are ~/.thumbnails/

Lots from my Trash, too.

In the end there are 357120 lines. The file is 34 MB. Perhaps that IS a lot actually. Most are "2 extents found, perfection would be 1 extent" A few are 3, 4 and 5, but just browsing through I don't see any bigger numbers than that.

H_TeXMeX_H · 12-28-2011, 02:18 PM

Fragmentation doesn't seem to be a problem. I never get 1, in fact I'm happy with 20-40 extents.

It must be something else.

Here, I ran the script:

Code:

bash-4.1$ ./test
Wed Dec 28 21:09:40 CET 2011
Wed Dec 28 21:16:06 CET 2011
finished
Wed Dec 28 21:16:06 CET 2011
-rw-r--r-- 1 demonslayer users 98888905 Dec 28 21:16 lines.txt

it took about 7 minutes.

H_TeXMeX_H · 12-29-2011, 08:50 AM

I would uses sysbench for more benchmarks:
http://sourceforge.net/projects/sysbench/

You may need to replace 'AC_PROG_LIBTOOL' with 'AC_PROG_RANLIB' in configure.ac and run 'autoconf' and './augen.sh' again for it to compile.

Your script uses a lot of CPU usage, so it may actually be testing how good your CPU is.

FredJones · 12-29-2011, 10:34 AM

I installed sysbench 0.4.12-1, a build for ArchLinux so I presume it matches my setup.

Using this bash script:

Code:

date
sysbench --num-threads=16 --test=fileio --file-total-size=3G --file-test-mode=rndrw prepare
date
sysbench --num-threads=16 --test=fileio --file-total-size=3G --file-test-mode=rndrw run
date
sysbench --num-threads=16 --test=fileio --file-total-size=3G --file-test-mode=seqrd run
date
sysbench --num-threads=16 --test=fileio --file-total-size=3G --file-test-mode=rndrw cleanup
date

I see this:

Code:

$ ./mysysb.sh
Thu Dec 29 18:20:02 IST 2011
sysbench 0.4.12:  multi-threaded system evaluation benchmark

128 files, 24576Kb each, 3072Mb total
Creating files for the test...
Thu Dec 29 18:20:42 IST 2011
sysbench 0.4.12:  multi-threaded system evaluation benchmark

Running the test with following options:
Number of threads: 16

Extra file open flags: 0
128 files, 24Mb each
3Gb total file size
Block size 16Kb
Number of random requests for random IO: 10000
Read/Write ratio for combined random IO test: 1.50
Periodic FSYNC enabled, calling fsync() each 100 requests.
Calling fsync() at the end of test, Enabled.
Using synchronous I/O mode
Doing random r/w test
Threads started!
Done.

Operations performed:  6013 Read, 3998 Write, 12802 Other = 22813 Total
Read 93.953Mb  Written 62.469Mb  Total transferred 156.42Mb  (2.3925Mb/sec)
  153.12 Requests/sec executed

Test execution summary:
    total time:                          65.3801s
    total number of events:              10011
    total time taken by event execution: 605.9701
    per-request statistics:
         min:                                  0.00ms
         avg:                                 60.53ms
         max:                               1031.56ms
         approx.  95 percentile:             379.01ms

Threads fairness:
    events (avg/stddev):           625.6875/67.78
    execution time (avg/stddev):   37.8731/0.85

Thu Dec 29 18:21:48 IST 2011
sysbench 0.4.12:  multi-threaded system evaluation benchmark

Running the test with following options:
Number of threads: 16

Extra file open flags: 0
128 files, 24Mb each
3Gb total file size
Block size 16Kb
Periodic FSYNC enabled, calling fsync() each 100 requests.
Calling fsync() at the end of test, Enabled.
Using synchronous I/O mode
Doing sequential read test
Threads started!
Done.

Operations performed:  196608 Read, 0 Write, 0 Other = 196608 Total
Read 3Gb  Written 0b  Total transferred 3Gb  (103.59Mb/sec)
 6629.87 Requests/sec executed

Test execution summary:
    total time:                          29.6549s
    total number of events:              196608
    total time taken by event execution: 474.2813
    per-request statistics:
         min:                                  0.00ms
         avg:                                  2.41ms
         max:                                319.21ms
         approx.  95 percentile:               5.15ms

Threads fairness:
    events (avg/stddev):           12288.0000/382.08
    execution time (avg/stddev):   29.6426/0.00

Thu Dec 29 18:22:18 IST 2011
sysbench 0.4.12:  multi-threaded system evaluation benchmark

Removing test files...
Thu Dec 29 18:22:18 IST 2011

and then I did this:

Code:

date
sysbench --test=memory --memory-block-size=128K --memory-scope=global --memory-total-size=100G --memory-oper=write --num-threads=1 run
sysbench --test=memory --memory-block-size=128K --memory-scope=global --memory-total-size=100G --memory-oper=write --num-threads=2 run
sysbench --test=memory --memory-block-size=128K --memory-scope=global --memory-total-size=100G --memory-oper=write --num-threads=4 run
sysbench --test=memory --memory-block-size=128K --memory-scope=global --memory-total-size=100G --memory-oper=write --num-threads=32 run
sysbench --test=memory --memory-block-size=128K --memory-scope=global --memory-total-size=100G --memory-oper=write --num-threads=64 run
sysbench --test=memory --memory-block-size=128K --memory-scope=global --memory-total-size=100G --memory-oper=write --num-threads=128 run
sysbench --test=memory --memory-block-size=16K --memory-scope=global --memory-total-size=100G --memory-oper=write --num-threads=1 run
sysbench --test=memory --memory-block-size=16K --memory-scope=global --memory-total-size=100G --memory-oper=write --num-threads=2 run
sysbench --test=memory --memory-block-size=16K --memory-scope=global --memory-total-size=100G --memory-oper=write --num-threads=4 run
sysbench --test=memory --memory-block-size=16K --memory-scope=global --memory-total-size=100G --memory-oper=write --num-threads=32 run
sysbench --test=memory --memory-block-size=16K --memory-scope=global --memory-total-size=100G --memory-oper=write --num-threads=64 run
sysbench --test=memory --memory-block-size=16K --memory-scope=global --memory-total-size=100G --memory-oper=write --num-threads=128 run
date

and get this:

Code:

$ ./mysysb.sh
Thu Dec 29 18:26:33 IST 2011
sysbench 0.4.12:  multi-threaded system evaluation benchmark

Running the test with following options:
Number of threads: 1

Doing memory operations speed test
Memory block size: 128K

Memory transfer size: 0M

Memory operations type: write
Memory scope type: global
Threads started!
Done.

Operations performed: 0 (    0.00 ops/sec)

0.00 MB transferred (0.00 MB/sec)


Test execution summary:
    total time:                          0.0001s
    total number of events:              0
    total time taken by event execution: 0.0000
    per-request statistics:
         min:                            18446744073709.55ms
         avg:                                  0.00ms
         max:                                  0.00ms

Threads fairness:
    events (avg/stddev):           0.0000/0.00
    execution time (avg/stddev):   0.0000/0.00

sysbench 0.4.12:  multi-threaded system evaluation benchmark

Running the test with following options:
Number of threads: 2

Doing memory operations speed test
Memory block size: 128K

Memory transfer size: 0M

Memory operations type: write
Memory scope type: global
Threads started!
Done.

Operations performed: 0 (    0.00 ops/sec)

0.00 MB transferred (0.00 MB/sec)


Test execution summary:
    total time:                          0.0001s
    total number of events:              0
    total time taken by event execution: 0.0000
    per-request statistics:
         min:                            18446744073709.55ms
         avg:                                  0.00ms
         max:                                  0.00ms

Threads fairness:
    events (avg/stddev):           0.0000/0.00
    execution time (avg/stddev):   0.0000/0.00

sysbench 0.4.12:  multi-threaded system evaluation benchmark

Running the test with following options:
Number of threads: 4

Doing memory operations speed test
Memory block size: 128K

Memory transfer size: 0M

Memory operations type: write
Memory scope type: global
Threads started!
Done.

Operations performed: 0 (    0.00 ops/sec)

0.00 MB transferred (0.00 MB/sec)


Test execution summary:
    total time:                          0.0001s
    total number of events:              0
    total time taken by event execution: 0.0000
    per-request statistics:
         min:                            18446744073709.55ms
         avg:                                  0.00ms
         max:                                  0.00ms

Threads fairness:
    events (avg/stddev):           0.0000/0.00
    execution time (avg/stddev):   0.0000/0.00

sysbench 0.4.12:  multi-threaded system evaluation benchmark

Running the test with following options:
Number of threads: 32

Doing memory operations speed test
Memory block size: 128K

Memory transfer size: 0M

Memory operations type: write
Memory scope type: global
Threads started!
Done.

Operations performed: 0 (    0.00 ops/sec)

0.00 MB transferred (0.00 MB/sec)


Test execution summary:
    total time:                          0.0008s
    total number of events:              0
    total time taken by event execution: 0.0000
    per-request statistics:
         min:                            18446744073709.55ms
         avg:                                  0.00ms
         max:                                  0.00ms

Threads fairness:
    events (avg/stddev):           0.0000/0.00
    execution time (avg/stddev):   0.0000/0.00

sysbench 0.4.12:  multi-threaded system evaluation benchmark

Running the test with following options:
Number of threads: 64

Doing memory operations speed test
Memory block size: 128K

Memory transfer size: 0M

Memory operations type: write
Memory scope type: global
Threads started!
Done.

Operations performed: 0 (    0.00 ops/sec)

0.00 MB transferred (0.00 MB/sec)


Test execution summary:
    total time:                          0.0012s
    total number of events:              0
    total time taken by event execution: 0.0000
    per-request statistics:
         min:                            18446744073709.55ms
         avg:                                  0.00ms
         max:                                  0.00ms

Threads fairness:
    events (avg/stddev):           0.0000/0.00
    execution time (avg/stddev):   0.0000/0.00

sysbench 0.4.12:  multi-threaded system evaluation benchmark

Running the test with following options:
Number of threads: 128

Doing memory operations speed test
Memory block size: 128K

Memory transfer size: 0M

Memory operations type: write
Memory scope type: global
Threads started!
Done.

Operations performed: 0 (    0.00 ops/sec)

0.00 MB transferred (0.00 MB/sec)


Test execution summary:
    total time:                          0.0024s
    total number of events:              0
    total time taken by event execution: 0.0000
    per-request statistics:
         min:                            18446744073709.55ms
         avg:                                  0.00ms
         max:                                  0.00ms

Threads fairness:
    events (avg/stddev):           0.0000/0.00
    execution time (avg/stddev):   0.0000/0.00

sysbench 0.4.12:  multi-threaded system evaluation benchmark

Running the test with following options:
Number of threads: 1

Doing memory operations speed test
Memory block size: 16K

Memory transfer size: 0M

Memory operations type: write
Memory scope type: global
Threads started!
Done.

Operations performed: 0 (    0.00 ops/sec)

0.00 MB transferred (0.00 MB/sec)


Test execution summary:
    total time:                          0.0000s
    total number of events:              0
    total time taken by event execution: 0.0000
    per-request statistics:
         min:                            18446744073709.55ms
         avg:                                  0.00ms
         max:                                  0.00ms

Threads fairness:
    events (avg/stddev):           0.0000/0.00
    execution time (avg/stddev):   0.0000/0.00

sysbench 0.4.12:  multi-threaded system evaluation benchmark

Running the test with following options:
Number of threads: 2

Doing memory operations speed test
Memory block size: 16K

Memory transfer size: 0M

Memory operations type: write
Memory scope type: global
Threads started!
Done.

Operations performed: 0 (    0.00 ops/sec)

0.00 MB transferred (0.00 MB/sec)


Test execution summary:
    total time:                          0.0000s
    total number of events:              0
    total time taken by event execution: 0.0000
    per-request statistics:
         min:                            18446744073709.55ms
         avg:                                  0.00ms
         max:                                  0.00ms

Threads fairness:
    events (avg/stddev):           0.0000/0.00
    execution time (avg/stddev):   0.0000/0.00

sysbench 0.4.12:  multi-threaded system evaluation benchmark

Running the test with following options:
Number of threads: 4

Doing memory operations speed test
Memory block size: 16K

Memory transfer size: 0M

Memory operations type: write
Memory scope type: global
Threads started!
Done.

Operations performed: 0 (    0.00 ops/sec)

0.00 MB transferred (0.00 MB/sec)


Test execution summary:
    total time:                          0.0001s
    total number of events:              0
    total time taken by event execution: 0.0000
    per-request statistics:
         min:                            18446744073709.55ms
         avg:                                  0.00ms
         max:                                  0.00ms

Threads fairness:
    events (avg/stddev):           0.0000/0.00
    execution time (avg/stddev):   0.0000/0.00

sysbench 0.4.12:  multi-threaded system evaluation benchmark

Running the test with following options:
Number of threads: 32

Doing memory operations speed test
Memory block size: 16K

Memory transfer size: 0M

Memory operations type: write
Memory scope type: global
Threads started!
Done.

Operations performed: 0 (    0.00 ops/sec)

0.00 MB transferred (0.00 MB/sec)


Test execution summary:
    total time:                          0.0006s
    total number of events:              0
    total time taken by event execution: 0.0000
    per-request statistics:
         min:                            18446744073709.55ms
         avg:                                  0.00ms
         max:                                  0.00ms

Threads fairness:
    events (avg/stddev):           0.0000/0.00
    execution time (avg/stddev):   0.0000/0.00

sysbench 0.4.12:  multi-threaded system evaluation benchmark

Running the test with following options:
Number of threads: 64

Doing memory operations speed test
Memory block size: 16K

Memory transfer size: 0M

Memory operations type: write
Memory scope type: global
Threads started!
Done.

Operations performed: 0 (    0.00 ops/sec)

0.00 MB transferred (0.00 MB/sec)


Test execution summary:
    total time:                          0.0012s
    total number of events:              0
    total time taken by event execution: 0.0000
    per-request statistics:
         min:                            18446744073709.55ms
         avg:                                  0.00ms
         max:                                  0.00ms

Threads fairness:
    events (avg/stddev):           0.0000/0.00
    execution time (avg/stddev):   0.0000/0.00

sysbench 0.4.12:  multi-threaded system evaluation benchmark

Running the test with following options:
Number of threads: 128

Doing memory operations speed test
Memory block size: 16K

Memory transfer size: 0M

Memory operations type: write
Memory scope type: global
Threads started!
Done.

Operations performed: 0 (    0.00 ops/sec)

0.00 MB transferred (0.00 MB/sec)


Test execution summary:
    total time:                          0.0024s
    total number of events:              0
    total time taken by event execution: 0.0000
    per-request statistics:
         min:                            18446744073709.55ms
         avg:                                  0.00ms
         max:                                  0.00ms

Threads fairness:
    events (avg/stddev):           0.0000/0.00
    execution time (avg/stddev):   0.0000/0.00

Thu Dec 29 18:26:33 IST 2011

Not really sure what tests I should be running here, but anyhow this whole second set seems odd. No operations performed. I got the above command from http://milek.blogspot.com/2007/12/sy...n-niagara.html

Then I tried

Code:

$ sysbench --test=memory --memory-block-size=16K --memory-total-size=100G --memory-scope=global --memory-hugetlb=off --memory-oper=write --memory-access-mode=seq run
sysbench 0.4.12:  multi-threaded system evaluation benchmark

Running the test with following options:
Number of threads: 1

Doing memory operations speed test
Memory block size: 16K

Memory transfer size: 0M

Memory operations type: write
Memory scope type: global
Threads started!
Done.

Operations performed: 0 (    0.00 ops/sec)

0.00 MB transferred (0.00 MB/sec)


Test execution summary:
    total time:                          0.0002s
    total number of events:              0
    total time taken by event execution: 0.0000
    per-request statistics:
         min:                            18446744073709.55ms
         avg:                                  0.00ms
         max:                                  0.00ms

Threads fairness:
    events (avg/stddev):           0.0000/0.00
    execution time (avg/stddev):   0.0000/0.00

from http://pastebin.com/H66QTu55 (random page I found) and also I see "Operations performed: 0 ( 0.00 ops/sec)"

Now here http://forums.tabcrawler.com/index.php?topic=4933.0 (another random page) I did this:

Code:

$ sysbench --test=cpu --cpu-max-prime=20000 run
sysbench 0.4.12:  multi-threaded system evaluation benchmark

Running the test with following options:
Number of threads: 1

Doing CPU performance benchmark

Threads started!
Done.

Maximum prime number checked in CPU test: 20000


Test execution summary:
    total time:                          26.3220s
    total number of events:              10000
    total time taken by event execution: 26.3184
    per-request statistics:
         min:                                  2.59ms
         avg:                                  2.63ms
         max:                                  4.44ms
         approx.  95 percentile:               2.78ms

Threads fairness:
    events (avg/stddev):           10000.0000/0.00
    execution time (avg/stddev):   26.3184/0.00

At least it DID something.