[SOLVED] Disk space usage extremely slow

mogmog · 06-17-2015, 05:13 AM

Hi, relative newbie to Linux (running Mint 17) & fairly lightweight. All of the utilities I've found for looking at disk usage are all painfully slow (presumably use the same underlying commands?) - taking 10, maybe 20 minutes to display my disk. On Windows, SequoiaView takes 30-60 seconds to analyse the disk. Is this normal?

In case it affects this, I'm dual booting W7 on a 320gb HDD & Linux on a separate 70gb HDD.

Many thanks

TobiSGD · 06-17-2015, 05:19 AM

I don't know about other programs, but ncdu scans the disk of my fileserver (about 1TB used on a 2TB disk, about 80,000 files) in under 20 seconds.

syg00 · 06-17-2015, 05:31 AM

Which utilities (GUI presumably) ? What filesystems (NTFS, FAT, ext4, xfs ... ?)

mogmog · 06-17-2015, 05:34 AM

Quote:

Originally Posted by TobiSGD

I don't know about other programs, but ncdu scans the disk of my fileserver (about 1TB used on a 2TB disk, about 80,000 files) in under 20 seconds.

Thanks. I'm looking into it. As mentioned, I'm a bit lightweight with Linux & trying to work out how to navigate to the system disk... I'm not sure that it is working as I can't get it to scan or abort, in fact it seems totally unresponsive. I'll leave it for half an hour to see if it is the same as the others.
Ta

mogmog · 06-17-2015, 06:08 AM

Quote:

Originally Posted by syg00

Which utilities (GUI presumably) ? What filesystems (NTFS, FAT, ext4, xfs ... ?)

Hi, Not entirely sure I understand the question: utilities tried? Disk Usage Analyser, Graphical Disk Map even KDirStat (before I understood the implications of adding KDE programs to Gnome.)

FS of Linux disk is ext4

FS of 320gb Storage disk is NTFS (original W7 disk) This latter is mounted through the Media directory which may be sub-optimal but I haven't managed/got round to mount it in a more useful way yet.
Ta

syg00 · 06-17-2015, 06:59 AM

That's what I was asking - Disk Usage Analyser is baobab, a gnome utility. It's slow, but not as bad as you are seeing. That's ridiculous. Post the output from this (in a terminal) to see if the Linux disk is ok (you may have to install smartmontools)

Code:

sudo smartctl -AH /dev/sda

Don't worry about the /media mounts - Ubuntu (hence Mint) do that by default - it's fine.

mogmog · 06-17-2015, 07:07 AM

syg00
Thanks. here it is. It is a very old disk but I have SSD to install when I can make the time.

Code:

 === START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x002f   200   200   051    Pre-fail  Always       -       5
  3 Spin_Up_Time            0x0027   134   130   021    Pre-fail  Always       -       4275
  4 Start_Stop_Count        0x0032   099   099   000    Old_age   Always       -       1310
  5 Reallocated_Sector_Ct   0x0033   199   199   140    Pre-fail  Always       -       1
  7 Seek_Error_Rate         0x002e   100   253   000    Old_age   Always       -       0
  9 Power_On_Hours          0x0032   087   087   000    Old_age   Always       -       9635
 10 Spin_Retry_Count        0x0032   100   100   000    Old_age   Always       -       0
 11 Calibration_Retry_Count 0x0032   100   100   000    Old_age   Always       -       0
 12 Power_Cycle_Count       0x0032   099   099   000    Old_age   Always       -       1223
192 Power-Off_Retract_Count 0x0032   200   200   000    Old_age   Always       -       151
193 Load_Cycle_Count        0x0032   200   200   000    Old_age   Always       -       1310
194 Temperature_Celsius     0x0022   108   090   000    Old_age   Always       -       35
196 Reallocated_Event_Count 0x0032   199   199   000    Old_age   Always       -       1
197 Current_Pending_Sector  0x0032   200   200   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0030   200   200   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x0032   200   200   000    Old_age   Always       -       0
200 Multi_Zone_Error_Rate   0x0008   200   200   000    Old_age   Offline      -       1
240 Head_Flying_Hours       0x0032   088   088   000    Old_age   Always       -       9449
241 Total_LBAs_Written      0x0032   200   200   000    Old_age   Always       -       9514083152
242 Total_LBAs_Read         0x0032   200   200   000    Old_age   Always       -       19336878527

syg00 · 06-17-2015, 07:22 AM

Looks ok - dunno what's the problem. Try this as well - and wrap [code] .... [/code] tags around any output you post to preserve formatting (go back and edit that last post pls)

Code:

vmstat 2 10

mogmog · 06-17-2015, 08:19 AM

Thank you for this. This is the output.
Regards

Code:

vmstat 2 10
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 4  0   3052 362984 1234308 1497144    0    0   110  2143   58  130 17  4 70 10  0
 0  0   3052 362056 1234344 1497196    0    0     0   356 1968 18340 14  4 81  2  0
 1  0   3052 361904 1234348 1497192    0    0     0   198 2201 20060 13  5 82  0  0
 1  0   3052 460412 1234348 1399452    0    0     0     0 2315 19751 29  6 65  0  0
 0  0   3052 361404 1234360 1497332    0    0     0 58094 2126 18875 12  5 60 23  0
 1  0   3052 361116 1234376 1497276    0    0     0   168 1948 18796 12  4 83  1  0
 1  0   3052 361808 1234424 1497312    0    0     0   350 1739 17952 12  4 83  2  0
 0  0   3052 362624 1234424 1497312    0    0     0    20 1730 17946 12  3 85  0  0
 2  0   3052 361148 1234440 1497304    0    0     0    62 1833 18836 15  4 81  0  0
 1  0   3052 355260 1234444 1497312    0    0     0   176 2441 24369 18  6 77  0  0

syg00 · 06-17-2015, 08:35 AM

Hmmm - that's starting to look more interesting - ignore the first line of numbers. The numbers don't line up properly (not your fault), but the interesting ones are under "in" (interrupts) and "cs" (context switches - switching from userspace to kernelspace or vice-versa).
Way too high - I'd say you have some hardware not properly seated (cables, disk), or more likely a dodgy driver. Very hard to track down.

mogmog · 06-18-2015, 02:11 AM

Quote:

Originally Posted by syg00

Hmmm - that's starting to look more interesting - ignore the first line of numbers. The numbers don't line up properly (not your fault), but the interesting ones are under "in" (interrupts) and "cs" (context switches - switching from userspace to kernelspace or vice-versa).
Way too high - I'd say you have some hardware not properly seated (cables, disk), or more likely a dodgy driver. Very hard to track down.

Thanks - that is a useful insight - what sort of values would you expect? Could this cause random momentary pauses in the system?
I've now also got the way paved for my SSD install so I'll reseat everything whilst I'm in there.

syg00 · 06-18-2015, 05:05 AM

Entirely dependent on machine, kernel, workload. But say divide your interrupt counts by 10, and "cs" say double that number. See the problem ?.
It could certainly affect things - interrupts run higher priority than kernel threads. And context switch stops whatever is doing the switch. Just bad all round. Try another (mates ?) machine and check.

mogmog · 06-26-2015, 01:40 PM

Hi, have migrated to SSD & all went pretty smoothly. Baobab took a reasonable amount of time - a long blink.
FWIW, I ran vmstat again...

Code:

procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 1  0      0 8207668 313884 2263480    0    0    24    57  223   34  4  1 94  0  0
 0  0      0 8207296 313892 2263512    0    0     0    26  521 2890  2  1 98  0  0
 0  0      0 8206800 313892 2263544    0    0     0     0  603 3363  2  0 98  0  0
 0  0      0 8206428 313892 2263544    0    0     0     0  529 2909  2  1 98  0  0
 1  0      0 8206188 313892 2263544    0    0     0     0  533 2908  1  1 98  0  0
 1  0      0 8205312 313896 2263544    0    0     0     6  668 4474  3  1 97  0  0
 0  0      0 8204336 313896 2263544    0    0     0    92  901 5854  6  2 92  0  0
 0  0      0 8207008 313896 2263544    0    0     0     0  554 3067  2  1 98  0  0
 0  0      0 8206700 313896 2263608    0    0     0     0  556 3050  2  1 97  0  0
 0  0      0 8208312 313896 2263608    0    0     0    16  538 2941  2  1 97  0  0

Thanks for your help.

Rambo1234 · 11-06-2017, 10:15 AM

While this thread is not a new one, i thought it was worth writing a response to it

to format vmstat in a nicer way try vmstat 1 10 -w

its worth noting the first line of the vmstat output provides a summary since boot,
& in the information supplied there is no information on uptime.

So trying to decide what is "normal" behavior needs to be seen in the context of

1)whats happening on the machine in question (backups/heavy read/writes)
2)how long the server has been up

and in my opinion you should dig a lot deeper than looking at 20 (vmstat 2 10 will run vmstat 10 times every 2 seconds, 10*2 =20) seconds worth of info from vmstat before replacing/buying new hardware. You also need to check the slowness actually happens when context switches are high and is quick when they are low.

Needless to say installing a new disk would have required a reboot and new updates i.e kernel could have been applied.

There is so much troubleshooting which could be done using sar/checking ps/checking crons/checking dmesg/.... (for more reading check out google searches for things like brendan gregg 60 second analysis)

Without clearly understanding the cause there is no guaranty buying new/installing hardware will solve the issue and stop it reoccurring.