LinuxQuestions.org
Share your knowledge at the LQ Wiki.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Server
User Name
Password
Linux - Server This forum is for the discussion of Linux Software used in a server related context.

Notices


Reply
  Search this Thread
Old 09-02-2019, 08:57 AM   #1
virtualmin2018
LQ Newbie
 
Registered: Jun 2019
Posts: 15

Rep: Reputation: Disabled
How are RAM used and how do you keep track on it?


Hello guys, recently I've uploaded a huge number of files to my website which crashed the SQL. It's used 63% of the total hard disk drive but the RAM usage is 100%. I discovered whilst I was uploading stuff somehow some of the files manage to slip into the RAM usage, is there a way to prevent this? thx.

OS:Centos 7
Panel: Virtualmin/webmin
 
Old 09-02-2019, 09:29 AM   #2
BW-userx
LQ Guru
 
Registered: Sep 2013
Location: Somewhere in my head.
Distribution: FreeBSD/Slackware-14.2+/ArcoLinux
Posts: 9,071

Rep: Reputation: 1901Reputation: 1901Reputation: 1901Reputation: 1901Reputation: 1901Reputation: 1901Reputation: 1901Reputation: 1901Reputation: 1901Reputation: 1901Reputation: 1901
upload files onto a server or just copying files onto a different drive (same thing). all files will go to disk. Maybe it was putting them into RAM because write was not fast enough to keep up with transferrer speeds. Therefore overloading RAM causing it to crash. BUFFER swap.. not large enough to compensate for the lack of amount of RAM needed to keep track of all of the files being sent over until it has time to complete writes to the disk of all of the files.

Last edited by BW-userx; 09-02-2019 at 09:41 AM.
 
2 members found this post helpful.
Old 09-02-2019, 09:38 AM   #3
scasey
Senior Member
 
Registered: Feb 2013
Location: Tucson, AZ, USA
Distribution: CentOS 7.6
Posts: 3,629

Rep: Reputation: 1204Reputation: 1204Reputation: 1204Reputation: 1204Reputation: 1204Reputation: 1204Reputation: 1204Reputation: 1204Reputation: 1204
Also, if your "upload" is also loading into a database, that process can consume RAM.
I think we need more, specific, information about what you were attempting to do.
 
Old 09-02-2019, 10:34 AM   #4
virtualmin2018
LQ Newbie
 
Registered: Jun 2019
Posts: 15

Original Poster
Rep: Reputation: Disabled
Quote:
Originally Posted by BW-userx View Post
upload files onto a server or just copying files onto a different drive (same thing). all files will go to disk. Maybe it was putting them into RAM because write was not fast enough to keep up with transferrer speeds. Therefore overloading RAM causing it to crash. BUFFER swap.. not large enough to compensate for the lack of amount of RAM needed to keep track of all of the files being sent over until it has time to complete writes to the disk of all of the files.
This explains a lot of stuff thanks! What would you recommend when uploading images/files to keep it up with transfer speeds?

Quote:
Originally Posted by scasey View Post
Also, if your "upload" is also loading into a database, that process can consume RAM.
I think we need more, specific, information about what you were attempting to do.
I was uploading huge files about 100MB for users to download using a wordpress plugin, however I added images to the post which were roughly in the 200s some of which is 1MB per file.
 
Old 09-02-2019, 11:39 AM   #5
BW-userx
LQ Guru
 
Registered: Sep 2013
Location: Somewhere in my head.
Distribution: FreeBSD/Slackware-14.2+/ArcoLinux
Posts: 9,071

Rep: Reputation: 1901Reputation: 1901Reputation: 1901Reputation: 1901Reputation: 1901Reputation: 1901Reputation: 1901Reputation: 1901Reputation: 1901Reputation: 1901Reputation: 1901
not sure how you're sending. However, try setting it up to send in limited bulk sends at x amount of files for each send until finished.
 
Old 09-12-2019, 01:46 AM   #6
MadeInGermany
Senior Member
 
Registered: Dec 2011
Location: Simplicity
Posts: 1,178

Rep: Reputation: 532Reputation: 532Reputation: 532Reputation: 532Reputation: 532Reputation: 532
If there is a shortage of RAM then the kernel should drastically shrink I/O buffers - eventually it means to pause a data transfer - but not deny an application‘s request for RAM.
Things like overcommit work out better if you have swap.
If you do not have swap space configured - configure it!
If you have enough swap, reach out for a better kernel!
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
LXer: Mozilla changes Firefox policy from 'do not track' to 'will not track' LXer Syndicated Linux News 0 09-02-2018 03:20 PM
C++, keep track of memory used by a certain class simopal6 Programming 1 02-11-2012 07:29 AM
LXer: Track Me! Just Track Me, GNOME Project! LXer Syndicated Linux News 0 03-02-2011 01:41 AM
LXer: Basenji Helps You Keep Track Of Your CDs/DVDs And USB Sticks Data LXer Syndicated Linux News 0 10-06-2010 08:00 AM
[SOLVED] FC13 all ram used, but swap not used - suspicious.. unkie888 Fedora 2 09-23-2010 10:42 AM

LinuxQuestions.org > Forums > Linux Forums > Linux - Server

All times are GMT -5. The time now is 12:15 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration