LinuxQuestions.org
Visit Jeremy's Blog.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices


Reply
  Search this Thread
Old 05-22-2019, 03:09 AM   #1
AnjaliRaj
LQ Newbie
 
Registered: May 2019
Posts: 2

Rep: Reputation: Disabled
Unhappy ETL jobs not getting triggered in Linux after using SWAP Memory


Hello All,

I'm new to Linux. I have been facing an issue recently.
I have few PDI jobs scheduled in the server. Initially the jobs were failing due to error "Cant Allocate Memory". It was when we decided to add Swap memory.

The RAM is 16 GB . we added Swap Memory of 32 GB and later changed it to 64GB as well
We have tried setting Swappiness = 60 as well as 30. Even after trying all this the jobs drains out RAM memory, and use few MB from SWAP. But the jobs wont even trigger or its very slow.

Please advice how we can resolve this issue.

Many Thanks,
Anjali
 
Old 05-22-2019, 04:11 AM   #2
berndbausch
LQ Addict
 
Registered: Nov 2013
Location: Tokyo
Distribution: Mostly Ubuntu and Centos
Posts: 6,316

Rep: Reputation: 1994Reputation: 1994Reputation: 1994Reputation: 1994Reputation: 1994Reputation: 1994Reputation: 1994Reputation: 1994Reputation: 1994Reputation: 1994Reputation: 1994
Let me try to understand.

You run PDI jobs (I don't know what's PDI, but it's probably not relevant right now). Originally, you could not run them because you didn't have enough memory. You added swap space, and now you don't experience memory shortage, but jobs are very slow.

If I understand you correctly, my opinion is that the jobs overload the system. You need more than 16GB of RAM.

You can double-check this with a command like vmstat, which reports virtual memory statistics. Run it like this:
Code:
vmstat 3 10
which means that it will display statistics ten times, and every three seconds. Pay particular attention to the swap and io values.

Or
Code:
sar -B 3 10
Here, the pg... metrics are interesting, in particular pgpgin/s and pgpgout/s, meaning pages paged in and out, respectively.

You can also use the free command to see how much RAM is available, but it's easy to interpret its output incorrectly.

sar and vmstat may not be on your system. In this case, use your software management tool to install them.

Last edited by berndbausch; 05-22-2019 at 04:14 AM.
 
Old 05-22-2019, 10:56 AM   #3
TB0ne
LQ Guru
 
Registered: Jul 2003
Location: Birmingham, Alabama
Distribution: SuSE, RedHat, Slack,CentOS
Posts: 24,303

Rep: Reputation: 7147Reputation: 7147Reputation: 7147Reputation: 7147Reputation: 7147Reputation: 7147Reputation: 7147Reputation: 7147Reputation: 7147Reputation: 7147Reputation: 7147
Quote:
Originally Posted by AnjaliRaj View Post
Hello All,
I'm new to Linux. I have been facing an issue recently. I have few PDI jobs scheduled in the server. Initially the jobs were failing due to error "Cant Allocate Memory". It was when we decided to add Swap memory.

The RAM is 16 GB . we added Swap Memory of 32 GB and later changed it to 64GB as well We have tried setting Swappiness = 60 as well as 30. Even after trying all this the jobs drains out RAM memory, and use few MB from SWAP. But the jobs wont even trigger or its very slow. Please advice how we can resolve this issue.
If you're new to Linux, how did you get the task of troubleshooting/resolving this issue??

Best advice would be to stop using swap (it is ALWAYS going to be slow), and add more real memory into the system. However, you don't say ANYTHING about what kind of system this actually is, what else is running on it, version/distro of Linux, etc.

Any Pentaho jobs need to be run via the command line with "kitchen"...are you using it? https://help.pentaho.com/Documentati...and_Line_Tools
How do you have the jobs scheduled? What do these jobs actually DO?
 
Old 05-27-2019, 02:25 AM   #4
AnjaliRaj
LQ Newbie
 
Registered: May 2019
Posts: 2

Original Poster
Rep: Reputation: Disabled
Thanks you so much for the quick response.

I have checked the possible options like SWAP and one thing I understood was that its important for us to increase the RAM size.

So we are planning to go ahead with that plan and analyze the server .

Thanks a lot for your support and suggestions
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
LXer: MapR, Syncsort Partner on Big Data ETL Hadoop Solution LXer Syndicated Linux News 0 06-05-2014 10:00 PM
Free ETL tool using postgreSQL under Linux Benetl2009 Linux - News 0 11-22-2008 05:11 PM
LXer: The case for open source ETL LXer Syndicated Linux News 0 12-30-2005 05:46 AM

LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie

All times are GMT -5. The time now is 06:20 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration