LinuxQuestions.org
Download your favorite Linux distribution at LQ ISO.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Distributions > Slackware
User Name
Password
Slackware This Forum is for the discussion of Slackware Linux.

Notices


Reply
  Search this Thread
Old 03-26-2021, 10:16 AM   #16
bassmadrigal
LQ Guru
 
Registered: Nov 2003
Location: West Jordan, UT, USA
Distribution: Slackware
Posts: 8,008

Original Poster
Rep: Reputation: 5556Reputation: 5556Reputation: 5556Reputation: 5556Reputation: 5556Reputation: 5556Reputation: 5556Reputation: 5556Reputation: 5556Reputation: 5556Reputation: 5556

Quote:
Originally Posted by phenixia2003 View Post
Hello,

I can be wrong but I guess that filebot uses java nio API caches which can lead to native (off-heap) memory leak.

To workaround this, you can bound the size of buffers cached in the per-thread buffer caches with property jdk.nio.maxCachedBufferSize. So, try to add -Djdk.nio.maxCachedBufferSize=262144 to the java command line.

Hope this helps.

--
SeB
Unfortunately, it didn't seem to do anything. I closed filebot and added that option to the launch options and reopened it. After moving back those 100 files and using filebot to move them again, memory usage climbed again to 16.7G VIRT and 10.2G RES.

I also noticed that jconsole (which I left up overnight while attached to the process) has climbed to 21.8G VIRT (only 360M RES). So, it seems that jconsole invalidates that the VIRT memory usage is tied to file operations as it wasn't doing anything (but maybe the RES memory usage is tied to it).

Any other suggestions?
 
Old 03-26-2021, 10:52 AM   #17
phenixia2003
Senior Member
 
Registered: May 2006
Location: France
Distribution: Slackware
Posts: 1,011

Rep: Reputation: 978Reputation: 978Reputation: 978Reputation: 978Reputation: 978Reputation: 978Reputation: 978Reputation: 978
Hello,

Quote:
Originally Posted by bassmadrigal View Post
Unfortunately, it didn't seem to do anything. I closed filebot and added that option to the launch options and reopened it. After moving back those 100 files and using filebot to move them again, memory usage climbed again to 16.7G VIRT and 10.2G RES.

I also noticed that jconsole (which I left up overnight while attached to the process) has climbed to 21.8G VIRT (only 360M RES). So, it seems that jconsole invalidates that the VIRT memory usage is tied to file operations as it wasn't doing anything (but maybe the RES memory usage is tied to it).

Any other suggestions?
Run a second (and maybe a third) instance of filebot to see if the memory used by the 1st one is freed (or not) by the system when under pressure.

--
SeB
 
Old 03-26-2021, 11:35 AM   #18
bassmadrigal
LQ Guru
 
Registered: Nov 2003
Location: West Jordan, UT, USA
Distribution: Slackware
Posts: 8,008

Original Poster
Rep: Reputation: 5556Reputation: 5556Reputation: 5556Reputation: 5556Reputation: 5556Reputation: 5556Reputation: 5556Reputation: 5556Reputation: 5556Reputation: 5556Reputation: 5556
Quote:
Originally Posted by phenixia2003 View Post
Run a second (and maybe a third) instance of filebot to see if the memory used by the 1st one is freed (or not) by the system when under pressure.
Didn't make a difference. I tried 3 total filebot instances... each moving 100 files. Started first, moved 100 files, started the next, moved 100 files, and the same for the 3rd. No process lowered memory usage as the newer java process increased usage (and interestingly, each newer process used more than the previous... but I'm not sure if that's just a coincidence).

1st Filebot - 16.7G VIRT | 10.2G RES
2nd Filebot - 17.8G VIRT | 11.4G RES
3rd Filebot - 22.5G VIRT | 16.1G RES

This leaves me with about 4.8GB of available RAM left with my swap maxed out (I have 16GB more swap that can be enabled if needed for testing).

Code:
jbhansen@craven-moorhead:~/slackbuilds$ free -h                                                                                                                                                                                                                                
              total        used        free      shared  buff/cache   available
Mem:           62Gi        56Gi       543Mi       917Mi       5.9Gi       4.8Gi
Swap:         8.0Gi       8.0Gi       0.0Ki
 
Old 03-26-2021, 11:52 AM   #19
phenixia2003
Senior Member
 
Registered: May 2006
Location: France
Distribution: Slackware
Posts: 1,011

Rep: Reputation: 978Reputation: 978Reputation: 978Reputation: 978Reputation: 978Reputation: 978Reputation: 978Reputation: 978
Hello,

Quote:
Originally Posted by bassmadrigal View Post
Didn't make a difference. I tried 3 total filebot instances... each moving 100 files. Started first, moved 100 files, started the next, moved 100 files, and the same for the 3rd. No process lowered memory usage as the newer java process increased usage (and interestingly, each newer process used more than the previous... but I'm not sure if that's just a coincidence).
I found this thread on filebot forum. Maybe the options used in this post, and especially -XX:MaxMetaspaceSize, -XX:MaxRAM and -XX:MaxRAMPercentage could help.

--
SeB
 
Old 03-26-2021, 01:39 PM   #20
Martinus2u
Member
 
Registered: Apr 2010
Distribution: Slackware
Posts: 476

Rep: Reputation: 115Reputation: 115
Quote:
Originally Posted by BrunoLafleur View Post
Very good article. According to that, java.nio.DirectByteBuffer etc. is not the culprit here, because it would be reported by Native Memory Tracking under "Internal".

Quote:
Originally Posted by Jeebizz View Post
The question is who's fault is this? The actual java program or the java VM?
Seemingly neither.

Quote:
Originally Posted by Jeebizz View Post
I remember how java is touted for its "memory clean up" or whatever. Sorry that I don't have anything actually useful to contribute, it is the morning - and when it comes to java, I like to shit on it every chance I get; because I just never liked java to begin with.

"B-b-b-ut muh garbage collecsthun."
You're entitled to an opinion, and I would even mildly agree that Java runtimes are a bit heavy at times. However, from a software development point of view, Java at least gives us a chance to write well-structured and maintainable code (even if some developers blow this chance). Unlike many of the modern script languages that are designed to produce unmaintainable trash code.
 
1 members found this post helpful.
Old 03-26-2021, 02:56 PM   #21
bassmadrigal
LQ Guru
 
Registered: Nov 2003
Location: West Jordan, UT, USA
Distribution: Slackware
Posts: 8,008

Original Poster
Rep: Reputation: 5556Reputation: 5556Reputation: 5556Reputation: 5556Reputation: 5556Reputation: 5556Reputation: 5556Reputation: 5556Reputation: 5556Reputation: 5556Reputation: 5556
Quote:
Originally Posted by phenixia2003 View Post
I found this thread on filebot forum. Maybe the options used in this post, and especially -XX:MaxMetaspaceSize, -XX:MaxRAM and -XX:MaxRAMPercentage could help.
No go... I tried that, switching the MaxRAMPercentage to 3.0 (which would be just under 2GB of RAM and kept the MaxRAM to 1GB) and after the last run, VIRT was 11.5G and RES was 7.8G.
 
1 members found this post helpful.
Old 03-27-2021, 09:52 AM   #22
phenixia2003
Senior Member
 
Registered: May 2006
Location: France
Distribution: Slackware
Posts: 1,011

Rep: Reputation: 978Reputation: 978Reputation: 978Reputation: 978Reputation: 978Reputation: 978Reputation: 978Reputation: 978
Hello,

Quote:
Originally Posted by bassmadrigal View Post
No go... I tried that, switching the MaxRAMPercentage to 3.0 (which would be just under 2GB of RAM and kept the MaxRAM to 1GB) and after the last run, VIRT was 11.5G and RES was 7.8G.
Maybe this can help (from the document pointed out in this post) :

Quote:
The glibc memory allocator, at least in some versions of Linux, has an optimization to improve speed by avoiding contention when a process has a large number of concurrent threads. The supposed speedup is achieved by maintaining per-core memory pools. Essentially, with this optimization, the OS grabs memory for a given process in pretty big same-size (64MB) chunks called arenas, which are clearly visible when process memory is analyzed with pmap. Each arena is available only to its respective CPU core, so no more than one thread at a time can operate on it. Then, individual malloc() calls reserve memory within these arenas. Up to a certain maximum number of arenas (8 by default) can be allocated per each CPU core. Looks like this is maxed out when the number of threads is high and/or threads are created and destroyed frequently. The actual amount of memory utilized by the application within these arenas can be quite small. However, if an application has a large number of threads, and the machine has a large number of CPU cores, the total amount of memory reserved in this way can grow really high. For example, on a machine with 16 cores, this number would be 16 * 8 * 64MB = 8GB.

Fortunately, the maximum number of arenas can be adjusted via the MALLOC_ARENA_MAX environment variable
--
SeB

Last edited by phenixia2003; 03-27-2021 at 10:58 AM.
 
Old 03-28-2021, 06:43 PM   #23
bassmadrigal
LQ Guru
 
Registered: Nov 2003
Location: West Jordan, UT, USA
Distribution: Slackware
Posts: 8,008

Original Poster
Rep: Reputation: 5556Reputation: 5556Reputation: 5556Reputation: 5556Reputation: 5556Reputation: 5556Reputation: 5556Reputation: 5556Reputation: 5556Reputation: 5556Reputation: 5556
Quote:
Originally Posted by phenixia2003 View Post
Maybe this can help (from the document pointed out in this post)
I was hoping, but exporting it as 1 and then starting filebot still led to similar memory usage in moving 100 files.
 
Old 04-04-2021, 06:24 PM   #24
notzed
LQ Newbie
 
Registered: Dec 2020
Location: South Australia
Distribution: slackware64-current
Posts: 15

Rep: Reputation: Disabled
I would say try a newer openjdk as 8 is ancient and a lot of things wrt to native memory have been improved significantly. But I couldn't get it to work to verify - it just crashes whenever I try to do anything interesting. I tried openjdk 8, 15, and 16 and they all segfaulted I think it's related to libmediainfo.

Maybe you can try ulimit to force a maximum? (-v 20000000 seems about the minimum for virtual memory) Or (yuck) a virtual machine with a fixed memory size and no swap.

Looking at the last available public source[1] it looks like it's using java.nio.Files to do file copies/moves, these [2] appear to be mostly native calls and wouldn't use nio ByteBuffer as mentioned above.

Also free memory is wasted memory so if it's just the number you're worried about and not system performance it's not very important. On the other hand this just seems like a bug somewhere but there's not much you can do with unsupported software.

[1] https://sourceforge.net/p/filebot/code/3968/tree/trunk/
[2] https://github.com/frohoff/jdk8u-jdk...xCopyFile.java
 
Old 04-04-2021, 09:09 PM   #25
Richard Cranium
Senior Member
 
Registered: Apr 2009
Location: Carrollton, Texas
Distribution: Slackware64 14.2
Posts: 3,746

Rep: Reputation: 2083Reputation: 2083Reputation: 2083Reputation: 2083Reputation: 2083Reputation: 2083Reputation: 2083Reputation: 2083Reputation: 2083Reputation: 2083Reputation: 2083
I was going to look at this, but the idiot behind this application was still using ant to build stuff in 2019 (date of the his last free version). (Not to mention that he's now decided that he needs to be paid for his work; good luck with that.)

His paid version has 4 shared libraries packaged with it; you could certainly have memory leaks from them.

The last free version that I found is at https://github.com/deleted-repo/filebot

Since he provided no documentation on how to build this, his You may build FileBot for private use on unsupported platforms. is a bit of horse dung. However, he could have deleted the entire repository when he decided to go private; he didn't, so there's that positive bit (and it really is a positive bit). So, if you really like the early version of the app and you enjoy spelunking how to operate a rather obsolete build system, have a go at it.

I have better things to do with my time to figure out how some random ant project can be built. (I've written one build system myself and have used ant in the past. I've been doing Java development for 20 years now and would not think of using either the build system that I had written or ant for anything since 2013 or so) I believe that the OP should either:
  1. Pay for the non-free version, where you can complain to the person who wrote that code.
  2. Find some other application which does the same thing.
 
1 members found this post helpful.
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
Newbie wants to install ubuntu on windows xp sp3 with 256 mb ram 40gb hd JaniceM Ubuntu 14 11-13-2008 08:31 AM
Sunami External USB 2.0 HDD caddy, Hitachi 40gb drive nickpye Linux - Hardware 1 02-16-2004 05:36 PM
RAID 1 on 2*40GB hdd, what is the best solution? Ivan Slackware 0 01-15-2004 06:38 AM
Help mounting 40GB Iomega USB external HD. Whitehat Linux - General 4 07-31-2003 09:07 PM
Installing GHL on 40Gb HDD atc24amit Linux - General 1 02-26-2003 02:44 AM

LinuxQuestions.org > Forums > Linux Forums > Linux - Distributions > Slackware

All times are GMT -5. The time now is 01:04 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration