LinuxQuestions.org
Latest LQ Deal: Latest LQ Deals
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Software
User Name
Password
Linux - Software This forum is for Software issues.
Having a problem installing a new program? Want to know which application is best for the job? Post your question in this forum.

Notices


Reply
  Search this Thread
Old 12-10-2018, 10:37 AM   #1
MrBamse
LQ Newbie
 
Registered: Dec 2018
Posts: 15

Rep: Reputation: Disabled
rTorrent max_open_files


Hi, I created another post the other day but it seems like it never got past admin. So this time I'll keep it short.

Official documentation says "The library dynamically closes the least used files as needed." but my experience is that every file downloaded need to be "open" whether or not it is active (dl/ul).

Can anyone comment on this? Currently, my 25000+ files, need a max_open_files value of 25000+

Thank you
 
Old 12-11-2018, 03:07 AM   #2
ondoho
LQ Addict
 
Registered: Dec 2013
Posts: 13,400
Blog Entries: 9

Rep: Reputation: 3655Reputation: 3655Reputation: 3655Reputation: 3655Reputation: 3655Reputation: 3655Reputation: 3655Reputation: 3655Reputation: 3655Reputation: 3655Reputation: 3655
question unclear (maybe the longer version would have been better).
apparently this is about torrenting.
what do you mean "25000 files" - files or torrents?
25000 torrents would bring the most advanced torrent software to its knees i should think.
otoh, you do not need to bother about the individual files each torrent consists of. the software does that.

beyond that, all torrent software has configuration options, and documentation for that.
 
Old 12-11-2018, 04:02 AM   #3
MrBamse
LQ Newbie
 
Registered: Dec 2018
Posts: 15

Original Poster
Rep: Reputation: Disabled
Hi ondo, maybe the mods/admins will approve my previous post, i redacted info by replacing it with the letter X three times, might have been that that set of some alarms
Anyway, i mean maybe 100-150 torrents and they themself consist of 25000-35000 files. And as the documentation says it should handle this by deciding which files are active (dl/ul) and not keeping them "open" but in my case that does not seem to work. Because if i dl a torrent with, lets say, 75 files it will add 75 to the "open files counter" and they are never removed, even if i pull the network cable :/

Im running Debian 9.6 and rTorrent 0.9.6/0.13.6 if that helps. Would my config help?`(think its safer to put it in a separate post this time
I reinstalled the entire system a few months back but with same results.
 
Old 12-11-2018, 04:06 AM   #4
MrBamse
LQ Newbie
 
Registered: Dec 2018
Posts: 15

Original Poster
Rep: Reputation: Disabled
This is my current config

Quote:
directory = /[redacted]/temp/

session = /[redacted]/.rtorrent/

check_hash = no

schedule = untied_directory,5,10,remove_untied=
schedule = low_diskspace,5,20,close_low_diskspace=500M
schedule = filter_active,30,30,"view_filter = active,"or={d.get_up_rate=,d.get_down_rate=}""

schedule = watch_directory_1,5,10,"load_start=/[redacted]/*.torrent,d.set_custom1=/[redacted]/"
schedule = watch_directory_2,5,10,"load_start=/[redacted]/*.torrent,d.set_custom1=/[redacted]/"
schedule = watch_directory_3,5,10,"load_start=/[redacted]/*.torrent,d.set_custom1=/[redacted]/"

schedule = watch_directory_4,5,10,"load_start=/[redacted]/*.torrent,d.set_custom1=noaction"

system.method.set_key = event.download.finished,finished_torrent,"execute=/[redacted].sh,$d.get_base_path=,$d.get_custom1="

method.set_key = event.download.erased, remove_file,"execute={rm,-drf,--,$d.get_base_path=}"


# Get public IP address without the need of having dynamic DNS service, also works from behind NAT, through tunnel
method.insert = get_public_ip_address, simple|private, "execute.capture=bash,-c,"eval echo -n \$(dig -4 TXT +short o-o.myaddr.l.google.com @ns1.google.com)""
# The IP address reported to the tracker. (ip) This handles dynamic IP's as well.
schedule2 = ip_tick, 0, 600, "network.local_address.set=(get_public_ip_address)"

port_range = [redacted]
port_random = yes

use_udp_trackers = yes

encryption = allow_incoming,try_outgoing,enable_retry

dht = off
dht_port = 6881

peer_exchange = no

network.max_open_files.set = 45000
 
Old 12-12-2018, 02:16 AM   #5
ondoho
LQ Addict
 
Registered: Dec 2013
Posts: 13,400
Blog Entries: 9

Rep: Reputation: 3655Reputation: 3655Reputation: 3655Reputation: 3655Reputation: 3655Reputation: 3655Reputation: 3655Reputation: 3655Reputation: 3655Reputation: 3655Reputation: 3655
Quote:
Originally Posted by MrBamse View Post
Hi ondo, maybe the mods/admins will approve my previous post, i redacted info by replacing it with the letter X three times, might have been that that set of some alarms
no, it didn't.
it is normal that your first post awaits approval.
it now shows, if you care to look at your own history.

Quote:
it should handle this by deciding which files are active (dl/ul) and not keeping them "open" but in my case that does not seem to work. Because if i dl a torrent with, lets say, 75 files it will add 75 to the "open files counter" and they are never removed, even if i pull the network cable :/
i see.
anyhow, the solution lies within rtorrent configuration, and the solution to understanding it lies in reading the documentation.
looking at this search, the first two results are what you need, i think.
 
Old 12-12-2018, 03:29 AM   #6
MrBamse
LQ Newbie
 
Registered: Dec 2018
Posts: 15

Original Poster
Rep: Reputation: Disabled
Thanks, but over the years I've read the github-pages "a thousand" times and even though there are terms I don't understand (epoll) I like yo believe I understand most of the text. Any idea what could be the cause of my issue?
 
Old 12-12-2018, 03:47 AM   #7
KIOPRET
LQ Newbie
 
Registered: Nov 2018
Posts: 20

Rep: Reputation: 0
I agree that not every software can process 25000+ файлов, try processing files in small portions to find out what's the maximum
 
Old 12-12-2018, 06:09 AM   #8
MrBamse
LQ Newbie
 
Registered: Dec 2018
Posts: 15

Original Poster
Rep: Reputation: Disabled
Smile

Quote:
Originally Posted by KIOPRET View Post
I agree that not every software can process 25000+ файлов, try processing files in small portions to find out what's the maximum
Thanks for the input KIOPRET but the issue isn't how many files it can handle (so far i've had 50000+ simultaneously without any problem.
It's that rTorrent doesn't seem to be "closing the connection" to files, aka marking them as inactive, like documentation (https://github.com/rakshasa/rtorrent/wiki/User-Guide) says: "The library dynamically closes the least used files as needed."

So I was wondering if anybody either has experience with rTorrent "closing" inactive files and/or sees a problem with my configuration regarding this issue?
 
Old 12-14-2018, 01:30 AM   #9
ondoho
LQ Addict
 
Registered: Dec 2013
Posts: 13,400
Blog Entries: 9

Rep: Reputation: 3655Reputation: 3655Reputation: 3655Reputation: 3655Reputation: 3655Reputation: 3655Reputation: 3655Reputation: 3655Reputation: 3655Reputation: 3655Reputation: 3655
interestingly, i once had a similar problem with trasnsmission:
Paused torrents still continued to congest the network connection.
Found out that there's a hidden setting
Code:
"scrape-paused-torrents-enabled": false,
that fixes that. see: https://trac.transmissionbt.com/ticket/3219

maybe this helps you shed some light on the situation, though i'm not even sure we're talking about the same thing here.

(and yes, i know you're using different torrent software)
 
Old 12-14-2018, 02:13 AM   #10
ehartman
Senior Member
 
Registered: Jul 2007
Location: Delft, The Netherlands
Distribution: Slackware
Posts: 1,170

Rep: Reputation: 657Reputation: 657Reputation: 657Reputation: 657Reputation: 657Reputation: 657
Quote:
Originally Posted by MrBamse View Post
"The library dynamically closes the least used files as needed."
The problem seems to be what rTorrent defines to be "as needed". If that means: when it runs out of file descripters you essentially do not have "enough" files open yet for that behaviour to be activated.
So you would have to seach in the source to find when this closing of least used files will be activated.

Last edited by ehartman; 12-24-2018 at 04:05 AM.
 
Old 12-23-2018, 02:57 AM   #11
MrBamse
LQ Newbie
 
Registered: Dec 2018
Posts: 15

Original Poster
Rep: Reputation: Disabled
Thanks for helping everyone.
We didn't figure this one out (yet?) and it will continue to baffle me how rTorrent handles its "open files".
Take care
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
Rtorrent keyboard commands unresponsive when starting rtorrent in detached screen MheAd Linux - Software 1 09-05-2010 08:00 AM
cant compile rtorrent ruskie Linux - Software 3 07-08-2010 02:57 PM
About rtorrent satimis Linux - Software 8 02-19-2008 09:36 AM
rtorrent "max_open_files" explanation truthe Linux - Software 0 10-17-2006 03:51 AM
rtorrent/libtorrent error truthe Linux - Software 0 06-26-2005 03:41 AM

LinuxQuestions.org > Forums > Linux Forums > Linux - Software

All times are GMT -5. The time now is 08:48 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration