rTorrent max_open_files

MrBamse · 12-10-2018, 09:37 AM

Hi, I created another post the other day but it seems like it never got past admin. So this time I'll keep it short.

Official documentation says "The library dynamically closes the least used files as needed." but my experience is that every file downloaded need to be "open" whether or not it is active (dl/ul).

Can anyone comment on this? Currently, my 25000+ files, need a max_open_files value of 25000+

Thank you

ondoho · 12-11-2018, 02:07 AM

question unclear (maybe the longer version would have been better).
apparently this is about torrenting.
what do you mean "25000 files" - files or torrents?
25000 torrents would bring the most advanced torrent software to its knees i should think.
otoh, you do not need to bother about the individual files each torrent consists of. the software does that.

beyond that, all torrent software has configuration options, and documentation for that.

MrBamse · 12-11-2018, 03:02 AM

Hi ondo, maybe the mods/admins will approve my previous post, i redacted info by replacing it with the letter X three times, might have been that that set of some alarms

Anyway, i mean maybe 100-150 torrents and they themself consist of 25000-35000 files. And as the documentation says it should handle this by deciding which files are active (dl/ul) and not keeping them "open" but in my case that does not seem to work. Because if i dl a torrent with, lets say, 75 files it will add 75 to the "open files counter" and they are never removed, even if i pull the network cable :/

Im running Debian 9.6 and rTorrent 0.9.6/0.13.6 if that helps. Would my config help?`(think its safer to put it in a separate post this time

I reinstalled the entire system a few months back but with same results.

MrBamse · 12-11-2018, 03:06 AM

This is my current config

Quote:

directory = /[redacted]/temp/

session = /[redacted]/.rtorrent/

check_hash = no

schedule = untied_directory,5,10,remove_untied=
schedule = low_diskspace,5,20,close_low_diskspace=500M
schedule = filter_active,30,30,"view_filter = active,"or={d.get_up_rate=,d.get_down_rate=}""

schedule = watch_directory_1,5,10,"load_start=/[redacted]/*.torrent,d.set_custom1=/[redacted]/"
schedule = watch_directory_2,5,10,"load_start=/[redacted]/*.torrent,d.set_custom1=/[redacted]/"
schedule = watch_directory_3,5,10,"load_start=/[redacted]/*.torrent,d.set_custom1=/[redacted]/"

schedule = watch_directory_4,5,10,"load_start=/[redacted]/*.torrent,d.set_custom1=noaction"

system.method.set_key = event.download.finished,finished_torrent,"execute=/[redacted].sh,$d.get_base_path=,$d.get_custom1="

method.set_key = event.download.erased, remove_file,"execute={rm,-drf,--,$d.get_base_path=}"

# Get public IP address without the need of having dynamic DNS service, also works from behind NAT, through tunnel
method.insert = get_public_ip_address, simple|private, "execute.capture=bash,-c,"eval echo -n \$(dig -4 TXT +short o-o.myaddr.l.google.com @ns1.google.com)""
# The IP address reported to the tracker. (ip) This handles dynamic IP's as well.
schedule2 = ip_tick, 0, 600, "network.local_address.set=(get_public_ip_address)"

port_range = [redacted]
port_random = yes

use_udp_trackers = yes

encryption = allow_incoming,try_outgoing,enable_retry

dht = off
dht_port = 6881

peer_exchange = no

network.max_open_files.set = 45000

ondoho · 12-12-2018, 01:16 AM

Quote:

Originally Posted by MrBamse

Hi ondo, maybe the mods/admins will approve my previous post, i redacted info by replacing it with the letter X three times, might have been that that set of some alarms

no, it didn't.
it is normal that your first post awaits approval.
it now shows, if you care to look at your own history.

Quote:

it should handle this by deciding which files are active (dl/ul) and not keeping them "open" but in my case that does not seem to work. Because if i dl a torrent with, lets say, 75 files it will add 75 to the "open files counter" and they are never removed, even if i pull the network cable :/

i see.
anyhow, the solution lies within rtorrent configuration, and the solution to understanding it lies in reading the documentation.
looking at this search, the first two results are what you need, i think.

MrBamse · 12-12-2018, 02:29 AM

Thanks, but over the years I've read the github-pages "a thousand" times and even though there are terms I don't understand (epoll) I like yo believe I understand most of the text. Any idea what could be the cause of my issue?

KIOPRET · 12-12-2018, 02:47 AM

I agree that not every software can process 25000+ файлов, try processing files in small portions to find out what's the maximum

MrBamse · 12-12-2018, 05:09 AM

Quote:

Originally Posted by KIOPRET

I agree that not every software can process 25000+ файлов, try processing files in small portions to find out what's the maximum

Thanks for the input KIOPRET but the issue isn't how many files it can handle (so far i've had 50000+ simultaneously without any problem.
It's that rTorrent doesn't seem to be "closing the connection" to files, aka marking them as inactive, like documentation (https://github.com/rakshasa/rtorrent/wiki/User-Guide) says: "The library dynamically closes the least used files as needed."

So I was wondering if anybody either has experience with rTorrent "closing" inactive files and/or sees a problem with my configuration regarding this issue?

ondoho · 12-14-2018, 12:30 AM

interestingly, i once had a similar problem with trasnsmission:
Paused torrents still continued to congest the network connection.
Found out that there's a hidden setting

Code:

"scrape-paused-torrents-enabled": false,

that fixes that. see: https://trac.transmissionbt.com/ticket/3219

maybe this helps you shed some light on the situation, though i'm not even sure we're talking about the same thing here.

(and yes, i know you're using different torrent software)

ehartman · 12-14-2018, 01:13 AM

Quote:

Originally Posted by MrBamse

"The library dynamically closes the least used files as needed."

The problem seems to be what rTorrent defines to be "as needed". If that means: when it runs out of file descripters you essentially do not have "enough" files open yet for that behaviour to be activated.
So you would have to seach in the source to find when this closing of least used files will be activated.

MrBamse · 12-23-2018, 01:57 AM

Thanks for helping everyone.
We didn't figure this one out (yet?) and it will continue to baffle me how rTorrent handles its "open files".

Take care