LinuxQuestions.org
Download your favorite Linux distribution at LQ ISO.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices


Reply
  Search this Thread
Old 10-05-2018, 08:40 PM   #1
ktoonsez
LQ Newbie
 
Registered: Oct 2018
Posts: 4

Rep: Reputation: Disabled
Getting total bytes of all files in a directory (NOT ANYTHING IN SUB DIRECTORIES)


I wrote a program in C# for winndows that is backing up my windows server to a remote linux box over the internet. So I am trying to find the most efficient way to get the total bytes of all files in a directory (NOT INCLUDING ANYTHING IN SUB DIRECTORIES). There are lots of options using the "du" command but I cannot find any option that does not interrogate sub directories, reason is my main folder that I am wanting to get just the files has over 80,000 sub directories and 2.8 million files inside of the those sub directories. So all du commands even using the --max-depth end up taking hours to return an answer. I am currently using du and adding all the files from each folder to the command, which creates a lot of traffic in and out (the linux box is at a remote location). So hopefully someone has a good solution using du or maybe even ls. Just remember I am trying to keep network traffic small and tight for speed.

Last edited by ktoonsez; 10-05-2018 at 10:05 PM.
 
Old 10-05-2018, 09:22 PM   #2
AwesomeMachine
LQ Guru
 
Registered: Jan 2005
Location: USA and Italy
Distribution: Debian testing/sid; OpenSuSE; Fedora; Mint
Posts: 5,513

Rep: Reputation: 1004Reputation: 1004Reputation: 1004Reputation: 1004Reputation: 1004Reputation: 1004Reputation: 1004Reputation: 1004
Hi ktoonsez,

Welcome!

You can try rsync --dry-run, which will output the total size of everything it would have transferred. To save bandwidth you can use rsync -c, to compress everything before transferring it. There's many tricks using rsync. Debian has the ftpsync scripts that are used to mirror the debian archive. I mirror the i386 and amd64 architectures, which together are about 500,000 objects, and the machine doesn't even break a sweat.

You can just take the ftpsync package and customize it to your needs.
 
Old 10-05-2018, 10:04 PM   #3
ktoonsez
LQ Newbie
 
Registered: Oct 2018
Posts: 4

Original Poster
Rep: Reputation: Disabled
Quote:
Originally Posted by AwesomeMachine View Post
Hi ktoonsez,

Welcome!

You can try rsync --dry-run, which will output the total size of everything it would have transferred. To save bandwidth you can use rsync -c, to compress everything before transferring it. There's many tricks using rsync. Debian has the ftpsync scripts that are used to mirror the debian archive. I mirror the i386 and amd64 architectures, which together are about 500,000 objects, and the machine doesn't even break a sweat.

You can just take the ftpsync package and customize it to your needs.
Shoot, I should have mentioned that the data I am backing up is from a windows machine with a .NET program I wrote that is is executing SSH command from a dll so rsync isn't going to work. Thanks
 
Old 10-05-2018, 10:10 PM   #4
syg00
LQ Veteran
 
Registered: Aug 2003
Location: Australia
Distribution: Lots ...
Posts: 18,065

Rep: Reputation: 2897Reputation: 2897Reputation: 2897Reputation: 2897Reputation: 2897Reputation: 2897Reputation: 2897Reputation: 2897Reputation: 2897Reputation: 2897Reputation: 2897
You could use "find" with maxdepth and print the size and simply sum it yourself, but simpler to just use "ls -l" and then sum it ignoring the dir entries. Trivial awk one-liner will do it.
 
1 members found this post helpful.
Old 10-05-2018, 10:15 PM   #5
berndbausch
Senior Member
 
Registered: Nov 2013
Location: Tokyo
Distribution: Redhat/Centos, Ubuntu, Raspbian, Fedora, Alpine, Cirros, OpenSuse/SLES
Posts: 3,279

Rep: Reputation: 853Reputation: 853Reputation: 853Reputation: 853Reputation: 853Reputation: 853Reputation: 853
i see this du option:
Code:
-S, --separate-dirs
    do not include size of subdirectories
Source: https://linux.die.net/man/1/du
 
Old 10-05-2018, 10:26 PM   #6
syg00
LQ Veteran
 
Registered: Aug 2003
Location: Australia
Distribution: Lots ...
Posts: 18,065

Rep: Reputation: 2897Reputation: 2897Reputation: 2897Reputation: 2897Reputation: 2897Reputation: 2897Reputation: 2897Reputation: 2897Reputation: 2897Reputation: 2897Reputation: 2897
strace confirms what the OP reported; "du -S" descends into subdirs and stats every file. "ls -l" doesn't.
 
Old 10-05-2018, 10:37 PM   #7
ktoonsez
LQ Newbie
 
Registered: Oct 2018
Posts: 4

Original Poster
Rep: Reputation: Disabled
Quote:
Originally Posted by berndbausch View Post
i see this du option:
Code:
-S, --separate-dirs
    do not include size of subdirectories
Source: https://linux.die.net/man/1/du
Ya, you would think that works but it does not, it still dives into the sub directories
 
Old 10-05-2018, 11:38 PM   #8
ktoonsez
LQ Newbie
 
Registered: Oct 2018
Posts: 4

Original Poster
Rep: Reputation: Disabled
Quote:
Originally Posted by syg00 View Post
You could use "find" with maxdepth and print the size and simply sum it yourself, but simpler to just use "ls -l" and then sum it ignoring the dir entries. Trivial awk one-liner will do it.
Actually this might work pretty good. Using the command below that will give me only the sizes which I think will work great and keep the data down to a minimum:

ls -1pea | grep -v / | awk '{ print $3 }'


Thanks :-)
 
Old 10-07-2018, 01:14 AM   #9
AwesomeMachine
LQ Guru
 
Registered: Jan 2005
Location: USA and Italy
Distribution: Debian testing/sid; OpenSuSE; Fedora; Mint
Posts: 5,513

Rep: Reputation: 1004Reputation: 1004Reputation: 1004Reputation: 1004Reputation: 1004Reputation: 1004Reputation: 1004Reputation: 1004
You can use rsync on windows: http://www.aboutmyip.com/AboutMyXApp/DeltaCopy.jsp
 
Old 10-07-2018, 01:25 AM   #10
putney53
LQ Newbie
 
Registered: Oct 2018
Posts: 1

Rep: Reputation: Disabled
i am runing ubuntu for the first time. its my first time installing lenux ever. how do i install sound drivers for gigabyte ax370 gaming 5. this is all new to me so i dont know any code. can anybody help a beginer-
 
Old 10-07-2018, 11:41 PM   #11
berndbausch
Senior Member
 
Registered: Nov 2013
Location: Tokyo
Distribution: Redhat/Centos, Ubuntu, Raspbian, Fedora, Alpine, Cirros, OpenSuse/SLES
Posts: 3,279

Rep: Reputation: 853Reputation: 853Reputation: 853Reputation: 853Reputation: 853Reputation: 853Reputation: 853
Quote:
Originally Posted by putney53 View Post
i am runing ubuntu for the first time. its my first time installing lenux ever. how do i install sound drivers for gigabyte ax370 gaming 5. this is all new to me so i dont know any code. can anybody help a beginer-
To ask a new question, start a new thread.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
total number of files in mount point from all directories rookie11 Linux - Newbie 2 06-14-2013 04:51 PM
Copy a directory into another directory while ignoring specific directories or files wh33t Linux - Newbie 14 05-16-2012 08:13 PM
Total particular files size in a directory kirukan Linux - Newbie 3 01-03-2010 02:48 PM
subtract total numbers of files from 2 directories Michael235 Linux - Newbie 2 03-28-2009 05:49 PM
Count the number of files in a directory and sub-directories within that directory soumyajit.haldar Linux - Software 4 03-20-2007 06:22 AM

LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie

All times are GMT -5. The time now is 06:29 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration