LinuxQuestions.org
Share your knowledge at the LQ Wiki.
Home Forums Tutorials Articles Register
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Server
User Name
Password
Linux - Server This forum is for the discussion of Linux Software used in a server related context.

Notices


Reply
  Search this Thread
Old 03-26-2013, 10:47 AM   #1
anon091
Senior Member
 
Registered: Jun 2009
Posts: 1,795

Rep: Reputation: 49
For ext4, what's "better", one large file or multiple smaller files


hello everyone. Someone on here showed me this nifty command to create a 100MB "filler" file.

dd if=/dev/zero of=filename bs=1M count=100

But then it got me thinking. Say I wanted to occupy 10GB of space on a volume. From an ext4 filesystem perspective, is one way better than the other: one 10GB file vs. 100 100MB files, to occupy the same space?

Not sure what's better/easier for the filesystem to keep track of or if it really matters, or if one way would cause an fsck of a volume to take longer over the other way.
 
Old 03-26-2013, 07:37 PM   #2
MS3FGX
LQ Guru
 
Registered: Jan 2004
Location: NJ, USA
Distribution: Slackware, Debian
Posts: 5,852

Rep: Reputation: 361Reputation: 361Reputation: 361Reputation: 361
I'm not sure which one is "better" as I don't understand the goal of the exercise. If you are just looking to fill up a drive arbitrarily, then I can't imagine it would matter.
 
Old 03-26-2013, 07:53 PM   #3
anon091
Senior Member
 
Registered: Jun 2009
Posts: 1,795

Original Poster
Rep: Reputation: 49
Well, not really arbitrarily. I have a server that I can't do quotas on yet, so I'm implemented the old trick of putting filler files on the volume the share is on to limit how much free space the users have available to them. those users will use the whole thing even if you tell them not to ;-)

But I guess the real question is I'm not sure if there's any more "strain" on a file system in those couple ways between one big file, or a lot of smaller files, that both take up the same amount of disk space. Hope that makes sense.
 
Old 03-28-2013, 02:23 AM   #4
chrism01
LQ Guru
 
Registered: Aug 2004
Location: Sydney
Distribution: Rocky 9.2
Posts: 18,359

Rep: Reputation: 2751Reputation: 2751Reputation: 2751Reputation: 2751Reputation: 2751Reputation: 2751Reputation: 2751Reputation: 2751Reputation: 2751Reputation: 2751Reputation: 2751
Well, if you fill up the extra space with dummy files, what's the point in doing so? The disk space is still gone...
 
Old 03-28-2013, 04:37 AM   #5
David the H.
Bash Guru
 
Registered: Jun 2004
Location: Osaka, Japan
Distribution: Arch + Xfce
Posts: 6,852

Rep: Reputation: 2037Reputation: 2037Reputation: 2037Reputation: 2037Reputation: 2037Reputation: 2037Reputation: 2037Reputation: 2037Reputation: 2037Reputation: 2037Reputation: 2037
I'm no expert here, but AIUI, the size of the file really only matters during read/write operations. When it's just sitting on the disk it has little effect at all. Disk space is disk space.

I know some filesystems are designed for high performance in certain situations, such as reiserfs, which is optimized for handling many small files, but my impression is that the ext* series is designed to be a general, roundly balanced FS, with no real emphasis placed on performance in any specific situation.

The only other things I can think of are that many small files occupy more inodes, but that isn't much of a problem with modern filesystems in most cases. And I also suppose that many very small files could lead to excess space consumption over and above their actual number, if they're smaller than the block size.

Finally, I'd think that one big file is just easier to handle than many small ones. Although on the other hand multiple smaller files could make it easier to manage the desired space manually, by simply copying or deleting as many as you need to get to the approximate size you want.
 
1 members found this post helpful.
Old 03-28-2013, 04:50 AM   #6
jschiwal
LQ Guru
 
Registered: Aug 2001
Location: Fargo, ND
Distribution: SuSE AMD64
Posts: 15,733

Rep: Reputation: 682Reputation: 682Reputation: 682Reputation: 682Reputation: 682Reputation: 682
If you want to limit each user to 100M, the way to do it is to use quotas. However, you could create an empty file for each user, format the files with a filesystem, and mount them. Then the user will be limited by the size of their own filesystem.
 
2 members found this post helpful.
Old 03-28-2013, 08:03 AM   #7
anon091
Senior Member
 
Registered: Jun 2009
Posts: 1,795

Original Poster
Rep: Reputation: 49
Chris, my point in doing so is to limit the amount of space the users have available on the server, and having the dummy files not accessible to them through a share lets them not delete them. Basically so if I have a 2TB volume, but I only want a samba share to appear to have 1TB total space, I'd have to fill up 1TB with other stuff (unless there's a way to configure Samba to do this).

David, that was my general take on the situation as well, but I wasn't sure if I was missing something, but seems like I wasn't.

jschiwal, what I'm really trying to accomplish with my filler files is something similar to the example I gave in the first paragraph of this post. I have a volume, with just one samba share on it, but I don't want the users to have all the volume's space available to them to use, at least at this time. Reasoning behind that is my backup to disk server doesn't have enough space to back up the user data in the samba share when they would eventually fill up that volume, so until I can add storage to my backup server, I need to curtail their available space to an acceptable level in the meantime.
 
Old 04-02-2013, 01:37 AM   #8
chrism01
LQ Guru
 
Registered: Aug 2004
Location: Sydney
Distribution: Rocky 9.2
Posts: 18,359

Rep: Reputation: 2751Reputation: 2751Reputation: 2751Reputation: 2751Reputation: 2751Reputation: 2751Reputation: 2751Reputation: 2751Reputation: 2751Reputation: 2751Reputation: 2751
I'd say you have (at least) 3 options:

1. quotas (ideal)
2. one large file to fill 'empty' space
3. change (reduce) the partition size
4. bind-mounts ??
 
1 members found this post helpful.
Old 04-02-2013, 07:44 AM   #9
anon091
Senior Member
 
Registered: Jun 2009
Posts: 1,795

Original Poster
Rep: Reputation: 49
Yeah, I really need to read up more on the quota stuff, as that sounds like the best option. Option 2 I can handle, just wasn't sure if one huge file or several smaller but still pretty big files were better on ext4. I'm not sure how to do options 3 or 4, so probably best I focus on 1 and 2.
 
Old 04-02-2013, 08:20 PM   #10
chrism01
LQ Guru
 
Registered: Aug 2004
Location: Sydney
Distribution: Rocky 9.2
Posts: 18,359

Rep: Reputation: 2751Reputation: 2751Reputation: 2751Reputation: 2751Reputation: 2751Reputation: 2751Reputation: 2751Reputation: 2751Reputation: 2751Reputation: 2751Reputation: 2751
Ok; for quotas, Chap 7 here http://www.linuxtopia.org/online_boo...ion/index.html goes through it pretty clearly.
You should have no problem
 
  


Reply



Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
k3b says "your mkisofs doesn't support large files"etc grissiom Slackware 8 11-27-2014 09:02 PM
[SOLVED] Ho to delete "orphaned" files after switching /tmp from ext4 to tmpfs? JZL240I-U Linux - Software 2 12-14-2012 06:29 AM
[SOLVED] "Lost space" for Ext3 and Ext4 file systems on hard drive Steve W Linux - Hardware 2 10-16-2012 01:20 PM
Split large file into smaller files mikes88 Programming 29 03-22-2012 10:14 AM
"Best file systems" for only large (10-30GB) files ? petepdx Linux - Server 3 03-24-2008 01:17 PM

LinuxQuestions.org > Forums > Linux Forums > Linux - Server

All times are GMT -5. The time now is 11:53 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration