For ext4, what's "better", one large file or multiple smaller files
Linux - ServerThis forum is for the discussion of Linux Software used in a server related context.
Notices
Welcome to LinuxQuestions.org, a friendly and active Linux Community.
You are currently viewing LQ as a guest. By joining our community you will have the ability to post topics, receive our newsletter, use the advanced search, subscribe to threads and access many other special features. Registration is quick, simple and absolutely free. Join our community today!
Note that registered members see fewer ads, and ContentLink is completely disabled once you log in.
If you have any problems with the registration process or your account login, please contact us. If you need to reset your password, click here.
Having a problem logging in? Please visit this page to clear all LQ-related cookies.
Get a virtual cloud desktop with the Linux distro that you want in less than five minutes with Shells! With over 10 pre-installed distros to choose from, the worry-free installation life is here! Whether you are a digital nomad or just looking for flexibility, Shells can put your Linux machine on the device that you want to use.
Exclusive for LQ members, get up to 45% off per month. Click here for more info.
For ext4, what's "better", one large file or multiple smaller files
hello everyone. Someone on here showed me this nifty command to create a 100MB "filler" file.
dd if=/dev/zero of=filename bs=1M count=100
But then it got me thinking. Say I wanted to occupy 10GB of space on a volume. From an ext4 filesystem perspective, is one way better than the other: one 10GB file vs. 100 100MB files, to occupy the same space?
Not sure what's better/easier for the filesystem to keep track of or if it really matters, or if one way would cause an fsck of a volume to take longer over the other way.
I'm not sure which one is "better" as I don't understand the goal of the exercise. If you are just looking to fill up a drive arbitrarily, then I can't imagine it would matter.
Well, not really arbitrarily. I have a server that I can't do quotas on yet, so I'm implemented the old trick of putting filler files on the volume the share is on to limit how much free space the users have available to them. those users will use the whole thing even if you tell them not to ;-)
But I guess the real question is I'm not sure if there's any more "strain" on a file system in those couple ways between one big file, or a lot of smaller files, that both take up the same amount of disk space. Hope that makes sense.
I'm no expert here, but AIUI, the size of the file really only matters during read/write operations. When it's just sitting on the disk it has little effect at all. Disk space is disk space.
I know some filesystems are designed for high performance in certain situations, such as reiserfs, which is optimized for handling many small files, but my impression is that the ext* series is designed to be a general, roundly balanced FS, with no real emphasis placed on performance in any specific situation.
The only other things I can think of are that many small files occupy more inodes, but that isn't much of a problem with modern filesystems in most cases. And I also suppose that many very small files could lead to excess space consumption over and above their actual number, if they're smaller than the block size.
Finally, I'd think that one big file is just easier to handle than many small ones. Although on the other hand multiple smaller files could make it easier to manage the desired space manually, by simply copying or deleting as many as you need to get to the approximate size you want.
If you want to limit each user to 100M, the way to do it is to use quotas. However, you could create an empty file for each user, format the files with a filesystem, and mount them. Then the user will be limited by the size of their own filesystem.
Chris, my point in doing so is to limit the amount of space the users have available on the server, and having the dummy files not accessible to them through a share lets them not delete them. Basically so if I have a 2TB volume, but I only want a samba share to appear to have 1TB total space, I'd have to fill up 1TB with other stuff (unless there's a way to configure Samba to do this).
David, that was my general take on the situation as well, but I wasn't sure if I was missing something, but seems like I wasn't.
jschiwal, what I'm really trying to accomplish with my filler files is something similar to the example I gave in the first paragraph of this post. I have a volume, with just one samba share on it, but I don't want the users to have all the volume's space available to them to use, at least at this time. Reasoning behind that is my backup to disk server doesn't have enough space to back up the user data in the samba share when they would eventually fill up that volume, so until I can add storage to my backup server, I need to curtail their available space to an acceptable level in the meantime.
Yeah, I really need to read up more on the quota stuff, as that sounds like the best option. Option 2 I can handle, just wasn't sure if one huge file or several smaller but still pretty big files were better on ext4. I'm not sure how to do options 3 or 4, so probably best I focus on 1 and 2.
LinuxQuestions.org is looking for people interested in writing
Editorials, Articles, Reviews, and more. If you'd like to contribute
content, let us know.