LinuxQuestions.org
Share your knowledge at the LQ Wiki.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices


Reply
  Search this Thread
Old 03-09-2012, 04:34 PM   #1
daisychick
Member
 
Registered: Nov 2006
Location: Texas
Distribution: ubuntu 12.04 LTS
Posts: 154

Rep: Reputation: 0
ext4 block size


so, I have some guys that are only using files 1TB or higher. I was reading that increasing the block size would help increase throughput. I found this
Code:
mkfs.ext4 -j -b 1024 /dev/sde1
so I know I can do it. The question is, what's a good block size for a drive that's only going to be having 1TB-4TB files stored on it?
 
Old 03-09-2012, 05:13 PM   #2
T3RM1NVT0R
Senior Member
 
Registered: Dec 2010
Location: Internet
Distribution: Linux Mint, SLES, CentOS, Red Hat
Posts: 2,385

Rep: Reputation: 476Reputation: 476Reputation: 476Reputation: 476Reputation: 476
@ Reply

Hi daisychick,

With ext4 filesystem I think max block size is 4096. I tried on my machine to use larger block size say 8192 and it gave me the following warning:

Code:
Warning: blocksize 8192 not usable on most systems.
mke2fs 1.41.12 (17-May-2010)
mkfs.ext4: 8192-byte blocks too big for system (max 4096)
Proceed anyway? (y,n)
So it appears to me that the maximum block size that you can define is 4096 KB.
 
Old 03-09-2012, 05:26 PM   #3
daisychick
Member
 
Registered: Nov 2006
Location: Texas
Distribution: ubuntu 12.04 LTS
Posts: 154

Original Poster
Rep: Reputation: 0
I admit I'm a little confused by all this. what is the inode size? and if I make that bigger, will that help?
 
Old 03-09-2012, 05:33 PM   #4
daisychick
Member
 
Registered: Nov 2006
Location: Texas
Distribution: ubuntu 12.04 LTS
Posts: 154

Original Poster
Rep: Reputation: 0
I'm dealing with big big files. An example of some of the smallest ones...
Code:
[root@server californo]# ls -lah
total 19G
drwxrwxr-x 3 temp temp 4.0K Sep 23 16:26 .
drwxrwxr-x 4 temp temp 4.0K Sep 22 12:26 ..
-rw-rw-r-- 1 temp temp 1.2G Sep 22 12:29 001_3_f.fastq
-rw-rw-r-- 1 temp temp 1.2G Sep 22 12:29 001_3_r.fastq
-rw-rw-r-- 1 temp temp 1.3G Sep 22 12:28 002_2_f.fastq
-rw-rw-r-- 1 temp temp 1.3G Sep 22 12:28 002_2_r.fastq
-rw-rw-r-- 1 temp temp 1.5G Sep 22 12:30 002_3_f.fastq
-rw-rw-r-- 1 temp temp 1.5G Sep 22 12:30 002_3_r.fastq
-rw-rw-r-- 1 temp temp 1.8G Sep 22 13:07 003_3_f.fastq
-rw-rw-r-- 1 temp temp 1.8G Sep 22 13:07 003_3_r.fastq
-rw-rw-r-- 1 temp temp 1.7G Sep 22 12:29 005_3_f.fastq
-rw-rw-r-- 1 temp temp 1.7G Sep 22 12:29 005_3_r.fastq
-rw-rw-r-- 1 temp temp 2.2G Sep 22 12:30 006_3_f.fastq
-rw-rw-r-- 1 temp temp 2.2G Sep 22 12:30 006_3_r.fastq
-rw-rw-r-- 1 temp temp  550 Sep 22 12:26 demultiplexed_sampleSheet.csv
drwxrwxr-x 6 temp temp 4.0K Nov  9 14:17 processed
[root@server californo]#
 
Old 03-09-2012, 05:45 PM   #5
T3RM1NVT0R
Senior Member
 
Registered: Dec 2010
Location: Internet
Distribution: Linux Mint, SLES, CentOS, Red Hat
Posts: 2,385

Rep: Reputation: 476Reputation: 476Reputation: 476Reputation: 476Reputation: 476
@ Reply

To reply to your initial question I would say 4 KB is the maximum block size that you can use. Have a look at the following link: http://en.wikipedia.org/wiki/Ext4

As far as inode is concerned it is basically a term used for index-node. inode keeps a track of files / directories hierarchy on a file system. It also keep information about files / directories attributes like time stamps, location etc. Default inode size for ext4 file system is 256 bytes. You can increase the size of the inode to accomodate additional attribute that it can keep track of. inode also defines the maximum number of files that you can have a on a file system.

For me inode is basically a feature or algorith that keeps track of indexing and metadata of files/directories on a file system.

Edit

If you want to find out the current inode size on a particular partition you can use the following command:

Code:
tune2fs -l /dev/sdxx
Where first x will be the slot on which your hdd is connected and the second x will be partition number.

Last edited by T3RM1NVT0R; 03-09-2012 at 05:48 PM. Reason: Additional information.
 
Old 03-09-2012, 05:58 PM   #6
anomie
Senior Member
 
Registered: Nov 2004
Location: Texas
Distribution: RHEL, Scientific Linux, Debian, Fedora
Posts: 3,935
Blog Entries: 5

Rep: Reputation: Disabled
[ Caveat: I haven't tested this. Just explaining what I would try. ]

Given files of the size you're working with, I would likely just use some variation of:
Code:
# mkfs.ext4 -T largefile /dev/here1
 
Old 03-09-2012, 10:24 PM   #7
pinga123
Member
 
Registered: Sep 2009
Posts: 684
Blog Entries: 2

Rep: Reputation: 36
Ok if i understand correctly you need to determine the best block size according to your environment.

The information you have given is incomplete to determine the best block size .

Generally the block size for the partition is determine by Two Things.
1)Type of data.
2)Application used to store the data.

For example if you are using the partition to store data for database ,you might need to first determine which database it is ?e.g oracle,sybase,sqlserver .
the next step will be what is the best block size for each of them.

If your harddisk is only meant to store media files such as images ,video then a slightly bigger block size is recommended .
 
Old 03-10-2012, 12:12 AM   #8
daisychick
Member
 
Registered: Nov 2006
Location: Texas
Distribution: ubuntu 12.04 LTS
Posts: 154

Original Poster
Rep: Reputation: 0
If my understanding is correct, it's large files of text and / or images. It's genome sequence analysis and DNA mapping. The application is a proprietary application that parses data and analyzes it. The larger block size is definitely what we're looking for.
 
Old 03-11-2012, 11:12 PM   #9
pinga123
Member
 
Registered: Sep 2009
Posts: 684
Blog Entries: 2

Rep: Reputation: 36
Quote:
Originally Posted by daisychick View Post
If my understanding is correct, it's large files of text and / or images. It's genome sequence analysis and DNA mapping. The application is a proprietary application that parses data and analyzes it. The larger block size is definitely what we're looking for.
I would recommend to create two partitions with different block size and derive at conclusion based on the end result.

First make sure What block size your application is using to write the data?
then substitute that block size to below dd command and compare the result of both partition.

Code:
time dd if=/dev/zero of=<partitionpath>/Test.img bs=<As described above> count=<Make sure end file should be > 1GB so substitute value accordingly>
If you are confused over the above command just provide here with the value of application block size for writing.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
store a block device information in sb_info struct of ext4 file system shwety1085 Linux - Newbie 0 09-05-2011 10:09 PM
Extremely Large Metadata size for directory on an ext4 filesystem mdpolaris Linux - Server 2 06-27-2011 05:47 AM
Help resizing Ext4 partition size NightSky Slackware 14 11-13-2010 10:59 PM
Expanding parition ext4 size leblinux Fedora 4 10-25-2010 11:55 PM
Upgrading Redhat to support ext4 for files over 2TB in size rog_ski Linux - General 7 09-28-2010 10:27 AM


All times are GMT -5. The time now is 10:45 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration