LinuxQuestions.org
Latest LQ Deal: Linux Power User Bundle
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices


Reply
  Search this Thread
Old 01-08-2012, 10:52 PM   #1
ust
Senior Member
 
Registered: Mar 2003
Location: fasdf
Distribution: Debian / Suse /RHEL
Posts: 1,130

Rep: Reputation: 30
separate a file


I have a large text file , I would like to separate it into many files , each file have 1000 lines , for example , the file have 90000 lines then separtate it into 90 files ,

can advise what can i do ?

Thanks.
 
Old 01-08-2012, 11:11 PM   #2
micxz
Senior Member
 
Registered: Oct 2002
Location: CA
Distribution: openSuSE, Cent OS, Slackware
Posts: 1,131

Rep: Reputation: 75
You can get the first 1000 lines like so:
Code:
cat largefile | sed -n '1,1000p'
Then maybe write something to loop and grab the next thousand and so on;
 
Old 01-08-2012, 11:14 PM   #3
ust
Senior Member
 
Registered: Mar 2003
Location: fasdf
Distribution: Debian / Suse /RHEL
Posts: 1,130

Original Poster
Rep: Reputation: 30
Quote:
Originally Posted by micxz View Post
You can get the first 1000 lines like so:
Code:
cat largefile | sed -n '1,1000p'
Then maybe write something to loop and grab the next thousand and so on;
thx reply ,

if I want it separate the whole file into different files , rather the first 1000 lines , what can i do ?

thanks.
 
Old 01-09-2012, 12:12 AM   #4
Telengard
Member
 
Registered: Apr 2007
Location: USA
Distribution: Kubuntu 8.04
Posts: 579
Blog Entries: 8

Rep: Reputation: 147Reputation: 147
Here is an awk program which I believe will do what you ask.

Code:
# split.awk
#
# Splits a large file into files of chunkSize lines each.
# Minimally tested with mawk.

BEGIN {
    chunkSize=1000	# Set number of lines per file.
    chunkNum=1
}

{
    print > (FILENAME "." chunkNum)
}

! (NR % chunkSize) {
    close((FILENAME "." chunkNum++))
}
Here is a sample data file of ninety thousand lines.

Code:
test$ wc -l numbers.txt
90000 numbers.txt
test$ cat numbers.txt
1
2
3
4
5
6
7
8
9
10
11
... goes on ...
89995
89996
89997
89998
89999
90000
test$
Here is how to process the file with the awk program.

Code:
awk -f split.awk numbers.txt
Here are the resulting files.

Code:
test$ ls -v numbers.txt.*                                     
numbers.txt.1   numbers.txt.19  numbers.txt.37  numbers.txt.55  numbers.txt.73
numbers.txt.2   numbers.txt.20  numbers.txt.38  numbers.txt.56  numbers.txt.74
numbers.txt.3   numbers.txt.21  numbers.txt.39  numbers.txt.57  numbers.txt.75
numbers.txt.4   numbers.txt.22  numbers.txt.40  numbers.txt.58  numbers.txt.76
numbers.txt.5   numbers.txt.23  numbers.txt.41  numbers.txt.59  numbers.txt.77
numbers.txt.6   numbers.txt.24  numbers.txt.42  numbers.txt.60  numbers.txt.78
numbers.txt.7   numbers.txt.25  numbers.txt.43  numbers.txt.61  numbers.txt.79
numbers.txt.8   numbers.txt.26  numbers.txt.44  numbers.txt.62  numbers.txt.80
numbers.txt.9   numbers.txt.27  numbers.txt.45  numbers.txt.63  numbers.txt.81
numbers.txt.10  numbers.txt.28  numbers.txt.46  numbers.txt.64  numbers.txt.82
numbers.txt.11  numbers.txt.29  numbers.txt.47  numbers.txt.65  numbers.txt.83
numbers.txt.12  numbers.txt.30  numbers.txt.48  numbers.txt.66  numbers.txt.84
numbers.txt.13  numbers.txt.31  numbers.txt.49  numbers.txt.67  numbers.txt.85
numbers.txt.14  numbers.txt.32  numbers.txt.50  numbers.txt.68  numbers.txt.86
numbers.txt.15  numbers.txt.33  numbers.txt.51  numbers.txt.69  numbers.txt.87
numbers.txt.16  numbers.txt.34  numbers.txt.52  numbers.txt.70  numbers.txt.88
numbers.txt.17  numbers.txt.35  numbers.txt.53  numbers.txt.71  numbers.txt.89
numbers.txt.18  numbers.txt.36  numbers.txt.54  numbers.txt.72  numbers.txt.90
test$ wc -l numbers.txt.*                                     
  1000 numbers.txt.1
  1000 numbers.txt.10
  1000 numbers.txt.11
  1000 numbers.txt.12
  1000 numbers.txt.13
  1000 numbers.txt.14
  1000 numbers.txt.15
  1000 numbers.txt.16
  1000 numbers.txt.17
  1000 numbers.txt.18
  1000 numbers.txt.19
  1000 numbers.txt.2
  1000 numbers.txt.20
  1000 numbers.txt.21
  1000 numbers.txt.22
  1000 numbers.txt.23
  1000 numbers.txt.24
  1000 numbers.txt.25
  1000 numbers.txt.26
  1000 numbers.txt.27
  1000 numbers.txt.28
  1000 numbers.txt.29
  1000 numbers.txt.3
  1000 numbers.txt.30
  1000 numbers.txt.31
  1000 numbers.txt.32
  1000 numbers.txt.33
  1000 numbers.txt.34
  1000 numbers.txt.35
  1000 numbers.txt.36
  1000 numbers.txt.37
  1000 numbers.txt.38
  1000 numbers.txt.39
  1000 numbers.txt.4
  1000 numbers.txt.40
  1000 numbers.txt.41
  1000 numbers.txt.42
  1000 numbers.txt.43
  1000 numbers.txt.44
  1000 numbers.txt.45
  1000 numbers.txt.46
  1000 numbers.txt.47
  1000 numbers.txt.48
  1000 numbers.txt.49
  1000 numbers.txt.5
  1000 numbers.txt.50
  1000 numbers.txt.51
  1000 numbers.txt.52
  1000 numbers.txt.53
  1000 numbers.txt.54
  1000 numbers.txt.55
  1000 numbers.txt.56
  1000 numbers.txt.57
  1000 numbers.txt.58
  1000 numbers.txt.59
  1000 numbers.txt.6
  1000 numbers.txt.60
  1000 numbers.txt.61
  1000 numbers.txt.62
  1000 numbers.txt.63
  1000 numbers.txt.64
  1000 numbers.txt.65
  1000 numbers.txt.66
  1000 numbers.txt.67
  1000 numbers.txt.68
  1000 numbers.txt.69
  1000 numbers.txt.7
  1000 numbers.txt.70
  1000 numbers.txt.71
  1000 numbers.txt.72
  1000 numbers.txt.73
  1000 numbers.txt.74
  1000 numbers.txt.75
  1000 numbers.txt.76
  1000 numbers.txt.77
  1000 numbers.txt.78
  1000 numbers.txt.79
  1000 numbers.txt.8
  1000 numbers.txt.80
  1000 numbers.txt.81
  1000 numbers.txt.82
  1000 numbers.txt.83
  1000 numbers.txt.84
  1000 numbers.txt.85
  1000 numbers.txt.86
  1000 numbers.txt.87
  1000 numbers.txt.88
  1000 numbers.txt.89
  1000 numbers.txt.9
  1000 numbers.txt.90
 90000 total
test$
Surely there is a simpler solution

HTH
 
Old 01-09-2012, 12:35 AM   #5
Dark_Helmet
Senior Member
 
Registered: Jan 2003
Posts: 2,786

Rep: Reputation: 370Reputation: 370Reputation: 370Reputation: 370
My gut tells me this is a "homework" type of problem. It sounds too much like a contrived, textbook situation. I could be wrong though. I'll play it safe and point you in the right direction, but I'm not going to write the full command

You need to look at the split command (man split).

EDIT:
Not to mention the fact that the split command has an oh-so-coincidental default in the first line of its description.

Last edited by Dark_Helmet; 01-09-2012 at 12:39 AM.
 
1 members found this post helpful.
Old 01-09-2012, 12:40 AM   #6
micxz
Senior Member
 
Registered: Oct 2002
Location: CA
Distribution: openSuSE, Cent OS, Slackware
Posts: 1,131

Rep: Reputation: 75
Or how about "split". I think this is "simpler";
Code:
split -l 1000 bigfile chunk


---------- Post added 01-08-12 at 09:40 PM ----------

opps you beat me to the post. Did I just do someones homework?

Last edited by micxz; 01-09-2012 at 01:08 AM.
 
1 members found this post helpful.
Old 01-09-2012, 12:58 AM   #7
Dark_Helmet
Senior Member
 
Registered: Jan 2003
Posts: 2,786

Rep: Reputation: 370Reputation: 370Reputation: 370Reputation: 370
Quote:
Originally Posted by micxz
Did I just do someones homework?
lol

If you did, you would not be the first. Nor would you be the last.

I know you're not the first because I've (unwittingly) done it before. Though, if it is homework, here's hoping Telengard's awk script was crammed into the blank line for the answer!
 
Old 01-09-2012, 01:12 AM   #8
Telengard
Member
 
Registered: Apr 2007
Location: USA
Distribution: Kubuntu 8.04
Posts: 579
Blog Entries: 8

Rep: Reputation: 147Reputation: 147
It hadn't occurred to me that this might be homework (grr!)

Quote:
Originally Posted by Dark_Helmet View Post
Though, if it is homework, here's hoping Telengard's awk script was crammed into the blank line for the answer!
That would be pretty funny

Want another reason to laugh? I didn't even think of split, clearly a superior solution.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
[SOLVED] Booting LFS on separate USB HDD + separate /boot parition nivwusquorum Linux From Scratch 33 12-14-2011 07:38 AM
Each mail to be in Separate File..!!! anishkumarv Linux - Newbie 3 03-10-2011 11:15 PM
File name in separate column jeesun Linux - General 5 04-26-2010 06:27 AM
Looking for a tool to auto crop and separate images in to separate files.. mlsfit Linux - Software 2 08-06-2006 04:13 PM
file accidentally appended to itself, how to separate? sqrt7744 Linux - General 5 04-29-2006 04:30 PM


All times are GMT -5. The time now is 04:22 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration