[SOLVED] use sed to break up a csv file into several smaller files

saphil · 05-22-2013, 04:01 PM

I have a large csv file that I want to break into smaller files
There is a title block, that does not share the index labeling of the rest of the file.
There is a labels line, that I want to separate out so I can call it for reports of each individual row in the content section, and there are 2 lines on the end that also do not share the indexing of the content section.

I have gotten this far, and the first 2 sections create their own file properly, but sed does not like variables at all, and it is failing to expand them.

echo "enter input file"; read infile
ct=`wc -l $infile` # I thought I could find out how many
# lines the file had and do an "end minus" to
# get the entire content section and the last 2 lines.
echo $ct # This didn't work, of course.

sed -n '1,7p' $infile > $infile"_title.csv"

sed -n '8p' $infile > $infile"_labels.csv"

sed -n '9,$ct-3p' $infile > $infile"_content.csv"

sed -n '$ct-1,$ctp' $infile > $infile"_endblock.csv"

cfajohnson · 05-22-2013, 07:45 PM

Quote:

Originally Posted by saphil

I have a large csv file that I want to break into smaller files
There is a title block, that does not share the index labeling of the rest of the file.
There is a labels line, that I want to separate out so I can call it for reports of each individual row in the content section, and there are 2 lines on the end that also do not share the indexing of the content section.

I have gotten this far, and the first 2 sections create their own file properly, but sed does not like variables at all, and it is failing to expand them.

Code:

echo "enter input file"; read infile
ct=`wc -l $infile`  # I thought I could find out how many 
                    #  lines the file had and do an "end minus" to 
                    #  get the entire content section and the last 2 lines. 
echo $ct            # This didn't work, of course.

sed -n '1,7p' $infile > $infile"_title.csv"

You are quoting the wrong thing. Quote the variable not the literal:
Code:
sed -n '1,7p' "$infile" > "$infile"_title.csv

Quote:

Code:

sed -n '8p' $infile > $infile"_labels.csv"

sed -n '9,$ct-3p' $infile > $infile"_content.csv"

sed -n '$ct-1,$ctp' $infile > $infile"_endblock.csv"

Variables inside single quotes are not expanded; use double quotes instead:
Code:
sed -n "$ct-1,$ctp" "$infile" > "$infile"_endblock.csv

saphil · 05-23-2013, 07:09 AM

Quote:

Originally Posted by cfajohnson

You are quoting the wrong thing. Quote the variable not the literal:
Code:
sed -n '1,7p' "$infile" > "$infile"_title.csv

Variables inside single quotes are not expanded; use double quotes instead:
Code:
sed -n "$ct-1,$ctp" "$infile" > "$infile"_endblock.csv

Thanks! This helped a lot.

My code that works looks like this:

Code:

echo "enter input file"; read infile

ct=`cat $infile |wc -l`
ct1=`expr $ct - 1`
ct2=`expr $ct - 2`
echo $ct, $ct1, $ct2

sed -n '1,7p' "$infile" > "$infile"_title.csv

sed -n "8p" "$infile" > "$infile"_labels.csv

sed -n "9,$ct2"p "$infile" > "$infile"_content.csv

sed -n "$ct1,$ct"p "$infile" > "$infile"_endblock.csv

Code:

mhalton@mint-141 ~/Desktop/appy $ ./spliff.sh -x 
+ echo 'enter input file'
enter input file
+ read infile
test1.csv
++ wc -l
++ cat test1.csv
+ ct=67396
++ expr 67396 - 0
+ ct0=67396
++ expr 67396 - 1
+ ct1=67395
++ expr 67396 - 2
+ ct2=67394
+ echo 67396, 67395, 67394
67396, 67395, 67394
+ sed -n 1,7p test1.csv                   # made test1.csv_title.csv
+ sed -n 8p test1.csv                     # made test1.csv_labels.csv
+ sed -n 9,67394p test1.csv               # made test1.csv_content.csv
+ sed -n 67395,67396p test1.csv           # made test1.csv_endblock.csv

Added the following after line 6 to clean up the output filenames:

Code:

slim="`echo $infile | cut -d '.' -f 1 `"
echo "$slim is the stub"
slim="test1"

This takes the infile and cuts it at the '.' delimiter and displays field 1.