Extract data from a file using bash then use part of that data to rename the input fi

mfarch99 · 08-29-2014, 09:29 AM

I have a file that has data that changes consistently. I need to make the file name unique and use a portion of the data that is always constant as part of the new filename. Basically, this is what is needed:
1) read the file line by line
2) find " 70 " in the first 5 columns
3) Once found, start extracting data that starts in column 12 for 18
4) Use that data found and place in a variable
5) take the variable and append to the end of the input file name.
6) result would be a new file name

I have the bash code to read a file line by line, but where I'm failing is gathering the data from the file to place in a variable.

As an example, the input file name would be RPT70
I need to have the result of this script create a file name that looks something like this:

RPT70.$variablename

schneidz · 08-29-2014, 09:33 AM

can you post your code so far so someone can help debug it.

i suggest looking into awk.

szboardstretcher · 08-29-2014, 09:35 AM

Also a piece of the file would be helpful.

mfarch99 · 08-29-2014, 12:02 PM

Sorry,
here is the code and a piece if the file:

#!/bin/bash
filename="mike.txt"
while read -r line
do
name=$line
echo "Name read from file - $name"

done < "$filename"

1SASSHIS8 CA WORKLOAD AUTOMATION SE CONTROL CARD EDIT 08/28/14 12:50 PAGE 1
0------------------------------------------------------------------------------------------------------------------------------------
0 1 1 2 2 3 3 4 4 5 5 6 6 7 7 8 EDIT COMMENTS
....5....0....5....0....5....0....5....0....5....0....5....0....5....0....5....0----------------------------------------------------
070 142401230142401250 HIS8-09 NO CONTROL CARD ERRORS DETECTED.
70 142401230142401250 HIS8-18 CONTROL CARD EDIT FINAL RESULTS.
1SASSHIS8 CA WORKLOAD AUTOMATION SE CONTROL CARD EDIT 08/28/14 12:50 PAGE 2
0------------------------------------------------------------------------------------------------------------------------------------
0 1 1 2 2 3 3 4 4 5 5 6 6 7 7 8 EDIT COMMENTS
....5....0....5....0....5....0....5....0....5....0....5....0....5....0....5....0----------------------------------------------------
- *** T O T A L S ***
0 001 CARDS READ.
0 001 CARDS ACCEPTED.
0 000 CARDS REJECTED.
-
-2471291 ...LOG RECORDS READ.
00011526 ...RECORDS EXTRACTED FOR --- 70
1

grail · 08-29-2014, 12:18 PM

Based on the data provided:

1. What is your definition of a column?
2. What would be the output based on the supplied data?
3. What does 'column 12 for 18' mean?
4. The string being searched for ' 70 ', does it have spaces and/or tabs either side of it?

mfarch99 · 08-29-2014, 12:39 PM

It is hard to identify the data in this thread. The file is very long. and about 133 columns wide. The data I'm looking for is in the file and is a parameter card representation. The information would look like this 70 142401050142401110. The first column or character would be a space followed by 70 then 8 spaces and then the data I need to retrieve. The data being retrieved changes. The only constant would be " 70 "

keefaz · 08-29-2014, 12:48 PM

NO CONTROL CARD ERRORS DETECTED.

Is it 1 column or 5 ?

mfarch99 · 08-29-2014, 07:40 PM

The data that wiil be searched starts in column(position) 1 of the string. All counts be off that position. I mentioned the columns, positions earlier in the thread. So basically the search string to lokk for starts in position 1 as a space the next 2 are numeric 70 and then the next two are spaces. Like this-- " 70 " without the quotes.

The data to be extracted would be on that same line but would start around 8 positions in and go for 12 positions. That data I need to extract to a variable and use as part of the new file name. An example was provided above. Thank you..

grail · 08-30-2014, 02:22 AM

Ok ... I think I am getting a better picture (I hope)

So as far as columns go we are simply looking at 1 column is 1 character (yes)

So to restate the question, we are searching for lines that start with a space followed by the number 70 and some more white space. We are trying to retrieve data from the the 8th
character on the line and the subsequent 12 characters.

Based on this the current data return no data, however I believe that due to you pasting directly within your question we have lost information about white space (in the future
if you place code or data in [code][/code] tags, this type of information will be preserved).

So i have edited the information to look as below:

Code:

1SASSHIS8 CA WORKLOAD AUTOMATION SE CONTROL CARD EDIT 08/28/14 12:50 PAGE 1
0------------------------------------------------------------------------------------------------------------------------------------
0 1 1 2 2 3 3 4 4 5 5 6 6 7 7 8 EDIT COMMENTS
....5....0....5....0....5....0....5....0....5....0....5....0....5....0....5....0----------------------------------------------------
070 142401230142401250 HIS8-09 NO CONTROL CARD ERRORS DETECTED.
 70 142401230142401250 HIS8-18 CONTROL CARD EDIT FINAL RESULTS.
1SASSHIS8 CA WORKLOAD AUTOMATION SE CONTROL CARD EDIT 08/28/14 12:50 PAGE 2
0------------------------------------------------------------------------------------------------------------------------------------
0 1 1 2 2 3 3 4 4 5 5 6 6 7 7 8 EDIT COMMENTS
....5....0....5....0....5....0....5....0....5....0....5....0....5....0....5....0----------------------------------------------------
- *** T O T A L S *** 
0 001 CARDS READ.
0 001 CARDS ACCEPTED.
0 000 CARDS REJECTED.
-
-2471291 ...LOG RECORDS READ.
00011526 ...RECORDS EXTRACTED FOR --- 70
1

Here we now see on line 6 we have the scenario where the following grep will pick up data we are looking for:

Code:

$ grep -E '^ 70 ' in_file
 70 142401230142401250 HIS8-18 CONTROL CARD EDIT FINAL RESULTS.

Assuming I am on the correct path here, would the data to be returned be:

Code:

301424012

mfarch99 · 09-05-2014, 07:52 AM

Sorry I did not het back to you sooner. This is what would be needed from that line 142401230142401250.
I know if I just do an awk '{print $2}' I can retrieve the data I want. But how to I place that data into a variable so I can use that to append to the end of the original file name.

mfarch99 · 09-05-2014, 10:33 AM

Looks like I have what is needed. Thanks. Here is what I have as the final code. Crude but works:
#!/bin/bash

RPT70EXT=$(grep -E '^ 70 ' RPT70 | awk '{print $2}')

mv -f RPT70 RPT70.$RPT70EXT

grail · 09-05-2014, 10:51 AM

Well that would be a grep and awk solution, not a bash one.

Also, awk can already match regular expressions, so grep is not needed.

Here are some other alternatives to return the required data:

Code:

# awk
awk '/^ 70 /{print $2}' RPT70

# sed
sed -rn 's/^ 70 *([^ ]*).*/\1/p' RPT70

# bash
#!/usr/bin/env bash

regex='^ 70 *([^ ]*)'

while IFS=$'\n' read -r line
do
	[[ "$line" =~ $regex ]] && echo "${BASH_REMATCH[1]}"
done<RPT70

# alternate bash
#!/usr/bin/env bash

while read -r num data _
do
	[[ "$num" == 70 ]] && echo "$data"
done<RPT70