how do i remove the first 5 numbers from my data string ??

charlyb21 · 04-26-2012, 08:39 AM

Hi,

I am trying to process twelve .csv files that contain 9 columns of 20,000 data points each.
I have got a piece of Fortran code to perform an FFT on this data and it runs.
I also have a Bash command file to run the FFT on all my data sets one after the other and output the results to a .dat file.

BUT i have a problem, the data does not open in excel so im trying to use gnuplot to produce graphs of the .dat files but the first column of my data is time and due to the logger used to take the data the time values have 12 digits. Of these 12 digits the first 5 are all the same so i wish to remove the first 5 digits from the string so that gnuplot is able to plot it as at the moment it is cutting off the last 2 digits which are the key ones.

Any help with this would be greatly appreciated!!
Please let me know if anything is not clear

danielbmartin · 04-26-2012, 09:30 AM

Quote:

Originally Posted by charlyb21

... i wish to remove the first 5 digits from the string ...

Try this ...

Code:

sed 's/,/\n/g' $InFile \
|cut -c6-              \
|paste -s -d,          \
> $OutFile

Explanation:
If the input file looks like this ...

Code:

123456789,123456789,123456789,123456789,123456789,123456789

... the sed changes it to this ...

Code:

... then cut changes it to this ...

Code:

... then paste changes it to this ...

Code:

6789,6789,6789,6789,6789,6789

Daniel B. Martin

grail · 04-26-2012, 10:42 AM

hmmm ... my understanding is that it is the first 5 from each line and not from each field (maybe OP can clear that up). Also you do not indicate if the numbers start at the
beginning of the line or if there is possibly white space prior? (I will assume there is none)

So borrowing Daniel's example:

Code:

echo '123456789,123456789,123456789,123456789,123456789,123456789' | sed -r 's/^[0-9]{5}//'

If running on a file:

Code:

sed -ri 's/^[0-9]{5}//' file

This will edit the file in place.

You also do not say why Excel had issues?

danielbmartin · 04-26-2012, 11:03 AM

Quote:

Originally Posted by grail

... my understanding is that it is the first 5 from each line and not from each field ...

If that is the case, then maybe we need nothing fancier than ...

Code:

cut -c6- $InFile > $OutFile

Quote:

This will edit the file in place.

Not to quarrel with the guru, I always avoid editing an input file in place. I want to be able to run and re-run a program as my code evolves, always starting with the same input file. In other words, it is dangerous to tamper with evidence!

Daniel B. Martin

grail · 04-26-2012, 12:01 PM

Quote:

Not to quarrel with the guru, I always avoid editing an input file in place. I want to be able to run and re-run a program as my code evolves, always starting with the same input file. In other words, it is dangerous to tamper with evidence!

No probs, supply -i an extension:

Code:

sed -ri.bak 's/^[0-9]{5}//' file

Now you have the edited file called "file" and the old file stored in "file.bak"

And definitely no quarrel ... I learn something different every time I look at questions