using ed, sed to correct database

ifeatu · 11-15-2008, 04:17 PM

Okay, I dunno if I'm posting in the correct Forum but here it goes:

I have a huge tape library that has some unique attributes; all of the tracks are separated into folders by date (mostly)...each folder is labeled in this syntax by default:

LOCATION-YEAR-MN-DY- Subject -SPK (The last three letters are the initials of the speaker on the track) I have about a thousand of these folders and there are more to come...I want to make a file with a list of the directories and replace the Location-YEAR-MN-DY syntax with a more conventional syntax of DY-MN-YEAR so that I can redirect to an even larger flat file (excel) and compare data...so the creating a file part is simple (a simple ls redirection) but coming up with the syntax for sed is eluding me...can anyone help...here is a sample of the file:

Code:

1984-04-20- Ordanation!!!!!-BLH
1999-08-08- The Holy Ghost! - BCH
Albuquerque-2004-07-10- Being a Dedicated Christian -BLH
Baton Rouge-2003-4-13 -Unity_Building the Body-BLJ
Berkeley-06-03- Revelations 138 Worlds -BLH
Berkeley-1976-04-25- Ressurection -BLH
Berkeley-1976-11-25- Fulfillment of Pr
Berkeley-1976-11-25- Time -BLH
Berkeley-1978-11-25 Early Church -BLH
Berkeley-1983 - 10-21 -Attibutes of Christ - BLH
Berkeley-1983-02-02- Perfection Overcoming -BLH
Berkeley-1983-02-04- Perfection _ Brotherly Love - BLH
Berkeley-1983-06-19 - Making the Bride - BCH
Berkeley-1983-10-21- Jesus Christ - BLH
Berkeley-1983-10-23- Washing yourself from the World - BCH
Berkeley-1983-11-04- Trying to Build the body - BJ
Berkeley-1983-11-04- Trying to Build the body - BJS
Berkeley-1983-11-06- Fireside about importance of family -BS
Berkeley-1983-12-09- Daniel 8 -BLH
Berkeley-1983-12-09- Q&A - Daniel 8 -BLH
Berkeley-1983-12-23- Restoration of the Church - BLH

syg00 · 11-15-2008, 06:17 PM

That is *UGLY* data - not consistent at all. Some leading zeroes missing, imbedded blanks in the date sometimes - one line doesn't even have the year. Try this to get you started

Code:

 sed -r 's/(.*)([0-9]{4})\s*-\s*([0-9]{1,2})\s*-\s*([0-9]{1,2})(.*)/\4-\3-\2\5/' junk.txt

ifeatu · 11-15-2008, 06:19 PM

Couldn't I truncate all of the spaces and such before doing the search?

I will run that command and get back to you with my results.

ifeatu · 11-15-2008, 06:40 PM

Ok I just ran the command and the good news is there were no errors...the bad news is that is didn't make a single change....when I cat the file the format remained the same...
Here is what I typed "directory" being the name of the file that contains the directory listing..

Code:

$ sed -r 's/ (.*) ([0-9] {4} )\s*-\s*([0-9]{1,2})\s*-\s*([0-9]{1,2})(.*)/\4-\3-\2\5/' directory

syg00 · 11-15-2008, 06:48 PM

Write it out to a new file "... > newfile.txt"

ifeatu · 11-15-2008, 07:22 PM

hmmm same thing...I redirected output to another file called directory2:

Code:

$ sed -r 's/ (.*) ([0-9] {4} )\s*-\s*([0-9]{1,2})\s*-\s*([0-9]{1,2})(.*)/\4-\3-
\2\5/' directory > directory2

can I forward you a copy of the file to try the command on it yourself?

syg00 · 11-15-2008, 07:27 PM

I would suggest you try the command I posted - unmodified.

ifeatu · 11-15-2008, 07:40 PM

You are correct sir. Now I'll just spend some time decrypting your syntax so that I might replicate it in the future. Thank you very much!!

ifeatu · 11-15-2008, 07:42 PM

How might I append the location to the end (since the former command removed the location?)

syg00 · 11-15-2008, 08:58 PM

There's plenty of ideas for you to look at in that command - it does what you asked for initially. The other data is still available if you didn't destroy the source. I'll leave it as an exercise for you to finish it off. There is good doco on the sed.sf.net site.

ifeatu · 11-15-2008, 09:23 PM

Quote:

Originally Posted by syg00

That is *UGLY* data - not consistent at all. Some leading zeroes missing, imbedded blanks in the date sometimes - one line doesn't even have the year. Try this to get you started

Code:

 sed -r 's/(.*)([0-9]{4})\s*-\s*([0-9]{1,2})\s*-\s*([0-9]{1,2})(.*)/\4-\3-\2\5/' junk.txt

Okay I'm going to attempt to ask an educated question about what you've done with this command...its very impressive to me that it didn't take you like 3 days to figure out that expression...here it goes:

Question 1: At the beginning of your expression you used sed -r 's/
Why would you need the '-r' option...from the man pages it says its for using extended regular expressions in the script. What extended expressions did you use?

Question 2: When you cached the output of ([0-9]{4}) in what memory block did you store it in? I dont see a \1 or \2 until later on in the expression but I do see several instances of \s* what does that do?

Thanks for the clarification.

syg00 · 11-15-2008, 10:05 PM

1) Supposedly the "-r" isn't required, but often the substitution of subexpressions on RHS is rejected without it. Try it and see. Becomes habit after a while.
2) \s* is any number (zero or more) whitespace characters - tabs, blanks. Very handy when you aren't sure what the actual "space" is. Now about that "-r" ...

ifeatu · 11-15-2008, 10:08 PM

Quote:

Originally Posted by syg00

There's plenty of ideas for you to look at in that command - it does what you asked for initially. The other data is still available if you didn't destroy the source. I'll leave it as an exercise for you to finish it off. There is good doco on the sed.sf.net site.

Thank you for the website reference...I'll begin my studying. Thanks for your help.

ifeatu · 12-02-2008, 03:51 PM

I purchased the O'Reilly Book by J. Freidl on RegEx (ISBN: 9780596528126) I have no idea how I got along this far without this book...thanks for your advice....I also joined a sed yahoo group.

Thanks!

pixellany · 12-02-2008, 04:03 PM

Holy cow!!! I thought I was a SED fanatic.

Another good reference:
http://www.grymoire.com/Unix/