Hi All,

I'd be grateful for some guidance as my Bash skills aren't cutting the mustard!

I have a need to process a large number (thousands) of small text files, to find the total numbers of files that contain certain combinations of text strings. The files are spread through multiple subdirectories.

The 'variables' in each file are a device number such as [001, 002, 003... 006], a date (mm/yy) string in the format "/10/14" (October 14) and a text string, which will either be present or not. If present it will read 'Settled'.

My need is to produce a simple report that says, for example:

001, /09/14 - Settled: 325
001, /09/14 - Total files: 817
001, /10/14 - Settled: 842
001, /10/14 - Total files: 2812
002, /09/14 - Settled: 665
002, /09/14 - Total files: 1823
etc etc.
(Format isn't important - I just need a count for each category).

I'm guessing I need to do something like:

I tried using piped grep commands, moving matching files to a directory, grepping them again and then filtering the output list through wc -l, but this got messy fast!

I'm looking for a way to input the device numbers (001, 002 etc) and the valid month dates (from a file maybe?) and loop through them counting the matches.

I'm starting to think that Bash isn't the right tool, and something like Perl might be better, but I'm clueless with Perl! Anyone have suggestions please?

Thanks all.

Frustrated from Wales.

hm. something like this:
grep -rH '[0-9][0-9][0-9], \/..\/..'
will collect all the files/lines with the required info (probably the pattern is not ok)
pipe the output into awk, where you can easily count devices, device and date or anything else.

No need for grep, awk has (regex) pattern patching - likewise for perl.
Depending on data structure - is it well formatted, known location(s) in each record for the data, consistent field separator(s), device number always present ? ... - awk would be my weapon of choice. Good logic tests, easy counters.
Likewise for perl of course ....

1 members found this post helpful.

The files are variable length, but the field separators are consistent and all the information (apart from the 'settled' string) are always present. Think of a layout like a cash till receipt - header, variable length body, footer. I don't care about the content of the body, but I can't always rely on exactly where the data I want will be, hence my first effort using grep.

I'll brush up on awk - it's been a while!

Thanks for the input folks.

Post some representative records - cleansed in need if the data is sensitive.

Thanks. Here's a sample of one that does NOT contain the 'settled' string:


I need to filter/group by a substring within "772/080/004/121" (The '004' could be '001' to '006' or so). Also I need the month and year from the datestamp (06.14, 07.14, 08.14 for example), shown in bold. (I mistakenly described this as being dd/mm format rather than dd.mm previously - sorry!).

In some cases there will be an additional string on the bottom line, like this:

(The c0! is part of the text file formatting)

No way you can do that without some logic capabilities - awk or perl or similar.
You need to find the line with device/date and capture that data - and set a flag. Then go look for "Settled" on subsequent records. Maybe you could use this as a start (lots of presumptions and no manipulation on my part)

Got bored while watching the soccer - how about something like this

Wow. It's gonna take a while to grok that, but many thanks! (I'm gonna assume the soccer wasn't great!).

I'll give it a whirl on some sample data and report back.