[SOLVED] text munging problem

sharky · 04-12-2023, 06:30 PM

Input data is a text file about 300k - almost 7000 lines.

Using python 3.

I want to append every term in the text file that contains MAX (just an example) with _33.

For example:

"left -21 exMAX_14" would become "left -21 exMAX_14_33"

There are several search terms besides MAX but the append term will always be _33.

syg00 · 04-12-2023, 06:48 PM

I'm sure python can handle it, but this is what sed was built for.

Code:

sed 's/MAX/MAX_33/' some.file

Presumes your description of requirements is robust. You can do several in one invocation, use regex ...
Looks simple, can be sophisticated.

Can be done in-place, but I always redirect to a new file for sanity.

sharky · 04-12-2023, 07:14 PM

I was thinking python would be the better solution because I want to append every term that contains MAX. I do not necessarily want to append MAX directly as your sed example shows. MAX could be at the beginning, middle, or end of the term but I always want to append the end of the term.

Examples
MAX -> MAX_33
MAX_XXX -> MAX_XXX_33
XXX_MAX23 -> XXX_MAX23_33
N0MIX -> N0MIX

For that sed might still be the better solution but I am more familiar with python.

syg00 · 04-12-2023, 07:52 PM

Certainly simple enough to accommodate - good luck with your python adventures. Gotta get round to getting in it one day ...

NevemTeve · 04-12-2023, 09:50 PM

Code:

sed 's/^.*MAX.*$/&_33/' inputfile >outputfile

michaelk · 04-12-2023, 10:17 PM

Maybe a better way using awk but:

Code:

awk ' { if ( $0 ~ /MAX/ )  printf "%s_33\n",$0; else printf "%s\n",$0 }' /path/to/file

syg00 · 04-12-2023, 11:14 PM

I'm guessing that isn't what the OP is looking for. Seems the OP wants to learn python - after s?he/they furnishes a working python solution, I'll provide a trivial sed one.

sharky · 04-12-2023, 11:57 PM

Quote:

Originally Posted by NevemTeve

Code:

sed 's/^.*MAX.*$/&_33/' inputfile >outputfile

This is close.

Input:

Quote:

not MAX_PIN1 MAX_drawing bad_MAX_PIN
select -interact -not MAX_PIN1 bad_MAX_PIN -outputlayer MAX_PIN
select -interact METAL2_slot M2_PIN1 M2_PIN1_slot
select -interact M2_PIN1 METAL2_slot M2_slot_PIN

Output:

Quote:

not MAX_PIN1 MAX_drawing bad_MAX_PIN_33
select -interact -not MAX_PIN1 bad_MAX_PIN -outputlayer MAX_PIN_33
select -interact METAL2_slot M2_PIN1 M2_PIN1_slot
select -interact M2_PIN1 METAL2_slot M2_slot_PIN

Needed:

Quote:

not MAX_PIN1_33 MAX_drawing_33 bad_MAX_PIN_33
select -interact -not MAX_PIN1_33 bad_MAX_PIN_33 -outputlayer MAX_PIN_33
select -interact METAL2_slot M2_PIN1 M2_PIN1_slot
select -interact M2_PIN1 METAL2_slot M2_slot_PIN

Not just at the end of the line but appended to every term.

NevemTeve · 04-13-2023, 02:29 AM

With GNU!sed:

Code:

sed -E '/MAX/{ s/\b(\w*)(_33)*\b/\1_33/g }'

But most certainly, you can do it with Python. Some help can be found here:
https://docs.python.org/3/library/re.html
https://www.hackerrank.com/domains/p...5B%5D=py-regex

sharky · 04-13-2023, 09:50 AM

Quote:

Originally Posted by NevemTeve

With GNU!sed:

Code:

sed -E '/MAX/{ s/\b(\w*)(_33)*\b/\1_33/g }'

But most certainly, you can do it with Python. Some help can be found here:
https://docs.python.org/3/library/re.html
https://www.hackerrank.com/domains/p...5B%5D=py-regex

Appreciate the helpful links.

The sed command you suggest appends every term. Only the terms containing 'MAX' should be appended.

Quote:

not_33 MAX_PIN1_33 MAX_drawing_33 bad_MAX_PIN_33
select_33 -interact_33 -not_33 MAX_PIN1_33 bad_MAX_PIN_33 -outputlayer_33 MAX_PIN_33
select -interact METAL2_slot M2_PIN1 M2_PIN1_slot
select -interact M2_PIN1 METAL2_slot M2_slot_PIN

michaelk · 04-13-2023, 10:03 AM

Code:

import os
import sys

txt = "select -interact -not MAX_PIN1 bad_MAX_PIN -outputlayer MAX_PIN"
x = txt.split(" ")
y = len(x)
print(x) 
for m in range(0,y-1):
    if "MAX" in x[m]:
        sys.stdout.write(x[m]+"_33 ")
    else:
        sys.stdout.write(x[m]+" ")

if "MAX" in x[y-1]:
   print(x[y-1]+"_33")
else:
   print(x[y-1])

Just throwing something out...

NevemTeve · 04-13-2023, 10:35 AM

@OP Would you mind showing what you have done so far and where you're stuck?

teckk · 04-13-2023, 12:05 PM

cat 1test.txt

Code:

left -21 exMAX_14
left -22 exMAX_15
left -23 exMAX_16
left -24 exMAX_17

Code:

def green(text):
    return '\033[0;32m' + text + '\033[0m'
        
with open('1test.txt', 'r') as f:
    for i in f:
        print(i.rstrip()+f'{green("_33")}')

teckk · 04-13-2023, 12:10 PM

Actually that is a little confusing with all the f's, how about

Code:

with open('1test.txt', 'r') as x:
    for i in x:
        print(i.rstrip()+f'{green("_33")}')

danielbmartin · 04-14-2023, 11:08 AM

With this InFile ...

Code:

not MAX_PIN1 MAX_drawing bad_MAX_PIN
select -interact -not MAX_PIN1 bad_MAX_PIN -outputlayer MAX_PIN
select -interact METAL2_slot M2_PIN1 M2_PIN1_slot
select -interact M2_PIN1 METAL2_slot M2_slot_PIN

... this sed ...

Code:

sed -r 's/(MAX[_[:alnum:]]*)/\1_33/g' $InFile >$OutFile

... produced this OutFile ...

Code:

not MAX_PIN1_33 MAX_drawing_33 bad_MAX_PIN_33
select -interact -not MAX_PIN1_33 bad_MAX_PIN_33 -outputlayer MAX_PIN_33
select -interact METAL2_slot M2_PIN1 M2_PIN1_slot
select -interact M2_PIN1 METAL2_slot M2_slot_PIN

I cannot take credit for this solution. A Google search led me to a similar solution written by mashuptwice at https://stackoverflow.com/questions/71731677/

Daniel B. Martin

.