[SOLVED] ssed removes \r?\n

littlebigman · 05-27-2020, 06:01 AM

Hello,

I'm using super-sed 3.62, and noticed that ssed seems to remove (\r)\n from the regex before proceeding :

Code:

#NOK
ssed -R "s@^.+<time>.+</time>\r?\n@@g" < test.gpx > test.TIME.gpx

#OK
ssed -R "s@^.+<time>.+</time>@@g" < test.gpx > test.TIME.gpx

Is there a work-around?

Thank you.

syg00 · 05-27-2020, 06:13 AM

Stream-based tools (almost) never see the newline. Try your first test without it.

shruggy · 05-27-2020, 06:29 AM

ssed is obsoleted by GNU sed anyway. It was a fork of GNU sed to test some experimental features that long since found its way back into the GNU sed. What distro are you using? I suppose, something Debian-based? Or is it FreeBSD? It seems nobody else packages ssed anymore. Unless it's a very old release like Debian 8 (Jessie) there's no point in using ssed.

It seems like you're trying to parse an XML/SGML document with sed. Please don't. sed is a line-oriented tool, HTML/XML/SGML is in no way line-oriented.

If \n is at the end of line your sed expression won't work, but sed can handle \n embedded into the string by other means, e.g. via its commands N and G.

littlebigman · 05-27-2020, 06:29 AM

Debian.

A regex is fine when parsing a single line, like it is here.

But I'd like to also remove the \r?\n. Without it, I end up with a blank line.

boughtonp · 05-27-2020, 06:35 AM

Regular sed allows changing the line delimiter from \n to \0 (null character), using the -z option, but that was added in 2012 and the last release of super-sed appears to be 2005 so I guess it doesn't support that.

You could try comparing the codebases to see how much effort it is to update super-sed with new features.

Or, for this specific example that doesn't appear to be doing anything special, just use regular sed.

shruggy · 05-27-2020, 06:41 AM

I still don't understand what are you trying to do. Could you provide a sample of the input data?

Flag M can be specified for the s command in GNU sed. Is it what you're looking for?

littlebigman · 05-27-2020, 06:53 AM

I used ssed because it supports non-greediness, but I'll be happy to use GNU sed (4.4) if it does what I need.

I'm simply tring to remove all the lines that contain time in GPX files, including the carriage return:

Code:

<trk>
	<trkseg>
		<trkpt lon="2.325975" lat="48.821938">
			<ele>71.75</ele>
			<time>2020-05-13T15:31:04Z</time>
		</trkpt>

pan64 · 05-27-2020, 06:55 AM

Quote:

Originally Posted by littlebigman

Debian.

A regex is fine when parsing a single line, like it is here.

But I'd like to also remove the \r?\n. Without it, I end up with a blank line.

Again, sed does not see \n, because that is the delimiter. With other words: input is splitted into lines and sed commands work on these lines.
If you want to make sed work differently you need to check the option -z (see man sed).

littlebigman · 05-27-2020, 06:58 AM

Because you replied before my own reply.

The magic of non-face to face conversations :-)

Code:

NOK sed -E "s@^.+<time>.+</time>\r?\n@@g" < test.gpx > test.TIME.gpx

sed -E "s@^.+<time>.+</time>\r$@@g" < test.gpx > test.TIME.gpx
sed: -e expression #1, char 24: unterminated `s' command

NOK sed -E "s@^.+<time>.+</time>\n@@g" < test.gpx > test.TIME.gpx

CTRL+V CTRL+M
NOK sed -E "s@^.+<time>.+</time>^M@@g" < test.gpx > test.TIME.gpx
NOK sed -e "s@^.+<time>.+</time>^M@@g" < test.gpx > test.TIME.gpx

sed -i "s@^.+<time>.+</time>$@@g" < test.gpx > test.TIME.gpx
sed: -e expression #1, char 22: unterminated `s' command

NOK sed -z "s@^.+<time>.+</time>\r?\n@@g" < test.gpx > test.TIME.gpx

NOK sed -z "s@^.+<time>.+</time>\n@@g" < test.gpx > test.TIME.gpx
NOK sed -z 's@^.+<time>.+</time>\n@@g' < test.gpx > test.TIME.gpx

shruggy · 05-27-2020, 07:20 AM

Code:

xml2 <test.gpx|grep -v ^/gpx/trk/trkseg/trkpt/time=|2xml

littlebigman · 05-27-2020, 07:29 AM

Thank you. I'll try xml2/2xml if I can't get sed to work.

shruggy · 05-27-2020, 08:30 AM

xmlstarlet is also an option. Actually, it was my first choice, but it's picky about namespaces. E.g the sample GPX document from Wikipedia includes xmlns attributes, so you'll need to run it like this:

Code:

xmlstarlet ed -d //_:time test.gpx

OTOH, the example from OSM Wiki doesn't, so:

Code:

xmlstarlet ed -d //time test.gpx

Since I know nothing about your gpx data, I opted for the dumber xml2

.

littlebigman · 05-27-2020, 09:39 AM

Thanks.

The first examples I found about sed on the Net didn't mention that it removed carriage returns before handling the regex.

This will remove all empty lines, which is fine in my particular case:

Code:

sed -r "s@^.+<time>.+</time>@@g" < test.CRLF.gpx > test.CRLF.TIME.gpx
sed -r "/^$/d" test.CRLF.TIME.gpx > test.CRLF.TIME.stripped.gpx

shruggy · 05-27-2020, 09:44 AM

This can be done in one sed command:

Code:

sed -E 's@^.+<time>.+</time>@@g;/^$/d'

I still don't understand why you cannot just remove the lines in question:

Code:

sed '\@<time>.\+</time>@d'

boughtonp · 05-27-2020, 09:47 AM

Quote:

Originally Posted by littlebigman

I used ssed because it supports non-greediness...

Fair enough, but I'd rather go with Perl itself in that situation. (You're not using non-greediness here, and you don't need to.)

For basic substitutions it can be a simple case of switching from "sed 'EXPR'" to "perl -pe 'EXPR'".

For example:

Code:

perl -pe 's@^.+<time>.+</time>\r?\n@@g' < test.gpx > test.TIME.gpx

(That appears to remove newlines despite being in sed-like mode.)

Of course, that should be written as:

Code:

perl -pe 's@^[\t ]*<time>[^<]+</time>\r?\n@@g' < test.gpx > test.TIME.gpx

Because otherwise the .+ is likely to match to the end of the input before backtracking, whereas using accurate character classes will match only the characters needed, (which is both more efficient and makes intent clearer).

As various people have said, parsing markup with regex can be difficult and is not the best option - but there's a difference between handling wild HTML and simply removing lines from consistent well-formed XML - so this specific example is possibly ok - but if you're not 100% certain that's what you have and will always have, use a markup-aware tool instead.