[SOLVED] Is it possible to easily detect a space at the end of lines in a long list?

linustalman · 01-03-2021, 03:13 AM

Hi.

I have a large file of words (17000+ lines) and their translations. I sort them with Pluma and check the 'remove duplicates' option. However, I'm sure there's a fair few with a space at the end of the line and would be seen as not a duplicate. Is there some easy way to find them?

Example of what I mean (notice how the second line has a space at the end so these would be seen as 2 different lines in Pluma):
dog - madra
dog - madra

BTW - 'madra' is the Irish word for 'dog'. :-)

Thanks.

syg00 · 01-03-2021, 03:42 AM

Of course - simple regex wit anchors. Whether pluma offers it I don't know, but if you can (pre-)edit those files, simple enough with sed (for example) to remove any trailing spaces on any/all lines.

shruggy · 01-03-2021, 03:43 AM

To see them, cat -A|less. To trim them, sed 's/ *$//'.

BTW, I'm using this Vim plugin. It highlights trailing whitespace in red.

pan64 · 01-03-2021, 04:11 AM

Quote:

Originally Posted by shruggy

To see them, cat -A|less. To trim them, sed 's/ *$//'.

BTW, I'm using this Vim plugin. It highlights trailing whitespace in red.

you can highlight those spaces without any plugin, and also you can remove them (in vi).
But it is not really important and offtopic.

linustalman · 01-03-2021, 11:41 AM

I should have added that not all lines end in a full stop (most don't).

Examples:
dogs - madraí
Does anyone here speak Irish? - An labhraíonn éinne anseo Gaeilge?
Do it this way. - Déan an bealach seo é.

pan64 · 01-03-2021, 11:45 AM

full stop is not a white space char, so it is now irrelevant.
did you try what was suggested?

linustalman · 01-03-2021, 02:32 PM

Quote:

Originally Posted by pan64

full stop is not a white space char, so it is now irrelevant.
did you try what was suggested?

Hi pan64.

Code:

cat -A|less irish.txt

seems to just list the lines in the terminal. As did

Code:

sed 's/ *$//' irish.txt

shruggy · 01-04-2021, 12:43 AM

Quote:

Originally Posted by linustalman

Code:

cat -A|less irish.txt

This should have been

Code:

cat -A irish.txt|less

Quote:

Originally Posted by linustalman

Code:

sed 's/ *$//' irish.txt

And this should have been

Code:

sed -i.bak 's/ *$//' irish.txt

You always look up manpages of the commands you were suggested to run, don't you?

ondoho · 01-04-2021, 05:15 AM

Quote:

Originally Posted by shruggy

You always look up manpages of the commands you were suggested to run, don't you?

No, I think they prefer that you do it for them!

pan64 · 01-04-2021, 11:01 AM

interestingly

Code:

cat -A | less irish.txt

will work (at least it will show the file), but anyway this is incorrect.

Code:

sed 's/ *$//' irish.txt

will remove spaces at the end of the lines, probably there are tabs or others, so you may need to modify that regexp (if you wish to remove them too).
but return back to your original title: if you wish to detect only those lines a grep would be sufficient:

Code:

grep '  *$' irish.txt  # <<-- there are two spaces inside !!

I still don't know what is the goal here....

MadeInGermany · 01-06-2021, 11:41 AM

Trim the file with sed:

Code:

sed -i.bak 's/ *$//' irish.txt

leaves a .bak backup i.e. irish.txt.bak

Trim the file with vim:
Load the file

Code:

vim irish.txt

Visualize special characters

Code:

:set list

Trim

Code:

:%s/ *$//

Save and quit

Code:

:wq

See the similarity with the sed command!
The % means "all lines"; this is default in sed.

linustalman · 01-12-2021, 02:41 AM

Quote:

Originally Posted by MadeInGermany

Trim the file with sed:

Code:

sed -i.bak 's/ *$//' irish.txt

leaves a .bak backup i.e. irish.txt.bak

Trim the file with vim:
Load the file

Code:

vim irish.txt

Visualize special characters

Code:

:set list

Trim

Code:

:%s/ *$//

Save and quit

Code:

:wq

See the similarity with the sed command!
The % means "all lines"; this is default in sed.

Thank you, MIG. Your sed command did exactly what I wanted. 👍🏻☺️

linustalman · 01-12-2021, 07:14 AM

If anyone was unsure what I meant. Here's an example. [see attached image]

linustalman · 01-13-2021, 12:09 PM

Quote:

Originally Posted by ondoho

No, I think they prefer that you do it for them!

That's your contribution? 😏

ondoho · 01-13-2021, 01:28 PM

Quote:

Originally Posted by linustalman

That's your contribution? 😏

Yep.
Ain't I right though.
As your quoted post proves, again.