[SOLVED] Linux command to delete the last non empty line from a file

Swathi Prasad · 01-23-2018, 05:12 AM

Hi ,

Code:

$ cat  file1
PGW|PGW_CDR_|2017-06-23 141946|2017-07-17 131633|2017-08-21 
PGW|PGW_CDR_|2017-06-23 141946|2017-07-17 131633|2017-08-21 
PGW|PGW_CDR_|2017-06-23 141946|2017-07-17 131633|2017-08-21 
PGW|PGW_CDR_|2017-06-23 141946|2017-07-17 131633|2017-08-21 
PGW|PGW_CDR_|2017-06-23 141946|2017-07-17 131633|2017-08-21 
Total number of rows: 5
<<Blank line >>
<<Blank line >>
<<Blank line >>

I want to delete the last non empty line(Total number of rows: 5) from the file and copy the remaining lines to new file(file2)

File 2 should be:

Code:

$ cat  file2
PGW|PGW_CDR_|2017-06-23 141946|2017-07-17 131633|2017-08-21 
PGW|PGW_CDR_|2017-06-23 141946|2017-07-17 131633|2017-08-21 
PGW|PGW_CDR_|2017-06-23 141946|2017-07-17 131633|2017-08-21 
PGW|PGW_CDR_|2017-06-23 141946|2017-07-17 131633|2017-08-21 
PGW|PGW_CDR_|2017-06-23 141946|2017-07-17 131633|2017-08-21 
<<Blank line >>
<<Blank line >>
<<Blank line >>

I tried with the below command

Code:

sed '$d' file1>file2

But the above command instead deleting the last line and copying to the new file,like below.

Code:

$ cat  file2
PGW|PGW_CDR_|2017-06-23 141946|2017-07-17 131633|2017-08-21 
PGW|PGW_CDR_|2017-06-23 141946|2017-07-17 131633|2017-08-21 
PGW|PGW_CDR_|2017-06-23 141946|2017-07-17 131633|2017-08-21 
PGW|PGW_CDR_|2017-06-23 141946|2017-07-17 131633|2017-08-21 
PGW|PGW_CDR_|2017-06-23 141946|2017-07-17 131633|2017-08-21 
Total number of rows: 5
<<Blank line >>
<<Blank line >>

I am new to Linux and seeking help from the forum.

Thanks.

TB0ne · 01-23-2018, 07:25 AM

Quote:

Originally Posted by Swathi Prasad

Hi ,

Code:

$ cat  file1
PGW|PGW_CDR_|2017-06-23 141946|2017-07-17 131633|2017-08-21 
PGW|PGW_CDR_|2017-06-23 141946|2017-07-17 131633|2017-08-21 
PGW|PGW_CDR_|2017-06-23 141946|2017-07-17 131633|2017-08-21 
PGW|PGW_CDR_|2017-06-23 141946|2017-07-17 131633|2017-08-21 
PGW|PGW_CDR_|2017-06-23 141946|2017-07-17 131633|2017-08-21 
Total number of rows: 5

I want to delete the last non empty line(Total number of rows: 5) from the file and copy the remaining lines to new file(file2) File 2 should be:

Code:

$ cat  file2
PGW|PGW_CDR_|2017-06-23 141946|2017-07-17 131633|2017-08-21 
PGW|PGW_CDR_|2017-06-23 141946|2017-07-17 131633|2017-08-21 
PGW|PGW_CDR_|2017-06-23 141946|2017-07-17 131633|2017-08-21 
PGW|PGW_CDR_|2017-06-23 141946|2017-07-17 131633|2017-08-21 
PGW|PGW_CDR_|2017-06-23 141946|2017-07-17 131633|2017-08-21

I tried with the below command

Code:

sed '$d' file1>file2

But the above command instead deleting the last line and copying to the new file,like below.

Code:

$ cat  file2
PGW|PGW_CDR_|2017-06-23 141946|2017-07-17 131633|2017-08-21 
PGW|PGW_CDR_|2017-06-23 141946|2017-07-17 131633|2017-08-21 
PGW|PGW_CDR_|2017-06-23 141946|2017-07-17 131633|2017-08-21 
PGW|PGW_CDR_|2017-06-23 141946|2017-07-17 131633|2017-08-21 
PGW|PGW_CDR_|2017-06-23 141946|2017-07-17 131633|2017-08-21 
Total number of rows: 5
<<Blank line >>
<<Blank line >>

I am new to Linux and seeking help from the forum.

The sed command you posted only does one thing; delete the last line of a file. If you're talking about more complex operations, like examining that line to see if it's blank, and copying/appending output to a different file, you will have to write a small script to do so. Read the "Question Guidelines" link in my posting signature...there are many easily-found bash scripting tutorials that can help you get started, but we aren't going to write your scripts for you.

To give you a hint, this:

Code:

[ -z "$line" ] && continue

..is a small bash-script line that will check the variable ($line) to see if it's empty and if so, skip it.

keefaz · 01-23-2018, 07:35 AM

If the line always contains the same text, you could use grep -v to remove it from output as well

rtmistler · 01-23-2018, 07:42 AM

I think I'd like to know if it is adamant that those blank lines remain.

However brutish it is, re-use the original sed command to remove the file's last line until it is done.

Or better yet, I'm sure there's a form of sed which can first remove all blank lines.

Then use that original command to remove the last line of the file.

Quick web search Using sed to delete empty lines gives me the following suggestion:

Code:

$ sed '/^$/d' /tmp/data.txt > /tmp/output.txt

Also in that result is a command to search for and remove based on a pattern, thus if that one line with it's summary comments is desired to be removed, then find something unique in that, search for that pattern, and remove the line matching that pattern.

allend · 01-23-2018, 08:39 AM

Sometimes it helps to read last to first.

Code:

tac file1 | awk '!NF {print ""};NF {if (n) {print} else {n=1}}' | tac > file2

fatmac · 01-23-2018, 09:36 AM

Open it with vi, & use dd to delete that line, save it to file2.

rknichols · 01-23-2018, 12:24 PM

How messy it's going to be depends on whether the file contains other blank lines besides the ones at the end. If there are none, then the task is pretty straightforward with awk or perl, doable with some difficulty with sed (the sed language is, after all, Turing-complete).

If there are embedded blank lines, then processing the lines in reverse order is the only single-step solution that comes to mind. (Multi-step method: Pre-process the file to locate the trailing blank lines, then use sed to delete the specific line number that precedes them.)

syg00 · 01-23-2018, 04:06 PM

.... and of course it depends on what one considers "blank lines" ...

MadeInGermany · 01-23-2018, 04:23 PM

The following does it in one stroke

Code:

awk 'NR>1 && /./ { print buf; buf=rs=""} { buf=(buf rs $0); rs=RS } END { if (i=index(buf,RS)) print substr(buf,i+1) }' file1

The END section is only needed if trailing empty lines must be retained.
Regarding empty versus blank,
/./ means non-empty, replace it with
NF for non-blank

Swathi Prasad · 01-23-2018, 08:04 PM

It is working sir!

Could you please decode the command and explain.

Thanks in advance.

Swathi Prasad · 01-23-2018, 08:54 PM

Quote:

Originally Posted by allend

Sometimes it helps to read last to first.

Code:

tac file1 | awk '!NF {print ""};NF {if (n) {print} else {n=1}}' | tac > file2

It is working ,but could you please decode and explain.

Swathi Prasad · 01-23-2018, 08:56 PM

Quote:

Originally Posted by MadeInGermany

The following does it in one stroke

Code:

awk 'NR>1 && /./ { print buf; buf=rs=""} { buf=(buf rs $0); rs=RS } END { if (i=index(buf,RS)) print substr(buf,i+1) }' file1

The END section is only needed if trailing empty lines must be retained.
Regarding empty versus blank,
/./ means non-empty, replace it with
NF for non-blank

It is fine,could you please decode and explain

allend · 01-23-2018, 10:28 PM

Quote:

It is working ,but could you please decode and explain.

Quote:

tac file1 | awk '!NF {print ""};NF {if (n) {print} else {n=1}}' | tac > file2

The tac command is like cat, but reads in reverse, last line to first line.
The awk command sets an internal variable NF containing the number of fields when it reads a line. The ! is the NOT operator, so !NF is true when no fields are found in the line and an empty line is printed.
When an awk variable is initialised without a value, it defaults to zero, so n=0. When the first line containing fields is found, then n is set to 1 and nothing is printed. Subsequently all lines containing fields are printed.

I will let MadeInGermany explain his clever creation manipulating the record separator variable in conjunction with a buffer.

giis · 01-23-2018, 10:29 PM

You can also use grep. Just search for lines beginning (^) with required word (PGW) and redirect the output to new file.

Quote:

grep ^PGW /path/to/original.txt > new_file.txt

Swathi Prasad · 01-24-2018, 12:39 AM

Quote:

Originally Posted by allend

The tac command is like cat, but reads in reverse, last line to first line.
The awk command sets an internal variable NF containing the number of fields when it reads a line. The ! is the NOT operator, so !NF is true when no fields are found in the line and an empty line is printed.
When an awk variable is initialised without a value, it defaults to zero, so n=0. When the first line containing fields is found, then n is set to 1 and nothing is printed. Subsequently all lines containing fields are printed.

I will let MadeInGermany explain his clever creation manipulating the record separator variable in conjunction with a buffer.

What difference does it make if the code is changed to (I need to understand it as my sas code is not allowing the double codes,but the functionality if the code should not change) as below.

Code:

tac file1 | awk '!NF {print};NF {if (n) {print} else {n=1}}' | tac > file2

as per my understanding it does not make any difference,Am I correct?

Thanks in advance