LinuxQuestions.org
Latest LQ Deal: Linux Power User Bundle
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices


Reply
  Search this Thread
Old 11-04-2014, 04:29 PM   #1
soph75
LQ Newbie
 
Registered: Jun 2011
Posts: 4

Rep: Reputation: Disabled
Join Lines between spaces


Hello, I would like join lines horizontally based on blocks below. can someone please provide a command on how to do this? the file size is about 1gb.

Current log file
================

g-Entry-Offset/Size: 0/0001010^M
Content-ID: 11633/48425_2255560634 HU^M
MailHost: alum.mit.edu(18.7.68.15)^M
Addressee: aetallian@alum.mit.edu^M
Status-Attempt-Session-Error: 1:1:ac7:250^M
Date: Wed Oct 29 11:39:21 EDT 2014^M
At 2014-10-29 11:39:21.432: 220 alum-mailsec-scanner-4.mit.edu ESMTP Symantec Messaging Gateway^M
At 2014-10-29 11:39:21.432: Sent: EHLO dc1-lcmeprd01.worldwidevacationtravel.com^M

At 2014-10-29 11:39:21.447: 250-alum-mailsec-scanner-4.mit.edu says EHLO to 12.42.131.15:63146^M
250-PIPELINING^M
250-SIZE 31457280^M
250-8BITMIME^M
250-ENHANCEDSTATUSCODES^M
250 STARTTLS^M
At 2014-10-29 11:39:21.467: Sent: MAIL FROM:<news@go.intervalworld.com>^M
RCPT TO:<aetallian@alum.mit.edu>^M
DATA^M

At 2014-10-29 11:39:21.482: 250 2.0.0 MAIL FROM accepted^M
At 2014-10-29 11:39:21.537: 250 2.0.0 RCPT TO accepted^M
At 2014-10-29 11:39:21.537: 354 3.0.0 continue. finished with "\r\n.\r\n"^M
At 2014-10-29 11:39:21.582: Sent 28612 bytes of data.
At 2014-10-29 11:39:21.614: 250 2.0.0 OK 2A/EB-20338-E1A01545^M
Log-Entry-Offset/Size: 1010/0000646^M
Content-ID: 11635/48425_2157584815 HU^M
MailHost: alum.mit.edu(0.0.0.0)^M
Addressee: batespkjr@alum.mit.edu^M
Status-Attempt-Session-Error: 1:1:ac7:250^M
Date: Wed Oct 29 11:39:21 EDT 2014^M
At 2014-10-29 11:39:21.630: Sent: MAIL FROM:<news@go.intervalworld.com>^M
RCPT TO:<batespkjr@alum.mit.edu>^M
DATA^M

At 2014-10-29 11:39:21.646: 250 2.0.0 MAIL FROM accepted^M
At 2014-10-29 11:39:21.733: 250 2.0.0 RCPT TO accepted^M
At 2014-10-29 11:39:21.733: 354 3.0.0 continue. finished with "\r\n.\r\n"^M
At 2014-10-29 11:39:21.733: Sent 24794 bytes of data.
At 2014-10-29 11:39:21.765: 250 2.0.0 OK 3A/EB-20338-E1A01545^M
Log-Entry-Offset/Size: 1656/0000646^M
Content-ID: 11636/48425_2252191673 HU^M
MailHost: alum.mit.edu(0.0.0.0)^M
Addressee: belinda94@alum.mit.edu^M
Status-Attempt-Session-Error: 1:1:ac7:250^M
Date: Wed Oct 29 11:39:21 EDT 2014^M
At 2014-10-29 11:39:21.775: Sent: MAIL FROM:<news@go.intervalworld.com>^M
RCPT TO:<belinda94@alum.mit.edu>^M
DATA^M




Thanks,
Soph
 
Old 11-04-2014, 07:12 PM   #2
grail
LQ Guru
 
Registered: Sep 2009
Location: Perth
Distribution: Manjaro
Posts: 9,506

Rep: Reputation: 2890Reputation: 2890Reputation: 2890Reputation: 2890Reputation: 2890Reputation: 2890Reputation: 2890Reputation: 2890Reputation: 2890Reputation: 2890Reputation: 2890
A couple of things:

1. please use [code][/code] tags around code or data

2. please provide what the output would like based on your example

3. as the file was created in Windows you would first want to run something like dos2unix over the file. This will remove the Windows based line endings (^M) which many linux utils have issue with
 
Old 11-05-2014, 11:49 AM   #3
soph75
LQ Newbie
 
Registered: Jun 2011
Posts: 4

Original Poster
Rep: Reputation: Disabled
Okay, inserted the code tags in the data. let me know if I'm missing anything.

Code:
 
g-Entry-Offset/Size: 0/0001010
Content-ID: 11633/48425_2255560634 HU
MailHost: alum.mit.edu(18.7.68.15)
Addressee: aetallian@alum.mit.edu
Status-Attempt-Session-Error: 1:1:ac7:250
Date: Wed Oct 29 11:39:21 EDT 2014
At 2014-10-29 11:39:21.432: 220 alum-mailsec-scanner-4.mit.edu ESMTP Symantec Messaging Gateway
At 2014-10-29 11:39:21.432: Sent: EHLO dc1-lcmeprd01.worldwidevacationtravel.com
Code:
At 2014-10-29 11:39:21.447: 250-alum-mailsec-scanner-4.mit.edu says EHLO to 12.42.131.15:63146
250-PIPELINING
250-SIZE 31457280
250-8BITMIME
250-ENHANCEDSTATUSCODES
250 STARTTLS
At 2014-10-29 11:39:21.467: Sent: MAIL FROM:<news@go.intervalworld.com>
RCPT TO:<aetallian@alum.mit.edu>
DATA
Code:
At 2014-10-29 11:39:21.482: 250 2.0.0 MAIL FROM accepted
At 2014-10-29 11:39:21.537: 250 2.0.0 RCPT TO accepted
At 2014-10-29 11:39:21.537: 354 3.0.0 continue. finished with "\r\n.\r\n"
At 2014-10-29 11:39:21.582: Sent 28612 bytes of data.
At 2014-10-29 11:39:21.614: 250 2.0.0 OK 2A/EB-20338-E1A01545
Log-Entry-Offset/Size: 1010/0000646
Content-ID: 11635/48425_2157584815 HU
MailHost: alum.mit.edu(0.0.0.0)
Addressee: batespkjr@alum.mit.edu
Status-Attempt-Session-Error: 1:1:ac7:250
Date: Wed Oct 29 11:39:21 EDT 2014
At 2014-10-29 11:39:21.630: Sent: MAIL FROM:<news@go.intervalworld.com>
RCPT TO:<batespkjr@alum.mit.edu>
DATA
Code:
At 2014-10-29 11:39:21.646: 250 2.0.0 MAIL FROM accepted
At 2014-10-29 11:39:21.733: 250 2.0.0 RCPT TO accepted
At 2014-10-29 11:39:21.733: 354 3.0.0 continue. finished with "\r\n.\r\n"
At 2014-10-29 11:39:21.733: Sent 24794 bytes of data.
At 2014-10-29 11:39:21.765: 250 2.0.0 OK 3A/EB-20338-E1A01545
Log-Entry-Offset/Size: 1656/0000646
Content-ID: 11636/48425_2252191673 HU
MailHost: alum.mit.edu(0.0.0.0)
Addressee: belinda94@alum.mit.edu
Status-Attempt-Session-Error: 1:1:ac7:250
Date: Wed Oct 29 11:39:21 EDT 2014
At 2014-10-29 11:39:21.775: Sent: MAIL FROM:<news@go.intervalworld.com>
RCPT TO:<belinda94@alum.mit.edu>
DATA

Desired Output example:

At 2014-10-29 11:39:21.646: 250 2.0.0 MAIL FROM accepted At 2014-10-29 11:39:21.733: 250 2.0.0 RCPT TO accepted At 2014-10-29 11:39:21.733: 354 3.0.0 continue. finished with "\r\n.\r\n" At 2014-10-29 11:39:21.733: Sent 24794 bytes of data. At 2014-10-29 11:39:21.765: 250 2.0.0 OK 3A/EB-20338-E1A01545 Log-Entry-Offset/Size: 1656/0000646 Content-ID: 11636/48425_2252191673 HU MailHost: alum.mit.edu(0.0.0.0) Addressee: belinda94@alum.mit.edu Status-Attempt-Session-Error: 1:1:ac7:250 Date: Wed Oct 29 11:39:21 EDT 2014 At 2014-10-29 11:39:21.775: Sent: MAIL FROM:<news@go.intervalworld.com> RCPT TO:<belinda94@alum.mit.edu> DATA

Needs to be in 1 line per stack.


Thanks,
Soph
 
Old 11-05-2014, 07:07 PM   #4
grail
LQ Guru
 
Registered: Sep 2009
Location: Perth
Distribution: Manjaro
Posts: 9,506

Rep: Reputation: 2890Reputation: 2890Reputation: 2890Reputation: 2890Reputation: 2890Reputation: 2890Reputation: 2890Reputation: 2890Reputation: 2890Reputation: 2890Reputation: 2890
Ok ... thanks for using the code tags. I had meant that you just place one at the start and one at the end so we could see the data in total

Also, you should also place your output in code tags, as again it makes it clearer (btw. you can edit your current post, you do not need to redo an entire new one to do this)


So as to the solution I would need to confirm at least 2 things:

1. the file has been parsed by dos2unix to remove unwanted Windows line endings

2. the chunks currently displayed are all in a single file (I assume), so what delimits one chunk from the next? (ie. is it 1 or more blank lines ... or some other delimiter?)
 
Old 11-05-2014, 07:31 PM   #5
NoStressHQ
Member
 
Registered: Apr 2010
Location: Geneva - Switzerland ( Bordeaux - France / Montreal - QC - Canada)
Distribution: Slackware 14.2 - 32/64bit
Posts: 558

Rep: Reputation: 187Reputation: 187
Using BASH you can:

Code:
$ while IFS= read line ; do
  [[ -n "$line" ]] && printf '%s' "$line" || echo
done <"yourfile"
Cheers

Garry

PS/ As grail stated, if it's a windows generated log, and need to "clean" the line endings, just use this instead:

Code:
$ <"yourfile" dos2unix | while IFS= read line ; do
  [[ -n "$line" ]] && printf '%s' "$line" || echo
done

Last edited by NoStressHQ; 11-05-2014 at 07:47 PM. Reason: fixed wrong redirection... :)
 
Old 11-06-2014, 11:21 AM   #6
soph75
LQ Newbie
 
Registered: Jun 2011
Posts: 4

Original Poster
Rep: Reputation: Disabled
Hi thanks for The code but it's not working in bash mode


-bash-3.2$ $ while IFS= read line ; do [[ -n "$line" ]] && printf '%s' "$line" || echo done <"test.joined.log"
-bash: $: command not found
done

Then removed $ sign

-bash-3.2$ while IFS= read line ; do [[ -n "$line" ]] && printf '%s' "$line" || echo done <"test.joined.log"
>

What is this mean >
 
Old 11-06-2014, 06:52 PM   #7
grail
LQ Guru
 
Registered: Sep 2009
Location: Perth
Distribution: Manjaro
Posts: 9,506

Rep: Reputation: 2890Reputation: 2890Reputation: 2890Reputation: 2890Reputation: 2890Reputation: 2890Reputation: 2890Reputation: 2890Reputation: 2890Reputation: 2890Reputation: 2890
If you are going to run at the command prompt you need to insert a semi-colon prior to the word done, other wise the word done is passed to echo. This then means there is no end to your while loop,
hence you receive the line continuation symbol (>) which is asking for you to complete the while loop.

So it should look like:
Code:
while IFS= read line ; do [[ -n "$line" ]] && printf '%s' "$line" || echo ;done <"test.joined.log"
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
[SOLVED] Join with unpairable lines danielbmartin Programming 4 08-05-2012 01:45 PM
Cutting fields from lines with multiple spaces 915086731 Linux - Software 6 09-07-2011 09:51 AM
join lines s_linux Programming 5 04-11-2011 10:00 AM
[SOLVED] Removing lines and spaces - Formatting Text vikas027 Programming 8 10-10-2008 03:06 PM
Join all lines using sed chipix Programming 3 04-03-2007 09:55 AM


All times are GMT -5. The time now is 02:48 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration