LinuxQuestions.org
Download your favorite Linux distribution at LQ ISO.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Software
User Name
Password
Linux - Software This forum is for Software issues.
Having a problem installing a new program? Want to know which application is best for the job? Post your question in this forum.

Notices


Reply
  Search this Thread
Old 12-04-2014, 11:20 AM   #1
lhiggie1
LQ Newbie
 
Registered: Oct 2010
Location: Taylorsville, KY, USA
Distribution: RHEL, Fedora, CentOS, Ubuntu & HP-UX
Posts: 22

Rep: Reputation: 0
sed replace \n gives "_" in results


Good morning everyone,

I have a bash script that I have written on my Centos 6.6 server to take the results from the following command:

wbinfo --group-info "Employees" | cut -d ':' --complement -f -3 > /home/user/users_group/users.txt

which provides me a listing of the AD Employees. That is working perfectly. The problem is that when I want to take that comma delimited list and replace the commas with new lines with the following command:

sed 's/\,/\n/g' diff.txt > diff_clean.txt

The output includes an underscore after each user ID with the only exception being the last as there is no comma after the last entry:

user1_
user2_
user3_
user4

I have spent the last 2 hours googling this and have yet to find an answer as to why.

I'm currently using the following versions of bash and sed:

bash-4.1.2-15.el6_5.2.x86_64
sed-4.2.1-10.el6.x86_64

I get even a more interesting result when I try to build in removal of the "_" with the following command:

sed 's/\,/\n/g' diff.txt | sed 's/\_//g' > diff_clean.txt

The file is there, but I can no longer cat it and when I edit it with vi, I receive exactly what I want except with a bunch of ^H after every character in the list.

So any help at this time would be awesome. Thank you in advance
 
Old 12-04-2014, 11:40 AM   #2
pan64
LQ Addict
 
Registered: Mar 2012
Location: Hungary
Distribution: debian/ubuntu/suse ...
Posts: 21,523

Rep: Reputation: 7246Reputation: 7246Reputation: 7246Reputation: 7246Reputation: 7246Reputation: 7246Reputation: 7246Reputation: 7246Reputation: 7246Reputation: 7246Reputation: 7246
would be nice to see the original source file.
 
Old 12-04-2014, 12:38 PM   #3
lhiggie1
LQ Newbie
 
Registered: Oct 2010
Location: Taylorsville, KY, USA
Distribution: RHEL, Fedora, CentOS, Ubuntu & HP-UX
Posts: 22

Original Poster
Rep: Reputation: 0
Quote:
Originally Posted by pan64 View Post
would be nice to see the original source file.
Pan64,

I cannot share it as it is actual user names. Literally in the following format:

user,user1,user2,user3,user4

There are approximately 150 user names.

Sincerely,
Lee
 
Old 12-04-2014, 12:42 PM   #4
lhiggie1
LQ Newbie
 
Registered: Oct 2010
Location: Taylorsville, KY, USA
Distribution: RHEL, Fedora, CentOS, Ubuntu & HP-UX
Posts: 22

Original Poster
Rep: Reputation: 0
Quote:
Originally Posted by pan64 View Post
would be nice to see the original source file.
Pan64,

Here is the script:

#!/bin/bash
#################################################################################################### #############
# #
# This script will pull the most recent wbinfo for the rdusers group for chef to deploy to all servers. #
# #
# Created by Lee Higginbotham #
# Created on 09NOV14 #
# All rights reserved by #
# #
# This script will also generate a log file named new_user_for_chef.log' #
# This script is to be run by root #
# #
#################################################################################################### #############
# #
#################################################################################################### #############
# #
# Changing directory to the working directory. #
# #
#################################################################################################### #############
# #
echo "Changing the directory to /opt/chef/bin/users_group." >> /var/log/new_user_for_chef.log
echo $PWD >> /var/log/new_user_for_chef.log
cd /opt/chef/bin/users_group
# #
#################################################################################################### #############
# #
# Backing up the previous users.txt file for comparison and removing the diff.txt and diff_clean.txt #
# #
#################################################################################################### #############
# #
echo "------------------------------------------------------------------------------------------------" >> /var/log/new_user_for_chef.log
echo "-----------------------------------NEW USERS RUN--------------------------------------------" >> /var/log/new_user_for_chef.log
echo "------------------------------------------------------------------------------------------------" >> /var/log/new_user_for_chef.log
echo "Copying the old file to users_old.txt." >> /var/log/new_user_for_chef.log
ls -al users* >> /var/log/new_user_for_chef.log
cp users.txt users_old.txt
echo "Removing the diff.txt and diff_clean.txt and touching them to create a clean environment" >> /var/log/new_user_for_chef.log
rm -rf diff.txt
rm -rf diff_clean.txt
ls -al >> /var/log/new_user_for_chef.log
touch diff.txt
touch diff_clean.txt
ls -al >> /var/log/new_user_for_chef.log
# #
#################################################################################################### #############
# #
# Pulling in the new AD group memebership info. #
# #
#################################################################################################### #############
# #
echo "Querying AD for the updated domain rdusers group members and saving it to users.txt." >> /var/log/new_user_for_chef.log
wbinfo --group-info "Employees" | cut -d ':' --complement -f -3 | iconv -f utf-8 -t ascii//translit > /opt/chef/bin/users_group/users.txt
# #
#################################################################################################### #############
# #
# Performing the comparison of the files. #
# #
#################################################################################################### #############
# #
echo "Performing the comparison of the files." >> /var/log/new_user_for_chef.log
dwdiff -Pl -3 users.txt users_old.txt > diff.txt
echo "Here is the result of the comparison:" >> /var/log/new_user_for_chef.log
cat diff.txt >> /var/log/new_user_for_chef.log
# #
#################################################################################################### #############
# #
# Cleaning the file. #
# First I'm going to remove the two lines that contain a double equal signs lines. #
# Secondly, I'm going to remove the leading #
# and trailing commas to get the new user name #
# #
#################################################################################################### #############
# #
#iconv -f utf-8 -t ascii//translit users.txt
sed -i -e "/=/d" diff.txt
sed 's/\,/\n/g' diff.txt | sed 's/\_//g' > diff_clean.txt
#sed 's/\,/\n/g' diff.txt > diff_clean.txt
# #
#################################################################################################### #############
# #
# Here is the file after removing the commas to provide a list of new domain users. #
# #
#################################################################################################### #############
# #
cat diff_clean.txt >> /var/log/new_user_for_chef.log
echo "The new user import is ready to run in Chef." >> /var/log/new_user_for_chef.log
echo "------------------------------------------------------------------------------------------------" >> /var/log/new_user_for_chef.log
echo "-----------------------------------END USERS RUN--------------------------------------------" >> /var/log/new_user_for_chef.log
echo "------------------------------------------------------------------------------------------------" >> /var/log/new_user_for_chef.log

Sincerely,
Lee
 
Old 12-04-2014, 12:53 PM   #5
pan64
LQ Addict
 
Registered: Mar 2012
Location: Hungary
Distribution: debian/ubuntu/suse ...
Posts: 21,523

Rep: Reputation: 7246Reputation: 7246Reputation: 7246Reputation: 7246Reputation: 7246Reputation: 7246Reputation: 7246Reputation: 7246Reputation: 7246Reputation: 7246Reputation: 7246
I still think there is something uncommon (not visible chars?) in that diff.txt. But I have no idea.
probably you can try od -xc diff.txt to check the content.


# First I'm going to remove the two lines that contain a double equal signs lines. #
your sed contains single equal sign
 
Old 12-04-2014, 01:31 PM   #6
lhiggie1
LQ Newbie
 
Registered: Oct 2010
Location: Taylorsville, KY, USA
Distribution: RHEL, Fedora, CentOS, Ubuntu & HP-UX
Posts: 22

Original Poster
Rep: Reputation: 0
Quote:
Originally Posted by pan64 View Post
I still think there is something uncommon (not visible chars?) in that diff.txt. But I have no idea.
probably you can try od -xc diff.txt to check the content.


# First I'm going to remove the two lines that contain a double equal signs lines. #
your sed contains single equal sign
pan64,

Thank you for your very quick responses. It is greatly appreciated. O.k., with the od-xc on users.txt and diff.txt...users.txt looks fine. The conversion to the diff.txt is where things go wonky. Here is the first two lines of users.txt:

:0000000 646b 6168 6972 2c61 636a 7572 6573 722c
k d h a r i a , j c r u s e , r



Here is the first two lines of diff.txt:

0000000 085f 5f6b 6408 085f 5f68 6108 085f 5f72
_ \b k _ \b d _ \b h _ \b a _ \b r _

This is just too strange to me.

Sincerely,
Lee
 
Old 12-04-2014, 01:39 PM   #7
pan64
LQ Addict
 
Registered: Mar 2012
Location: Hungary
Distribution: debian/ubuntu/suse ...
Posts: 21,523

Rep: Reputation: 7246Reputation: 7246Reputation: 7246Reputation: 7246Reputation: 7246Reputation: 7246Reputation: 7246Reputation: 7246Reputation: 7246Reputation: 7246Reputation: 7246
I think that is generated by dwdiff (that is something like annotation). That is probalby caused by the -P flag, but I'm not really sure about that. You may try to use different flags to get a simple result.
 
1 members found this post helpful.
Old 12-04-2014, 03:00 PM   #8
lhiggie1
LQ Newbie
 
Registered: Oct 2010
Location: Taylorsville, KY, USA
Distribution: RHEL, Fedora, CentOS, Ubuntu & HP-UX
Posts: 22

Original Poster
Rep: Reputation: 0
Quote:
Originally Posted by pan64 View Post
I think that is generated by dwdiff (that is something like annotation). That is probalby caused by the -P flag, but I'm not really sure about that. You may try to use different flags to get a simple result.
pan64,

You are exactly correct. Now I just have to figure out either how to remove the markers that dwdiff puts in the output file or figure out another direction to get the desired result.

Thank you very much for your assistance. I will mark this as solved.

Sincerely,
Lee
 
Old 12-04-2014, 03:41 PM   #9
pan64
LQ Addict
 
Registered: Mar 2012
Location: Hungary
Distribution: debian/ubuntu/suse ...
Posts: 21,523

Rep: Reputation: 7246Reputation: 7246Reputation: 7246Reputation: 7246Reputation: 7246Reputation: 7246Reputation: 7246Reputation: 7246Reputation: 7246Reputation: 7246Reputation: 7246
if you really want to say thanks just press YES
you can try to remove those things by:
sed 's/_\x08//g', but I'm not really sure if that works properly (it is just an idea)
 
1 members found this post helpful.
  


Reply

Tags
bash sed find, centos6, sed bash


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
unable to replace "ö" to "p" in shell scripting with "sed" meninmech Programming 5 06-22-2012 03:58 PM
Sed scripting "how to replace value for only last part of a pattern " alok.rhct Linux - General 3 04-28-2010 08:10 AM
sed - use sed to replace multiple instances from "x" to "y" in a line mrodmac Linux - General 4 02-02-2010 12:37 PM
can I replace text with the result of "wc" using sed? BrianK Linux - General 1 04-21-2004 02:15 PM

LinuxQuestions.org > Forums > Linux Forums > Linux - Software

All times are GMT -5. The time now is 09:49 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration