LinuxQuestions.org
Share your knowledge at the LQ Wiki.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices


Reply
  Search this Thread
Old 04-04-2020, 06:54 AM   #1
296.saurabh
LQ Newbie
 
Registered: Apr 2020
Posts: 8

Rep: Reputation: Disabled
Lightbulb Inserting a new line at a specific row with a tab in bash script using sed, along with env variable


I have a part of json which looks like below:
Code:
{
  "openstack": {
    "admin": {
      "username": "admin",
      "password": "password",
      "tenant_name": "test"
    },
I want to add specific env variables at row no 3, 4, and 5 like $auth_url, $region_name and $endpoint Something like:
Code:
{
  "openstack": {
    "auth_url": $auth_url,
    "region_name": $region_name,
    "endpoint_type": $endpoint,
    "admin": {
      "username": "admin",
      "password": "password",
      "tenant_name": "test"
    },
How can I do this in bash with sed, Here $auth_url and other $ are basically env variables that are to be replaced.
Using
set -ex
sed -e '3i\\t"auth_url":$AUTH,' -i account_2.json
sed -e '4i\\t"region_name":$REGION,' -i account_2.json
sed -e '5i\\t"endpoint_type":$ENDPOINT,' -i account_2.json

It is not replacing $auth_url with env variables

Last edited by 296.saurabh; 04-04-2020 at 08:11 AM.
 
Old 04-04-2020, 07:06 AM   #2
pan64
LQ Guru
 
Registered: Mar 2012
Location: Hungary
Distribution: debian/ubuntu/suse ...
Posts: 15,971

Rep: Reputation: 5285Reputation: 5285Reputation: 5285Reputation: 5285Reputation: 5285Reputation: 5285Reputation: 5285Reputation: 5285Reputation: 5285Reputation: 5285Reputation: 5285
Please use code tags to post your script, otherwise hard to read and understand your post.
I would rather suggest you to do it in python, there is no reliable way to edit json files with sed.
From the other hand:
Code:
sed -e '3i\\t"auth_url":$AUTH,' -i account_2.json
# the quoting is not correct (that's why it does not work)
sed -e '3i\\t"auth_url":'$AUTH',' -i account_2.json
probably better

Last edited by pan64; 04-04-2020 at 07:08 AM.
 
Old 04-04-2020, 07:09 AM   #3
syg00
LQ Veteran
 
Registered: Aug 2003
Location: Australia
Distribution: Lots ...
Posts: 19,480

Rep: Reputation: 3454Reputation: 3454Reputation: 3454Reputation: 3454Reputation: 3454Reputation: 3454Reputation: 3454Reputation: 3454Reputation: 3454Reputation: 3454Reputation: 3454
Single quotes inhibit interpolation - first suggestion would be to get rid of them, certainly as outer-most quotes. Things can get really ugly with quotes and escaped characters (that tab for example). Trial and error is the best teacher.

... too slow again ...

Last edited by syg00; 04-04-2020 at 07:10 AM.
 
Old 04-04-2020, 07:15 AM   #4
Turbocapitalist
LQ Guru
 
Registered: Apr 2005
Distribution: Linux Mint, Devuan, OpenBSD
Posts: 5,441
Blog Entries: 3

Rep: Reputation: 2699Reputation: 2699Reputation: 2699Reputation: 2699Reputation: 2699Reputation: 2699Reputation: 2699Reputation: 2699Reputation: 2699Reputation: 2699Reputation: 2699
Python or perl would be the right way to do it. JSON is complex enough that it must be parsed in order to acheive correct results. Otherwise if a line or even sometimes a space is off, you'll just break the output.

However if you are writing a script in sed, and using the -i option, then you should feed the -i option a string so that a backup file is saved with that string tacked onto the end of the old name.

Code:
sed -i.orig -e '...'
Or

Code:
sed -i.bak -e '...'
Or anything. But if you leave it blank, then -i will overwrite the original file without leaving a copy of the original.
 
Old 04-04-2020, 08:07 AM   #5
296.saurabh
LQ Newbie
 
Registered: Apr 2020
Posts: 8

Original Poster
Rep: Reputation: Disabled
I was able to get it going till I dont use, (comma). once I use it to add to sed like below:
Code:
sed "3i\    \"auth_url\":$OS_AUTH_URL,\n    \"region_name\":$OS_REGION_NAME,\n    \"endpoint_type\":$OS_INTERFACE," -i "${ACCOUNTS_JSON}"
It gives an error:
sed: -e expression #1, char 45: unknown command: `,'

Last edited by 296.saurabh; 04-04-2020 at 08:35 AM.
 
Old 04-04-2020, 08:19 AM   #6
Turbocapitalist
LQ Guru
 
Registered: Apr 2005
Distribution: Linux Mint, Devuan, OpenBSD
Posts: 5,441
Blog Entries: 3

Rep: Reputation: 2699Reputation: 2699Reputation: 2699Reputation: 2699Reputation: 2699Reputation: 2699Reputation: 2699Reputation: 2699Reputation: 2699Reputation: 2699Reputation: 2699
Quotes are read in series by the shell, not from the outside. Read them from left to right. The first one starts, the next one stops, the one after that starts again, then one after that stops again, and so on. So if you want to put double quotes inside of a pair of double quotes, the inner set has to be escaped:

https://mywiki.wooledge.org/Quotes

The dollar signs will also have to be escaped when inside double quotes.

Which shell are you using and which distro is this on?
 
Old 04-04-2020, 08:30 AM   #7
296.saurabh
LQ Newbie
 
Registered: Apr 2020
Posts: 8

Original Poster
Rep: Reputation: Disabled
I am using bash and sed with versions
Code:
aq@9f766bdee2e2:~$ sed --version
sed (GNU sed) 4.4

aq@9f766bdee2e2:~$ bash --version
GNU bash, version 4.4.20(1)-release (x86_64-pc-linux-gnu)
Copyright (C) 2016 Free Software Foundation, Inc.
I mistakenly left back slash while copying the sed command, corrected now

Last edited by 296.saurabh; 04-04-2020 at 08:40 AM.
 
Old 04-04-2020, 08:39 AM   #8
Turbocapitalist
LQ Guru
 
Registered: Apr 2005
Distribution: Linux Mint, Devuan, OpenBSD
Posts: 5,441
Blog Entries: 3

Rep: Reputation: 2699Reputation: 2699Reputation: 2699Reputation: 2699Reputation: 2699Reputation: 2699Reputation: 2699Reputation: 2699Reputation: 2699Reputation: 2699Reputation: 2699
What is inside the quotes and what is outside of them?

Code:
sed "3i\    "auth_url":$OS_AUTH_URL,\n    "region_name":$OS_REGION_NAME,\n    "endpoint_type":$OS_INTERFACE," -i "${ACCOUNTS_JSON}"
The parts outside of the quotes will be parsed by the shell. Any escapes \\ or dollar signs $ will also be interpreted by the shell inside double quotes.
 
Old 04-04-2020, 08:56 AM   #9
296.saurabh
LQ Newbie
 
Registered: Apr 2020
Posts: 8

Original Poster
Rep: Reputation: Disabled
How to make this work without Error
Code:
sed "3i\    \"auth_url\":$OS_AUTH_URL,\n    \"region_name\":$OS_REGION_NAME,\n    \"endpoint_type\":$OS_INTERFACE," -i "${ACCOUNTS_JSON}"
Error:
Code:
+ sed 3i\    "auth_url":https://identity-nc,\n    "region_name":test,\n    "endpoint_type":public, -i /home/aqua/accounts_automated.json:
sed: -e expression #1, char 45: unknown command: `,'
 
Old 04-04-2020, 09:05 AM   #10
shruggy
Senior Member
 
Registered: Mar 2020
Posts: 1,799

Rep: Reputation: Disabled
How about
Code:
jq "{openstack:{auth_url:\"$OS_AUTH_URL\",region_name:\"$OS_REGION_NAME\",endpoint_type:\"$OS_INTERFACE\"}}*." <"$ACCOUNTS_JSON"

Last edited by shruggy; 04-04-2020 at 09:53 AM.
 
2 members found this post helpful.
Old 04-04-2020, 09:52 AM   #11
pan64
LQ Guru
 
Registered: Mar 2012
Location: Hungary
Distribution: debian/ubuntu/suse ...
Posts: 15,971

Rep: Reputation: 5285Reputation: 5285Reputation: 5285Reputation: 5285Reputation: 5285Reputation: 5285Reputation: 5285Reputation: 5285Reputation: 5285Reputation: 5285Reputation: 5285
probably you can get better result with (something like):
Code:
LINE="\    \"auth_url\":$OS_AUTH_URL,\n    \"region_name\":$OS_REGION_NAME,\n    \"endpoint_type\":$OS_INTERFACE,"
sed "3i$LINE" -i file
(not tested)
but as it was mentioned this is not really a task for sed. Use perl/python or jq as it was already suggested.
 
Old 04-04-2020, 09:58 AM   #12
296.saurabh
LQ Newbie
 
Registered: Apr 2020
Posts: 8

Original Poster
Rep: Reputation: Disabled
@shruggy: This works, thank you
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
[SOLVED] Using a variable in sed to pick a specific line from a text file, bash Sunvic Linux - Newbie 10 08-12-2012 12:03 PM
[SOLVED] Insert line using sed or awk at line using line number as variable sunilsagar Programming 11 02-03-2012 11:48 AM
[SOLVED] Sed - print line number along with the line ? ntpntp Linux - Newbie 6 01-30-2011 06:58 AM
AWK a variable Ouptut to a new variable and using the new variable with the old one alertroshannow Linux - Newbie 4 02-16-2009 01:08 AM
Shell script to parse csv-like output, row by row utahnix Linux - General 8 12-08-2007 06:03 AM

LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie

All times are GMT -5. The time now is 10:57 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration