LinuxQuestions.org

LinuxQuestions.org (/questions/)
-   Programming (https://www.linuxquestions.org/questions/programming-9/)
-   -   add literal string with simple and double quotes to a file (https://www.linuxquestions.org/questions/programming-9/add-literal-string-with-simple-and-double-quotes-to-a-file-4175675574/)

vincix 05-20-2020 04:48 AM

add literal string with simple and double quotes to a file
 
I'm trying to add the following string to a file:
Code:

sed 's|\$MYSQL_PASSWORD|'"$MYSQL_PASSWORD"'|'
I'm using the simple and double quote syntax so that sed would use the MYSQL_PASSWORD variable (in the second part; the first part is literally $MYSQL_PASSWORD) when sed is run in the file I'm trying to add the line to.

What I've tried to far is:
Code:

echo sed 's|\$MYSQL_PASSWORD|'"\$MYSQL_PASSWORD"'|' > file
But unfortunately it drops the quote altogether:
Code:

sed s|\$MYSQL_PASSWORD|$MYSQL_PASSWORD|
Somewhat closer is:
Code:

echo sed 's|\$MYSQL_PASSWORD|'"'"\$MYSQL_PASSWORD"'"'|'
But this drops the double quotes:
Code:

sed s|\$MYSQL_PASSWORD|'$MYSQL_PASSWORD'|
This is becoming a little bit retarded, so if there are more elegant solutions, I'm looking forward to it :)

Ok, I was actually able to find a horrible solution:
Code:

echo sed \''s|\$MYSQL_PASSWORD|'"'\""\$MYSQL_PASSWORD"'"\"'|'\'
But the question remains: is there any other way to achieve this?
Thanks!

pan64 05-20-2020 05:06 AM

that can be quite tricky. I don't know how did you check if the result is ok
Code:

TEXT="sed 's|\$MYSQL_PASSWORD|${MYSQL_PASSWORD}|'"
echo ">${TEXT}<"

would be probably the first step.
If that works you can safely redirect it into file too.

vincix 05-20-2020 05:21 AM

Actually I made a mistake in the final result. It should have been:
Code:

echo sed \''s|\$MYSQL_PASSWORD|'"'\""\$MYSQL_PASSWORD\""'"'|'\'
I had misplaced the \ " (if I write \ " together, then the backslash is removed from the post, because it's interpreted as an escape character! :D)
I tested it by simply redirecting it into the file and it worked as expected:
Code:

root@oms:~# echo sed \''s|\$MYSQL_PASSWORD|'"'\""\$MYSQL_PASSWORD\""'"'|'\' > file
root@oms:~# cat file
sed 's|\$MYSQL_PASSWORD|'"$MYSQL_PASSWORD"'|'


The problem with this solution
Code:

TEXT="sed 's|\$MYSQL_PASSWORD|${MYSQL_PASSWORD}|'"
is that sed interprets the ${MYSQL_PASSWORD} literally, because of the simple quotes. This is why I was trying to use '"$MYSQL_PASSWORD"' (so between simple + double quotes; so I'm escaping the bash variable that way and sed will replace it with the value of the variable, instead of the literal name).

Can you explain what the meaning of the angle brackets in echo ">${TEXT}<" is?

shruggy 05-20-2020 05:37 AM

Your first version should work with a HERE document:
Code:

cat <<EOF >file
sed 's|\$MYSQL_PASSWORD|'"\$MYSQL_PASSWORD"'|'
EOF


ntubski 05-20-2020 05:57 AM

Quote:

Originally Posted by shruggy (Post 6125269)
Your first version should work with a HERE document:
Code:

cat <<EOF >file
sed 's|\$MYSQL_PASSWORD|'"\$MYSQL_PASSWORD"'|'
EOF


You need to use
Code:

cat <<'EOF' > file
sed 's|\$MYSQL_PASSWORD|'"\$MYSQL_PASSWORD"'|'
EOF

Note the quotes around EOF.

https://www.gnu.org/software/bash/ma...Here-Documents
Quote:

If any part of word is quoted, the delimiter is the result of quote removal on word, and the lines in the here-document are not expanded. If word is unquoted, all lines of the here-document are subjected to parameter expansion

shruggy 05-20-2020 06:09 AM

^ With quotes around EOF you don't need to escape $ in the replacement string:
Code:

cat <<'EOF' > file
sed 's|\$MYSQL_PASSWORD|'"$MYSQL_PASSWORD"'|'
EOF


pan64 05-20-2020 06:13 AM

So your problem is that the password itself contains something which can be interpreted by sed.
You cannot protect it that way there is no fool-proof solution. You need to process the password string itself to escape all the suspicious chars.

> and < are just markers to see what is "inside". Nothing else.

shruggy 05-20-2020 06:49 AM

I assume the file you're writing your sed expression to is a shell script. A reasonably recent bash should have a nice facility to cope with the problem pan64 describes in form of ${var@Q}. From the Bash Reference Manual:
Quote:

${parameter@operator}
The expansion is either a transformation of the value of parameter or information about parameter itself, depending on the value of operator. Each operator is a single letter:

Q
The expansion is a string that is the value of parameter quoted in a format that can be reused as input.


pan64 05-20-2020 09:23 AM

Quote:

Originally Posted by shruggy (Post 6125289)
I assume the file you're writing your sed expression to is a shell script. A reasonably recent bash should have a nice facility to cope with the problem pan64 describes in form of ${var@Q}. From the Bash Reference Manual:

It is something new, somewhere bash 4.4 was implemented.

vincix 05-24-2020 07:04 AM

I wanted to restrict the thread only to this particular problem without giving the bigger context, but now the context has become much more relevant.
The point was to do something somewhat intentionally dumb, that is to say, to change an existing docker-entrypoint from an official docker image (of php-apache) through an additional Dockerfile, where I'd run the sed commands.
The here document would have been a great solution under normal circumstances, and it hadn't occurred to me anyway, but unfortunately it doesn't work in Dockerfile :)

What I did eventually (which I find much more sensible) was to actually copy only the entrypoint from github, add the sed lines that would replace the php configuration settings, and then copy it into the new image, ensuring, of course, that it has execution permissions (being an entrypoint).

@pan64
The issue with this: TEXT="sed 's|\$MYSQL_PASSWORD|${MYSQL_PASSWORD}|'" is that sed is not going to interpret the bash variable as it is, regardless of what the variable contains. This is why it needs to be escaped with '"${MYSQL_PASSWORD}"'. What the actual value of the variable is, is indeed a problem in itself and needs to be dealt with carefully, I agree, sure. That's why I chose a longer string (length of 40-50 characters) but with standard characters, as it were.


Anyway, thanks for the help, I've come away with lots of interesting information! :)

pan64 05-24-2020 07:57 AM

sed will never be able to use bash variables, that is a different language.
https://stackoverflow.com/questions/...itution-in-sed
If you wish to do that you need to use perl/awk/python/... something, which can handle environment variables properly.

vincix 05-24-2020 08:00 AM

Ok, so what about this?
Code:

root@kube:~# var=crap
root@kube:~# echo "stuff" > file.txt
root@kube:~# sed 's/stuff/'"$var"'/' file.txt
crap


pan64 05-24-2020 08:14 AM

In that case the shell will evaluate $var and will pass the result to sed. As always, the command evaluated first (before execution). You can check it by executing set -x (before executing your sed)
Code:

$ set -x
$ var=cap
+ var=cap
$ echo "stuff" > file.txt
+ echo stuff
$ sed 's/stuff/'"$var"'/' file.txt
+ sed s/stuff/cap/ file.txt
cap

black was entered
blue was actually executed (after evaluation)
green is the result

Also [re]read the link I posted.

vincix 05-24-2020 08:17 AM

Ok, I see what you mean (I also hadn't read the link entirely before answering), but that works for all intents and purposes. And that's the most important thing, so that's more of a theoretical problem (yes, not necessarily, because it depends on what the variable contains, but I've already cleared that up). There's no point in switching to another tool when doing such substitutions - they are being used widely.

pan64 05-24-2020 08:22 AM

the problem is when the variable contains something which can be interpreted by sed. And (if the content of the variable is unpredictable) this substitution is just unreliable.

vincix 05-24-2020 08:31 AM

Well, awk doesn't seem to be any smarter, to be honest. Or am I mistaken?

Code:

root@kube:~# cat file.txt
$
root@kube:~# echo $var
$
root@kube:~# echo $var2
crap2
root@kube:~# awk -v var="${var}" -v var2="${var2}" '{sub(var, var2)}1' file.txt
$crap2

So instead of replacing $ with crap2, it just suffixes, for some reason.
Of course, a normal string substitution works fine:
Code:

root@kube:~# cat file.txt
whatever
root@kube:~# var=whatever
root@kube:~# echo $var2
crap2
root@kube:~# awk -v var="${var}" -v var2="${var2}" '{sub(var, var2)}1' file.txt
crap2


shruggy 05-24-2020 08:59 AM

Quote:

Originally Posted by vincix (Post 6126680)
So instead of replacing $ with crap2, it just suffixes, for some reason.

For quite an obvious reason, actually. The first argument to sub/gsub is considered to be a regular expression. Putting it in a variable changes nothing in this respect.

pan64 05-24-2020 09:03 AM

you may need to export variables, otherwise (as in your example) that will be again evaluated by the shell.
sub (in awk) works on regexp, that's why it replaced the end of the line (when var=$)
If you wish to use simple substitution/replacement you need to find another solution.
Code:

awk -v a="${var}" -v b="${var2}" {
  while(i=index($0,a))
    $0=substr($0,1,i-1) b substr($0,i+length(a))
  print $0
}


vincix 05-24-2020 09:06 AM

Right, that's correct, this is unrelated to any interpretation of the bash variables.
What about this? Now I'm curious, while we're at it:
Code:

root@kube:~# echo $var
$!:.%^&
root@kube:~# echo $var2
crap2
root@kube:~# awk -v var="${var}" -v var2="${var2}" '{sub(var2, var)}1' file.txt
$!:.%^crap2

Why is awk interpreting & that way? I guess it's like in sed, & means the whole matched regex, right?

EDIT: I've posted it without seeing pan64's latest reply.

pan64 05-24-2020 09:13 AM

which awk is it exactly? what is in file.txt?

vincix 05-24-2020 09:18 AM

It's gawk (comes with most Linux distros - here running under Ubuntu 18.04)
In file.txt there's only "crap2".
I couldn't get your command to work, I'm not sure why. I get this:
Code:

awk: cmd. line:1: {
awk: cmd. line:1:  ^ unexpected newline or end of string
awk.sh: line 2: syntax error near unexpected token `('
awk.sh: line 2: `  while(i=index($0,a))'


shruggy 05-24-2020 09:23 AM

Quote:

Originally Posted by vincix (Post 6126690)
Why is awk interpreting & that way? I guess it's like in sed, & means the whole matched regex, right?

Exactly. There's a pretty long treatment of it in the Gawk Manual.

shruggy 05-24-2020 09:24 AM

Quote:

Originally Posted by vincix (Post 6126698)
I couldn't get your command to work, I'm not sure why.

The whole awk expression should be quoted:
Code:

awk -v a="${var}" -v b="${var2}" '{
  while(i=index($0,a))
    $0=substr($0,1,i-1) b substr($0,i+length(a))
  print $0
}'


vincix 05-24-2020 09:28 AM

It doesn't work either way :) Adding double quotes shouldn't work because they are already being used, and it might screw up the interpretation. Single quotes don't work anyway. I placed the whole command in a file, and it's not working.

pan64 05-24-2020 09:35 AM

Quote:

Originally Posted by vincix (Post 6126703)
It doesn't work either way :) Adding double quotes shouldn't work because they are already being used, and it might screw up the interpretation. Single quotes don't work anyway. I placed the whole command in a file, and it's not working.

so check it again. The last post from shruggy looks correct, I missed the two ' .

shruggy 05-24-2020 09:40 AM

I've just tried it out, it works:
Code:

$ cat file.txt
crap2
$
awk -v b='$!:.%^&' -v a='crap2' -f pan64.awk file.txt
$!:.%^&


vincix 05-24-2020 09:45 AM

Yes, I missed that. It does work as expected, indeed!
Code:

root@kube:~# cat file.txt
crap2
root@kube:~# echo $var
crap2
root@kube:~# echo $var2
$!:.%^&
awk -v a="${var}" -v b="${var2}" '{
  while(i=index($0,a))
    $0=substr($0,1,i-1) b substr($0,i+length(a))
  print $0
}' file.txt
$!:.%^&

So that would be a completely literal substitution, and I wouldn't care about what characters are found in the variable.

But if I wanted to use regex for matching the pattern and then substitute it with a literal string, like the sub command does, then all of a sudden I have a problem, in that I yet again have to be careful about what the variable contains, right?

shruggy 05-24-2020 09:54 AM

Yes, and you should be aware of how regex in a string differs from a regex literal between slashes.

pan64 05-24-2020 10:01 AM

and additionally you may also kill the awk process (with an invalid regexp).

vincix 05-24-2020 10:02 AM

Right, but then I return to the initial problem related to sed and the interpretation of bash variables and I see that awk is susceptible to very similar issues, regardless of when or by what the variable is converted into its value.

Maybe perl would do the job, but I don't see lots of people writing perl substitutions in their docker entrypoints, to be honest. Installing python to create entrypoint containers could be ok, depending on the situation, but normally you don't want your image to grow by hundreds of mb just for a substitution.

ntubski 05-24-2020 06:50 PM

Quote:

Originally Posted by vincix (Post 6126713)
But if I wanted to use regex for matching the pattern and then substitute it with a literal string, like the sub command does, then all of a sudden I have a problem, in that I yet again have to be careful about what the variable contains, right?

Not necessarily:

Code:

awk -v a="${var}" -v b="${var2}" '{
  while(i=match($0,a))
    $0=substr($0,1,i-1) b substr($0,i+RLENGTH)
  print $0
}' file.txt


boughtonp 05-25-2020 05:31 AM

Quote:

Originally Posted by vincix (Post 6126640)
I wanted to restrict the thread only to this particular problem without giving the bigger context, but now the context has become much more relevant.

It usually does, and should always be mentioned so people who know the answer to your real problem can go straight to providing the simpler solution that everyone else is already using, instead of either reinventing the wheel or using an ugly hack.


Quote:

The point was to do something somewhat intentionally dumb, that is to say, to change an existing docker-entrypoint from an official docker image (of php-apache) through an additional Dockerfile, where I'd run the sed commands.
The here document would have been a great solution under normal circumstances, and it hadn't occurred to me anyway, but unfortunately it doesn't work in Dockerfile :)

What I did eventually (which I find much more sensible) was to actually copy only the entrypoint from github, add the sed lines that would replace the php configuration settings, and then copy it into the new image, ensuring, of course, that it has execution permissions (being an entrypoint).
I haven't needed to use Docker yet, so that doesn't make complete sense to me, but it sounds like you want to change the password in a config file, which seems a sufficiently common problem to have a common solution.

How does everyone else using Docker solve it?

What are you doing differently that stops you using their solution?


vincix 05-25-2020 12:33 PM

Well, if you're not familiar with docker it's a little bit harder to explain, I guess, but the solution I've chosen, given that I don't have other orchestration tools (like swarm or kubernetes, which offer so called config files that can be dynamically mounted into the container and being used as such even after you've built the containers), should be the right one in this context. It's just a sed replacement based on environmental variables - this is rather common.

What was weird about what I was trying to do (I'd have stuck with it only if the solution had turned out to be much easier, or maybe not even then), was changing an existing entrypoint (which is just a bash script running every time the container starts), which had already been built in the container image, by adding a few sed lines before the last line. Yeah, I don't think people do that. So that's why I simply copied the whole entrypoint, adjusted to my liking, and made it part of the build (using a Dockerfile of my own - meaning adding another container layer in the image).

I hope it made a little bit of sense :)


As far as the context is concerned, sure, I agree 100%, I just thought this time I could get away with it :D

@ntubski, thanks for the answer!

pan64 05-26-2020 01:36 AM

Quote:

Originally Posted by vincix (Post 6127178)
Well, if you're not familiar with docker it's a little bit harder to explain, I guess, but the solution I've chosen, given that I don't have other orchestration tools (like swarm or kubernetes, which offer so called config files that can be dynamically mounted into the container and being used as such even after you've built the containers), should be the right one in this context. It's just a sed replacement based on environmental variables - this is rather common.

That can be made with docker too. You can "mount" even a simple file into the docker image. Also you can pass environment variables to the docker image. But obviously it all depends on the details....


All times are GMT -5. The time now is 08:26 AM.