How to decode DES encrypted string

mfoley · 11-18-2017, 04:32 PM

I've been using x11vnc for quite a while, but without any password authentication. I'm trying to add this security now. The man page says,

Quote:

-passwdfile filename
:
If filename is prefixed with "custom:" then a custom password checker is supplied as an external command following
the ":". The command will be run when a client authenticates. If the command exits with 0 the client is accepted,
otherwise it is rejected. The environment variables are set as in -accept.

The standard input to the custom command will be a decimal digit "len" followed by a newline. "len" specifies the
challenge size and is usually 16 (the VNC spec). Then follows len bytes which is the random challenge string that
was sent to the client. This is then followed by len more bytes holding the client's response (i.e. the challenge
string encrypted via DES with the user password in the standard situation).

I'm trying to figure out how this works. I ran:

Code:

# /usr/local/bin/x11vnc -auth guess -passwdfile custom:/user/util/bin/vncchxpw -repeat -modtweak

The vncchxpw file is just a bash script to capture the passed information.

I ran x11vnc as shown above and got a screen on the client asking for a password. I just entered "any". A hex dump of the /tmp/vncchxpw.log file shows:

Code:

     0: 0A 6E 65 78 74 0A 31 36 0A 23 36 F4 E1 03 EE 30    .next.16.#6....0
    10: 16 85 FC E9 4C F1 F5 16 5C 2C D5 5C 93 C2 21 29    ....L...\,.\..!)
    20: 3A DF C2 A2 7C E9 1F 1A D7                         :...|....

The word "next" is from my script. As the man page says, "the standard input to the custom command will be a decimal digit "len" followed by a newline. "len" specifies the challenge size and is usually 16 (the VNC spec)." You can see the number "16" following my "next" string. Here's where I get lost. The next 16 bytes is the random challenge string that was sent to the client." This is:

Code:

23 36 F4 E1 03 EE 30 16 85 FC E9 4C F1 F5 16 5C

The next 16 bytes is "the client's response (i.e. the challenge string encrypted via DES with the user password in the standard situation)." These bytes are:

Code:

2C D5 5C 93 C2 21 29 3A DF C2 A2 7C E9 1F 1A D7

How do I use the challenge string to decrypt the "client response" string and end up with "any" (the string I typed into the password dialog)? I've looked at using the openssl command, but I've no idea what encryption algorithm to use, salt, pass, ... how do the challenge string and client response fit here, if at all?

Turbocapitalist · 11-19-2017, 12:44 PM

DES or 3DES? Either way, openssl is probably going to work:

Code:

openssl des3 -d -in input.txt -out output.txt

The ciphers are listed in "man openssl"

mfoley · 11-19-2017, 06:35 PM

So, if the 16 byte "challenge" string is A and the response string is B, I suppose I need something like:

Code:

echo "B" | openssl DES3 -d

but what do I do with string A which I assume is the key?

Turbocapitalist · 11-19-2017, 11:36 PM

I would try feeding it to openssl using one of the many options provided by -passin.

mfoley · 11-20-2017, 10:39 AM

Not having much luck. I've tried:

Code:

challenge='#6ôá^Cî0^VüéLñõ^V\'
response=',Õ\Â!):ßÂ¢|é^_^Z×'

echo "$response" | openssl des -passin "pass:$challenge" -d

And I get, "unknown option '-passin'"

Turbocapitalist · 11-20-2017, 11:02 AM

Hmm. The manual page is wrong there.

Code:

echo "$response" | openssl des -in - -pass "env:challenge" -d

You'll have to try the DES ciphers to see if one works or find a good way of capturing the data to
be decrypted.

mfoley · 11-20-2017, 01:18 PM

Yes, I'll try different DES ciphers when I get past the needed command syntax and if plain 'ole DES doesn't work. Meanwhile, I tried your new suggested syntax and got:

Code:

# echo "$response" | openssl des -in - -pass "env:challenge" -d
-: No such file or directory
140456559851160:error:02001002:system library:fopen:No such file or directory:bss_file.c:402:fopen('-','r')
140456559851160:error:20074002:BIO routines:FILE_CTRL:system lib:bss_file.c:404:

Without the - after -in I got "unknown option 'env:challenge'".

Interesting that the man page is wrong. One would think openssl is a active enough program that this would be updated. My openssl version is 1.0.2K.