Probem Ipsec - racoon roadwarrior client - main mode - hybrid_rsa authentication
 

My client is on Ubuntu Lucid 10.04, I installed ipsec-tools and racoon from the repositories. The gateway is installed on a CentOS machine.

I've configured everything to get a working roadwarrior configuration with authentication_method hybrid_rsa client and server. It's working in aggressive mode, but in main mode I can't get it working. I delivered new CA and certificates several times but I'm still stuck.

It seems that it comes from my client not supporting the certificate sent by the server.
The client contains a copy of the CA, whereas server has a private key and a certificate signed by the CA.


on the client I get the following odd message during connection (both in aggressive and main mode) :
2011-06-28 09:30:15: DEBUG: peer transmitted CR: X.509 Certificate Signature
2011-06-28 09:30:15: ERROR: such a cert type isn't supported: 4

On the server I have the following :
2011-06-28 11:04:38: INFO: received Vendor ID: DPD
2011-06-28 11:04:38: INFO: Selected NAT-T version: RFC 3947
2011-06-28 11:04:38: DEBUG: total SA len=52
2011-06-28 11:04:38: DEBUG:
00000001 00000001 0000002c 01010001 00000024 01010000 800b0001 800c7080
80010007 800e0080 8003fadd 80020001 80040002
2011-06-28 11:04:38: DEBUG: begin.
2011-06-28 11:04:38: DEBUG: seen nptype=2(prop)
2011-06-28 11:04:38: DEBUG: succeed.
2011-06-28 11:04:38: DEBUG: proposal #1 len=44
2011-06-28 11:04:38: DEBUG: begin.
2011-06-28 11:04:38: DEBUG: seen nptype=3(trns)
2011-06-28 11:04:38: DEBUG: succeed.
2011-06-28 11:04:38: DEBUG: transform #1 len=36
2011-06-28 11:04:38: DEBUG: type=Life Type, flag=0x8000, lorv=seconds
2011-06-28 11:04:38: DEBUG: type=Life Duration, flag=0x8000, lorv=28800
2011-06-28 11:04:38: DEBUG: type=Encryption Algorithm, flag=0x8000, lorv=7
2011-06-28 11:04:38: DEBUG: encryption(aes)
2011-06-28 11:04:38: DEBUG: type=Key Length, flag=0x8000, lorv=128
2011-06-28 11:04:38: DEBUG: type=Authentication Method, flag=0x8000, lorv=Hybrid RSA server
2011-06-28 11:04:38: DEBUG: type=Hash Algorithm, flag=0x8000, lorv=MD5
2011-06-28 11:04:38: DEBUG: hash(md5)
2011-06-28 11:04:38: DEBUG: type=Group Description, flag=0x8000, lorv=1024-bit MODP group
2011-06-28 11:04:38: DEBUG: hmac(modp1024)
2011-06-28 11:04:38: DEBUG: pair 1:
2011-06-28 11:04:38: DEBUG: 0x2b69133f5380: next=(nil) tnext=(nil)
2011-06-28 11:04:38: DEBUG: proposal #1: 1 transform
2011-06-28 11:04:38: DEBUG: prop#=1, prot-id=ISAKMP, spi-size=0, #trns=1
2011-06-28 11:04:38: DEBUG: trns#=1, trns-id=IKE
2011-06-28 11:04:38: DEBUG: type=Life Type, flag=0x8000, lorv=seconds
2011-06-28 11:04:38: DEBUG: type=Life Duration, flag=0x8000, lorv=28800
2011-06-28 11:04:38: DEBUG: type=Encryption Algorithm, flag=0x8000, lorv=7
2011-06-28 11:04:38: DEBUG: type=Key Length, flag=0x8000, lorv=128
2011-06-28 11:04:38: DEBUG: type=Authentication Method, flag=0x8000, lorv=Hybrid RSA server
2011-06-28 11:04:38: DEBUG: type=Hash Algorithm, flag=0x8000, lorv=MD5
2011-06-28 11:04:38: DEBUG: type=Group Description, flag=0x8000, lorv=1024-bit MODP group
2011-06-28 11:04:38: DEBUG: Compared: DB:Peer
2011-06-28 11:04:38: DEBUG: (lifetime = 28800:28800)
2011-06-28 11:04:38: DEBUG: (lifebyte = 0:0)
2011-06-28 11:04:38: DEBUG: enctype = 7:7
2011-06-28 11:04:38: DEBUG: (encklen = 128:128)
2011-06-28 11:04:38: DEBUG: hashtype = MD5:MD5
2011-06-28 11:04:38: DEBUG: authmethod = Hybrid RSA server:Hybrid RSA server
2011-06-28 11:04:38: DEBUG: dh_group = 1024-bit MODP group:1024-bit MODP group
2011-06-28 11:04:38: DEBUG: an acceptable proposal found.
2011-06-28 11:04:38: DEBUG: hmac(modp1024)
2011-06-28 11:04:38: DEBUG: new cookie:
cf694b856ad13bd9
2011-06-28 11:04:38: DEBUG: add payload of len 52, next type 13
2011-06-28 11:04:38: DEBUG: add payload of len 16, next type 13
2011-06-28 11:04:38: DEBUG: add payload of len 16, next type 0
2011-06-28 11:04:38: DEBUG: 124 bytes from x.x.x.15[500] to x.x.x.200[500]
2011-06-28 11:04:38: DEBUG: sockname x.x.x.15[500]
2011-06-28 11:04:38: DEBUG: send packet from x.x.x.15[500]
2011-06-28 11:04:38: DEBUG: send packet to x.x.x.200[500]
2011-06-28 11:04:38: DEBUG: src4 x.x.x.15[500]
2011-06-28 11:04:38: DEBUG: dst4 x.x.x.200[500]
2011-06-28 11:04:38: DEBUG: 1 times of 124 bytes message will be sent to x.x.x.200[500]
2011-06-28 11:04:38: DEBUG:
a8629059 64cca031 cf694b85 6ad13bd9 01100200 00000000 0000007c 0d000038
00000001 00000001 0000002c 01010001 00000024 01010000 800b0001 800c7080
80010007 800e0080 8003fadd 80020001 80040002 0d000014 4a131c81 07035845
5c5728f2 0e95452f 00000014 afcad713 68a1f1c9 6b8696fc 77570100
2011-06-28 11:04:38: DEBUG: resend phase1 packet a862905964cca031:cf694b856ad13bd9
2011-06-28 11:04:38: DEBUG: ===
2011-06-28 11:04:38: DEBUG: 220 bytes message received from x.x.x.200[500] to x.x.x.15[500]
2011-06-28 11:04:38: DEBUG:
a8629059 64cca031 cf694b85 6ad13bd9 04100200 00000000 000000dc 0a000084
03bc0534 ca269bcc 5c705fa5 a1960378 6f3d3987 3693a723 946dfcb4 afa3838e
bb42e8a4 01f11ac5 b82b308e 0df8f750 379ba57e 4bdbceff 6ce2e91e 05f0b738
73f1bde0 65475e79 0dc58006 779caa03 baf884a1 4a65f927 e17378c4 06a96a2f
a56a2d0e 3f074998 909411db 140306a5 b99da0f4 94810f44 88f3f597 023ee8d5
14000014 ba91d639 0c765786 6c757740 d4224814 14000014 cf36b772 c87dee08
d6f5744a 08d98747 00000014 d8974740 3476fce1 906a917a e15ed864
2011-06-28 11:04:38: DEBUG: begin.
2011-06-28 11:04:38: DEBUG: seen nptype=4(ke)
2011-06-28 11:04:38: DEBUG: seen nptype=10(nonce)
2011-06-28 11:04:38: DEBUG: seen nptype=20(nat-d)
2011-06-28 11:04:38: DEBUG: seen nptype=20(nat-d)
2011-06-28 11:04:38: DEBUG: succeed.
2011-06-28 11:04:38: INFO: Hashing x.x.x.15[500] with algo #1
2011-06-28 11:04:38: DEBUG: hash(md5)
2011-06-28 11:04:38: INFO: NAT-D payload #0 verified
2011-06-28 11:04:38: INFO: Hashing x.x.x.200[500] with algo #1
2011-06-28 11:04:38: DEBUG: hash(md5)
2011-06-28 11:04:38: INFO: NAT-D payload #1 verified
2011-06-28 11:04:38: INFO: NAT not detected
2011-06-28 11:04:38: DEBUG: ===
2011-06-28 11:04:38: DEBUG: compute DH's private.
2011-06-28 11:04:38: DEBUG:
69aa17df 9300eac9 2f154ca3 45e3bdd3 2a9509eb fd6c736e c7ff7fe6 9f5a6cb3
c91d4fca cd9666c2 f9da668b ef1b0c75 9686c4e3 e6d61656 b48c4610 2626dfd1
70fec872 ba72732e 5cf65298 b9c73b5b 7cb8885d 0f9a89f0 e137c090 1f6f80e4
c3bce65a 1304b7c3 8c7ae9e9 89f9b3ba 60041e48 9f24392f 7d0f4f67 2cd448ae
2011-06-28 11:04:38: DEBUG: compute DH's public.
2011-06-28 11:04:38: DEBUG:
49b04ee3 d65c6ef6 efb61865 31266abb 1a54d32a c5412235 c9a39154 b39aea4c
7188dd81 e3a83942 e9cbb4a6 37fe4da8 a3d105b8 22aef120 2eda1344 c6fc9594
ba9ba0b0 0ada7f1b 0ca0cbd3 ecbb3b85 688beb2c fcedf2eb 5f9b0e61 501b9ed4
c91ffd4d 982d3d4d 530051ff 1f1dd2e8 33d01bc4 5138adcf 1bf8e64f 26ae8139
2011-06-28 11:04:38: DEBUG: create my CR: X.509 Certificate Signature
2011-06-28 11:04:38: INFO: Hashing x.x.x.200[500] with algo #1
2011-06-28 11:04:38: DEBUG: hash(md5)
2011-06-28 11:04:38: INFO: Hashing x.x.x.15[500] with algo #1
2011-06-28 11:04:38: DEBUG: hash(md5)
2011-06-28 11:04:38: INFO: Adding remote and local NAT-D payloads.
2011-06-28 11:04:38: DEBUG: add payload of len 128, next type 10
2011-06-28 11:04:38: DEBUG: add payload of len 16, next type 7
2011-06-28 11:04:38: DEBUG: add payload of len 1, next type 20
2011-06-28 11:04:38: DEBUG: add payload of len 16, next type 20
2011-06-28 11:04:38: DEBUG: add payload of len 16, next type 0
2011-06-28 11:04:38: DEBUG: 225 bytes from x.x.x.15[500] to x.x.x.200[500]
2011-06-28 11:04:38: DEBUG: sockname x.x.x.15[500]
2011-06-28 11:04:38: DEBUG: send packet from x.x.x.15[500]
2011-06-28 11:04:38: DEBUG: send packet to x.x.x.200[500]
2011-06-28 11:04:38: DEBUG: src4 x.x.x.15[500]
2011-06-28 11:04:38: DEBUG: dst4 x.x.x.200[500]
2011-06-28 11:04:38: DEBUG: 1 times of 225 bytes message will be sent to x.x.x.200[500]
2011-06-28 11:04:38: DEBUG:
a8629059 64cca031 cf694b85 6ad13bd9 04100200 00000000 000000e1 0a000084
49b04ee3 d65c6ef6 efb61865 31266abb 1a54d32a c5412235 c9a39154 b39aea4c
7188dd81 e3a83942 e9cbb4a6 37fe4da8 a3d105b8 22aef120 2eda1344 c6fc9594
ba9ba0b0 0ada7f1b 0ca0cbd3 ecbb3b85 688beb2c fcedf2eb 5f9b0e61 501b9ed4
c91ffd4d 982d3d4d 530051ff 1f1dd2e8 33d01bc4 5138adcf 1bf8e64f 26ae8139
07000014 9d8aa197 defc4e54 c595bb3c b3927a3a 14000005 04140000 14d89747
403476fc e1906a91 7ae15ed8 64000000 14cf36b7 72c87dee 08d6f574 4a08d987
47
2011-06-28 11:04:38: DEBUG: resend phase1 packet a862905964cca031:cf694b856ad13bd9
2011-06-28 11:04:38: DEBUG: compute DH's shared.
2011-06-28 11:04:38: DEBUG:
f5c43acb 0ef7737d 930315c3 0a532149 21bad054 d9c44746 9ce99c32 42d03efb
4d907cfa 4ca21325 14637328 b947815b 46c70cce bb0f471c 027fc9d1 1e895c7b
5242871a 6395912d ebbe6058 2d260cd0 02974f3a e36cb574 0ce51266 99d3ff85
87a8e006 b9022888 87862004 ffc1e548 994d2183 85024f33 a3c79637 7aed8aaf
2011-06-28 11:04:38: DEBUG: nonce1: 2011-06-28 11:04:38: DEBUG:
ba91d639 0c765786 6c757740 d4224814
2011-06-28 11:04:38: DEBUG: nonce2: 2011-06-28 11:04:38: DEBUG:
9d8aa197 defc4e54 c595bb3c b3927a3a
2011-06-28 11:04:38: DEBUG: hmac(hmac_md5)
2011-06-28 11:04:38: DEBUG: SKEYID computed:
2011-06-28 11:04:38: DEBUG:
f7b4e6be 7fa25878 e3e7bb33 53d166f4
2011-06-28 11:04:38: DEBUG: hmac(hmac_md5)
2011-06-28 11:04:38: DEBUG: SKEYID_d computed:
2011-06-28 11:04:38: DEBUG:
d84dbd06 1919a96f 14803a24 92f2f9a2
2011-06-28 11:04:38: DEBUG: hmac(hmac_md5)
2011-06-28 11:04:38: DEBUG: SKEYID_a computed:
2011-06-28 11:04:38: DEBUG:
f3f5bc97 eb67dad1 8d3a8cba bed39b6b
2011-06-28 11:04:38: DEBUG: hmac(hmac_md5)
2011-06-28 11:04:38: DEBUG: SKEYID_e computed:
2011-06-28 11:04:38: DEBUG:
ecdc77c4 6cfe9e41 b2739943 076ac617
2011-06-28 11:04:38: DEBUG: encryption(aes)
2011-06-28 11:04:38: DEBUG: hash(md5)
2011-06-28 11:04:38: DEBUG: final encryption key computed:
2011-06-28 11:04:38: DEBUG:
ecdc77c4 6cfe9e41 b2739943 076ac617
2011-06-28 11:04:38: DEBUG: hash(md5)
2011-06-28 11:04:38: DEBUG: encryption(aes)
2011-06-28 11:04:38: DEBUG: IV computed:
2011-06-28 11:04:38: DEBUG:
a3ee97a9 884653c9 72cd77c6 75cdcf81
2011-06-28 11:04:38: DEBUG: ===
2011-06-28 11:04:38: DEBUG: 76 bytes message received from x.x.x.200[500] to x.x.x.15[500]
2011-06-28 11:04:38: DEBUG:
a8629059 64cca031 cf694b85 6ad13bd9 05100201 00000000 0000004c 96efcc6c
6199bc6d 9ecd7172 637a9477 f2eefff1 d5b29d2c c2262973 d6dba364 ad7263b3
678f581e 592f1b18 0722d898
2011-06-28 11:04:38: DEBUG: begin decryption.
2011-06-28 11:04:38: DEBUG: encryption(aes)
2011-06-28 11:04:38: DEBUG: IV was saved for next processing:
2011-06-28 11:04:38: DEBUG:
ad7263b3 678f581e 592f1b18 0722d898
2011-06-28 11:04:38: DEBUG: encryption(aes)
2011-06-28 11:04:38: DEBUG: with key:
2011-06-28 11:04:38: DEBUG:
ecdc77c4 6cfe9e41 b2739943 076ac617
2011-06-28 11:04:38: DEBUG: decrypted payload by IV:
2011-06-28 11:04:38: DEBUG:
a3ee97a9 884653c9 72cd77c6 75cdcf81
2011-06-28 11:04:38: DEBUG: decrypted payload, but not trimed.
2011-06-28 11:04:38: DEBUG:
0800000c 011101f4 c0a814a6 00000014 8d7abf3f 33849c00 9eb424a2 85c257e6
a0e9bc87 e8b1b1b4 abf2e1dc a8f4c70f
2011-06-28 11:04:38: DEBUG: padding len=16
2011-06-28 11:04:38: DEBUG: skip to trim padding.
2011-06-28 11:04:38: DEBUG: decrypted.
2011-06-28 11:04:38: DEBUG:
a8629059 64cca031 cf694b85 6ad13bd9 05100201 00000000 0000004c 0800000c
011101f4 c0a814a6 00000014 8d7abf3f 33849c00 9eb424a2 85c257e6 a0e9bc87
e8b1b1b4 abf2e1dc a8f4c70f
2011-06-28 11:04:38: DEBUG: begin.
2011-06-28 11:04:38: DEBUG: seen nptype=5(id)
2011-06-28 11:04:38: DEBUG: seen nptype=8(hash)
2011-06-28 11:04:38: DEBUG: succeed.
2011-06-28 11:04:38: ERROR: invalid authmethod 64221 why ?
2011-06-28 11:04:43: DEBUG: ===


I haven't found anything related to that issue on Internet, I don't know what to do...

Here are my racoon.conf :

Gateway :
# Racoon IKE daemon configuration file.
# See 'man racoon.conf' for a description of the format and entries.

path include "/etc/racoon";
#path pre_shared_key "/etc/racoon/psk.txt";
path certificate "/etc/racoon/certs";


#option of controlling racoon by racoonctl tool is disabled
listen {
adminsock disabled;
}


#remote section - anonymous address of roadwarrior client
remote anonymous {
## work mode in IKE first phase
## agggressive is for testing purpose
exchange_mode main;

## certificate type, certificate + public key and secret key file name
certificate_type x509 "cert.pem" "privkey.pem";
# ca_type x509 "cacert.pem";

## Identifier sent to the remote host and type to use in phase 1
## asn1dn is type ASN.1 distinguished name (to put right after)
## If string is omitted, racoon will get the DN from the subject
## field in the certificate

my_identifier asn1dn;
# peers_identifier asn1dn;
# verify_identifier off;

## claiming the options requested by other peer
proposal_check claim;

## automatic generation of SPs from the initial connection request
generate_policy on;

## tells racoon to remain passive and wait for new connection
## to be started from the outside
passive on;

## verifying certificates set to off
verify_cert on;

## nat-t set to on
nat_traversal on;

## DPD activation and 20 sec. delay allowed between 2 proof of liveness requests
dpd_delay 20;

## IKE fragmentation enabled
ike_frag on;

## agreement proposal in IKE first phase
proposal {
## cryptography and hash algorithm
encryption_algorithm aes;
hash_algorithm md5;

## authentication method
authentication_method hybrid_rsa_server;

#Diffie-Hellman exponential group
dh_group modp1024;
}
}

# Local network information
# RoadWarrior acceptation among LAN users
mode_cfg {
#starting address of the IP address pool
# LAN
network4 x.x.x.21;

#maximum number of clients
pool_size 10;

#network mask
netmask4 255.255.255.0;

#authentication source - user database on the system
auth_source system;

#configuration source - from data given in this section
conf_source local;

#DNS and WINS servers IP addresses
dns4 x.x.x.36;
wins4 x.x.x.36;

#banner file - welcome message
banner "/etc/racoon/motd";
}


#############################################################

## SA information for IKE second phase

sainfo anonymous {
## Diffie-Hellman exponential group
pfs_group modp1024;

## second phase information lifetime
lifetime time 1 hour;

## cryptography, authentication and compression algorithm
encryption_algorithm aes;

authentication_algorithm hmac_md5;

compression_algorithm deflate;
}




And on the roadwarrior client :

path certificate "/etc/racoon/certs";

listen {
adminsock "/var/run/racoon/racoon.sock" "root" "operator" 0660;
}


#############################################################

## Roadwarrior

## Adresse connue de la passerelle VPN

remote x.x.x.15 {

exchange_mode main;

ca_type x509 "/etc/racoon/certs/cacert.pem";

# obeying the options requested by other peer
proposal_check obey;
# nat-t set to on
nat_traversal on;
# IKE fragmentation enabled
ike_frag on;
# accepting information about the network being connected to
mode_cfg on;
# verifying certificates set to off
verify_cert on;
# IKE first phase starting script
script "/etc/racoon/phase1-up.sh" phase1_up;
# IKE first phase ending script
script "/etc/racoon/phase1-down.sh" phase1_down;

peers_identifier asn1dn;
# verify_identifier off;


proposal {
## cryptographiy and hash algorithm
encryption_algorithm aes;
hash_algorithm md5;

authentication_method hybrid_rsa_client;

## Diffie-Hellman exponential group
dh_group modp1024;
}
}

sainfo anonymous {
## Diffie-Hellman exponential group
pfs_group modp1024;

##second phase information lifetime
lifetime time 1 hour;

##cryptography, authentication and compression algorithm
encryption_algorithm aes;
authentication_algorithm hmac_md5;
compression_algorithm deflate ;
}

Any help would be greatly appreciated.

My bad
 

Seems like you HAVE to run in aggressive mode if you are in roadwarrior mode or using Nat. I found this on Symantec :

Main Mode
Main Mode uses a 6-message exchange between the initiator and responder during a dynamic tunnel negotiation. Main Mode is more secure since it provides Denial of Service protection and more flexibility in the key negotiation. But because it uses the IP address as part of the exchange for identification it cannot be used in configuration where the IP address of a VPN endpoint may change (NAT) or when the IP address of a VPN endpoint is not known ahead of time (telecommuters).

Aggressive Mode
Aggressive Mode uses a 3-message exchange that eliminates the dependency on the IP address for identification. However, it does not afford the same level of Denial-of-Service protection provided by Main Mode. Aggressive Mode is useful for client-to-gateway tunnels where the IP address of the remote user is not known ahead of time, or for configurations where the IP address of the initiator/responder is changed by a NATting device along the way.

I'll mark the thread as solved.