🚀 go-pugleaf

Hi Daniel,

Thank you so much for taking the time to recreate my (rather large) setup, and for posting it.  I double checked my setup compared to your examples here to make sure all was equivalent.  I dumbed down my pf.conf as you suggested.  Still the same symptoms occur.

Then, as a last resort, I went and looked at my sysctl.conf file and started playing with some of my custom settings.  Voila!  net.inet.ip.fastforwarding is the culprit.  Switching net.inet.ip.fastforwarding: 1 -> 0 fixes the issue and allows all packets to match and be encrypted.  Switching net.inet.ip.fastforwarding: 0 -> 1 and the encryption breaks again.

My use of this sysctl dates back to some optimization I'd done when running FreeBSD v7.1.  Does anyone here have a current understanding/opinion of its role and necessity on a FreeBSD v8.3 router/gateway?

Best,
Daniel



On May 17, 2013, at 4:34 AM, Daniel Hartmeier <daniel@benzedrine.cx> wrote:

> I rebuilt your setup but can't reproduce the problem.
> 
> I picked A.A.A.A=3.3.3.3 and B.B.B.B=4.4.4.4 and used FreeBSD 8.3-STABLE
> i386 with GENERIC plus IPSEC, and installed ipsec-tools-0.8.0_3.
> 
> ------------------------------ gatewayA ------------------------------
> 
> /etc/rc.conf
> ifconfig_em0="inet 1.1.1.254 netmask 255.255.255.0"
> ifconfig_em1="inet 3.3.3.3 netmask 255.255.255.0"
> gif_interfaces="gif0"
> gifconfig_gif0="3.3.3.3 4.4.4.4"
> ifconfig_gif0="1.1.1.254 2.2.2.254 netmask 255.255.255.0"
> defaultrouter="3.3.3.1"
> static_routes="gif"
> route_gif="-net 2.2.2.0/24 2.2.2.254"
> gateway_enable="YES"
> racoon_enable="YES"
> pf_enable="YES"
> 
> # ifconfig gif0
> gif0: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> metric 0 mtu 1280
>        tunnel inet 3.3.3.3 --> 4.4.4.4
>        inet 1.1.1.254 --> 2.2.2.254 netmask 0xffffff00
>        options=1<ACCEPT_REV_ETHIP_VER>
> 
> # netstat -anr
> Destination        Gateway            Flags    Refs      Use  Netif Expire
> default            3.3.3.1            UGS         0     1948    em1
> 1.1.1.0/24         link#1             U           0     1270    em0
> 1.1.1.254          link#1             UHS         1        0    lo0
> 2.2.2.0/24         2.2.2.254          UGS         0     1873   gif0
> 2.2.2.254          link#5             UH          0       39   gif0
> 3.3.3.0/24         link#2             U           0        0    em1
> 3.3.3.3            link#2             UHS         0        0    lo0
> 127.0.0.1          link#4             UH          0        0    lo0
> 
> /etc/pf.conf is a simple (and identical on gatewayB)
> set state-policy if-bound
> set skip on { lo }
> scrub in log all fragment reassemble
> block log
> pass
> 
> # pfctl -ss
> em0 icmp 2.2.2.2:25352 <- 1.1.1.1       0:0
> em1 esp 3.3.3.3 -> 4.4.4.4       MULTIPLE:MULTIPLE
> 
> /usr/local/etc/racoon/psk.txt
> 4.4.4.4 topsecret
> 
> /usr/local/etc/racoon/ipsec.conf
> flush;
> spdflush;
> spdadd 1.1.1.0/24 2.2.2.0/24 any -P out ipsec esp/tunnel/3.3.3.3-4.4.4.4/use;
> spdadd 2.2.2.0/24 1.1.1.0/24 any -P in ipsec esp/tunnel/4.4.4.4-3.3.3.3/use;
> 
> /usr/local/etc/racoon/racoon.conf
> (exact copy of handbook example, only differences:)
> listen
>        isakmp          3.3.3.3 [500];
>        isakmp_natt     3.3.3.3 [4500];
> remote  4.4.4.4 [500]
>        my_identifier           address 3.3.3.3;
>        peers_identifier        address 4.4.4.4;
> sainfo  (address 1.1.1.0/24 any address 2.2.2.0/24 any)
> 
> ------------------------------ gatewayB ------------------------------
> 
> ifconfig_em0="inet 2.2.2.254 netmask 255.255.255.0"
> ifconfig_em1="inet 4.4.4.4 netmask 255.255.255.0"
> gif_interfaces="gif0"
> gifconfig_gif0="4.4.4.4 3.3.3.3"
> ifconfig_gif0="2.2.2.254 1.1.1.254 netmask 255.255.255.0"
> defaultrouter="4.4.4.1"
> static_routes="gif"
> route_gif="-net 1.1.1.0/24 1.1.1.254"
> gateway_enable="YES"
> racoon_enable="NO"
> pf_enable="YES"
> 
> # ifconfig gif0
> gif0: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> metric 0 mtu 1280
>        tunnel inet 4.4.4.4 --> 3.3.3.3
>        inet 2.2.2.254 --> 1.1.1.254 netmask 0xffffff00
>        options=1<ACCEPT_REV_ETHIP_VER>
> 
> # netstat -anr
> Destination        Gateway            Flags    Refs      Use  Netif Expire
> default            4.4.4.1            UGS         0     2066    em1
> 1.1.1.0/24         1.1.1.254          UGS         0     2023   gif0
> 1.1.1.254          link#5             UH          0        0   gif0
> 2.2.2.0/24         link#1             U           0     1984    em0
> 2.2.2.254          link#1             UHS         1        0    lo0
> 4.4.4.0/24         link#2             U           0        0    em1
> 4.4.4.4            link#2             UHS         0        0    lo0
> 127.0.0.1          link#4             UH          0        0    lo0
> 
> # pfctl -ss
> em1 esp 4.4.4.4 <- 3.3.3.3       MULTIPLE:MULTIPLE
> em0 icmp 1.1.1.1:25352 -> 2.2.2.2       0:0
> 
> /usr/local/etc/racoon/psk.txt
> 3.3.3.3 topsecret
> 
> /usr/local/etc/racoon/ipsec.conf
> flush;
> spdflush;
> spdadd 2.2.2.0/24 1.1.1.0/24 any -P out ipsec esp/tunnel/4.4.4.4-3.3.3.3/use;
> spdadd 1.1.1.0/24 2.2.2.0/24 any -P in ipsec esp/tunnel/3.3.3.3-4.4.4.4/use;
> 
> /usr/local/etc/racoon/racoon.conf
> listen
>        isakmp          4.4.4.4 [500];
>        isakmp_natt     4.4.4.4 [4500];
> remote  3.3.3.3 [500]
>        my_identifier           address 4.4.4.4;
>        peers_identifier        address 3.3.3.3;
> sainfo  (address 2.2.2.0/24 any address 1.1.1.0/24 any)
> 
> ------------------------------ router ------------------------------
> 
> When I ping from gatewayB to 1.1.1.1 (or from 1.1.1.1 to 2.2.2.2),
> I see only encrypted packets:
> 
> 13:23:52.800285 IP (tos 0x0, ttl 63, id 6391, offset 0, flags [none], proto ESP (50), length 136)
>    4.4.4.4 > 3.3.3.3: ESP(spi=0x016bdbe7,seq=0x5e), length 116
> 13:23:52.801401 IP (tos 0x0, ttl 64, id 5827, offset 0, flags [none], proto ESP (50), length 136)
>    3.3.3.3 > 4.4.4.4: ESP(spi=0x04049e8b,seq=0x5e), length 116
> 13:23:53.820296 IP (tos 0x0, ttl 63, id 6394, offset 0, flags [none], proto ESP (50), length 136)
>    4.4.4.4 > 3.3.3.3: ESP(spi=0x016bdbe7,seq=0x5f), length 116
> 13:23:53.821230 IP (tos 0x0, ttl 64, id 5829, offset 0, flags [none], proto ESP (50), length 136)
>    3.3.3.3 > 4.4.4.4: ESP(spi=0x04049e8b,seq=0x5f), length 116
> 
> There must be something in your setup that causes the difference.
> 
> If there's a non-trivial pf.conf, maybe try with a trivial one first.
> 
> Kind regards,
> Daniel



--
daniel duerr   |   president   |   ouido.net
dd@ouido.net  |  +1 (831) 531-2272 x103
Managed hosting services for Business