[RP-PPPoE] Multilink PPPoE

[Gordon Henderson]

On Tue, 1 Nov 2011, Insane Laughing Clown wrote:

> On 11/01/2011 03:35 PM, Gordon Henderson wrote:
>> 
>> A bit stuck here wondering what to do... Wondering if there is anyone
>> here who might even give me a pointer to the source of the issue I'm
>> having...
>> 
>> Anyway - I have a Linux router, multiple Ethernet ports, and 2 ADSL
>> lines which the ISP has bonded. I need to get multilink PPPoE going.
>> 
>> And I can - that's not the issue - the issue is that the router seems to
>> corrupt outgoing SYN packets when the lines are bonded.
>> 
>> I can bring either line up on it's own with the multilink setting in the
>> config file and it's fine - can transfer data OK, but when I bring the
>> 2nd line in, it joins the bundle, and it's still fine - any transfer in
>> progress keeps on going - double the speed - until something sends a SYN
>> up or down the line, then the router sends out a corrupted packet out on
>> one link and the far-end doesn't like it.
>> 
>
> So, do you actually have a packet capture showing this 'corrupted packet'? In 
> what way is it corrupt - eg: wrong source address, fails checksum, bizzare 
> flag settings, etc?

It appears to be sending a zero-length header and it's 100% repeatable. 
Here are some examples:

The link is up, or appears to be - from syslog:

   Nov  1 14:38:06 waveguide pppd[9591]: Using interface ppp1
   Nov  1 14:38:06 waveguide pppd[9591]: New bundle ppp1 created

then

   Nov  1 14:38:08 waveguide pppd[9649]: Using interface ppp1
   Nov  1 14:38:08 waveguide pppd[9649]: Link attached to ppp1

I'm on a remote server (195.10.225.68) and I want to ssh into the site 
with the bonded ADSL (93.89.81.142). Quick ping check:

   --- 93.89.81.142 ping statistics ---
   5 packets transmitted, 5 received, 0% packet loss, time 4001ms
   rtt min/avg/max/mdev = 26.232/26.720/27.584/0.518 ms

So the link is up and a basic ping works.

Now, ssh:

On the remote server, the ssh hangs and I see this via tshark on it's 
Ethernet interface:

This is the opening connection out:

   0.000000 195.10.225.68 -> 93.89.81.142 TCP 54944 > ssh [SYN] Seq=0
 	Win=5840 Len=0 TSV=1385479723 TSER=0 MSS=1460 WS=6

and this is what I see back:

   0.026286 93.89.81.142 -> 195.10.225.68 TCP ssh > 54944 [FIN, RST] Seq=1
 	Win=22944, bogus TCP header length (0, must be at least 20)

On the router with the ADSL line end, if I run tshark on the Ethernet 
interfaces that go to the ADSL modems, I see this on the one that sends 
the reply back:

   1.023964 Draytek_39:82:8f -> Fabiatec_0a:02:86 PPP LCP Echo Request
   1.024370 Fabiatec_0a:02:86 -> Draytek_39:82:8f PPP LCP Echo Reply
   1.906267 93.89.81.142 -> 195.10.225.68 TCP ssh > 55193[Malformed Packet]

(There are LCP echo requests going on constantly - I guess the ISP is 
doing this to monitor the links)

If I run tshark on the PPP interface, then it looks OK during the ssh 
startup:

   0.000000 195.10.225.68 -> 93.89.81.142 TCP 45572 > 22 [SYN] Seq=0
 	Win=5840 Len=0 TSV=1386552063 TSER=0 MSS=1446 WS=6

   0.000053 93.89.81.142 -> 195.10.225.68 TCP 22 > 45572 [SYN, ACK] Seq=0
 	Ack=1 Win=14340 Len=0 MSS=1446 TSV=214111862 TSER=1386552063 WS=4


There is some basic iptables firewalling on the interface, but it's the 
same script applied to the other, working, PPPoE line to another ISP. 
There is also a MSS clamping rule added in by the startup of the link, but 
I've tried it with and without these rules to no avail. Other than that, 
there is some simple iproute rules to control routing - to make sure a 
packet that comes in one one itnerface goes out on that interface - this 
is right out of the LARTC site and I've used it in the past with multiple 
non-bonded links without any issues.

I've tried it on a 2nd (Linux) router just in-case it was an issue with 
running a bonded session with a non-bonded session on the same router, but 
it didn't have any effect.

Does this help? Let me know if you want anything more.

Thanks,

Gordon