[RP-PPPoE] RP-PPPoE downlink performance on iOS/iPhoneOS

[Nathan Anderson]

On Thu, Oct 7, 2010 at 12:44 PM, David F. Skoll <dfs at roaringpenguin.com> wrote:

>> You'd think so, logically...but I just observed load on the device
>> during a download, and it's coming nowhere close to maxing out the CPU
>> (20-30% load on the processor).
>
> Huh, weird... I have no clue.

(No response necessary or expected...I'm just reporting my findings so
far/thinking out loud.)

I'm beginning to think it might possibly be a combination of userland
implementation & something funny going on on this particular platform
(poor performance on the part of the OS X BPF implementation maybe?)

RP-PPPoE on desktop OS X definitely is doing something eerily similar,
with a twist (one which I suspect is also going to be observable under
iOS; it's just that my bandwidth saturation test options are more
limited on that device): the downlink throughput limit is not a solid
one.  I get different results depending on what host I am receiving
data from.  If I test to the same internet host that I was testing to
from the iPhone and getting 300kbps from, I get roughly 1Mbps on Mac
OS X using RP-PPPoE (and roughly 10Mbps up).  If I test to an Ubuntu
Server box I have on my network that I installed Speedtest.NET Mini
to, I get between 5-7Mbps down and about 10+Mbps up using RP-PPPoE on
Mac OS X.  If I use the native Mac OS PPPoE client, I get 50+Mbps
bidirectionally to the local server (and, quite frankly, the
web/Flash-based Speedtest.NET Mini app doesn't give very accurate
downlink throughput results past 20Mbps because the data sample size
is just too small), and about 10Mbps to the internet speedtest host.
(PPP concentration is being handled by a box running MikroTik RouterOS
in all instances, just FYI.)

I finally got the bright idea to run iperf between the Ubuntu Server
and the Mac.  With the native Mac PPPoE client, on a 100Mbps ethernet
network, I see 70+Mbps TCP transfers consistently.  When I switch to
RP-PPPoE, that plummets to 30Mbps TCP.  But keep in mind that this is
to the same box that Speedtest.NET Mini only shows 5-7Mbps.  So,
that's somewhat interesting.

The fact that I was seeing slower downlink transfers the more remote
the host was made it feel to me like TCP slow start/congestion
avoidance was kicking in for some reason.  But why?  I wasn't seeing
any packet loss between the Ubuntu Server box and the Mac when the Mac
was connected using RP-PPPoE, and although the latency between hosts
was slightly higher with RP-PPPoE vs. the native Mac client, it wasn't
abnormally high, either.

So I snooped in on the session with Wireshark (with an external host),
and discovered that during a download, when using RP-PPPoE on the Mac,
I was seeing a LOT of TCP ACK duplicates/retransmits from the Mac to
the remote host during the downlink part of the speedtests.  These
duplicate ACKs were NOT present when using the native client.

This would go a long way to explaining the slow downloads: TCP on the
remote end was reacting (correctly) to this.

So the big question is, why is this happening?  Is this just a
necessary result of having the PPPoE client run entirely in userspace
and incoming buffer data not being received in a timely manner?  I
decided to see if things could be improved by renice-ing the pppoe
process to -20, and it actually DID help.  But not as much as I hoped:
downloads to the local host on the network went up to 10Mbps (vs. 5-7)
on Speedtest.NET Mini, and up to 3Mbps (vs. 1) on the remote speedtest
host.  Values drop back to their originals when I renice the running
pppoe process back to 0.  I tried this 2-3 times, and the difference
is observable each time.

I tried renice-ing pppoe on the iPhone to see if that would help,
though, and it didn't have a measurable impact.  Hrm.

I'm going to run some more tests on a couple different networks, and
I'm going to put iperf on the iPhone and see what results I get with
that.  If anybody has any ideas about how the ACK duplicates can be
avoided or what might be causing them, I'm all-ears.

Thanks again,

-- 
Nathan Anderson
nathan at anderson-net.com