[Bloat] different speeds on different ports? (benchmarking fun)

[Aaron Wood]

I don't have a full writeup yet, but wanted to ask if people on here have
run into this.

I'm seeing a disparity between flent and the dslreports speed tests. On my
connection at home (Comcast 150/12), I figured it was something related to
the test implementations, but minor. But on a connect at a friend with
business-class Comcast (300/12), we're seeing a huge difference. Flent
can't seem to achieve more than 120Mbps, often with an early, couple-second
hump at a much higher speed. But dslreports' speed tests gets the full
300Mbps.

In looking closer at my connection, with sqm (cake) turned off, I'm seeing
~180Mbps download with 500ms of bufferbloat when I use the dslreports test (
http://www.dslreports.com/speedtest/20805152).

Yet flent can't come close to that, even with the tcp_12down test:

​
The current hypothesis that we have is that this is due to either traffic
class, or the ports that traffic are running on. I've ruled out the ping
streams, as a parallel set of netperf tcp_maerts downloads has the same
120Mbps roof.

It would be interesting if we could run some netperf tests using port
80/443 for the listening socket for the data connection (although if doing
deep-packet inspection, we might need to use an actual HTTP transfer).

-Aaron

[Dave Taht]

Well, at these higher speeds the distance to the server and the quality of
the server and its cloud connection start to matter. I'm assuming you are
using flent-fremont which should only be 10-12 ms away from you?

[Dave Taht]

A packet capture from dslreports might be useful also.

[Stefan Alfredsson]

Post by Aaron Wood
The current hypothesis that we have is that this is due to either
traffic class, or the ports that traffic are running on.  I've ruled
out the ping streams, as a parallel set of netperf tcp_maerts
downloads has the same 120Mbps roof.

Also think of server differentiation (maybe full capacity to known
measurement servers, shaped for other destinations). Does dslreports
have flent/netperf responders on their servers as well?

Post by Aaron Wood
It would be interesting if we could run some netperf tests using port
80/443 for the listening socket for the data connection (although if
doing deep-packet inspection, we might need to use an actual HTTP
transfer).

UDP 443 (QUIC) would also be interesting for comparison.

/Stefan

[Toke Høiland-Jørgensen]

Post by Aaron Wood
It would be interesting if we could run some netperf tests using port
80/443 for the listening socket for the data connection (although if
doing deep-packet inspection, we might need to use an actual HTTP
transfer).

Trouble with this is that netserver would have to run as root to be able
to bind to the ports; it does that dynamically, so binding at startup
and dropping privileges won't work. Also it would mean that no other
services could run on those ports.

For your use case, my guess would also be differences in the server
connection (or peering), though. I do believe dslreports will open
connections to multiple servers; you could try running one of the
rtt_fair Flent tests against multiple servers to see if you can push
more data that way...

-Toke

[Sebastian Moeller]

Hi Toke, Stefan, Aaron,

Post by Toke Høiland-Jørgensen

Post by Aaron Wood
It would be interesting if we could run some netperf tests using port
80/443 for the listening socket for the data connection (although if
doing deep-packet inspection, we might need to use an actual HTTP
transfer).

Trouble with this is that netserver would have to run as root to be able
to bind to the ports; it does that dynamically, so binding at startup
and dropping privileges won't work. Also it would mean that no other
services could run on those ports.
For your use case, my guess would also be differences in the server
connection (or peering), though. I do believe dslreports will open
connections to multiple servers; you could try running one of the
rtt_fair Flent tests against multiple servers to see if you can push
more data that way...

After a free registration you can actually select the servers you want to test against as well as the number of concurrent streams. There even is a command line version of the dslreports speedtest (https://www.dslreports.com/forum/speedtestbinary) which offers the following options:
bash-3.2$ ./dslrcli-darwin-amd64 --help
NAME:
Dslrcli - Test network speed

USAGE:
dslrcli-darwin-amd64 [global options] command [command options] [arguments...]

VERSION:
0.1 - 13-Dec-2016

AUTHOR(S):
DSLReports.com <www.dslreports.com>

COMMANDS:
daemonmode, W run in daemonmode
help, h Shows a list of commands or help for one command

GLOBAL OPTIONS:
--debug, -d
--printlog, -l Prints the log on the screen.
--uname value Your DslReports.com username - OPTIONAL.
--upassword value Your DslReports.com password - OPTIONAL.
--machinename value A machine name for the daemon mode. (default: "machine1")
--apikey value Your DslReports.com API Key - OPTIONAL.
--up value Number of streams to use for upload tests (default: "4")
--uploadtime value Time limit to end the upload test in. (default: "15")
--down value Number of streams to use for download tests. (default: "6")
--hz value The frequency (in hz) at which speed samples are to be taken. (default: 1)
--downloadtime value Time limit to end the download test in. (default: "15")
--perserver value The number of streams per server. (default: "3")
--totalstreams value The total number of streams to use for the speedtests - OPTIONAL (default: "X")
--comment value, -c value Include comment with test result.
--output value, -o value Specify type of output . 'json' and 'csv' are currently supported. (default: "default")
--upload-only Performs the Upload test only.
--download-only Performs the Download test only.
--daemon Runs the DslReports client in daemon mode.
--daemon-cmd value Daemon sub command install, uninstall.
--authfile value A json file containing username and password for authentication.
--platform value Specify a platform (GPRS, 3G, 4G, WiFi, Wireless, Satellite, DSL, Cable, Fiber, Unsure). (default: "Unsure")
--latlong value Specify a latitude longitude of the region to select servers from. In the form -latlong a,b
--ipv4 IPv4
--ipv6 IPv6
--help, -h show help
--version, -v print the version

So you can specify the number of servers (--totalstreams MM; totalstreams/perserver shoukld equal number of servers) as well as the number of flows per server (--perserver NN). Anyway, no back to the regularly scheduled programming...

Best Regards
Sebastian

Post by Toke Høiland-Jørgensen
-Toke
_______________________________________________
Bloat mailing list
https://lists.bufferbloat.net/listinfo/bloat

[David Lang]

Post by Toke Høiland-Jørgensen

Post by Aaron Wood
It would be interesting if we could run some netperf tests using port
80/443 for the listening socket for the data connection (although if
doing deep-packet inspection, we might need to use an actual HTTP
transfer).

Trouble with this is that netserver would have to run as root to be able
to bind to the ports; it does that dynamically, so binding at startup
and dropping privileges won't work.

you can also give the binary the capability to bind to low ports without running
as root.

setcap 'cap_net_bind_service=+ep' /path/to/program

Post by Toke Høiland-Jørgensen
Also it would mean that no other services could run on those ports.

that is true.

David Lang

[Colin Dearborn]

This is my guess.
DSL reports uses many streams from different servers to achieve these speeds.
I’m assuming flent is a single stream, so you’re at the mercy of TCP receive windows and latency limiting how fast you can go on that single stream.

From: Bloat [mailto:bloat-***@lists.bufferbloat.net] On Behalf Of Aaron Wood
Sent: Tuesday, August 29, 2017 11:16 PM
To: bloat <***@lists.bufferbloat.net>
Subject: [Bloat] different speeds on different ports? (benchmarking fun)

I don't have a full writeup yet, but wanted to ask if people on here have run into this.

I'm seeing a disparity between flent and the dslreports speed tests. On my connection at home (Comcast 150/12), I figured it was something related to the test implementations, but minor. But on a connect at a friend with business-class Comcast (300/12), we're seeing a huge difference. Flent can't seem to achieve more than 120Mbps, often with an early, couple-second hump at a much higher speed. But dslreports' speed tests gets the full 300Mbps.

In looking closer at my connection, with sqm (cake) turned off, I'm seeing ~180Mbps download with 500ms of bufferbloat when I use the dslreports test (http://www.dslreports.com/speedtest/20805152).

Yet flent can't come close to that, even with the tcp_12down test:
[cid:***@01D332B6.95FED960]
​
The current hypothesis that we have is that this is due to either traffic class, or the ports that traffic are running on. I've ruled out the ping streams, as a parallel set of netperf tcp_maerts downloads has the same 120Mbps roof.

It would be interesting if we could run some netperf tests using port 80/443 for the listening socket for the data connection (although if doing deep-packet inspection, we might need to use an actual HTTP transfer).

-Aaron

[Aaron Wood]

I'd wondered about single vs. multiple, but I'm getting pretty consistent
speeds from the flent-fremont node irrespective of the number of streams
that I use (1, 4, 12, etc).

Post by Colin Dearborn
This is my guess.
DSL reports uses many streams from different servers to achieve these speeds.
I’m assuming flent is a single stream, so you’re at the mercy of TCP
receive windows and latency limiting how fast you can go on that single
stream.
Wood
*Sent:* Tuesday, August 29, 2017 11:16 PM
*Subject:* [Bloat] different speeds on different ports? (benchmarking fun)
I don't have a full writeup yet, but wanted to ask if people on here have run into this.
I'm seeing a disparity between flent and the dslreports speed tests. On
my connection at home (Comcast 150/12), I figured it was something related
to the test implementations, but minor. But on a connect at a friend with
business-class Comcast (300/12), we're seeing a huge difference. Flent
can't seem to achieve more than 120Mbps, often with an early, couple-second
hump at a much higher speed. But dslreports' speed tests gets the full
300Mbps.
In looking closer at my connection, with sqm (cake) turned off, I'm seeing
~180Mbps download with 500ms of bufferbloat when I use the dslreports test (
http://www.dslreports.com/speedtest/20805152).
​
The current hypothesis that we have is that this is due to either traffic
class, or the ports that traffic are running on. I've ruled out the ping
streams, as a parallel set of netperf tcp_maerts downloads has the same
120Mbps roof.
It would be interesting if we could run some netperf tests using port
80/443 for the listening socket for the data connection (although if doing
deep-packet inspection, we might need to use an actual HTTP transfer).
-Aaron

[Aaron Wood]

The friend of mine that I've been working with brought up a cloud node
somewhere with ubuntu and netperf on it, and from another location
(business internet) able to consistently get better throughput from his
cloud node setup than from the flent-fremont node. We're starting to think
that it's something about that node in particular. It seems to have a
125Mbps cap (so I guess about a 140-150Mbps line-rate cap?).

What kind of node is it running on?

Post by Aaron Wood
I'd wondered about single vs. multiple, but I'm getting pretty consistent
speeds from the flent-fremont node irrespective of the number of streams
that I use (1, 4, 12, etc).

Post by Colin Dearborn
This is my guess.
DSL reports uses many streams from different servers to achieve these speeds.
I’m assuming flent is a single stream, so you’re at the mercy of TCP
receive windows and latency limiting how fast you can go on that single
stream.
*Aaron Wood
*Sent:* Tuesday, August 29, 2017 11:16 PM
*Subject:* [Bloat] different speeds on different ports? (benchmarking fun)
I don't have a full writeup yet, but wanted to ask if people on here have run into this.
I'm seeing a disparity between flent and the dslreports speed tests. On
my connection at home (Comcast 150/12), I figured it was something related
to the test implementations, but minor. But on a connect at a friend with
business-class Comcast (300/12), we're seeing a huge difference. Flent
can't seem to achieve more than 120Mbps, often with an early, couple-second
hump at a much higher speed. But dslreports' speed tests gets the full
300Mbps.
In looking closer at my connection, with sqm (cake) turned off, I'm
seeing ~180Mbps download with 500ms of bufferbloat when I use the
dslreports test (http://www.dslreports.com/speedtest/20805152).
​
The current hypothesis that we have is that this is due to either traffic
class, or the ports that traffic are running on. I've ruled out the ping
streams, as a parallel set of netperf tcp_maerts downloads has the same
120Mbps roof.
It would be interesting if we could run some netperf tests using port
80/443 for the listening socket for the data connection (although if doing
deep-packet inspection, we might need to use an actual HTTP transfer).
-Aaron

[Aaron Wood]

I dug out the info from the friend of mine doing this testing, and he was
about to get the dslreports_8dn test to deliver ~800Mbps from his server,
but only ~122Mbps from flent-freemont.bufferbloat.net. Which is awfully
suspicious that this is a setup issue with the node in fremont.

Post by Aaron Wood
The friend of mine that I've been working with brought up a cloud node
somewhere with ubuntu and netperf on it, and from another location
(business internet) able to consistently get better throughput from his
cloud node setup than from the flent-fremont node. We're starting to think
that it's something about that node in particular. It seems to have a
125Mbps cap (so I guess about a 140-150Mbps line-rate cap?).
What kind of node is it running on?

Post by Aaron Wood
I'd wondered about single vs. multiple, but I'm getting pretty consistent
speeds from the flent-fremont node irrespective of the number of streams
that I use (1, 4, 12, etc).

Post by Colin Dearborn
This is my guess.
DSL reports uses many streams from different servers to achieve these speeds.
I’m assuming flent is a single stream, so you’re at the mercy of TCP
receive windows and latency limiting how fast you can go on that single
stream.
Of *Aaron Wood
*Sent:* Tuesday, August 29, 2017 11:16 PM
*Subject:* [Bloat] different speeds on different ports? (benchmarking fun)
I don't have a full writeup yet, but wanted to ask if people on here
have run into this.
I'm seeing a disparity between flent and the dslreports speed tests. On
my connection at home (Comcast 150/12), I figured it was something related
to the test implementations, but minor. But on a connect at a friend with
business-class Comcast (300/12), we're seeing a huge difference. Flent
can't seem to achieve more than 120Mbps, often with an early, couple-second
hump at a much higher speed. But dslreports' speed tests gets the full
300Mbps.
In looking closer at my connection, with sqm (cake) turned off, I'm
seeing ~180Mbps download with 500ms of bufferbloat when I use the
dslreports test (http://www.dslreports.com/speedtest/20805152).
​
The current hypothesis that we have is that this is due to either
traffic class, or the ports that traffic are running on. I've ruled out
the ping streams, as a parallel set of netperf tcp_maerts downloads has the
same 120Mbps roof.
It would be interesting if we could run some netperf tests using port
80/443 for the listening socket for the data connection (although if doing
deep-packet inspection, we might need to use an actual HTTP transfer).
-Aaron