[Cerowrt-devel] Latest build test - new sqm-scripts seem to work; "cake overhead 40" didn't

[Alan Jenkins]

Hi

I guess I've done the complementary half to Seb's test :). Basically
"cake overhead 40" didn't work, but that's the fault of cake in this
build. Or tc, as Johnathan suggested. (The "cake atm" part seems to
work, as per my previous test).

"tc qdisc" says "cake overhead 0", as Sebastian noticed. And the test
results show "cake overhead 40" does not give a measurable
improvement. But "tc stab overhead 40" does.

I ran this test with the updated sqm-scripts and I think they're doing
the right thing.


Method:

I used the updated files from sqm-scripts,

(once I remembered to mark them executable. Lacking that causes a
failure with no error messages, because sqm-scripts checks before
running them :)

but didn't bother updating & using luci-app-sqm.

The test was to compare netperf-runner results - ping during combined
upload & download - for "overhead 40" and "overhead 0". I tested both
values of linklayer_adaptation_mechanism.

I had to repeat 6 times (60s per run for each overhead) because of
random variation in the range of 3-4ms. I alternated "overhead 40"
and "overhead 0" to try and exclude longer-term variation effects.

With "stab overhead 40", median latency was better by about 3-4ms.
With "cake overhead 40", there is no such effect.

I'm confident of the result. I just need to set up the scripting
capability in flent now. Running this manually takes too long!

Alan


On 18/06/2015, Sebastian Moeller <***@gmx.de> wrote:
>
> Not sure I can test functionality, but at least I could confirm the
> following:
> 1) using the tc stab link layer adjustment method with cake could work
> (needs testing, td-d qdisc reports all the right things).
>
> 2) the advanced qdisc option strings are passed to cake now, which might
> increase the potential tester pool.
> 2.1) these option strings are truly dangerous: one typo or invalid keyword
> and cake will not start up in that direction without any notice; “no error
> checking” indeed.
>
> 3) somehow the installed cake version has issues with numeric overheads, or
> at least with reporting them.
> cake will accept “overhead N” arguments so it clearly understands they are
> valid (otherwise see 2.1).
> But it does keep reporting “raw” in “tc -d qdisc” while it should report
> some number istead of raw there.
>
> “atm overhead 40” gets reported as “qdisc cake 802d: dev ifb4eth1 root
> refcnt 2 bandwidth 45Mbit besteffort flows atm overhead 0“
> So clearly something is off with either cake or tc on that host.
> @Jonathan, any idea what this might be caused by?
>
> 4) My changes not made the router go off-line...
>
> 5) I wonder whether we should not give cake its own script cake.qos, that
> would keep simple and simplest more readable and the cake script should be
> refreshingly concise ;)
>
>
>>
>> So you can't count on getting accurate results this way, but can at
>> least test functionality.
>
> The main tests that needed doing, is do the changes from the GUI propagate
> into the enabled sqm instances. It seems they do, so my changes are save
> enough for the greater/smaller hard-core tester community. For actual
> functional tests we need an actual ADSL-link where link-layer adjustments
> issues can readily be observed…
>
>>
>> My plan, such as it was, was to get back to 3 source specific gateways
>> but have run into barrier after barrier.
>
> Maybe we should not have “forked” cerowrt before BB was released, by the
> looks of it you could use a barrier breaker. (Sorry can not resist a bad
> pun/joke, was a long day today…)
>
> Best Regards
> Sebastian
>
>>
>>
>> On Thu, Jun 18, 2015 at 1:24 PM, Sebastian Moeller <***@gmx.de>
>> wrote:
>>> Hi Dave,
>>>
>>>
>>> On Jun 18, 2015, at 22:18 , Dave Taht <***@gmail.com> wrote:
>>>
>>>> On Thu, Jun 18, 2015 at 1:04 PM, Sebastian Moeller <***@gmx.de>
>>>> wrote:
>>>>> Hi Dave,
>>>>>
>>>>>
>>>>> On Jun 18, 2015, at 21:33 , Dave Taht <***@gmail.com> wrote:
>>>>>
>>>>>> My present box under test is at 2601:646:8300:2cba::129
>>>>>>
>>>>>> You know the default password for ssh. It is also accessible via
>>>>>> https://[2601:646:8300:2cba::129/]
>>>>>
>>>>> Just the joy of being able to play with cake, made me commit a
>>>>> new change ;) This changes a get_cake_lla_string to
>>>>> `get_cake_lla_string` which in turn might actually do something. I want
>>>>> to note that currently this only passes the numerically defined
>>>>> overhead variable from the GUI to the cake call and side steps the
>>>>> issue how to present the symbolic options. I think this should be fine
>>>>> as current cake users should be seasoned enough to just shrug off this
>>>>> slight inconvenience ;)
>>>>> I only noticed that as I just added the tc_stab link layer
>>>>> adjustment method to the cake calls in simple.qos/simplest.qos so that
>>>>> the different methods can be validated against each other (which was
>>>>> helpful when we had issues with HTB’s private link layer adjustments).
>>>>
>>>> My favorite thing with cake is doing a watch tc -s qdisc show dev
>>>> whatever and looking at the sp(arse) statistic under load. My (ietf)
>>>> world is full of people that think 100ms delay is ok, seeing usec
>>>> makes me happy, and knowing that cake is "peeling" apart the often
>>>> huge packets is quite nice also.
>>>
>>> A good to know.
>>>
>>>>
>>>> High on my list now that I am heads down in getting snmpd to work
>>>> again is to somehow parse various outputs to show drop and CE events
>>>> in mrtg and/or cacti.
>>>>
>>>> I do wish I could find ways to make everyone more productive. I could
>>>> put a box up in my co-lo next week, if that would help, but reflashing
>>>> is a problem no matter where I go.
>>>>
>>>>>>
>>>>>> It is one iteration behind jonathon's cake, and one iteration now
>>>>>> behind sebastian.
>>>>>
>>>>> Make that two iterations ;)
>>>>
>>>> K. Me busy. I will try to keep this box up on this ip for as long as I
>>>> can. I had hoped to flash 6 routers this week.
>>>
>>> Ooops, so I just manually hoisted that boxes sqm-scripts and GUI
>>> to my current development state (I overwrote sqm.lua for luck-app-sqm,
>>> and functions.sh under /usr/lib/sqm and added simple_WIP4cake.qos, these
>>> changes should be confined to simple_WIP4cake (the one extra function in
>>> functions.sh is only called by simple_WIP4cake and hence things should
>>> work for you as before)).
>>> I would love to see wether this can be used to set up cake on
>>> eth0, so I would like to ask for permission to test (for all I know a
>>> reboot might be required, so this only works if you are near that box and
>>> there is nothing important using it right now).
>>>
>>> Best Regards
>>> Sebastian
>>>
>>>
>>>>
>>>>>
>>>>> This also means I would love experienced testers for the latest
>>>>> sqm-scripts changes...
>>>>>
>>>>> Best Regards
>>>>> Sebastian
>>>>>
>>>>>> At the moment I am trying to fix snmpd (massive upgrade + musl
>>>>>> support) and won't be touching that box for a while. It IS a dynamic
>>>>>> ipv6 address....
>>>>>>
>>>>>> Can anyone here push out tc-adv, kmod-* into openwrt's routing repo?
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Thu, Jun 18, 2015 at 12:19 PM, Sebastian Moeller <***@gmx.de>
>>>>>> wrote:
>>>>>>> Hi All,
>>>>>>>
>>>>>>> I just committed a few cake related changes to luci-app-sqm and
>>>>>>> sqm-scripts in ceropackages 3.10. Mainly it is about passing link
>>>>>>> layer and numeric overhead on to cake (if selected as qdisc), but it
>>>>>>> also passed the two string fields ion the GUI aptly named “Advanced
>>>>>>> option string to pass to the [ingress|egress} queueing disciplines;
>>>>>>> no error checking, use very carefully.” to the [ingress|egress] cake
>>>>>>> instances. This should be helpful in testing cake options under
>>>>>>> openwrt builds. BUT this is totally untested, as I have not managed
>>>>>>> to build cake locally let alone install it. So please, anybody,
>>>>>>> please test the changes and report failure and/or success back. Thank
>>>>>>> you very much in advance.
>>>>>>>
>>>>>>> Best Regards
>>>>>>> Sebastian

[Sebastian Moeller]

Hi Alan,

excellent, thanks a million.

On Jun 19, 2015, at 16:44 , Alan Jenkins <***@gmail.com> wrote:

> Hi
>
> I guess I've done the complementary half to Seb's test :). Basically
> "cake overhead 40" didn't work, but that's the fault of cake in this
> build. Or tc, as Johnathan suggested. (The "cake atm" part seems to
> work, as per my previous test).

Great!

>
> "tc qdisc" says "cake overhead 0", as Sebastian noticed. And the test
> results show "cake overhead 40" does not give a measurable
> improvement. But "tc stab overhead 40" does.
>
> I ran this test with the updated sqm-scripts and I think they're doing
> the right thing.

Thanks for testing this, especially as I can not due to a lack of an ADSL-link (and lack of cake actually, last I looked all I could find was cookies in my browser and a promise of pie in my router)

>
>
> Method:
>
> I used the updated files from sqm-scripts,
>
> (once I remembered to mark them executable. Lacking that causes a
> failure with no error messages, because sqm-scripts checks before
> running them :)
>
> but didn't bother updating & using luci-app-sqm.

Ah, okay, I guess I did test this part with Dave’s help, so this should work with the most recent sqm.lua.

>
> The test was to compare netperf-runner results - ping during combined
> upload & download - for "overhead 40" and "overhead 0". I tested both
> values of linklayer_adaptation_mechanism.
>
> I had to repeat 6 times (60s per run for each overhead) because of
> random variation in the range of 3-4ms. I alternated "overhead 40"
> and "overhead 0" to try and exclude longer-term variation effects.
>
> With "stab overhead 40", median latency was better by about 3-4ms.
> With "cake overhead 40", there is no such effect.

Intersting, when I still had a 6M/1M ADSL link, I saw much larger latency under load increases when setting the per packet overhead to small, but I had my egress shaper running at 100% of line rate, so the system was rigged for maximum effect that way. How are your shapers typically set?

>
> I'm confident of the result. I just need to set up the scripting
> capability in flent now. Running this manually takes too long!
>


Best Regards
Sebastian


> Alan
>
>
> On 18/06/2015, Sebastian Moeller <***@gmx.de> wrote:
>>
>> Not sure I can test functionality, but at least I could confirm the
>> following:
>> 1) using the tc stab link layer adjustment method with cake could work
>> (needs testing, td-d qdisc reports all the right things).
>>
>> 2) the advanced qdisc option strings are passed to cake now, which might
>> increase the potential tester pool.
>> 2.1) these option strings are truly dangerous: one typo or invalid keyword
>> and cake will not start up in that direction without any notice; “no error
>> checking” indeed.
>>
>> 3) somehow the installed cake version has issues with numeric overheads, or
>> at least with reporting them.
>> cake will accept “overhead N” arguments so it clearly understands they are
>> valid (otherwise see 2.1).
>> But it does keep reporting “raw” in “tc -d qdisc” while it should report
>> some number istead of raw there.
>>
>> “atm overhead 40” gets reported as “qdisc cake 802d: dev ifb4eth1 root
>> refcnt 2 bandwidth 45Mbit besteffort flows atm overhead 0“
>> So clearly something is off with either cake or tc on that host.
>> @Jonathan, any idea what this might be caused by?
>>
>> 4) My changes not made the router go off-line...
>>
>> 5) I wonder whether we should not give cake its own script cake.qos, that
>> would keep simple and simplest more readable and the cake script should be
>> refreshingly concise ;)
>>
>>
>>>
>>> So you can't count on getting accurate results this way, but can at
>>> least test functionality.
>>
>> The main tests that needed doing, is do the changes from the GUI propagate
>> into the enabled sqm instances. It seems they do, so my changes are save
>> enough for the greater/smaller hard-core tester community. For actual
>> functional tests we need an actual ADSL-link where link-layer adjustments
>> issues can readily be observed…
>>
>>>
>>> My plan, such as it was, was to get back to 3 source specific gateways
>>> but have run into barrier after barrier.
>>
>> Maybe we should not have “forked” cerowrt before BB was released, by the
>> looks of it you could use a barrier breaker. (Sorry can not resist a bad
>> pun/joke, was a long day today…)
>>
>> Best Regards
>> Sebastian
>>
>>>
>>>
>>> On Thu, Jun 18, 2015 at 1:24 PM, Sebastian Moeller <***@gmx.de>
>>> wrote:
>>>> Hi Dave,
>>>>
>>>>
>>>> On Jun 18, 2015, at 22:18 , Dave Taht <***@gmail.com> wrote:
>>>>
>>>>> On Thu, Jun 18, 2015 at 1:04 PM, Sebastian Moeller <***@gmx.de>
>>>>> wrote:
>>>>>> Hi Dave,
>>>>>>
>>>>>>
>>>>>> On Jun 18, 2015, at 21:33 , Dave Taht <***@gmail.com> wrote:
>>>>>>
>>>>>>> My present box under test is at 2601:646:8300:2cba::129
>>>>>>>
>>>>>>> You know the default password for ssh. It is also accessible via
>>>>>>> https://[2601:646:8300:2cba::129/]
>>>>>>
>>>>>> Just the joy of being able to play with cake, made me commit a
>>>>>> new change ;) This changes a get_cake_lla_string to
>>>>>> `get_cake_lla_string` which in turn might actually do something. I want
>>>>>> to note that currently this only passes the numerically defined
>>>>>> overhead variable from the GUI to the cake call and side steps the
>>>>>> issue how to present the symbolic options. I think this should be fine
>>>>>> as current cake users should be seasoned enough to just shrug off this
>>>>>> slight inconvenience ;)
>>>>>> I only noticed that as I just added the tc_stab link layer
>>>>>> adjustment method to the cake calls in simple.qos/simplest.qos so that
>>>>>> the different methods can be validated against each other (which was
>>>>>> helpful when we had issues with HTB’s private link layer adjustments).
>>>>>
>>>>> My favorite thing with cake is doing a watch tc -s qdisc show dev
>>>>> whatever and looking at the sp(arse) statistic under load. My (ietf)
>>>>> world is full of people that think 100ms delay is ok, seeing usec
>>>>> makes me happy, and knowing that cake is "peeling" apart the often
>>>>> huge packets is quite nice also.
>>>>
>>>> A good to know.
>>>>
>>>>>
>>>>> High on my list now that I am heads down in getting snmpd to work
>>>>> again is to somehow parse various outputs to show drop and CE events
>>>>> in mrtg and/or cacti.
>>>>>
>>>>> I do wish I could find ways to make everyone more productive. I could
>>>>> put a box up in my co-lo next week, if that would help, but reflashing
>>>>> is a problem no matter where I go.
>>>>>
>>>>>>>
>>>>>>> It is one iteration behind jonathon's cake, and one iteration now
>>>>>>> behind sebastian.
>>>>>>
>>>>>> Make that two iterations ;)
>>>>>
>>>>> K. Me busy. I will try to keep this box up on this ip for as long as I
>>>>> can. I had hoped to flash 6 routers this week.
>>>>
>>>> Ooops, so I just manually hoisted that boxes sqm-scripts and GUI
>>>> to my current development state (I overwrote sqm.lua for luck-app-sqm,
>>>> and functions.sh under /usr/lib/sqm and added simple_WIP4cake.qos, these
>>>> changes should be confined to simple_WIP4cake (the one extra function in
>>>> functions.sh is only called by simple_WIP4cake and hence things should
>>>> work for you as before)).
>>>> I would love to see wether this can be used to set up cake on
>>>> eth0, so I would like to ask for permission to test (for all I know a
>>>> reboot might be required, so this only works if you are near that box and
>>>> there is nothing important using it right now).
>>>>
>>>> Best Regards
>>>> Sebastian
>>>>
>>>>
>>>>>
>>>>>>
>>>>>> This also means I would love experienced testers for the latest
>>>>>> sqm-scripts changes...
>>>>>>
>>>>>> Best Regards
>>>>>> Sebastian
>>>>>>
>>>>>>> At the moment I am trying to fix snmpd (massive upgrade + musl
>>>>>>> support) and won't be touching that box for a while. It IS a dynamic
>>>>>>> ipv6 address....
>>>>>>>
>>>>>>> Can anyone here push out tc-adv, kmod-* into openwrt's routing repo?
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Thu, Jun 18, 2015 at 12:19 PM, Sebastian Moeller <***@gmx.de>
>>>>>>> wrote:
>>>>>>>> Hi All,
>>>>>>>>
>>>>>>>> I just committed a few cake related changes to luci-app-sqm and
>>>>>>>> sqm-scripts in ceropackages 3.10. Mainly it is about passing link
>>>>>>>> layer and numeric overhead on to cake (if selected as qdisc), but it
>>>>>>>> also passed the two string fields ion the GUI aptly named “Advanced
>>>>>>>> option string to pass to the [ingress|egress} queueing disciplines;
>>>>>>>> no error checking, use very carefully.” to the [ingress|egress] cake
>>>>>>>> instances. This should be helpful in testing cake options under
>>>>>>>> openwrt builds. BUT this is totally untested, as I have not managed
>>>>>>>> to build cake locally let alone install it. So please, anybody,
>>>>>>>> please test the changes and report failure and/or success back. Thank
>>>>>>>> you very much in advance.
>>>>>>>>
>>>>>>>> Best Regards
>>>>>>>> Sebastian
> _______________________________________________
> Cerowrt-devel mailing list
> Cerowrt-***@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cerowrt-devel

[Alan Jenkins]

On 19/06/2015, Sebastian Moeller <***@gmx.de> wrote:
> Hi Alan,
>
> excellent, thanks a million.
>
> On Jun 19, 2015, at 16:44 , Alan Jenkins
> <***@gmail.com> wrote:
>
>> Hi
>>
>> I guess I've done the complementary half to Seb's test :). Basically
>> "cake overhead 40" didn't work, but that's the fault of cake in this
>> build. Or tc, as Johnathan suggested. (The "cake atm" part seems to
>> work, as per my previous test).
>
> Great!
>
>>
>> "tc qdisc" says "cake overhead 0", as Sebastian noticed. And the test
>> results show "cake overhead 40" does not give a measurable
>> improvement. But "tc stab overhead 40" does.
>>
>> I ran this test with the updated sqm-scripts and I think they're doing
>> the right thing.
>
> Thanks for testing this, especially as I can not due to a lack of an
> ADSL-link (and lack of cake actually, last I looked all I could find was
> cookies in my browser and a promise of pie in my router)
>
>>
>>
>> Method:
>>
>> I used the updated files from sqm-scripts,
>>
>> (once I remembered to mark them executable. Lacking that causes a
>> failure with no error messages, because sqm-scripts checks before
>> running them :)
>>
>> but didn't bother updating & using luci-app-sqm.
>
> Ah, okay, I guess I did test this part with Dave’s help, so this should
> work with the most recent sqm.lua.
>
>>
>> The test was to compare netperf-runner results - ping during combined
>> upload & download - for "overhead 40" and "overhead 0". I tested both
>> values of linklayer_adaptation_mechanism.
>>
>> I had to repeat 6 times (60s per run for each overhead) because of
>> random variation in the range of 3-4ms. I alternated "overhead 40"
>> and "overhead 0" to try and exclude longer-term variation effects.
>>
>> With "stab overhead 40", median latency was better by about 3-4ms.
>> With "cake overhead 40", there is no such effect.
>
> Intersting, when I still had a 6M/1M ADSL link, I saw much larger latency
> under load increases when setting the per packet overhead to small, but I
> had my egress shaper running at 100% of line rate, so the system was rigged
> for maximum effect that way. How are your shapers typically set?

For this test I try to push it, today I used 95%. I started trying
100%, which is still much better than unshaped. I was scared off by
the random variation, I think it was higher at 100%.

For long term use I reduce it, because I've seen the line rate vary
slightly. (1020k up today, 912 a while back. Currently it reports a
"max" figure I don't understand, it's about 1100 despite being
rebooted daily. 16390k down).

Alan

[Dave Taht]

re: keywords: it may be that the tc-adv package is not correctly
overriding the tc package. Sigh.

try a opkg update; opkg remove tc-adv; opkg install tc-adv and see if
you get the keywords.

[Alan Jenkins]

On 19/06/15 21:12, Dave Taht wrote:
> re: keywords: it may be that the tc-adv package is not correctly
> overriding the tc package. Sigh.
>
> try a opkg update; opkg remove tc-adv; opkg install tc-adv and see if
> you get the keywords.

Or I screwed up and the box is running the previous build?

$ ssh ***@mortar
BusyBox v1.23.2 (2015-06-12 19:22:15 PDT) built-in shell (ash)


I hope so because the alternative is debugging this-

***@mortar:~# strace
-ash: strace: not found
***@mortar:~# echo $?
127

(same happens when installing tc-adv)

Alan

[Dave Taht]

hmm?

opkg update; opkg install strace ?

I have done several builds in this dir in the last few days, however,
and that could be messing you up. I am trying to get a good build now,
but it is blowing up on needing libssp for some reason.

I really picked the wrong time to try and get something stable built.
This was my last big chance to deploy a few routers in test before
starting to travel extensively.



On Fri, Jun 19, 2015 at 1:40 PM, Alan Jenkins
<***@gmail.com> wrote:
> On 19/06/15 21:12, Dave Taht wrote:
>>
>> re: keywords: it may be that the tc-adv package is not correctly
>> overriding the tc package. Sigh.
>>
>> try a opkg update; opkg remove tc-adv; opkg install tc-adv and see if
>> you get the keywords.
>
>
> Or I screwed up and the box is running the previous build?
>
> $ ssh ***@mortar
> BusyBox v1.23.2 (2015-06-12 19:22:15 PDT) built-in shell (ash)
>
>
> I hope so because the alternative is debugging this-
>
> ***@mortar:~# strace
> -ash: strace: not found
> ***@mortar:~# echo $?
> 127
>
> (same happens when installing tc-adv)
>
> Alan



--
Dave Täht
worldwide bufferbloat report:
http://www.dslreports.com/speedtest/results/bufferbloat
And:
What will it take to vastly improve wifi for everyone?
https://plus.google.com/u/0/explore/makewififast

[Dave Taht]

probably easy to fix. Not that pimd is well maintained in the first place.

make[4]: Entering directory
`/build/cero3/src/ubnt/build_dir/target-mips_34kc_musl-1.1.10/pimd-2.1.8'
CC pim.o
pim.c: In function 'pim_read':
pim.c:128:5: error: implicit declaration of function 'sigblock'
[-Werror=implicit-function-declaration]
omask = sigblock(sigmask(SIGALRM));
^
pim.c:128:5: error: implicit declaration of function 'sigmask'
[-Werror=implicit-function-declaration]
pim.c:136:5: error: implicit declaration of function 'sigsetmask'
[-Werror=implicit-function-declaration]
(void)sigsetmask(omask);
^
cc1: all warnings being treated as errors
make[4]: *** [pim.o] Error 1
make[4]: Leaving directory
`/build/cero3/src/ubnt/build_dir/target-mips_34kc_musl-1.1.10/pimd-2.1.8'
make[3]: *** [/build/cero3/src/ubnt/build_dir/target-mips_34kc_musl-1.1.10/pimd-2.1.8/.built]
Error 2
make[3]: Leaving directory `/build/cero3/src/ubnt/feeds/cero/net/pimd'
make[2]: *** [package/feeds/cero/pimd/compile] Error 2
make[2]: Leaving directory `/build/cero3/src/ubnt'
make[1]: *** [/build/cero3/src/ubnt/staging_dir/target-mips_34kc_musl-1.1.10/stamp/.package_compile]
Error 2
make[1]: Leaving directory `/build/cero3/src/ubnt'
make: *** [world] Error 2

On Fri, Jun 19, 2015 at 1:51 PM, Dave Taht <***@gmail.com> wrote:
> hmm?
>
> opkg update; opkg install strace ?
>
> I have done several builds in this dir in the last few days, however,
> and that could be messing you up. I am trying to get a good build now,
> but it is blowing up on needing libssp for some reason.
>
> I really picked the wrong time to try and get something stable built.
> This was my last big chance to deploy a few routers in test before
> starting to travel extensively.
>
>
>
> On Fri, Jun 19, 2015 at 1:40 PM, Alan Jenkins
> <***@gmail.com> wrote:
>> On 19/06/15 21:12, Dave Taht wrote:
>>>
>>> re: keywords: it may be that the tc-adv package is not correctly
>>> overriding the tc package. Sigh.
>>>
>>> try a opkg update; opkg remove tc-adv; opkg install tc-adv and see if
>>> you get the keywords.
>>
>>
>> Or I screwed up and the box is running the previous build?
>>
>> $ ssh ***@mortar
>> BusyBox v1.23.2 (2015-06-12 19:22:15 PDT) built-in shell (ash)
>>
>>
>> I hope so because the alternative is debugging this-
>>
>> ***@mortar:~# strace
>> -ash: strace: not found
>> ***@mortar:~# echo $?
>> 127
>>
>> (same happens when installing tc-adv)
>>
>> Alan
>
>
>
> --
> Dave Täht
> worldwide bufferbloat report:
> http://www.dslreports.com/speedtest/results/bufferbloat
> And:
> What will it take to vastly improve wifi for everyone?
> https://plus.google.com/u/0/explore/makewififast



--
Dave Täht
worldwide bufferbloat report:
http://www.dslreports.com/speedtest/results/bufferbloat
And:
What will it take to vastly improve wifi for everyone?
https://plus.google.com/u/0/explore/makewififast

[Alan Jenkins]

On 19/06/15 21:51, Dave Taht wrote:
> hmm?
>
> opkg update; opkg install strace ?

sorry. Yes, that's what I did; I left that part out. strace isn't
installed by default, I tried to install it at some point, got "not found"

same error with tc command from tc-adv if I try to reinstall it


> I have done several builds in this dir in the last few days, however,
> and that could be messing you up.

Ok, that's it.

***@mortar:~# grep musl `which strace`
/lib/ld-musl-mips-sf.so.1

Don't think that's going to work on my box somehow :)

***@mortar:~# grep -i uclibc `which which`
/lib/ld-uClibc.so.0
__uClibc_main



> I am trying to get a good build now,
> but it is blowing up on needing libssp for some reason.
>
> I really picked the wrong time to try and get something stable built.
> This was my last big chance to deploy a few routers in test before
> starting to travel extensively.
>
>
>
> On Fri, Jun 19, 2015 at 1:40 PM, Alan Jenkins
> <***@gmail.com> wrote:
>> On 19/06/15 21:12, Dave Taht wrote:
>>> re: keywords: it may be that the tc-adv package is not correctly
>>> overriding the tc package. Sigh.
>>>
>>> try a opkg update; opkg remove tc-adv; opkg install tc-adv and see if
>>> you get the keywords.
>>
>> Or I screwed up and the box is running the previous build?
>>
>> $ ssh ***@mortar
>> BusyBox v1.23.2 (2015-06-12 19:22:15 PDT) built-in shell (ash)
>>
>>
>> I hope so because the alternative is debugging this-
>>
>> ***@mortar:~# strace
>> -ash: strace: not found
>> ***@mortar:~# echo $?
>> 127
>>
>> (same happens when installing tc-adv)
>>
>> Alan
>
>

[Dave Taht]

I am making one last effort to do a new build, please give me 10
minutes to see if it completes.

if it doesn't there will be much gnashing of teeth, and grumbling you
should probably be able to hear from your location.


On Fri, Jun 19, 2015 at 1:57 PM, Alan Jenkins
<***@gmail.com> wrote:
> On 19/06/15 21:51, Dave Taht wrote:
>>
>> hmm?
>>
>> opkg update; opkg install strace ?
>
>
> sorry. Yes, that's what I did; I left that part out. strace isn't
> installed by default, I tried to install it at some point, got "not found"
>
> same error with tc command from tc-adv if I try to reinstall it
>
>
>> I have done several builds in this dir in the last few days, however,
>> and that could be messing you up.
>
>
> Ok, that's it.
>
> ***@mortar:~# grep musl `which strace`
> /lib/ld-musl-mips-sf.so.1
>
> Don't think that's going to work on my box somehow :)
>
> ***@mortar:~# grep -i uclibc `which which`
> /lib/ld-uClibc.so.0
> __uClibc_main
>
>
>
>
>> I am trying to get a good build now,
>> but it is blowing up on needing libssp for some reason.
>>
>> I really picked the wrong time to try and get something stable built.
>> This was my last big chance to deploy a few routers in test before
>> starting to travel extensively.
>>
>>
>>
>> On Fri, Jun 19, 2015 at 1:40 PM, Alan Jenkins
>> <***@gmail.com> wrote:
>>>
>>> On 19/06/15 21:12, Dave Taht wrote:
>>>>
>>>> re: keywords: it may be that the tc-adv package is not correctly
>>>> overriding the tc package. Sigh.
>>>>
>>>> try a opkg update; opkg remove tc-adv; opkg install tc-adv and see if
>>>> you get the keywords.
>>>
>>>
>>> Or I screwed up and the box is running the previous build?
>>>
>>> $ ssh ***@mortar
>>> BusyBox v1.23.2 (2015-06-12 19:22:15 PDT) built-in shell (ash)
>>>
>>>
>>> I hope so because the alternative is debugging this-
>>>
>>> ***@mortar:~# strace
>>> -ash: strace: not found
>>> ***@mortar:~# echo $?
>>> 127
>>>
>>> (same happens when installing tc-adv)
>>>
>>> Alan
>>
>>
>>
>



--
Dave Täht
worldwide bufferbloat report:
http://www.dslreports.com/speedtest/results/bufferbloat
And:
What will it take to vastly improve wifi for everyone?
https://plus.google.com/u/0/explore/makewififast

[Dave Taht]

Fresh bits! Get your fresh bits here! (totally untested)

http://snapon.lab.bufferbloat.net/~cero3/lupin/ar71xx/

[Luis E. Garcia]

Dave,
Just a quick question:
Cake is not yet available through the LUCI interface, right?

Regards,
Luis

On Fri, Jun 19, 2015 at 4:24 PM, Dave Taht <***@gmail.com> wrote:

> Fresh bits! Get your fresh bits here! (totally untested)
>
> http://snapon.lab.bufferbloat.net/~cero3/lupin/ar71xx/
> _______________________________________________
> Cerowrt-devel mailing list
> Cerowrt-***@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cerowrt-devel
>

[Dave Taht]

On Fri, Jun 19, 2015 at 2:52 PM, Luis E. Garcia <***@bitamins.net> wrote:
> Dave,
> Just a quick question:
> Cake is not yet available through the LUCI interface, right?

There is no gui support as yet. Patches gladly accepted, as always.
>
> Regards,
> Luis
>
> On Fri, Jun 19, 2015 at 4:24 PM, Dave Taht <***@gmail.com> wrote:
>>
>> Fresh bits! Get your fresh bits here! (totally untested)
>>
>> http://snapon.lab.bufferbloat.net/~cero3/lupin/ar71xx/
>> _______________________________________________
>> Cerowrt-devel mailing list
>> Cerowrt-***@lists.bufferbloat.net
>> https://lists.bufferbloat.net/listinfo/cerowrt-devel
>
>



--
Dave Täht
What will it take to vastly improve wifi for everyone?
https://plus.google.com/u/0/explore/makewififast

[Sebastian Moeller]

Hi Luis,

it should be possible to select cake from the list of qdiscs. And then you should get a somewhat working cake setup. The most recent version also wired up a few more fields for cake including the advanced configuration string fields in the dangerous configuration options in the Queue Discipline tab. If you test it, please post your result (failure or success) here on the list....

Best Regards
Sebastian

On June 19, 2015 11:52:43 PM GMT+02:00, "Luis E. Garcia" <***@bitamins.net> wrote:
>Dave,
>Just a quick question:
>Cake is not yet available through the LUCI interface, right?
>
>Regards,
>Luis
>
>On Fri, Jun 19, 2015 at 4:24 PM, Dave Taht <***@gmail.com> wrote:
>
>> Fresh bits! Get your fresh bits here! (totally untested)
>>
>> http://snapon.lab.bufferbloat.net/~cero3/lupin/ar71xx/
>> _______________________________________________
>> Cerowrt-devel mailing list
>> Cerowrt-***@lists.bufferbloat.net
>> https://lists.bufferbloat.net/listinfo/cerowrt-devel
>>
>
>
>------------------------------------------------------------------------
>
>_______________________________________________
>Cerowrt-devel mailing list
>Cerowrt-***@lists.bufferbloat.net
>https://lists.bufferbloat.net/listinfo/cerowrt-devel

--
Sent from my Android device with K-9 Mail. Please excuse my brevity.

[Jim Reisert AD1C]

On Fri, Jun 19, 2015 at 3:24 PM, Dave Taht wrote:

> Fresh bits! Get your fresh bits here! (totally untested)
> http://snapon.lab.bufferbloat.net/~cero3/lupin/ar71xx/

I installed the factory image on a WNDR3800 and all came up as it
should. WINS networking is working, both Dish Network and Sony
blu-ray players have wireless Internet access.

Are the "out-of-the-box" settings for SQM acceptable for Comcast (60
Mbs down/6 Mbs up)?
Should I check the "Enable" box on the SQM "Basic Settings" tab?
Should I change any settings on the "Queue Discipline" tab?

Thanks - Jim

--
Jim Reisert AD1C, <***@alum.mit.edu>, http://www.ad1c.us

[Sebastian Moeller]

Hi Jim,


On Jun 23, 2015, at 04:41 , Jim Reisert AD1C <***@alum.mit.edu> wrote:

> On Fri, Jun 19, 2015 at 3:24 PM, Dave Taht wrote:
>
>> Fresh bits! Get your fresh bits here! (totally untested)
>> http://snapon.lab.bufferbloat.net/~cero3/lupin/ar71xx/
>
> I installed the factory image on a WNDR3800 and all came up as it
> should. WINS networking is working, both Dish Network and Sony
> blu-ray players have wireless Internet access.
>
> Are the "out-of-the-box" settings for SQM acceptable for Comcast (60
> Mbs down/6 Mbs up)?

Most likely not. Check http://wiki.openwrt.org/doc/howto/sqm . Rich published a great set of instructions for setting up sqm-scripts under openwrt proper.

> Should I check the "Enable" box on the SQM "Basic Settings" tab?

You certain should if you want it to be active, make sure though, to at least select the correct interface and put in rates for the shaper below your link rates, so probably 56Mbps for downlink and 5.7Mbps for uplink as first approximation.

> Should I change any settings on the "Queue Discipline" tab?

You need to select the qos script of your choice, simple.qos if you want to have three priority tiers, simplest.qos if you do not.

Best Regards
Sebastian

>
> Thanks - Jim
>
> --
> Jim Reisert AD1C, <***@alum.mit.edu>, http://www.ad1c.us
> _______________________________________________
> Cerowrt-devel mailing list
> Cerowrt-***@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cerowrt-devel

[Mikael Abrahamsson]

On Tue, 23 Jun 2015, Sebastian Moeller wrote:

> Most likely not. Check http://wiki.openwrt.org/doc/howto/sqm .
> Rich published a great set of instructions for setting up sqm-scripts
> under openwrt proper.

I tried it on Linksys WRT1200AC with OpenWrt CC RC2. I configured sqm to
have 800 megabit/s each direction, and ran iperf3 over IPv4 with NAT44
from Linux box behind WRT1200AC to an OSX macbook connected on a switch on
the same L2 subnet as the WAN port.

Linux <->WRT1200AC<->switch<->OSX

I get 765 megabit/s of throughput using single session, at sirq load of
around 25%. If I lower the mss to 300 (to generate higher pps) I get
around 560 megabit/s of throughput at 50% sirq. With 10 parallel TCP
sessions, I get about the same. At MSS of 200 bytes, I get 400 megabit/s
at 70% sirq.

If I turn off SQM completely, I get 600 megabit/s at 200 byte MSS single
session at 80% sirq and 930 megabit/s at 26% sirq with default MSS.

So if you want high performing device that is OpenWRT compatible and still
does forwarding using CPU so you can test queuing algorithms, the
WRT1200AC and WRT1900ACv2 is the best I have been able to find currently
(unless you go for x86 platform).

--
Mikael Abrahamsson email: ***@swm.pp.se

[Dave Taht]

On Tue, Jun 23, 2015 at 5:55 AM, Mikael Abrahamsson <***@swm.pp.se> wrote:
> On Tue, 23 Jun 2015, Sebastian Moeller wrote:
>
>> Most likely not. Check http://wiki.openwrt.org/doc/howto/sqm .
>> Rich published a great set of instructions for setting up sqm-scripts under
>> openwrt proper.
>
>
> I tried it on Linksys WRT1200AC with OpenWrt CC RC2. I configured sqm to
> have 800 megabit/s each direction, and ran iperf3 over IPv4 with NAT44 from
> Linux box behind WRT1200AC to an OSX macbook connected on a switch on the
> same L2 subnet as the WAN port.
>
> Linux <->WRT1200AC<->switch<->OSX
>
> I get 765 megabit/s of throughput using single session, at sirq load of
> around 25%. If I lower the mss to 300 (to generate higher pps) I get around
> 560 megabit/s of throughput at 50% sirq. With 10 parallel TCP sessions, I
> get about the same. At MSS of 200 bytes, I get 400 megabit/s at 70% sirq.
>
> If I turn off SQM completely, I get 600 megabit/s at 200 byte MSS single
> session at 80% sirq and 930 megabit/s at 26% sirq with default MSS.

Yer missing the more important figure. What is the induced latency in
all these cases? With the system being software limited, I would
imagine that other oddities arise. Running out of cpu, additional
oddities.

When going at hardware line rate, is fq_codel enabled? Does it ever
engage? (My limited testing showed that lacking BQL, delay accrued big
time in the drivers themselves on this platform)

Good idea on using a reduced MSS size!

I would really like to get to the point where cake ran on this
platform, but thus far we have not managed to get a working build, nor
push cake into mainline openwrt.... my observation is that a lot of
the overhead of sqm comes from tc filters, iptables rules and NAT.

> So if you want high performing device that is OpenWRT compatible and still
> does forwarding using CPU so you can test queuing algorithms, the WRT1200AC
> and WRT1900ACv2 is the best I have been able to find currently (unless you
> go for x86 platform).

yes, x86 is fastest. I have not tried the reduced mss idea on my
rangeley box, I will check that out!

> --
> Mikael Abrahamsson email: ***@swm.pp.se
> _______________________________________________
> Cerowrt-devel mailing list
> Cerowrt-***@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cerowrt-devel



--
Dave Täht
worldwide bufferbloat report:
http://www.dslreports.com/speedtest/results/bufferbloat
And:
What will it take to vastly improve wifi for everyone?
https://plus.google.com/u/0/explore/makewififast

[Sebastian Moeller]

Hi Mikael,


On Jun 23, 2015, at 14:55 , Mikael Abrahamsson <***@swm.pp.se> wrote:

> On Tue, 23 Jun 2015, Sebastian Moeller wrote:
>
>> Most likely not. Check http://wiki.openwrt.org/doc/howto/sqm . Rich published a great set of instructions for setting up sqm-scripts under openwrt proper.
>
> I tried it on Linksys WRT1200AC with OpenWrt CC RC2. I configured sqm to have 800 megabit/s each direction, and ran iperf3 over IPv4 with NAT44 from Linux box behind WRT1200AC to an OSX macbook connected on a switch on the same L2 subnet as the WAN port.
>
> Linux <->WRT1200AC<->switch<->OSX

Thanks a lot, interesting data! Was this test stressing both directions at the same time? (My guess is if the test was UDP i don’t know, for a TCP test I am quite confident that it was uni-directional as the @full MTU data does not show enough loss to accommodate the roughly 2% reverse ACK traffic for the opposite direction).

>
> I get 765 megabit/s of throughput using single session, at sirq load of around 25%. If I lower the mss to 300 (to generate higher pps) I get around 560 megabit/s of throughput at 50% sirq. With 10 parallel TCP sessions, I get about the same. At MSS of 200 bytes, I get 400 megabit/s at 70% sirq.

I assume iperf3 uses TCP or UDP streams and reports the payload rate, correct? Then we have a MSS of 1460 (with 20 bytes IPv4 header and 20 bytes for TCP or UDP).


@full MTU
MSS:
1500 - 20 - 20 = 1460 byte
Number of packets at 765 Mbps goodput:
(765 * 1000^2) / ((1500 - 20 - 20) * 8) = 65496.5753425 = 65K
On-the-wire packet size (OTWPS) assuming ethernet with FCS and no VLAN tag)s):
1500 + 14 + 4 = 1518 bytes
MSS to OTWPS ratio:
(1500 - 20 - 20) / (1500 + 14 + 4) = 0.961791831357
raw bandwidth consumed by 765 Mbps good put:
765 / ((1500 - 20 - 20) / (1500 + 14 + 4)) = ((765 * 1000^2) / ((1500 - 20 - 20) * 8)) * ((1500 + 14 + 4) * 8) = 795.390410959 Mbps
So basically (1 - (795.390410959/800))*100 = 0.58 % unexplained loss, not bad

@MSS 300
MSS:
300 byte
Number of packets at 560 Mbps goodput:
(560 * 1000^2) / ((300) * 8) = 233333.333333 = 233K
On-the-wire packet size (OTWPS) assuming ethernet with FCS:
300 + 20 + 20 + 14 + 4 = 358 bytes
MSS to OTWPS ratio:
(300) / (300 + 20 + 20 + 14 + 4) = 0.837988826816
raw bandwidth consumed by 560 Mbps good put:
560 / ((300) / (300 + 20 + 20 + 14 + 4 )) = ((560 * 1000^2) / ((300) * 8)) * ((300 + 20 + 20 + 14 + 4 ) * 8) = 668.266666667 Mbps
So basically (1 - (668.266666667/800))*100 = 16.4666666666 % unexplained loss, not pretty but bearable I guess

@MSS 200
MSS:
200 byte
Number of packets at 400 Mbps goodput:
(400 * 1000^2) / ((200) * 8) = 250000 = 250K
On-the-wire packet size (OTWPS) assuming ethernet with FCS:
200 + 20 + 20 + 14 + 4 = 258 bytes
MSS to OTWPS ratio:
(200) / (200 + 20 + 20 + 14 + 4) = 0.77519379845
raw bandwidth consumed by 400 Mbps good put:
400 / ((200) / (200 + 20 + 20 + 14 + 4 )) = ((400 * 1000^2) / ((200) * 8)) * ((200 + 20 + 20 + 14 + 4 ) * 8) = 516 Mbps
So basically (1 - (516/800))*100 = 35.5 % unexplained loss, that is sad. But the packet rate is still at 250K, I winder how this router does with 64 byte ethernet frames


>
> If I turn off SQM completely, I get 600 megabit/s at 200 byte MSS single session at 80% sirq and 930 megabit/s at 26% sirq with default MSS.

Since no shaper was used, I think we need to include the inter-frame-gap and preamble to calculate the maximal payload rates for different packet sizes.

@1Gbps
MSS
(1500 - 20 - 20) = 1460 byte
Number of packets at 930 Mbps goodput:
(930 * 1000^2) / ((1500 - 20 - 20) * 8) = 79623.2876712 = 80K
To asses the maximum achievable at 1 GBE we need to take IFG and preamble into account I think
On-the-wire packet size (OTWPS) assuming ethernet with FCS plus inter frame gap and preamble:
1500 + 14 + 4 + 12 + 8 = 1538 bytes
MSS to OTWPS ratio:
(1500 - 20 - 20) / (1500 + 14 + 4 + 12 + 8) = 0.949284785436
raw bandwidth consumed by 930 Mbps good put:
930 / ((1500 - 20 - 20) / (1500 + 14 + 4 + 12 + 8)) = ((930 * 1000^2) / ((1500 - 20 - 20) * 8)) * ((1500 + 14 + 4 + 12 + 8) * 8) = 979.684931507 Mbps
So basically (1 - (979.684931507/1000))*100 = 2.0315068493 % unexplained loss, not bad.

@1Gbps
MSS
200 = 1460 byte
Number of packets at 600 Mbps goodput:
(600 * 1000^2) / ((200) * 8) = 375000 = 375K
On-the-wire packet size (OTWPS) assuming ethernet with FCS plus inter frame gap and preamble:
200 + 20 + 20 + 14 + 4 + 12 + 8 = 278 bytes
MSS to OTWPS ratio:
200 / (200 + 20 + 20 + 14 + 4 + 12 + 8) = 0.719424460432
raw bandwidth consumed by 600 Mbps good put:
600 / ((200) / (200 + 20 + 20 + 14 + 4 + 12 + 8)) = ((600 * 1000^2) / ((200) * 8)) * ((200 + 20 + 20 + 14 + 4 + 12 + 8) * 8) = 834 Mbps
So basically (1 - (834/1000))*100 = 16.6 % unexplained loss, not bad.

As Dave said it would be nice see RRUL data from the same testbed. It would be so nice if flint had a way to send different sized TCP packets… (I guess this might be faked with MSS clamping in the router and relaying on path MTU discovery?)


>
> So if you want high performing device that is OpenWRT compatible and still does forwarding using CPU so you can test queuing algorithms, the WRT1200AC and WRT1900ACv2 is the best I have been able to find currently (unless you go for x86 platform).

The 1200AC retailed for around 200EUR in Germany the 1900ACv2 will be closer to 300EUR I guess, not too expensive but certainly above my impulse buy limit ;)


tack så mycket & Best Regards
Sebastian

>
> --
> Mikael Abrahamsson email: ***@swm.pp.se
> _______________________________________________
> Cerowrt-devel mailing list
> Cerowrt-***@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cerowrt-devel

[Jonathan Morton]

Not so easy to find those in Finland, it seems, but I assume Amazon carry
them.

- Jonathan Morton

[Mikael Abrahamsson]

On Tue, 23 Jun 2015, Jonathan Morton wrote:

> Not so easy to find those in Finland, it seems, but I assume Amazon
> carry them.

www.webhallen.com is the only retailer in Sweden that currently have them
in stock. They just now becoming available, so I would imagine in the next
few weeks they'll be more widely available.

--
Mikael Abrahamsson email: ***@swm.pp.se

[Mikael Abrahamsson]

On Tue, 23 Jun 2015, Sebastian Moeller wrote:

> Thanks a lot, interesting data! Was this test stressing both
> directions at the same time? (My guess is if the test was UDP i don’t
> know, for a TCP test I am quite confident that it was uni-directional as
> the @full MTU data does not show enough loss to accommodate the roughly
> 2% reverse ACK traffic for the opposite direction).

It was TCP using iperf3, so it was larger data packets one direction and
ACKs going the other direction.

> So basically (1 - (795.390410959/800))*100 = 0.58 % unexplained loss, not bad

Oh, iperf3 can tell you packets lost, so I can re-run the tests and tell
you exactly how many packets were lost and within what 1 second interval
they were lost. My guess is that it would be better to run something else.
My numbers were not meant to be as exact as you have used for calculation,
I'd say I aimed for approximate number. I'll do a test run and give you
more exact data.

> As Dave said it would be nice see RRUL data from the same testbed. It
> would be so nice if flint had a way to send different sized TCP packets…
> (I guess this might be faked with MSS clamping in the router and
> relaying on path MTU discovery?)

That is my next thing to test.

> The 1200AC retailed for around 200EUR in Germany the 1900ACv2 will
> be closer to 300EUR I guess, not too expensive but certainly above my
> impulse buy limit ;)

Yes, I paid 195 EUR for it (1790 SEK), which includes 25% VAT. It's around
150 USD (not including tax) so I'm hoping it'll drop 10-20% in price when
it's more widely available in Europe.

--
Mikael Abrahamsson email: ***@swm.pp.se

[Mikael Abrahamsson]

On Tue, 23 Jun 2015, Sebastian Moeller wrote:

> As Dave said it would be nice see RRUL data from the same testbed. It
> would be so nice if flint had a way to send different sized TCP packets…
> (I guess this might be faked with MSS clamping in the router and
> relaying on path MTU discovery?)

What kind of tests should I run? I have rrul results now, both at the same
time as running iperf3. I set iperf3 to run with 10 parallel streams,
different MSS per test, and let it run for 30 seconds into the rrul test.
So in all the tests, iperf3 session stops running half way into the rrul
test.

I set sqm to 500M up and down on eth0, ECN up and down, and not to squash
DSCP in any direction.

http://swm.pp.se/aqm/rrul-wrt1200ac-150624.pdf

Tell me if there is more information I can provide or tests to run.

--
Mikael Abrahamsson email: ***@swm.pp.se

[Dave Taht]

On Wed, Jun 24, 2015 at 4:31 AM, Mikael Abrahamsson <***@swm.pp.se> wrote:
> On Tue, 23 Jun 2015, Sebastian Moeller wrote:
>
>> As Dave said it would be nice see RRUL data from the same testbed. It
>> would be so nice if flint had a way to send different sized TCP packets… (I
>> guess this might be faked with MSS clamping in the router and relaying on
>> path MTU discovery?)
>
>
> What kind of tests should I run? I have rrul results now, both at the same
> time as running iperf3. I set iperf3 to run with 10 parallel streams,
> different MSS per test, and let it run for 30 seconds into the rrul test. So
> in all the tests, iperf3 session stops running half way into the rrul test.
>
> I set sqm to 500M up and down on eth0, ECN up and down, and not to squash
> DSCP in any direction.
>
> http://swm.pp.se/aqm/rrul-wrt1200ac-150624.pdf

From what I see here you are rarely, if ever, engaging fq_codel
properly. Latencies are pretty high. In particular, I would suspect
you are hitting offloads hard, and the current (fixed in linux 4.1)
codel drop algorithm stops dropping below "maxpacket", which was meant
in the ns2 code to be a MTU, but in the linux code ended up being a
TSO sized (64k!) packet.

tc -s qdisc show # will show the maxpacket.

> Tell me if there is more information I can provide or tests to run.

0) I tend to have a script for everything... I can give you an example
script in another email.

1) tc -s qdisc show dev whatever # will show actuall drop/mark statistics

2) Please run your flent tests with -x --disable-log

-x collects more metadata, --disable-log disables the automatic log
scaling which is so hard to discern on these plots.

Use -t "title" to differentiate between variables under test.

3) I also tend to use flent's --remote-metadata=***@your_openwrtbox
to get the stats on that box into the metadata. You have to add your
local .ssh/id_rsa.pub key to
your_openwrtbox:/etc/dropbear/authorized_keys file to do this.

4) With all that in hand, sticking up a tarball of the results makes
for easy plotting of various other graphs, and using the flent-gui,
you can combine results from each run easily, also.

5) try disabling offloads on all interfaces on the router (or running cake)

My usual suite of tests is rrul, rrul_be, tcp_1up, tcp_1down, and tcp_2up_delay.

and rtt_fair (if you have more than one target server available)...
all without the iperf stuff....

Your mss changing idea is interesting, and I would like to do a mod to
tcp_2up_delay to sort of match your iperf combination + flent to
totally capture all data.

6) I am pretty interested as to what happens *without* sqm at the max
forwarding rate with fq_codel engaged on all these tests.

>
> --
> Mikael Abrahamsson email: ***@swm.pp.se
>
> _______________________________________________
> Cerowrt-devel mailing list
> Cerowrt-***@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cerowrt-devel
>



--
Dave Täht
worldwide bufferbloat report:
http://www.dslreports.com/speedtest/results/bufferbloat
And:
What will it take to vastly improve wifi for everyone?
https://plus.google.com/u/0/explore/makewififast

[Aaron Wood]

Here's a long-range (indoors) test result with the 1900AC. I have about
9dB snr at the client (all the way across my house, which has plaster
walls):

http://www.dslreports.com/speedtest/737736

download buffering is the 1900ac, upload buffering is the mac laptop's
driver. Both are trying too hard to deliver those packets, I think.

The driver has crazy over-buffering in the face of poor signal conditions.
OTOH, I'm amazed at the throughput numbers. But this is an 80Mhz wide 5Ghz
channel, in a quiet environment, so it has LOTS of airspace to work in.

-Aaron

On Wed, Jun 24, 2015 at 9:32 AM, Dave Taht <***@gmail.com> wrote:

> On Wed, Jun 24, 2015 at 4:31 AM, Mikael Abrahamsson <***@swm.pp.se>
> wrote:
> > On Tue, 23 Jun 2015, Sebastian Moeller wrote:
> >
> >> As Dave said it would be nice see RRUL data from the same testbed. It
> >> would be so nice if flint had a way to send different sized TCP
> packets
 (I
> >> guess this might be faked with MSS clamping in the router and relaying
> on
> >> path MTU discovery?)
> >
> >
> > What kind of tests should I run? I have rrul results now, both at the
> same
> > time as running iperf3. I set iperf3 to run with 10 parallel streams,
> > different MSS per test, and let it run for 30 seconds into the rrul
> test. So
> > in all the tests, iperf3 session stops running half way into the rrul
> test.
> >
> > I set sqm to 500M up and down on eth0, ECN up and down, and not to squash
> > DSCP in any direction.
> >
> > http://swm.pp.se/aqm/rrul-wrt1200ac-150624.pdf
>
> From what I see here you are rarely, if ever, engaging fq_codel
> properly. Latencies are pretty high. In particular, I would suspect
> you are hitting offloads hard, and the current (fixed in linux 4.1)
> codel drop algorithm stops dropping below "maxpacket", which was meant
> in the ns2 code to be a MTU, but in the linux code ended up being a
> TSO sized (64k!) packet.
>
> tc -s qdisc show # will show the maxpacket.
>
> > Tell me if there is more information I can provide or tests to run.
>
> 0) I tend to have a script for everything... I can give you an example
> script in another email.
>
> 1) tc -s qdisc show dev whatever # will show actuall drop/mark statistics
>
> 2) Please run your flent tests with -x --disable-log
>
> -x collects more metadata, --disable-log disables the automatic log
> scaling which is so hard to discern on these plots.
>
> Use -t "title" to differentiate between variables under test.
>
> 3) I also tend to use flent's --remote-metadata=***@your_openwrtbox
> to get the stats on that box into the metadata. You have to add your
> local .ssh/id_rsa.pub key to
> your_openwrtbox:/etc/dropbear/authorized_keys file to do this.
>
> 4) With all that in hand, sticking up a tarball of the results makes
> for easy plotting of various other graphs, and using the flent-gui,
> you can combine results from each run easily, also.
>
> 5) try disabling offloads on all interfaces on the router (or running cake)
>
> My usual suite of tests is rrul, rrul_be, tcp_1up, tcp_1down, and
> tcp_2up_delay.
>
> and rtt_fair (if you have more than one target server available)...
> all without the iperf stuff....
>
> Your mss changing idea is interesting, and I would like to do a mod to
> tcp_2up_delay to sort of match your iperf combination + flent to
> totally capture all data.
>
> 6) I am pretty interested as to what happens *without* sqm at the max
> forwarding rate with fq_codel engaged on all these tests.
>
> >
> > --
> > Mikael Abrahamsson email: ***@swm.pp.se
> >
> > _______________________________________________
> > Cerowrt-devel mailing list
> > Cerowrt-***@lists.bufferbloat.net
> > https://lists.bufferbloat.net/listinfo/cerowrt-devel
> >
>
>
>
> --
> Dave TÀht
> worldwide bufferbloat report:
> http://www.dslreports.com/speedtest/results/bufferbloat
> And:
> What will it take to vastly improve wifi for everyone?
> https://plus.google.com/u/0/explore/makewififast
> _______________________________________________
> Cerowrt-devel mailing list
> Cerowrt-***@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cerowrt-devel
>

[Mikael Abrahamsson]

On Wed, 24 Jun 2015, Aaron Wood wrote:

> Here's a long-range (indoors) test result with the 1900AC. I have about
> 9dB snr at the client (all the way across my house, which has plaster
> walls):
>
> http://www.dslreports.com/speedtest/737736
>
> download buffering is the 1900ac, upload buffering is the mac laptop's
> driver. Both are trying too hard to deliver those packets, I think.
>
> The driver has crazy over-buffering in the face of poor signal conditions.
> OTOH, I'm amazed at the throughput numbers. But this is an 80Mhz wide 5Ghz
> channel, in a quiet environment, so it has LOTS of airspace to work in.

Is this over a 25/5 Internet connection? Because I wouldn't imagine that
you would get any significant buffering on the wifi with that kind of
Internet access speed. I regularily get hundreds of megabit/s over
802.11ac wifi.

Also, I guess this is WRT1900ACv1? Running what OS?

--
Mikael Abrahamsson email: ***@swm.pp.se

[Aaron Wood]

On Wed, Jun 24, 2015 at 8:07 PM, Mikael Abrahamsson <***@swm.pp.se>
wrote:

> On Wed, 24 Jun 2015, Aaron Wood wrote:
>
> Here's a long-range (indoors) test result with the 1900AC. I have about
>> 9dB snr at the client (all the way across my house, which has plaster
>> walls):
>>
>

> Is this over a 25/5 Internet connection? Because I wouldn't imagine that
> you would get any significant buffering on the wifi with that kind of
> Internet access speed. I regularily get hundreds of megabit/s over 802.11ac
> wifi.
>

Comcast cable, with ~150Mbps down and 12Mbps up. here's a close-range
result for comparison:

http://www.dslreports.com/speedtest/675012



> Also, I guess this is WRT1900ACv1? Running what OS?


Yep, running OpenWRT CC RC1, with sqm-scripts installed, setup as in:

http://burntchrome.blogspot.com/2015/06/sqmscripts-before-and-after-at-160mbps.html

-Aaron

[Mikael Abrahamsson]

On Wed, 24 Jun 2015, Dave Taht wrote:

> From what I see here you are rarely, if ever, engaging fq_codel
> properly. Latencies are pretty high. In particular, I would suspect
> you are hitting offloads hard, and the current (fixed in linux 4.1)
> codel drop algorithm stops dropping below "maxpacket", which was meant
> in the ns2 code to be a MTU, but in the linux code ended up being a
> TSO sized (64k!) packet.
>
> tc -s qdisc show # will show the maxpacket.

http://swm.pp.se/aqm/qdisc-show.txt

I discovered also that I wasn't running ECN, I had set tcp_ecn = 2 on the
linux box. All the tests done in
http://swm.pp.se/aqm/flent-mikabr-150625-1.tar are now done with ECN
working, without iperf3 running, and with and without SQM, but with
default offload setting.

Also, now iperf3 says "0 packet lost" when I use that.

> 2) Please run your flent tests with -x --disable-log

Done.

> Use -t "title" to differentiate between variables under test.

Done.

> 3) I also tend to use flent's --remote-metadata=***@your_openwrtbox
> to get the stats on that box into the metadata. You have to add your
> local .ssh/id_rsa.pub key to
> your_openwrtbox:/etc/dropbear/authorized_keys file to do this.

Done.

> 4) With all that in hand, sticking up a tarball of the results makes
> for easy plotting of various other graphs, and using the flent-gui,
> you can combine results from each run easily, also.

See above.

> 5) try disabling offloads on all interfaces on the router (or running cake)

This is my next thing to test, I have some other things I need to try
first.

> My usual suite of tests is rrul, rrul_be, tcp_1up, tcp_1down, and tcp_2up_delay.

Done.

> and rtt_fair (if you have more than one target server available)...
> all without the iperf stuff....

I do not have another machine readily available here at home.

> 6) I am pretty interested as to what happens *without* sqm at the max
> forwarding rate with fq_codel engaged on all these tests.

This is included. Why did you want to do this test? Just to see if the
WRT1200AC can do wirespeed forwarding with fq_codel on? Because with the
setup I have, I don't see how there can be any buffering going on because
it's a single gig port both in and out.

--
Mikael Abrahamsson email: ***@swm.pp.se

[Mikael Abrahamsson]

On Thu, 25 Jun 2015, Mikael Abrahamsson wrote:

>> 5) try disabling offloads on all interfaces on the router (or running cake)
>
> This is my next thing to test, I have some other things I need to try first.

http://swm.pp.se/aqm/flent-mikabr-150625-2-sqm-tsogsogro-off.tar

It includes an ethtool -k eth0 and eth1 to show what the settings were.

--
Mikael Abrahamsson email: ***@swm.pp.se

[Dave Taht]

On Thu, Jun 25, 2015 at 2:12 AM, Mikael Abrahamsson <***@swm.pp.se> wrote:
> On Wed, 24 Jun 2015, Dave Taht wrote:
>
>> From what I see here you are rarely, if ever, engaging fq_codel
>> properly. Latencies are pretty high. In particular, I would suspect
>> you are hitting offloads hard, and the current (fixed in linux 4.1)
>> codel drop algorithm stops dropping below "maxpacket", which was meant
>> in the ns2 code to be a MTU, but in the linux code ended up being a
>> TSO sized (64k!) packet.
>>
>> tc -s qdisc show # will show the maxpacket.
>
>
> http://swm.pp.se/aqm/qdisc-show.txt

Ugh. 64k maxpacket. I did not even know that was possible until I saw
it (GRO is usually for tcp acks, and peaks at 24k or so).

I will check to see if openwrt backported that crucial codel patch. Or
we can switch sqm "simplest" to use tbf, or we can do cake´s
peeling....

> I discovered also that I wasn't running ECN, I had set tcp_ecn = 2 on the
> linux box. All the tests done in
> http://swm.pp.se/aqm/flent-mikabr-150625-1.tar are now done with ECN
> working, without iperf3 running, and with and without SQM, but with default
> offload setting.
>
> Also, now iperf3 says "0 packet lost" when I use that.

cool. :) Your router has it off, too, and if you plan to use it for
other things than routing, you might want to turn it on (gleaned from
your metadata, thx!)

>> 2) Please run your flent tests with -x --disable-log
>
>
> Done.
>
>> Use -t "title" to differentiate between variables under test.
>
>
> Done.
>
>> 3) I also tend to use flent's --remote-metadata=***@your_openwrtbox
>> to get the stats on that box into the metadata. You have to add your
>> local .ssh/id_rsa.pub key to
>> your_openwrtbox:/etc/dropbear/authorized_keys file to do this.
>
>
> Done.

Unfortunately the core piece of metadata I wanted from the router was
the qdisc statistics. Didnt parse. Will file bug.

>
>> 4) With all that in hand, sticking up a tarball of the results makes
>> for easy plotting of various other graphs, and using the flent-gui,
>> you can combine results from each run easily, also.
>
>
> See above.
>
>> 5) try disabling offloads on all interfaces on the router (or running
>> cake)
>
>
> This is my next thing to test, I have some other things I need to try first.
>
>> My usual suite of tests is rrul, rrul_be, tcp_1up, tcp_1down, and
>> tcp_2up_delay.
>
>
> Done.
>
>> and rtt_fair (if you have more than one target server available)...
>> all without the iperf stuff....
>
>
> I do not have another machine readily available here at home.
>
>> 6) I am pretty interested as to what happens *without* sqm at the max
>> forwarding rate with fq_codel engaged on all these tests.
>
>
> This is included. Why did you want to do this test? Just to see if the
> WRT1200AC can do wirespeed forwarding with fq_codel on? Because with the
> setup I have, I don't see how there can be any buffering going on because
> it's a single gig port both in and out.

Well, it was good to see fq_codel actually engaging on a few of your
hardware queues. There are plenty of reasons why your egress might not
match your ingress periodically and vice versa.

To quote from the BQL commit: ( https://lwn.net/Articles/454378/ )

"Hardware queuing limits are typically specified in terms of a number
hardware descriptors, each of which has a variable size. The variability
of the size of individual queued items can have a very wide range. For
instance with the e1000 NIC the size could range from 64 bytes to 4K
(with TSO enabled). This variability makes it next to impossible to
choose a single queue limit that prevents starvation and provides lowest
possible latency.

The objective of byte queue limits is to set the limit to be the
minimum needed to prevent starvation between successive transmissions to
the hardware. The latency between two transmissions can be variable in a
system. It is dependent on interrupt frequency, NAPI polling latencies,
scheduling of the queuing discipline, lock contention, etc. Therefore we
propose that byte queue limits should be dynamic and change in
iaccordance with networking stack latencies a system encounters."

In particular, aggressive NAPI settings bug me, in routers. And I am
unfond of hardware multiqueue as presently implemented. Sometimes I
think it would be better to use them for QoS rather than fq with
birthday problems.

Another test idea for you would be to enable fq_codel on just the main
ethernet devices on the router, and not use hw mq on it at all. (tc
qdisc add dev each_ethernet_device_on_router root fq_codel)

2) You still have 15ms of delay at various rates. That is quite a lot
more than what I see on a rangeley (where we get below 5ms on sparse
traffic (partially due to local cpu scheduling delays), and 200*us* or
so when measured at the router with cake) I also see packet loss of
the measurement flows in most tests, which are possibly due to gro
forcing out smaller packets, or some other factor.

It is possible however, that the observed buffering here, is actually
on your host, server, or switch. (you have pfifo_fast on your host)
Try fq_codel on host and server (and/or sch_fq) and see what happens.
Disable tso/gro/gso on your server/host also. That leaves the switch
which I have no insight into. What switch chip is it? (see
/etc/config/network) - on the cerowrt project we got less buffering
out of the switch by enabling jumbo frames.

3) As for latency damage:

http://snapon.lab.bufferbloat.net/~d/gro_damage_to_latency.png # this
can also be somewhat due to ecn.

4) The diffserv marking behavior here was puzzling (losing BE
entirely) - my assumption is this was with also the simultaneous iperf
flows (280mbit on the uplink)?

http://snapon.lab.bufferbloat.net/~d/puzzling_diffserv_marking_behavior.png

5) Also on your hosts and servers, two sysctls seem to be helping
reduce the local bloat.

net.ipv4.tcp_limit_output_bytes = 8192 # I have seen 4k
net.ipv4.tcp_notsent_lowat = 8192 # or 16k - I do not know the right
settings really for either of these

The default for the first was originally proposed to be 4k or so, but
that interacted badly with aggregating wifi drivers on single threaded
benchmarks. (sigh) More recently it was bumped to 256k to make the xen
people happier. (vms suck) On raw hardware, it seems like the lower
settings are pretty optimal in my range of tests....

The second is cheshire´s big change to osx and available in linux for
ages - It increases the number of context switches but keeps way more
traffic in the app, not the kernel. I think the latter can also be set
via a setsockopt.

Have fun exploring more variables! :)

6) I do sometimes find it hard to care about the last 15ms, given the
100s or 1000s of ms we are saving elsewhere.... and how rotten wifi
and wireless are... but I suppose in a world that is moving towards
2-4ms e2e latency along the ISP link on fiber, that 15ms is a lot. (I
am pretty sure everyone here is aware that my personal end-goal for
this work is to get to where I could play music with a drummer down
the street, and that that is about 2.7ms... and I have been waiting
for 20 years to get to be able to do that. I will probably be too old
and deaf and crippled by the time that happens to be able play
anything but chopsticks, and too poor to live near anyplace with
fiber... but...)

> --
> Mikael Abrahamsson email: ***@swm.pp.se



--
Dave Täht
worldwide bufferbloat report:
http://www.dslreports.com/speedtest/results/bufferbloat
And:
What will it take to vastly improve wifi for everyone?
https://plus.google.com/u/0/explore/makewififast

[Dave Taht]

Did you restart sqm after turning off offloads? maxpacket is
accumulative and wont change back otherwise.

[Toke Høiland-Jørgensen]

Dave Taht <***@gmail.com> writes:

> Unfortunately the core piece of metadata I wanted from the router was
> the qdisc statistics. Didnt parse. Will file bug.

My guess is that in this case it's due to the openwrt box missing the tc
and ip binaries - those are not installed by default...

Or rather, sqm-scripts pulls in the tc binary, I think, but Flent uses
the ip binary to get the ip and route information to know from which
interfaces to pull the qdisc information from...

-Toke

[Dave Taht]

On Thu, Jun 25, 2015 at 1:24 PM, Toke Høiland-Jørgensen <***@toke.dk> wrote:
> Dave Taht <***@gmail.com> writes:
>
>> Unfortunately the core piece of metadata I wanted from the router was
>> the qdisc statistics. Didnt parse. Will file bug.
>
> My guess is that in this case it's due to the openwrt box missing the tc
> and ip binaries - those are not installed by default...
>
> Or rather, sqm-scripts pulls in the tc binary, I think, but Flent uses
> the ip binary to get the ip and route information to know from which
> interfaces to pull the qdisc information from...
>
> -Toke

I keep wanting to have a way to have the metadata occupy a bottom
panel, and be pre-expanded, rather than the side. Metadata is becoming
increasingly useful...

In other news: Thank you for this commit! I still keep wanting a
rewrite in something faster than python, or to buy myself a faster pc
when I load up 100 datasets, or find some parallization
opportunities... but.... this was noticibly faster.

Author: Toke Høiland-Jørgensen <***@toke.dk>
Date: Tue Jun 9 13:04:58 2015 +0200

Don't copy raw_values when creating ResultSet.

~3.5x speed-up of loading data files with raw values. Not bad for a
two-line patch... :)



--
Dave Täht
worldwide bufferbloat report:
http://www.dslreports.com/speedtest/results/bufferbloat
And:
What will it take to vastly improve wifi for everyone?
https://plus.google.com/u/0/explore/makewififast

[Mikael Abrahamsson]

On Thu, 25 Jun 2015, Toke Høiland-Jørgensen wrote:

> Dave Taht <***@gmail.com> writes:
>
>> Unfortunately the core piece of metadata I wanted from the router was
>> the qdisc statistics. Didnt parse. Will file bug.
>
> My guess is that in this case it's due to the openwrt box missing the tc
> and ip binaries - those are not installed by default...

I did have tc, but I didn't have ip. Do you need "ip" or "ip-full"? I
installed "ip".

> Or rather, sqm-scripts pulls in the tc binary, I think, but Flent uses
> the ip binary to get the ip and route information to know from which
> interfaces to pull the qdisc information from...

We'll see if my next test run includes the correct information then.

--
Mikael Abrahamsson email: ***@swm.pp.se

[Mikael Abrahamsson]

On Thu, 25 Jun 2015, Dave Taht wrote:

> on your host, server, or switch. (you have pfifo_fast on your host) Try
> fq_codel on host and server (and/or sch_fq) and see what happens.
> Disable tso/gro/gso on your server/host also. That leaves the switch
> which I have no insight into. What switch chip is it? (see
> /etc/config/network) - on the cerowrt project we got less buffering out
> of the switch by enabling jumbo frames.

So the complete physical setup is:

Linux laptop <cable> WRT1200AC <cable> TP-Link TL-SG3424 L2 switch <cable> <macbook pro thunderbolt port>

This means there are multiple places buffering can occur, that doesn't
have fq_codel.

I can't tell what switch chip is being used, it's not on
http://techinfodepot.info/wiki/Linksys_WRT1200AC and I don't know where to
look to find it. In /etc/config/network there is just br-lan and eth0 and
eth1.

--
Mikael Abrahamsson email: ***@swm.pp.se

[Sebastian Moeller]

Hi Mikael,

thanks a lot.

On Jun 24, 2015, at 13:31 , Mikael Abrahamsson <***@swm.pp.se> wrote:

> On Tue, 23 Jun 2015, Sebastian Moeller wrote:
>
>> As Dave said it would be nice see RRUL data from the same testbed. It would be so nice if flint had a way to send different sized TCP packets… (I guess this might be faked with MSS clamping in the router and relaying on path MTU discovery?)
>
> What kind of tests should I run? I have rrul results now, both at the same time as running iperf3.

This is interesting. I also like the way iperf3 reports retransmits (BTW way how does it know this, I thought the kernel does the retransmits under the hood out of sight of applications). Since RRUL does not have such a nice report as iperf3 I will not be able to calculate the difference to the theoretical maximum transfer rates (also I lack insight on the ACK traffic).

> I set iperf3 to run with 10 parallel streams, different MSS per test, and let it run for 30 seconds into the rrul test.

The plots are interesting, even though I have a hard time with the logarithmic scale.


> So in all the tests, iperf3 session stops running half way into the rrul test.

I like how the RRUL flows soak up the newly available bandwidth.

>
> I set sqm to 500M up and down on eth0, ECN up and down, and not to squash DSCP in any direction.

I will have a look at the flint files, but it looks like the 1200ac does 500Mbps bidirectionally with SQM, so this looks like a decent machine for the present and (near?) future. Now I just need to wait until prices come down a tad (I hope that the 1900AC v2 release should drop the 1200ac’s process a bit).

Thanks for the tests, now I know what router to try next (the edgerouterX, which I had eyed as a replacement for the shaper in the wndr3700 tops out at 130K packets per second and hence will not really work that well for a 100/40 Mbps link).

Best Regards
Sebastian


>
> http://swm.pp.se/aqm/rrul-wrt1200ac-150624.pdf
>
> Tell me if there is more information I can provide or tests to run.
>
> --
> Mikael Abrahamsson email: ***@swm.pp.se

[Mikael Abrahamsson]

On Fri, 26 Jun 2015, Sebastian Moeller wrote:

> Thanks for the tests, now I know what router to try next (the
> edgerouterX, which I had eyed as a replacement for the shaper in the
> wndr3700 tops out at 130K packets per second and hence will not really
> work that well for a 100/40 Mbps link).

I did some more tests and it seems with SQM at 500 megabit/s I start to
lose packets at around 250-300k PPS. At these speeds, the rrul_be test is
"only" 85k PPS at 500 megabit/s bidirectional with large packets.

Also, I have an Edgerouter ER-5, but as soon as it does CPU based
forwarding it's really weak, easily under 100 megabit/s even with large
packets. OpenVPN without encryption is less than 20 megabit/s.

Btw, the WRT1200AC is now becoming more widely available and it's 150 EUR
incl 25% VAT and shipping here in Sweden now.

Btw, I tried WNDR3800 setting it to 100/100 SQM. It seems to max out
around 25-30k PPS, but the difference is that when the CPU is full, it
seems to delay/ECN-mark packets because there are no packets lost. When
the WRT1200AC runs out of CPU it starts dropping packets. I always have 0
packets lost with the WNDR3800 when doing iperf3 testing. I found this
difference interesting, wonder where in the forwarding path the WRT1200AC
loses packets?

--
Mikael Abrahamsson email: ***@swm.pp.se

[Sebastian Moeller]

Hi Mikael,

On Jun 26, 2015, at 14:26 , Mikael Abrahamsson <***@swm.pp.se> wrote:

> On Fri, 26 Jun 2015, Sebastian Moeller wrote:
>
>> Thanks for the tests, now I know what router to try next (the edgerouterX, which I had eyed as a replacement for the shaper in the wndr3700 tops out at 130K packets per second and hence will not really work that well for a 100/40 Mbps link).
>
> I did some more tests and it seems with SQM at 500 megabit/s I start to lose packets at around 250-300k PPS. At these speeds, the rrul_be test is "only" 85k PPS at 500 megabit/s bidirectional with large packets.

Ah, that sounds like there 1200ac is a practical solution today; at 500Mbps there are 500^3/(64*8) = 244 Kpps at the smallest size (since the wire is at 1000 Mbps I think we should and can ignore inter frame gap and preamble, SQM certainly ignores them) so it looks like the router is really close to the theoretical maximum, and with larger packets things get way more relaxed (500^3/(1518*8) = 10 Kpps). Pretty decent for a router using no proprietary offload features.

>
> Also, I have an Edgerouter ER-5, but as soon as it does CPU based forwarding it's really weak, easily under 100 megabit/s even with large packets. OpenVPN without encryption is less than 20 megabit/s.

Ah, interesting, so neither the affordable edge router lite nor the (similar) ER-5 will cut it unless the offloads are used; I think the newer edgerouterX is rated for slightly higher speeds without offloads, but not nearly close to what your 1200ac does...

>
> Btw, the WRT1200AC is now becoming more widely available and it's 150 EUR incl 25% VAT and shipping here in Sweden now.

In Germany it still retails for around 200EUR with the cheapest offer at 180EUR (but that is an outlier); guess I should visit Sweden then ;)

>
> Btw, I tried WNDR3800 setting it to 100/100 SQM.

Yeah, not pretty, huh?

> It seems to max out around 25-30k PPS, but the difference is that when the CPU is full, it seems to delay/ECN-mark packets because there are no packets lost. When the WRT1200AC runs out of CPU it starts dropping packets. I always have 0 packets lost with the WNDR3800 when doing iperf3 testing. I found this difference interesting, wonder where in the forwarding path the WRT1200AC loses packets?

Interesting observation, no idea, but intrigued ;)

Best Regards
Sebastian

>
> --
> Mikael Abrahamsson email: ***@swm.pp.se

[Mikael Abrahamsson]

On Fri, 26 Jun 2015, Mikael Abrahamsson wrote:

> Btw, I tried WNDR3800 setting it to 100/100 SQM. It seems to max out
> around 25-30k PPS, but the difference is that when the CPU is full, it
> seems to delay/ECN-mark packets because there are no packets lost. When
> the WRT1200AC runs out of CPU it starts dropping packets. I always have
> 0 packets lost with the WNDR3800 when doing iperf3 testing. I found this
> difference interesting, wonder where in the forwarding path the
> WRT1200AC loses packets?

I checked again, and my WDR4900 with same setup doesn't lose packets
either. Even at 99% sirq, no packets are lost.

WRT1200AC starts to lose packets at 500 megabit/s SQM around MSS 300 and
lower. If I turn off gso, tso and gro, I have to go to MSS 600 and above
to avoid packet loss.

Does flent check for packet loss at all? Perhaps it's something to look
into, because with ECN we really don't want to see any packets lost and
this might be good to include test results for.

--
Mikael Abrahamsson email: ***@swm.pp.se

[Jonathan Morton]

Hypothesis: this might have to do with the receive path. Some devices might
have more capacity than others to buffer inbound packets until the CPU can
get around to servicing them.

- Jonathan Morton

[Mikael Abrahamsson]

On Fri, 26 Jun 2015, Jonathan Morton wrote:

> Hypothesis: this might have to do with the receive path. Some devices might
> have more capacity than others to buffer inbound packets until the CPU can
> get around to servicing them.

Is there a way to tell? I am better at diagnosing Cisco CPU based routers
than Linux ones. I looked in /sys/class/net/ethX/statistics but these
drops are not recorded there.

--
Mikael Abrahamsson email: ***@swm.pp.se

[Jonathan Morton]

These would be hardware tail drops - there might not be a physical counter
recording them. But you could instrument three driver to see whether the
receive buffer is full when serviced.

- Jonathan Morton

[Dave Taht]

On Fri, Jun 26, 2015 at 9:35 AM, Jonathan Morton <***@gmail.com> wrote:
> These would be hardware tail drops - there might not be a physical counter
> recording them. But you could instrument three driver to see whether the
> receive buffer is full when serviced.

from drivers/net/ethernet/marvel/mvneta.c:

/* Max number of Rx descriptors */
#define MVNETA_MAX_RXD 128

this is probably too small, especially given the 64 it is willing to
wait for. At the same time, it is too large, as there are 8 hardware
queues in play here. So you get a huge burst from one flow, it gros it
all together.... aggghh...

/* Max number of Tx descriptors */
#define MVNETA_MAX_TXD 532

this realllllly needs BQL. Same problem(s). Only worse.


>
> - Jonathan Morton
>
>
> _______________________________________________
> Cerowrt-devel mailing list
> Cerowrt-***@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cerowrt-devel
>



--
Dave Täht
worldwide bufferbloat report:
http://www.dslreports.com/speedtest/results/bufferbloat
And:
What will it take to vastly improve wifi for everyone?
https://plus.google.com/u/0/explore/makewififast

[Dave Taht]

Mikael, a simple test of the analysis I just did would be to use
ethtool to set your server to 100mbits (ethtool -s
your_ethernet_device advertise 0x008 and turn on fq_codel on both the
client and server.

if it is using classification for the hw mq, the rrul test from the
client will blow up half the queues. If it is doing hw fq instead,
the rrul_50_up test will blow up all 8 queues.

For giggles, try 0x002 (10mbit)

my own attempts to get a compile for this platform consistently blow
up here similarly for both musl and uclibc.

checking for arm-openwrt-linux-muslgnueabi-gcc...
/build/cero3/src/ac1900/build_dir/toolchain-arm_cortex-a9+vfpv3_gcc-4.8-linaro_musl-1.1.10_eabi/gcc-linaro-4.8-2014.04-minimal/./gcc/xgcc
-B/build/cero3/src/ac1900/build_dir/toolchain-arm_cortex-a9+vfpv3_gcc-4.8-linaro_musl-1.1.10_eabi/gcc-linaro-4.8-2014.04-minimal/./gcc/
-B/build/cero3/src/ac1900/staging_dir/toolchain-arm_cortex-a9+vfpv3_gcc-4.8-linaro_musl-1.1.10_eabi/arm-openwrt-linux-muslgnueabi/bin/
-B/build/cero3/src/ac1900/staging_dir/toolchain-arm_cortex-a9+vfpv3_gcc-4.8-linaro_musl-1.1.10_eabi/arm-openwrt-linux-muslgnueabi/lib/
-isystem /build/cero3/src/ac1900/staging_dir/toolchain-arm_cortex-a9+vfpv3_gcc-4.8-linaro_musl-1.1.10_eabi/arm-openwrt-linux-muslgnueabi/include
-isystem /build/cero3/src/ac1900/staging_dir/toolchain-arm_cortex-a9+vfpv3_gcc-4.8-linaro_musl-1.1.10_eabi/arm-openwrt-linux-muslgnueabi/sys-include
checking for suffix of object files... configure: error: in
`/build/cero3/src/ac1900/build_dir/toolchain-arm_cortex-a9+vfpv3_gcc-4.8-linaro_musl-1.1.10_eabi/gcc-linaro-4.8-2014.04-minimal/arm-openwrt-linux-muslgnueabi/libgcc':
configure: error: cannot compute suffix of object files: cannot compile







On Fri, Jun 26, 2015 at 10:04 AM, Dave Taht <***@gmail.com> wrote:
> On Fri, Jun 26, 2015 at 9:35 AM, Jonathan Morton <***@gmail.com> wrote:
>> These would be hardware tail drops - there might not be a physical counter
>> recording them. But you could instrument three driver to see whether the
>> receive buffer is full when serviced.
>
> from drivers/net/ethernet/marvel/mvneta.c:
>
> /* Max number of Rx descriptors */
> #define MVNETA_MAX_RXD 128
>
> this is probably too small, especially given the 64 it is willing to
> wait for. At the same time, it is too large, as there are 8 hardware
> queues in play here. So you get a huge burst from one flow, it gros it
> all together.... aggghh...
>
> /* Max number of Tx descriptors */
> #define MVNETA_MAX_TXD 532
>
> this realllllly needs BQL. Same problem(s). Only worse.
>
>
>>
>> - Jonathan Morton
>>
>>
>> _______________________________________________
>> Cerowrt-devel mailing list
>> Cerowrt-***@lists.bufferbloat.net
>> https://lists.bufferbloat.net/listinfo/cerowrt-devel
>>
>
>
>
> --
> Dave Täht
> worldwide bufferbloat report:
> http://www.dslreports.com/speedtest/results/bufferbloat
> And:
> What will it take to vastly improve wifi for everyone?
> https://plus.google.com/u/0/explore/makewififast



--
Dave Täht
worldwide bufferbloat report:
http://www.dslreports.com/speedtest/results/bufferbloat
And:
What will it take to vastly improve wifi for everyone?
https://plus.google.com/u/0/explore/makewififast

[Mikael Abrahamsson]

On Fri, 26 Jun 2015, Dave Taht wrote:

> Mikael, a simple test of the analysis I just did would be to use
> ethtool to set your server to 100mbits (ethtool -s
> your_ethernet_device advertise 0x008 and turn on fq_codel on both the
> client and server.

Hm what do you mean by "client" and "server"?

Where do you want the queueing to happen? Egress from the WRT1200AC
towards the server? Then setting the WAN port of the WRT1200AC to 100
megabit/s would work, yes.

--
Mikael Abrahamsson email: ***@swm.pp.se

[Dave Taht]

On Fri, Jun 26, 2015 at 11:38 AM, Mikael Abrahamsson <***@swm.pp.se> wrote:
> On Fri, 26 Jun 2015, Dave Taht wrote:
>
>> Mikael, a simple test of the analysis I just did would be to use
>> ethtool to set your server to 100mbits (ethtool -s
>> your_ethernet_device advertise 0x008 and turn on fq_codel on both the
>> client and server.
>
>
> Hm what do you mean by "client" and "server"?
>
> Where do you want the queueing to happen? Egress from the WRT1200AC towards
> the server?

Yes.

> Then setting the WAN port of the WRT1200AC to 100 megabit/s
> would work, yes.

Yes, but I am unsure from looking at the driver that using ethtool on
the egress on the wrt1200ac will actually work, but
pretty sure it will work if you set it on the server. feel free to try both. :)

>
> --
> Mikael Abrahamsson email: ***@swm.pp.se



--
Dave Täht
worldwide bufferbloat report:
http://www.dslreports.com/speedtest/results/bufferbloat
And:
What will it take to vastly improve wifi for everyone?
https://plus.google.com/u/0/explore/makewififast

[Dave Taht]

On Fri, Jun 26, 2015 at 11:58 AM, Dave Taht <***@gmail.com> wrote:
> On Fri, Jun 26, 2015 at 11:38 AM, Mikael Abrahamsson <***@swm.pp.se> wrote:
>> On Fri, 26 Jun 2015, Dave Taht wrote:
>>
>>> Mikael, a simple test of the analysis I just did would be to use
>>> ethtool to set your server to 100mbits (ethtool -s
>>> your_ethernet_device advertise 0x008 and turn on fq_codel on both the
>>> client and server.
>>
>>
>> Hm what do you mean by "client" and "server"?

your topology is:

client box - router - server

forcing the router - server link to 100mbit will push the egress
buffering into the router for the rrul_50_up test in particular.

>> Where do you want the queueing to happen? Egress from the WRT1200AC towards
>> the server?
>
> Yes.
>
>> Then setting the WAN port of the WRT1200AC to 100 megabit/s
>> would work, yes.
>
> Yes, but I am unsure from looking at the driver that using ethtool on
> the egress on the wrt1200ac will actually work, but
> pretty sure it will work if you set it on the server. feel free to try both. :)
>
>>
>> --
>> Mikael Abrahamsson email: ***@swm.pp.se
>
>
>
> --
> Dave Täht
> worldwide bufferbloat report:
> http://www.dslreports.com/speedtest/results/bufferbloat
> And:
> What will it take to vastly improve wifi for everyone?
> https://plus.google.com/u/0/explore/makewififast



--
Dave Täht
worldwide bufferbloat report:
http://www.dslreports.com/speedtest/results/bufferbloat
And:
What will it take to vastly improve wifi for everyone?
https://plus.google.com/u/0/explore/makewififast

[Mikael Abrahamsson]

On Fri, 26 Jun 2015, Dave Taht wrote:

> On Fri, Jun 26, 2015 at 11:58 AM, Dave Taht <***@gmail.com> wrote:
>> On Fri, Jun 26, 2015 at 11:38 AM, Mikael Abrahamsson <***@swm.pp.se> wrote:
>>> On Fri, 26 Jun 2015, Dave Taht wrote:
>>>
>>>> Mikael, a simple test of the analysis I just did would be to use
>>>> ethtool to set your server to 100mbits (ethtool -s
>>>> your_ethernet_device advertise 0x008 and turn on fq_codel on both the
>>>> client and server.
>>>
>>>
>>> Hm what do you mean by "client" and "server"?
>
> your topology is:
>
> client box - router - server
>
> forcing the router - server link to 100mbit will push the egress
> buffering into the router for the rrul_50_up test in particular.

No, my topology is <client - router - switch - server>, that is what made
me confused.

So if I forced the eth0 router-switch link into 100M I will break the
server->client direction (it'll be shallow buffer fifo) but at least
client->server direction will exercise fq_codel on the router.

--
Mikael Abrahamsson email: ***@swm.pp.se

[Dave Taht]

On Fri, Jun 26, 2015 at 12:11 PM, Mikael Abrahamsson <***@swm.pp.se> wrote:
> On Fri, 26 Jun 2015, Dave Taht wrote:
>
>> On Fri, Jun 26, 2015 at 11:58 AM, Dave Taht <***@gmail.com> wrote:
>>>
>>> On Fri, Jun 26, 2015 at 11:38 AM, Mikael Abrahamsson <***@swm.pp.se>
>>> wrote:
>>>>
>>>> On Fri, 26 Jun 2015, Dave Taht wrote:
>>>>
>>>>> Mikael, a simple test of the analysis I just did would be to use
>>>>> ethtool to set your server to 100mbits (ethtool -s
>>>>> your_ethernet_device advertise 0x008 and turn on fq_codel on both the
>>>>> client and server.
>>>>
>>>>
>>>>
>>>> Hm what do you mean by "client" and "server"?
>>
>>
>> your topology is:
>>
>> client box - router - server
>>
>> forcing the router - server link to 100mbit will push the egress
>> buffering into the router for the rrul_50_up test in particular.
>
>
> No, my topology is <client - router - switch - server>, that is what made me
> confused.

> So if I forced the eth0 router-switch link into 100M I will break the
> server->client direction (it'll be shallow buffer fifo) but at least
> client->server direction will exercise fq_codel on the router.

well, maybe the driver will take ethtool on the mvneta. try it. :)

>
>
> --
> Mikael Abrahamsson email: ***@swm.pp.se



--
Dave Täht
worldwide bufferbloat report:
http://www.dslreports.com/speedtest/results/bufferbloat
And:
What will it take to vastly improve wifi for everyone?
https://plus.google.com/u/0/explore/makewififast

[Mikael Abrahamsson]

On Fri, 26 Jun 2015, Dave Taht wrote:

> Yes, but I am unsure from looking at the driver that using ethtool on
> the egress on the wrt1200ac will actually work, but pretty sure it will
> work if you set it on the server. feel free to try both. :)

I set speed 100 on my switch and did some new tests, I left SQM at 500M,
don't know what happens then, thought it was worthwile to test? Remember,
now I don't have fq_codel in the server->client direction, but there it's
an low buffered switchport that is doing the rate adaptation.

Btw, now I am running a nightly build that I compiled for myself for the
WRT1200AC, so if there is anything you would like me to try to change in
the mvneta driver, I can certainly test that. I can do serial console
access to the WRT1200AC if needed as well, I already "unbricked" it once.

Btw, disregard the title in the flent files, I just realised I forgot to
change the title argument. This is without iperf3, with SQM set to 500M,
but eth0 at 100megabit/full duplex.

http://swm.pp.se/aqm/wrt1200ac-150627-100m-sqm.tar

--
Mikael Abrahamsson email: ***@swm.pp.se

[Dave Taht]

your results are showing basically tail drop behavior. Although I
would have expected intrinsic delay on the link to crack 100mbits on
the rrul test, not 20ms (which is still high), and you only hit 7 on
the single threaded tcp up test, based on what I saw in the driver.

turn off sqm, stay at 100mbit, let fq_codel ride, try the rrul_50_up
test. Also do a capture to see if CE was ever exerted.

I do not know why my linksys ac1200 build does not work. I generally
suspect it is because my big build server is getting ancient. Can you
send me your .config file for openwrt?

It would be VERY helpful if you pulled from ceropackages-3.14, and
added and built kmod-sched-cake and tc-adv and switched to using the
ceropackages feed also for luci-app-sqm and sqm-scripts. There are a
couple people here that would probably leap on that! :)

add to your feeds.conf

src-git https://github.com/dtaht/ceropackages-3.10.git

./scripts feeds update
./scripts feeds install kmod-sched-cake tc-adv kmod-sched-fq_pie
edit the .config file to make them modules or installed by default
make menuconfig then save
make

I went through hell trying to to figure out how to switch over to the
cero versions of these. If yer doing the builds, I could try hacking
the BQL. Maybe. In sweden.



On Fri, Jun 26, 2015 at 10:03 PM, Mikael Abrahamsson <***@swm.pp.se> wrote:
> On Fri, 26 Jun 2015, Dave Taht wrote:
>
>> Yes, but I am unsure from looking at the driver that using ethtool on the
>> egress on the wrt1200ac will actually work, but pretty sure it will work if
>> you set it on the server. feel free to try both. :)
>
>
> I set speed 100 on my switch and did some new tests, I left SQM at 500M,
> don't know what happens then, thought it was worthwile to test? Remember,
> now I don't have fq_codel in the server->client direction, but there it's an
> low buffered switchport that is doing the rate adaptation.
>
> Btw, now I am running a nightly build that I compiled for myself for the
> WRT1200AC, so if there is anything you would like me to try to change in the
> mvneta driver, I can certainly test that. I can do serial console access to
> the WRT1200AC if needed as well, I already "unbricked" it once.
>
> Btw, disregard the title in the flent files, I just realised I forgot to
> change the title argument. This is without iperf3, with SQM set to 500M, but
> eth0 at 100megabit/full duplex.
>
> http://swm.pp.se/aqm/wrt1200ac-150627-100m-sqm.tar
>
>
> --
> Mikael Abrahamsson email: ***@swm.pp.se



--
Dave Täht
worldwide bufferbloat report:
http://www.dslreports.com/speedtest/results/bufferbloat
And:
What will it take to vastly improve wifi for everyone?
https://plus.google.com/u/0/explore/makewififast

[Mikael Abrahamsson]

On Fri, 26 Jun 2015, Dave Taht wrote:

> your results are showing basically tail drop behavior. Although I
> would have expected intrinsic delay on the link to crack 100mbits on
> the rrul test, not 20ms (which is still high), and you only hit 7 on
> the single threaded tcp up test, based on what I saw in the driver.
>
> turn off sqm, stay at 100mbit, let fq_codel ride, try the rrul_50_up
> test. Also do a capture to see if CE was ever exerted.

http://swm.pp.se/aqm/rrul_50_up-2015-06-27T072819.166452.iperf3_150626_9_100M.flent.gz

I have a capture of this as well, it's 1.5gigs. It has lots of mentions of
ECT(0) but 0 mention of ECT(1). TOS should be 0x3 when the EC=1 ? I see no
such packets. All the packets are 0x2.

http://swm.pp.se/aqm/rrul_50up-100M.pcap.bz2

I went back to gig and set SQM to 50 megabit/s bidirectional and looking
at tc qdisc -s eth0 I see ecn_mark 10339 on a single queue, none of the
others have non-zero ecn_mark.

> I do not know why my linksys ac1200 build does not work. I generally
> suspect it is because my big build server is getting ancient. Can you
> send me your .config file for openwrt?

http://swm.pp.se/aqm/wrt1200ac.config

This was my first try, I haven't fine tuned it yet, but it works (boots
and moves packets anyway :P )

> It would be VERY helpful if you pulled from ceropackages-3.14, and
> added and built kmod-sched-cake and tc-adv and switched to using the
> ceropackages feed also for luci-app-sqm and sqm-scripts. There are a
> couple people here that would probably leap on that! :)

I will give it a go in the next few days.

--
Mikael Abrahamsson email: ***@swm.pp.se

[Dave Taht]

On Fri, Jun 26, 2015 at 10:50 PM, Mikael Abrahamsson <***@swm.pp.se> wrote:
> On Fri, 26 Jun 2015, Dave Taht wrote:
>
>> your results are showing basically tail drop behavior. Although I
>> would have expected intrinsic delay on the link to crack 100mbits on
>> the rrul test, not 20ms (which is still high), and you only hit 7 on
>> the single threaded tcp up test, based on what I saw in the driver.
>>
>> turn off sqm, stay at 100mbit, let fq_codel ride, try the rrul_50_up
>> test. Also do a capture to see if CE was ever exerted.
>
>
> http://swm.pp.se/aqm/rrul_50_up-2015-06-27T072819.166452.iperf3_150626_9_100M.flent.gz
>
> I have a capture of this as well, it's 1.5gigs. It has lots of mentions of
> ECT(0) but 0 mention of ECT(1). TOS should be 0x3 when the EC=1 ? I see no
> such packets. All the packets are 0x2.
>
> http://swm.pp.se/aqm/rrul_50up-100M.pcap.bz2

Maybe you are dropping at the internal switch? A lot of manufacturers
ran everything through the switch, even the uplink, in recent years.
This will make things hard to fix.

> I went back to gig and set SQM to 50 megabit/s bidirectional and looking at
> tc qdisc -s eth0 I see ecn_mark 10339 on a single queue, none of the others
> have non-zero ecn_mark.

Yea, well, you need to use a diffserv enabled test to see marks or
drops in other queues.

Sort of my hope is that cake can run at 940mbit with software rate
limiting. But we probably need to find ways of parallizing the needed
softirq.

>
>> I do not know why my linksys ac1200 build does not work. I generally
>> suspect it is because my big build server is getting ancient. Can you
>> send me your .config file for openwrt?
>
>
> http://swm.pp.se/aqm/wrt1200ac.config
>
> This was my first try, I haven't fine tuned it yet, but it works (boots and
> moves packets anyway :P )

Must be nice to win on the first try.

>
>> It would be VERY helpful if you pulled from ceropackages-3.14, and
>> added and built kmod-sched-cake and tc-adv and switched to using the
>> ceropackages feed also for luci-app-sqm and sqm-scripts. There are a
>> couple people here that would probably leap on that! :)
>
>
> I will give it a go in the next few days.
>
>
> --
> Mikael Abrahamsson email: ***@swm.pp.se



--
Dave Täht
worldwide bufferbloat report:
http://www.dslreports.com/speedtest/results/bufferbloat
And:
What will it take to vastly improve wifi for everyone?
https://plus.google.com/u/0/explore/makewififast

[Mikael Abrahamsson]

On Sat, 27 Jun 2015, Dave Taht wrote:

> Maybe you are dropping at the internal switch? A lot of manufacturers
> ran everything through the switch, even the uplink, in recent years.
> This will make things hard to fix.

Looking at the vlans and pvids it looks like they've connected the SoC to
port 5 and 6 on the switch, port 4 is the "Internet" port, and 0-3 is
LAN1-4 port. Sigh, I had hoped at least eth0 (WAN/Internet) was a
physically dedicated port.

***@OpenWrt:~# swconfig dev switch0 show
Global attributes:
enable_vlan: 0
Port 0:
mask: 0x004e: (0) 1 2 3 6
qmode: 0
status: link: up, speed: 1000 Mbps, duplex: full
link: 1000
pvid: 0
Port 1:
mask: 0x004d: 0 (1) 2 3 6
qmode: 0
status: link: down
link: 0
pvid: 0
Port 2:
mask: 0x004b: 0 1 (2) 3 6
qmode: 0
status: link: down
link: 0
pvid: 0
Port 3:
mask: 0x0047: 0 1 2 (3) 6
qmode: 0
status: link: up, speed: 1000 Mbps, duplex: full
link: 1000
pvid: 0
Port 4:
mask: 0x0020: (4) 5
qmode: 0
status: link: up, speed: 1000 Mbps, duplex: full
link: 1000
pvid: 0
Port 5:
mask: 0x0010: 4 (5)
qmode: 0
status: link: up, speed: 1000 Mbps, duplex: full
link: 1000
pvid: 0
Port 6:
mask: 0x000f: 0 1 2 3 (6)
qmode: 0
status: link: up, speed: 1000 Mbps, duplex: full
link: 1000
pvid: 0

--
Mikael Abrahamsson email: ***@swm.pp.se

[Dave Taht]

Sigh.

It is possible, maybe, to build something using this chipset that does
not connect the wan port through the switch, built by someone else.
Maybe some other manufacturer did that. I had first evaluated this
chipset on the dual port mirabox, which as best as I recall had no
switch, two genuine ethernet ports (but it ran WAY too hot and at the
time, the kernel was ancient).

On Sat, Jun 27, 2015 at 11:23 AM, Mikael Abrahamsson <***@swm.pp.se> wrote:
> On Sat, 27 Jun 2015, Dave Taht wrote:
>
>> Maybe you are dropping at the internal switch? A lot of manufacturers
>> ran everything through the switch, even the uplink, in recent years.
>> This will make things hard to fix.
>
>
> Looking at the vlans and pvids it looks like they've connected the SoC to
> port 5 and 6 on the switch, port 4 is the "Internet" port, and 0-3 is LAN1-4
> port. Sigh, I had hoped at least eth0 (WAN/Internet) was a physically
> dedicated port.
>
> ***@OpenWrt:~# swconfig dev switch0 show
> Global attributes:
> enable_vlan: 0
> Port 0:
> mask: 0x004e: (0) 1 2 3 6
> qmode: 0
> status: link: up, speed: 1000 Mbps, duplex: full
> link: 1000
> pvid: 0
> Port 1:
> mask: 0x004d: 0 (1) 2 3 6
> qmode: 0
> status: link: down
> link: 0
> pvid: 0
> Port 2:
> mask: 0x004b: 0 1 (2) 3 6
> qmode: 0
> status: link: down
> link: 0
> pvid: 0
> Port 3:
> mask: 0x0047: 0 1 2 (3) 6
> qmode: 0
> status: link: up, speed: 1000 Mbps, duplex: full
> link: 1000
> pvid: 0
> Port 4:
> mask: 0x0020: (4) 5
> qmode: 0
> status: link: up, speed: 1000 Mbps, duplex: full
> link: 1000
> pvid: 0
> Port 5:
> mask: 0x0010: 4 (5)
> qmode: 0
> status: link: up, speed: 1000 Mbps, duplex: full
> link: 1000
> pvid: 0
> Port 6:
> mask: 0x000f: 0 1 2 3 (6)
> qmode: 0
> status: link: up, speed: 1000 Mbps, duplex: full
> link: 1000
> pvid: 0
>
>
> --
> Mikael Abrahamsson email: ***@swm.pp.se



--
Dave Täht
worldwide bufferbloat report:
http://www.dslreports.com/speedtest/results/bufferbloat
And:
What will it take to vastly improve wifi for everyone?
https://plus.google.com/u/0/explore/makewififast

[Sebastian Moeller]

Hi Dave, hi Mikael,

On Jun 27, 2015, at 19:59 , Dave Taht <***@gmail.com> wrote:

>> [...]
>
> Yea, well, you need to use a diffserv enabled test to see marks or
> drops in other queues.
>
> Sort of my hope is that cake can run at 940mbit with software rate
> limiting. But we probably need to find ways of parallizing the needed
> softirq.
[…]

If we go that route we need to remember adding the 20 bytes worth of inter frame gap and preamble to the per packet overhead (in addition to the 18 bytes of “normal” ethernet header with frame check sequence), otherwise the shaper will not do what we expect it to at small packet sizes.

Best Regards
Sebastian

[Mikael Abrahamsson]

On Fri, 26 Jun 2015, Dave Taht wrote:

> src-git https://github.com/dtaht/ceropackages-3.10.git
>
> ./scripts feeds update
> ./scripts feeds install kmod-sched-cake tc-adv kmod-sched-fq_pie
> edit the .config file to make them modules or installed by default
> make menuconfig then save
> make

I have now done this. I have sch_cake, and I have a tc I can add an qdisc
"cake" with to for instance eth1 and see stats from it with "tc -s qdisc"

I however do not have anything "cake" in luci-app-sqm. I see luci-app-sqm
in my "cero" feed, but it seems to install luci-app-sqm from regular
OpenWrt instead.

So either if someone has any tips on how to fix this, or can just give me
how to configure qdiscs manually, I will be able to do cake testing on the
WRT1200AC.

--
Mikael Abrahamsson email: ***@swm.pp.se

[Sebastian Moeller]

Hi Mikael,


On Jun 28, 2015, at 09:06 , Mikael Abrahamsson <***@swm.pp.se> wrote:

> On Fri, 26 Jun 2015, Dave Taht wrote:
>
>> src-git https://github.com/dtaht/ceropackages-3.10.git
>>
>> ./scripts feeds update
>> ./scripts feeds install kmod-sched-cake tc-adv kmod-sched-fq_pie
>> edit the .config file to make them modules or installed by default
>> make menuconfig then save
>> make
>
> I have now done this. I have sch_cake, and I have a tc I can add an qdisc "cake" with to for instance eth1 and see stats from it with "tc -s qdisc"
>
> I however do not have anything "cake" in luci-app-sqm. I see luci-app-sqm in my "cero" feed, but it seems to install luci-app-sqm from regular OpenWrt instead.

Ah, I see the latest changes have not yet made it into the openwrt repository (due to lack of testing). As I do not have permissions/authority to push changes into the openwrt repository, there is nothing I can do to expedite the process. You could replace /usr/lib/lua/luci/model/cbi/sqm.lua on your router with the following (attached file)

[Mikael Abrahamsson]

On Sun, 28 Jun 2015, Sebastian Moeller wrote:

> Ah, I see the latest changes have not yet made it into the openwrt
> repository (due to lack of testing). As I do not have
> permissions/authority to push changes into the openwrt repository, there
> is nothing I can do to expedite the process. You could replace
> /usr/lib/lua/luci/model/cbi/sqm.lua on your router with the following
> (attached file)

Ok, I have done that, run a battery of tests both at 50/50 and 500/500
with cake.

http://swm.pp.se/aqm/wrt1200ac-flent-cake-150628-1.tar

--
Mikael Abrahamsson email: ***@swm.pp.se

[Sebastian Moeller]

Hi Mikael,

On Jun 28, 2015, at 12:29 , Mikael Abrahamsson <***@swm.pp.se> wrote:

> On Sun, 28 Jun 2015, Sebastian Moeller wrote:
>
>> Ah, I see the latest changes have not yet made it into the openwrt repository (due to lack of testing). As I do not have permissions/authority to push changes into the openwrt repository, there is nothing I can do to expedite the process. You could replace /usr/lib/lua/luci/model/cbi/sqm.lua on your router with the following (attached file)
>
> Ok, I have done that, run a battery of tests both at 50/50 and 500/500 with cake.
>
> http://swm.pp.se/aqm/wrt1200ac-flent-cake-150628-1.tar

This looks great, could you by any chance confirm that the GUI does allow to configure cake and that you can or can not set the overhead for cake in the link layer adjustments (LLA) tab? (select cake as link layer adjustment method, and put 42 into the overhead field and report the output of “tc -d qdisc” before and after selecting cake as LLA). @Toke: If that works, I think we can safely push these changes into the openwrt repositories...


Best Regards
Sebastian

>
> --
> Mikael Abrahamsson email: ***@swm.pp.se

[Mikael Abrahamsson]

On Sun, 28 Jun 2015, Sebastian Moeller wrote:

> This looks great, could you by any chance confirm that the GUI
> does allow to configure cake and that you can or can not set the
> overhead for cake in the link layer adjustments (LLA) tab? (select cake
> as link layer adjustment method, and put 42 into the overhead field and
> report the output of “tc -d qdisc” before and after selecting cake as
> LLA). @Toke: If that works, I think we can safely push these changes
> into the openwrt repositories...

Here is the output. What I don't see is both ingress and egress ECN
markings even though I have selected this in the advanced configuration
under Queue Discipline.

Before changing LLA:

***@OpenWrt:~# tc -d qdisc
qdisc htb 1: dev eth0 root refcnt 9 r2q 10 default 12 direct_packets_stat
0 ver 3.17 direct_qlen 532
qdisc cake 110: dev eth0 parent 1:11 unlimited diffserv4 flows raw
qdisc cake 120: dev eth0 parent 1:12 unlimited diffserv4 flows raw
qdisc cake 130: dev eth0 parent 1:13 unlimited diffserv4 flows raw
qdisc ingress ffff: dev eth0 parent ffff:fff1 ----------------
qdisc mq 0: dev eth1 root
qdisc fq_codel 0: dev eth1 parent :1 limit 1024p flows 1024 quantum 300
target 5.0ms interval 100.0ms ecn
qdisc fq_codel 0: dev eth1 parent :2 limit 1024p flows 1024 quantum 300
target 5.0ms interval 100.0ms ecn
qdisc fq_codel 0: dev eth1 parent :3 limit 1024p flows 1024 quantum 300
target 5.0ms interval 100.0ms ecn
qdisc fq_codel 0: dev eth1 parent :4 limit 1024p flows 1024 quantum 300
target 5.0ms interval 100.0ms ecn
qdisc fq_codel 0: dev eth1 parent :5 limit 1024p flows 1024 quantum 300
target 5.0ms interval 100.0ms ecn
qdisc fq_codel 0: dev eth1 parent :6 limit 1024p flows 1024 quantum 300
target 5.0ms interval 100.0ms ecn
qdisc fq_codel 0: dev eth1 parent :7 limit 1024p flows 1024 quantum 300
target 5.0ms interval 100.0ms ecn
qdisc fq_codel 0: dev eth1 parent :8 limit 1024p flows 1024 quantum 300
target 5.0ms interval 100.0ms ecn
qdisc htb 1: dev ifb4eth0 root refcnt 2 r2q 10 default 12
direct_packets_stat 0 ver 3.17 direct_qlen 32
qdisc cake 110: dev ifb4eth0 parent 1:11 unlimited diffserv4 flows raw
qdisc cake 120: dev ifb4eth0 parent 1:12 unlimited diffserv4 flows raw
qdisc cake 130: dev ifb4eth0 parent 1:13 unlimited diffserv4 flows raw

After changing LLA:

***@OpenWrt:~# tc -d qdisc
qdisc htb 1: dev eth0 root refcnt 9 r2q 10 default 12 direct_packets_stat
0 ver 3.17 direct_qlen 532
linklayer ethernet overhead 42
qdisc cake 110: dev eth0 parent 1:11 unlimited diffserv4 flows raw
qdisc cake 120: dev eth0 parent 1:12 unlimited diffserv4 flows raw
qdisc cake 130: dev eth0 parent 1:13 unlimited diffserv4 flows raw
qdisc ingress ffff: dev eth0 parent ffff:fff1 ----------------
qdisc mq 0: dev eth1 root
qdisc fq_codel 0: dev eth1 parent :1 limit 1024p flows 1024 quantum 300
target 5.0ms interval 100.0ms ecn
qdisc fq_codel 0: dev eth1 parent :2 limit 1024p flows 1024 quantum 300
target 5.0ms interval 100.0ms ecn
qdisc fq_codel 0: dev eth1 parent :3 limit 1024p flows 1024 quantum 300
target 5.0ms interval 100.0ms ecn
qdisc fq_codel 0: dev eth1 parent :4 limit 1024p flows 1024 quantum 300
target 5.0ms interval 100.0ms ecn
qdisc fq_codel 0: dev eth1 parent :5 limit 1024p flows 1024 quantum 300
target 5.0ms interval 100.0ms ecn
qdisc fq_codel 0: dev eth1 parent :6 limit 1024p flows 1024 quantum 300
target 5.0ms interval 100.0ms ecn
qdisc fq_codel 0: dev eth1 parent :7 limit 1024p flows 1024 quantum 300
target 5.0ms interval 100.0ms ecn
qdisc fq_codel 0: dev eth1 parent :8 limit 1024p flows 1024 quantum 300
target 5.0ms interval 100.0ms ecn
qdisc htb 1: dev ifb4eth0 root refcnt 2 r2q 10 default 12
direct_packets_stat 0 ver 3.17 direct_qlen 32
linklayer ethernet overhead 42
qdisc cake 110: dev ifb4eth0 parent 1:11 unlimited diffserv4 flows raw
qdisc cake 120: dev ifb4eth0 parent 1:12 unlimited diffserv4 flows raw
qdisc cake 130: dev ifb4eth0 parent 1:13 unlimited diffserv4 flows raw

--
Mikael Abrahamsson email: ***@swm.pp.se

[Jonathan Morton]

To be honest, HTB + cake isn't really the preferred configuration.

- Jonathan Morton

[Dave Taht]

htb + cake is the wrong configuration. :)

cake has an integral bandwidth limiter and diffserv, there should be
no htb here and it looked like their was. I have only been using the
"simplest" qdisc, possibly the script for the simple qdisc is wrong?

On Sun, Jun 28, 2015 at 10:58 AM, Jonathan Morton <***@gmail.com> wrote:
> To be honest, HTB + cake isn't really the preferred configuration.
>
> - Jonathan Morton
>
>
> _______________________________________________
> Cerowrt-devel mailing list
> Cerowrt-***@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cerowrt-devel
>



--
Dave Täht
worldwide bufferbloat report:
http://www.dslreports.com/speedtest/results/bufferbloat
And:
What will it take to vastly improve wifi for everyone?
https://plus.google.com/u/0/explore/makewififast

[Sebastian Moeller]

Hi Dave, hi List,

On Jun 28, 2015, at 20:04 , Dave Taht <***@gmail.com> wrote:

> htb + cake is the wrong configuration. :)

“Wrong” might be a bit hard, but certainly not the preferred solution. During my tests on your box simple.qos resulted in the following:
***@davedesk2:/usr/lib/sqm# tc -d qdisc
qdisc fq_codel 0: dev eth0 root refcnt 2 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
qdisc cake 8030: dev eth1 root refcnt 2 bandwidth 4500Kbit diffserv4 flows raw
qdisc ingress ffff: dev eth1 parent ffff:fff1 ----------------
qdisc cake 8031: dev ifb4eth1 root refcnt 2 bandwidth 45Mbit besteffort flows atm overhead 0
qdisc bfifo 800b: dev wlan1 root refcnt 5 limit 16Kb
qdisc cake 8008: dev wlan0 root refcnt 5 unlimited diffserv4 flows raw

So either I mixed things up later or Mikael’s simple.qos somehow got stale. Anyway, there is work thee for me to do to fix this up properly.

Best Regards
Sebastian



>
> cake has an integral bandwidth limiter and diffserv, there should be
> no htb here and it looked like their was. I have only been using the
> "simplest" qdisc, possibly the script for the simple qdisc is wrong?
>
> On Sun, Jun 28, 2015 at 10:58 AM, Jonathan Morton <***@gmail.com> wrote:
>> To be honest, HTB + cake isn't really the preferred configuration.
>>
>> - Jonathan Morton
>>
>>
>> _______________________________________________
>> Cerowrt-devel mailing list
>> Cerowrt-***@lists.bufferbloat.net
>> https://lists.bufferbloat.net/listinfo/cerowrt-devel
>>
>
>
>
> --
> Dave Täht
> worldwide bufferbloat report:
> http://www.dslreports.com/speedtest/results/bufferbloat
> And:
> What will it take to vastly improve wifi for everyone?
> https://plus.google.com/u/0/explore/makewififast
> _______________________________________________
> Cerowrt-devel mailing list
> Cerowrt-***@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cerowrt-devel

[Mikael Abrahamsson]

On Sun, 28 Jun 2015, Sebastian Moeller wrote:

> So either I mixed things up later or Mikael’s simple.qos somehow got
> stale. Anyway, there is work thee for me to do to fix this up properly.

I believe this is simple.qos from whatever nightly OpenWRT build there is.
It's quite likely it didn't pull any of the sqm-scripts from the CeroWrt
feed at all.

--
Mikael Abrahamsson email: ***@swm.pp.se

[Sebastian Moeller]

Hi Mikael,

[Mikael Abrahamsson]

On Sun, 28 Jun 2015, Sebastian Moeller wrote:

> Hi Mikael,

***@OpenWrt:~# tc -d qdisc
qdisc htb 1: dev eth0 root refcnt 9 r2q 10 default 12 direct_packets_stat
0 ver 3.17 direct_qlen 532
qdisc fq_codel 110: dev eth0 parent 1:11 limit 1001p flows 1024 quantum
300 target 5.0ms interval 100.0ms ecn
qdisc fq_codel 120: dev eth0 parent 1:12 limit 1001p flows 1024 quantum
300 target 5.0ms interval 100.0ms ecn
qdisc fq_codel 130: dev eth0 parent 1:13 limit 1001p flows 1024 quantum
300 target 5.0ms interval 100.0ms ecn
qdisc ingress ffff: dev eth0 parent ffff:fff1 ----------------
qdisc mq 0: dev eth1 root
qdisc fq_codel 0: dev eth1 parent :1 limit 1024p flows 1024 quantum 300
target 5.0ms interval 100.0ms ecn
qdisc fq_codel 0: dev eth1 parent :2 limit 1024p flows 1024 quantum 300
target 5.0ms interval 100.0ms ecn
qdisc fq_codel 0: dev eth1 parent :3 limit 1024p flows 1024 quantum 300
target 5.0ms interval 100.0ms ecn
qdisc fq_codel 0: dev eth1 parent :4 limit 1024p flows 1024 quantum 300
target 5.0ms interval 100.0ms ecn
qdisc fq_codel 0: dev eth1 parent :5 limit 1024p flows 1024 quantum 300
target 5.0ms interval 100.0ms ecn
qdisc fq_codel 0: dev eth1 parent :6 limit 1024p flows 1024 quantum 300
target 5.0ms interval 100.0ms ecn
qdisc fq_codel 0: dev eth1 parent :7 limit 1024p flows 1024 quantum 300
target 5.0ms interval 100.0ms ecn
qdisc fq_codel 0: dev eth1 parent :8 limit 1024p flows 1024 quantum 300
target 5.0ms interval 100.0ms ecn
qdisc htb 1: dev ifb4eth0 root refcnt 2 r2q 10 default 12
direct_packets_stat 0 ver 3.17 direct_qlen 32
qdisc fq_codel 110: dev ifb4eth0 parent 1:11 limit 1001p flows 1024
quantum 500 target 5.0ms interval 100.0ms ecn
qdisc fq_codel 120: dev ifb4eth0 parent 1:12 limit 1001p flows 1024
quantum 1500 target 5.0ms interval 100.0ms ecn
qdisc fq_codel 130: dev ifb4eth0 parent 1:13 limit 1001p flows 1024
quantum 300 target 5.0ms interval 100.0ms ecn

then I change to cake and add 42 as overhead:

***@OpenWrt:~# tc -d qdisc
qdisc cake 8003: dev eth0 root refcnt 9 bandwidth 500Mbit diffserv4 flows
raw
linklayer ethernet overhead 42
qdisc ingress ffff: dev eth0 parent ffff:fff1 ----------------
qdisc mq 0: dev eth1 root
qdisc fq_codel 0: dev eth1 parent :1 limit 1024p flows 1024 quantum 300
target 5.0ms interval 100.0ms ecn
qdisc fq_codel 0: dev eth1 parent :2 limit 1024p flows 1024 quantum 300
target 5.0ms interval 100.0ms ecn
qdisc fq_codel 0: dev eth1 parent :3 limit 1024p flows 1024 quantum 300
target 5.0ms interval 100.0ms ecn
qdisc fq_codel 0: dev eth1 parent :4 limit 1024p flows 1024 quantum 300
target 5.0ms interval 100.0ms ecn
qdisc fq_codel 0: dev eth1 parent :5 limit 1024p flows 1024 quantum 300
target 5.0ms interval 100.0ms ecn
qdisc fq_codel 0: dev eth1 parent :6 limit 1024p flows 1024 quantum 300
target 5.0ms interval 100.0ms ecn
qdisc fq_codel 0: dev eth1 parent :7 limit 1024p flows 1024 quantum 300
target 5.0ms interval 100.0ms ecn
qdisc fq_codel 0: dev eth1 parent :8 limit 1024p flows 1024 quantum 300
target 5.0ms interval 100.0ms ecn
qdisc fq_codel 0: dev ifb4eth0 root refcnt 2 limit 1024p flows 1024
quantum 300 target 5.0ms interval 100.0ms ecn

***@OpenWrt:~# tc -d class show dev eth0
class cake 8003:508 parent 8003:
class cake 8003:944 parent 8003:
***@OpenWrt:~# tc -d class show dev ifb4eth0
***@OpenWrt:~#

--
Mikael Abrahamsson email: ***@swm.pp.se

[Sebastian Moeller]

Hi Mikael,

On Jun 28, 2015, at 22:48 , Mikael Abrahamsson <***@swm.pp.se> wrote:

> On Sun, 28 Jun 2015, Sebastian Moeller wrote:
>
>> Hi Mikael,
>
> ***@OpenWrt:~# tc -d qdisc
> qdisc htb 1: dev eth0 root refcnt 9 r2q 10 default 12 direct_packets_stat 0 ver 3.17 direct_qlen 532
> qdisc fq_codel 110: dev eth0 parent 1:11 limit 1001p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
> qdisc fq_codel 120: dev eth0 parent 1:12 limit 1001p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
> qdisc fq_codel 130: dev eth0 parent 1:13 limit 1001p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
> qdisc ingress ffff: dev eth0 parent ffff:fff1 ----------------
> qdisc mq 0: dev eth1 root
> qdisc fq_codel 0: dev eth1 parent :1 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
> qdisc fq_codel 0: dev eth1 parent :2 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
> qdisc fq_codel 0: dev eth1 parent :3 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
> qdisc fq_codel 0: dev eth1 parent :4 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
> qdisc fq_codel 0: dev eth1 parent :5 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
> qdisc fq_codel 0: dev eth1 parent :6 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
> qdisc fq_codel 0: dev eth1 parent :7 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
> qdisc fq_codel 0: dev eth1 parent :8 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
> qdisc htb 1: dev ifb4eth0 root refcnt 2 r2q 10 default 12 direct_packets_stat 0 ver 3.17 direct_qlen 32
> qdisc fq_codel 110: dev ifb4eth0 parent 1:11 limit 1001p flows 1024 quantum 500 target 5.0ms interval 100.0ms ecn
> qdisc fq_codel 120: dev ifb4eth0 parent 1:12 limit 1001p flows 1024 quantum 1500 target 5.0ms interval 100.0ms ecn
> qdisc fq_codel 130: dev ifb4eth0 parent 1:13 limit 1001p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
>
> then I change to cake and add 42 as overhead:
>
> ***@OpenWrt:~# tc -d qdisc
> qdisc cake 8003: dev eth0 root refcnt 9 bandwidth 500Mbit diffserv4 flows raw

Cake still did not take the overhead argument (or raw would have gone from the output)

> linklayer ethernet overhead 42

This looks a bit like the output I would expect from using tic’s stab mechanism to define link layer adjustments and per packet overhead.

Could you post the result of:

cat /etc/config/sqm

from the router, please? And the output of

/etc/init.d/sqm stop ; /etc/init.d/sqm start

this might give me a clue where to look...





> qdisc ingress ffff: dev eth0 parent ffff:fff1 ----------------
> qdisc mq 0: dev eth1 root
> qdisc fq_codel 0: dev eth1 parent :1 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
> qdisc fq_codel 0: dev eth1 parent :2 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
> qdisc fq_codel 0: dev eth1 parent :3 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
> qdisc fq_codel 0: dev eth1 parent :4 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
> qdisc fq_codel 0: dev eth1 parent :5 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
> qdisc fq_codel 0: dev eth1 parent :6 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
> qdisc fq_codel 0: dev eth1 parent :7 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
> qdisc fq_codel 0: dev eth1 parent :8 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
> qdisc fq_codel 0: dev ifb4eth0 root refcnt 2 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn

The ingress setup did not go through, which most likely means tc got unhappy about one of the options.

Thanks for your time in testing this, I fear sqm-scripts is not really in a good shape for cake testing yet.

Best Regars
Sebastian

>
> ***@OpenWrt:~# tc -d class show dev eth0
> class cake 8003:508 parent 8003:
> class cake 8003:944 parent 8003:
> ***@OpenWrt:~# tc -d class show dev ifb4eth0
> ***@OpenWrt:~#
>
> --
> Mikael Abrahamsson email: ***@swm.pp.se

[Mikael Abrahamsson]

On Sun, 28 Jun 2015, Sebastian Moeller wrote:

> Could you post the result of:
>
> cat /etc/config/sqm
>
> from the router, please? And the output of
>
> /etc/init.d/sqm stop ; /etc/init.d/sqm start
>
> this might give me a clue where to look...

***@OpenWrt:~# cat /etc/config/sqm

config queue 'eth1'
option script 'simple.qos'
option interface 'eth0'
option qdisc_advanced '1'
option squash_dscp '0'
option squash_ingress '0'
option ingress_ecn 'ECN'
option egress_ecn 'ECN'
option qdisc_really_really_advanced '0'
option enabled '1'
option download '500000'
option upload '500000'
option qdisc 'cake'
option linklayer 'ethernet'
option overhead '42'

***@OpenWrt:~# /etc/init.d/sqm stop
SQM: Trying to start/stop SQM on all interfaces.
SQM: run.sh stop
SQM: /usr/lib/sqm/run.sh Stopping SQM on interface: eth0
SQM: ifb associated with interface eth0:
SQM: trying to create new IFB: ifb4eth0
RTNETLINK answers: File exists
SQM: /usr/lib/sqm/stop.sh: Stopping eth0
SQM: ifb associated with interface eth0:
SQM: /usr/lib/sqm/run.sh SQM qdiscs on eth0 removed
***@OpenWrt:~# /etc/init.d/sqm start
SQM: Trying to start/stop SQM on all interfaces.
SQM: /usr/lib/sqm/run.sh Queue Setup Script: /usr/lib/sqm/simple.qos
SQM: ifb associated with interface eth0:
SQM: trying to create new IFB: ifb4eth0
RTNETLINK answers: File exists
SQM: cake
SQM: Keeping differentiated services code points (DSCP) from ingress.
SQM: STAB: stab mtu 2047 tsize 512 mpu 0 overhead 42 linklayer ethernet

This command never completes. I added -x to simple.qos and the RTNETLINK
is from:

+ /usr/sbin/ip link add name ifb4eth0 type ifb
RTNETLINK answers: File exists

and the last commands that execute are:

SQM: STAB: stab mtu 2047 tsize 512 mpu 0 overhead 42 linklayer ethernet
+ echo stab mtu 2047 tsize 512 mpu 0 overhead 42 linklayer ethernet
+ get_cake_lla_string
+ STABSTRING=
+ [ tc_stab = cake -a ethernet != none ]
+ sqm_logger
+ logger -t SQM -s

--
Mikael Abrahamsson email: ***@swm.pp.se

[Mikael Abrahamsson]

On Mon, 29 Jun 2015, Mikael Abrahamsson wrote:

> SQM: STAB: stab mtu 2047 tsize 512 mpu 0 overhead 42 linklayer ethernet
> + echo stab mtu 2047 tsize 512 mpu 0 overhead 42 linklayer ethernet
> + get_cake_lla_string
> + STABSTRING=
> + [ tc_stab = cake -a ethernet != none ]
> + sqm_logger
> + logger -t SQM -s

If I modify line 155 (the sqm_logger "${STABSTRING}" line) to add SQM:
as well within the quotes so it can't get passed an empty string, the
command completes.

--
Mikael Abrahamsson email: ***@swm.pp.se

[Sebastian Moeller]

HI MIkael,


On Jun 29, 2015, at 07:11 , Mikael Abrahamsson <***@swm.pp.se> wrote:

> On Mon, 29 Jun 2015, Mikael Abrahamsson wrote:
>
>> SQM: STAB: stab mtu 2047 tsize 512 mpu 0 overhead 42 linklayer ethernet
>> + echo stab mtu 2047 tsize 512 mpu 0 overhead 42 linklayer ethernet
>> + get_cake_lla_string
>> + STABSTRING=
>> + [ tc_stab = cake -a ethernet != none ]
>> + sqm_logger
>> + logger -t SQM -s
>
> If I modify line 155 (the sqm_logger "${STABSTRING}" line) to add SQM: as well within the quotes so it can't get passed an empty string, the command completes.

Good work-around; not a real solution as sqm_logger() will also add "SQM:”. I think the solution most likely is to remove line 155 completely. But weirdly this seems to work with cerowrt…

Best Regards
Sebastian

>
> --
> Mikael Abrahamsson email: ***@swm.pp.se

[Mikael Abrahamsson]

On Mon, 29 Jun 2015, Sebastian Moeller wrote:

> Good work-around; not a real solution as sqm_logger() will also
> add "SQM:”. I think the solution most likely is to remove line 155
> completely. But weirdly this seems to work with cerowrt…

May I suggest a wrapper around the logger part that handles this case, ie
if the string is empty? Because when the string is empty and it gets
stuck, luci just waits and waits and waits. I mean, this specific case can
be fixed, but for the future it would be more robust if this error case
can't happen at all.

--
Mikael Abrahamsson email: ***@swm.pp.se

[Sebastian Moeller]

HI Mikael,


On Jun 29, 2015, at 09:54 , Mikael Abrahamsson <***@swm.pp.se> wrote:

> On Mon, 29 Jun 2015, Sebastian Moeller wrote:
>
>> Good work-around; not a real solution as sqm_logger() will also add "SQM:”. I think the solution most likely is to remove line 155 completely. But weirdly this seems to work with cerowrt…
>
> May I suggest a wrapper around the logger part that handles this case, ie if the string is empty? Because when the string is empty and it gets stuck, luci just waits and waits and waits. I mean, this specific case can be fixed, but for the future it would be more robust if this error case can't happen at all.

Sounds reasonably defensive; what irk me is that in my testbed that specific error does not crop up; one more reason to code defensive I guess…

Best Regards
Sebastian

>
> --
> Mikael Abrahamsson email: ***@swm.pp.se

[Sebastian Moeller]

HI Mikael,


On Jun 29, 2015, at 09:54 , Mikael Abrahamsson <***@swm.pp.se> wrote:

> On Mon, 29 Jun 2015, Sebastian Moeller wrote:
>
>> Good work-around; not a real solution as sqm_logger() will also add "SQM:”. I think the solution most likely is to remove line 155 completely. But weirdly this seems to work with cerowrt…

Correction, calling logger with an empty string works on cero (as well as on openwrt), but passing an empty string to sqm_logger results in the observed bug, so I can reproduce this locally.

>
> May I suggest a wrapper around the logger part that handles this case, ie if the string is empty? Because when the string is empty and it gets stuck, luci just waits and waits and waits. I mean, this specific case can be fixed, but for the future it would be more robust if this error case can't happen at all.
>
> --
> Mikael Abrahamsson email: ***@swm.pp.se

I had a look at sqm_logger():
sqm_logger() {
logger -t SQM -s ${1}
}

and hope that
sqm_logger() {
logger -t SQM -s "${1}”
}

Fixes the issue for me and is more concise than trying to introduce real error handling in sqm_logger. Does this also work for you?

Best Regards
Sebastian

[Mikael Abrahamsson]

On Mon, 29 Jun 2015, Sebastian Moeller wrote:

> and hope that
> sqm_logger() {
> logger -t SQM -s "${1}”
> }
>
> Fixes the issue for me and is more concise than trying to introduce real error handling in sqm_logger. Does this also work for you?

Yes, I reverted my previous change, reproduced the hang, then changed the
above, which does not hang. Seems like the most elegant way to solve this!

Thanks!

--
Mikael Abrahamsson email: ***@swm.pp.se

[Sebastian Moeller]

On Jun 29, 2015, at 10:17 , Mikael Abrahamsson <***@swm.pp.se> wrote:

> On Mon, 29 Jun 2015, Sebastian Moeller wrote:
>
>> and hope that
>> sqm_logger() {
>> logger -t SQM -s "${1}”
>> }
>>
>> Fixes the issue for me and is more concise than trying to introduce real error handling in sqm_logger. Does this also work for you?
>
> Yes, I reverted my previous change, reproduced the hang, then changed the above, which does not hang. Seems like the most elegant way to solve this!

Thanks for finding, reporting, debugging, fixing, and testing this. This change is committed to the ceropackages 3.10 repository.

Best Regards
Sebastian

>
> Thanks!
>
> --
> Mikael Abrahamsson email: ***@swm.pp.se

[Sebastian Moeller]

Hi Mikael,


On Jun 29, 2015, at 06:58 , Mikael Abrahamsson <***@swm.pp.se> wrote:

> On Sun, 28 Jun 2015, Sebastian Moeller wrote:
>
>> Could you post the result of:
>>
>> cat /etc/config/sqm
>>
>> from the router, please? And the output of
>>
>> /etc/init.d/sqm stop ; /etc/init.d/sqm start
>>
>> this might give me a clue where to look...
>
> ***@OpenWrt:~# cat /etc/config/sqm
>
> config queue 'eth1'
> option script 'simple.qos'
> option interface 'eth0'
> option qdisc_advanced '1'
> option squash_dscp '0'
> option squash_ingress '0'
> option ingress_ecn 'ECN'
> option egress_ecn 'ECN'
> option qdisc_really_really_advanced '0'
> option enabled '1'
> option download '500000'
> option upload '500000'
> option qdisc 'cake'
> option linklayer 'ethernet'
> option overhead ’42'

Ah, I see, you are still using tc’s stab mechanism for account of per packet overhead and link layer adjustments instead of cake’s (you need to check the advanced options check box in the link layer adjustments tab, and then select “cake” as the lnk layer adjustment mechanism).

>
> ***@OpenWrt:~# /etc/init.d/sqm stop
> SQM: Trying to start/stop SQM on all interfaces.
> SQM: run.sh stop
> SQM: /usr/lib/sqm/run.sh Stopping SQM on interface: eth0
> SQM: ifb associated with interface eth0:
> SQM: trying to create new IFB: ifb4eth0
> RTNETLINK answers: File exists

We can ignore this as it is cosmetic.

> SQM: /usr/lib/sqm/stop.sh: Stopping eth0
> SQM: ifb associated with interface eth0:
> SQM: /usr/lib/sqm/run.sh SQM qdiscs on eth0 removed
> ***@OpenWrt:~# /etc/init.d/sqm start
> SQM: Trying to start/stop SQM on all interfaces.
> SQM: /usr/lib/sqm/run.sh Queue Setup Script: /usr/lib/sqm/simple.qos
> SQM: ifb associated with interface eth0:
> SQM: trying to create new IFB: ifb4eth0
> RTNETLINK answers: File exists
> SQM: cake
> SQM: Keeping differentiated services code points (DSCP) from ingress.
> SQM: STAB: stab mtu 2047 tsize 512 mpu 0 overhead 42 linklayer ethernet

Again, this is tc’s stab mechanism and not cake’s. But it should not break I guess...

>
> This command never completes. I added -x to simple.qos and the RTNETLINK is from:
>
> + /usr/sbin/ip link add name ifb4eth0 type ifb
> RTNETLINK answers: File exists
>
> and the last commands that execute are:
>
> SQM: STAB: stab mtu 2047 tsize 512 mpu 0 overhead 42 linklayer ethernet
> + echo stab mtu 2047 tsize 512 mpu 0 overhead 42 linklayer ethernet
> + get_cake_lla_string
> + STABSTRING=
> + [ tc_stab = cake -a ethernet != none ]
> + sqm_logger
> + logger -t SQM -s

This is weird, under cerowrt this works; but that call to sqm_logger is not useful anyways, it is either repetitive or empty, so I will just delete it

Best Regards
Sebastian


>
> --
> Mikael Abrahamsson email: ***@swm.pp.se

[Mikael Abrahamsson]

On Mon, 29 Jun 2015, Sebastian Moeller wrote:

> Ah, I see, you are still using tc’s stab mechanism for account of
> per packet overhead and link layer adjustments instead of cake’s (you
> need to check the advanced options check box in the link layer
> adjustments tab, and then select “cake” as the lnk layer adjustment
> mechanism).

Ok, so when I changed "stab" to cake, I get the following, but it still
only shapes one way. If I change to discipline to "fq_codel" or "pie" I
get bidirectonal shaping with otherwise same settings.

***@OpenWrt:~# cat /etc/config/sqm

config queue 'eth1'
option interface 'eth0'
option qdisc_advanced '1'
option squash_dscp '0'
option squash_ingress '0'
option ingress_ecn 'ECN'
option egress_ecn 'ECN'
option qdisc_really_really_advanced '0'
option download '50000'
option upload '50000'
option linklayer 'ethernet'
option overhead '42'
option linklayer_advanced '1'
option tcMTU '2047'
option tcTSIZE '128'
option tcMPU '0'
option linklayer_adaptation_mechanism 'cake'
option enabled '1'
option qdisc 'cake'
option script 'simple.qos'

***@OpenWrt:~# tc -d qdisc
qdisc cake 8002: dev eth0 root refcnt 9 bandwidth 50Mbit diffserv4 flows
noatm overhead 42
qdisc ingress ffff: dev eth0 parent ffff:fff1 ----------------
qdisc mq 0: dev eth1 root
qdisc fq_codel 0: dev eth1 parent :1 limit 1024p flows 1024 quantum 300
target 5.0ms interval 100.0ms ecn
qdisc fq_codel 0: dev eth1 parent :2 limit 1024p flows 1024 quantum 300
target 5.0ms interval 100.0ms ecn
qdisc fq_codel 0: dev eth1 parent :3 limit 1024p flows 1024 quantum 300
target 5.0ms interval 100.0ms ecn
qdisc fq_codel 0: dev eth1 parent :4 limit 1024p flows 1024 quantum 300
target 5.0ms interval 100.0ms ecn
qdisc fq_codel 0: dev eth1 parent :5 limit 1024p flows 1024 quantum 300
target 5.0ms interval 100.0ms ecn
qdisc fq_codel 0: dev eth1 parent :6 limit 1024p flows 1024 quantum 300
target 5.0ms interval 100.0ms ecn
qdisc fq_codel 0: dev eth1 parent :7 limit 1024p flows 1024 quantum 300
target 5.0ms interval 100.0ms ecn
qdisc fq_codel 0: dev eth1 parent :8 limit 1024p flows 1024 quantum 300
target 5.0ms interval 100.0ms ecn
qdisc fq_codel 0: dev ifb4eth0 root refcnt 2 limit 1024p flows 1024
quantum 300 target 5.0ms interval 100.0ms ecn

***@OpenWrt:~# tc -d class show dev eth0
class cake 8002:4e1 parent 8002:
class cake 8002:b89 parent 8002:

***@OpenWrt:~# tc -d class show dev ifb4eth0
class fq_codel :12c parent none
class fq_codel :222 parent none

***@OpenWrt:~# /etc/init.d/sqm restart
SQM: Trying to start/stop SQM on all interfaces.
SQM: /usr/lib/sqm/run.sh Stopping SQM on interface: eth0
SQM: ifb associated with interface eth0:
SQM: trying to create new IFB: ifb4eth0
RTNETLINK answers: File exists
SQM: /usr/lib/sqm/stop.sh: Stopping eth0
SQM: ifb associated with interface eth0:
SQM: /usr/lib/sqm/run.sh Queue Setup Script: /usr/lib/sqm/simple.qos
+ . /usr/lib/sqm/functions.sh
+ [ -z 50000 ]
+ [ -z 50000 ]
+ [ -z eth0 ]
+ [ -z cake ]
+ [ -z cake ]
+ [ -z ethernet ]
+ [ -z 42 ]
+ [ -z 2047 ]
+ [ -z 0 ]
+ [ -z 128 ]
+ [ -z ]
+ AUTOFLOW=0
+ [ -z ]
+ LIMIT=1001
+ [ -z ]
+ ILIMIT=
+ [ -z ]
+ ELIMIT=
+ [ -z ]
+ ITARGET=
+ [ -z ]
+ ETARGET=
+ [ -z ECN ]
+ [ -z ECN ]
+ [ -z 0 ]
+ [ -z 0 ]
+ [ -z ]
+ IQDISC_OPTS=
+ [ -z ]
+ EQDISC_OPTS=
+ [ -z ]
+ which tc
+ TC=/usr/sbin/tc
+ [ -z ]
+ which ip
+ IP=/usr/sbin/ip
+ [ -z ]
+ which insmod
+ INSMOD=/usr/sbin/insmod
+ [ -z ]
+ TARGET=5ms
+ [ -z ]
+ IPT_MASK=0xff
+ [ -z ]
+ IPT_MASK_STRING=/0xff
+ [ -z ]
+ get_ifb_for_if eth0
+ CUR_IF=eth0
+ get_ifb_associated_with_if eth0
+ CUR_IF=eth0
+ tc+ -p filtergrep -o -e ifb[^)]\+
show parent ffff: dev eth0
+ CUR_IFB=
+ sqm_logger ifb associated with interface eth0:
+ logger -t SQM -s ifb associated with interface eth0:
SQM: ifb associated with interface eth0:
+ echo
+ CUR_IFB=
+ [ -z ]
+ create_new_ifb_for_if eth0
+ CUR_IF=eth0
+ MAX_IF_NAME_LENGTH=15
+ IFB_PREFIX=ifb4
+ NEW_IFB=ifb4eth0
+ IFB_NAME_LENGTH=8
+ [ 8 -gt 15 ]
+ sqm_logger trying to create new IFB: ifb4eth0
+ logger -t SQM -s trying to create new IFB: ifb4eth0
SQM: trying to create new IFB: ifb4eth0
+ /usr/sbin/ip link add name ifb4eth0 type ifb
RTNETLINK answers: File exists
+ echo ifb4eth0
+ CUR_IFB=ifb4eth0
+ [ -z ifb4eth0 ]
+ echo ifb4eth0
+ DEV=ifb4eth0
+ do_modules
+ insmod act_ipt
+ lsmod+ grep
-q ^act_ipt
+ insmod sch_cake
+ + grep -q ^sch_cake
lsmod
+ insmod sch_ingress
+ + greplsmod
-q ^sch_ingress
+ insmod act_mirred
+ + greplsmod -q ^act_mirred

+ insmod cls_fw
+ + grep -q ^cls_fw
lsmod
+ insmod sch_htb
+ + lsmodgrep -q ^sch_htb

+ ipt_setup
+ ipt -t mangle -N QOS_MARK_eth0
+ echo -t mangle -N QOS_MARK_eth0
+ sed s/-A/-D/g
+ d=-t mangle -N QOS_MARK_eth0
+ [ -t mangle -N QOS_MARK_eth0 != -t mangle -N QOS_MARK_eth0 ]
+ echo -t mangle -N QOS_MARK_eth0
+ sed s/-I/-D/g
+ d=-t mangle -N QOS_MARK_eth0
+ [ -t mangle -N QOS_MARK_eth0 != -t mangle -N QOS_MARK_eth0 ]
+ iptables -t mangle -N QOS_MARK_eth0
+ ip6tables -t mangle -N QOS_MARK_eth0
+ sqm_logger cake does all the diffserv work - no need for iptables rules
+ logger -t SQM -s cake
SQM: cake
+ [ 0 = 1 ]
+ sqm_logger Keeping differentiated services code points (DSCP) from
ingress.
+ logger -t SQM -s Keeping differentiated services code points (DSCP) from
ingress.
SQM: Keeping differentiated services code points (DSCP) from ingress.
+ CAKE_OPTS=
+ ipt -t mangle -A PREROUTING -i eth0 -m mark --mark 0x00/0xff -g
QOS_MARK_eth0
+ echo+ sed s/-A/-D/g
-t mangle -A PREROUTING -i eth0 -m mark --mark 0x00/0xff -g QOS_MARK_eth0
+ d=-t mangle -D PREROUTING -i eth0 -m mark --mark 0x00/0xff -g
QOS_MARK_eth0
+ [ -t mangle -D PREROUTING -i eth0 -m mark --mark 0x00/0xff -g
QOS_MARK_eth0 != -t mangle -A PREROUTING -i eth0 -m mark --mark 0x00/0xff
-g QOS_MARK_eth0 ]
+ iptables -t mangle -D PREROUTING -i eth0 -m mark --mark 0x00/0xff -g
QOS_MARK_eth0
+ ip6tables -t mangle -D PREROUTING -i eth0 -m mark --mark 0x00/0xff -g
QOS_MARK_eth0
+ echo+ sed s/-I/-D/g
-t mangle -A PREROUTING -i eth0 -m mark --mark 0x00/0xff -g QOS_MARK_eth0
+ d=-t mangle -A PREROUTING -i eth0 -m mark --mark 0x00/0xff -g
QOS_MARK_eth0
+ [ -t mangle -A PREROUTING -i eth0 -m mark --mark 0x00/0xff -g
QOS_MARK_eth0 != -t mangle -A PREROUTING -i eth0 -m mark --mark 0x00/0xff
-g QOS_MARK_eth0 ]
+ iptables -t mangle -A PREROUTING -i eth0 -m mark --mark 0x00/0xff -g
QOS_MARK_eth0
+ ip6tables -t mangle -A PREROUTING -i eth0 -m mark --mark 0x00/0xff -g
QOS_MARK_eth0
+ ipt -t mangle -A POSTROUTING -o eth0 -m mark --mark 0x00/0xff -g
QOS_MARK_eth0
+ echo+ sed s/-A/-D/g
-t mangle -A POSTROUTING -o eth0 -m mark --mark 0x00/0xff -g
QOS_MARK_eth0
+ d=-t mangle -D POSTROUTING -o eth0 -m mark --mark 0x00/0xff -g
QOS_MARK_eth0
+ [ -t mangle -D POSTROUTING -o eth0 -m mark --mark 0x00/0xff -g
QOS_MARK_eth0 != -t mangle -A POSTROUTING -o eth0 -m mark --mark 0x00/0xff
-g QOS_MARK_eth0 ]
+ iptables -t mangle -D POSTROUTING -o eth0 -m mark --mark 0x00/0xff -g
QOS_MARK_eth0
+ ip6tables -t mangle -D POSTROUTING -o eth0 -m mark --mark 0x00/0xff -g
QOS_MARK_eth0
+ echo+ sed s/-I/-D/g
-t mangle -A POSTROUTING -o eth0 -m mark --mark 0x00/0xff -g
QOS_MARK_eth0
+ d=-t mangle -A POSTROUTING -o eth0 -m mark --mark 0x00/0xff -g
QOS_MARK_eth0
+ [ -t mangle -A POSTROUTING -o eth0 -m mark --mark 0x00/0xff -g
QOS_MARK_eth0 != -t mangle -A POSTROUTING -o eth0 -m mark --mark 0x00/0xff
-g QOS_MARK_eth0 ]
+ iptables -t mangle -A POSTROUTING -o eth0 -m mark --mark 0x00/0xff -g
QOS_MARK_eth0
+ ip6tables -t mangle -A POSTROUTING -o eth0 -m mark --mark 0x00/0xff -g
QOS_MARK_eth0
+ ipt -t mangle -I PREROUTING -i vtun+ -p tcp -j MARK --set-mark 0x2/0xff
+ echo+ sed s/-A/-D/g
-t mangle -I PREROUTING -i vtun+ -p tcp -j MARK --set-mark 0x2/0xff
+ d=-t mangle -I PREROUTING -i vtun+ -p tcp -j MARK --set-mark 0x2/0xff
+ [ -t mangle -I PREROUTING -i vtun+ -p tcp -j MARK --set-mark 0x2/0xff !=
-t mangle -I PREROUTING -i vtun+ -p tcp -j MARK --set-mark 0x2/0xff ]
+ echo+ sed s/-I/-D/g
-t mangle -I PREROUTING -i vtun+ -p tcp -j MARK --set-mark 0x2/0xff
+ d=-t mangle -D PREROUTING -i vtun+ -p tcp -j MARK --set-mark 0x2/0xff
+ [ -t mangle -D PREROUTING -i vtun+ -p tcp -j MARK --set-mark 0x2/0xff !=
-t mangle -I PREROUTING -i vtun+ -p tcp -j MARK --set-mark 0x2/0xff ]
+ iptables -t mangle -D PREROUTING -i vtun+ -p tcp -j MARK --set-mark
0x2/0xff
+ ip6tables -t mangle -D PREROUTING -i vtun+ -p tcp -j MARK --set-mark
0x2/0xff
+ iptables -t mangle -I PREROUTING -i vtun+ -p tcp -j MARK --set-mark
0x2/0xff
+ ip6tables -t mangle -I PREROUTING -i vtun+ -p tcp -j MARK --set-mark
0x2/0xff
+ ipt -t mangle -A OUTPUT -p udp -m multiport --ports 123,53 -j DSCP
--set-dscp-class AF42
+ echo+ sed s/-A/-D/g
-t mangle -A OUTPUT -p udp -m multiport --ports 123,53 -j DSCP
--set-dscp-class AF42
+ d=-t mangle -D OUTPUT -p udp -m multiport --ports 123,53 -j DSCP
--set-dscp-class AF42
+ [ -t mangle -D OUTPUT -p udp -m multiport --ports 123,53 -j DSCP
--set-dscp-class AF42 != -t mangle -A OUTPUT -p udp -m multiport --ports
123,53 -j DSCP --set-dscp-class AF42 ]
+ iptables -t mangle -D OUTPUT -p udp -m multiport --ports 123,53 -j DSCP
--set-dscp-class AF42
+ ip6tables -t mangle -D OUTPUT -p udp -m multiport --ports 123,53 -j DSCP
--set-dscp-class AF42
+ echo+ sed s/-I/-D/g
-t mangle -A OUTPUT -p udp -m multiport --ports 123,53 -j DSCP
--set-dscp-class AF42
+ d=-t mangle -A OUTPUT -p udp -m multiport --ports 123,53 -j DSCP
--set-dscp-class AF42
+ [ -t mangle -A OUTPUT -p udp -m multiport --ports 123,53 -j DSCP
--set-dscp-class AF42 != -t mangle -A OUTPUT -p udp -m multiport --ports
123,53 -j DSCP --set-dscp-class AF42 ]
+ iptables -t mangle -A OUTPUT -p udp -m multiport --ports 123,53 -j DSCP
--set-dscp-class AF42
+ ip6tables -t mangle -A OUTPUT -p udp -m multiport --ports 123,53 -j DSCP
--set-dscp-class AF42
+ [ 50000 -ne 0 ]
+ egress
+ CEIL=50000
+ expr 50000 / 3
+ PRIO_RATE=16666
+ expr 50000 / 6
+ BE_RATE=8333
+ expr 50000 / 6
+ BK_RATE=8333
+ expr 50000 - 16
+ BE_CEIL=49984
+ get_mtu eth0 50000
+ BW=50000
+ cat /sys/class/net/eth0/mtu
+ F=1500
+ [ -z 1500 ]
+ [ 50000 -gt 20000 ]
+ F=3000
+ [ 50000 -gt 30000 ]
+ F=6000
+ [ 50000 -gt 40000 ]
+ F=12000
+ [ 50000 -gt 50000 ]
+ [ 50000 -gt 60000 ]
+ [ 50000 -gt 80000 ]
+ echo 12000
+ LQ=quantum 12000
+ /usr/sbin/tc qdisc del dev eth0 root
+ get_stab_string
+ STABSTRING=
+ [ cake = tc_stab -a ethernet != none ]
+ echo
+ get_cake_lla_string
+ STABSTRING=
+ [ cake = cake -a ethernet != none ]
+ [ ethernet = atm ]
+ STABSTRING= overhead 42
+ sqm_logger cake link layer adjustments: overhead 42
+ logger -t SQM -s cake link layer adjustments: overhead 42
SQM: cake link layer adjustments: overhead 42
+ sqm_logger SQM: overhead 42
+ logger -t SQM -s SQM: overhead 42
SQM: SQM: overhead 42
+ echo overhead 42
+ /usr/sbin/tc qdisc add dev eth0 root cake bandwidth 50000kbit overhead
42
+ sqm_logger egress shaping activated
+ logger -t SQM -s egress shaping activated
SQM: egress shaping activated
+ [ 50000 -ne 0 ]
+ ingress
+ CEIL=50000
+ expr 50000 / 3
+ PRIO_RATE=16666
+ expr 50000 / 6
+ BE_RATE=8333
+ expr 50000 / 6
+ BK_RATE=8333
+ expr 50000 - 16
+ BE_CEIL=49984
+ get_mtu eth0 50000
+ BW=50000
+ cat /sys/class/net/eth0/mtu
+ F=1500
+ [ -z 1500 ]
+ [ 50000 -gt 20000 ]
+ F=3000
+ [ 50000 -gt 30000 ]
+ F=6000
+ [ 50000 -gt 40000 ]
+ F=12000
+ e 50000 -gt 50000 ]
+ [ 50000 -gt 60000 ]
+ [ 50000 -gt 80000 ]
+ echo 12000
+ LQ=quantum 12000
+ /usr/sbin/tc qdisc del dev eth0 handle ffff: ingress
+ /usr/sbin/tc qdisc add dev eth0 handle ffff: ingress
+ /usr/sbin/tc qdisc del dev ifb4eth0 root
+ [ 0 = 1 ]
+ sqm_logger Perform DSCP based filtering on ingress. (3-tier
classification)
+ logger -t SQM -s Perform DSCP based filtering on ingress. (3-tier
classification)
SQM: Perform DSCP based filtering on ingress. (3-tier classification)
+ get_stab_string
+ STABSTRING=
+ [ cake = tc_stab -a ethernet != none ]
+ echo
+ get_cake_lla_string
+ STABSTRING=
+ [ cake = cake -a ethernet != none ]
+ [ ethernet = atm ]
+ STABSTRING= overhead 42
+ sqm_logger cake link layer adjustments: overhead 42
+ logger -t SQM -s cake link layer adjustments: overhead 42
SQM: cake link layer adjustments: overhead 42
+ sqm_logger SQM: overhead 42
+ logger -t SQM -s SQM: overhead 42
SQM: SQM: overhead 42
+ echo overhead 42
+ /usr/sbin/tc qdisc add dev eth0 root cake bandwidth 50000kbit overhead
42
RTNETLINK answers: File exists
+ ifconfig ifb4eth0 up
+ /usr/sbin/tc filter add dev eth0 parent ffff: protocol all prio 10 u32
match u32 0 0 flowid 1:1 action mirred egress redirect dev ifb4eth0
+ sqm_logger ingress shaping activated
+ logger -t SQM -s ingress shaping activated
SQM: ingress shaping activated

--
Mikael Abrahamsson email: ***@swm.pp.se

[Sebastian Moeller]

HI Mikael,


On Jun 29, 2015, at 10:09 , Mikael Abrahamsson <***@swm.pp.se> wrote:

> On Mon, 29 Jun 2015, Sebastian Moeller wrote:
>
>> Ah, I see, you are still using tc’s stab mechanism for account of per packet overhead and link layer adjustments instead of cake’s (you need to check the advanced options check box in the link layer adjustments tab, and then select “cake” as the lnk layer adjustment mechanism).
>
> Ok, so when I changed "stab" to cake, I get the following, but it still only shapes one way. If I change to discipline to "fq_codel" or "pie" I get bidirectonal shaping with otherwise same settings.

Probabkr breakage I introduced...

>
> ***@OpenWrt:~# cat /etc/config/sqm
>
> config queue 'eth1'
> option interface 'eth0'
> option qdisc_advanced '1'
> option squash_dscp '0'
> option squash_ingress '0'
> option ingress_ecn 'ECN'
> option egress_ecn 'ECN'
> option qdisc_really_really_advanced '0'
> option download '50000'
> option upload '50000'
> option linklayer 'ethernet'
> option overhead '42'
> option linklayer_advanced '1'
> option tcMTU '2047'
> option tcTSIZE '128'
> option tcMPU ‘0'

we can ignore the last 3 savely.

> option linklayer_adaptation_mechanism 'cake'
> option enabled '1'
> option qdisc 'cake'
> option script 'simple.qos’

Okay, it really is trying cake’s LLA now.

>
> ***@OpenWrt:~# tc -d qdisc
> qdisc cake 8002: dev eth0 root refcnt 9 bandwidth 50Mbit diffserv4 flows noatm overhead 42

And that works on egress, excellent so bot sch_cake and tc are recent enough.

> qdisc ingress ffff: dev eth0 parent ffff:fff1 ----------------
> qdisc mq 0: dev eth1 root
> qdisc fq_codel 0: dev eth1 parent :1 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
> qdisc fq_codel 0: dev eth1 parent :2 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
> qdisc fq_codel 0: dev eth1 parent :3 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
> qdisc fq_codel 0: dev eth1 parent :4 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
> qdisc fq_codel 0: dev eth1 parent :5 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
> qdisc fq_codel 0: dev eth1 parent :6 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
> qdisc fq_codel 0: dev eth1 parent :7 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
> qdisc fq_codel 0: dev eth1 parent :8 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
> qdisc fq_codel 0: dev ifb4eth0 root refcnt 2 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn

Oops, ingress is not right, yet.

>
> ***@OpenWrt:~# tc -d class show dev eth0
> class cake 8002:4e1 parent 8002:
> class cake 8002:b89 parent 8002:
>
> ***@OpenWrt:~# tc -d class show dev ifb4eth0
> class fq_codel :12c parent none
> class fq_codel :222 parent none
>
> ***@OpenWrt:~# /etc/init.d/sqm restart
> SQM: Trying to start/stop SQM on all interfaces.
> SQM: /usr/lib/sqm/run.sh Stopping SQM on interface: eth0
> SQM: ifb associated with interface eth0:
> SQM: trying to create new IFB: ifb4eth0
> RTNETLINK answers: File exists
> SQM: /usr/lib/sqm/stop.sh: Stopping eth0
> SQM: ifb associated with interface eth0:
> SQM: /usr/lib/sqm/run.sh Queue Setup Script: /usr/lib/sqm/simple.qos
> + . /usr/lib/sqm/functions.sh
> + [ -z 50000 ]
> + [ -z 50000 ]
> + [ -z eth0 ]
> + [ -z cake ]
> + [ -z cake ]
> + [ -z ethernet ]
> + [ -z 42 ]
> + [ -z 2047 ]
> + [ -z 0 ]
> + [ -z 128 ]
> + [ -z ]
> + AUTOFLOW=0
> + [ -z ]
> + LIMIT=1001
> + [ -z ]
> + ILIMIT=
> + [ -z ]
> + ELIMIT=
> + [ -z ]
> + ITARGET=
> + [ -z ]
> + ETARGET=
> + [ -z ECN ]
> + [ -z ECN ]
> + [ -z 0 ]
> + [ -z 0 ]
> + [ -z ]
> + IQDISC_OPTS=
> + [ -z ]
> + EQDISC_OPTS=
> + [ -z ]
> + which tc
> + TC=/usr/sbin/tc
> + [ -z ]
> + which ip
> + IP=/usr/sbin/ip
> + [ -z ]
> + which insmod
> + INSMOD=/usr/sbin/insmod
> + [ -z ]
> + TARGET=5ms
> + [ -z ]
> + IPT_MASK=0xff
> + [ -z ]
> + IPT_MASK_STRING=/0xff
> + [ -z ]
> + get_ifb_for_if eth0
> + CUR_IF=eth0
> + get_ifb_associated_with_if eth0
> + CUR_IF=eth0
> + tc+ -p filtergrep -o -e ifb[^)]\+
> show parent ffff: dev eth0
> + CUR_IFB=
> + sqm_logger ifb associated with interface eth0:
> + logger -t SQM -s ifb associated with interface eth0:
> SQM: ifb associated with interface eth0:
> + echo
> + CUR_IFB=
> + [ -z ]
> + create_new_ifb_for_if eth0
> + CUR_IF=eth0
> + MAX_IF_NAME_LENGTH=15
> + IFB_PREFIX=ifb4
> + NEW_IFB=ifb4eth0
> + IFB_NAME_LENGTH=8
> + [ 8 -gt 15 ]
> + sqm_logger trying to create new IFB: ifb4eth0
> + logger -t SQM -s trying to create new IFB: ifb4eth0
> SQM: trying to create new IFB: ifb4eth0
> + /usr/sbin/ip link add name ifb4eth0 type ifb
> RTNETLINK answers: File exists
> + echo ifb4eth0
> + CUR_IFB=ifb4eth0
> + [ -z ifb4eth0 ]
> + echo ifb4eth0
> + DEV=ifb4eth0
> + do_modules
> + insmod act_ipt
> + lsmod+ grep
> -q ^act_ipt
> + insmod sch_cake
> + + grep -q ^sch_cake
> lsmod
> + insmod sch_ingress
> + + greplsmod
> -q ^sch_ingress
> + insmod act_mirred
> + + greplsmod -q ^act_mirred
>
> + insmod cls_fw
> + + grep -q ^cls_fw
> lsmod
> + insmod sch_htb
> + + lsmodgrep -q ^sch_htb
>
> + ipt_setup
> + ipt -t mangle -N QOS_MARK_eth0
> + echo -t mangle -N QOS_MARK_eth0
> + sed s/-A/-D/g
> + d=-t mangle -N QOS_MARK_eth0
> + [ -t mangle -N QOS_MARK_eth0 != -t mangle -N QOS_MARK_eth0 ]
> + echo -t mangle -N QOS_MARK_eth0
> + sed s/-I/-D/g
> + d=-t mangle -N QOS_MARK_eth0
> + [ -t mangle -N QOS_MARK_eth0 != -t mangle -N QOS_MARK_eth0 ]
> + iptables -t mangle -N QOS_MARK_eth0
> + ip6tables -t mangle -N QOS_MARK_eth0
> + sqm_logger cake does all the diffserv work - no need for iptables rules
> + logger -t SQM -s cake
> SQM: cake
> + [ 0 = 1 ]
> + sqm_logger Keeping differentiated services code points (DSCP) from ingress.
> + logger -t SQM -s Keeping differentiated services code points (DSCP) from ingress.
> SQM: Keeping differentiated services code points (DSCP) from ingress.
> + CAKE_OPTS=
> + ipt -t mangle -A PREROUTING -i eth0 -m mark --mark 0x00/0xff -g QOS_MARK_eth0
> + echo+ sed s/-A/-D/g
> -t mangle -A PREROUTING -i eth0 -m mark --mark 0x00/0xff -g QOS_MARK_eth0
> + d=-t mangle -D PREROUTING -i eth0 -m mark --mark 0x00/0xff -g QOS_MARK_eth0
> + [ -t mangle -D PREROUTING -i eth0 -m mark --mark 0x00/0xff -g QOS_MARK_eth0 != -t mangle -A PREROUTING -i eth0 -m mark --mark 0x00/0xff -g QOS_MARK_eth0 ]
> + iptables -t mangle -D PREROUTING -i eth0 -m mark --mark 0x00/0xff -g QOS_MARK_eth0
> + ip6tables -t mangle -D PREROUTING -i eth0 -m mark --mark 0x00/0xff -g QOS_MARK_eth0
> + echo+ sed s/-I/-D/g
> -t mangle -A PREROUTING -i eth0 -m mark --mark 0x00/0xff -g QOS_MARK_eth0
> + d=-t mangle -A PREROUTING -i eth0 -m mark --mark 0x00/0xff -g QOS_MARK_eth0
> + [ -t mangle -A PREROUTING -i eth0 -m mark --mark 0x00/0xff -g QOS_MARK_eth0 != -t mangle -A PREROUTING -i eth0 -m mark --mark 0x00/0xff -g QOS_MARK_eth0 ]
> + iptables -t mangle -A PREROUTING -i eth0 -m mark --mark 0x00/0xff -g QOS_MARK_eth0
> + ip6tables -t mangle -A PREROUTING -i eth0 -m mark --mark 0x00/0xff -g QOS_MARK_eth0
> + ipt -t mangle -A POSTROUTING -o eth0 -m mark --mark 0x00/0xff -g QOS_MARK_eth0
> + echo+ sed s/-A/-D/g
> -t mangle -A POSTROUTING -o eth0 -m mark --mark 0x00/0xff -g QOS_MARK_eth0
> + d=-t mangle -D POSTROUTING -o eth0 -m mark --mark 0x00/0xff -g QOS_MARK_eth0
> + [ -t mangle -D POSTROUTING -o eth0 -m mark --mark 0x00/0xff -g QOS_MARK_eth0 != -t mangle -A POSTROUTING -o eth0 -m mark --mark 0x00/0xff -g QOS_MARK_eth0 ]
> + iptables -t mangle -D POSTROUTING -o eth0 -m mark --mark 0x00/0xff -g QOS_MARK_eth0
> + ip6tables -t mangle -D POSTROUTING -o eth0 -m mark --mark 0x00/0xff -g QOS_MARK_eth0
> + echo+ sed s/-I/-D/g
> -t mangle -A POSTROUTING -o eth0 -m mark --mark 0x00/0xff -g QOS_MARK_eth0
> + d=-t mangle -A POSTROUTING -o eth0 -m mark --mark 0x00/0xff -g QOS_MARK_eth0
> + [ -t mangle -A POSTROUTING -o eth0 -m mark --mark 0x00/0xff -g QOS_MARK_eth0 != -t mangle -A POSTROUTING -o eth0 -m mark --mark 0x00/0xff -g QOS_MARK_eth0 ]
> + iptables -t mangle -A POSTROUTING -o eth0 -m mark --mark 0x00/0xff -g QOS_MARK_eth0
> + ip6tables -t mangle -A POSTROUTING -o eth0 -m mark --mark 0x00/0xff -g QOS_MARK_eth0
> + ipt -t mangle -I PREROUTING -i vtun+ -p tcp -j MARK --set-mark 0x2/0xff
> + echo+ sed s/-A/-D/g
> -t mangle -I PREROUTING -i vtun+ -p tcp -j MARK --set-mark 0x2/0xff
> + d=-t mangle -I PREROUTING -i vtun+ -p tcp -j MARK --set-mark 0x2/0xff
> + [ -t mangle -I PREROUTING -i vtun+ -p tcp -j MARK --set-mark 0x2/0xff != -t mangle -I PREROUTING -i vtun+ -p tcp -j MARK --set-mark 0x2/0xff ]
> + echo+ sed s/-I/-D/g
> -t mangle -I PREROUTING -i vtun+ -p tcp -j MARK --set-mark 0x2/0xff
> + d=-t mangle -D PREROUTING -i vtun+ -p tcp -j MARK --set-mark 0x2/0xff
> + [ -t mangle -D PREROUTING -i vtun+ -p tcp -j MARK --set-mark 0x2/0xff != -t mangle -I PREROUTING -i vtun+ -p tcp -j MARK --set-mark 0x2/0xff ]
> + iptables -t mangle -D PREROUTING -i vtun+ -p tcp -j MARK --set-mark 0x2/0xff
> + ip6tables -t mangle -D PREROUTING -i vtun+ -p tcp -j MARK --set-mark 0x2/0xff
> + iptables -t mangle -I PREROUTING -i vtun+ -p tcp -j MARK --set-mark 0x2/0xff
> + ip6tables -t mangle -I PREROUTING -i vtun+ -p tcp -j MARK --set-mark 0x2/0xff
> + ipt -t mangle -A OUTPUT -p udp -m multiport --ports 123,53 -j DSCP --set-dscp-class AF42
> + echo+ sed s/-A/-D/g
> -t mangle -A OUTPUT -p udp -m multiport --ports 123,53 -j DSCP --set-dscp-class AF42
> + d=-t mangle -D OUTPUT -p udp -m multiport --ports 123,53 -j DSCP --set-dscp-class AF42
> + [ -t mangle -D OUTPUT -p udp -m multiport --ports 123,53 -j DSCP --set-dscp-class AF42 != -t mangle -A OUTPUT -p udp -m multiport --ports 123,53 -j DSCP --set-dscp-class AF42 ]
> + iptables -t mangle -D OUTPUT -p udp -m multiport --ports 123,53 -j DSCP --set-dscp-class AF42
> + ip6tables -t mangle -D OUTPUT -p udp -m multiport --ports 123,53 -j DSCP --set-dscp-class AF42
> + echo+ sed s/-I/-D/g
> -t mangle -A OUTPUT -p udp -m multiport --ports 123,53 -j DSCP --set-dscp-class AF42
> + d=-t mangle -A OUTPUT -p udp -m multiport --ports 123,53 -j DSCP --set-dscp-class AF42
> + [ -t mangle -A OUTPUT -p udp -m multiport --ports 123,53 -j DSCP --set-dscp-class AF42 != -t mangle -A OUTPUT -p udp -m multiport --ports 123,53 -j DSCP --set-dscp-class AF42 ]
> + iptables -t mangle -A OUTPUT -p udp -m multiport --ports 123,53 -j DSCP --set-dscp-class AF42
> + ip6tables -t mangle -A OUTPUT -p udp -m multiport --ports 123,53 -j DSCP --set-dscp-class AF42
> + [ 50000 -ne 0 ]
> + egress
> + CEIL=50000
> + expr 50000 / 3
> + PRIO_RATE=16666
> + expr 50000 / 6
> + BE_RATE=8333
> + expr 50000 / 6
> + BK_RATE=8333
> + expr 50000 - 16
> + BE_CEIL=49984
> + get_mtu eth0 50000
> + BW=50000
> + cat /sys/class/net/eth0/mtu
> + F=1500
> + [ -z 1500 ]
> + [ 50000 -gt 20000 ]
> + F=3000
> + [ 50000 -gt 30000 ]
> + F=6000
> + [ 50000 -gt 40000 ]
> + F=12000
> + [ 50000 -gt 50000 ]
> + [ 50000 -gt 60000 ]
> + [ 50000 -gt 80000 ]
> + echo 12000
> + LQ=quantum 12000
> + /usr/sbin/tc qdisc del dev eth0 root
> + get_stab_string
> + STABSTRING=
> + [ cake = tc_stab -a ethernet != none ]
> + echo
> + get_cake_lla_string
> + STABSTRING=
> + [ cake = cake -a ethernet != none ]
> + [ ethernet = atm ]
> + STABSTRING= overhead 42
> + sqm_logger cake link layer adjustments: overhead 42
> + logger -t SQM -s cake link layer adjustments: overhead 42
> SQM: cake link layer adjustments: overhead 42
> + sqm_logger SQM: overhead 42
> + logger -t SQM -s SQM: overhead 42
> SQM: SQM: overhead 42
> + echo overhead 42
> + /usr/sbin/tc qdisc add dev eth0 root cake bandwidth 50000kbit overhead 42

As seen above egress works.

> + sqm_logger egress shaping activated
> + logger -t SQM -s egress shaping activated
> SQM: egress shaping activated
> + [ 50000 -ne 0 ]
> + ingress
> + CEIL=50000
> + expr 50000 / 3
> + PRIO_RATE=16666
> + expr 50000 / 6
> + BE_RATE=8333
> + expr 50000 / 6
> + BK_RATE=8333
> + expr 50000 - 16
> + BE_CEIL=49984
> + get_mtu eth0 50000
> + BW=50000
> + cat /sys/class/net/eth0/mtu
> + F=1500
> + [ -z 1500 ]
> + [ 50000 -gt 20000 ]
> + F=3000
> + [ 50000 -gt 30000 ]
> + F=6000
> + [ 50000 -gt 40000 ]
> + F=12000
> + e 50000 -gt 50000 ]
> + [ 50000 -gt 60000 ]
> + [ 50000 -gt 80000 ]
> + echo 12000
> + LQ=quantum 12000
> + /usr/sbin/tc qdisc del dev eth0 handle ffff: ingress
> + /usr/sbin/tc qdisc add dev eth0 handle ffff: ingress
> + /usr/sbin/tc qdisc del dev ifb4eth0 root
> + [ 0 = 1 ]
> + sqm_logger Perform DSCP based filtering on ingress. (3-tier classification)
> + logger -t SQM -s Perform DSCP based filtering on ingress. (3-tier classification)
> SQM: Perform DSCP based filtering on ingress. (3-tier classification)
> + get_stab_string
> + STABSTRING=
> + [ cake = tc_stab -a ethernet != none ]
> + echo
> + get_cake_lla_string
> + STABSTRING=
> + [ cake = cake -a ethernet != none ]
> + [ ethernet = atm ]
> + STABSTRING= overhead 42
> + sqm_logger cake link layer adjustments: overhead 42
> + logger -t SQM -s cake link layer adjustments: overhead 42
> SQM: cake link layer adjustments: overhead 42
> + sqm_logger SQM: overhead 42
> + logger -t SQM -s SQM: overhead 42
> SQM: SQM: overhead 42
> + echo overhead 42
> + /usr/sbin/tc qdisc add dev eth0 root cake bandwidth 50000kbit overhead 42

As seen above egress works. What the F. There is an embarrassing copy and paste error in simple.qosline 180, instead of:
$TC qdisc add dev $IFACE root `get_stab_string` $QDISC bandwidth ${DOWNLINK}kbit `get_cake_lla_string` $CAKE_OPTS ${IQDISC_OPTS}

this line should obviously read:
$TC qdisc add dev $DEV root `get_stab_string` $QDISC bandwidth ${DOWNLINK}kbit `get_cake_lla_string` $CAKE_OPTS ${IQDISC_OPTS}

so simple.qos set-up egress twice, once with the egress parameters and once with the ingress parameters, I guess I did not test 3-tier ingress classification on Dave’s test machine.
Also I should rename a few variables. $DEV and $IFACE are not the best names. (Also cake needs its own .qos file(s) instead of simple and simplest becoming rather “complex”…) Thanks for keeping testing this.

Best Regards
Sebastian

> RTNETLINK answers: File exists
> + ifconfig ifb4eth0 up
> + /usr/sbin/tc filter add dev eth0 parent ffff: protocol all prio 10 u32 match u32 0 0 flowid 1:1 action mirred egress redirect dev ifb4eth0
> + sqm_logger ingress shaping activated
> + logger -t SQM -s ingress shaping activated
> SQM: ingress shaping activated




>
> --
> Mikael Abrahamsson email: ***@swm.pp.se

[Sebastian Moeller]

Hi Mikael,

since coke overhead seems to work, you can switch back to link layer none or set the overhead to 0 again. Thanks for testing this, which helps getting sqm-scripts into better shape, but will not be helpful fur your tests. In case you want to try fancy options for cake the advances qdisc option strings in the queue discipline tab should work (caveat no error checking so one typo and the qdisc will most likely not be set-up properly).

Best Regards
Sebastian



On Jun 29, 2015, at 10:09 , Mikael Abrahamsson <***@swm.pp.se> wrote:

> On Mon, 29 Jun 2015, Sebastian Moeller wrote:
>
>> Ah, I see, you are still using tc’s stab mechanism for account of per packet overhead and link layer adjustments instead of cake’s (you need to check the advanced options check box in the link layer adjustments tab, and then select “cake” as the lnk layer adjustment mechanism).
>
> Ok, so when I changed "stab" to cake, I get the following, but it still only shapes one way. If I change to discipline to "fq_codel" or "pie" I get bidirectonal shaping with otherwise same settings.
>
> ***@OpenWrt:~# cat /etc/config/sqm
>
> config queue 'eth1'
> option interface 'eth0'
> option qdisc_advanced '1'
> option squash_dscp '0'
> option squash_ingress '0'
> option ingress_ecn 'ECN'
> option egress_ecn 'ECN'
> option qdisc_really_really_advanced '0'
> option download '50000'
> option upload '50000'
> option linklayer 'ethernet'
> option overhead '42'
> option linklayer_advanced '1'
> option tcMTU '2047'
> option tcTSIZE '128'
> option tcMPU '0'
> option linklayer_adaptation_mechanism 'cake'
> option enabled '1'
> option qdisc 'cake'
> option script 'simple.qos'
>
> ***@OpenWrt:~# tc -d qdisc
> qdisc cake 8002: dev eth0 root refcnt 9 bandwidth 50Mbit diffserv4 flows noatm overhead 42
> qdisc ingress ffff: dev eth0 parent ffff:fff1 ----------------
> qdisc mq 0: dev eth1 root
> qdisc fq_codel 0: dev eth1 parent :1 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
> qdisc fq_codel 0: dev eth1 parent :2 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
> qdisc fq_codel 0: dev eth1 parent :3 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
> qdisc fq_codel 0: dev eth1 parent :4 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
> qdisc fq_codel 0: dev eth1 parent :5 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
> qdisc fq_codel 0: dev eth1 parent :6 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
> qdisc fq_codel 0: dev eth1 parent :7 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
> qdisc fq_codel 0: dev eth1 parent :8 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
> qdisc fq_codel 0: dev ifb4eth0 root refcnt 2 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
>
> ***@OpenWrt:~# tc -d class show dev eth0
> class cake 8002:4e1 parent 8002:
> class cake 8002:b89 parent 8002:
>
> ***@OpenWrt:~# tc -d class show dev ifb4eth0
> class fq_codel :12c parent none
> class fq_codel :222 parent none
>
> ***@OpenWrt:~# /etc/init.d/sqm restart
> SQM: Trying to start/stop SQM on all interfaces.
> SQM: /usr/lib/sqm/run.sh Stopping SQM on interface: eth0
> SQM: ifb associated with interface eth0:
> SQM: trying to create new IFB: ifb4eth0
> RTNETLINK answers: File exists
> SQM: /usr/lib/sqm/stop.sh: Stopping eth0
> SQM: ifb associated with interface eth0:
> SQM: /usr/lib/sqm/run.sh Queue Setup Script: /usr/lib/sqm/simple.qos
> + . /usr/lib/sqm/functions.sh
> + [ -z 50000 ]
> + [ -z 50000 ]
> + [ -z eth0 ]
> + [ -z cake ]
> + [ -z cake ]
> + [ -z ethernet ]
> + [ -z 42 ]
> + [ -z 2047 ]
> + [ -z 0 ]
> + [ -z 128 ]
> + [ -z ]
> + AUTOFLOW=0
> + [ -z ]
> + LIMIT=1001
> + [ -z ]
> + ILIMIT=
> + [ -z ]
> + ELIMIT=
> + [ -z ]
> + ITARGET=
> + [ -z ]
> + ETARGET=
> + [ -z ECN ]
> + [ -z ECN ]
> + [ -z 0 ]
> + [ -z 0 ]
> + [ -z ]
> + IQDISC_OPTS=
> + [ -z ]
> + EQDISC_OPTS=
> + [ -z ]
> + which tc
> + TC=/usr/sbin/tc
> + [ -z ]
> + which ip
> + IP=/usr/sbin/ip
> + [ -z ]
> + which insmod
> + INSMOD=/usr/sbin/insmod
> + [ -z ]
> + TARGET=5ms
> + [ -z ]
> + IPT_MASK=0xff
> + [ -z ]
> + IPT_MASK_STRING=/0xff
> + [ -z ]
> + get_ifb_for_if eth0
> + CUR_IF=eth0
> + get_ifb_associated_with_if eth0
> + CUR_IF=eth0
> + tc+ -p filtergrep -o -e ifb[^)]\+
> show parent ffff: dev eth0
> + CUR_IFB=
> + sqm_logger ifb associated with interface eth0:
> + logger -t SQM -s ifb associated with interface eth0:
> SQM: ifb associated with interface eth0:
> + echo
> + CUR_IFB=
> + [ -z ]
> + create_new_ifb_for_if eth0
> + CUR_IF=eth0
> + MAX_IF_NAME_LENGTH=15
> + IFB_PREFIX=ifb4
> + NEW_IFB=ifb4eth0
> + IFB_NAME_LENGTH=8
> + [ 8 -gt 15 ]
> + sqm_logger trying to create new IFB: ifb4eth0
> + logger -t SQM -s trying to create new IFB: ifb4eth0
> SQM: trying to create new IFB: ifb4eth0
> + /usr/sbin/ip link add name ifb4eth0 type ifb
> RTNETLINK answers: File exists
> + echo ifb4eth0
> + CUR_IFB=ifb4eth0
> + [ -z ifb4eth0 ]
> + echo ifb4eth0
> + DEV=ifb4eth0
> + do_modules
> + insmod act_ipt
> + lsmod+ grep
> -q ^act_ipt
> + insmod sch_cake
> + + grep -q ^sch_cake
> lsmod
> + insmod sch_ingress
> + + greplsmod
> -q ^sch_ingress
> + insmod act_mirred
> + + greplsmod -q ^act_mirred
>
> + insmod cls_fw
> + + grep -q ^cls_fw
> lsmod
> + insmod sch_htb
> + + lsmodgrep -q ^sch_htb
>
> + ipt_setup
> + ipt -t mangle -N QOS_MARK_eth0
> + echo -t mangle -N QOS_MARK_eth0
> + sed s/-A/-D/g
> + d=-t mangle -N QOS_MARK_eth0
> + [ -t mangle -N QOS_MARK_eth0 != -t mangle -N QOS_MARK_eth0 ]
> + echo -t mangle -N QOS_MARK_eth0
> + sed s/-I/-D/g
> + d=-t mangle -N QOS_MARK_eth0
> + [ -t mangle -N QOS_MARK_eth0 != -t mangle -N QOS_MARK_eth0 ]
> + iptables -t mangle -N QOS_MARK_eth0
> + ip6tables -t mangle -N QOS_MARK_eth0
> + sqm_logger cake does all the diffserv work - no need for iptables rules
> + logger -t SQM -s cake
> SQM: cake
> + [ 0 = 1 ]
> + sqm_logger Keeping differentiated services code points (DSCP) from ingress.
> + logger -t SQM -s Keeping differentiated services code points (DSCP) from ingress.
> SQM: Keeping differentiated services code points (DSCP) from ingress.
> + CAKE_OPTS=
> + ipt -t mangle -A PREROUTING -i eth0 -m mark --mark 0x00/0xff -g QOS_MARK_eth0
> + echo+ sed s/-A/-D/g
> -t mangle -A PREROUTING -i eth0 -m mark --mark 0x00/0xff -g QOS_MARK_eth0
> + d=-t mangle -D PREROUTING -i eth0 -m mark --mark 0x00/0xff -g QOS_MARK_eth0
> + [ -t mangle -D PREROUTING -i eth0 -m mark --mark 0x00/0xff -g QOS_MARK_eth0 != -t mangle -A PREROUTING -i eth0 -m mark --mark 0x00/0xff -g QOS_MARK_eth0 ]
> + iptables -t mangle -D PREROUTING -i eth0 -m mark --mark 0x00/0xff -g QOS_MARK_eth0
> + ip6tables -t mangle -D PREROUTING -i eth0 -m mark --mark 0x00/0xff -g QOS_MARK_eth0
> + echo+ sed s/-I/-D/g
> -t mangle -A PREROUTING -i eth0 -m mark --mark 0x00/0xff -g QOS_MARK_eth0
> + d=-t mangle -A PREROUTING -i eth0 -m mark --mark 0x00/0xff -g QOS_MARK_eth0
> + [ -t mangle -A PREROUTING -i eth0 -m mark --mark 0x00/0xff -g QOS_MARK_eth0 != -t mangle -A PREROUTING -i eth0 -m mark --mark 0x00/0xff -g QOS_MARK_eth0 ]
> + iptables -t mangle -A PREROUTING -i eth0 -m mark --mark 0x00/0xff -g QOS_MARK_eth0
> + ip6tables -t mangle -A PREROUTING -i eth0 -m mark --mark 0x00/0xff -g QOS_MARK_eth0
> + ipt -t mangle -A POSTROUTING -o eth0 -m mark --mark 0x00/0xff -g QOS_MARK_eth0
> + echo+ sed s/-A/-D/g
> -t mangle -A POSTROUTING -o eth0 -m mark --mark 0x00/0xff -g QOS_MARK_eth0
> + d=-t mangle -D POSTROUTING -o eth0 -m mark --mark 0x00/0xff -g QOS_MARK_eth0
> + [ -t mangle -D POSTROUTING -o eth0 -m mark --mark 0x00/0xff -g QOS_MARK_eth0 != -t mangle -A POSTROUTING -o eth0 -m mark --mark 0x00/0xff -g QOS_MARK_eth0 ]
> + iptables -t mangle -D POSTROUTING -o eth0 -m mark --mark 0x00/0xff -g QOS_MARK_eth0
> + ip6tables -t mangle -D POSTROUTING -o eth0 -m mark --mark 0x00/0xff -g QOS_MARK_eth0
> + echo+ sed s/-I/-D/g
> -t mangle -A POSTROUTING -o eth0 -m mark --mark 0x00/0xff -g QOS_MARK_eth0
> + d=-t mangle -A POSTROUTING -o eth0 -m mark --mark 0x00/0xff -g QOS_MARK_eth0
> + [ -t mangle -A POSTROUTING -o eth0 -m mark --mark 0x00/0xff -g QOS_MARK_eth0 != -t mangle -A POSTROUTING -o eth0 -m mark --mark 0x00/0xff -g QOS_MARK_eth0 ]
> + iptables -t mangle -A POSTROUTING -o eth0 -m mark --mark 0x00/0xff -g QOS_MARK_eth0
> + ip6tables -t mangle -A POSTROUTING -o eth0 -m mark --mark 0x00/0xff -g QOS_MARK_eth0
> + ipt -t mangle -I PREROUTING -i vtun+ -p tcp -j MARK --set-mark 0x2/0xff
> + echo+ sed s/-A/-D/g
> -t mangle -I PREROUTING -i vtun+ -p tcp -j MARK --set-mark 0x2/0xff
> + d=-t mangle -I PREROUTING -i vtun+ -p tcp -j MARK --set-mark 0x2/0xff
> + [ -t mangle -I PREROUTING -i vtun+ -p tcp -j MARK --set-mark 0x2/0xff != -t mangle -I PREROUTING -i vtun+ -p tcp -j MARK --set-mark 0x2/0xff ]
> + echo+ sed s/-I/-D/g
> -t mangle -I PREROUTING -i vtun+ -p tcp -j MARK --set-mark 0x2/0xff
> + d=-t mangle -D PREROUTING -i vtun+ -p tcp -j MARK --set-mark 0x2/0xff
> + [ -t mangle -D PREROUTING -i vtun+ -p tcp -j MARK --set-mark 0x2/0xff != -t mangle -I PREROUTING -i vtun+ -p tcp -j MARK --set-mark 0x2/0xff ]
> + iptables -t mangle -D PREROUTING -i vtun+ -p tcp -j MARK --set-mark 0x2/0xff
> + ip6tables -t mangle -D PREROUTING -i vtun+ -p tcp -j MARK --set-mark 0x2/0xff
> + iptables -t mangle -I PREROUTING -i vtun+ -p tcp -j MARK --set-mark 0x2/0xff
> + ip6tables -t mangle -I PREROUTING -i vtun+ -p tcp -j MARK --set-mark 0x2/0xff
> + ipt -t mangle -A OUTPUT -p udp -m multiport --ports 123,53 -j DSCP --set-dscp-class AF42
> + echo+ sed s/-A/-D/g
> -t mangle -A OUTPUT -p udp -m multiport --ports 123,53 -j DSCP --set-dscp-class AF42
> + d=-t mangle -D OUTPUT -p udp -m multiport --ports 123,53 -j DSCP --set-dscp-class AF42
> + [ -t mangle -D OUTPUT -p udp -m multiport --ports 123,53 -j DSCP --set-dscp-class AF42 != -t mangle -A OUTPUT -p udp -m multiport --ports 123,53 -j DSCP --set-dscp-class AF42 ]
> + iptables -t mangle -D OUTPUT -p udp -m multiport --ports 123,53 -j DSCP --set-dscp-class AF42
> + ip6tables -t mangle -D OUTPUT -p udp -m multiport --ports 123,53 -j DSCP --set-dscp-class AF42
> + echo+ sed s/-I/-D/g
> -t mangle -A OUTPUT -p udp -m multiport --ports 123,53 -j DSCP --set-dscp-class AF42
> + d=-t mangle -A OUTPUT -p udp -m multiport --ports 123,53 -j DSCP --set-dscp-class AF42
> + [ -t mangle -A OUTPUT -p udp -m multiport --ports 123,53 -j DSCP --set-dscp-class AF42 != -t mangle -A OUTPUT -p udp -m multiport --ports 123,53 -j DSCP --set-dscp-class AF42 ]
> + iptables -t mangle -A OUTPUT -p udp -m multiport --ports 123,53 -j DSCP --set-dscp-class AF42
> + ip6tables -t mangle -A OUTPUT -p udp -m multiport --ports 123,53 -j DSCP --set-dscp-class AF42
> + [ 50000 -ne 0 ]
> + egress
> + CEIL=50000
> + expr 50000 / 3
> + PRIO_RATE=16666
> + expr 50000 / 6
> + BE_RATE=8333
> + expr 50000 / 6
> + BK_RATE=8333
> + expr 50000 - 16
> + BE_CEIL=49984
> + get_mtu eth0 50000
> + BW=50000
> + cat /sys/class/net/eth0/mtu
> + F=1500
> + [ -z 1500 ]
> + [ 50000 -gt 20000 ]
> + F=3000
> + [ 50000 -gt 30000 ]
> + F=6000
> + [ 50000 -gt 40000 ]
> + F=12000
> + [ 50000 -gt 50000 ]
> + [ 50000 -gt 60000 ]
> + [ 50000 -gt 80000 ]
> + echo 12000
> + LQ=quantum 12000
> + /usr/sbin/tc qdisc del dev eth0 root
> + get_stab_string
> + STABSTRING=
> + [ cake = tc_stab -a ethernet != none ]
> + echo
> + get_cake_lla_string
> + STABSTRING=
> + [ cake = cake -a ethernet != none ]
> + [ ethernet = atm ]
> + STABSTRING= overhead 42
> + sqm_logger cake link layer adjustments: overhead 42
> + logger -t SQM -s cake link layer adjustments: overhead 42
> SQM: cake link layer adjustments: overhead 42
> + sqm_logger SQM: overhead 42
> + logger -t SQM -s SQM: overhead 42
> SQM: SQM: overhead 42
> + echo overhead 42
> + /usr/sbin/tc qdisc add dev eth0 root cake bandwidth 50000kbit overhead 42
> + sqm_logger egress shaping activated
> + logger -t SQM -s egress shaping activated
> SQM: egress shaping activated
> + [ 50000 -ne 0 ]
> + ingress
> + CEIL=50000
> + expr 50000 / 3
> + PRIO_RATE=16666
> + expr 50000 / 6
> + BE_RATE=8333
> + expr 50000 / 6
> + BK_RATE=8333
> + expr 50000 - 16
> + BE_CEIL=49984
> + get_mtu eth0 50000
> + BW=50000
> + cat /sys/class/net/eth0/mtu
> + F=1500
> + [ -z 1500 ]
> + [ 50000 -gt 20000 ]
> + F=3000
> + [ 50000 -gt 30000 ]
> + F=6000
> + [ 50000 -gt 40000 ]
> + F=12000
> + e 50000 -gt 50000 ]
> + [ 50000 -gt 60000 ]
> + [ 50000 -gt 80000 ]
> + echo 12000
> + LQ=quantum 12000
> + /usr/sbin/tc qdisc del dev eth0 handle ffff: ingress
> + /usr/sbin/tc qdisc add dev eth0 handle ffff: ingress
> + /usr/sbin/tc qdisc del dev ifb4eth0 root
> + [ 0 = 1 ]
> + sqm_logger Perform DSCP based filtering on ingress. (3-tier classification)
> + logger -t SQM -s Perform DSCP based filtering on ingress. (3-tier classification)
> SQM: Perform DSCP based filtering on ingress. (3-tier classification)
> + get_stab_string
> + STABSTRING=
> + [ cake = tc_stab -a ethernet != none ]
> + echo
> + get_cake_lla_string
> + STABSTRING=
> + [ cake = cake -a ethernet != none ]
> + [ ethernet = atm ]
> + STABSTRING= overhead 42
> + sqm_logger cake link layer adjustments: overhead 42
> + logger -t SQM -s cake link layer adjustments: overhead 42
> SQM: cake link layer adjustments: overhead 42
> + sqm_logger SQM: overhead 42
> + logger -t SQM -s SQM: overhead 42
> SQM: SQM: overhead 42
> + echo overhead 42
> + /usr/sbin/tc qdisc add dev eth0 root cake bandwidth 50000kbit overhead 42
> RTNETLINK answers: File exists
> + ifconfig ifb4eth0 up
> + /usr/sbin/tc filter add dev eth0 parent ffff: protocol all prio 10 u32 match u32 0 0 flowid 1:1 action mirred egress redirect dev ifb4eth0
> + sqm_logger ingress shaping activated
> + logger -t SQM -s ingress shaping activated
> SQM: ingress shaping activated
>
> --
> Mikael Abrahamsson email: ***@swm.pp.se

[Mikael Abrahamsson]

On Mon, 29 Jun 2015, Sebastian Moeller wrote:

> Hi Mikael,
>
> since coke overhead seems to work, you can switch back to link layer
> none or set the overhead to 0 again. Thanks for testing this, which
> helps getting sqm-scripts into better shape, but will not be helpful fur
> your tests. In case you want to try fancy options for cake the advances
> qdisc option strings in the queue discipline tab should work (caveat no
> error checking so one typo and the qdisc will most likely not be set-up
> properly).

What I would like to get working is that it builds a fully working image
out of the box. I have a problem with this, I believe it's because the
ceropackages and openwrt standard packages have overlapping names:

#openwrt/feeds$ grep -i sqm-scripts *
grep: cero: Is a directory
cero.index:Depends: +libc +SSP_SUPPORT:libssp +USE_GLIBC:librt +USE_GLIBC:libpthread lua luci-base +sqm-scripts
cero.index:Source-Makefile: feeds/cero/net/sqm-scripts/Makefile
cero.index:Package: sqm-scripts
grep: cero.tmp: Is a directory
grep: luci: Is a directory
grep: luci.tmp: Is a directory
grep: management: Is a directory
grep: management.tmp: Is a directory
grep: packages: Is a directory
packages.index:Depends: +libc +SSP_SUPPORT:libssp +USE_GLIBC:librt +USE_GLIBC:libpthread lua luci-base +sqm-scripts
packages.index:Source-Makefile: feeds/packages/net/sqm-scripts/Makefile
packages.index:Package: sqm-scripts
grep: packages.tmp: Is a directory
grep: routing: Is a directory
grep: routing.tmp: Is a directory
grep: targets: Is a directory
grep: targets.tmp: Is a directory
grep: telephony: Is a directory
grep: telephony.tmp: Is a directory

How can I tell which one of these actually is included? Tried moving the
feed statement so ceropackages was first but that doesn't seem to have
helped, I still get OpenWrt regular sqm-scripts (no cake in luci-app-sqm).

#openwrt/feeds$ find . -name luci-app-sqm
./packages/net/luci-app-sqm
./cero/luci/luci-app-sqm

--
Mikael Abrahamsson email: ***@swm.pp.se

[Toke Høiland-Jørgensen]

Mikael Abrahamsson <***@swm.pp.se> writes:

> How can I tell which one of these actually is included? Tried moving the feed
> statement so ceropackages was first but that doesn't seem to have helped, I
> still get OpenWrt regular sqm-scripts (no cake in luci-app-sqm).

You need to have the cero feed first in feeds.conf, then do
./script/feeds uninstall sqm-scripts; ./script/feeds install sqm-scripts
-- that should pull it from the cero feed. If it doesn't work, try doing
./scripts/feeds update first

That seemed to work for me when I just tested it now, at least :)

-Toke

[Mikael Abrahamsson]

On Mon, 29 Jun 2015, Toke Høiland-Jørgensen wrote:

> Mikael Abrahamsson <***@swm.pp.se> writes:
>
>> How can I tell which one of these actually is included? Tried moving the feed
>> statement so ceropackages was first but that doesn't seem to have helped, I
>> still get OpenWrt regular sqm-scripts (no cake in luci-app-sqm).
>
> You need to have the cero feed first in feeds.conf, then do
> ./script/feeds uninstall sqm-scripts; ./script/feeds install sqm-scripts
> -- that should pull it from the cero feed. If it doesn't work, try doing
> ./scripts/feeds update first
>
> That seemed to work for me when I just tested it now, at least :)

Hi,

Ok, yes, this worked, I must have forgotten do to update after I moved
ceropackages to the top of the list before. Thanks!

So now I have a sysupgrade image for the wrt1200ac that out of the box
comes with CeroPackages and working bidirectional shaping for cake (don't
know why it didn't work before, it might have to do with my modifications.
This time I didn't modify anything on-disk, this is purely from the
CeroPackages feed).

I did try to get Kernel 4.1 to compile but that didn't work even though I
removed some packages that didn't compile, I ended up with no .dts file
and nothing to me obvious in scrollback to fix. So this is with 3.18.

Here are the cake 50M and 500M results and output from the router:

***@OpenWrt:~# cat /etc/config/sqm

config queue 'eth1'
option interface 'eth0'
option qdisc_advanced '1'
option squash_dscp '0'
option squash_ingress '0'
option ingress_ecn 'ECN'
option egress_ecn 'ECN'
option qdisc_really_really_advanced '0'
option linklayer 'ethernet'
option overhead '42'
option linklayer_advanced '1'
option tcMTU '2047'
option tcTSIZE '128'
option tcMPU '0'
option enabled '1'
option script 'simple.qos'
option qdisc 'cake'
option linklayer_adaptation_mechanism 'cake'
option download '500000'
option upload '500000'

***@OpenWrt:~# tc -d qdisc
qdisc cake 8009: dev eth0 root refcnt 9 bandwidth 500Mbit diffserv4 flows
noatm overhead 42
qdisc ingress ffff: dev eth0 parent ffff:fff1 ----------------
qdisc mq 0: dev eth1 root
qdisc fq_codel 0: dev eth1 parent :1 limit 1024p flows 1024 quantum 300
target 5.0ms interval 100.0ms ecn
qdisc fq_codel 0: dev eth1 parent :2 limit 1024p flows 1024 quantum 300
target 5.0ms interval 100.0ms ecn
qdisc fq_codel 0: dev eth1 parent :3 limit 1024p flows 1024 quantum 300
target 5.0ms interval 100.0ms ecn
qdisc fq_codel 0: dev eth1 parent :4 limit 1024p flows 1024 quantum 300
target 5.0ms interval 100.0ms ecn
qdisc fq_codel 0: dev eth1 parent :5 limit 1024p flows 1024 quantum 300
target 5.0ms interval 100.0ms ecn
qdisc fq_codel 0: dev eth1 parent :6 limit 1024p flows 1024 quantum 300
target 5.0ms interval 100.0ms ecn
qdisc fq_codel 0: dev eth1 parent :7 limit 1024p flows 1024 quantum 300
target 5.0ms interval 100.0ms ecn
qdisc fq_codel 0: dev eth1 parent :8 limit 1024p flows 1024 quantum 300
target 5.0ms interval 100.0ms ecn
qdisc cake 800a: dev ifb4eth0 root refcnt 2 bandwidth 500Mbit diffserv4
flows noatm overhead 42

These are the results from 50M and 500M, also including 50up and 50down
that I added to my test suite script.

http://swm.pp.se/aqm/rrul_150629-cake-4.tar

Then I re-did the test that Dave asked before, I set the wan port to 100
megabit/s in my switch, and removed the SQM. It resulted in the following
config:

***@OpenWrt:~# cat /etc/config/sqm

config queue 'eth1'
option interface 'eth0'
option qdisc_advanced '1'
option squash_dscp '0'
option squash_ingress '0'
option ingress_ecn 'ECN'
option egress_ecn 'ECN'
option qdisc_really_really_advanced '0'
option linklayer 'ethernet'
option overhead '42'
option linklayer_advanced '1'
option tcMTU '2047'
option tcTSIZE '128'
option tcMPU '0'
option script 'simple.qos'
option qdisc 'cake'
option linklayer_adaptation_mechanism 'cake'
option download '50000'
option upload '50000'
option enabled '0'

***@OpenWrt:~# tc -d qdisc
qdisc mq 0: dev eth0 root
qdisc fq_codel 0: dev eth0 parent :1 limit 1024p flows 1024 quantum 300
target 5.0ms interval 100.0ms ecn
qdisc fq_codel 0: dev eth0 parent :2 limit 1024p flows 1024 quantum 300
target 5.0ms interval 100.0ms ecn
qdisc fq_codel 0: dev eth0 parent :3 limit 1024p flows 1024 quantum 300
target 5.0ms interval 100.0ms ecn
qdisc fq_codel 0: dev eth0 parent :4 limit 1024p flows 1024 quantum 300
target 5.0ms interval 100.0ms ecn
qdisc fq_codel 0: dev eth0 parent :5 limit 1024p flows 1024 quantum 300
target 5.0ms interval 100.0ms ecn
qdisc fq_codel 0: dev eth0 parent :6 limit 1024p flows 1024 quantum 300
target 5.0ms interval 100.0ms ecn
qdisc fq_codel 0: dev eth0 parent :7 limit 1024p flows 1024 quantum 300
target 5.0ms interval 100.0ms ecn
qdisc fq_codel 0: dev eth0 parent :8 limit 1024p flows 1024 quantum 300
target 5.0ms interval 100.0ms ecn
qdisc mq 0: dev eth1 root
qdisc fq_codel 0: dev eth1 parent :1 limit 1024p flows 1024 quantum 300
target 5.0ms interval 100.0ms ecn
qdisc fq_codel 0: dev eth1 parent :2 limit 1024p flows 1024 quantum 300
target 5.0ms interval 100.0ms ecn
qdisc fq_codel 0: dev eth1 parent :3 limit 1024p flows 1024 quantum 300
target 5.0ms interval 100.0ms ecn
qdisc fq_codel 0: dev eth1 parent :4 limit 1024p flows 1024 quantum 300
target 5.0ms interval 100.0ms ecn
qdisc fq_codel 0: dev eth1 parent :5 limit 1024p flows 1024 quantum 300
target 5.0ms interval 100.0ms ecn
qdisc fq_codel 0: dev eth1 parent :6 limit 1024p flows 1024 quantum 300
target 5.0ms interval 100.0ms ecn
qdisc fq_codel 0: dev eth1 parent :7 limit 1024p flows 1024 quantum 300
target 5.0ms interval 100.0ms ecn
qdisc fq_codel 0: dev eth1 parent :8 limit 1024p flows 1024 quantum 300
target 5.0ms interval 100.0ms ecn
qdisc cake 800c: dev ifb4eth0 root refcnt 2 bandwidth 50Mbit diffserv4
flows noatm overhead 42
***@OpenWrt:~# tc -d class show dev eth0
class mq :1 root
class mq :2 root
class mq :3 root
class mq :4 root
class mq :5 root
class mq :6 root
class mq :7 root
class mq :8 root

what worries me is this:

***@OpenWrt:~# ethtool eth0
Settings for eth0:
Supported ports: [ TP MII ]
Supported link modes: 1000baseT/Half 1000baseT/Full
Supported pause frame use: No
Supports auto-negotiation: Yes
Advertised link modes: 1000baseT/Half 1000baseT/Full
Advertised pause frame use: No
Advertised auto-negotiation: Yes
Link partner advertised link modes: 1000baseT/Full
Link partner advertised pause frame use: No
Link partner advertised auto-negotiation: No
Speed: 1000Mb/s
Duplex: Full
Port: MII
PHYAD: 0
Transceiver: external
Auto-negotiation: on
Link detected: yes

So basically even after the wan port went to 100/full, eth0 doesn't know
about it (and it only supports gig speed (probably to the local switch)
anyway. I am seeing dropped packets, so this would support this theory.

First I ran some tests with only that, then I set SQM to 90 megabit/s
results here:

http://swm.pp.se/aqm/rrul_150629-cake-5.tar
http://swm.pp.se/aqm/rrul_150629-cake-6.tar

Next I am going to test wireless but it seems something has gone wrong
because I can't get the wireless to enable properly, so that'll have to be
for the next email after I fix that problem.

--
Mikael Abrahamsson email: ***@swm.pp.se

[Sebastian Moeller]

Hi Mikael,

On Jun 29, 2015, at 15:00 , Mikael Abrahamsson <***@swm.pp.se> wrote:
> [...]
> Hi,
>
> Ok, yes, this worked, I must have forgotten do to update after I moved ceropackages to the top of the list before. Thanks!
>
> So now I have a sysupgrade image for the wrt1200ac that out of the box comes with CeroPackages and working bidirectional shaping for cake (don't know why it didn't work before, it might have to do with my modifications. This time I didn't modify anything on-disk, this is purely from the CeroPackages feed).

Erm, I committed the fix for the $DEV $IFACE confusion to the ceropackages repository, so you got the fixed version you helped fix ;) (that or you deselected ingress 3-tier classification).

>
> I did try to get Kernel 4.1 to compile but that didn't work even though I removed some packages that didn't compile, I ended up with no .dts file and nothing to me obvious in scrollback to fix. So this is with 3.18.
>
> Here are the cake 50M and 500M results and output from the router:
>
> ***@OpenWrt:~# cat /etc/config/sqm
>
> config queue 'eth1'
> option interface 'eth0'
> option qdisc_advanced '1'
> option squash_dscp '0'
> option squash_ingress '0'
> option ingress_ecn 'ECN'
> option egress_ecn 'ECN'
> option qdisc_really_really_advanced '0'
> option linklayer 'ethernet'
> option overhead '42'
> option linklayer_advanced '1'
> option tcMTU '2047'
> option tcTSIZE '128'
> option tcMPU '0'
> option enabled '1'
> option script 'simple.qos'
> option qdisc 'cake'
> option linklayer_adaptation_mechanism 'cake'
> option download '500000'
> option upload '500000'
>
> ***@OpenWrt:~# tc -d qdisc
> qdisc cake 8009: dev eth0 root refcnt 9 bandwidth 500Mbit diffserv4 flows noatm overhead 42
> qdisc ingress ffff: dev eth0 parent ffff:fff1 ----------------
> qdisc mq 0: dev eth1 root
> qdisc fq_codel 0: dev eth1 parent :1 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
> qdisc fq_codel 0: dev eth1 parent :2 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
> qdisc fq_codel 0: dev eth1 parent :3 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
> qdisc fq_codel 0: dev eth1 parent :4 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
> qdisc fq_codel 0: dev eth1 parent :5 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
> qdisc fq_codel 0: dev eth1 parent :6 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
> qdisc fq_codel 0: dev eth1 parent :7 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
> qdisc fq_codel 0: dev eth1 parent :8 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
> qdisc cake 800a: dev ifb4eth0 root refcnt 2 bandwidth 500Mbit diffserv4 flows noatm overhead 42

I love that passing per packet overheads to cake works, but for your testing it is going to reduce the maximum achievable speed from:

TCP/IPv4 Payload at 500Mbps shaping:
1500 - 20 - 20 = 1460 Byte
payload to OTWS ratio
1460/1518 = 0.961791831357
Payload Bandwidth
500*1460/1518 = 480.90 Mbps

to:

TCP/IPv4 Payload at 500Mbps shaping: with overhead 42
MTU: 1500
MSS: 1500 - 20 - 20 = 1460 Byte
PerPacketOverhead: 42
configured On-The-Wire Size: 1500 + 42 = 1542 Byte
payload to OTWS ratio (due to the explicitly configured overhead42)
1460/1542 = 0.961791831357
Payload Bandwidth
500*1460/1542 = 473.411154345 Mbps

I really ned to get a new router to take part in all the fun…

Best Regards
Sebastian



>
> These are the results from 50M and 500M, also including 50up and 50down that I added to my test suite script.
>
> http://swm.pp.se/aqm/rrul_150629-cake-4.tar
>
> Then I re-did the test that Dave asked before, I set the wan port to 100 megabit/s in my switch, and removed the SQM. It resulted in the following config:
>
> ***@OpenWrt:~# cat /etc/config/sqm
>
> config queue 'eth1'
> option interface 'eth0'
> option qdisc_advanced '1'
> option squash_dscp '0'
> option squash_ingress '0'
> option ingress_ecn 'ECN'
> option egress_ecn 'ECN'
> option qdisc_really_really_advanced '0'
> option linklayer 'ethernet'
> option overhead '42'
> option linklayer_advanced '1'
> option tcMTU '2047'
> option tcTSIZE '128'
> option tcMPU '0'
> option script 'simple.qos'
> option qdisc 'cake'
> option linklayer_adaptation_mechanism 'cake'
> option download '50000'
> option upload '50000'
> option enabled '0'
>
> ***@OpenWrt:~# tc -d qdisc
> qdisc mq 0: dev eth0 root
> qdisc fq_codel 0: dev eth0 parent :1 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
> qdisc fq_codel 0: dev eth0 parent :2 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
> qdisc fq_codel 0: dev eth0 parent :3 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
> qdisc fq_codel 0: dev eth0 parent :4 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
> qdisc fq_codel 0: dev eth0 parent :5 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
> qdisc fq_codel 0: dev eth0 parent :6 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
> qdisc fq_codel 0: dev eth0 parent :7 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
> qdisc fq_codel 0: dev eth0 parent :8 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
> qdisc mq 0: dev eth1 root
> qdisc fq_codel 0: dev eth1 parent :1 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
> qdisc fq_codel 0: dev eth1 parent :2 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
> qdisc fq_codel 0: dev eth1 parent :3 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
> qdisc fq_codel 0: dev eth1 parent :4 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
> qdisc fq_codel 0: dev eth1 parent :5 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
> qdisc fq_codel 0: dev eth1 parent :6 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
> qdisc fq_codel 0: dev eth1 parent :7 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
> qdisc fq_codel 0: dev eth1 parent :8 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
> qdisc cake 800c: dev ifb4eth0 root refcnt 2 bandwidth 50Mbit diffserv4 flows noatm overhead 42
> ***@OpenWrt:~# tc -d class show dev eth0
> class mq :1 root
> class mq :2 root
> class mq :3 root
> class mq :4 root
> class mq :5 root
> class mq :6 root
> class mq :7 root
> class mq :8 root
>
> what worries me is this:
>
> ***@OpenWrt:~# ethtool eth0
> Settings for eth0:
> Supported ports: [ TP MII ]
> Supported link modes: 1000baseT/Half 1000baseT/Full
> Supported pause frame use: No
> Supports auto-negotiation: Yes
> Advertised link modes: 1000baseT/Half 1000baseT/Full
> Advertised pause frame use: No
> Advertised auto-negotiation: Yes
> Link partner advertised link modes: 1000baseT/Full
> Link partner advertised pause frame use: No
> Link partner advertised auto-negotiation: No
> Speed: 1000Mb/s
> Duplex: Full
> Port: MII
> PHYAD: 0
> Transceiver: external
> Auto-negotiation: on
> Link detected: yes
>
> So basically even after the wan port went to 100/full, eth0 doesn't know about it (and it only supports gig speed (probably to the local switch) anyway. I am seeing dropped packets, so this would support this theory.
>
> First I ran some tests with only that, then I set SQM to 90 megabit/s results here:
>
> http://swm.pp.se/aqm/rrul_150629-cake-5.tar
> http://swm.pp.se/aqm/rrul_150629-cake-6.tar
>
> Next I am going to test wireless but it seems something has gone wrong because I can't get the wireless to enable properly, so that'll have to be for the next email after I fix that problem.
>
> --
> Mikael Abrahamsson email: ***@swm.pp.se_______________________________________________
> Cerowrt-devel mailing list
> Cerowrt-***@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cerowrt-devel

[d***@reed.com]

I would love to try out cake in my environment. However, as a non-combatant, it would be nice to have an instruction sheet on how to set the latest version up, and what hardware it works best on (WRT1200AC?). Obviously this is a work in progress, so that will change, but it would be nice to have a summarized wiki page.

[Jonathan Morton]

I'd also like to be able to try it out on CPE hardware. However, what I've
got is a Buffalo H300N, so I'll need build instructions (preferably
starting from an existing stock build) as well as setup.

The Buffalo isn't as powerful as some others, being based around a 34K core.

- Jonathan Morton

[Sebastian Moeller]

HI Mikael,, hi Jonathan,

> [...]
>
> These are the results from 50M and 500M, also including 50up and 50down that I added to my test suite script.
>
> http://swm.pp.se/aqm/rrul_150629-cake-4.tar
>

Now both ingress and egress are up to roughly 455Mbps from roughly 360 with cake just playing leaf qdisc for HTB. This looks even better than before…

Best Regards
Sebastian

[Dave Taht]

On Mon, Jun 29, 2015 at 6:42 AM, Sebastian Moeller <***@gmx.de> wrote:
> HI Mikael,, hi Jonathan,
>
>> [...]
>>
>> These are the results from 50M and 500M, also including 50up and 50down that I added to my test suite script.
>>
>> http://swm.pp.se/aqm/rrul_150629-cake-4.tar
>>
>
> Now both ingress and egress are up to roughly 455Mbps from roughly 360 with cake just playing leaf qdisc for HTB. This looks even better than before


350 *usec* induced delay on the 50mbit rrul_be test. w00t!

Most of the tests compare well with the reference rangeley data now. I
would like a 900mbit soft shaped result.

1.2ms at 500mbit. Less of a w00t. Possible it is coming from elsewhere
on that path (fq or fq_codel on the server and client?)

cake currently peels at 1ms / flows (more or less)... NAPI is an
issue... hw mq an issue...

There are a half dozen things in the mvneta driver I would try to
reduce it's latency more. The easy ones:

reduce this to 16:

netif_napi_add(dev, &pp->napi, mvneta_poll, NAPI_POLL_WEIGHT);

Reduce this to 24: (this will also reduce the max outstanding stuff in
the driver by a LOT, but is still not BQL!)

/* Max number of allowed TCP segments for software TSO */
#define MVNETA_MAX_TSO_SEGS 100

Both of the will improve read side latency at the cost of more sirqs.

I do not know what reducing these will do, and would test both of the
above separately.

/* Coalescing */
#define MVNETA_TXDONE_COAL_PKTS 1
#define MVNETA_RX_COAL_PKTS 32
#define MVNETA_RX_COAL_USEC 100

As for cake itself, eric dumazet told us we dont need atomic ops in it,
and peeling at at even lower threshold has some appeal (to me, anyway)

attached is a patch for that, put it in your feeds/cero/kmod_sched_cake/patches
directory, rebuild (make package/kmod-sched-cake/{clean,compile,install})

(bump up the makefile rel number also, if you want)





> Best Regards
> Sebastian
> _______________________________________________
> Cerowrt-devel mailing list
> Cerowrt-***@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cerowrt-devel



--
Dave TÀht
worldwide bufferbloat report:
http://www.dslreports.com/speedtest/results/bufferbloat
And:
What will it take to vastly improve wifi for everyone?
https://plus.google.com/u/0/explore/makewififast

[Sebastian Moeller]

Hi List,

On Jun 29, 2015, at 18:44 , Dave Taht <***@gmail.com> wrote:

> On Mon, Jun 29, 2015 at 6:42 AM, Sebastian Moeller <***@gmx.de> wrote:
>> HI Mikael,, hi Jonathan,
>>
>>> [...]
>>>
>>> These are the results from 50M and 500M, also including 50up and 50down that I added to my test suite script.
>>>
>>> http://swm.pp.se/aqm/rrul_150629-cake-4.tar
>>>
>>
>> Now both ingress and egress are up to roughly 455Mbps from roughly 360 with cake just playing leaf qdisc for HTB. This looks even better than before…
>
> 350 *usec* induced delay on the 50mbit rrul_be test. w00t!
>
> Most of the tests compare well with the reference rangeley data now. I
> would like a 900mbit soft shaped result.

Make sure to use the correct per packet overhead of 18 + 20, on gigabit ethernet inter-frame gap and preamble cost 20 Bytes worth of data. So with a MTU of 1500 thee is no issue (900 * (1538/1518) = 911.85770751 Mbps) but the smaller the packets get:

900 * (MTU+38/MTU+18) = 1000
(MTU+38/MTU+18) = (1000 / 900)
MTU+38 = (10 / 9) * (MTU + 18)
MTU + 38 = (10/9) * MTU + (10/9)*18
38 - (10/9)*18 = (10/9) * MTU - (9/9)MTU
38 - (10/9)*18 = (1/9) MTU
MTU = (38 - ((10/9)*18))*9 = 162
So for TCP/IPv4 MSS < 122 the shaper will not keep the ethernet hardware queues empty…

On the other hand shaping at
1000/(88/64) = 727.272727273 Mbps
should make sure that even at minimal packet size of 64byte shaping would still be keeping the ethernet queues “empty-ish”. If the 1200ac can shape at 900 I would rather specify the correct overhead though.


To make things a bit trickier, depending on the interface used the kernel will already account for the standards ethernet header without the frame check sequence, so I would guess in the 900Mbps soft shaper on ethN scenario one would need to add a per packet overhead of 24 bytes. If someone in the know could double check that reasoning I would be much obliged…


Best Regards
Sebastian



>
> 1.2ms at 500mbit. Less of a w00t. Possible it is coming from elsewhere
> on that path (fq or fq_codel on the server and client?)
>
> cake currently peels at 1ms / flows (more or less)... NAPI is an
> issue... hw mq an issue...
>
> There are a half dozen things in the mvneta driver I would try to
> reduce it's latency more. The easy ones:
>
> reduce this to 16:
>
> netif_napi_add(dev, &pp->napi, mvneta_poll, NAPI_POLL_WEIGHT);
>
> Reduce this to 24: (this will also reduce the max outstanding stuff in
> the driver by a LOT, but is still not BQL!)
>
> /* Max number of allowed TCP segments for software TSO */
> #define MVNETA_MAX_TSO_SEGS 100
>
> Both of the will improve read side latency at the cost of more sirqs.
>
> I do not know what reducing these will do, and would test both of the
> above separately.
>
> /* Coalescing */
> #define MVNETA_TXDONE_COAL_PKTS 1
> #define MVNETA_RX_COAL_PKTS 32
> #define MVNETA_RX_COAL_USEC 100
>
> As for cake itself, eric dumazet told us we dont need atomic ops in it,
> and peeling at at even lower threshold has some appeal (to me, anyway)
>
> attached is a patch for that, put it in your feeds/cero/kmod_sched_cake/patches
> directory, rebuild (make package/kmod-sched-cake/{clean,compile,install})
>
> (bump up the makefile rel number also, if you want)
>
>
>
>
>
>> Best Regards
>> Sebastian
>> _______________________________________________
>> Cerowrt-devel mailing list
>> Cerowrt-***@lists.bufferbloat.net
>> https://lists.bufferbloat.net/listinfo/cerowrt-devel
>
>
>
> --
> Dave Täht
> worldwide bufferbloat report:
> http://www.dslreports.com/speedtest/results/bufferbloat
> And:
> What will it take to vastly improve wifi for everyone?
> https://plus.google.com/u/0/explore/makewififast

[Mikael Abrahamsson]

On Mon, 29 Jun 2015, Dave Taht wrote:

> Most of the tests compare well with the reference rangeley data now. I
> would like a 900mbit soft shaped result.

http://swm.pp.se/aqm/rrul_150629-cake-9.tar

Above is with cake as link layer adaptation and pie, fq_codel and cake as
900 meg bidirectional shaped.

Then I removed the ll adaptation and changed to simplest.qos and ran
another test with cake:

http://swm.pp.se/aqm/rrul_150629-cake-10.tar

Now I'm going to try out if my new version with your cake patch will boot
and work.

--
Mikael Abrahamsson email: ***@swm.pp.se

[Mikael Abrahamsson]

On Mon, 29 Jun 2015, Dave Taht wrote:

> attached is a patch for that, put it in your feeds/cero/kmod_sched_cake/patches
> directory, rebuild (make package/kmod-sched-cake/{clean,compile,install})

Test results: http://swm.pp.se/aqm/rrul_150629-cake-11.tar

--
Mikael Abrahamsson email: ***@swm.pp.se

[Mikael Abrahamsson]

On Sun, 28 Jun 2015, Sebastian Moeller wrote:

After the modification, the installed policy looks better, I am now not
getting any drops in iperf3 (=ECN is working). Please see link to test
results. From my bw monitoring script, it looks like I am receiving more
than 500 megabit/s though, even though sending is spot on at 500
megabit/s. Yeah, the receiving limiter isn't working at all, I see now in
rrul_be that I am getting full gige speed download.

***@OpenWrt:~# tc -d qdisc
qdisc cake 8005: dev eth0 root refcnt 9 bandwidth 500Mbit diffserv4 flows
raw
linklayer ethernet overhead 42
qdisc ingress ffff: dev eth0 parent ffff:fff1 ----------------
qdisc mq 0: dev eth1 root
qdisc fq_codel 0: dev eth1 parent :1 limit 1024p flows 1024 quantum 300
target 5.0ms interval 100.0ms ecn
qdisc fq_codel 0: dev eth1 parent :2 limit 1024p flows 1024 quantum 300
target 5.0ms interval 100.0ms ecn
qdisc fq_codel 0: dev eth1 parent :3 limit 1024p flows 1024 quantum 300
target 5.0ms interval 100.0ms ecn
qdisc fq_codel 0: dev eth1 parent :4 limit 1024p flows 1024 quantum 300
target 5.0ms interval 100.0ms ecn
qdisc fq_codel 0: dev eth1 parent :5 limit 1024p flows 1024 quantum 300
target 5.0ms interval 100.0ms ecn
qdisc fq_codel 0: dev eth1 parent :6 limit 1024p flows 1024 quantum 300
target 5.0ms interval 100.0ms ecn
qdisc fq_codel 0: dev eth1 parent :7 limit 1024p flows 1024 quantum 300
target 5.0ms interval 100.0ms ecn
qdisc fq_codel 0: dev eth1 parent :8 limit 1024p flows 1024 quantum 300
target 5.0ms interval 100.0ms ecn
qdisc fq_codel 0: dev ifb4eth0 root refcnt 2 limit 1024p flows 1024
quantum 300 target 5.0ms interval 100.0ms ecn
***@OpenWrt:~# tc -d class show dev eth0
class cake 8005:6 parent 8005:
class cake 8005:44b parent 8005:
class cake 8005:50c parent 8005:
class cake 8005:5fc parent 8005:
class cake 8005:716 parent 8005:
class cake 8005:7b3 parent 8005:
class cake 8005:b43 parent 8005:
class cake 8005:c7f parent 8005:
class cake 8005:c87 parent 8005:
class cake 8005:d8d parent 8005:
***@OpenWrt:~# tc -d class show dev ifb4eth0
class fq_codel :32c parent none

I have put the results at http://swm.pp.se/aqm/rrul_150629-cake-1.tar

I also have a small shell script that prints out pps and bw once per
second that I run on my ubuntu machine, just to see approx how much
traffic is being used. Might be useful for someone.

http://swm.pp.se/aqm/measurebw.sh

--
Mikael Abrahamsson email: ***@swm.pp.se

[Sebastian Moeller]

Hi Mikael,

thanks a lot.

On Jun 28, 2015, at 19:32 , Mikael Abrahamsson <***@swm.pp.se> wrote:

> On Sun, 28 Jun 2015, Sebastian Moeller wrote:
>
>> This looks great, could you by any chance confirm that the GUI does allow to configure cake and that you can or can not set the overhead for cake in the link layer adjustments (LLA) tab? (select cake as link layer adjustment method, and put 42 into the overhead field and report the output of “tc -d qdisc” before and after selecting cake as LLA). @Toke: If that works, I think we can safely push these changes into the openwrt repositories...
>
> Here is the output. What I don't see is both ingress and egress ECN markings even though I have selected this in the advanced configuration under Queue Discipline.

Good point, I have not connected these fields with cake yet, will do in the near future. I believe cake defaults to ECN, but if no packets are marked during a test, but rather dropped, something is fishy...

>
> Before changing LLA:
>
> ***@OpenWrt:~# tc -d qdisc
> qdisc htb 1: dev eth0 root refcnt 9 r2q 10 default 12 direct_packets_stat 0 ver 3.17 direct_qlen 532
> qdisc cake 110: dev eth0 parent 1:11 unlimited diffserv4 flows raw
> qdisc cake 120: dev eth0 parent 1:12 unlimited diffserv4 flows raw
> qdisc cake 130: dev eth0 parent 1:13 unlimited diffserv4 flows raw
> qdisc ingress ffff: dev eth0 parent ffff:fff1 ----------------
> qdisc mq 0: dev eth1 root
> qdisc fq_codel 0: dev eth1 parent :1 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
> qdisc fq_codel 0: dev eth1 parent :2 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
> qdisc fq_codel 0: dev eth1 parent :3 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
> qdisc fq_codel 0: dev eth1 parent :4 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
> qdisc fq_codel 0: dev eth1 parent :5 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
> qdisc fq_codel 0: dev eth1 parent :6 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
> qdisc fq_codel 0: dev eth1 parent :7 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
> qdisc fq_codel 0: dev eth1 parent :8 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
> qdisc htb 1: dev ifb4eth0 root refcnt 2 r2q 10 default 12 direct_packets_stat 0 ver 3.17 direct_qlen 32
> qdisc cake 110: dev ifb4eth0 parent 1:11 unlimited diffserv4 flows raw
> qdisc cake 120: dev ifb4eth0 parent 1:12 unlimited diffserv4 flows raw
> qdisc cake 130: dev ifb4eth0 parent 1:13 unlimited diffserv4 flows raw
>
> After changing LLA:
>
> ***@OpenWrt:~# tc -d qdisc
> qdisc htb 1: dev eth0 root refcnt 9 r2q 10 default 12 direct_packets_stat 0 ver 3.17 direct_qlen 532
> linklayer ethernet overhead 42
> qdisc cake 110: dev eth0 parent 1:11 unlimited diffserv4 flows raw
> qdisc cake 120: dev eth0 parent 1:12 unlimited diffserv4 flows raw
> qdisc cake 130: dev eth0 parent 1:13 unlimited diffserv4 flows raw
> qdisc ingress ffff: dev eth0 parent ffff:fff1 ----------------
> qdisc mq 0: dev eth1 root
> qdisc fq_codel 0: dev eth1 parent :1 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
> qdisc fq_codel 0: dev eth1 parent :2 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
> qdisc fq_codel 0: dev eth1 parent :3 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
> qdisc fq_codel 0: dev eth1 parent :4 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
> qdisc fq_codel 0: dev eth1 parent :5 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
> qdisc fq_codel 0: dev eth1 parent :6 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
> qdisc fq_codel 0: dev eth1 parent :7 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
> qdisc fq_codel 0: dev eth1 parent :8 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
> qdisc htb 1: dev ifb4eth0 root refcnt 2 r2q 10 default 12 direct_packets_stat 0 ver 3.17 direct_qlen 32
> linklayer ethernet overhead 42
> qdisc cake 110: dev ifb4eth0 parent 1:11 unlimited diffserv4 flows raw
> qdisc cake 120: dev ifb4eth0 parent 1:12 unlimited diffserv4 flows raw
> qdisc cake 130: dev ifb4eth0 parent 1:13 unlimited diffserv4 flows raw

This is showing tha the cake setup went wrong. It is using HTB as shaper instead of cake. There should be no cake line. Could you post your /usr/lib/sqm/simple.qos file please.

Best Regards
Sebastian

>
> --
> Mikael Abrahamsson email: ***@swm.pp.se

[Dave Taht]

On Fri, Jun 26, 2015 at 9:18 AM, Jonathan Morton <***@gmail.com> wrote:
> Hypothesis: this might have to do with the receive path. Some devices might
> have more capacity than others to buffer inbound packets until the CPU can
> get around to servicing them.

*Good* hypothesis. I am certain I have seen this on multiple occasions
on other hardware. Hard to confirm.

Wet paint... so I finally got off my arse and looked at the driver this morning.

given that this is a multi-core box, I would lean towards a smaller
napi_poll_weight, which unfortunately is a constant (64) in the code.
4 cores can take interrupts faster. (and I hate napi on routers) I
have sometimes longed for an IQL (ingress queue limits) to also handle
differences in packet size, dynamically changing the poll weight based
on load - increasing it for loads with lots of small packets,
decreasing it for lots of big packets.

Furthermore this thing is doing software gro (up to 64 packets at a
time) which is a LOT of processing at this layer in the stack.

Its a two line patch to cut the weight to 16, but I have never managed
to get a working build for this platform.

> - Jonathan Morton
>
>
> _______________________________________________
> Cerowrt-devel mailing list
> Cerowrt-***@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cerowrt-devel
>



--
Dave Täht
worldwide bufferbloat report:
http://www.dslreports.com/speedtest/results/bufferbloat
And:
What will it take to vastly improve wifi for everyone?
https://plus.google.com/u/0/explore/makewififast

[Sebastian Moeller]

HI Mikael,


On Jun 26, 2015, at 16:49 , Mikael Abrahamsson <***@swm.pp.se> wrote:

> On Fri, 26 Jun 2015, Mikael Abrahamsson wrote:
>
>> Btw, I tried WNDR3800 setting it to 100/100 SQM. It seems to max out around 25-30k PPS, but the difference is that when the CPU is full, it seems to delay/ECN-mark packets because there are no packets lost. When the WRT1200AC runs out of CPU it starts dropping packets. I always have 0 packets lost with the WNDR3800 when doing iperf3 testing. I found this difference interesting, wonder where in the forwarding path the WRT1200AC loses packets?
>
> I checked again, and my WDR4900 with same setup doesn't lose packets either. Even at 99% sirq, no packets are lost.

Tangent: What is the shaper rate the wdr4900 can push with sqm-scripts? (Before your 1200ac results the ppc-soc in the wdr4900 looked like the finest little router platform in the last years, too bad it was ignored by the mass market...)

>
> WRT1200AC starts to lose packets at 500 megabit/s SQM around MSS 300 and lower. If I turn off gso, tso and gro, I have to go to MSS 600 and above to avoid packet loss.

So the offloads buy roughly a doubling of the achievable packet rate, not bad, but as your results show also knot essential.


Best Regards
Sebastian

>
> Does flent check for packet loss at all? Perhaps it's something to look into, because with ECN we really don't want to see any packets lost and this might be good to include test results for.
>
> --
> Mikael Abrahamsson email: ***@swm.pp.se

[Mikael Abrahamsson]

On Fri, 26 Jun 2015, Sebastian Moeller wrote:

> Tangent: What is the shaper rate the wdr4900 can push with
> sqm-scripts? (Before your 1200ac results the ppc-soc in the wdr4900
> looked like the finest little router platform in the last years, too bad
> it was ignored by the mass market...)

Well, if I set SQM to 500 megabit/s and MSS to 300 I am able to get 70
megabit/s of iperf3 through it at 42k PPS. At default MSS I get 150
megabit/s at 25k PPS through it. This is at 100% sirq.

Does this help, or do you want me to do any other tests. I have the
WDR4900 powered up and on my desk right now.

--
Mikael Abrahamsson email: ***@swm.pp.se

[Dave Taht]

On Fri, Jun 26, 2015 at 9:36 AM, Mikael Abrahamsson <***@swm.pp.se> wrote:
> On Fri, 26 Jun 2015, Sebastian Moeller wrote:
>
>> Tangent: What is the shaper rate the wdr4900 can push with
>> sqm-scripts? (Before your 1200ac results the ppc-soc in the wdr4900 looked
>> like the finest little router platform in the last years, too bad it was
>> ignored by the mass market...)
>
>
> Well, if I set SQM to 500 megabit/s and MSS to 300 I am able to get 70
> megabit/s of iperf3 through it at 42k PPS. At default MSS I get 150
> megabit/s at 25k PPS through it. This is at 100% sirq.
>
> Does this help, or do you want me to do any other tests. I have the WDR4900
> powered up and on my desk right now.

I am never allergic to somene running a comprehensive flent suite
through something, and sticking the results up somewhere.

:)

There are always more surprises and more things we seem to have to wackamole.

>
>
> --
> Mikael Abrahamsson email: ***@swm.pp.se
> _______________________________________________
> Cerowrt-devel mailing list
> Cerowrt-***@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cerowrt-devel



--
Dave Täht
worldwide bufferbloat report:
http://www.dslreports.com/speedtest/results/bufferbloat
And:
What will it take to vastly improve wifi for everyone?
https://plus.google.com/u/0/explore/makewififast

[Mikael Abrahamsson]

On Fri, 26 Jun 2015, Dave Taht wrote:

> I am never allergic to somene running a comprehensive flent suite
> through something, and sticking the results up somewhere.

http://swm.pp.se/aqm/wdr4900-150626-9.tar

Happy no sneezing!

--
Mikael Abrahamsson email: ***@swm.pp.se

[Jim Reisert AD1C]

On Tue, Jun 23, 2015 at 1:20 AM, Sebastian Moeller wrote:

> Most likely not. Check http://wiki.openwrt.org/doc/howto/sqm.
> Rich published a great set of instructions for setting up sqm-scripts under OpenWRT proper.

Thanks, Sebastian. I did as you suggested.

It was eye-opening looking at the results from
http://www.dslreports.com/speedtest both before and after enabling
SQM. Wow!

- Jim

--
Jim Reisert AD1C, <***@alum.mit.edu>, http://www.ad1c.us

[Dave Taht]

On Tue, Jun 23, 2015 at 7:35 AM, Jim Reisert AD1C
<***@alum.mit.edu> wrote:
> On Tue, Jun 23, 2015 at 1:20 AM, Sebastian Moeller wrote:
>
>> Most likely not. Check http://wiki.openwrt.org/doc/howto/sqm.
>> Rich published a great set of instructions for setting up sqm-scripts under OpenWRT proper.
>
> Thanks, Sebastian. I did as you suggested.
>
> It was eye-opening looking at the results from
> http://www.dslreports.com/speedtest both before and after enabling
> SQM. Wow!

I think one of the most effective graphs we ever did was the "you are
here. You could be here." one.

I would like to add it to this:

http://www.dslreports.com/speedtest/results/bufferbloat?up=1

>
> - Jim
>
> --
> Jim Reisert AD1C, <***@alum.mit.edu>, http://www.ad1c.us



--
Dave Täht
worldwide bufferbloat report:
http://www.dslreports.com/speedtest/results/bufferbloat
And:
What will it take to vastly improve wifi for everyone?
https://plus.google.com/u/0/explore/makewififast

[Sebastian Moeller]

Hi Alan,

On Jun 19, 2015, at 19:35 , Alan Jenkins <***@gmail.com> wrote:

> On 19/06/2015, Sebastian Moeller <***@gmx.de> wrote:
>> Hi Alan,
>>
>> excellent, thanks a million.
>>
>> On Jun 19, 2015, at 16:44 , Alan Jenkins
>> <***@gmail.com> wrote:
>>
>>> Hi
>>>
>>> I guess I've done the complementary half to Seb's test :). Basically
>>> "cake overhead 40" didn't work, but that's the fault of cake in this
>>> build. Or tc, as Johnathan suggested. (The "cake atm" part seems to
>>> work, as per my previous test).
>>
>> Great!
>>
>>>
>>> "tc qdisc" says "cake overhead 0", as Sebastian noticed. And the test
>>> results show "cake overhead 40" does not give a measurable
>>> improvement. But "tc stab overhead 40" does.
>>>
>>> I ran this test with the updated sqm-scripts and I think they're doing
>>> the right thing.
>>
>> Thanks for testing this, especially as I can not due to a lack of an
>> ADSL-link (and lack of cake actually, last I looked all I could find was
>> cookies in my browser and a promise of pie in my router)
>>
>>>
>>>
>>> Method:
>>>
>>> I used the updated files from sqm-scripts,
>>>
>>> (once I remembered to mark them executable. Lacking that causes a
>>> failure with no error messages, because sqm-scripts checks before
>>> running them :)
>>>
>>> but didn't bother updating & using luci-app-sqm.
>>
>> Ah, okay, I guess I did test this part with Dave’s help, so this should
>> work with the most recent sqm.lua.
>>
>>>
>>> The test was to compare netperf-runner results - ping during combined
>>> upload & download - for "overhead 40" and "overhead 0". I tested both
>>> values of linklayer_adaptation_mechanism.
>>>
>>> I had to repeat 6 times (60s per run for each overhead) because of
>>> random variation in the range of 3-4ms. I alternated "overhead 40"
>>> and "overhead 0" to try and exclude longer-term variation effects.
>>>
>>> With "stab overhead 40", median latency was better by about 3-4ms.
>>> With "cake overhead 40", there is no such effect.
>>
>> Intersting, when I still had a 6M/1M ADSL link, I saw much larger latency
>> under load increases when setting the per packet overhead to small, but I
>> had my egress shaper running at 100% of line rate, so the system was rigged
>> for maximum effect that way. How are your shapers typically set?
>
> For this test I try to push it, today I used 95%. I started trying
> 100%, which is still much better than unshaped. I was scared off by
> the random variation, I think it was higher at 100%.

Fair enough, but only at 100% you will notice if the overhead is off by a single byte… You are running sqm on the pppoe interface, correct?

>
> For long term use I reduce it, because I've seen the line rate vary
> slightly. (1020k up today, 912 a while back. Currently it reports a
> "max" figure I don't understand, it's about 1100 despite being
> rebooted daily. 16390k down).

A pity that your line is that flaky, but it only differs between reboots and does not change during the day? As far as I know only SRA would allow rate changes while the sync stays up. So you either have unscheduled resyncs during the day which drag in a rate change or between days. Anyway aiming with the shaper rate at below the best case sync seems wise unless you want to fiddle with modem and router daily. (It would be sweet if one could query the actual sync speed from XDSL-modems from the LAN side in a standardized fashion, so sqm could learn to set the shaper automatically).

Best Regards
Sebastian

>
> Alan