[Cerowrt-devel] Will there ever be a stable version of cerowrt?

Discussion:

Stephen Hemminger

2014-02-07 17:01:46 UTC

Any progress on getting a stable version? Recent versions seem to be targeted
at IPv6 and other things which do not impact me for home use. What does matter
is fixing the wireless hanging issue on 2.4Ghz.

Jim Gettys

2014-02-07 17:45:07 UTC

Permalink

On Fri, Feb 7, 2014 at 12:01 PM, Stephen Hemminger <

Post by Stephen Hemminger
Any progress on getting a stable version? Recent versions seem to be targeted
at IPv6 and other things which do not impact me for home use. What does matter
is fixing the wireless hanging issue on 2.4Ghz.

Dave's (and I) are pedalling as hard and fast as he can at the moment.
And Dave has other responsibilitys that consume time.

The IPv6 problem consumed a lot of time, and you do care.

Comcast has lit up a big chunk of their network with Native IPv6 and
tickled a latent disastrous bug (now fixed upstream in OpenWrt). That we
simultaneously had a disk failure on our machine that does much of
OpenWrt's and CeroWrt's builds is more than unfortunate and so we end up
spending our time on trying to get the build situation fixed once and for
all (by getting machines and people to set up and manage the build system
going forward at a scale previously unimaginable to OpenWrt/CeroWrt).

And yes, we're making progress on all fronts. 2 juicy servers are teed up
awaiting a single signature (thanks Mikael!) and I have a strong lead on
both large amounts of additional cloud cycles and volunteers to help admin
all this so it doesn't happen again and the build system is as fast as it
should be (continuous across all supported architectures)

It would be very help if this mailing list could at least nominate bugs
that they think are showstoppers before declaring it "soup"....

But please don't jiggle the chef (Dave's) arm expecting immediate involved
response in that discussion while sorting through it; it will reduce the
time to see what if anything remains.

We keep finding more and more people running CeroWrt (not subscribed to
this list necessarily) we had no clue were doing so.

I know I have one test I have to run on the first build he can give me (one
Mike Frankston ran into) that might be in that category if I can easily
reproduce it, as I think I ran into it myself recently. It would cause a
support nightmare as it's a "new to CeroWrt" thing triggered from an
obvious thing you want and often need to configure.

But we gotta be able to build it, close out those show stoppers and test
the fixes, and get that build installed on enough boxes in different
environments for Dave send out an announcement that people should try an
upgrade.

We know it has to be ASAP.

- Jim

Post by Stephen Hemminger
_______________________________________________
Cerowrt-devel mailing list
https://lists.bufferbloat.net/listinfo/cerowrt-devel

Dave Taht

2014-02-07 17:57:53 UTC

Permalink

On Fri, Feb 7, 2014 at 12:01 PM, Stephen Hemminger

Do I detect a plainative note? I share it...

I'm not aware of any problems on 2.4ghz in the last specific release I
did for comcast. It's been in day-to-day use here since I put it out.

http://snapon.lab.bufferbloat.net/~cero2/cerowrt/wndr/comcast/3.10.28-4/

Feel free to give it a shot. The stuff that made it comcast specific
(co-existence with HE tunnels) is solved, so the next gen will be
generic.

I AM painfully aware that ht40+ at 5ghz appears to have broken on
several channels (Seems to work on 36 and fail on 44) There was a
fresh merge from wireless-testing recently, that I hope fixes it.

I DO plan a stable release this month if possible, and at the very
least a code freeze fairly soon.

in cerowrt head are fixes for ipv6 tunnel co-existence, mostly
everything now using procd, fixed upnp support (think that landed in
the release above), cleanups of the SQM code for eventual submittal to
mainline landed also.

*Please note* that a source of hangs in the last several cero releases
was an oopsy in turning off logging for ahcp and babeld, which filled
up available memory rapidly. EVERYBODY: Do turn send those logs to
/dev/null if you haven't already in the relevant /etc/config/ files
and look for other sources of memory growth from logs.

the remaining problems are:

got competing work requirements terribly pressing, final pie release
needs to be added, (kind of wanted sch_fq and the other new qdisc from
3.13, too) a bunch of ietf drafts are in the nascent stage and stuck
there, a build server crashed and a whole string of openwrt builds
failed to build, still need /etc/init.d/babeld support for source
specific routing and babel to run out of procd, package signing is
broken, and worst of the cerowrt build broke last week due to some
changes in the openwrt build process I still don't understand.

There is still one unaligned_instruction trap left to beat that
appears to be triggered by odhcpd.

***@comcast-gw:~# cat /sys/kernel/debug/mips/unaligned_instructions
18000

One of the things that scares me is that a bunch of openwrt targets
are migrating to 3.13 and if I miss a window the very stable 3.10.28
release is going to get replaced upstream with 3.13 and resultant
headaches. (and benefits)

As my own deadlines slip everywhere, I made a big push earlier this
week to appeal to many folk to get dnsmasq + dnssec more widely tested
on other platforms in the hope that whenever I got unburied it would
be stable enough to toss into cero. (that's going along swimmingly on
x86 and arm so far)

I hope to replace the failed server this tonight and burn sunday on
trying to resolve the problems above. The currently borked srcs are on
github.

--
Dave Täht

Fixing bufferbloat with cerowrt: http://www.teklibre.com/cerowrt/subscribe.html

Sebastian Moeller

2014-02-07 18:37:46 UTC

Permalink

Hi Dave,

Post by Jim Gettys
On Fri, Feb 7, 2014 at 12:01 PM, Stephen Hemminger

With ht40+ on 44 I get MCS index 7 with transmigrate 150 reported on my macbook, that according to the table in http://en.wikipedia.org/wiki/IEEE_802.11n-2009 means 40MHz channel, with netperf-wrapper I get an accumulate of ~100Mbits/s (which actually sounds too good to be true I would have expected 75 tops, but with the macbook hogging all tops, there seems to be less wasted airtime for direction changes). So at least with country code GE channel 44 is capable of operating in ht40+ mode.

Post by Jim Gettys
I DO plan a stable release this month if possible, and at the very
least a code freeze fairly soon.
in cerowrt head are fixes for ipv6 tunnel co-existence, mostly
everything now using procd, fixed upnp support (think that landed in
the release above), cleanups of the SQM code for eventual submittal to
mainline landed also.
*Please note* that a source of hangs in the last several cero releases
was an oopsy in turning off logging for ahcp and babeld, which filled
up available memory rapidly. EVERYBODY: Do turn send those logs to
/dev/null if you haven't already in the relevant /etc/config/ files
and look for other sources of memory growth from logs.
got competing work requirements terribly pressing, final pie release
needs to be added, (kind of wanted sch_fq and the other new qdisc from
3.13, too) a bunch of ietf drafts are in the nascent stage and stuck
there, a build server crashed and a whole string of openwrt builds
failed to build, still need /etc/init.d/babeld support for source
specific routing and babel to run out of procd, package signing is
broken, and worst of the cerowrt build broke last week due to some
changes in the openwrt build process I still don't understand.
There is still one unaligned_instruction trap left to beat that
appears to be triggered by odhcpd.
18000

So with my non-working IP6 on 3.10.28-1, I get:
***@nacktmulle:~# cat /sys/kernel/debug/mips/unaligned_instructions
0
***@nacktmulle:~# uptime
19:35:13 up 11 days, 22:46, load average: 0.09, 0.43, 1.06

so most of the traps seem fixed!

Post by Jim Gettys
One of the things that scares me is that a bunch of openwrt targets
are migrating to 3.13 and if I miss a window the very stable 3.10.28
release is going to get replaced upstream with 3.13 and resultant
headaches. (and benefits)

Do yu know what their plan is? Put out a stable version an maintain that or switch to a rolling release cycle?

Best Regards & many Thanks for doing all this
Sebastian

Post by Jim Gettys
As my own deadlines slip everywhere, I made a big push earlier this
week to appeal to many folk to get dnsmasq + dnssec more widely tested
on other platforms in the hope that whenever I got unburied it would
be stable enough to toss into cero. (that's going along swimmingly on
x86 and arm so far)
I hope to replace the failed server this tonight and burn sunday on
trying to resolve the problems above. The currently borked srcs are on
github.
--
Dave Täht
Fixing bufferbloat with cerowrt: http://www.teklibre.com/cerowrt/subscribe.html
_______________________________________________
Cerowrt-devel mailing list
https://lists.bufferbloat.net/listinfo/cerowrt-devel

Dave Taht

2014-02-07 18:43:41 UTC

Permalink

Post by Sebastian Moeller
Hi Dave,

Post by Jim Gettys
On Fri, Feb 7, 2014 at 12:01 PM, Stephen Hemminger

What release exactly? I confess to have merely seen the bug on a release
or two back.

You should get MCS15, btw.

Post by Sebastian Moeller

Post by Jim Gettys
There is still one unaligned_instruction trap left to beat that
appears to be triggered by odhcpd.
18000

0
19:35:13 up 11 days, 22:46, load average: 0.09, 0.43, 1.06

No, not fixed, it's triggered by ipv6 traffic.

Post by Sebastian Moeller
so most of the traps seem fixed!

Do yu know what their plan is? Put out a stable version an maintain that or switch to a rolling release cycle?

I have had few detailed conversations with the #openwrt-devel folk of
late. I have been ignoring irc in favor of doing work.

There is a ton of good stuff landing in openwrt, let me forward
something in a second...

Post by Sebastian Moeller
Best Regards & many Thanks for doing all this
Sebastian

--
Dave Täht

Fixing bufferbloat with cerowrt: http://www.teklibre.com/cerowrt/subscribe.html

Sebastian Moeller

2014-02-07 18:57:10 UTC

Permalink

Hi Dave,

Post by Dave Taht

Post by Sebastian Moeller
Hi Dave,

Post by Jim Gettys
On Fri, Feb 7, 2014 at 12:01 PM, Stephen Hemminger

What release exactly? I confess to have merely seen the bug on a release
or two back.

Oh 3.10.28-1, my working place and the router position are badly matched, so I only get 80 to 150 mbits/s there, but it is nice to cover the full flat with one router, so I will have to live with that. Question to all, I have set up my WNDR3700 on the vertical stand on a shelf above my head, is that the ideal position or has anyone other recommendations how to rotate the router? (The overhead position is a given, since I want this to be safely out of reach of my two toddlers).

Post by Dave Taht
You should get MCS15, btw.

On the occasional day I get a bit more, also dependent on how the other people in the apartment are located ;)

Post by Dave Taht

Post by Sebastian Moeller

Post by Jim Gettys
There is still one unaligned_instruction trap left to beat that
appears to be triggered by odhcpd.
18000

0
19:35:13 up 11 days, 22:46, load average: 0.09, 0.43, 1.06

No, not fixed, it's triggered by ipv6 traffic.

Ah, okay I will then dart looking at those again once I can exercise odhcp with ip6 traffic...

Post by Dave Taht

Post by Sebastian Moeller
so most of the traps seem fixed!

Do yu know what their plan is? Put out a stable version an maintain that or switch to a rolling release cycle?

I have had few detailed conversations with the #openwrt-devel folk of
late. I have been ignoring irc in favor of doing work.
There is a ton of good stuff landing in openwrt, let me forward
something in a second…

Much appreciated.

Best Regards
Sebastian

Post by Dave Taht

Post by Sebastian Moeller
Best Regards & many Thanks for doing all this
Sebastian

--
Dave Täht
Fixing bufferbloat with cerowrt: http://www.teklibre.com/cerowrt/subscribe.html