Ethernet Switch Framework

[Aleksandr Rybalko]

Hi,

Sorry for cross posting. I just thought this might interesting to
review for subscribers on both MLs.

I glad to introduce working version of Ethernet Switch Framework. of
course, here is so many things to do, but it already work and can help
us in some situations for embedded devices.

here is the patch:
http://my.ddteam.net/files/2012-01-20_switch_framework.patch

It include sys/mips/conf/AR7240, that together with hints file is good
example for typical AR7240 setup.

Since it still don't have any documentation, I will try to explain
it here.


The framework contain 3 basic parts:
1. core module, which handle ioctl calls and interact with driver.
2. Various bus glue (now it is obio(mem), mii(MDIO) and IIC in near
future)
3. drivers. Atheros AR8x16, Broadcom BCM53xx, Ralink RT305xF, Realtek
RTL8305/09
4. FloatPHY, pseudo driver which find master switchX device and ask his
PHY reg's.
5. switchctl utility.

Utility.
Currently can do
switchctl /dev/switch0 (get|set) (reg|port|vlan) [flags]

get/set port: get or set port flags:
"IngressCheck" - put port into VLAN mode, drop packets which have 802.1q
tag with value != PVID
"Q-in-Q" - enable double tag, add second tag to already tagged
packets.
"LAN" and "WAN" - flags, for switches which have special function for
LAN-WAN processing.
"Tagged" and "Untagged" - mark port Tagged/Untagged, used if switch
using Global Tag flag (one flag for all VLANs)
"pvid" - set Port VLAN ID
Example:
switchctl /dev/switch0 set port 2 pvid 2 flags IngressCheck LAN Tagged

get/set vlan V:
add N (tag|untag|forbid) - add port N to VLAN member ports as (tag|
untag| forbid)
del N - delete port N from VLAN member ports
vid N - assign VLAN ID to internal index V
Example:
switchctl /dev/switch0 set vlan 2 vid 12
switchctl /dev/switch0 set vlan 2 add 2 u
switchctl /dev/switch0 set vlan 2 del 1

get/set reg:
Generic access to registers. Have 2 address modifiers:
0x00000000(no modifiers) - access to parent space (parent MDIO bus, if
switchX attaches to miibusX)
0x40000000 - access to switch MDIO bus
0x80000000 - access to switch registers.
Example:
# switchctl /dev/switch0 get reg 0x80000008
Reg 0x80000008 Value = 0x012603e2
# switchctl /dev/switch0 get reg 0x00000008
Reg 0x00000008 Value = 0x0000ffff
# switchctl /dev/switch0 get reg 0x40000001
Reg 0x40000001 Value = 0x00007949
# switchctl /dev/switch0 set reg 0x80000008 0x012603e2
Reg 0x80000008 Value = 0x012603e2 (Old value = 0x012603e2)

FloatPHY
pseudo driver which attach to miibus like normal PHY, but do find
master switchX device and ask his PHY reg's. Main problem with that
driver - is usage of newbus calls between independent device (not a
parent <-> child), since floatphyX query set/get methods of switchX.

Use hints:
"master" - to set master name.
"master_unit" - master unit.
"master_phys" - bitmap of PHY numbers on which get link status/speed.
"flags" - see driver (dev/switch/floatphy.c).
"speed" - initial link speed value, used when no access to master.

I will describe as example 4 situation that current framework is cover:
1. Ralink RT305X SoC, internal switch, one NIC with two paths.
2. Atheros AR7240 SoC, two NIC, but MDIO routed only from second
3. Cavium Octeon CN5010, one NIC with three paths + BCM53115 switch +
some Broadcom PHY

1. Ralink RT305X - is simple one
Attached by obio0 if driver present in kernel.

VLAN features: it have so called global untagged flag, so port can be
member of any VLAN but may be tagged or untagged in all VLAN in same
time.

How to enable: just add following into kernel config file.
------------------------------------------------
device switch
device switch_rt305x
------------------------------------------------

2. Atheros AR7240 - very interesting. have AR8216 internal switch and
two arge NICs.

arge0 MII bus connected to PHY4 which configured as separate PHY(not
attached to switch core). But PHY4 reg's can be accessible only via
switch MDIO bus access.
arge1 MII bus connected to switch MII.
But MDIO bus wired only on arge0, so if arge0 want to know speed and
link status, it must ask switch connected to miibus attached to arge1.
[1] Page 3

VLAN features: Like RT395x use global untagged like flag.

How to enable (example config for AR7240 in patch)

Config:
------------------------------------------------
device mii
device switch
device switch_ar8x16
------------------------------------------------

hints:
------------------------------------------------
# No probe at all
# First MDIO connected to switch which not have real PHY regs
hint.miibus.0.phymask="0x00000000"
# Second MDIO not wired at all
hint.miibus.1.phymask="0x00000000"

# Connect pseudo PHY driver to miibus0
hint.floatphy.0.at="miibus0"
hint.floatphy.0.phyno=0
# floatphy0 will ask switch0
hint.floatphy.0.master="switch"
hint.floatphy.0.master_unit=0
# and get link status from PHYs masked by 0x00000010
hint.floatphy.0.master_phys=0x00000010 # Sense PHY4
hint.floatphy.0.flags=0x00000000
# Default link speed 100 (if no access to master)
hint.floatphy.0.speed=100

# Switch attached to MDIO bus on arge0
hint.switch.0.at="miibus0"
hint.switch.0.phyno=1
# AR8x16 Magic register which can configure PHY4 as a standalone PHY
hint.ar8x16_switch.0.mii_mode=0x012603e2

hint.floatphy.1.at="miibus1"
hint.floatphy.1.phyno=0
hint.floatphy.1.master="switch"
hint.floatphy.1.master_unit=0
# check link on PHY0-PHY3 (link on any rise link on arge1)
hint.floatphy.1.master_phys=0x0000000f # Link Sensing PHY0-PHY3
hint.floatphy.1.flags=0x00000004
# "Link on any PHYs" | "Static link speed"
hint.floatphy.1.speed=1000
------------------------------------------------

3. Cavium Octeon CN5010, one NIC with three paths + BCM53115 switch +
some Broadcom PHY.
Since here is required a lot of magic to attach anything that Cavium
was not expect as "can be attached" (i.e. Cavium SDK limitation), I
made patch which allow to attach one PHY driver per NIC path (per
octe0 iface).

VLAN features: it seems have most clear VLAN implementation, except
some things like remapped some regs which have port bit maps. (seems
forget to think about bigger port count when design small switches)

How to enable:
Config:
------------------------------------------------
device brgphy
device switch
device switch_bcm5325
------------------------------------------------

Hints:
------------------------------------------------
hint.miibus.0.phymask="0x00000000"
hint.miibus.1.phymask="0x00000000"
hint.miibus.3.phymask="0x00000100"
# brgphy will attach here

hint.floatphy.0.at="miibus0"
hint.floatphy.0.phyno=0
hint.floatphy.0.master="switch"
hint.floatphy.0.master_unit=0
# Check link on any ports
hint.floatphy.0.master_phys=0x0000001f # Sense PHY0
hint.floatphy.0.flags=0x00000000
hint.floatphy.0.speed=1000

hint.switch.0.at="miibus1"
hint.switch.0.phyno=30
------------------------------------------------

Huh, hope that is enough for first release of manual :)

Will wait for any comments and suggestions!

We're committing this in 5 days unless anyone objects in any meaningful
way.



1. http://my.ddteam.net/files/Switch_Framework.pdf

WBW

[Stefan Bethke]

Thank you for the update, that clears up a number of questions I had.

Here's a couple of questions and comments.

Which devices can this code be run on? For example, can the AR7240 config run on an TL-WR3420?

Would you mind giving an overview of how the various parts fit together, how they are probed and attached? I think I understand from you explanations on IRC, but not everbody had that chance.

Post by Aleksandr Rybalko
0x00000000(no modifiers) - access to parent space (parent MDIO bus, if
switchX attaches to miibusX)
0x40000000 - access to switch MDIO bus
0x80000000 - access to switch registers.

Wouldn't it be better to have a proper API to select the various busses and device register files?

Post by Aleksandr Rybalko
FloatPHY
pseudo driver which attach to miibus like normal PHY, but do find
master switchX device and ask his PHY reg's. Main problem with that
driver - is usage of newbus calls between independent device (not a
parent <-> child), since floatphyX query set/get methods of switchX.

The general approach has a number of problems, as far as I understand the miibus code. miibus assumes that only one of the PHYs on the MII is active concurrently (since that's a requirement of the MII/GMII/... bus). If I read your code correctly, you have one miibus that has all the PHYs for all switch ports on them. Won't miibus just isolate all but one PHY?

In your current implementation, you've reimplented ukphy (in a limited fashion). One of the advantages of reusing the mii framework is being able to use all the PHYs, in case a switch chip would include a quirky PHY. Extending floatphy to work as a full proxy for any phy driver looks non-trivial to me due to the API constraints the mii framework imposes.


Stefan

[Stefan Bethke]

Post by Aleksandr Rybalko
It include sys/mips/conf/AR7240, that together with hints file is good
example for typical AR7240 setup.

IÄm heaving trouble getting this to work. The patch applies cleanly and I can get a kernel compiled and booted, but neither arge0 nor arge1 appear to be functional. I had to roll my own kernel config as your AR7240 hangs before printing anything on my TL-MR3420.

dmesg and devinfo below.


Stefan

CPU platform: Atheros AR7241 rev 1
CPU Frequency=400 MHz
CPU DDR Frequency=400 MHz
CPU AHB Frequency=200 MHz
platform frequency: 400000000
arguments:
a0 = 00000008
a1 = a1f87fb0
a2 = a1f88470
a3 = 00000004
Cmd line:argv is invalid
Environment:
envp is invalid
Cache info:
picache_stride = 4096
picache_loopcount = 16
pdcache_stride = 4096
pdcache_loopcount = 8
cpu0: MIPS Technologies processor v116.147
MMU: Standard TLB, 16 entries
L1 i-cache: 4 ways of 512 sets, 32 bytes per line
L1 d-cache: 4 ways of 256 sets, 32 bytes per line
Config1=0x9ee3519e<PerfCount,WatchRegs,MIPS16,EJTAG>
Config3=0x20
KDB: debugger backends: ddb
KDB: current backend: ddb
Copyright (c) 1992-2012 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 10.0-CURRENT #1: Thu Jan 1 01:00:00 CET 1970
***@dummy:/home/stb/working/fe/obj/mipseb/mips.mipseb/home/stb/working/fe/freebsd/sys/TL-MR3420D mips
WARNING: WITNESS option enabled, expect reduced performance.
real memory = 33554432 (32768K bytes)
avail memory = 25567232 (24MB)
random device not loaded; using insecure entropy
nexus0: <MIPS32 root nexus>
nexus0: failed to add child: arge0
nexus0: failed to add child: arge1
clock0: <Generic MIPS32 ticker> on nexus0
Timecounter "MIPS32" frequency 200000000 Hz quality 800
Event timer "MIPS32" frequency 200000000 Hz quality 800
apb0 at irq 4 on nexus0
uart0: <16550 or compatible> on apb0
uart0: console (115200,n,8,1)
gpio0: <Atheros AR71XX GPIO driver> on apb0
gpio0: [GIANT-LOCKED]
gpio0: function_set: 0x0
gpio0: function_clear: 0x0
gpio0: gpio pinmask=0x1943
gpioc0: <GPIO controller> on gpio0
gpiobus0: <GPIO bus> on gpio0
gpioled0: <GPIO led> at pin(s) 0 on gpiobus0
gpioled1: <GPIO led> at pin(s) 1 on gpiobus0
gpioled2: <GPIO led> at pin(s) 3 on gpiobus0
ehci0: <AR71XX Integrated USB 2.0 controller> at mem 0x1b000100-0x1bffffff irq 1 on nexus0
usbus0: set host controller mode
usbus0: EHCI version 1.0
usbus0: set host controller mode
usbus0: <AR71XX Integrated USB 2.0 controller> on ehci0
arge0: <Atheros AR71xx built-in ethernet interface> at mem 0x19000000-0x19000fff irq 2 on nexus0
arge0: Overriding MAC from EEPROM
arge0: No PHY specified, using mask 16
miibus0: <MII bus> on arge0
floatphy0 PHY 0 on miibus0
floatphy0: none, 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseSX, 1000baseSX-FDX, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, auto
switch0 PHY 1 on miibus0
switch0: 100baseTX, 100baseTX-FDX, 1000baseSX, 1000baseSX-FDX, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master
ar8x16_switch0: <Unknown AR8x16 like Ethernet switch (00.00)> on switch0
arge0: Ethernet address: ff:ff:ff:ff:ff:ff
arge1: <Atheros AR71xx built-in ethernet interface> at mem 0x1a000000-0x1a000fff irq 3 on nexus0
arge1: No PHY specified, using mask 15
arge1: Ethernet address: ff:ff:ff:ff:ff:00
spi0: <AR71XX SPI> at mem 0x1f000000-0x1f00000f on nexus0
spibus0: <spibus bus> on spi0
mx25l0: <M25Pxx Flash Family> at cs 0 on spibus0
mx25l0: s25s1032, sector 65536 bytes, 64 sectors
ar71xx_wdog0: <Atheros AR71XX watchdog timer> on nexus0
ar71xx_wdog0: Previous reset was due to watchdog timeout
Timecounters tick every 1.000 msec
usbus0: 480Mbps High Speed USB v2.0
ugen0.1: <Atheros> at usbus0
uhub0: <Atheros EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus0
WARNING: WITNESS option enabled, expect reduced performance.
uhub0: 1 port with 1 removable, self powered
Root mount waiting for: usbus0
Root mount waiting for: usbus0
Root mount waiting for: usbus0
ugen0.2: <Kingston> at usbus0
umass0: <Kingston DataTraveler G3, class 0/0, rev 2.00/1.00, addr 2> on usbus0
umass0: SCSI over Bulk-Only; quirks = 0x0100
umass0:0:0:-1: Attached to scbus0
Trying to mount root from ufs:map/rootfs.uzip []...
mountroot: waiting for device map/rootfs.uzip ...
da0 at umass-sim0 bus 0 scbus0 target 0 lun 0
da0: <Kingston DataTraveler G3 PMAP> Removable Direct Access SCSI-0 device
da0: 40.000MB/s transfers
da0: 15252MB (31236096 512 byte sectors: 255H 63S/T 1944C)
Trying to mount root from ufs:da0s1a []...
warning: no time-of-day clock registered, system time will not be set accurately
Setting hostuuid: aed4c502-193a-11e1-b662-74ea3ae4d920.
Setting hostid: 0x6a714343.
Entropy harvesting: interrupts ethernet point_to_point kickstart.
Starting file system checks:
/dev/da0s1a: FILE SYSTEM CLEAN; SKIPPING CHECKS
/dev/da0s1a: clean, 3849245 free (1085 frags, 481020 blocks, 0.0% fragmentation)
lock order reversal:
1st 0x8095691c ufs (ufs) @ /home/stb/working/fe/freebsd/sys/kern/vfs_mount.c:1245
2nd 0x80956e94 devfs (devfs) @ /home/stb/working/fe/freebsd/sys/kern/vfs_subr.c:2167
KDB: stack backtrace:
db_trace_thread+30 (?,?,?,?) ra c3f597c000000018 sp 0 sz 0
db_trace_self+1c (?,?,?,?) ra c3f597d800000018 sp 0 sz 0
800776d8+34 (?,?,?,?) ra c3f597f0000001a0 sp 0 sz 0
kdb_backtrace+44 (?,?,?,?) ra c3f5999000000018 sp 0 sz 0
801b3bbc+34 (?,?,?,?) ra c3f599a800000020 sp 0 sz 0
witness_checkorder+9cc (?,?,803b1f7c,877) ra c3f599c800000050 sp 0 sz 1
__lockmgr_args+948 (?,?,80956eb4,?) ra c3f59a1800000070 sp 0 sz 1
vop_stdlock+4c (?,?,?,?) ra c3f59a8800000028 sp 0 sz 0
VOP_LOCK1_APV+e4 (?,?,?,?) ra c3f59ab000000020 sp 0 sz 0
_vn_lock+84 (?,?,?,?) ra c3f59ad000000048 sp 0 sz 0
vget+c8 (?,?,?,?) ra c3f59b1800000030 sp 0 sz 0
devfs_allocv+100 (?,?,?,?) ra c3f59b4800000038 sp 0 sz 0
800ba9b4+4c (?,?,?,?) ra c3f59b8000000028 sp 0 sz 0
vflush+6c (?,?,0,80991300) ra c3f59ba8000000f8 sp 0 sz 1
800baa74+54 (?,?,?,?) ra c3f59ca000000028 sp 0 sz 0
dounmount+3f0 (809e8000,?,?,?) ra c3f59cc800000050 sp 100000000 sz 0
sys_unmount+39c (?,?,?,?) ra c3f59d18000000b0 sp 0 sz 0
trap+7f4 (?,?,?,?) ra c3f59dc8000000b8 sp 0 sz 0
MipsUserGenException+10c (?,?,?,404b53c0) ra c3f59e8000000000 sp 0 sz 0
pid 70
Mounting local file systems:.
Setting hostname: whitebox.lassitu.de.
miibus0: mii_mediachg: can't handle non-zero PHY instance 1
floatphy0: found master switch0
miibus0: mii_mediachg: can't handle non-zero PHY instance 1
arge0: link state changed to DOWN
Starting Network: lo0 arge0 arge1.
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
options=3<RXCSUM,TXCSUM>
inet6 ::1 prefixlen 128
inet6 fe80::1%lo0 prefixlen 64 scopeid 0x4
inet 127.0.0.1 netmask 0xff000000
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
arge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=80000<LINKSTATE>
ether ff:ff:ff:ff:ff:ff
inet 192.168.1.1 netmask 0xffffff00 broadcast 192.168.1.255
inet6 fe80::481f:1b46:dbf1:bd78%arge0 prefixlen 64 scopeid 0x2
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
media: Ethernet autoselect (100baseTX <full-duplex>)
status: no carrier
arge1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
ether ff:ff:ff:ff:ff:00
inet 44.128.65.33 netmask 0xffffffc0 broadcast 44.128.65.63
inet6 fe80::481f:1b46:dbf1:bd78%arge1 prefixlen 64 scopeid 0x3
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
media: Ethernet 100baseTX <full-duplex>
status: active
add net default: gateway 44.128.65.1
add net ::ffff:0.0.0.0: gateway ::1
add net ::0.0.0.0: gateway ::1
add net fe80::: gateway ::1
add net ff02::: gateway ::1
Mounting NFS file systems:mount_nfs: diesel: hostname nor servname provided, or not known
.
Creating and/or trimming log files.
No core dumps found.
ELF ldconfig path: /lib /usr/lib /usr/lib/compat
Setting date via ntp.
Error : hostname nor servname provided, or not known
22 Jan 15:01:04 ntpdate[566]: can't find host diesel

22 Jan 15:01:04 ntpdate[566]: no servers can be used, exiting
Clearing /tmp (X related).
Mounting late file systems:mount_nfs: diesel: hostname nor servname provided, or not known
.
Mounting /etc/fstab filesystems failed,Jan 22 15:01:12 init: /bin/sh on /etc/rc terminated abnormally, going to single user mode
Enter full pathname of shell or RETURN for /bin/sh:
# devinfo
nexus0
clock0
apb0
uart0
gpio0
gpioc0
gpiobus0
gpioled0
gpioled1
gpioled2
ehci0
usbus0
uhub0
umass0
arge0
miibus0
floatphy0
switch0
ar8x16_switch0
arge1
spi0
spibus0
mx25l0
ar71xx_wdog0
#

[Aleksandr Rybalko]

On Sun, 22 Jan 2012 16:31:06 +0100

Post by Stefan Bethke

Post by Aleksandr Rybalko
It include sys/mips/conf/AR7240, that together with hints file is
good example for typical AR7240 setup.

IÄm heaving trouble getting this to work. The patch applies cleanly
and I can get a kernel compiled and booted, but neither arge0 nor
arge1 appear to be functional. I had to roll my own kernel config as
your AR7240 hangs before printing anything on my TL-MR3420.

Yeah, I know where is problem, to proper attach switch framework to
arge, arge must be regular NIC. Here is the patch for that:
http://my.ddteam.net/files/2012-01-22_arge.patch
Hope it will apply cleanly.

Patch have fixed both arge problems (problem for allocation of ring
buffer, and stray interrupts) + remove most phymask bits + whitespace
cleanup.

Thank you for testing that Stefan.

P.S. I can't test clear SoC config on my board, because my board id
D-Link DIR-615_E4 with modified U-Boot in it, which able to load only
FW images, but not ELF kernel. So I test it with ZRouter.org FW image
instead.

P.P.S. can you also show me network part of your config and hints files.

P.P.P.S. still working on your previous question about subj, already
begin work on more wide documentation on wiki, but still far enough :)
"http://wiki.freebsd.org/AleksandrRybalko/Switch Framework"

Post by Stefan Bethke
dmesg and devinfo below.
Stefan
CPU platform: Atheros AR7241 rev 1
CPU Frequency=400 MHz
CPU DDR Frequency=400 MHz
CPU AHB Frequency=200 MHz
platform frequency: 400000000
a0 = 00000008
a1 = a1f87fb0
a2 = a1f88470
a3 = 00000004
Cmd line:argv is invalid
envp is invalid
picache_stride = 4096
picache_loopcount = 16
pdcache_stride = 4096
pdcache_loopcount = 8
cpu0: MIPS Technologies processor v116.147
MMU: Standard TLB, 16 entries
L1 i-cache: 4 ways of 512 sets, 32 bytes per line
L1 d-cache: 4 ways of 256 sets, 32 bytes per line
Config1=0x9ee3519e<PerfCount,WatchRegs,MIPS16,EJTAG>
Config3=0x20
KDB: debugger backends: ddb
KDB: current backend: ddb
Copyright (c) 1992-2012 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993,
1994 The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 10.0-CURRENT #1: Thu Jan 1 01:00:00 CET 1970
mips WARNING: WITNESS option enabled, expect reduced performance.
real memory = 33554432 (32768K bytes)
avail memory = 25567232 (24MB)
random device not loaded; using insecure entropy
nexus0: <MIPS32 root nexus>
nexus0: failed to add child: arge0
nexus0: failed to add child: arge1
clock0: <Generic MIPS32 ticker> on nexus0
Timecounter "MIPS32" frequency 200000000 Hz quality 800
Event timer "MIPS32" frequency 200000000 Hz quality 800
apb0 at irq 4 on nexus0
uart0: <16550 or compatible> on apb0
uart0: console (115200,n,8,1)
gpio0: <Atheros AR71XX GPIO driver> on apb0
gpio0: [GIANT-LOCKED]
gpio0: function_set: 0x0
gpio0: function_clear: 0x0
gpio0: gpio pinmask=0x1943
gpioc0: <GPIO controller> on gpio0
gpiobus0: <GPIO bus> on gpio0
gpioled0: <GPIO led> at pin(s) 0 on gpiobus0
gpioled1: <GPIO led> at pin(s) 1 on gpiobus0
gpioled2: <GPIO led> at pin(s) 3 on gpiobus0
ehci0: <AR71XX Integrated USB 2.0 controller> at mem
0x1b000100-0x1bffffff irq 1 on nexus0 usbus0: set host controller mode
usbus0: EHCI version 1.0
usbus0: set host controller mode
usbus0: <AR71XX Integrated USB 2.0 controller> on ehci0
arge0: <Atheros AR71xx built-in ethernet interface> at mem
0x19000000-0x19000fff irq 2 on nexus0 arge0: Overriding MAC from
EEPROM arge0: No PHY specified, using mask 16
miibus0: <MII bus> on arge0
floatphy0 PHY 0 on miibus0
floatphy0: none, 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX,
1000baseSX, 1000baseSX-FDX, 1000baseT, 1000baseT-master,
1000baseT-FDX, 1000baseT-FDX-master, auto switch0 PHY 1 on miibus0
switch0: 100baseTX, 100baseTX-FDX, 1000baseSX, 1000baseSX-FDX,
1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master
ar8x16_switch0: <Unknown AR8x16 like Ethernet switch (00.00)> on
switch0 arge0: Ethernet address: ff:ff:ff:ff:ff:ff arge1: <Atheros
AR71xx built-in ethernet interface> at mem 0x1a000000-0x1a000fff irq
3 on nexus0 arge1: No PHY specified, using mask 15 arge1: Ethernet
address: ff:ff:ff:ff:ff:00 spi0: <AR71XX SPI> at mem
<M25Pxx Flash Family> at cs 0 on spibus0 mx25l0: s25s1032, sector
65536 bytes, 64 sectors ar71xx_wdog0: <Atheros AR71XX watchdog timer>
on nexus0 ar71xx_wdog0: Previous reset was due to watchdog timeout
Timecounters tick every 1.000 msec
usbus0: 480Mbps High Speed USB v2.0
ugen0.1: <Atheros> at usbus0
uhub0: <Atheros EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on
usbus0 WARNING: WITNESS option enabled, expect reduced performance.
uhub0: 1 port with 1 removable, self powered
Root mount waiting for: usbus0
Root mount waiting for: usbus0
Root mount waiting for: usbus0
ugen0.2: <Kingston> at usbus0
umass0: <Kingston DataTraveler G3, class 0/0, rev 2.00/1.00, addr 2>
on usbus0 umass0: SCSI over Bulk-Only; quirks = 0x0100
umass0:0:0:-1: Attached to scbus0
Trying to mount root from ufs:map/rootfs.uzip []...
mountroot: waiting for device map/rootfs.uzip ...
da0 at umass-sim0 bus 0 scbus0 target 0 lun 0
da0: <Kingston DataTraveler G3 PMAP> Removable Direct Access SCSI-0
device da0: 40.000MB/s transfers
da0: 15252MB (31236096 512 byte sectors: 255H 63S/T 1944C)
Trying to mount root from ufs:da0s1a []...
warning: no time-of-day clock registered, system time will not be set
accurately Setting hostuuid: aed4c502-193a-11e1-b662-74ea3ae4d920.
Setting hostid: 0x6a714343.
Entropy harvesting: interrupts ethernet point_to_point kickstart.
/dev/da0s1a: FILE SYSTEM CLEAN; SKIPPING CHECKS
/dev/da0s1a: clean, 3849245 free (1085 frags, 481020 blocks, 0.0%
1st 0x8095691c ufs (ufs)
@ /home/stb/working/fe/freebsd/sys/kern/vfs_mount.c:1245 2nd
0x80956e94 devfs (devfs)
@ /home/stb/working/fe/freebsd/sys/kern/vfs_subr.c:2167 KDB: stack
backtrace: db_trace_thread+30 (?,?,?,?) ra c3f597c000000018 sp 0 sz 0
db_trace_self+1c (?,?,?,?) ra c3f597d800000018 sp 0 sz 0 800776d8+34
(?,?,?,?) ra c3f597f0000001a0 sp 0 sz 0 kdb_backtrace+44 (?,?,?,?) ra
c3f5999000000018 sp 0 sz 0 801b3bbc+34 (?,?,?,?) ra c3f599a800000020
sp 0 sz 0 witness_checkorder+9cc (?,?,803b1f7c,877) ra
c3f599c800000050 sp 0 sz 1 __lockmgr_args+948 (?,?,80956eb4,?) ra
c3f59a1800000070 sp 0 sz 1 vop_stdlock+4c (?,?,?,?) ra
c3f59a8800000028 sp 0 sz 0 VOP_LOCK1_APV+e4 (?,?,?,?) ra
c3f59ab000000020 sp 0 sz 0 _vn_lock+84 (?,?,?,?) ra c3f59ad000000048
sp 0 sz 0 vget+c8 (?,?,?,?) ra c3f59b1800000030 sp 0 sz 0
devfs_allocv+100 (?,?,?,?) ra c3f59b4800000038 sp 0 sz 0
800ba9b4+4c (?,?,?,?) ra c3f59b8000000028 sp 0 sz 0
vflush+6c (?,?,0,80991300) ra c3f59ba8000000f8 sp 0 sz 1
800baa74+54 (?,?,?,?) ra c3f59ca000000028 sp 0 sz 0
dounmount+3f0 (809e8000,?,?,?) ra c3f59cc800000050 sp 100000000 sz 0
sys_unmount+39c (?,?,?,?) ra c3f59d18000000b0 sp 0 sz 0
trap+7f4 (?,?,?,?) ra c3f59dc8000000b8 sp 0 sz 0
MipsUserGenException+10c (?,?,?,404b53c0) ra c3f59e8000000000 sp 0 sz
0 pid 70
Mounting local file systems:.
Setting hostname: whitebox.lassitu.de.
miibus0: mii_mediachg: can't handle non-zero PHY instance 1
floatphy0: found master switch0
miibus0: mii_mediachg: can't handle non-zero PHY instance 1
arge0: link state changed to DOWN
Starting Network: lo0 arge0 arge1.
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
options=3<RXCSUM,TXCSUM>
inet6 ::1 prefixlen 128
inet6 fe80::1%lo0 prefixlen 64 scopeid 0x4
inet 127.0.0.1 netmask 0xff000000
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
arge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0
mtu 1500 options=80000<LINKSTATE>
ether ff:ff:ff:ff:ff:ff
inet 192.168.1.1 netmask 0xffffff00 broadcast 192.168.1.255
inet6 fe80::481f:1b46:dbf1:bd78%arge0 prefixlen 64 scopeid
0x2 nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
media: Ethernet autoselect (100baseTX <full-duplex>)
status: no carrier
arge1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0
mtu 1500 ether ff:ff:ff:ff:ff:00
inet 44.128.65.33 netmask 0xffffffc0 broadcast 44.128.65.63
inet6 fe80::481f:1b46:dbf1:bd78%arge1 prefixlen 64 scopeid
0x3 nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
media: Ethernet 100baseTX <full-duplex>
status: active
add net default: gateway 44.128.65.1
add net ::ffff:0.0.0.0: gateway ::1
add net ::0.0.0.0: gateway ::1
add net fe80::: gateway ::1
add net ff02::: gateway ::1
Mounting NFS file systems:mount_nfs: diesel: hostname nor servname
provided, or not known .
Creating and/or trimming log files.
No core dumps found.
ELF ldconfig path: /lib /usr/lib /usr/lib/compat
Setting date via ntp.
Error : hostname nor servname provided, or not known
22 Jan 15:01:04 ntpdate[566]: can't find host diesel
22 Jan 15:01:04 ntpdate[566]: no servers can be used, exiting
Clearing /tmp (X related).
Mounting late file systems:mount_nfs: diesel: hostname nor servname
provided, or not known .
Mounting /etc/fstab filesystems failed,Jan 22 15:01:12 init: /bin/sh
on /etc/rc terminated abnormally, going to single user mode Enter
# devinfo
nexus0
clock0
apb0
uart0
gpio0
gpioc0
gpiobus0
gpioled0
gpioled1
gpioled2
ehci0
usbus0
uhub0
umass0
arge0
miibus0
floatphy0
switch0
ar8x16_switch0
arge1
spi0
spibus0
mx25l0
ar71xx_wdog0
#
--

[Adrian Chadd]

So when will you two have something consensus-y to commit? :-)

What I'm hoping for is:

* some traction on the MII bus / MDIO bus split and tidyup from stb, which
is nice;
* ray's switch API for speaking to userland with;
* agreeing on whether the correct place to put the driver(s) is where stb,
ray, or a mix of both approaches says so.

I've been (mostly) trying to stay out of this to see where both of you have
gone. I think we've made some good progress; now it's time to solidify a
design for the first pass of what we want in -HEAD and figure out how to
move forward.


Adrian

Post by Aleksandr Rybalko
On Sun, 22 Jan 2012 16:31:06 +0100

Post by Stefan Bethke

Post by Aleksandr Rybalko
It include sys/mips/conf/AR7240, that together with hints file is
good example for typical AR7240 setup.

IÄm heaving trouble getting this to work. The patch applies cleanly
and I can get a kernel compiled and booted, but neither arge0 nor
arge1 appear to be functional. I had to roll my own kernel config as
your AR7240 hangs before printing anything on my TL-MR3420.

Yeah, I know where is problem, to proper attach switch framework to
http://my.ddteam.net/files/2012-01-22_arge.patch
Hope it will apply cleanly.
Patch have fixed both arge problems (problem for allocation of ring
buffer, and stray interrupts) + remove most phymask bits + whitespace
cleanup.
Thank you for testing that Stefan.
P.S. I can't test clear SoC config on my board, because my board id
D-Link DIR-615_E4 with modified U-Boot in it, which able to load only
FW images, but not ELF kernel. So I test it with ZRouter.org FW image
instead.
P.P.S. can you also show me network part of your config and hints files.
P.P.P.S. still working on your previous question about subj, already
begin work on more wide documentation on wiki, but still far enough :)
"http://wiki.freebsd.org/AleksandrRybalko/Switch Framework"

Post by Stefan Bethke
dmesg and devinfo below.
Stefan
CPU platform: Atheros AR7241 rev 1
CPU Frequency=400 MHz
CPU DDR Frequency=400 MHz
CPU AHB Frequency=200 MHz
platform frequency: 400000000
a0 = 00000008
a1 = a1f87fb0
a2 = a1f88470
a3 = 00000004
Cmd line:argv is invalid
envp is invalid
picache_stride = 4096
picache_loopcount = 16
pdcache_stride = 4096
pdcache_loopcount = 8
cpu0: MIPS Technologies processor v116.147
MMU: Standard TLB, 16 entries
L1 i-cache: 4 ways of 512 sets, 32 bytes per line
L1 d-cache: 4 ways of 256 sets, 32 bytes per line
Config1=0x9ee3519e<PerfCount,WatchRegs,MIPS16,EJTAG>
Config3=0x20
KDB: debugger backends: ddb
KDB: current backend: ddb
Copyright (c) 1992-2012 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993,
1994 The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 10.0-CURRENT #1: Thu Jan 1 01:00:00 CET 1970

:/home/stb/working/fe/obj/mipseb/mips.mipseb/home/stb/working/fe/freebsd/sys/TL-MR3420D

Post by Stefan Bethke
mips WARNING: WITNESS option enabled, expect reduced performance.
real memory = 33554432 (32768K bytes)
avail memory = 25567232 (24MB)
random device not loaded; using insecure entropy
nexus0: <MIPS32 root nexus>
nexus0: failed to add child: arge0
nexus0: failed to add child: arge1
clock0: <Generic MIPS32 ticker> on nexus0
Timecounter "MIPS32" frequency 200000000 Hz quality 800
Event timer "MIPS32" frequency 200000000 Hz quality 800
apb0 at irq 4 on nexus0
uart0: <16550 or compatible> on apb0
uart0: console (115200,n,8,1)
gpio0: <Atheros AR71XX GPIO driver> on apb0
gpio0: [GIANT-LOCKED]
gpio0: function_set: 0x0
gpio0: function_clear: 0x0
gpio0: gpio pinmask=0x1943
gpioc0: <GPIO controller> on gpio0
gpiobus0: <GPIO bus> on gpio0
gpioled0: <GPIO led> at pin(s) 0 on gpiobus0
gpioled1: <GPIO led> at pin(s) 1 on gpiobus0
gpioled2: <GPIO led> at pin(s) 3 on gpiobus0
ehci0: <AR71XX Integrated USB 2.0 controller> at mem
0x1b000100-0x1bffffff irq 1 on nexus0 usbus0: set host controller mode
usbus0: EHCI version 1.0
usbus0: set host controller mode
usbus0: <AR71XX Integrated USB 2.0 controller> on ehci0
arge0: <Atheros AR71xx built-in ethernet interface> at mem
0x19000000-0x19000fff irq 2 on nexus0 arge0: Overriding MAC from
EEPROM arge0: No PHY specified, using mask 16
miibus0: <MII bus> on arge0
floatphy0 PHY 0 on miibus0
floatphy0: none, 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX,
1000baseSX, 1000baseSX-FDX, 1000baseT, 1000baseT-master,
1000baseT-FDX, 1000baseT-FDX-master, auto switch0 PHY 1 on miibus0
switch0: 100baseTX, 100baseTX-FDX, 1000baseSX, 1000baseSX-FDX,
1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master
ar8x16_switch0: <Unknown AR8x16 like Ethernet switch (00.00)> on
switch0 arge0: Ethernet address: ff:ff:ff:ff:ff:ff arge1: <Atheros
AR71xx built-in ethernet interface> at mem 0x1a000000-0x1a000fff irq
3 on nexus0 arge1: No PHY specified, using mask 15 arge1: Ethernet
address: ff:ff:ff:ff:ff:00 spi0: <AR71XX SPI> at mem
<M25Pxx Flash Family> at cs 0 on spibus0 mx25l0: s25s1032, sector
65536 bytes, 64 sectors ar71xx_wdog0: <Atheros AR71XX watchdog timer>
on nexus0 ar71xx_wdog0: Previous reset was due to watchdog timeout
Timecounters tick every 1.000 msec
usbus0: 480Mbps High Speed USB v2.0
ugen0.1: <Atheros> at usbus0
uhub0: <Atheros EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on
usbus0 WARNING: WITNESS option enabled, expect reduced performance.
uhub0: 1 port with 1 removable, self powered
Root mount waiting for: usbus0
Root mount waiting for: usbus0
Root mount waiting for: usbus0
ugen0.2: <Kingston> at usbus0
umass0: <Kingston DataTraveler G3, class 0/0, rev 2.00/1.00, addr 2>
on usbus0 umass0: SCSI over Bulk-Only; quirks = 0x0100
umass0:0:0:-1: Attached to scbus0
Trying to mount root from ufs:map/rootfs.uzip []...
mountroot: waiting for device map/rootfs.uzip ...
da0 at umass-sim0 bus 0 scbus0 target 0 lun 0
da0: <Kingston DataTraveler G3 PMAP> Removable Direct Access SCSI-0
device da0: 40.000MB/s transfers
da0: 15252MB (31236096 512 byte sectors: 255H 63S/T 1944C)
Trying to mount root from ufs:da0s1a []...
warning: no time-of-day clock registered, system time will not be set
accurately Setting hostuuid: aed4c502-193a-11e1-b662-74ea3ae4d920.
Setting hostid: 0x6a714343.
Entropy harvesting: interrupts ethernet point_to_point kickstart.
/dev/da0s1a: FILE SYSTEM CLEAN; SKIPPING CHECKS
/dev/da0s1a: clean, 3849245 free (1085 frags, 481020 blocks, 0.0%
1st 0x8095691c ufs (ufs)
@ /home/stb/working/fe/freebsd/sys/kern/vfs_mount.c:1245 2nd
0x80956e94 devfs (devfs)
@ /home/stb/working/fe/freebsd/sys/kern/vfs_subr.c:2167 KDB: stack
backtrace: db_trace_thread+30 (?,?,?,?) ra c3f597c000000018 sp 0 sz 0
db_trace_self+1c (?,?,?,?) ra c3f597d800000018 sp 0 sz 0 800776d8+34
(?,?,?,?) ra c3f597f0000001a0 sp 0 sz 0 kdb_backtrace+44 (?,?,?,?) ra
c3f5999000000018 sp 0 sz 0 801b3bbc+34 (?,?,?,?) ra c3f599a800000020
sp 0 sz 0 witness_checkorder+9cc (?,?,803b1f7c,877) ra
c3f599c800000050 sp 0 sz 1 __lockmgr_args+948 (?,?,80956eb4,?) ra
c3f59a1800000070 sp 0 sz 1 vop_stdlock+4c (?,?,?,?) ra
c3f59a8800000028 sp 0 sz 0 VOP_LOCK1_APV+e4 (?,?,?,?) ra
c3f59ab000000020 sp 0 sz 0 _vn_lock+84 (?,?,?,?) ra c3f59ad000000048
sp 0 sz 0 vget+c8 (?,?,?,?) ra c3f59b1800000030 sp 0 sz 0
devfs_allocv+100 (?,?,?,?) ra c3f59b4800000038 sp 0 sz 0
800ba9b4+4c (?,?,?,?) ra c3f59b8000000028 sp 0 sz 0
vflush+6c (?,?,0,80991300) ra c3f59ba8000000f8 sp 0 sz 1
800baa74+54 (?,?,?,?) ra c3f59ca000000028 sp 0 sz 0
dounmount+3f0 (809e8000,?,?,?) ra c3f59cc800000050 sp 100000000 sz 0
sys_unmount+39c (?,?,?,?) ra c3f59d18000000b0 sp 0 sz 0
trap+7f4 (?,?,?,?) ra c3f59dc8000000b8 sp 0 sz 0
MipsUserGenException+10c (?,?,?,404b53c0) ra c3f59e8000000000 sp 0 sz
0 pid 70
Mounting local file systems:.
Setting hostname: whitebox.lassitu.de.
miibus0: mii_mediachg: can't handle non-zero PHY instance 1
floatphy0: found master switch0
miibus0: mii_mediachg: can't handle non-zero PHY instance 1
arge0: link state changed to DOWN
Starting Network: lo0 arge0 arge1.
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
options=3<RXCSUM,TXCSUM>
inet6 ::1 prefixlen 128
inet6 fe80::1%lo0 prefixlen 64 scopeid 0x4
inet 127.0.0.1 netmask 0xff000000
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
arge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0
mtu 1500 options=80000<LINKSTATE>
ether ff:ff:ff:ff:ff:ff
inet 192.168.1.1 netmask 0xffffff00 broadcast 192.168.1.255
inet6 fe80::481f:1b46:dbf1:bd78%arge0 prefixlen 64 scopeid
0x2 nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
media: Ethernet autoselect (100baseTX <full-duplex>)
status: no carrier
arge1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0
mtu 1500 ether ff:ff:ff:ff:ff:00
inet 44.128.65.33 netmask 0xffffffc0 broadcast 44.128.65.63
inet6 fe80::481f:1b46:dbf1:bd78%arge1 prefixlen 64 scopeid
0x3 nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
media: Ethernet 100baseTX <full-duplex>
status: active
add net default: gateway 44.128.65.1
add net ::ffff:0.0.0.0: gateway ::1
add net ::0.0.0.0: gateway ::1
add net fe80::: gateway ::1
add net ff02::: gateway ::1
Mounting NFS file systems:mount_nfs: diesel: hostname nor servname
provided, or not known .
Creating and/or trimming log files.
No core dumps found.
ELF ldconfig path: /lib /usr/lib /usr/lib/compat
Setting date via ntp.
Error : hostname nor servname provided, or not known
22 Jan 15:01:04 ntpdate[566]: can't find host diesel
22 Jan 15:01:04 ntpdate[566]: no servers can be used, exiting
Clearing /tmp (X related).
Mounting late file systems:mount_nfs: diesel: hostname nor servname
provided, or not known .
Mounting /etc/fstab filesystems failed,Jan 22 15:01:12 init: /bin/sh
on /etc/rc terminated abnormally, going to single user mode Enter
# devinfo
nexus0
clock0
apb0
uart0
gpio0
gpioc0
gpiobus0
gpioled0
gpioled1
gpioled2
ehci0
usbus0
uhub0
umass0
arge0
miibus0
floatphy0
switch0
ar8x16_switch0
arge1
spi0
spibus0
mx25l0
ar71xx_wdog0
#
--

--
_______________________________________________
http://lists.freebsd.org/mailman/listinfo/freebsd-arch

[Stefan Bethke]

Post by Adrian Chadd
So when will you two have something consensus-y to commit? :-)
* some traction on the MII bus / MDIO bus split and tidyup from stb, which is nice;
* ray's switch API for speaking to userland with;
* agreeing on whether the correct place to put the driver(s) is where stb, ray, or a mix of both approaches says so.
I've been (mostly) trying to stay out of this to see where both of you have gone. I think we've made some good progress; now it's time to solidify a design for the first pass of what we want in -HEAD and figure out how to move forward.

My suggestion is to take my bus attachment code (incl. mdio and miiproxy) and ray's ioctl and userland code.

Aleksandr's approach for the driver attachment is to have a generic switch "bus" driver that abstracts the mii, i2c, memory mapped I/O, etc. busses the devices are physically attached to, and present a uniform register file to the chip-specific switch driver. I believe that this is unnecessarily complicated for two reasons: newbus already provides that abstraction, and chip-specific drivers usually differ in so many aspects, including their register files, that code sharing between chips will be somewhat limited anyway.

One aspect that I would enjoy looking into in more detail is how register accesses on, for example, MDIO, can be provided through the bus space API. From my cursory reading, it seems that the code currently is tailored towards register accesses that can be implemented through CPU native instructions, instead of indirectly through a controller.

Aleksandr has defined a quite comprehensive ethernet switch control API that the framework provides towards in-kernel clients as well as userland. I think it would be really helpful if we could concentrate on those API functions that can be controlled through the userland utility, have immediate use cases (for example, VLAN configuration on the TL-WR1043ND to separate the WAN from the LAN ports), and we have test hardware for. In short, don't commit dead code.

Having a description of the generic switch model that the API assumes and driver-specific documentation also wouldn't hurt. (Yes, I'm volunteering.)


Stefan

[Adrian Chadd]

Ok, I do like the idea of:

* mdiobus/miibus proxy tidyup;
* then the switch API;
* then the switch devices themselves.

Can we get some consensus/agreement from Marius (and others) about the
first step?


Adrian

[Aleksandr Rybalko]

On Wed, 25 Jan 2012 09:57:32 +0100

Post by Stefan Bethke
My suggestion is to take my bus attachment code (incl. mdio and
miiproxy) and ray's ioctl and userland code.

As I see from your patch, mdio/miiproxy require special bits in MAC
driver. When I design switch framework, I keeping in mind that
MAC drivers should be standard as possible, that why I send you patch
http://my.ddteam.net/files/2012-01-22_arge.patch
which clean most phymask features from if_arge driver.
You may ask me why I do so? It is because arge H/W is not the single
implementation, it is just FPGA design that also used in many other
devices, f.e. if_et. Look into dev/et/if_etreg.h, begin from line:
#define ET_MAC_CFG1 0x5000
There is the same registers, same logic, just mapped at 0x5000 in
device PCI BAR, instead of 0x19000000(arge) and I bet it is not last
time when that FPGA design used :)

Post by Stefan Bethke
Aleksandr's approach for the driver attachment is to have a generic
switch "bus" driver that abstracts the mii, i2c, memory mapped I/O,
etc. busses the devices are physically attached to, and present a
uniform register file to the chip-specific switch driver. I believe
that this is unnecessarily complicated for two reasons: newbus
already provides that abstraction, and chip-specific drivers usually
differ in so many aspects, including their register files, that code
sharing between chips will be somewhat limited anyway.

newbus allow attach anything to anything, but bus interfaces
implemented in different ways (for mem/mdio we call read/write, for
SPI/IIC we call transfer). When we made that interfaces consistent we
be able really forget about "bus glue".
While we still not done it(even still not doing it), model with single
parent (switch0) require bus glue for each supported interface (MDIO,
MEM, SPI, etc.).
But model with direct attach driver to bus will require bus glue per
driver. If only one interface is supported, then glue in driver file,
else - separate file per each supported interface.

And two words about "complicated":

1. If we about complicated structure of devices - yes switch0 with
bcm5325_switch0 more complicated, than just bcm5325_switch0, but device
tree will care about it.

2. If we about code size, then I will say my model much smaller even
having more drivers.

My personal decision - is 2, because - less code better for maintenance.

Post by Stefan Bethke
One aspect that I would enjoy looking into in more detail is how
register accesses on, for example, MDIO, can be provided through the
bus space API. From my cursory reading, it seems that the code
currently is tailored towards register accesses that can be
implemented through CPU native instructions, instead of indirectly
through a controller.
Aleksandr has defined a quite comprehensive ethernet switch control
API that the framework provides towards in-kernel clients as well as
userland. I think it would be really helpful if we could concentrate
on those API functions that can be controlled through the userland
utility, have immediate use cases (for example, VLAN configuration on
the TL-WR1043ND to separate the WAN from the LAN ports), and we have
test hardware for. In short, don't commit dead code.

It is not dead code, it is TODO :)

Post by Stefan Bethke
Having a description of the generic switch model that the API assumes
and driver-specific documentation also wouldn't hurt. (Yes, I'm
volunteering.)

It is also TODO :)

On Thu, 26 Jan 2012 22:03:58 -0800

Post by Stefan Bethke
* mdiobus/miibus proxy tidyup;
* then the switch API;
* then the switch devices themselves.
Can we get some consensus/agreement from Marius (and others) about the
first step?

I think we don't need to "rewrite" miibus now. :)

Post by Stefan Bethke
Adrian

WBW

[Juli Mallett]

Post by Aleksandr Rybalko
As I see from your patch, mdio/miiproxy require special bits in MAC
driver. When I design switch framework, I keeping in mind that
MAC drivers should be standard as possible

I don't understand why this is desirable in practice. It's a nice
theory, but it falls down when one thinks in depth about how Ethernet
interfaces are used and administered vs. how switches are used and
administered. What should media report? What should media changes
do? What is link status? Do you show the CPU-to-switch port, or all
switch ports?

It makes me wonder if the understanding of the relationship in FreeBSD
isn't backwards. Yes, the MAC sits on a bus and is memory-mapped, but
you can conceptualize of it as a child of the PHY, rather than the
parent of it, especially in systems with switch chipsets. Especially
in systems where there is a switch chipset attached to multiple MACs.

In that model, it makes sense to semi-generically attach a
CPU-to-switch port's pseudo-PHY (or actual PHY, depending on hardware)
to a MAC generically, but that doesn't meant that the switch itself is
attached generically to the MAC.

There are a lot of switches out there that don't look or act much like
MII-driven PHYs, but which are connected over MDIO, as I've tried to
stress before. I hope there will be as much separation between the
MII work that is being done and the switch work that is being done as
possible. I think anything else will prove rapidly-obsolete and
perhaps even obstructive as soon as anyone seeks to add support for
more switch chipsets to FreeBSD.

I suppose the most likely reality, though, is that people simply won't
add switch support to FreeBSD, and will administer them out-of-band
from userland, using gross kernel shims. That is probably true
whether any of the currently-outstanding work is committed or not,
unfortunately :( I just hope we'll end up with something flexible
enough, broad enough in applicability, narrow enough in requirements,
etc., that we'll have feature-rich support for a few chipsets in tree
that work in sufficiently-different manners that they can be models
for other drivers in the future.

Thanks,
Juli.

[Warner Losh]

Post by Juli Mallett
It makes me wonder if the understanding of the relationship in FreeBSD
isn't backwards. Yes, the MAC sits on a bus and is memory-mapped, but
you can conceptualize of it as a child of the PHY, rather than the
parent of it, especially in systems with switch chipsets. Especially
in systems where there is a switch chipset attached to multiple MACs.
In that model, it makes sense to semi-generically attach a
CPU-to-switch port's pseudo-PHY (or actual PHY, depending on hardware)
to a MAC generically, but that doesn't meant that the switch itself is
attached generically to the MAC.

I think that the main issue here is that we have an assumption that we have a tree structure. However, it is more of a DAG across different domains. The hierarchy works well when each device owns all the devices below it and only interacted with them. Most devices are that way. However, in the embedded world, there's lots of reach-accrosses that are expected that break the model.

Plus, MDIO is more than what we call mii/phy, so there's an imperfect match there.

Warner

[Aleksandr Rybalko]

On Sat, 28 Jan 2012 15:00:01 -0800

Post by Juli Mallett

Post by Aleksandr Rybalko
As I see from your patch, mdio/miiproxy require special bits in MAC
driver. When I design switch framework, I keeping in mind that
MAC drivers should be standard as possible

I don't understand why this is desirable in practice. It's a nice
theory, but it falls down when one thinks in depth about how Ethernet
interfaces are used and administered vs. how switches are used and
administered. What should media report? What should media changes
do? What is link status? Do you show the CPU-to-switch port, or all
switch ports?

IMO, if we about why to make MAC driver more "standard", it is desired
because most MAC, used in SoC's, used also in NIC cards or other SoC's
which have no switch, but MII and MDIO pins to attach external PHY.

Real example bfe(4), it is supported long ago as PCI card, but also
found in BCM4701 family SoC's. In BCM5354 it attached to embedded
switch. In BCM5836 it is two independent MAC.
Last one used in D-Link DIR-330, there is one MAC attached to PHY,
second to BCM5325 switch). Since BCM4701 family have internal SSB bus,
a.k.a siba(4), I add only siba(4) bus glue. So if we able to attach
switch driver to regular MAC drivers it will make our life simpler :)

Post by Juli Mallett
It makes me wonder if the understanding of the relationship in FreeBSD
isn't backwards. Yes, the MAC sits on a bus and is memory-mapped, but
you can conceptualize of it as a child of the PHY, rather than the
parent of it, especially in systems with switch chipsets. Especially
in systems where there is a switch chipset attached to multiple MACs.
In that model, it makes sense to semi-generically attach a
CPU-to-switch port's pseudo-PHY (or actual PHY, depending on hardware)
to a MAC generically, but that doesn't meant that the switch itself is
attached generically to the MAC.
There are a lot of switches out there that don't look or act much like
MII-driven PHYs, but which are connected over MDIO, as I've tried to
stress before.

Yeah, for that case Marius commit phymask support for our miibus, to
not touch reserved PHY IDs.

Post by Juli Mallett
I hope there will be as much separation between the
MII work that is being done and the switch work that is being done as
possible. I think anything else will prove rapidly-obsolete and
perhaps even obstructive as soon as anyone seeks to add support for
more switch chipsets to FreeBSD.
I suppose the most likely reality, though, is that people simply won't
add switch support to FreeBSD, and will administer them out-of-band
from userland, using gross kernel shims. That is probably true
whether any of the currently-outstanding work is committed or not,
unfortunately :( I just hope we'll end up with something flexible
enough, broad enough in applicability, narrow enough in requirements,
etc., that we'll have feature-rich support for a few chipsets in tree
that work in sufficiently-different manners that they can be models
for other drivers in the future.
Thanks,
Juli.

Thank you for comments Juli!

WBW

[Stefan Bethke]

Post by Juli Mallett

Post by Aleksandr Rybalko
As I see from your patch, mdio/miiproxy require special bits in MAC
driver. When I design switch framework, I keeping in mind that
MAC drivers should be standard as possible

I don't understand why this is desirable in practice. It's a nice
theory, but it falls down when one thinks in depth about how Ethernet
interfaces are used and administered vs. how switches are used and
administered. What should media report? What should media changes
do? What is link status? Do you show the CPU-to-switch port, or all
switch ports?

The main thrust here is to reuse the existing PHY code to be able to configure the PHYs that are embedded in the switch chips. To confuse things, one of these PHYs might be connected to the SoCs ethernet interface via MII, GMII, etc. To confuse things further, these PHYs are controlled by an MDIO bus that has it's master in the switch chip, while the switch chip is a slave to the MDIO master in the ethernet controller.

The goal is to be able to configure the switch ports and set media on one of them, for example. That code path could be entirely independent from the ethernet infrastructure, if dev/miibus didn't require if_media and hence a struct ifnet. This discussion is also about how to deal with this entanglement.

The MII connection between the ethernet controller and the switch chip (usually referred to as the "CPU" port) is hard-coded and has no media settings, so there's no question what if_media settings should be presented on the interface.

Post by Juli Mallett
are a lot of switches out there that don't look or act much like
MII-driven PHYs, but which are connected over MDIO, as I've tried to
stress before. I hope there will be as much separation between the
MII work that is being done and the switch work that is being done as
possible. I think anything else will prove rapidly-obsolete and
perhaps even obstructive as soon as anyone seeks to add support for
more switch chipsets to FreeBSD.
I suppose the most likely reality, though, is that people simply won't
add switch support to FreeBSD, and will administer them out-of-band
from userland, using gross kernel shims. That is probably true
whether any of the currently-outstanding work is committed or not,
unfortunately :( I just hope we'll end up with something flexible
enough, broad enough in applicability, narrow enough in requirements,
etc., that we'll have feature-rich support for a few chipsets in tree
that work in sufficiently-different manners that they can be models
for other drivers in the future.

Which is a valid approach from a vendor's viewpoint. The reason we're talking is to try and make it easier to write a switch driver for this framework than roll your own hack. :-)


Stefan

[Aleksandr Rybalko]

On Sun, 29 Jan 2012 13:26:00 +0100

Post by Stefan Bethke

Post by Juli Mallett

Post by Aleksandr Rybalko
As I see from your patch, mdio/miiproxy require special bits in MAC
driver. When I design switch framework, I keeping in mind that
MAC drivers should be standard as possible

I don't understand why this is desirable in practice. It's a nice
theory, but it falls down when one thinks in depth about how
Ethernet interfaces are used and administered vs. how switches are
used and administered. What should media report? What should
media changes do? What is link status? Do you show the
CPU-to-switch port, or all switch ports?

The main thrust here is to reuse the existing PHY code to be able to
configure the PHYs that are embedded in the switch chips. To confuse
things, one of these PHYs might be connected to the SoCs ethernet
interface via MII, GMII, etc. To confuse things further, these PHYs
are controlled by an MDIO bus that has it's master in the switch
chip, while the switch chip is a slave to the MDIO master in the
ethernet controller.
The goal is to be able to configure the switch ports and set media on
one of them, for example. That code path could be entirely
independent from the ethernet infrastructure, if dev/miibus didn't
require if_media and hence a struct ifnet. This discussion is also
about how to deal with this entanglement.
The MII connection between the ethernet controller and the switch
chip (usually referred to as the "CPU" port) is hard-coded and has no
media settings, so there's no question what if_media settings should
be presented on the interface.

Most switches (AR8x16 for example) have configurable MII port, so you
can choose to use MII/RMII/GMII/RGMII. If you decide to use MII
media can not be higher than 100baseTX. So it is possible to use some
auto-negotiation here.

Post by Stefan Bethke

Post by Juli Mallett
are a lot of switches out there that don't look or act much like
MII-driven PHYs, but which are connected over MDIO, as I've tried to
stress before. I hope there will be as much separation between the
MII work that is being done and the switch work that is being done
as possible. I think anything else will prove rapidly-obsolete and
perhaps even obstructive as soon as anyone seeks to add support for
more switch chipsets to FreeBSD.
I suppose the most likely reality, though, is that people simply
won't add switch support to FreeBSD, and will administer them
out-of-band from userland, using gross kernel shims. That is
probably true whether any of the currently-outstanding work is
committed or not, unfortunately :( I just hope we'll end up with
something flexible enough, broad enough in applicability, narrow
enough in requirements, etc., that we'll have feature-rich support
for a few chipsets in tree that work in sufficiently-different
manners that they can be models for other drivers in the future.

Which is a valid approach from a vendor's viewpoint. The reason
we're talking is to try and make it easier to write a switch driver
for this framework than roll your own hack. :-)
Stefan
--

[Stefan Bethke]

Post by Aleksandr Rybalko

Post by Stefan Bethke
The MII connection between the ethernet controller and the switch
chip (usually referred to as the "CPU" port) is hard-coded and has no
media settings, so there's no question what if_media settings should
be presented on the interface.

Most switches (AR8x16 for example) have configurable MII port, so you
can choose to use MII/RMII/GMII/RGMII. If you decide to use MII
media can not be higher than 100baseTX. So it is possible to use some
auto-negotiation here.

The point is that the admin has no need to change it, it's hard coded in the driver or statically configured via hints. And for all of this discussion, I'm using MII as a synonym for all xMII busses.


Stefan

[Stefan Bethke]

Post by Warner Losh
I think that the main issue here is that we have an assumption that we have a tree structure. However, it is more of a DAG across different domains. The hierarchy works well when each device owns all the devices below it and only interacted with them. Most devices are that way. However, in the embedded world, there's lots of reach-accrosses that are expected that break the model.

What do people think would be a good approach to solve the concrete issue without too much hackery? Aleksandr and I have presented three possible approaches:
1. have a PHY driver that acts as a proxy and talks to a separate MDIO master
2. have a proxy between the ethernet driver and the miibus driver
3. modify miibus to have a separate device for mediachg() etc.

All three suffer from a dependency problem: miibus attachment can only complete successfully when both the ethernet driver and the mdio driver have been attached. In the current model, the ethernet driver provides both the MDIO access as well as the target for the mediachg, etc. messages, so there's no attachment ordering issue.

In my miiproxy patch (2.), it is necessary for the ethernet driver to cooperate in this delayed attachment, by splitting out the miibus attachment from the device_attach method implementation. That's a fairly intrusive change, and that would need to be made to any ethernet driver that is to support such a split MDIO/MII setup.

I tried delaying just the call to mii_attach(), but it seems that interacts badly with an already attached interface. Specifically, I got a non-working interface when I tried to call mii_attach() long after if_etherattach(). Additionally, if_arge (and probably most other drivers) assumes that the miibus is attached after they've successfully attached themselves, and subsequently runs into null pointer issues when it isn't.

Would it be possible to attach miibus without having a working PHY, and only attach the PHYs once the MDIO device has been attached?

For the mips/atheros platforms, we could introduce ordering hints for nexus to make sure the mdio driver gets attached before the arge's. That's a fairly small changes (two lines), and does work, but it's more a hack than a general solution. With my proposed change to miibus (split devices), that would bring down the necessary changes to just a handful of lines.


Stefan

[Marius Strobl]

Post by Stefan Bethke

Post by Warner Losh
I think that the main issue here is that we have an assumption that we have a tree structure. However, it is more of a DAG across different domains. The hierarchy works well when each device owns all the devices below it and only interacted with them. Most devices are that way. However, in the embedded world, there's lots of reach-accrosses that are expected that break the model.

1. have a PHY driver that acts as a proxy and talks to a separate MDIO master
2. have a proxy between the ethernet driver and the miibus driver
3. modify miibus to have a separate device for mediachg() etc.
All three suffer from a dependency problem: miibus attachment can only complete successfully when both the ethernet driver and the mdio driver have been attached. In the current model, the ethernet driver provides both the MDIO access as well as the target for the mediachg, etc. messages, so there's no attachment ordering issue.
In my miiproxy patch (2.), it is necessary for the ethernet driver to cooperate in this delayed attachment, by splitting out the miibus attachment from the device_attach method implementation. That's a fairly intrusive change, and that would need to be made to any ethernet driver that is to support such a split MDIO/MII setup.
I tried delaying just the call to mii_attach(), but it seems that interacts badly with an already attached interface. Specifically, I got a non-working interface when I tried to call mii_attach() long after if_etherattach(). Additionally, if_arge (and probably most other drivers) assumes that the miibus is attached after they've successfully attached themselves, and subsequently runs into null pointer issues when it isn't.
Would it be possible to attach miibus without having a working PHY, and only attach the PHYs once the MDIO device has been attached?

This would break the concept of knowing that when mii_attach() returns
success you can talk to all the hardware necessary to present a working
interface. So you'll just need another way instead to ensure that there's
also at least a single PHY before attaching the whole thing which I don't
see getting us further with the attach ordering problem of the MDIO
provider.

Post by Stefan Bethke
For the mips/atheros platforms, we could introduce ordering hints for nexus to make sure the mdio driver gets attached before the arge's. That's a fairly small changes (two lines), and does work, but it's more a hack than a general solution. With my proposed change to miibus (split devices), that would bring down the necessary changes to just a handful of lines.

How about adding the MDIO provider via multi-pass probing? That idea
of that model was to attach things like drivers for interrupt controllers
before any other driver that requires that resource. This seems like a
perfect match here and requires neither a specific hack to the nexus
(the platform generally needs to be multi-pass aware though and the
thing enabled in subr_bus.c) nor a hack to miibus(4). We really need
to find a proper way of dealing with the constraints of the embedded-
world rather than to sprinkle hacks all over the place.

Marius

[Stefan Bethke]

Post by Marius Strobl
How about adding the MDIO provider via multi-pass probing? That idea
of that model was to attach things like drivers for interrupt controllers
before any other driver that requires that resource. This seems like a
perfect match here and requires neither a specific hack to the nexus
(the platform generally needs to be multi-pass aware though and the
thing enabled in subr_bus.c) nor a hack to miibus(4).

Please recall the devinfo graph that I posted earlier. The PHY arge0 needs to talk to is attached to the MDIO master on the switch controller, which in turn is attached to GE1's MDIO master. This would require early attachment of the switch, in turn requiring early attachment of arge_mdio, in turn requiring early attachment of mips/mips/nexus.

Post by Marius Strobl
We really need
to find a proper way of dealing with the constraints of the embedded-
world rather than to sprinkle hacks all over the place.

Why is the above is less of a hack than making the ordering in nexus configurable through a hint?


Stefan
--
Stefan Bethke <***@lassitu.de> Fon [phone]

diff --git a/sys/mips/mips/nexus.c b/sys/mips/mips/nexus.c
index b51357d..c4c207a 100644
--- a/sys/mips/mips/nexus.c
+++ b/sys/mips/mips/nexus.c
@@ -240,8 +240,11 @@ nexus_hinted_child(device_t bus, const char *dname, int dunit)
int result;
int irq;
int mem_hints_count;
+ int order;

- child = BUS_ADD_CHILD(bus, 0, dname, dunit);
+ order = 1000; /* there should be a define for the default order */
+ resource_int_value(dname, dunit, "order", &order);
+ child = BUS_ADD_CHILD(bus, order, dname, dunit);
if (child == NULL)
return;

[Marius Strobl]

Post by Stefan Bethke

Post by Marius Strobl
How about adding the MDIO provider via multi-pass probing? That idea
of that model was to attach things like drivers for interrupt controllers
before any other driver that requires that resource. This seems like a
perfect match here and requires neither a specific hack to the nexus
(the platform generally needs to be multi-pass aware though and the
thing enabled in subr_bus.c) nor a hack to miibus(4).

Please recall the devinfo graph that I posted earlier. The PHY arge0 needs to talk to is attached to the MDIO master on the switch controller, which in turn is attached to GE1's MDIO master. This would require early attachment of the switch, in turn requiring early attachment of arge_mdio, in turn requiring early attachment of mips/mips/nexus.

Yes

Post by Stefan Bethke

Post by Marius Strobl
We really need
to find a proper way of dealing with the constraints of the embedded-
world rather than to sprinkle hacks all over the place.

Why is the above is less of a hack than making the ordering in nexus configurable through a hint?

If it's generally true that driver A must be attached before driver
B as B has a dependency on A than this should be expressed and
configured in the drivers themselves and not need an extra hint to
configure it at the runtime of the kernel.

Marius

[Stefan Bethke]

Post by Marius Strobl

Post by Stefan Bethke

Post by Marius Strobl
We really need
to find a proper way of dealing with the constraints of the embedded-
world rather than to sprinkle hacks all over the place.

Why is the above is less of a hack than making the ordering in nexus configurable through a hint?

If it's generally true that driver A must be attached before driver
B as B has a dependency on A than this should be expressed and
configured in the drivers themselves and not need an extra hint to
configure it at the runtime of the kernel.

Except that it is not generally true, but only in specific configurations. Other boards have other combinations of devices. The Atheros family of switches is available both embedded into certain SoC as well as separate silicon.


Stefan

[Marius Strobl]

Post by Stefan Bethke

Post by Marius Strobl

Post by Stefan Bethke

Post by Marius Strobl
We really need
to find a proper way of dealing with the constraints of the embedded-
world rather than to sprinkle hacks all over the place.

Why is the above is less of a hack than making the ordering in nexus configurable through a hint?

If it's generally true that driver A must be attached before driver
B as B has a dependency on A than this should be expressed and
configured in the drivers themselves and not need an extra hint to
configure it at the runtime of the kernel.

Except that it is not generally true, but only in specific configurations. Other boards have other combinations of devices. The Atheros family of switches is available both embedded into certain SoC as well as separate silicon.

It still seems to be true to me that as soon as you have a
separate MDIO driver that this one needs to be attached before
any Ethernet, switch or whatever driver that needs to talk
via the MDIO lines of the former. Similarly, if the switch
attaches to an MDIO instance directly, that switch would need
to be attached before the Ethernet driver (if there is one)
that the switch is the MDIO master for (this isn't exactly
the problematic arge-case). Multi-pass is per driver module,
so if the switch attaches to a "regular" Ethernet driver
providing both the MAC and the MDIO master instead, you can
leave the default probe order and attach the switch after the
Ethernet driver. So for the problematic arge-case you would
attach the MDIO driver first (corresponding to the MDIO of
arge1), then the switch to the MDIO driver and both arge0
and arge1 in whatever order last. Allowing that one special
case to work properly shouldn't interfere with other device
combinations as it basically boils down to the presence of
a separate MDIO instance. Actually that should also work
just fine when the MDIO master sits on the IIC bus as that
again would mean that it needs to be attached before a
Ethernet or switch driver can use it.

Marius

[Adrian Chadd]

I think for switches, that doesn't necessarily hold.

ie, mii_attach() for single-PHY devices may work that way, but the weird
and wonderful world of embedded switch SoC's doesn't. You're lucky in most
instances since the bootloader does give you a mostly-working switch
config. But I doubt that's a guarantee on all platforms.

So for those (ethernet) devices, splitting out the mii/mdio stuff and _not_
necessarily presenting a working PHY may be acceptable. For now it's going
to be a subset, to having it export an MDIO bus and doing late MII
attachment may not be as architecturally "no-no" as I interpret your post.


Adrian

[Marius Strobl]

Post by Adrian Chadd
I think for switches, that doesn't necessarily hold.

Err, what exactly doesn't hold? The suggestion about using multi-pass
probing was just for the case when there's a separate MDIO master
other drivers depend on. The latter would just use the default ordering
(unless there are again ordering constraints whithin them). So if
there's no separate MDIO master driver invovled that is attached first,
the other drivers would still just be attached in the default ordering.

Post by Adrian Chadd
ie, mii_attach() for single-PHY devices may work that way, but the weird

What way?

Post by Adrian Chadd
and wonderful world of embedded switch SoC's doesn't. You're lucky in most
instances since the bootloader does give you a mostly-working switch
config. But I doubt that's a guarantee on all platforms.
So for those (ethernet) devices, splitting out the mii/mdio stuff and _not_
necessarily presenting a working PHY may be acceptable. For now it's going

If there isn't a single working PHY, why would one want to attach
miibus(4) in the first place?
What is about the opposite case, when all you have is a MDIO master
and a switch connected to it, but no MAC on the MII lines of the
switch?

Post by Adrian Chadd
to be a subset, to having it export an MDIO bus and doing late MII
attachment may not be as architecturally "no-no" as I interpret your post.

Originally, Stefan said that he wants to find a way to support all
the odd combinations found in the embedded-world and I try to come
up with model that is able to do that without needing hacks and
hints left and right. As imp@ said, interrupt controllers and
GPIO basically suffer the same dependency constraints across
otherwise unrelated devices there, so we really should find a way
to properly deal with that situation without again needing another
set of hacks and hints in other types of drivers.
As for MDIO you actually can have another layer of dependencies
between MDIO master, Ethernet switch and PHY, f.e. at work we use
a single MDIO master and switch it to different slave busses using
a multiplexer managed via GPIO ... not that I'd ever wanted to
support this via generic means in FreeBSD.

Marius