Post by Hoyer, Marko (ADITG/SW2)Post by Greg KHWhat do you mean by this? What is limiting this? What is your limit?
How large are these kernel modules that you are having a hard time to
build into your kernel image?
- As far as I remember, we have special fastboot aware partitions on the emmc that are available very fast. But those are very limited in size. But with this point I'm pretty much not sure. This is something I got told.
- targeted kernel size: 2-3MB packed
- we have heavy graphics drivers (~800kb, stripped), they are needed half the way at startup
- video processing unit drivers (don't know the size), they are needed half the way at startup
- wireless & bluetooth, they are needed very late
- usb subsystem, conventionally needed very late (but this finally depends on the concrete product)
- hot plug mass storage handling, conventionally needed very late (but this finally depends on the concrete product)
- audio driver, in most of our products needed very late
- some drivers for INC communication (partly needed very early -> we compiled in them, partly needed later -> we have them as module)
All in all I'd guess we are getting twice the size if we would compile in all the stuff.
All of those should dynamically be loaded when the hardware is found by
the system, so I don't see why you are trying to load them "by hand".
Post by Hoyer, Marko (ADITG/SW2)Post by Greg KHPost by Hoyer, Marko (ADITG/SW2)- Loading the image is a kind of monolithic block in terms of time
where you can hardly do things in parallel
How long does loading a tiny kernel image actually take?
I don't know exact numbers, sorry. I guess something between 50-200ms
plus time for unpacking. But this loading and unpacking job is
important since it is located directly on the critical path.
Exact numbers matter :)
Post by Hoyer, Marko (ADITG/SW2)Post by Greg KHPost by Hoyer, Marko (ADITG/SW2)- We are strongly following the idea from Umut (loading things not
before they are actually needed) to get up early services very early
(e.g. rendering a camera on a display in less than 2secs after power
on)
Ah, IVI, you all have some really strange hardware configurations :(
Yes IVI. Since we are developing our hardware as well as our software
(different department), I'm interested in getting more infos about
what is strange about IVI hardware configuration in general. Maybe we
can improve things to a certain extent. Could you go more into
details?
Traditionally IVI systems are _very_ underpowered, use old processors,
have tiny storage systems on very slow interfaces, very little memory,
huge numbers of IPC calls at startup due to legacy userspace programs
written originally for other operating systems, and expect to get
high-performance results out of bad hardware decisions.
Post by Hoyer, Marko (ADITG/SW2)Post by Greg KHThere is no reason you have to do a "cold reset" to get your boot times
down, there is the fun "resume from a system image" solution that
others have done that can get that camera up and displayed in
milliseconds.
I'm interested in this point.
- Are you talking about "Save To RAM", "Save to Disk", or a hybrid combination of both?
- Or do you have something completely different in mind?
A number of devices in the past have done a "save system image" to
flash, and then when starting up, just load the system image into memory
and jump into it, everything up and running with no "startup time"
needed other than the initial memory load.
When updating the system with new software, just be sure to write out a
new "initial system state" image at the same time, and you should be
fine for your next boot.
This idea / implementation has been around for a very long time, and has
shipped in lots of devices, solving the "image in x seconds from power
on" problem quite easily.
Post by Hoyer, Marko (ADITG/SW2)- either the resume solution is robust enough to guarantee to come up properly every boot up
- achieved for instance by a static system image that brings the system into a static state very fast, from which on a kind of conventional boot is going on then ...
This is the easiest to do.
Post by Hoyer, Marko (ADITG/SW2)- or the boot up after an actual "cold reset" is fast enough to at least guarantee the really important timing requirements in case the resume is not coming up properly
If you have good hardware, do this, but odds are, your hardware can't do
this, otherwise we wouldn't be having this whole conversation :)
Post by Hoyer, Marko (ADITG/SW2)Post by Greg KHPost by Hoyer, Marko (ADITG/SW2)- Some modules do time / CPU consuming things in init(), which would
delay the entry time into userspace
Then fix them, that's the best thing about Linux, you have the source
to not accept problems like this! And no module should do expensive
things in init(), we have been fixing issues like that for a while now.
This would be properly the cleanest solution. In a long term
perspective we are of course going this way and we are trying to get
suppliers to go this way with us as well. But finally, we have to
bring up now products at a fixed date. So it sometimes is easier, and
more stable to work around suboptimal things.
As you have the source to everything, this shouldn't be an issue.
Post by Hoyer, Marko (ADITG/SW2)- refactoring a driver that is doing lots of CPU intensive things in init()
No driver should be doing this, if it does, push back on the supplier of
it as unacceptable.
Post by Hoyer, Marko (ADITG/SW2)vs.
- taking the module as it is and using the time by loading things from emmc in parallel
Work around crappy code? No, push back, as it obviously doesn't work
properly.
Post by Hoyer, Marko (ADITG/SW2)Post by Greg KHPost by Hoyer, Marko (ADITG/SW2)-> deferred init calls are not really a solution because they
cannot
Post by Hoyer, Marko (ADITG/SW2)be controlled in the needed granularity
We have loads of granularity there, how much more do you need?
Post by Hoyer, Marko (ADITG/SW2)So finally it is actually a trade of between compiling things in and
spending the overhead of module loading to gain the flexibility to load
things later.
That's fine, but you will run into the kernel lock that prevents
modules loading at the same time for some critical sections, if your
I/O issues don't limit you already.
There are lots of areas you can work on to speed up boot times other
than worrying about multithreaded kernel module loading. I really
doubt this is going to be the final solution for your problems.
It is of course not. The initial intention to develop something new
here on top of kmod or systemd-modules-load was not to load kernel
modules in parallel. We found that for lots of our modules we can
actually gain some benefit by loading things in parallel so we decided
to include the threaded approach as well.
The initial motivation to develop something new here was to get rid of
using the "udevd" / "udevadm trigger" approach during startup. This
"setting up the system hardware completely in one early phase during
startup" approach is not going well with our timing requirements. So
we are setting up our static hardware piece by piece exactly at this
point in time when it is needed using our tool. Besides the actually
module loading, this tool provides a mechanism for synchronization,
and is doing some addition setup stuff on top. The threaded loading is
just a feature.
- in an idle system, systemd-udev-trigger.service takes by about
150-200ms just to get the complete device tree to send out "add"
uevents again (udevd was deactivated while doing this measure)
This all depends on the size of your device tree.
Post by Hoyer, Marko (ADITG/SW2)- the processing of the resulting uevents by udevd takes 1-2 seconds
(with the default rule set) again in an idle system
That's a very long time, something is wrong there, do you have too many
rules with lots of userspace processes being called out?
Post by Hoyer, Marko (ADITG/SW2)- in a general solution, we'd need to wait for udevd to completely
settle until we can start using the devices
As others have pointed out, you should never have to wait for 'udevd to
settle' if so, something is wrong with your kernel drivers, or your
system design.
best of luck,
greg k-h