Re: [cdi-devel] Proposed Dynamic Driver Loading System

[Kevin Wolf <kevin@xxxxxxxxxx>]

On Sat, Jan 02, 2010 at 04:58:39PM +1000, Matthew Iselin wrote:
> Hi,
> 
> I have been able to get access to a WiFi hotspot, so this mail comes a
> little earlier than I thought it would. :)
> 
> This proposal covers the addition of a feature to CDI which enables drivers
> to be loaded dynamically and on-demand rather than statically at a single
> point in time.
> 
> Now that drivers no longer do their own probe of devices, but rather provide
> a set of initialisation functions to the OS which are called at a convenient
> time by the OS, it is possible to implement an (optional) scheme through
> which drivers can be loaded on-demand. The benefits of this include reducing
> the memory footprint for the OS (especially in the case of monolithic OS's)
> and greater flexibility.
> 
> The idea is that the OS stores a map of device identifiers (at this stage the
> proposal considers the use of PCI vendor and device codes) which detail the
> name of a driver which is to be loaded if the given device is detected. This
> list may be OS-specific, or there may be a generic list provided as part of
> CDI. Either way, the OS is open to interpret the driver name as it sees fit:
> for example, appending ".o" in the case of Pedigree.
> 
> This map might look something like this:
> 
> /** OS-specific map of PCI identifiers -> driver names */
> 
> struct cdi_os_pci_driver_map = {
>     0x8086, 0x1230, "piix", // Vendor, Device, Driver name
>     ...
> };

This would be an additional top-level C file, right? I don't think it's
a good idea to manually create it and keep it in sync (especially
considering that many OSes don't use all drivers), but generating such a
file looks fine to me.

> I would like for this map to be implemented as part of CDI, in order to ensure
> that the list is kept up-to-date with drivers as they are written for CDI.
> Keeping the list in CDI also decreases the amount of work required to port CDI
> to a new OS and take advantage of these kind of features.
> 
> In this early draft of the concept, there is no support for non-PCI devices.
> Things such as USB with each device having a class code (eg, mass storage, or
> human interface device) may need their own list. This would be logical as the
> method for loading drivers dynamically upon a hotplug or similar event could
> be quite different to that of detecting static PCI devices at boot-time. Hot
> plug PCI devices are considered to be out of the scope of this proposal.

No, we need to do it right. And this definitely means considering more
than coldplugged PCI devices. So what I would like too see in drivers is
something like this:

static struct cdi_bus_data* supported_devices[] = {
    CDI_PCI_VENDOR_DEVICE(0x8086, 0x1230),
};

static struct cdi_driver piix_driver = {
    .name = "piix",
    .devices = &supported_devices,
    ...
}

Accordingly, I would change the generated file to contain cdi_bus_data
structs instead of a fixed vendor ID/device ID/name schema.

> At a time defined by the OS implementing CDI, the list of PCI devices will be
> iterated, and each device will be checked against the known PCI->driver list.
> Should a driver be found in this list, the driver will be loaded (if not
> already) and used to initialise the detected device.
> 
> Now, this brings up a small tangent: it is my belief that driver loading could
> be added to the CDI interface, as something like:
> 
> /* Returns -1 if the driver could not be loaded, zero otherwise */
> int cdi_load_driver(const char *);
> 
> However, this may be difficult to justify. After all, how many actual drivers
> really need to load another driver? The alternative is to have the OS perform
> driver loading itself, which effectively moves most of this subsystem to the
> OS side (making this hardly related to CDI at all :) )

Right, I'm pretty sure that it's hard to justify. Maybe I'm missing
something, so let me ask some questions: Which driver is supposed to
load the right other drivers? Does that mean that automatic loading is
only done by other drivers or would the OS still load some by itself?
And what is the benefit of having cdi_load_driver()? Can we share any
more code with it? In fact, the implementation would still be OS
specific.

I think we're slowly crossing to border to device initialisation here,
which is what I've been thinking about a lot. I agree that drivers need
some way to launch other drivers - however they should do so implicitly
rather that explicitly asking for a driver name. To be precise, what
drivers want to do is to say "Hey CDI, I can provide this and that
device" and the CDI implementation would load the driver and call
init_device() there.

> So the routine at boot-time might be something like this, for a monolithic
> kernel:
> 1. Probe the PCI bus and develop a list of devices that are present
> 2. Use this list to find drivers to load based on the PCI->driver map
> 3. Load each driver, calling the necessary initialisation function for each
>    relevant device.
> 4. (optional) Install hooks for hotplugging
> 
> Point 4 may need explanation. The kernel will need to provide some form of
> callback to be called when a new device becomes available (eg, a USB device
> is plugged in). Whilst non-PCI devices are out of the scope of this initial
> proposal, it cannot hurt to look into the crystal ball and add support for
> future expansion.

Personally, I tend to consider coldplugging some kind of hotplugging
that incidentally happens at driver startup.

> As an aside, this might also be a good time to mention that it might be nice
> to start a wiki or something somewhere to document CDI, how to port CDI to a
> new OS, and some of the optional features like the one proposed here.

Yeah, maybe. I guess the problem is to find someone to write the
documentation. ;-)

I'm not sure what kind of documentation you are aiming for. I think we
can still improve the Doxygen documentation a lot. My recent patch to
add documentation for the Core module was only a start. Doxygen has one
important advantage over a wiki: The documentation is inside the
repository. It's always available, even if you're offline (or the server
is), and I think external resources are more likely to outdate.

For more high-level tutorials or such I haven't had a look yet if this
can be integrated into the Doxygen documentation in a helpful way. We
could consider setting up a wiki for that, or - if it's only a single
tutorial - a wiki page in the osdev.org wiki.

What do other people think? I'm open for suggestions.

Kevin