Testing UBI, UBIFS, JFFS2 on top of MTD can be unpleasant. Various MTD emulator kernel modules exist, but none of them are entirely satisfactory. Richard is developing a new userspace NAND and NOR flash emulator using FUSE (CUSE) as the interface.

sched – no slides 🙁

For debugging an existing image, we have mtdram, block2mtd, nandsim.

mtdram is a randisk exposed as an mtd device. It’s only useful for small MTDs though. It also allows only a single instance. It also has no way to inject errors (e.g. inject a bitflip).

block2mtd is similar, but with a block device (e.g. loopbacked file) as backend rather than a vmalloc’ed area. It allows multiple instances (configured at module load time) and larger devices, good enough for basic UBI and UBIFS testing. It only emulates NOR flash though, so no wear levelling. Also no fault injection.

nandsim works with kmalloc’ed NAND pages which can be swapped out to a file passed in as a kernel module argument. It doesn’t allow to persist the swapped out data though, so not useful for an existing image. It emulates a parallel NAND chip, so it includes ECC, also delays and fault injection. It’s very slow. Its goal was to find bugs in MTD NAND and UBI subsystems, but nowadays it mostly finds bugs in nandsim itself. It is also very hard to configure, e.g. the geometry (eraseblock size etc.) is specified by a NAND ID.

For all of these, things have to be specified at module load time. This is mainly because the mtd framework doesn’t really support hotplugging.

It’s also possible to use QEMU, which can emulate flash devices. But it’s very inflexible, and also no fault injection.

Richard wants an emulator where you can add or remove MTD devices at runtime, where you can inject errors in various ways, He thought it best to do that in userspace, because that allows much more flexibility. He first tried to create an ad-hoc new mtd driver for userspace emulated flash. Then he realized that he was reinventing the wheel. He first looked at virtio in qemu. It turns out to be very complicated though, and it didn’t really fit his needs. Then he (re)discovered that FUSE has a mode called CUSE, character device in userspace. It was originally used back when OSS was still a thing, to allow OSS devices to be emulated in userspace.

CUSE gives the userspace interface to handle read, write, ioctl, .., but a real MTD device needs a lot more information. CUSE already has a lot though. Especially since MTD devices don’t need complicated things like zero-copy I/O.

The first thing in MUSE (MTD in USErspace) is to add new operations needed to make MTD happy: MUSE_READ, MUSE_WRITE, MUSE_ERASE, … A first prototype was ready in a single day. The kernel part is mostly done, it’s less than 1000 LoC. It’s not send upstream yet, because the new MUSE commands are new userspace ABI so he needs to be sure it’s good enough.

Current features (in progress): snapshots; custom image types (not just nanddump output), replay (to exactly reproduce all the operations, not just the final state). It of course has fault injection.

MUSE can also be useful beyond testing. It can be used to write a full userspace driver for an MTD device, e.g. with spidev or UIO for the low-level interface. Performance will obviously be abominable, but it can be good for prototyping.

FUSE is a client/server architecture, where userspace is the server and the kernel makes requests to it. The kernel part implements a specific class, e.g. VFS for FUSE and MTD for MUSE. Every request contains an operation, which is performed by userspace and a response is sent. The reply generally contains 3 iovecs: a general header, an operation-specific message, and a payload.

MUSE (or any new userspace driver framework) adds a few things to the kernel. It adds new FUSE operations and corresponding structures – these are UAPI. It also implements a new control character device (e.g. /dev/fusectl. Finally, there’s a driver itself that forwards requests and registers itself in the framework. There are plenty of fuse helpers to do that.

In userspace, libfuse_lowlevel has helpers to parse the requests and create responses.