Are there any Wayland compositor 'toolkits'? 
I'm thinking something that lets you design your own compositor by writing limited code and configuration.
Not locked into a specific paradigm and more capable than just "customize this off-the-shelf WM", but less work than rolling your own with a bunch of libraries.
(Customizable WMs *may* be suitable answers, if you can customize both their appearance and behaviour far enough that a user would not be able to identify the underlying WM from using a customized installation, as an imperfect yardstick.)
@joepie91@pixie.town The way river works might interest you? River has a compositor that's independent of what it calls a "layout generator" which is an external program which manages window positioning. River manages (and enforces) tags and outputs, but window positioning itself within an output is entirely delegated.
It seems there's a big rewrite on its way that includes renaming the role of layout generator to window manager, with all that implies (for example, among the functionality removed from the compositor in that pr is tags)
@joepie91@pixie.town I think "tiling only" is a trap. The faq has a "my windows are not tiled!" that mentions this happens without a layout generator. Not sure myself if a current layout generator can just handle floating windows atm but this sounds like something that will be very possible with 0.4 (the rewrite).