I know this is kind of playing with fire, but since NeXuS dock is the easiest to customize, I've been messing with some downloaded skins as well as my own.

I notice that different graphics can be used based on which orientation or screen edge the dock is set to through [BackgroundTop/Bottom/Left/Right]. I've also noticed some skins simply call [Background] but this doesn't seem to do anything different from [BackgroundBottom] since it always takes effect when the dock is floating. This also means that subdocks use the image of their respective orientation (floating being bottom-aligned).

Is there a way to set a background for a floating dock as well as for the subdocks regardless of position/orientation?

From your description it seems you are dealing mainly with the Stardock's ObjectDock skin format (Nexus is compatible with skins for all 3rd party docks, but it has its own skin format as well).

'BackgroundBottom' is the same as 'Background' only, since that is the default skin position for the dock.

Anyway, in the Appearance tab of the Dock Properties dialog there is a 'Do not flip background for docks at the top of the screen' setting. Have you tried playing with it already?

Yes, but this only make the issue more apparent. I'm using a dock with a rounded-corner background that's flat on the side docked to the edge of the screen. I made a flipped version of this background for the top edge and one completely rounded for left and right.

The problem is that subdocks with the horizontal orientation use the "flat" background and the dock itself uses the same when undocked, which looks ugly.

Probably not what you want, but you can force sub-docks to always open as vertical sub-docks in Dock Properties -> Behavior -> Sub-Docks -> How would you like your sub-docks to open? -> Always Vertical

Not exactly, but I suppose it's the next best thing if there's no alternative. Thanks!

Hmmm... there is another alternative, but it is a bit complicated.

If you open the Dock Properties dialog for the parent dock, go to the Behavior tab and then click the Sub-Docks button, you will find a 'Sub-docks inherit parent dock settings' option.

If you clear this option you will be able to manually set some properties for individual sub-docks, e.g.; each sub-dock can have its own settings AND theme.

With this option cleared if you right click the sub-dock icon you should now find a new 'Sub-dock Properties' option in the context menu.

From there you can apply different themes to different sub-docks, i.e.; you could make a theme that would apply to the sub-docks only, but you will have to apply this theme to each sub-dock individually by right clicking each sub-dock -> Sub-dock Properties-> Themes tab (and it would probably be a good idea to then lock this sub-dock theme so you don't accidentally change it when switching to a new main theme).