The Nebula3 Resource Subsystem

More info on the Resource subsystem. Since this is work in progress, details like class names may change. Generally speaking, the Nebula3 resource subsystem is more open, and gives the programmer much more control over how resources are created and managed as compared to N2.

Nebula3 Resources have the following properties:

• wrap some sort of data required by other Nebula subsystems
  
• can be shared by ResourceId
• can be loaded (initialized) and unloaded at any time
  
• can be loaded synchronously or asynchronously

Typical graphical resources are for instance Meshes and Textures, however the Resource subsystem is not limited to graphical resources.

The resource subsystem has 2 operational levels (maybe those will be placed into 2 different namespaces, at the moment they are both under namespace Resources):

The lower level provides actual resoure objects, handles resource sharing, loading and (less important) saving. The low level resource classes are:

• ResourceId
  
• Resource
• ResourceLoader
• ResourceSaver
• SharedResourceServer.

The higher level of the resource subsystem provides resource management, which means loading or unloading resources dynamically based on some feedback from resource users. The high level resource subsystem classes are:

• ResourceProxy (alternate class name: ManagedResource)
  
• ResourceProxyServer (alternate class name: ResourceManager)
  
• ResourceMapper

Here's how the resource subsystem classes actually work together:

A ResourceId is a unique identifier for a resource. The resource id is used for sharing, and for locating the resource data on disc (or wherever the data is stored). ResourceIds are string atoms. Atoms are unique 32-bit identifiers for constant strings (or other complex data types) which speed up copying and comparing alot, and also reduce the memory footprint since identical strings are stored only once. To locate the resource data on disc, ResourceIds usually resolve into a valid URL (a resource id may for instance look like "texture:materials/granite.dds", which would resolve into something like "file:///C:/Programme/[AppName]/export/textures/materials/granite.dds" at runtime.

A Resource object is the actual container for the resource data. Specific resource types like textures and meshes are subclasses of Resource with specialized class interfaces. Resource subclasses are often platform specific (for instance D3D9Texture), but are conditionally typedef'ed to platform-independent interfaces (for instance Texture). Unlike in Nebula2, resource objects do not know how to setup, load or save themselves. Instead, a suitable ResourceLoader and/or ResourceSaver object must be attached to the Resource object. Since Nebula applications rarely write out data, ResourceSavers exist more for completeness. On the other hand, ResourceLoaders are essential, since they are the only way to setup a Resource object for use. ResourceLoaders have total control over the resource setup process. They can be platform-specific, and may depend on an associated platformspecific Resource class. This gives the programmer much more control over the resource setup process compared to Nebula2. Example resource loader classes are StreamTextureLoader, StreamMeshLoader (setup textures and meshes from streams), MemoryVertexBufferLoader and MemoryIndexBufferLoader (setup vertex buffers and index buffer from data in memory).

The Resource class also provides a common interface for synchronous and asynchronous resource loading. Synchronous loading is done like this:

1. res-> SetResourceId("tex:system/white.dds");
2. res-> SetLoader(StreamTextureLoader::Create());
   
3. res-> SetAsyncEnabled(false)
4. res-> Load()
5. if (res-> IsValid()) ... then resource loading was successful, otherwise the method LoadFailed() will return true.
   

Asynchronous resource loading is very similar:

1. res->SetResourceId("tex:system/white.dds");
2. res->SetLoader(StreamTextureLoader::Create());
   
3. res->SetAsyncEnabled(true);
4. res->Load();
5. the resource will now go into pending state...
   
6. as long as IsPending() returns true, repeatedly call Load()... of course a real application would do something useful in the meantime
7. at some point in the future, after Load() is called, the state of the resource will either be Valid (resource is ready for use), Failed (loading the resource has failed) or Cancelled (the pending resource load has been cancelled)

An application or even the Nebula3 render code usually doesn't have to deal with this, since the resource management layer will take care of this and hide the details of asynchronous resource loading behind resource proxies.

The SharedResourceServer singleton offers resource sharing by ResourceId. Creating resources through the SharedResourceServer makes sure that a resource is only loaded exactly once into memory, regardless of the its client count. If the client count of a resource drops to zero, the resource is automatically unloaded (this is no substitute for proper resource management however, that's what the ResourceProxyServer cares about). Resource sharing can be bypassed completely by creating Resource objects directly with Nebula3's standard object creation mechanisms.

A ResourceProxy (or ManagedResource) is a resource management wrapper around an actual resource object. The idea is that the contained resource object may change under the control of a resource manager based on resource-usage-feedback. For instance, a texture proxy may provide a placeholder texture as long as the requested texture is background-loading, a lower resolution texture may be provided if all objects using the resource are very small on screen, the texture may be unloaded if it hasn't been rendered for X frames and so on...

The ResourceProxyServer (or ResourceManager) singleton is the frontend to the resource management system. It is the factory for ResourceProxies and associates ResourceMappers with Resource types, other then that it basically hands all work down to the attached ResourceMappers.

The ResourceMapper class is where the interesting stuff is happening. A ResourceMapper is associated with one resource type (e.g. Texture or Mesh) and attached to the ResourceProxyServer by the application. A ResourceMapper is responsible to load/unload resources based on resource usage feedback from the rendering code. Subclasses of ResourceMapper may implement different resource management strategies, and it should be possible to create completely customized, platform- and application-specific resource management scenarios by deriving specialized subclasses from ResourceMapper and probably ResourceLoader. The goal is of course, that Nebula3 provides a good set of ResourceMappers out of the box for simple cases (just load everything that is required) up to streaming scenarios for large worlds.

The resource usage feedback is written by the rendering code to ResourceProxy objects and should include stuff like whether the resource may be needed in the near future, whether the resource was visible at all, and a guesstimate of the screen space size of the object using the resource. However the specific feedback depends on the ResourceProxy subclass, there are no common feedback methods in the ResourceProxy class.

Based on the resource usage feedback, a ResourceMapper could implement the following operations (however that's completely up to the actual mapper):

• Load: asynchronously load a resource at a specific level-of-detail (for instance skipping higher res texture mipmaps if not needed), provide a placeholder for res0urces that are still loading
  
• Unload: completely unload a resource, freeing valuable memory
• Upgrade: increase level-of-detail for an already loaded resource (for instance loading higher-resolution mipmap-levels of a texture)
• Degrade: decrease level-of-detail for a loaded resource (for instance dropping higher-resolution mipmap levels of a texture)

That's it so far! I plan to do the first code drop with a very simple resource management (asynchronously load required resources, but don't care about on-screen-size or automatically unloading resources).