Most of the time this was sufficient, but under certain conditions artifacts were visible-- such as the surface of a smoothly varying sphere using triplanar texturing, whose weights are based on the normals.
Here is a visualization of the Triplanar weights generated from the Oct16 normals. |
Even switching to Oct20(10 bits per channel) does not completely solve this, the diamonds are much smaller, but they persist.
Oct16, but with custom scale/bias |
limited range of the world space normals.
I track min/max per channel for the entire patch, then encode the normals so that the full range of bits is used.
Decoding in the shader requires a custom scale and bias parameter per channel(4 floats for the two channel Oct16).
There are no extra instructions, as a fixed scale of 2 and bias of -1 was previously being used to transform from [0,1] to [-1,1] range.
The 2nd image was encoded this way, the normals are still using Oct16, so only 16 bits per normal, but with a custom scale/bias per patch.
In the majority of cases this provides many extra bits of precision, and in the worst case it degrades back to standard Oct16.