Add support for erofs and squashfs layer media types

[rchincha]

OCI artifacts support has landed in various OCI specs v1.1.0 which allows for arbitrary artifact types, small and large.

Large artifacts (even existing container images) pose a particular challenge that:

1. it takes too long to download
2. it takes too long to unpack

This PR begins to address 2. above.

squashfs (an older read-only) and erofs (newer) filesystems are already compressed full filesystems, and overlayfs compatible.

[sudo-bmitch]

This needs a rebase on main to be considered. Also, I don't think we need to include non-OCI media types, especially if they've been registered with IANA.

[rchincha]

#1190

[rchincha]

opencontainers/distribution-spec#537
to be combined with ^

[rchincha]

This needs a rebase on main to be considered. Also, I don't think we need to include non-OCI media types, especially if they've been registered with IANA.

This PR just represents possibilities. Needs a discussion if OCI wants to "adopt" this a recommended option. Almost all known solutions end up being a clever fuse or remotefs wrapper around a layer - so why not make the layer a filesystem itself! No further machinery needed.

[hsiangkao]

This needs a rebase on main to be considered. Also, I don't think we need to include non-OCI media types, especially if they've been registered with IANA.

This PR just represents possibilities. Needs a discussion if OCI wants to "adopt" this a recommended option. Almost all known solutions end up being a clever fuse or remotefs wrapper around a layer - so why not make the layer a filesystem itself! No further machinery needed.

JFYI, EROFS already has a IANA-registered media type "vnd.erofs" if some format is referred.

[rchincha]

Could also be application/vnd.oci.image.layer.erofs and application/vnd.oci.image.layer.squashfs
just so that we indicate this is OCI blessed.

Open question, whiteout handling

[syed]

The problems I see with large binary files (eg AI models) is that they don't tend well to compression. I would like to see if we can improve the distribution spec to allow parallel chunked upload/download. Right now we can only upload/download sequentially

[sudo-bmitch]

Right now we can only upload/download sequentially

I get the impression that most registries allow this already and we're just formalizing it.