Christian Ebner [Mon, 26 Feb 2024 11:08:30 +0000 (12:08 +0100)]
encoder/format: finish payload stream with marker
Mark the end of the optional payload stream, this makes sure that at
least some bytes are written to the stream (as empty archives are not
allowed by the proxmox backup server) and possible injected chunks
must be consumed.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Christian Ebner [Tue, 20 Feb 2024 13:07:14 +0000 (14:07 +0100)]
encoder: add payload advance capability
Allows to advance the payload writer position by a given size.
This is used to update the encoders payload input position when
injecting reused chunks for files with unchanged metadata.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Christian Ebner [Wed, 13 Mar 2024 08:00:21 +0000 (09:00 +0100)]
encoder: add payload position capability
Allows to read the current payload offset from the dedicated payload
input stream. This is required to get the current offset for calculation
of forced boundaries in the proxmox-backup-client, when injecting reused
payload chunks into the payload stream.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Christian Ebner [Mon, 29 Apr 2024 08:41:18 +0000 (10:41 +0200)]
decoder: set payload input range when decoding via accessor
When accessing the file contents via the sequential file restore
the range of the payload contents cannot be inferred a-priori but need
to be calculated based on the payload references encountered during
decoding.
Extending the `SeqRead` trait by the method `update_range` allows to
set the range in the payload reader instance by implementing the
method for `SeqReadAtAdapter`, thereby setting the correct content
range to be accessed.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Christian Ebner [Tue, 12 Mar 2024 14:06:44 +0000 (15:06 +0100)]
decoder/accessor: allow for split input stream variant
When a pxar archive was encoded using the split stream output
variant, access to the payload of regular files has to be redirected
to the corresponding dedicated input.
Allow to pass the split input variant to the decoder and accessor
instances to handle the split streams accordingly and decode split
stream archives.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Christian Ebner [Thu, 1 Feb 2024 08:47:18 +0000 (09:47 +0100)]
encoder: allow split output writer for archive creation
During regular pxar archive encoding, the payload of regular files is
written as part of the archive.
This patch introduces functionality to instead attach a writer variant
with a split payload writer instance to redirect the payload to a
different output.
The separation of data and metadata streams allows for efficient
reuse of payload data by referencing the payload writer byte offset,
without having to reencode it.
Whenever the payload of regular files is redirected to a dedicated
output writer, encode a payload reference header followed by the
required data to locate the data, instead of adding the regular payload
header followed by the encoded payload to the archive.
This is in preparation for reusing payload chunks for unchanged files
of backups created via the proxmox-backup-client.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Christian Ebner [Thu, 1 Feb 2024 11:51:56 +0000 (12:51 +0100)]
format/examples: add header type `PXAR_PAYLOAD_REF`
Introduces the header type `PXAR_PAYLOAD_REF` to mark regular file
entry payloads, not encoded within the regular pxar archive but
rather redirected to a dedicated payload output writer.
It therefore substitutes the `PXAR_PAYLOAD` header type for these
entries.
The header marks the start and size for a `PayloadRef` typed object
in the archive, storing the offset to the payload header offset in the
payload stream of the dedicated payload output as well as the payload
size.
The `PayloadRef` provides the means to store, serialize and
deserialize the entry.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Christian Ebner [Tue, 19 Mar 2024 13:51:08 +0000 (14:51 +0100)]
encoder: move to stack based state tracking
In preparation for the proxmox-backup-client look-ahead caching,
where a passing around of different encoder instances with internal
references is not feasible.
Instead of creating a new encoder instance for each directory level
and keeping references to the parent state, use an internal stack.
Adds additional helper functions to solve borrow issues, when both
the state and writers have to be accessed by a mutable reference.
This is a breaking change in the pxar library API.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Christian Ebner [Thu, 23 May 2024 06:54:58 +0000 (08:54 +0200)]
lib: add type for input/output variant differentiation
Introduce an enum which stores 2 different possible variants of
inputs or outputs to be passed to encoder and decoder/accessor
instances, depending whether to read/write a fully self contained
pxar archive or whether to split off the payload stream into a
separate input/output.
Co-authored-by: Dominik Csapak <d.csapak@proxmox.com> Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Max Carrara [Fri, 21 Jul 2023 15:58:19 +0000 (17:58 +0200)]
decoder: aio: Make `TokioReader` public
This exposes `decoder::aio::TokioReader<T>` in a similar manner to
`decoder::sync::StandardReader<T>`, which is necessary if one wants
to remain generic over `T: tokio::io::AsyncRead`, e.g.:
Stefan Reiter [Wed, 31 Mar 2021 10:21:43 +0000 (12:21 +0200)]
decoder/aio: add contents() and content_size() calls
Returns a decoder::Contents without a wrapper type, since in this case
we don't want to hide the SeqRead implementation (as done in
decoder::sync). For conviencience also implement AsyncRead if "tokio-io"
is enabled.
Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
Stefan Reiter [Tue, 9 Feb 2021 12:03:47 +0000 (13:03 +0100)]
make aio::Encoder actually behave with async
To really use the encoder with async/await, it needs to support
SeqWrite implementations that are Send. This requires changing a whole
bunch of '&mut dyn SeqWrite' trait objects to instead take a 'T:
SeqWrite' generic parameter directly instead. Most of this is quite
straightforward, though incurs a lot of churn (FileImpl needs a generic
parameter now for example).
The trickiest part is returning a new Encoder instance in
create_directory, as the trait object trick with
SeqWrite::as_trait_object doesn't work if SeqWrite is implemented for
generic '&mut S'.
Instead, work with the generic object directly, and express the
owned/borrowed state in the Encoder (to avoid nested borrowing) as an
enum EncoderOutput.
Add to the aio test to ensure the Encoder is now actually useable.
Signed-off-by: Stefan Reiter <s.reiter@proxmox.com> Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
unfortunately, futures::io::AsyncRead and tokio::io::AsyncRead no longer
share a do_poll_read signature, so we need to adapt one to the other
(and also no longer generate some wrapper implementations via macro).
When using the random accessor to access FIFOs or sockets,
the ranged reader limits the data to only that entry, and
the `decode_entry` will never see a `PAYLOAD` or
`GOODBYE_TABLE` item to finish the entry.
Instead, it'll reach EOF and we need to handle this.
The accessor now tells the decoder to expect EOF as a valid
condition for ending the entry.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
projects / pxar.git / log