It's not enough to check whether $! is set. From "perldoc perlvar":
> Many system or library calls set "errno" if they fail, to
> indicate the cause of failure. They usually do not set "errno"
> to zero if they succeed and may set "errno" to a non-zero value
> on success. This means "errno", hence $!, is meaningful only
> *immediately* after a failure:
To protect against potential issues, check the return value of unlink
and only check $! if it failed.
Adds the ability to change the owner of a guest image.
Btrfs does not need special commands to rename a subvolume and this can
be achieved the same as in Storage/plugin.pm's rename_volume taking
special care of how the directory structure used by Btrfs.
Daniel Kral [Wed, 21 Aug 2024 13:57:47 +0000 (15:57 +0200)]
esxi: fix #5587: add support for older version of vmx storage filepaths
Allow the ESXi storage disk entry property "fileName" to be flatcased
("filename") in addition to being camelcased ("fileName"). This adds
compatibility with older ESXi .vmx configuration files.
Fiona Ebner [Mon, 10 Jun 2024 09:04:15 +0000 (11:04 +0200)]
volume import: assume target API version is at least 9
The storage API version has been bumped to at least 9 since
libpve-storage = 7.0-4. If the source node is on Proxmox VE 8, where
this change will come in, then the target node can be assumed to be
running either Proxmox VE 8 or, during upgrade, the latest version of
Proxmox VE 7.4, so it's safe to assume a storage API version of at
least 9 in all cases.
As reported by Maximiliano, the fact that the 'apiinfo' call was
guarded with a quiet eval could lead to strange errors for replication
on a customer system where an SSH connection could not always be
established, because the target's API version would fall back to 1.
Because of that, the '-base' argument would be missing for the import
call on the target which would in turn lead to an error about the
target ZFS volume already existing (rather than doing an incremental
sync).
Reported-by: Maximiliano Sandoval <m.sandoval@proxmox.com> Signed-off-by: Fiona Ebner <f.ebner@proxmox.com> Reviewed-by: Max Carrara <m.carrara@proxmox.com>
plugin: move definition for 'port' option to base plugin
Commit 7020491 ("esxi: add 'port' config parameter") started using
the 'port' option in a second plugin, but the definition stayed in the
PBS plugin. Avoid the hidden dependency and move the definition to the
base plugin instead.
It is necessary to mark it as optional or it would be required always.
Clarify that the option is not used by NFS and CIFS.
This diverts stderr of the fuse process to a pipe, which makes no
sense as it runs daemonized in a scope, also, the pipe fd was used as
a ready-signal, which now does not trigger anymore.
Dominik Csapak [Fri, 10 May 2024 13:56:58 +0000 (15:56 +0200)]
esxi: improve error handling for fuse mount tool
if the fuse tool encounters an error early, it prints it like:
Error: some error message
on stderr.
Redirect STDERR of the child process (which mounts the ESXi instance) to
the pipe of the parent (API) process, so that it can pass a hopefully
more meaningful message to the user than just an erroneous return code.
This prevents importing from vmdks with whitespaces in file names.
Further, some operations that include file sizes (like listing disks)
would potentially fail entirely if a custom disk with a badly name
backing device exists in a VM images directory since they don't expect
this. Specifically, since we don't necessarily know the actual naming
scheme of the current storage in the plain Plugin.pm version, we don't
check the full name anyway, so why bother with whitespaces...
See-also: https://forum.proxmox.com/threads/new-import-wizard-available-for-migrating-vmware-esxi-based-virtual-machines.144023/page-16#post-658697 Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
Mira Limbeck [Wed, 17 Apr 2024 09:48:57 +0000 (11:48 +0200)]
fix insecure migration failing if waiting on lock
both STDOUT and STDERR are written into `$info` which is then parsed for
IP and port of the target socket listening.
when the ports file can't be locked immediately `trying to acquire
lock...` is printed on STDERR and in turn written into `$info`.
trying to parse the IP then fails, resulting in a migration or
replication failing.
the bare open3 call is replaced by the run_command wrapper from
pve-common to use a safe wrapper around open3 with the same
functionality.
STDERR is read separatey from STDOUT and the last line of STDERR is
kept in case of errors.
Fixes: 57acd6a ("fix #1452: also log stderr of remote command with
insecure storage migration")
Signed-off-by: Mira Limbeck <m.limbeck@proxmox.com>
Max Carrara [Tue, 2 Apr 2024 14:55:20 +0000 (16:55 +0200)]
cephconfig: align our parser with Ceph's parser
This commit rewrites the entire parser for ceph.conf, aligning its
behaviour as closely as possible with Ceph's parser grammar [0].
The most notable improvements are as follows:
1. The characters '#' and ';' now both mark comments, instead of
just the '#' character.
2. Any character, including comment literals ('#' and ';'), may now
be escaped.
3. Quoted values (single and double) are now supported.
4. Line continuations are now supported (lines ending with '\').
5. Repeated whitespace characters in keys are now treated as a
single space character.
6. Dashes '-' are not treated the same as spaces and underscores
anymore, as Ceph's grammar doesn't treat them that way.
* Paired with 5., this means that repeated whitespace is now
equivalent to a single underscore.
7. Escaped comment literals are now un-escaped.
8. Although not too crucial, the parser now also supports empty
sections and will just initialize them with an empty hash.
Furthermore, the original grammar's more quirky behaviours are also
respected where sanely possible.
This commit changes the code style of subroutine `write_ceph_config`
to match our style guide [0] more.
Furthermore, the repeated calls to the inner subroutine are replaced
with a loop, while the regular expressions used by the inner `sub` are
now quoted with `qr` to prevent any accidental mis-quotings in the
future.
Instead of just using it as a warning and then trying to parse an
empty string as json.
For example, trying to parse unsupported vmdks, previously we'd see
something like this:
qemu-img: Could not open
'/run/pve/import/esxi/foo/mnt/ha-datacenter/vsanDatastore/asdf/asdf-000001.vmdk':
Unsupported image type 'vsanSparse'
could not parse qemu-img info command output for
'/run/pve/import/esxi/foo/mnt/ha-datacenter/vsanDatastore/asdf/asdf-000001.vmdk'
- malformed JSON string, neither tag, array, object, number, string
or atom, at character offset 0 (before "(end of string)") at
src/PVE/Storage/Plugin.pm line 962, <DATA> line 960.
Now it simply shows:
qemu-img: Could not open
'/run/pve/import/esxi/foo/mnt/ha-datacenter/vsanDatastore/asdf/asdf-000001.vmdk':
Unsupported image type 'vsanSparse'
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
Thomas Lamprecht [Wed, 27 Mar 2024 12:11:22 +0000 (13:11 +0100)]
esxi: reduce cache invalidation time to 30s
Reduce the time the cache stays valid from 60s to 30s, while this
could double the amount of requests in the worst case, it's still not
that frequent and also halves the maximal time a user has to wait to
see changes on the ESXi side to appear here.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Gabriel Goller [Thu, 21 Mar 2024 09:07:52 +0000 (10:07 +0100)]
esxi: detect correct os type in 'other' family
This patch introduces the conversion table for all possible OS Types
that are in the VMWare 'other' family and sets the pve counterpart.
Our default OS Type is 'linux', so including mappings to 'other' makes
sense.
Signed-off-by: Gabriel Goller <g.goller@proxmox.com>
following pve-esxi-import-tools's commits: 3ee5c3b ("esxi-folder-fuse: add --insecure option") c292c67 ("listvms.py: add --insecure parameter, verify cert by
default") 34c87be ("rename --insecure option to --skip-cert-verification")
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
[ TL: rename 'insecure' to 'skip-cert-verification' to better convey
what it means ] Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
[ TL: fix wrong comparison with >= and avoid undef warning if file
does not yet exist at all ] Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
[ TL: squash removal of both subs into one commit ] Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Thomas Lamprecht [Sun, 10 Mar 2024 18:25:34 +0000 (19:25 +0100)]
api: import-metadata: make warnings structured & merge ignored-volumes
This allows the frontends to translate them and avoids somewhat
duplicated info by having some warnings explicitly (ignored-volumes)
while others are in the warnings array.
By passing along the key and the value the frontend can also show the
warnings in-line, e.g. by marking a disk-entry in a grid as having
potential problems.
Ideally we'd have a central list of known types used for the API
return schema enum and to check when calling the $warn closure, but as
we only got three warnings keep this as is and only add a comment.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
This will be used for returning the base meta information of a
external VM that is about to be imported into Proxmox VE.
A front-end can use this endpoint to show the proposed configs with
potential override switches to the user, so that they can adapt the
most important options to ensure that import can work.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
[ TL: add more commit message with some background ] Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Add a new 'import' content type which will be the corner stone for a
better API and UI integrated way to import virtual guests into Proxmox
VE.
For starters this will be used to implement a ESXi adapter, so that
those VMs can get imported nicely.
Later we want to integrate the OVF/OVA import skeletons we got in
qemu-server to something more usable here.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
[ TL: add more commit message with some background ] Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Thomas Lamprecht [Mon, 19 Feb 2024 16:13:41 +0000 (17:13 +0100)]
fix #5254: api: allow usage of download-url with Sys.AccessNetwork
The download-url API endpoint has some implications that admins are
unaware of, namely that it basically allow to scan the whole network
via HTTP URLs, and potentially even download some image that the user
should not have access to and adding to a VM that the user controls.
That's why in addition to the Datastore.AllocateTemplate privilege on
the storage, the Sys.Modify on the whole Cluster was required to use
the API call. That design was chosen as we were not fully sure if a
separate privilege is warranted, but user feedback has shown that the
(not so big) cost of adding such a new privilege is justified.
Change the permission check to allow the combination of
Datastore.AllocateTemplate on the storage and either 'Sys.Modify' on
/, for backwards compatibility, or the newer 'Sys.AccessNetwork' on
the node that handles the download.
Using a node-specific ACL path allows admins to e.g. prepare one
specific node's firewall so that pveproxy can access only a safe set
of hosts via outgoing HTTP (not stemming from valid connection
tracking to the PVE API), and thus even further limit the privileges
of users or tools that are trusted to download images to a storage.
Buglink: https://bugzilla.proxmox.com/show_bug.cgi?id=5254 Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com> Tested-by: Hannes Duerr <h.duerr@proxmox.com> Reviewed-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
Thomas Lamprecht [Sun, 19 Nov 2023 19:05:50 +0000 (20:05 +0100)]
btrfs: fix calling parent create_base method in fall-back
If we want to forward to the create_base of the directory plugin while
making that use our $class for the operations that call might do, we
cannot use the -> notation (which would resolve the next actual
implementation) but rather pass the class directly.
But, DirPlugin reuses the create_base method from the base Plugin
method, so we also need to call that, because on direct call notation
the inheritance fallback to super methods isn't available.
Reported in the forum:
https://forum.proxmox.com/threads/95684/post-606535
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
fix #254: iscsi: add support for multipath targets
With this patch Proxmox now tries to login to all discovered portals
in case some of them are not logged yet.
In case of multipath configuration when initially configured portal is
missing for some reason Proxmox don't lose iSCSI storage now and can
successfully restore iSCSI connection between reboots.
Fiona Ebner [Tue, 27 Jun 2023 07:48:49 +0000 (09:48 +0200)]
cifs: bubble up NT_STATUS_INVALID_PARAMETER during connection check
instead of claiming that the storage is not online.
Would've made the issue fixed by b27da68 ("cifs: fix check connection
call") more obvious, because (the UI passes along an empty string for
domain if not set and) the smbclient call returns that status with
> -W ''
in Bookworm.
projects / pve-storage.git / log