MrBlake/NetBox-VM-Import
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Remove Proxmox sync support

Browse Source
This commit is contained in:
MrBlake
2026-07-20 14:24:28 +02:00
parent 7d8819c374
commit 6ec4093c97
15 changed files with 192 additions and 752 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.gitignore
+1
View File
@@ -1,2 +1,3 @@
__pycache__/
*.py[cod]
*.egg-info/
COMPATIBILITY.md
+1
View File
@@ -18,3 +18,4 @@
| 0.2.4 | 4.4.0 | 4.6.x |
| 0.2.5 | 4.4.0 | 4.6.x |
| 0.2.6 | 4.4.0 | 4.6.x |
| 0.3.0 | 4.4.0 | 4.6.x |
README.md
+14 -27
View File
@@ -1,11 +1,10 @@
# NetBox VMware & Proxmox Importer
# NetBox VMware Importer
NetBox plugin zum Synchronisieren von VMware vSphere/vCenter und Proxmox VE VMs nach NetBox.
NetBox plugin zum Synchronisieren von VMware vSphere/vCenter VMs nach NetBox.
## Funktionen
- vCenter/ESXi-Ziele direkt in der NetBox-Weboberflaeche pflegen
- Proxmox VE-Ziele direkt in der NetBox-Weboberflaeche pflegen
- Pro Ziel Tenant, Site und NetBox-Cluster hinterlegen
- Manuelle Synchronisation per Button
- Optionale automatische Synchronisation ueber ein Minutenintervall
@@ -15,7 +14,7 @@ NetBox plugin zum Synchronisieren von VMware vSphere/vCenter und Proxmox VE VMs
- Primaere IPv4/IPv6 wird anhand der ersten gefundenen Gast-IP gesetzt
- Multi-Tenant-sicherer VM-Abgleich ueber `name + cluster`
Das Plugin loescht bewusst keine verwaisten VMs, Interfaces oder IP-Adressen. Damit bleibt der erste Rollout konservativ und vermeidet Datenverlust, wenn Hypervisor-Gastdaten unvollstaendig sind.
Das Plugin loescht bewusst keine verwaisten VMs, Interfaces oder IP-Adressen. Damit bleibt der erste Rollout konservativ und vermeidet Datenverlust, wenn VMware-Gastdaten unvollstaendig sind.
## Installation
@@ -25,7 +24,7 @@ Im Python-Environment der NetBox-Installation:
pip install git+https://git.mrblake.cc/MrBlake/NetBox-VM-Import.git
```
NetBox selbst ist keine `pip`-Dependency des Plugins. Das Plugin wird im vorhandenen NetBox-Virtualenv installiert und bringt nur externe Bibliotheken wie `pyvmomi`, `cryptography`, `requests` und `proxmox-sdk` mit.
NetBox selbst ist keine `pip`-Dependency des Plugins. Das Plugin wird im vorhandenen NetBox-Virtualenv installiert und bringt nur externe Bibliotheken wie `pyvmomi` und `cryptography` mit.
In `configuration.py` aktivieren:
@@ -44,36 +43,24 @@ sudo systemctl restart netbox netbox-rq
## Nutzung
1. In NetBox unter `VM Import > Endpoints` ein Ziel anlegen.
2. Provider auswaehlen.
3. VMware: vCenter Host, Port, Benutzername und Passwort eintragen.
4. Proxmox: Proxmox Host, Port `8006`, Benutzername inklusive Realm und Passwort oder API-Token eintragen, z. B. `root@pam`.
5. Tenant, Site und Cluster fuer den Kunden auswaehlen.
6. Optional Regex-Filter und Sync-Intervall setzen.
7. Auf der Detailseite `Sync jetzt starten` ausfuehren.
1. In NetBox unter `VMware Import > Endpoints` ein Ziel anlegen.
2. vCenter/ESXi Host, Port, Benutzername und Passwort eintragen.
3. Tenant, Site und Cluster fuer den Kunden auswaehlen.
4. Optional Regex-Filter und Sync-Intervall setzen.
5. Auf der Detailseite `Sync jetzt starten` ausfuehren.
Bei selbstsignierten Proxmox-Zertifikaten `Validate SSL` deaktiviert lassen. Wenn Proxmox langsam antwortet, kann `API timeout` am Endpoint erhoeht werden.
Wenn `Sync interval minutes` gesetzt ist, prueft ein Systemjob alle fuenf Minuten, welche VMware-Endpoints faellig sind, und stellt die eigentlichen Sync-Jobs in die Queue.
Der Proxmox-Client nutzt fuer API-Token bevorzugt `proxmox-sdk` wie `netbox-proxbox`/`proxbox-api`. Passwort-Logins laufen ueber den eigenen `requests`-Client, weil `proxmox-sdk==0.0.12` die Ticket-URL fuer Passwort-Authentifizierung ohne `/api2/json` aufbaut. Proxy-Umgebungsvariablen des NetBox-Dienstes werden im `requests`-Client ignoriert, damit interne Proxmox-Adressen nicht versehentlich ueber einen HTTPS-Proxy laufen.
## Legacy-Proxmox-Endpoints
Im `requests`-Fallback ist der unauthentifizierte Proxmox-`version`-Probe nur diagnostisch. Wenn er haengt, versucht das Plugin den eigentlichen Login trotzdem. Im bevorzugten SDK-Pfad wird `version` als erste SDK-Anfrage genutzt und authentifiziert dabei automatisch.
Proxmox-API-Verbindungen nutzen TLS 1.2-Kompatibilitaetsmodus, weil einige Firewalls oder TLS-Inspektionspfade moderne TLS-1.3-ClientHellos blockieren.
Bei Proxmox koennen `Connect timeout`, `API timeout`, `GET retries` und `Retry backoff` pro Endpoint gesetzt werden. Das entspricht dem Ansatz von Proxbox, Verbindungs- und Retry-Parameter pro Proxmox-Ziel zu speichern.
Wenn Passwort-Login auf `access/ticket` haengt, zuerst den Realm im Benutzernamen pruefen (`root@pam`, `user@pve`, `user@ldaprealm`). Fuer produktive Imports ist ein Proxmox API-Token meistens stabiler als Passwort-Login.
Wenn `Sync interval minutes` gesetzt ist, prueft ein Systemjob alle fuenf Minuten, welche Endpoints faellig sind, und stellt die eigentlichen Sync-Jobs in die Queue.
Ab Version `0.1.8` wird beim Speichern eines Endpoints zusaetzlich ein wiederkehrender NetBox-Job fuer genau diesen Endpoint geplant. Nach einem Update vorhandene Endpoints einmal speichern oder `netbox-rq` neu starten und bis zum naechsten Systemjob-Lauf warten.
Proxmox-Unterstuetzung wurde entfernt. Alte Proxmox-Endpoints aus frueheren Plugin-Versionen bleiben in der Liste und Detailansicht sichtbar, damit sie geloescht werden koennen. Sie koennen nicht mehr bearbeitet, neu angelegt oder synchronisiert werden.
## Sicherheit
Das Endpoint-Passwort bzw. Token-Secret wird verschluesselt in der Plugin-Tabelle gespeichert. Der Schluessel wird aus `SECRET_KEY` abgeleitet. Wenn `SECRET_KEY` rotiert wird, muessen die Secrets in den Endpoints erneut gesetzt werden.
Das Endpoint-Passwort wird verschluesselt in der Plugin-Tabelle gespeichert. Der Schluessel wird aus `SECRET_KEY` abgeleitet. Wenn `SECRET_KEY` rotiert wird, muessen die Secrets in den Endpoints erneut gesetzt werden.
Fuer sehr strenge Umgebungen ist ein externer Secret-Store wie Vault als naechster sinnvoller Ausbaupunkt vorgesehen.
## Datenqualitaet
IP-Adressen kommen bei VMware aus den VMware Guest Tools und bei Proxmox-QEMU aus dem QEMU Guest Agent. Ohne laufende bzw. aktuelle Guest Tools koennen Interfaces und IPs fehlen oder veraltet sein.
IP-Adressen kommen aus den VMware Guest Tools. Ohne laufende bzw. aktuelle Guest Tools koennen Interfaces und IPs fehlen oder veraltet sein.
netbox_vmware_importer/__init__.py
+3 -3
View File
@@ -3,9 +3,9 @@ from netbox.plugins import PluginConfig
class VMwareImporterConfig(PluginConfig):
name = "netbox_vmware_importer"
verbose_name = "Virtualization Importer"
description = "Synchronize VMware vSphere and Proxmox VE virtual machines into NetBox."
version = "0.2.6"
verbose_name = "VMware Importer"
description = "Synchronize VMware vSphere virtual machines into NetBox."
version = "0.3.0"
author = "Internal NetBox Team"
base_url = "vmware-importer"
min_version = "4.4.0"
netbox_vmware_importer/choices.py
+2 -2
View File
@@ -24,7 +24,7 @@ class EndpointProviderChoices(ChoiceSet):
CHOICES = [
(PROVIDER_VMWARE, _("VMware vSphere"), "blue"),
(PROVIDER_PROXMOX, _("Proxmox VE"), "orange"),
(PROVIDER_PROXMOX, _("Proxmox VE (legacy)"), "gray"),
]
@@ -34,5 +34,5 @@ class EndpointAuthMethodChoices(ChoiceSet):
CHOICES = [
(METHOD_PASSWORD, _("Username/password"), "blue"),
(METHOD_API_TOKEN, _("API token"), "green"),
(METHOD_API_TOKEN, _("API token (legacy)"), "gray"),
]
netbox_vmware_importer/forms.py
+5 -25
View File
@@ -29,16 +29,8 @@ class VCenterEndpointForm(NetBoxModelForm):
cluster = DynamicModelChoiceField(
queryset=Cluster.objects.all(),
)
provider = forms.ChoiceField(
choices=EndpointProviderChoices,
required=True,
)
auth_method = forms.ChoiceField(
choices=EndpointAuthMethodChoices,
required=True,
)
password = forms.CharField(
label=_("Password / token secret"),
label=_("Password"),
required=False,
widget=forms.PasswordInput(render_value=False),
help_text=_("Leave blank to keep the currently stored password."),
@@ -47,18 +39,11 @@ class VCenterEndpointForm(NetBoxModelForm):
fieldsets = (
FieldSet("name", "slug", "enabled", "comments", name=_("Endpoint")),
FieldSet(
"provider",
"host",
"port",
"username",
"auth_method",
"token_name",
"password",
"validate_ssl",
"request_timeout_seconds",
"connect_timeout_seconds",
"max_retries",
"retry_backoff_seconds",
name=_("Connection"),
),
FieldSet("tenant", "site", "cluster", name=_("NetBox target")),
@@ -69,7 +54,6 @@ class VCenterEndpointForm(NetBoxModelForm):
"sync_interfaces",
"sync_ip_addresses",
"sync_primary_ips",
"sync_lxc_containers",
"update_existing",
name=_("Sync behavior"),
),
@@ -83,18 +67,11 @@ class VCenterEndpointForm(NetBoxModelForm):
"name",
"slug",
"enabled",
"provider",
"host",
"port",
"username",
"auth_method",
"token_name",
"password",
"validate_ssl",
"request_timeout_seconds",
"connect_timeout_seconds",
"max_retries",
"retry_backoff_seconds",
"tenant",
"site",
"cluster",
@@ -104,7 +81,6 @@ class VCenterEndpointForm(NetBoxModelForm):
"sync_interfaces",
"sync_ip_addresses",
"sync_primary_ips",
"sync_lxc_containers",
"update_existing",
"default_ipv4_prefix_length",
"default_ipv6_prefix_length",
@@ -126,6 +102,10 @@ class VCenterEndpointForm(NetBoxModelForm):
obj = super().save(commit=False)
password = self.cleaned_data.get("password")
obj.provider = EndpointProviderChoices.PROVIDER_VMWARE
obj.auth_method = EndpointAuthMethodChoices.METHOD_PASSWORD
obj.token_name = ""
if password:
obj.set_password(password)
netbox_vmware_importer/jobs.py
+19 -2
View File
@@ -3,7 +3,7 @@ from core.exceptions import JobFailed
from django.utils import timezone
from netbox.jobs import JobRunner, system_job
from .choices import SyncStatusChoices
from .choices import EndpointProviderChoices, SyncStatusChoices
from .models import VCenterEndpoint
from .sync import VMwareImporter
@@ -16,6 +16,10 @@ def clear_endpoint_sync_schedule(endpoint):
def schedule_endpoint_sync(endpoint):
if endpoint.provider != EndpointProviderChoices.PROVIDER_VMWARE:
clear_endpoint_sync_schedule(endpoint)
return None
if not endpoint.enabled or not endpoint.sync_interval_minutes:
clear_endpoint_sync_schedule(endpoint)
return None
@@ -45,6 +49,12 @@ class SyncVCenterEndpointJob(JobRunner):
endpoint.mark_failure("Endpoint is disabled.")
raise JobFailed("Endpoint is disabled.")
if endpoint.provider != EndpointProviderChoices.PROVIDER_VMWARE:
message = "Legacy Proxmox endpoints are no longer supported. Delete this endpoint."
clear_endpoint_sync_schedule(endpoint)
endpoint.mark_failure(message)
raise JobFailed(message)
self.logger.info("Starting VM sync for %s (%s)", endpoint.name, endpoint.host)
endpoint.mark_running()
@@ -107,8 +117,15 @@ class ScheduleDueVCenterSyncsJob(JobRunner):
def run(self, *args, **kwargs):
now = timezone.now()
legacy_endpoints = VCenterEndpoint.objects.exclude(provider=EndpointProviderChoices.PROVIDER_VMWARE)
cleared = 0
for endpoint in legacy_endpoints:
deleted_count, _ = clear_endpoint_sync_schedule(endpoint)
cleared += deleted_count
enabled_endpoints = VCenterEndpoint.objects.filter(
enabled=True,
provider=EndpointProviderChoices.PROVIDER_VMWARE,
sync_interval_minutes__isnull=False,
)
@@ -120,4 +137,4 @@ class ScheduleDueVCenterSyncsJob(JobRunner):
if schedule_endpoint_sync(endpoint):
scheduled += 1
self.logger.info("Ensured %s endpoint sync schedule(s).", scheduled)
self.logger.info("Ensured %s VMware endpoint sync schedule(s); cleared %s legacy schedule(s).", scheduled, cleared)
netbox_vmware_importer/migrations/0006_remove_proxmox_sync_support.py
+106
View File
@@ -0,0 +1,106 @@
import decimal
import django.core.validators
import django.db.models.deletion
from django.db import migrations, models
class Migration(migrations.Migration):
dependencies = [
("netbox_vmware_importer", "0005_add_proxmox_connection_tuning"),
]
operations = [
migrations.AlterModelOptions(
name="vcenterendpoint",
options={
"ordering": ("name",),
"verbose_name": "VMware endpoint",
"verbose_name_plural": "VMware endpoints",
},
),
migrations.AlterField(
model_name="vcenterendpoint",
name="provider",
field=models.CharField(
choices=[
("vmware", "VMware vSphere"),
("proxmox", "Proxmox VE (legacy)"),
],
default="vmware",
max_length=30,
),
),
migrations.AlterField(
model_name="vcenterendpoint",
name="auth_method",
field=models.CharField(
choices=[
("password", "Username/password"),
("api_token", "API token (legacy)"),
],
default="password",
max_length=30,
),
),
migrations.AlterField(
model_name="vcenterendpoint",
name="host",
field=models.CharField(help_text="vCenter or ESXi hostname or IP address", max_length=255),
),
migrations.AlterField(
model_name="vcenterendpoint",
name="validate_ssl",
field=models.BooleanField(default=False, help_text="Validate the VMware endpoint certificate."),
),
migrations.AlterField(
model_name="vcenterendpoint",
name="cluster",
field=models.ForeignKey(
help_text="NetBox cluster into which VMware VMs will be synchronized.",
on_delete=django.db.models.deletion.PROTECT,
related_name="vmware_import_endpoints",
to="virtualization.cluster",
),
),
migrations.AlterField(
model_name="vcenterendpoint",
name="token_name",
field=models.CharField(
blank=True,
help_text="Legacy Proxmox API token ID. Kept only for deleting old endpoints.",
max_length=100,
),
),
migrations.AlterField(
model_name="vcenterendpoint",
name="max_retries",
field=models.PositiveSmallIntegerField(
default=0,
help_text="Legacy connection retry setting. Kept only for compatibility with older plugin versions.",
validators=[
django.core.validators.MinValueValidator(0),
django.core.validators.MaxValueValidator(10),
],
),
),
migrations.AlterField(
model_name="vcenterendpoint",
name="retry_backoff_seconds",
field=models.DecimalField(
decimal_places=2,
default=decimal.Decimal("0.50"),
help_text="Exponential retry backoff base delay in seconds.",
max_digits=5,
validators=[
django.core.validators.MinValueValidator(decimal.Decimal("0.00")),
django.core.validators.MaxValueValidator(decimal.Decimal("60.00")),
],
),
),
migrations.AlterField(
model_name="vcenterendpoint",
name="sync_lxc_containers",
field=models.BooleanField(default=True, verbose_name="Sync legacy Proxmox LXC containers"),
),
]
netbox_vmware_importer/models.py
+16 -26
View File
@@ -34,7 +34,7 @@ class VCenterEndpoint(JobsMixin, NetBoxModel):
)
host = models.CharField(
max_length=255,
help_text=_("vCenter, ESXi, or Proxmox VE hostname or IP address"),
help_text=_("vCenter or ESXi hostname or IP address"),
)
port = models.PositiveIntegerField(
default=443,
@@ -51,7 +51,7 @@ class VCenterEndpoint(JobsMixin, NetBoxModel):
token_name = models.CharField(
max_length=100,
blank=True,
help_text=_("Proxmox API token ID, e.g. netbox-import. The token secret is stored in the password field."),
help_text=_("Legacy Proxmox API token ID. Kept only for deleting old endpoints."),
)
password_ciphertext = models.TextField(
blank=True,
@@ -59,7 +59,7 @@ class VCenterEndpoint(JobsMixin, NetBoxModel):
)
validate_ssl = models.BooleanField(
default=False,
help_text=_("Validate the endpoint TLS certificate."),
help_text=_("Validate the VMware endpoint certificate."),
)
request_timeout_seconds = models.PositiveIntegerField(
default=120,
@@ -74,7 +74,7 @@ class VCenterEndpoint(JobsMixin, NetBoxModel):
max_retries = models.PositiveSmallIntegerField(
default=0,
validators=[MinValueValidator(0), MaxValueValidator(10)],
help_text=_("Retries for safe Proxmox GET requests when the endpoint returns transient errors."),
help_text=_("Legacy connection retry setting. Kept only for compatibility with older plugin versions."),
)
retry_backoff_seconds = models.DecimalField(
max_digits=5,
@@ -101,7 +101,7 @@ class VCenterEndpoint(JobsMixin, NetBoxModel):
to="virtualization.Cluster",
on_delete=models.PROTECT,
related_name="vmware_import_endpoints",
help_text=_("NetBox cluster into which hypervisor VMs will be synchronized."),
help_text=_("NetBox cluster into which VMware VMs will be synchronized."),
)
include_name_regex = models.CharField(
max_length=255,
@@ -128,7 +128,7 @@ class VCenterEndpoint(JobsMixin, NetBoxModel):
)
sync_lxc_containers = models.BooleanField(
default=True,
verbose_name=_("Sync Proxmox LXC containers"),
verbose_name=_("Sync legacy Proxmox LXC containers"),
)
update_existing = models.BooleanField(
default=True,
@@ -196,17 +196,10 @@ class VCenterEndpoint(JobsMixin, NetBoxModel):
clone_fields = (
"enabled",
"provider",
"host",
"port",
"username",
"auth_method",
"token_name",
"validate_ssl",
"request_timeout_seconds",
"connect_timeout_seconds",
"max_retries",
"retry_backoff_seconds",
"tenant",
"site",
"cluster",
@@ -216,7 +209,6 @@ class VCenterEndpoint(JobsMixin, NetBoxModel):
"sync_interfaces",
"sync_ip_addresses",
"sync_primary_ips",
"sync_lxc_containers",
"update_existing",
"default_ipv4_prefix_length",
"default_ipv6_prefix_length",
@@ -225,8 +217,8 @@ class VCenterEndpoint(JobsMixin, NetBoxModel):
class Meta:
ordering = ("name",)
verbose_name = _("virtualization endpoint")
verbose_name_plural = _("virtualization endpoints")
verbose_name = _("VMware endpoint")
verbose_name_plural = _("VMware endpoints")
def __str__(self):
return self.name
@@ -257,15 +249,13 @@ class VCenterEndpoint(JobsMixin, NetBoxModel):
if self.cluster.site_id != self.site_id:
raise ValidationError({"cluster": _("Selected cluster belongs to a different site.")})
if self.provider == EndpointProviderChoices.PROVIDER_VMWARE:
if self.auth_method != EndpointAuthMethodChoices.METHOD_PASSWORD:
raise ValidationError({"auth_method": _("VMware endpoints currently support username/password authentication only.")})
if self.provider != EndpointProviderChoices.PROVIDER_VMWARE:
raise ValidationError(
{"provider": _("Proxmox endpoints are legacy records and can only be deleted.")}
)
if self.provider == EndpointProviderChoices.PROVIDER_PROXMOX:
if "@" not in self.username:
raise ValidationError({"username": _("Proxmox usernames must include a realm, e.g. root@pam or user@pve.")})
if self.auth_method == EndpointAuthMethodChoices.METHOD_API_TOKEN and not self.token_name:
raise ValidationError({"token_name": _("A token name is required for Proxmox API token authentication.")})
if self.auth_method != EndpointAuthMethodChoices.METHOD_PASSWORD:
raise ValidationError({"auth_method": _("VMware endpoints support username/password authentication only.")})
def mark_queued(self):
self.last_status = SyncStatusChoices.STATUS_QUEUED
@@ -315,13 +305,13 @@ class VCenterEndpoint(JobsMixin, NetBoxModel):
self.save(update_fields=("last_sync_at", "last_status", "last_message", "next_sync_at", "last_updated"))
def set_next_sync(self, now=None):
if self.enabled and self.sync_interval_minutes:
if self.provider == EndpointProviderChoices.PROVIDER_VMWARE and self.enabled and self.sync_interval_minutes:
self.next_sync_at = (now or timezone.now()) + timedelta(minutes=self.sync_interval_minutes)
else:
self.next_sync_at = None
def save(self, *args, **kwargs):
if not self.enabled or not self.sync_interval_minutes:
if self.provider != EndpointProviderChoices.PROVIDER_VMWARE or not self.enabled or not self.sync_interval_minutes:
self.next_sync_at = None
elif not self.next_sync_at:
self.set_next_sync()
netbox_vmware_importer/navigation.py
+2 -2
View File
@@ -18,9 +18,9 @@ endpoint_item = PluginMenuItem(
)
menu = PluginMenu(
label="VM Import",
label="VMware Import",
groups=(
("Providers", (endpoint_item,)),
("VMware", (endpoint_item,)),
),
icon_class="mdi mdi-cloud-sync",
)
netbox_vmware_importer/signals.py
+1 -1
View File
@@ -6,7 +6,7 @@ from .jobs import clear_endpoint_sync_schedule, schedule_endpoint_sync
from .models import VCenterEndpoint
AUTOMATIC_SYNC_FIELDS = {"enabled", "sync_interval_minutes"}
AUTOMATIC_SYNC_FIELDS = {"enabled", "provider", "sync_interval_minutes"}
@receiver(post_save, sender=VCenterEndpoint)
netbox_vmware_importer/sync.py
+6 -637
View File
@@ -1,9 +1,7 @@
import ipaddress
import re
import ssl
import time
from dataclasses import dataclass, field
from urllib.parse import urlsplit
from django.contrib.contenttypes.models import ContentType
from django.db import transaction
@@ -12,29 +10,7 @@ from dcim.models import MACAddress, Platform
from ipam.models import IPAddress
from virtualization.models import VMInterface, VirtualDisk, VirtualMachine
from .choices import EndpointAuthMethodChoices, EndpointProviderChoices
try:
import requests
from requests.adapters import HTTPAdapter
from urllib3.exceptions import InsecureRequestWarning
except ImportError: # pragma: no cover - handled at runtime inside NetBox
HTTPAdapter = None
requests = None
InsecureRequestWarning = None
try:
from proxmox_sdk import ProxmoxSDK
from proxmox_sdk.sdk.exceptions import (
ProxmoxConnectionError as SDKConnectionError,
ProxmoxTimeoutError as SDKTimeoutError,
ResourceException as SDKResourceException,
)
except ImportError: # pragma: no cover - optional compatibility fallback
ProxmoxSDK = None
SDKConnectionError = None
SDKTimeoutError = None
SDKResourceException = None
from .choices import EndpointProviderChoices
try:
from pyVim.connect import Disconnect, SmartConnect
@@ -96,34 +72,6 @@ class VMwareConnectionError(RuntimeError):
pass
class ProxmoxConnectionError(RuntimeError):
pass
class ProxmoxTLSCompatibilityAdapter(HTTPAdapter):
def __init__(self, validate_ssl=True, *args, **kwargs):
self.validate_ssl = validate_ssl
super().__init__(*args, **kwargs)
def init_poolmanager(self, connections, maxsize, block=False, **pool_kwargs):
pool_kwargs["ssl_context"] = self._build_ssl_context()
return super().init_poolmanager(connections, maxsize, block=block, **pool_kwargs)
def proxy_manager_for(self, *args, **kwargs):
kwargs["ssl_context"] = self._build_ssl_context()
return super().proxy_manager_for(*args, **kwargs)
def _build_ssl_context(self):
context = ssl.create_default_context()
if hasattr(ssl, "TLSVersion"):
context.minimum_version = ssl.TLSVersion.TLSv1_2
context.maximum_version = ssl.TLSVersion.TLSv1_2
if not self.validate_ssl:
context.check_hostname = False
context.verify_mode = ssl.CERT_NONE
return context
class VMwareClient:
def __init__(self, endpoint):
self.endpoint = endpoint
@@ -264,605 +212,26 @@ class VMwareClient:
return interfaces
class ProxmoxClient:
DISK_KEY_PATTERN = re.compile(r"^(ide|sata|scsi|virtio)\d+$")
LXC_DISK_KEY_PATTERN = re.compile(r"^(rootfs|mp\d+)$")
NET_KEY_PATTERN = re.compile(r"^net\d+$")
FULL_TOKEN_PATTERN = re.compile(r"^(?:PVEAPIToken=)?(?P<user>[^!]+)!(?P<name>[^=]+)=(?P<value>.+)$")
RETRY_STATUSES = {502, 503, 504}
def __init__(self, endpoint):
self.endpoint = endpoint
self.host, self.port = self._parse_endpoint_address(endpoint.host, endpoint.port)
self.url_host = self._format_host_for_url(self.host)
self.session = None
self.sdk = None
self.uses_sdk = False
self.base_url = f"https://{self.url_host}:{self.port}/api2/json"
self.connect_timeout = endpoint.connect_timeout_seconds or 10
self.read_timeout = endpoint.request_timeout_seconds or 120
self.timeout = (self.connect_timeout, self.read_timeout)
self.max_retries = endpoint.max_retries or 0
self.retry_backoff = float(endpoint.retry_backoff_seconds or 0.5)
self.api_probe_succeeded = False
self.api_probe_error = ""
def __enter__(self):
if "@" not in self.endpoint.username:
raise ProxmoxConnectionError("Proxmox username must include a realm, e.g. root@pam or user@pve.")
if self._should_use_sdk():
self._connect_with_sdk()
return self
self._connect_with_requests()
return self
def __exit__(self, exc_type, exc_value, traceback):
if self.sdk is not None:
self.sdk.close()
if self.session is not None:
self.session.close()
@staticmethod
def _parse_endpoint_address(host, port):
raw_host = str(host or "").strip()
resolved_port = int(port or 8006)
if not raw_host:
return raw_host, resolved_port
parsed = urlsplit(raw_host if "://" in raw_host else f"//{raw_host}")
parsed_host = parsed.hostname
if parsed.port:
resolved_port = parsed.port
if parsed_host:
return parsed_host.strip("[]"), resolved_port
return raw_host.strip("[]"), resolved_port
@staticmethod
def _format_host_for_url(host):
if ":" in host and not host.startswith("["):
return f"[{host}]"
return host
def _should_use_sdk(self):
# proxmox-sdk 0.0.12 builds password ticket URLs without /api2/json.
# Keep it for stateless API-token auth, where no ticket endpoint is used.
return (
ProxmoxSDK is not None
and self.endpoint.auth_method == EndpointAuthMethodChoices.METHOD_API_TOKEN
)
def _connect_with_sdk(self):
user = (self.endpoint.username or "").strip()
kwargs = {
"host": self.host,
"backend": "https",
"service": "PVE",
"port": self.port,
"user": user,
"verify_ssl": self.endpoint.validate_ssl,
"timeout": self.read_timeout,
"connect_timeout": self.connect_timeout,
"max_retries": self.max_retries,
"retry_backoff": self.retry_backoff,
}
if self.endpoint.auth_method == EndpointAuthMethodChoices.METHOD_API_TOKEN:
user, token_name, token_value = self._normalized_token_credentials()
kwargs["user"] = user
kwargs.update(
{
"token_name": token_name,
"token_value": token_value,
}
)
else:
kwargs["password"] = self.endpoint.password
try:
self.sdk = ProxmoxSDK.sync(**kwargs)
self.uses_sdk = True
self._get("version")
self.api_probe_succeeded = True
except ProxmoxConnectionError:
if self.sdk is not None:
self.sdk.close()
self.sdk = None
raise
except Exception as exc:
if self.sdk is not None:
self.sdk.close()
self.sdk = None
raise ProxmoxConnectionError(self._describe_sdk_error("GET", "version", exc)) from exc
def _connect_with_requests(self):
if requests is None:
raise ProxmoxConnectionError(
"Neither proxmox-sdk nor requests is installed in the NetBox Python environment."
)
self.session = requests.Session()
self.session.trust_env = False
self.session.verify = self.endpoint.validate_ssl
if HTTPAdapter is not None:
self.session.mount("https://", ProxmoxTLSCompatibilityAdapter(validate_ssl=self.endpoint.validate_ssl))
self.session.headers.update(
{
"Accept": "application/json",
"Connection": "close",
"User-Agent": "netbox-vm-import/0.2",
}
)
if not self.endpoint.validate_ssl and InsecureRequestWarning is not None:
requests.packages.urllib3.disable_warnings(category=InsecureRequestWarning)
self._probe_api_availability()
if self.endpoint.auth_method == EndpointAuthMethodChoices.METHOD_API_TOKEN:
user, token_name, token_value = self._normalized_token_credentials()
self.session.headers.update({"Authorization": f"PVEAPIToken={user}!{token_name}={token_value}"})
else:
self._authenticate_with_password()
def _probe_api_availability(self):
try:
self._request("GET", "version", timeout=(5, 10))
self.api_probe_succeeded = True
self.api_probe_error = ""
except ProxmoxConnectionError as exc:
self.api_probe_succeeded = False
self.api_probe_error = str(exc)
def _normalized_token_credentials(self):
user = (self.endpoint.username or "").strip()
token_name = (self.endpoint.token_name or "").strip()
token_value = (self.endpoint.password or "").strip()
parsed = self._parse_full_token(token_value)
if parsed is not None:
user, token_name, token_value = parsed
else:
parsed = self._parse_full_token(token_name)
if parsed is not None:
parsed_user, parsed_name, parsed_value = parsed
user = parsed_user or user
token_name = parsed_name
if parsed_value and not token_value:
token_value = parsed_value
elif token_name.startswith("PVEAPIToken="):
token_name = token_name.removeprefix("PVEAPIToken=").strip()
if "!" in token_name:
parsed_user, parsed_name = token_name.rsplit("!", 1)
if parsed_user:
user = parsed_user
token_name = parsed_name
if not token_name or not token_value:
raise ProxmoxConnectionError(
"Proxmox API token authentication requires a token name and token secret."
)
return user, token_name, token_value
@classmethod
def _parse_full_token(cls, value):
match = cls.FULL_TOKEN_PATTERN.match(str(value or "").strip())
if not match:
return None
return match.group("user").strip(), match.group("name").strip(), match.group("value").strip()
def _authenticate_with_password(self):
response = self._request(
"POST",
"access/ticket",
data={
"username": self.endpoint.username,
"password": self.endpoint.password,
},
)
data = response.json().get("data") or {}
ticket = data.get("ticket")
csrf_token = data.get("CSRFPreventionToken")
if not ticket:
raise ProxmoxConnectionError("Proxmox authentication did not return a ticket.")
self.session.cookies.set("PVEAuthCookie", ticket)
if csrf_token:
self.session.headers.update({"CSRFPreventionToken": csrf_token})
def _url(self, path):
return f"{self.base_url}/{path.strip('/')}"
def _request(self, method, path, timeout=None, **kwargs):
url = self._url(path)
request_timeout = timeout or self.timeout
attempts = self.max_retries + 1 if method.upper() == "GET" else 1
last_error = None
for attempt in range(attempts):
if attempt:
time.sleep(min(self.retry_backoff * (2 ** (attempt - 1)), 30.0))
try:
response = self.session.request(method, url, timeout=request_timeout, **kwargs)
if response.status_code in self.RETRY_STATUSES and attempt < attempts - 1:
last_error = ProxmoxConnectionError(
f"Proxmox API {method} {path} returned transient HTTP {response.status_code}."
)
continue
response.raise_for_status()
return response
except requests.ConnectTimeout as exc:
last_error = ProxmoxConnectionError(
f"Proxmox API {method} {path} connect timed out after {request_timeout[0]} seconds."
)
if attempt >= attempts - 1:
raise last_error from exc
except requests.ReadTimeout as exc:
detail = f"Proxmox API {method} {path} read timed out after {request_timeout[1]} seconds."
if path == "access/ticket" and self.api_probe_succeeded:
detail += " API version endpoint responded, so the delay is likely in Proxmox authentication. Check username realm (e.g. root@pam), PAM/LDAP auth, two-factor auth, or use an API token."
elif getattr(self, "api_probe_error", "") and path != "version":
detail += f" Earlier unauthenticated version probe also failed: {self.api_probe_error}"
last_error = ProxmoxConnectionError(detail)
if attempt >= attempts - 1:
raise last_error from exc
except requests.exceptions.SSLError as exc:
raise ProxmoxConnectionError(
f"Proxmox API {method} {path} TLS validation failed. Disable Validate SSL for self-signed certificates or install the CA certificate. Details: {exc}"
) from exc
except requests.RequestException as exc:
last_error = ProxmoxConnectionError(f"Proxmox API {method} {path} failed: {exc}")
if attempt >= attempts - 1:
raise last_error from exc
if last_error is not None:
raise last_error
raise ProxmoxConnectionError(f"Proxmox API {method} {path} failed without a response.")
def _sdk_get(self, path, params=None):
try:
return self.sdk(path.strip("/")).get(**(params or {}))
except Exception as exc:
raise ProxmoxConnectionError(self._describe_sdk_error("GET", path, exc)) from exc
def _describe_sdk_error(self, method, path, exc):
if SDKTimeoutError is not None and isinstance(exc, SDKTimeoutError):
return f"Proxmox SDK {method} {path} timed out after {self.read_timeout} seconds."
if SDKConnectionError is not None and isinstance(exc, SDKConnectionError):
detail = str(exc).strip() or "connection failed"
return f"Proxmox SDK {method} {path} connection failed: {detail}"
if SDKResourceException is not None and isinstance(exc, SDKResourceException):
status_code = getattr(exc, "status_code", None)
status_message = getattr(exc, "status_message", "") or ""
content = (getattr(exc, "content", "") or "").strip()
errors = getattr(exc, "errors", None)
parts = []
if status_code:
parts.append(f"HTTP {status_code} {status_message}".strip())
elif status_message:
parts.append(status_message)
if content:
parts.append(content)
if errors:
parts.append(str(errors))
detail = " - ".join(parts) if parts else str(exc)
return f"Proxmox SDK {method} {path} failed: {detail}"
detail = str(exc).strip() or "unknown error"
return f"Proxmox SDK {method} {path} failed: {detail}"
def _get(self, path, params=None):
if self.uses_sdk:
return self._sdk_get(path, params=params)
response = self._request("GET", path, params=params)
return response.json().get("data")
def iter_virtual_machines(self):
resources = self._get("cluster/resources", params={"type": "vm"}) or []
for resource in resources:
vm_type = resource.get("type")
if vm_type == "lxc" and not self.endpoint.sync_lxc_containers:
continue
if vm_type not in ("qemu", "lxc"):
continue
yield self._build_vm_data(resource)
def _build_vm_data(self, resource):
vmid = resource["vmid"]
node = resource["node"]
vm_type = resource["type"]
config = self._get(f"nodes/{node}/{vm_type}/{vmid}/config") or {}
name = resource.get("name") or config.get("name") or config.get("hostname") or f"{vm_type}-{vmid}"
status = "active" if resource.get("status") == "running" else "offline"
memory_mb = self._get_memory_mb(resource, config)
vcpus = self._get_vcpus(resource, config)
if vm_type == "qemu":
disks = self._get_qemu_disks(config)
interfaces = self._get_qemu_interfaces(config)
self._merge_qemu_agent_interfaces(node, vmid, interfaces)
guest_os = self._get_qemu_guest_os(node, vmid)
else:
disks = self._get_lxc_disks(config)
interfaces = self._get_lxc_interfaces(config)
guest_os = "Linux Container"
disk_mb = sum(disk.size_mb for disk in disks) or self._bytes_to_mb(resource.get("maxdisk"))
return VMData(
name=str(name),
status=status,
vcpus=vcpus,
memory_mb=memory_mb,
disk_mb=disk_mb,
guest_os=guest_os,
disks=disks,
interfaces=interfaces,
)
@staticmethod
def _get_memory_mb(resource, config):
memory = config.get("memory")
if memory not in (None, ""):
try:
return int(memory)
except (TypeError, ValueError):
pass
return ProxmoxClient._bytes_to_mb(resource.get("maxmem"))
@staticmethod
def _get_vcpus(resource, config):
try:
sockets = int(config.get("sockets") or 1)
cores = int(config.get("cores") or resource.get("maxcpu") or 0)
return max(sockets * cores, 0)
except (TypeError, ValueError):
return int(resource.get("maxcpu") or 0)
@staticmethod
def _bytes_to_mb(value):
try:
return round(int(value or 0) / 1024 / 1024)
except (TypeError, ValueError):
return 0
def _get_qemu_disks(self, config):
disks = []
for key, value in sorted(config.items()):
if not self.DISK_KEY_PATTERN.match(key):
continue
parsed = self._parse_proxmox_value(value)
if parsed.get("media") == "cdrom":
continue
size_mb = self._parse_size_mb(parsed.get("size"))
if size_mb <= 0:
continue
disks.append(
DiskData(
name=key,
size_mb=size_mb,
description=f"Proxmox: {str(value)[:191]}",
)
)
return disks
def _get_lxc_disks(self, config):
disks = []
for key, value in sorted(config.items()):
if not self.LXC_DISK_KEY_PATTERN.match(key):
continue
parsed = self._parse_proxmox_value(value)
size_mb = self._parse_size_mb(parsed.get("size"))
if size_mb <= 0:
continue
disks.append(
DiskData(
name=key,
size_mb=size_mb,
description=f"Proxmox: {str(value)[:191]}",
)
)
return disks
def _get_qemu_interfaces(self, config):
interfaces = []
for key, value in sorted(config.items()):
if not self.NET_KEY_PATTERN.match(key):
continue
parsed = self._parse_proxmox_value(value)
mac_address = self._extract_proxmox_mac(parsed)
if not mac_address:
continue
bridge = parsed.get("bridge")
name = f"{key} ({bridge})" if bridge else key
interfaces.append(InterfaceData(name=name[:64], mac_address=mac_address))
return interfaces
def _get_lxc_interfaces(self, config):
interfaces = []
for key, value in sorted(config.items()):
if not self.NET_KEY_PATTERN.match(key):
continue
parsed = self._parse_proxmox_value(value)
mac_address = parsed.get("hwaddr")
if not mac_address:
continue
name = parsed.get("name") or key
ip_addresses = []
for ip_key in ("ip", "ip6"):
ip_value = parsed.get(ip_key)
if ip_value and ip_value not in ("dhcp", "manual", "auto"):
ip_addresses.append(ip_value)
interfaces.append(
InterfaceData(
name=str(name)[:64],
mac_address=mac_address,
ip_addresses=ip_addresses,
)
)
return interfaces
def _merge_qemu_agent_interfaces(self, node, vmid, interfaces):
try:
response = self._get(f"nodes/{node}/qemu/{vmid}/agent/network-get-interfaces") or {}
except (requests.RequestException, ProxmoxConnectionError):
return
agent_interfaces = response.get("result") if isinstance(response, dict) else response
if not isinstance(agent_interfaces, list):
return
interfaces_by_mac = {self._normalize_mac(interface.mac_address): interface for interface in interfaces}
for agent_interface in agent_interfaces:
mac_address = agent_interface.get("hardware-address")
normalized_mac = self._normalize_mac(mac_address)
if not normalized_mac:
continue
ip_addresses = []
for ip_data in agent_interface.get("ip-addresses") or []:
ip_address = ip_data.get("ip-address")
prefix = ip_data.get("prefix")
if not ip_address:
continue
if prefix is not None:
ip_addresses.append(f"{ip_address}/{prefix}")
else:
ip_addresses.append(ip_address)
if normalized_mac in interfaces_by_mac:
interfaces_by_mac[normalized_mac].ip_addresses = ip_addresses
else:
interfaces.append(
InterfaceData(
name=str(agent_interface.get("name") or f"NIC-{normalized_mac[-5:]}")[:64],
mac_address=normalized_mac,
ip_addresses=ip_addresses,
)
)
def _get_qemu_guest_os(self, node, vmid):
try:
response = self._get(f"nodes/{node}/qemu/{vmid}/agent/get-osinfo") or {}
except (requests.RequestException, ProxmoxConnectionError):
return ""
result = response.get("result") if isinstance(response, dict) else None
if not isinstance(result, dict):
return ""
return result.get("pretty-name") or result.get("name") or ""
@staticmethod
def _parse_proxmox_value(value):
parsed = {"_raw": str(value)}
parts = str(value).split(",")
if parts:
parsed["_first"] = parts[0]
for part in parts:
if "=" in part:
key, part_value = part.split("=", 1)
parsed[key.strip()] = part_value.strip()
return parsed
@staticmethod
def _extract_proxmox_mac(parsed):
for key in ("virtio", "e1000", "e1000e", "rtl8139", "vmxnet3", "hwaddr"):
if parsed.get(key):
return parsed[key]
return None
@staticmethod
def _parse_size_mb(value):
if value in (None, ""):
return 0
match = re.match(r"^\s*([0-9.]+)\s*([KMGTPE]?)(?:i?B)?\s*$", str(value), re.IGNORECASE)
if not match:
return 0
amount = float(match.group(1))
unit = match.group(2).upper()
factors = {
"": 1,
"K": 1 / 1024,
"M": 1,
"G": 1024,
"T": 1024 * 1024,
"P": 1024 * 1024 * 1024,
"E": 1024 * 1024 * 1024 * 1024,
}
if not unit and amount > 1024 * 1024:
return round(amount / 1024 / 1024)
return round(amount * factors.get(unit, 1))
@staticmethod
def _normalize_mac(mac_address):
if not mac_address:
return None
value = str(mac_address).strip().replace("-", ":").lower()
parts = value.split(":")
if len(parts) != 6:
return None
try:
return ":".join(f"{int(part, 16):02x}" for part in parts)
except ValueError:
return None
class VMwareImporter:
def __init__(self, endpoint, logger):
self.endpoint = endpoint
self.logger = logger
self.include_pattern = re.compile(endpoint.include_name_regex) if endpoint.include_name_regex else None
self.exclude_pattern = re.compile(endpoint.exclude_name_regex) if endpoint.exclude_name_regex else None
self.provider_name = "Proxmox" if endpoint.provider == EndpointProviderChoices.PROVIDER_PROXMOX else "VMware"
self.provider_name = "VMware"
self.interface_content_type = None
self.mac_address_content_type = None
def sync(self):
if self.endpoint.provider != EndpointProviderChoices.PROVIDER_VMWARE:
raise VMwareConnectionError("Legacy Proxmox endpoints are no longer supported. Delete this endpoint.")
result = SyncResult()
platforms = list(Platform.objects.all())
self.interface_content_type = ContentType.objects.get_for_model(VMInterface)
self.mac_address_content_type = ContentType.objects.get_for_model(VMInterface)
client_class = ProxmoxClient if self.endpoint.provider == EndpointProviderChoices.PROVIDER_PROXMOX else VMwareClient
with client_class(self.endpoint) as client:
with VMwareClient(self.endpoint) as client:
for vm_data in client.iter_virtual_machines():
result.seen += 1
netbox_vmware_importer/templates/netbox_vmware_importer/vcenterendpoint.html
+8 -22
View File
@@ -1,6 +1,12 @@
{% extends 'generic/object.html' %}
{% block content %}
{% if object.provider != "vmware" %}
<div class="alert alert-warning" role="alert">
This is a legacy Proxmox endpoint from an older plugin version. Proxmox sync is no longer supported; this endpoint can be deleted.
</div>
{% endif %}
<div class="row mb-3">
<div class="col col-md-6">
<div class="card">
@@ -22,16 +28,6 @@
<th scope="row">Username</th>
<td>{{ object.username }}</td>
</tr>
<tr>
<th scope="row">Auth method</th>
<td>{{ object.get_auth_method_display }}</td>
</tr>
{% if object.token_name %}
<tr>
<th scope="row">Token name</th>
<td>{{ object.token_name }}</td>
</tr>
{% endif %}
<tr>
<th scope="row">Enabled</th>
<td>{{ object.enabled|yesno:"Yes,No" }}</td>
@@ -40,18 +36,6 @@
<th scope="row">Validate SSL</th>
<td>{{ object.validate_ssl|yesno:"Yes,No" }}</td>
</tr>
<tr>
<th scope="row">API timeout</th>
<td>{{ object.request_timeout_seconds }} seconds</td>
</tr>
<tr>
<th scope="row">Connect timeout</th>
<td>{{ object.connect_timeout_seconds }} seconds</td>
</tr>
<tr>
<th scope="row">GET retries</th>
<td>{{ object.max_retries }} retries, {{ object.retry_backoff_seconds }}s backoff</td>
</tr>
</table>
</div>
</div>
@@ -110,6 +94,7 @@
<td>{{ object.last_message|default:"-" }}</td>
</tr>
</table>
{% if object.provider == "vmware" %}
{% if perms.netbox_vmware_importer.change_vcenterendpoint %}
<div class="card-footer">
<form method="post" action="{% url 'plugins:netbox_vmware_importer:vcenterendpoint_sync' pk=object.pk %}">
@@ -120,6 +105,7 @@
</form>
</div>
{% endif %}
{% endif %}
</div>
</div>
<div class="col col-md-6">
netbox_vmware_importer/views.py
+6 -1
View File
@@ -8,6 +8,7 @@ from netbox.views import generic
from .filtersets import VCenterEndpointFilterSet
from .forms import VCenterEndpointFilterForm, VCenterEndpointForm
from .jobs import SyncVCenterEndpointJob
from .choices import EndpointProviderChoices
from .models import VCenterEndpoint
from .tables import VCenterEndpointTable
@@ -29,7 +30,7 @@ class VCenterEndpointView(generic.ObjectView):
class VCenterEndpointEditView(generic.ObjectEditView):
queryset = VCenterEndpoint.objects.all()
queryset = VCenterEndpoint.objects.filter(provider=EndpointProviderChoices.PROVIDER_VMWARE)
form = VCenterEndpointForm
@@ -44,6 +45,10 @@ class VCenterEndpointSyncView(View):
if not request.user.has_perm("netbox_vmware_importer.change_vcenterendpoint"):
return HttpResponseForbidden()
if endpoint.provider != EndpointProviderChoices.PROVIDER_VMWARE:
messages.error(request, _("Legacy Proxmox endpoints can only be deleted."))
return redirect(endpoint)
SyncVCenterEndpointJob.enqueue(instance=endpoint)
endpoint.mark_queued()
messages.success(request, _("VM sync job queued for %(endpoint)s.") % {"endpoint": endpoint})
pyproject.toml
+2 -4
View File
@@ -4,8 +4,8 @@ build-backend = "setuptools.build_meta"
[project]
name = "netbox-vmware-importer"
version = "0.2.7"
description = "NetBox plugin to synchronize VMware vSphere and Proxmox VE virtual machines into NetBox."
version = "0.3.0"
description = "NetBox plugin to synchronize VMware vSphere virtual machines into NetBox."
readme = "README.md"
requires-python = ">=3.12"
authors = [
@@ -22,8 +22,6 @@ classifiers = [
dependencies = [
"pyvmomi>=8.0.3",
"cryptography>=42.0",
"requests>=2.32",
"proxmox-sdk==0.0.12",
]
[tool.setuptools.packages.find]