Technical Breakdown

Project Overview: Nested Virtualization & Automating IaaC

In highly automated DevOps and cloud platform environments, simulating target architectures before spinning up expensive cloud clusters is a critical design challenge. Testing clustering, networking protocols, and systems recovery models requires a local, self-contained development server that can emulate multiple distinct systems.

Nested Sandbox is an engineering implementation that resolves this challenge by enabling nested virtualization (VMs running inside another VM).

Deployed as a primary virtual machine inside Proxmox VE (Ubuntu 24.04 LTS) with CPU hardware virtualization passthrough enabled, this server acts as an independent virtualization host. It leverages Vagrant paired with the Libvirt (KVM/QEMU) provider to automatically declare, provision, and cluster a highly available, 3-node HashiCorp Consul cluster running within completely isolated nested network boundaries.

🗺️ Nested Virtualization Architecture

The system utilizes three distinct layers of hypervisor abstraction, passing hardware instructions seamlessly from bare-metal down to the inner guest kernels:

🛡️ Clustered Service Matrix & Deep Pros & Cons

To validate the nested virtualization engine, we deploy a highly available HashiCorp Consul service mesh inside the level-2 nested VMs. Below is a deep structural breakdown of this architecture and its system trade-offs:

1. Nested KVM Hypervisor passthrough (Intel vmx / AMD svm)

• Role: Exposes bare-metal CPU hardware acceleration flags from the host hypervisor (Proxmox) straight into the guest kernel (Ubuntu 24.04), enabling native nested VM performance.
• Pros:
  • Near-Native Performance: Uses hardware-managed page tables to execute nested instructions without software translation penalties.
  • Dev/Prod Parity: Enables engineers to run and test bare-metal cloud orchestrators (K3s, OpenStack) inside local sandboxes.
• Cons:
  • Live Migration Limitations: Enabling Host CPU passthrough prevents Proxmox from live-migrating the Level-1 VM to a physical cluster node running a different CPU model.
  • Strict BIOS Dependency: Requires VT-x/AMD-V to be explicitly enabled in the physical host BIOS.

2. High-Performance Virtualization Provider (Vagrant + Libvirt)

• Role: Replaces standard VirtualBox with Libvirt (Linux KVM) inside the nested host to gain extreme performance improvements and native integration with the Linux kernel.
• Pros:
  • Dynamic RAM Management: Features memory ballooning, freeing memory when guest VMs are idle.
  • Extreme I/O Throughput: Uses native KVM/QEMU drivers to read and write directly to physical block devices.
• Cons:
  • Configuration Overhead: Setting up Libvirt networking and bridges requires running commands via the Linux terminal, compared to VirtualBox GUI setups.

3. Highly Available Distributed Consensus (Consul HA Cluster)

• Role: Coordinates service discovery, checks, and configurations across the nested cluster using the Raft consensus algorithm.
• Pros:
  • Zero-Trust Identity: Distributes short-lived access tokens dynamically across the network.
  • Automated Failover: Maintains operations even if a server node crashes, electing a new leader inside 150ms.
• Cons:
  • Quorum Requirements: Demands a strict odd number of server nodes (minimum 3) to prevent "split-brain" routing lockups.

💻 Infrastructure-as-Code Configuration Blueprint

A. Vagrantfile: Declaring the 3-Node Nested Consul Cluster

Below is the highly optimized, declarative configuration mapping our private network bridge, CPU, and automated provisioning structures:

# Vagrantfile - Multi-VM Libvirt configuration
Vagrant.configure("2") do |config|
  # Define default Ubuntu 24.04 Noble Numbat Box
  config.vm.box = "generic/ubuntu2404"

  # Configure High-Performance KVM/Libvirt Provider
  config.vm.provider :libvirt do |libvirt|
    libvirt.driver = "kvm"
    libvirt.connect_uri = "qemu:///system"
    libvirt.cpus = 1
    libvirt.memory = 1024
    libvirt.nested = true # Enables Level-3 nesting capabilities inside inner VMs
  end

  # Consul Server 01
  config.vm.define "consul-01" do |c|
    c.vm.hostname = "consul-01"
    c.vm.network "private_network", ip: "10.10.10.11", libvirt__forward_mode: "nat"
    c.vm.provision "shell", path: "scripts/bootstrap.sh", args: ["consul-01", "10.10.10.11", "bootstrap"]
  end

  # Consul Server 02
  config.vm.define "consul-02" do |c|
    c.vm.hostname = "consul-02"
    c.vm.network "private_network", ip: "10.10.10.12", libvirt__forward_mode: "nat"
    c.vm.provision "shell", path: "scripts/bootstrap.sh", args: ["consul-02", "10.10.10.12", "join"]
  end

  # Consul Server 03
  config.vm.define "consul-03" do |c|
    c.vm.hostname = "consul-03"
    c.vm.network "private_network", ip: "10.10.10.13", libvirt__forward_mode: "nat"
    c.vm.provision "shell", path: "scripts/bootstrap.sh", args: ["consul-03", "10.10.10.13", "join"]
  end
end

📈 System Real-Time Performance & Quorum Scaling

Testing high availability is completely self-contained. The nested infrastructure runs smoothly on a single Proxmox VM with 4 CPU Cores and 6 GB of RAM, scaling resources dynamically:

Proxmox Host CPU Core Allocation:
[Core 0] ===> [ Ubuntu 24.04 L1 Host ] ───> [ Libvirt Scheduler ]
[Core 1] ===>                                  ├──> [ Consul Node 01 ] (1 Core, 1GB RAM)
[Core 2] ===>                                  ├──> [ Consul Node 02 ] (1 Core, 1GB RAM)
[Core 3] ===>                                  └──> [ Consul Node 03 ] (1 Core, 1GB RAM)

💡 Key Architectural Lessons Learned

1. Enforce Nested VMX Flags: Nested virtualization fails silently if the Level-1 VM CPU model in Proxmox is set to Default (kvm64). You must configure the CPU type to Host to expose the direct hardware virtualization layers.
2. Standardize MTU Boundaries: Libvirt's default bridge interfaces (virbr0) apply an MTU of 1500. Ensure your outer Proxmox bridges are fully synchronized to avoid packet dropouts under heavy data transfers.
3. Bootstrap Quorum Carefully: When clustering Consul, node consul-01 starts in a wait state. Nodes 02 and 03 must explicitly refer to the bootstrap node's IP to execute the Raft join cluster state cleanly.