Talks

Proposal Description

• Proxmox 9 was recently released and brings in a lot of features that might cater to the advanced. However, with this new release, they bring an end to one of my biggest complaints to Proxmox: the mobile UI - it simply sucked: both the mobile web and mobile app. The latter sees no improvement but the former has seen a lot of improvements, all of which combined warrant a fresh look at the product.
  
• This presentation will go over the following topics:
  • My journey with Proxmox - I went from being completely pro-baremetal to pro-virtualisation in the span of 2 years where many things increasingly convinced me that the latter approach suits a simultaneous project work-style far better. Even kubernetes recommends running “workers” as VMs!
  • What’s new with Proxmox 9 - ZFS improvements, new HA and SDN sauce, VM snapshots on thick-LVM on iSCSI drives
  • SImple ways to get started with Proxmox as a complete noob - Often the easiest way is to start small. All the usual switch-to-Linux rules apply: use a spare laptop, spare PC, dual boot, etc.
  • Usecases - Homelab, Enterprise
  • How virtualising your projects gives you infinite flexibility in terms of undo → redo → experiment, all while running actual production services - be it off of a minipc on top of a fridge or a rack full of AMD EPYC servers in a Datacenter in Gurgaon. 

Key Takeaways from this talk

• Familiarity with Proxmox, especially 9.0.
• Head-start to self-hosting services with the ability to make infinite mistakes and then recover with no trouble. All of this is on a separate host so no interference with the personal/work machine.
• Head-start to concepts like virtualisation, high-availability, HA groups, fencing, shared-storage: all of which obviously makes sense in an enterprise setup but might also make sense in a homelab depending on how seriously the audience takes their setup.

Proposal

Introduction

You’ve already learned about the basics of the shell, and now it’s time to make it practical. This presentation will contain lots of tips and tricks from the book, “Learn Bash the Hard Way.” We’ll explore Bash—a powerful shell you’ll encounter frequently in Linux. Rather than just memorizing commands, we’ll focus on learning by doing, making sure you feel confident experimenting on your own.

What Exactly is Bash?

Bash is simply the tool that translates your commands into actions the system understands. You type instructions, Bash executes them. We’ll briefly touch on how Bash compares to other shells like sh or zsh, but our main goal is to become familiar with the everyday use of Bash itself.

Working with Files and Commands

One of the first steps to mastering Bash is managing files effectively. We’ll discuss simple yet powerful techniques like using wildcards to quickly find or manage groups of files. You’ll also learn how to handle hidden files and comfortably move around the Linux file system without confusion.

Variables—Making Bash Do More

When commands become repetitive, variables are your friends. Think of them as containers that hold useful information. You’ll learn how to create variables, assign values, and use them efficiently. We’ll clear up common confusion around single and double quotes, making your Bash experience less frustrating.

Pipes, Redirects, and Command Chaining

Let’s simplify how you handle command outputs. Using pipes, you can take the output from one command and pass it straight into another. Redirects let you save outputs directly into files or discard unnecessary errors. These simple techniques dramatically speed up your work.

Writing Your First Scripts

Scripts help automate repeated tasks, saving you time and effort. Together, we’ll walk through creating basic scripts, setting them up to run smoothly, and understanding their structure. You’ll quickly see how scripts can make your daily Linux tasks easier and more organized.

Success, Failure, and Making Decisions

Every command you run in Bash gives feedback on whether it succeeded or not. Understanding these signals—called exit codes—allows you to write scripts that handle real-life situations gracefully. You’ll learn basic conditional logic, enabling your scripts to make intelligent decisions and handle unexpected situations without breaking.

Useful Everyday Tips

Finally, we’ll cover practical tips to make your daily interaction with Linux more comfortable. We’ll discuss simple customizations, ways to efficiently use command history, and methods for getting quick help when you need it. By the end, you’ll have plenty of tricks to make your Linux experience enjoyable and productive.

Wrapping Up

This approach is all about hands-on experience, providing you the confidence and clarity to continue exploring Linux on your own. With these skills, you’ll move beyond just typing commands—you’ll understand exactly what’s happening, giving you control and confidence in your Linux journey.

Intro (or the Problem)

Encryption is all the buzz these days. In a hypervisor environment, it is reasonable to have some kind of encryption. Should the hypervisor handle that or should the user? To be more specific, we are talking about server side encryption or SSE or encryption at rest. This encryption at rest can be achieved in the following ways:

• Let Linux handle it with LUKS - but if you do and you have redundancy with ZFS, you pay the N write overhead (data is encrypted N times instead of once, N being the number of disks).
• Let ZFS handle it - but if you do, you can't really migrate VMs on Proxmox because ZFS does not allow sending and receiving datasets with properties included if the dataset is encrypted. You can either send the dataset as a raw stream, that is, it turns up encrypted with the same key on the other side or you can send the dataset with no properties (compression, recordsize, etc) included, that is you have to reapply the specific properties of the dataset once it's fully received on the other side. However, Proxmox supports neither method. Proxmox only supports sending VMs (and thus zvols / datasets) with properties included.
• Let the user handle it...? Maybe the most convenient option for now, and the only viable option if you want to use and learn about Proxmox HA.

Possible Solutions

This talk is about moving VMs (and containers) that are already on such a ZFS native encrypted Proxmox setup to a Proxmox setup that is set up without encryption (one that puts the onus on the user instead). There are a few strategies we can go through here:

• Use the Proxmox Migration / Live Migration
  • Obviously not possible because of issues highlighted before.
• remote-migrate (a tool for migrating vms/cts to a remote cluster) could've been it but is blocked by the fact that:
  • cloud-init drives are not supported
  • linked clones are not supported
  • source and destination storages must support the exact same types
  • mismatched pve versions (between source and destination) are not supported
• Backup to a common destination, like PBS, and restore from that same PBS.
  • This not only works but is the most convenient solution. It requires the least administrator involvement but if the proxmox VMs have disks backed by HDDs, and they are in 100s of gigabytes or more, verifying the backup index takes a long time for these disks. As an example, for my nodes connected with gigabit fiber, my personal media VM took 39547 seconds just for the 2 TB disk (1.6T full) backed by HDD.
• Splitting redundant disks, creating a replica pool, moving each VM disk to the replica pool, deleting the old pool, renaming the new pool with the old pool's name, and then migrating the VMs (live migration would then be an option).
  • As the description would give it away, this is a very involved process but is faster than backup and restore. Downtime is likely less than expected because it opens up the pathway to live migration. However, this only makes sense for a Proxmox node with few VMs, as creating full clones of each disk would likely require multiple user interventions.

Takeaways

The idea is to make the audience familiar with Proxmox, and introduce a few intricacies even experienced users of Proxmox might face in the future. High Availability is a hot topic, and is a very useful practice by itself for small and large organisations alike. Even homelabbers such as myself employ high availability where they can. Employing HA at the hypervisor level, like in Proxmox, makes services reliable for any sort of disruption: upgrades/reboots, power loss, data corruption, and could be seen as the second level of defense on top of application level HA.

Intro

Formally described as "devops and a storage engineer," but I would say I am the devops who's not very keen on the cloud, and is more interested in making his cloud. The basis of any sort of distributed system is storage, and that's where my interests lie these days. I took to Ceph first and then Proxmox next. While both are key areas to where I am employed as a support guy (https://croit.io), Proxmox rules over my homelab while the density of my wallet doesn't really align with deploying Ceph, production style, for such a small scale.

If you liked that, there's more about me and what I do on my website: https://kayg.org

Talk Abstract

The cloud isn't all-consuming. Multiple clouds? Maybe. How do you sync your data between datacenters? Sure, rclone is a nice enough tool but then how do you integrate it with ceph if you want 2 way replication? Is there something easier that comes with ceph that does not require a lot of glue? Sure there is. Ceph's RADOS Gateway features multi-site that enables geographically distributed object storage deployments by allowing buckets to be replicated across multiple Ceph clusters. Not only does this allow a global namespace, but this also facilitates disaster recovery.

In this talk, we will glance over the architecture, synchronisation modes, fail-over/disaster recovery capabilities and new features introduced in new ceph release, Reef. Realms, zonegroups, and zones make up the multi-site architecture, allowing for asynchronous data replication, but restricting metadata operations to the master zone. The modes of synchronisation vary between a full sync, metadata only sync and selective sync. Each zone can operate independently without the other and recover once they are fully functional, making the fail-over part of it quite reliable. The shiny bits are the fresh additions in Reef which allow for compression of objects before they are server-side encrypted and dynamic resharding of buckets across zones.

Takeaways and Use-cases

Ceph's multi-site isn't an easy topic to grok, contrastingly so the documentation and the terms can be overwhelming to a new ceph user or even to somebody who's been working with Ceph, RGW but has not worked with multi-site. If everything goes well, within the length of the presentation, the aim is to have the audience grasp and leave with a deeper or at least a good enough understanding of how multi-site functions and what practical use-cases they might get out of it.

👓 References

https://docs.ceph.com/en/reef/radosgw/multisite/#zone-features

https://docs.ceph.com/en/latest/radosgw/multisite/#multisite

❓ What?

VLANs in Proxmox are a method to segment existing networking interfaces, bridges or bonds. If eth0 is the interface name, eth0.2 would be the vlan interface name where 2 is the vlan tag. The vlan tag 1 is reserved for untagged traffic and is often the default vlan in most networks, including Proxmox. VLAN IDs range from 0 - 4096.

❔ Why?

VLANs are used to separate traffic into different Broadcast Domains. This is useful in managing networking traffic, creating a logical separation between, say, different tenants on the same host. However, VLANs can also span multiple hosts if the hosts are connected by a switch with Trunk Port configuration. Intra-VLAN Traffic is permitted by default and Inter-VLAN Traffic is disallowed.

🎤 How?

Create VLANs

There are two ways to create a VLAN on Proxmox:

Transparent configuration

The VM’s virtual network device has to be assigned a network tag and the network device has to be marked VLAN aware.

1. Visit the Network tab under the node and create a Linux bridge.

2. Tick the “VLAN Aware” box.

3. Create a “Linux VLAN” too as we want the Proxmox Host to do the routing. Here we want to define a subnet as the VM will be permitted to set an IP within this subnet.

4. On the VM, goto the Hardware Tab. While creating the network device or editing a network device, add the vlan tag.

5. Under VM → Cloud-init, add IPs belonging the VLAN’s subnet to the VMs.

Make sure to click on “Regenerate Image” to apply changes.

6. Start both VMs and get them to ping each other.

Traditional configuration

The VLAN tagging is done directly on the interface and the VLAN interface is referred to by a linux bridge. This VLAN interface on the physical interface is created dynamically when the linux bride is active and is in use.

1. Create a Linux Bridge under System → Network that refers to a VLAN on the NIC:

This time the bridge is not marked VLAN-Aware, as the tagging is done directly on the NIC. The bridge port here is the vlan tag we want to create on the NIC - 2048 in this example. eno1.2048 will be created and destroyed dynamically.

2. Use the bridge vmbr2 on two test VMs to put them in the same VLAN. The VLAN tag on the VM itself is empty because the VM network device does not know it belongs to a VLAN, the bridge itself does.

3. Under VM → Cloud-init, make sure the VM has an IP that belongs to our VLAN bridge.

Make sure to click on “Regenerate Image” for changes to apply to the VM.

4. Start the VMs and try to ping each other.

Why does inter-VLAN traffic work?

However, now that we have two VLANs (one on a bridge, one on a NIC), when we ping from one VLAN to the other…. this happens.

This is because of a sysctl property that makes IP forwarding possible. The property is disabled by default on Proxmox but for a routed configuration, where traffic has to be forwarded from one interface to another, it needs to be enabled everytime on startup like this:

echo 1 > /proc/sys/net/ipv4/ip_forward

OR permanently like this:

echo 'net.ipv4.ip_forward = 1' | sudo tee -a /etc/sysctl.d/99-tailscale.conf
echo 'net.ipv6.conf.all.forwarding = 1' | sudo tee -a /etc/sysctl.d/99-tailscale.conf
sudo sysctl -p /etc/sysctl.d/99-tailscale.conf

Now Tailscale docs mention a very important side effect of allowing IP forwarding:

Thus, the host here acts like a router and forwards all traffic, even the unintended bits (between two VLANs).

Fixing inter-VLAN traffic

This can be solved by setting a default forwarding policy of DROP with iptables and then allowing traffic we intend to.

Drop everything else

iptables -A FORWARD -j DROP

Allow communication between the VLANs and the bridge.

iptables -A FORWARD -i vmbr5 -o vmbr5.+ -j ACCEPT
Iptables -A FORWARD -i vmbr5.+ -o vmbr5 -j ACCEPT

Allow communication between the NIC and the bridge.

iptables -A FORWARD -i eno1 -o vmbr+ -j ACCEPT
iptables -A FORWARD -i vmbr+ -o eno1 -j ACCEPT

👓 References

7.4. FORWARD and NAT Rules Red Hat Enterprise Linux 4 | Red Hat Customer Portal

Subnet routers and traffic relay nodes

Explain the term broadcast domain, and what are multicast and unicast?

Internal working of rules in forward chain for NAT

iptables: rules to forward incoming packets from a static IP on one interface to a dynamic IP on another interface