Nothing about resource (net, io, disk, cpu) isolation, limits, priorities, or guarantees. Not the same as a type 1 hypervisor. These qualities are needed to run things safely and predictably in the real world™, at scale. Also, accounting and multitenancy if it's going to be used as some sort VAR or VPS offering.
Fun fact, Incus is being used as underlying infrastructure for the NorthSec CTF, i.e. in an "as hostile as it can get" environment. If you have close to a hundred teams of hackers on your systems trying to break stuff, I think it speaks for Incus and its capabilities regarding isolation and limits.
In case you are interested, Zabbly has some interesting behind-the-scenes on Youtube (not affiliated).
Proxmox feels like a more apt comparison, as they both act like a controlplane for KVM virtual-machines and LXC containers across one or multiple hosts.
If you are interested in running kubernetes on top of incus, that is your kubernetes cluster nodes will be made up of KVM or LXC instances - I highly recommend the cluster-api provider incus https://github.com/lxc/cluster-api-provider-incus
This provider is really well done and maintained, including ClusterClass support and array of pre-built machine images for both KVM and LXC. It also supports pivoting the mgmt cluster on to a workload cluster, enabling the mgmt cluster to upgrade itself which is really cool.
How do you handle updating the machine that Incus itself runs on? I imagine you have to be super careful not to introduce any breakage, because then all the VMs/containers go down.
What about kernel updates that require reboots? I have heard of ksplice/kexec, but I have never seen them used anywhere.
Yes, it uses QEMU under the hood for VMs and runs LXC containers. But also, since recently, you can run docker images in it. Very handy, especially since it has 1st class remote support, meaning you can install only the incus client and when doing `incus launch` or whatever, it will transparently start the container/vm on your remote host
Incus is great when developing ansible playbooks. The main benefit for me over docker/podman is systemd works out of the box in incus containers.
I went through the online tutorial, but I'm not really seeing how it's different from docker?
Nothing about resource (net, io, disk, cpu) isolation, limits, priorities, or guarantees. Not the same as a type 1 hypervisor. These qualities are needed to run things safely and predictably in the real world™, at scale. Also, accounting and multitenancy if it's going to be used as some sort VAR or VPS offering.
Fun fact, Incus is being used as underlying infrastructure for the NorthSec CTF, i.e. in an "as hostile as it can get" environment. If you have close to a hundred teams of hackers on your systems trying to break stuff, I think it speaks for Incus and its capabilities regarding isolation and limits.
In case you are interested, Zabbly has some interesting behind-the-scenes on Youtube (not affiliated).
Took a few seconds of googling to find this: https://linuxcontainers.org/incus/docs/main/reference/instan...
A little bit of context about where Incus came from:
https://lwn.net/Articles/940684/
So it looks like a Proxmox alternative, this [0] goes into some reasons to switch. Main selling point seems to be fully OSS and no enterprise version.
[0]: https://tadeubento.com/2024/replace-proxmox-with-incus-lxd/
It’s more like a Kubernetes alternative
Proxmox feels like a more apt comparison, as they both act like a controlplane for KVM virtual-machines and LXC containers across one or multiple hosts.
If you are interested in running kubernetes on top of incus, that is your kubernetes cluster nodes will be made up of KVM or LXC instances - I highly recommend the cluster-api provider incus https://github.com/lxc/cluster-api-provider-incus
This provider is really well done and maintained, including ClusterClass support and array of pre-built machine images for both KVM and LXC. It also supports pivoting the mgmt cluster on to a workload cluster, enabling the mgmt cluster to upgrade itself which is really cool.
I was surprised to come across this provider by chance as for some reason it's not listed on the CAPI documentation provider list https://cluster-api.sigs.k8s.io/reference/providers
Not really, Kubernetes does a lot of different things that are out of scope for incus or lxd or docker compose for that matter or any hypervisor or …
like what? I'd love to hear some examples of things Kubernetes does that incus doesn't at this point
How do you handle updating the machine that Incus itself runs on? I imagine you have to be super careful not to introduce any breakage, because then all the VMs/containers go down.
What about kernel updates that require reboots? I have heard of ksplice/kexec, but I have never seen them used anywhere.
As with any such system, you need a spare box. Upgrade the spare, move the clients to it, upgrade the original.
But then the clients have downtime while they’re being moved.
I don't know about Incus, but on ProxMox the downtime when moving a VM is around 200 ms.
Isn't that the exact problem that k8s workloads solve by scaling onto new nodes first etc? No downtime required.
Right but incus is not k8s. You can stand up spares and switch traffic, but it’s not built in functionality and requires extra orchestration.
What can this work with? It says „Containers and VMs“ - I guess that’s LXCs and QEMU VMs?
Yes, it uses QEMU under the hood for VMs and runs LXC containers. But also, since recently, you can run docker images in it. Very handy, especially since it has 1st class remote support, meaning you can install only the incus client and when doing `incus launch` or whatever, it will transparently start the container/vm on your remote host
Not to be confused with the cirrus7 incus[0], which are fanless PC models based on the ASRock DeskMini series that I'm using right now.
[0] https://www.cirrus7.com/produkte/cirrus7-incus/