New server has been acquired. Debian 13 has been installed.
GS308EP switches have been acquired and installed.
Now, I’m working to migrate to the new machine. 3 1/2 years ago when I started futzing with Docker, I sorta followed guides and guessed, abused it trying to make it do things it wasn’t designed for, and flipped switches I likely shouldn’t have flipped, so the set up is more than a little shabby.
As a result, I’ll likely end more redeploying than migrating the containers.
So rather than go forward with Docker blindly, I want to reassess whether I shouldn’t look into Proxmox, LXC, or Podman instead of Docker, or maybe something else entirely?
Work is just about done dumping ESX for Nutanix, but both of those seem overkill for my needs.
Of course the forums for any of the solutions make their own out to be the best thing since sliced bread and the others useless, so I’m hoping to get a more nuanced answer here.
You must log in or # to comment.
There is no wrong answer but the right answer is Proxmox.
Definitely Proxmox as base OS, Docker inside.
Run Docker inside a LXC or VM
and run Proxmox inside that VM
What?
they claim 3% overhead, I bet you could get pretty deep before noticing.
Proxmox with all of its warts appears to be the better hypervisor. If your host has the headroom like others suggested: proxmox with a VM hosting docker/podman.
I have to say running proxmox saved me from running to the basement or opening up IPMI a few times already when making suspect changes to a VM that otherwise would have taken down the network of my host.
ATM I’m running Proxmox with Nixos VMs running mixed docker/podman containers. It works out pretty well for my use case and with some opentofu fiddling I have most of my infrastructure defined in config files if that’s a rabbit hole you want to go down.
As for my experience with docker vs podman I have to say podman can pretty much do whatever docker does with the exception of docker swarm. You may have to do some digging to handle more advanced networking/gpu setups.
The only thing I have running docker atm is a gluetun container because container to container networking took more than 15m of research with podman so I fell back to the very well documented path using docker.
Proxmox can run lxc containers natively.
Personally I keep a Debian VM for docker, a holdover from before hypervisors supported containers natively. I use docker compose and it Just Works™.
I have three docker nodes on three Debean VMs on three proxmox servers. I should play around with lxcs and native containers on proxmox but it would mess up with my tri lateral system.
Both. Depends on what you want to achieve
I run Proxmox in my small cluster. Proxmox runs multiple VMs that each run various groups of docker containers.
They’re on different levels.
Yea I’m the same. planning on making some VMS/docker nodes, Highly available while others can be kicked to the kerb if there are issues.
docker in a vm in proxmox!
no but really. I don’t like that the vm host’s kernel is shared with all the containers.
then maybe have 2 or 3 VMs, one of which hosts essential services (not like your nextcloud, but services that are necessary to make the (internal or whole home) network work like DNS and DHCP, maybe monitoring, and another that hosts your convenience services (jellyfin, nextcloud). the first is easier to fix if it breaks, the second is not the end if the world if it does. maybe also have a third for services that would be painful to have offline (nextcloud), but try to limit it to few tasks because the goal is simplicity. this setup can help with distro upgrade difficulties, and vm custom configuration issues when you have set up something that seems to work but will prevent VM startup sometime in the future.podman could be a good idea too.
Proxmox is a hypervisor. I would install that as your os if that is something you want.
If you just need containers, podman is good. Docker is fine, but podman is open source.
That being said, I run proxmox on my server and have various vms running. I have a laptop that I will use to play with containers and if it turns into a service I want I’ll make it run on proxmox, either as a container itself or as something running on a VM on the hypervisor
podman is also rootless.
Docker CE is also just as foss
Docket-compose is the easiest and most commonly used format imo
Proxmox and Docker don’t really do the same thing. They live in the same area, but the coverage is very different. You can always use docker when your host is running proxmox: either individually or in groups inside of an lxc, or all in w dedicated VM, or even natively on the same house if you prefer chaos. But you can’t do the opposite: Sometimes you just need a VM. Maybe you only need a couple of devices, and you know they run on or are even designed for docker, then that’s the better option. In all other cases, and when just getting started, proxmox is just the way more universal solution if you’re only planning on having a single host (for now).
The management tools in proxmox are great. The community scripts are a fantastic resource and only work with proxmox. I would suggest you set it up natively, not on top of Debian though, even if that’s already installed. Not the least of the reasons are to be able to use ZFS easily, including on on the boot partition (select that in the installer).
Finally, if you’re gonna stick with docker, like others said: consider podman. That really does the same thing docker does, but it’s fully open source. Arguably it’s better in some areas, but on the flip side might, in occasion, require fiddling with something intended specifically for docker and using advanced setups.
Also there really is no wrong answer, either. And you can always change whatever you choose.
Docker community edition is fully foss
I like Podman but In don’t think foss is the driving factor in this case
@Creat @JeanValjean Alot of apps seem to come with Docker images - can you use those with Podman?
Docker images are OCI compliant and are agnostic of the container runtime you use.
Yes. That’s basically the point. They call it a “drop in replacement”, but last I used it manually there were some extra steps for what I wanted to do. To be clear: not for every thing you want to setup, just one if the things I read don’t up required extra steps. But I also hear that those things have changed since then and it’s mostly seamless now.
It’ll never really be a perfect drop in replacement because Docker relies on its daemon for a lot of functionality and Podman is daemonless, so you have to work around that. But like you said it’s just a matter of learning how things work with Portman.
Proxmox is great. I run each service in its own lxc/vm. Many of the vms also run docker. Probably overkill, but backups and restores via Proxmox Backup Server are super easy when segregated this way
This. Unless you want to go through the hell of multiple redundant architecture to deal with updates breaking things and backups. If you are The only person who is maintaining the multiple services that you’re running the backup feature alone is the reason to go with Proxmox. Upgrade to a lcx container didn’t go like you thought it would, roll back the backup, 30 sec and done. Also making liberal use of alpine have made it smooth and simple.
where do you run PBS? in a vm on the main proxmox host?
Yes that’s exactly how I run it. I backup the PBS vm separately also, in case the host goes down
LXC on the host for me. Make sure it never backs itself up though.
@JeanValjean I am not in a position to judge but Id be interested to hear comments on this position.
From the Proxmox Discussion Forum
Linux Containers (= LXCs) and docker containers are something completely different. LXCs are system containers. These contain a full OS except for the kernel which is shared with the host. You will have to administrate them like you would do it with a VM.
Docker containers are application containers where you containerize a single appliance. You don’t individualize/upgrade them. You just throw them away and create a new one. So you are quite limited what you can do with them, as all you can configure is what the container creator wants you to be able to change.So if you want a service as secure and independent as possible, or if you want to use Win/Mac/FreeBSD, use a VM.
If you want something like a VM running Linux, but you want less overhead, at the cost of security, use a LXC.
If you just want to run some services but you don’t want to invest time on learning how they work or administrating them, then running a VM with docker would be a good choice.podman inside nixos inside lxc inside proxmox
Maybe use Docker with a UI like
lazydockerorPortainer.If you feel safe with Docker, perhaps experiment with Proxmox as a base. Then you can separate your environments into different VMs.
I use systemd-nspawn to absolutely pissoff everyone :)
systemd-nspawn launched virt-qemu-run instance of proxmox running a windows hyper-v vm
