> Firstly, can there be an easier way to stop a microVM mid execution in this single executable bottlefire format and then rerun that and it would start mid execution. (something akin to how criu does it?)

Not yet - Firecracker supports snapshotting so this should be doable though!

> if something like microvm could be run in normal cloud infrastructure?

Some cloud providers - like GCP and DigitalOcean - do support nested virtualization, and they work pretty well with Firecracker. Using VM migration to run stable workloads on spot instances sounds very interesting :)

You don't need it, it's essentially a hosted version of their tool bake[0], which is actually pretty slick - without any setup you can go from a system with just KVM to running any random docker container. I can think of a few use cases for this, especially anything where you want to use containers as the delivery mechanism

[0]: https://github.com/losfair/bake

I hope you have read the article. They literally say as the other commentor points, that they essentially use bake. You could rather host something like this pretty trivially I suppose.

Seems to use a SOCKS5 proxy over tun2socks[1] to then use the host's networking setup https://github.com/losfair/bake/blob/ddf1edaa9daf340bd167794... so I think the short version to your question is "proxy server"

1: https://github.com/xjasonlyu/tun2socks#readme

I don't know about the [1] but I think that [2] actually embed's crun https://github.com/containers/crun in the executable itself.

Crun can be/is used by podman/docker too so you could think that its essentially stripping docker down to its core of running oci containers and just taking the most lightweight and (fastest?) approach and embedding it directly into the executable.

I am sure that I might get somethings wrong, I usually do, so if I have an error, please do let me know as I don't wish to spread misinformation

This project on the other hand seems to be using bake https://github.com/losfair/bake which seems to take something like a docker container and I think essentially push it into something like .iso using squashfs The idea here is that they seem to embed firecracker itself into a single binary to actually just make it an executable that can run that microvm as it is.

And I think that bottlefire just seems to do something similar but they do it on their server side and seem to just provide you a binary to make things easy for you...

I am not sure with microvm's architecture but it seems that there are some additional features that you can get since they are VM, maybe you can get the ability of https://criu.org/Main_Page out of the box but this doesn't seem to be done right now but In my opinion is really very doable.

Another idea might be more security. I am going to refer to an article [1] which talks about flatpak but it has this line and I quote `containers are not sandboxed`, so I suppose that this can help blur the lines b/w containers by converting containers into microvm's and then leveraging vm's technology for sandboxing or some other mechanism rather easily as compared to docker (I see it use landlock which was new to me!)

article [1]: https://hanako.codeberg.page/

Would proxmox work?