Skip to content

Docker

Do NOT try this code on the cluster

Please be aware that examples below are intended for those users who want to prepare their own docker containers on their own laptop/workstation and convert them into singularity containers.

Please do NOT use these examples on the cluster. They are for apt-based systems - Ubuntu, Debian etc. Cluster is running CentOS 7 and you certainly do not have sudo rights on the cluster.

On the cluser you have to use Singularity, please see the Singularity page

If you need only to convert container from docker hub or Nvidia gpu cloud, you can do it directly on the cluster, the method is described in the Singularity page

Example how to run a code in a container

We will examine the construction of a singularity container for the COLMAP library. One way of doing it is to use the Docker container and build the Singularity out of it.

Docker can be installed according to the official documentation. GUI for Windows makes the work with Windows Subsystem Linux (WSL) easy. Thus one can compile and run all the Linux codes on Windows. Moreover, the recent update in the form of WSL2 should also allow GPU sharing between the Windows host and the Linux subsystem. It should allow running Tensor Flow etc., in a container on Windows. The two main structures are the images and containers. Images wrap all the libraries and are used for sharing. They can be transformed into the Singularity image. The containers are runtime versions of images, i.e., we can use them for running your codes. You can see the Docker tutorial for more details. PS. Some summary is also bellow.

The first step is to install the Singularity on your local machine because the build of an image requires sudo rights. I recommend working in some Ubuntu distribution under WLS and later follow the same steps as written for Linux in the documentation.

To run Ubuntu on windows in WSL2, follow these five steps, open PowerShell as Administrator, and run: 1. Enable WSL: dism.exe /online /enable-feature /featurename:Microsoft-Windows-Subsystem-Linux /all /norestart 2. Enable ‘Virtual Machine Platform’: dism.exe /online /enable-feature /featurename:VirtualMachinePlatform /all /norestart 3. Set WSL 2 as default: wsl --set-default-version 2 4. Install a Linux distro -> Ubuntu 20.04 and Windows Terminal 5. Use WSL 2: run Windows Terminal -> select an installed Linux at the top bar -> setup password and use it as you wish

Having the Linux running, we can install Singularity there following few steps here. In sum, there is need to install the dependences:

$ sudo apt-get update && sudo apt-get install -y \ build-essential \ uuid-dev \ libgpgme-dev \ squashfs-tools \ libseccomp-dev \ wget \ pkg-config \ git \ cryptsetup-bin

Install the Go compiler (you can select different version, see reales versions here):

$ export VERSION=1.15.6 OS=linux ARCH=amd64 && \ wget https://dl.google.com/go/go$VERSION.$OS-$ARCH.tar.gz && \ sudo tar -C /usr/local -xzvf go$VERSION.$OS-$ARCH.tar.gz && \ rm go$VERSION.$OS-$ARCH.tar.gz

$ echo 'export GOPATH=${HOME}/go' >> ~/.bashrc && \ echo 'export PATH=/usr/local/go/bin:${PATH}:${GOPATH}/bin' >> ~/.bashrc && \ source ~/.bashrc

Clone and install the Singularity:

$ git clone https://github.com/hpcng/singularity.git && \ cd singularity && \ git checkout v3.7.0

$ ./mconfig && \ make -C ./builddir && \ sudo make -C ./builddir install

$ ./mconfig --prefix=/opt/singularity

Try if Singularity is ready and available:

$ singularity

We have all the prerequisites now, so let's try to create some singularity image. The COLMAP already share the Docker image on DockerHub. Building the Singularity image is super simple now:

$ sudo singularity build colmap.simg docker://colmap/colmap

This way, we can build any docker image into Singularity by one command. This container is saved in the directory where you run the code. Thus you may use the full paths. Details are below (you need to call docker-archive for local images). The COLMAP can be executed on server by runing:

$ singularity exec --nv -B $(pwd):/host_pwd --pwd /host_pwd colmap.simg colmap

Just by running colmap instead of /bin/bash.

Run any code in a container

One trick may be to create the Docker image and convert it to Singularity. We can start by running a container out of an image, e.g., Ubuntu 20.04. A simple example which does that may be:

$ docker run -it ubuntu /bin/bash

It will download the latest release of Ubuntu image, and -it say to run the /bin/bash command interactively. Thus you can install and develop anything you need now. The interactive access can be stoped by Ctrl+d shortcut. Let's mention a few main commands of docker:

  • List available containers: $ docker ps -all
  • List running containers: $ docker ps
  • List available images: $ docker images
  • Start a container (not the image): $ docker start <container id or name>
  • Start a container from image: $ docker run [options] <container id or name> <command>
  • Stop a container: $ docker start <container id or name>
  • Run something in running container: $ docker exec [options] ubuntu <container id or name> <command>, e.g., $ docker exec -it ubuntu_container /bin/bash

And the important one for us:

  • Build an image from container: $ docker build -n <name of image> <container id or name>
  • Save the image into some directory (we use it for Singularity): $ docker save -o <some_path.tar> <image id or name>

Thus, you can create a docker image from a container (where you have sudo and install whatever you need) and save it locally to a tar file. Further, the command builds the Singularity image:

$ sudo singularity build <singularity_image_name>.simg docker-archive:<path_to_saved_docker_image.tar>

Note that you need to use docker-archive: for docker image saved in a local archive. Having a Singularity image, you can run it on the server, as mentioned before.