Posted on by maartenk

How to Convert a Conda Environment into an Apptainer Image with Snakemake: Improving Performance on Software-Defined Storage System

Managing complex workflows and dependencies in scientific computing can be challenging. Conda helps by managing dependencies and environments, while Apptainer (formerly Singularity) allows for containerized execution, especially in HPC environments. Snakemake, a powerful workflow management system, can integrate these tools to ensure reproducibility and scalability. Additionally, using Apptainer can significantly improve small file performance on software-defined storage systems like CEPHFS. This guide will walk you through converting a Conda environment into an Apptainer image using Snakemake.

Prerequisites

  1. Conda: Make sure Conda is installed.
  2. Apptainer or Singularity: Install Apptainer by following the official guide.
  3. Snakemake: Ensure Snakemake is installed.

Step-by-Step Guide

1. Generate a Dockerfile with Snakemake

First, use Snakemake to generate a Dockerfile for your workflow:

snakemake --containerize > Dockerfile

This command creates a Dockerfile that includes all the necessary dependencies for your Snakemake workflow. Please make sure your workflow itself does not print any text.

2. Create and Activate a Conda Environment for Spython

Spython is a tool that simplifies converting Dockerfiles to Apptainer definition files. Create and activate a Conda environment for Spython:

conda create -n spython spython
conda activate spython

3. Convert the Dockerfile to an Apptainer Definition File

Use Spython to convert the Dockerfile into an Apptainer definition file:

spython recipe Dockerfile container.def

4. Build the Apptainer Image

Now, use Apptainer to build the image from the definition file:

apptainer build  container.sif container.def

This command generates an Apptainer image (container.sif) from the container.def file.

5. Add container to Snakfile

Tell SnakeMake where to find the container. Add the following to your snakemake file in a global part of the workflow:

containerized: "/path/to/container.sif "

Please note, do use containerized, not conainerize.

6. Run Your Snakemake Workflow with Apptainer

Finally, run your Snakemake workflow using the generated Apptainer image. Ensure the necessary directories are bound to the Apptainer container using the APPTAINER_BIND environment variable:

export APPTAINER_BIND="/cvmfs/softdrive.nl"
snakemake -c1 --use-conda  --use-singularity

This command runs the Snakemake workflow, producing the desired VCF file while leveraging the Apptainer container for execution. For snakemake >8 use. snakemake -c1 --software-deployment-method conda apptainer

Addressing Small File Performance Issues

Using Apptainer containers can help alleviate performance issues associated with small files on software-defined storage systems like CEPHFS. These systems often struggle with the overhead of managing numerous small files, leading to degraded performance. By containerizing your environment, you encapsulate your dependencies and binaries into a single image file. This reduces the number of small files accessed directly from the storage system, thus improving I/O performance and overall workflow efficiency.

Conclusion

By following these steps, you can seamlessly integrate Conda environments into Apptainer images using Snakemake, which not only ensures reproducibility and scalability but also addresses small file performance issues on software-defined storage systems like CEPHFS. This workflow enhances your computational environment's portability and efficiency, especially in HPC settings. If you encounter any issues or have questions, feel free to reach out for assistance. Happy computing!