Operations Research

Logo

A site for hosting software and data repositories associated with papers appearing in the journal _Operations Research_

View source on GitHub

Isolated Environments for Reproducible Experiments

Creating reproducible experimental environments requires isolating dependencies, system configurations, and runtime environments. Here are the main approaches, from lightweight to comprehensive.

Language-Specific Virtual Environments

Python: venv / virtualenv

Isolates Python packages at the user level:

# Create virtual environment
python -m venv myenv

# Activate
source myenv/bin/activate  # Unix
myenv\Scripts\activate     # Windows

# Install dependencies
pip install -r requirements.txt

# Deactivate
deactivate

Pros:

Cons:

Python: conda

Isolates Python version and packages, including non-Python dependencies:

# Create environment with specific Python version
conda create -n myenv python=3.9 numpy scipy

# Activate
conda activate myenv

# Export environment
conda env export > environment.yml

# Recreate environment
conda env create -f environment.yml

Pros:

Cons:

R: renv

Project-specific R package libraries:

# Initialize
renv::init()

# Save state
renv::snapshot()

# Restore
renv::restore()

Pros:

Cons:

Julia: Pkg Environments

Built-in project environments:

# Activate project
using Pkg
Pkg.activate(".")

# Install packages (automatically tracked)
Pkg.add("DataFrames")

# Instantiate from manifest
Pkg.instantiate()

Pros:

Cons:

Node.js: npm / yarn

Project-specific Node dependencies:

# npm
npm install

# yarn
yarn install

Pros:

Cons:

Version Managers

Python: pyenv

Manages multiple Python versions:

# Install specific Python version
pyenv install 3.9.7

# Set version for directory
pyenv local 3.9.7

# Combine with venv
python -m venv myenv

Pros:

Cons:

Node.js: nvm / fnm

Manages multiple Node.js versions:

# nvm
nvm install 16.14.0
nvm use 16.14.0

# fnm (faster)
fnm install 16.14.0
fnm use 16.14.0

Pros:

Cons:

Ruby: rbenv / rvm

Manages multiple Ruby versions:

# rbenv
rbenv install 3.1.0
rbenv local 3.1.0

# rvm
rvm install 3.1.0
rvm use 3.1.0

Container Technologies

Docker

Full OS-level isolation with containers:

# Dockerfile
FROM python:3.9-slim

WORKDIR /app

COPY requirements.txt .
RUN pip install -r requirements.txt

COPY . .

CMD ["python", "experiment.py"]

# Build image
docker build -t myexperiment:v1 .

# Run container
docker run myexperiment:v1

# Interactive shell
docker run -it myexperiment:v1 bash

Pros:

Cons:

Docker Compose

Orchestrate multi-container setups:

# docker-compose.yml
version: '3'
services:
  experiment:
    build: .
    volumes:
      - ./data:/app/data
    environment:
      - EXPERIMENT_ID=exp001
  
  database:
    image: postgres:13
    environment:
      - POSTGRES_DB=experiments

docker-compose up

Pros:

Cons:

Podman

Docker alternative (daemonless, rootless):

# Similar commands to Docker
podman build -t myexperiment:v1 .
podman run myexperiment:v1

Pros:

Cons:

Singularity / Apptainer

Container system designed for HPC:

# Build from Docker image
singularity build myexperiment.sif docker://myexperiment:v1

# Run
singularity run myexperiment.sif

# Shell
singularity shell myexperiment.sif

Pros:

Cons:

Virtual Machines

Vagrant

Manages development VMs with code:

# Vagrantfile
Vagrant.configure("2") do |config|
  config.vm.box = "ubuntu/focal64"
  
  config.vm.provision "shell", inline: <<-SHELL
    apt-get update
    apt-get install -y python3-pip
    pip3 install -r /vagrant/requirements.txt
  SHELL
end

vagrant up
vagrant ssh

Pros:

Cons:

VirtualBox / VMware

Full virtualization platforms:

Pros:

Cons:

Cloud/Remote Options

Google Colab / Kaggle Notebooks

Cloud-based Jupyter notebooks:

Pros:

Cons:

Binder / MyBinder.org

Reproducible Jupyter environments from GitHub:

# environment.yml
name: myenv
dependencies:
  - python=3.9
  - numpy
  - matplotlib

Pros:

Cons:

Code Ocean / Gigantum

Platforms designed for computational reproducibility:

Pros:

Cons:

Specialized Tools

Nix / NixOS

Declarative package management and system configuration:

# shell.nix
{ pkgs ? import <nixpkgs> {} }:

pkgs.mkShell {
  buildInputs = [
    pkgs.python39
    pkgs.python39Packages.numpy
    pkgs.python39Packages.scipy
  ];
}

nix-shell

Pros:

Cons:

Guix

Similar to Nix with Scheme-based configuration:

Pros:

Cons:

Spack

Package manager for HPC:

spack install python@3.9.7 ^openmpi@4.1.0
spack load python@3.9.7

Pros:

Cons:

Workflow Management Systems

Snakemake

Workflow system with environment management:

# Snakefile
rule experiment:
    conda: "environment.yml"
    script: "experiment.py"

Pros:

Cons:

Nextflow

Workflow system with container support:

process experiment {
    container 'myexperiment:v1'
    
    script:
    "python experiment.py"
}

Pros:

Cons:

Comparison Matrix

Approach Isolation Level Reproducibility Setup Complexity Resource Overhead Best For
venv/virtualenv Packages only Low Very Low Minimal Quick Python experiments
conda Packages + Python version Medium Low Low Scientific Python work
Docker Complete OS High Medium Medium Cross-platform reproducibility
Singularity Complete OS High Medium Medium HPC environments
Vagrant Full VM High Medium High OS-level testing
Nix Bit-for-bit Very High High Low Maximum reproducibility
Language tools (renv, Pkg) Language packages Medium Very Low Minimal Language-specific projects

Recommendations by Use Case

Quick Local Experiment (Python)

# Lightweight approach
python -m venv venv
source venv/bin/activate
pip install -r requirements.txt

Shareable Experiment (Any Language)

# Docker for portability
FROM python:3.9
COPY requirements.txt .
RUN pip install -r requirements.txt
COPY . /app
WORKDIR /app

Academic Paper Reproduction

HPC Cluster

Maximum Reproducibility

Team Collaboration

Long-term Archival

Key Takeaways

Layered Approach

Combine multiple tools for complete reproducibility:

Trade-offs

Reproducibility Levels

  1. Code only: Not reproducible (dependencies change)
  2. Code + dependency list: Somewhat reproducible (versions drift)
  3. Code + lock file: Good reproducibility (specific versions)
  4. Code + lock file + container: Very reproducible (includes OS)
  5. Code + Nix/Guix: Bit-for-bit reproducible (all dependencies pinned)

Best Practices

Always capture:

Version control:

Document:

Modern Standard (2025)

For most scientific computing:

  1. Development: Language-specific tool (conda, renv, Pkg)
  2. Sharing: Docker container
  3. Publishing: Docker image + code on GitHub/Zenodo
  4. Optional: Nix for maximum reproducibility

Common Pitfalls

Bottom Line: For most reproducible experiments, use conda (Python scientific) or language-specific tools for development, then package in Docker for sharing and long-term reproducibility. For maximum reproducibility or HPC work, consider Singularity or Nix.