Frequently Asked Questions

This section addresses common questions about the ALMA and VLA pipelines.

General

What is the difference between PPR and Recipe workflows?

PPR (Pipeline Processing Request) is the standard execution method used in ALMA and VLA observatory operations. A PPR is a self-contained XML document that packages both observation metadata (project structure, scheduling block IDs, dataset filenames, session definitions) and processing instructions (ordered sequence of pipeline tasks and parameters).

PPRs are executed via standalone runner scripts in CASA:

# ALMA execution
casa --nogui --nologger -c pipeline/runpipeline.py ppr_alma.xml

# VLA execution
casa --nogui --nologger -c pipeline/runvlapipeline.py ppr_vla.xml

Recipe / Procedure (recipereducer) is a programmatic Python API workflow designed for development, validation testing, and custom data reduction. It is driven by generic procedure XML templates (located in pipeline/recipes/, e.g., procedure_hifa_calimage.xml or procedure_hifv.xml) that define only the sequence of pipeline tasks and default parameters.

Unlike a PPR, a recipe procedure contains no dataset metadata. Input measurement sets (vis), session assignments, and execution options are passed directly as Python arguments:

from pipeline import recipereducer
recipereducer.reduce(vis=['uid___A001_X123_X456.ms'], procedure='procedure_hifa_calimage.xml')

Note

In production operations, a dataset-specific PPR XML is dynamically generated online by merging a generic recipe procedure with observation project metadata.

What Python version does the pipeline require?

The latest pipeline version supports Python 3.12+. Check the branch-specific pyproject.toml for exact version constraints and compatible CASA versions.

Installation & Deployment

How do I build a CASA6+Pipeline monolithic tarball locally?

You can create a standalone CASA distribution with the pipeline pre-installed using a custom setup script. This is useful for creating reproducible environments to facilitate validation testing of specific CASA and pipeline version combinations not covered by the automated packaging matrix at NRAO.

Prerequisites:

  • ~3 GB of available disk space

  • Internet connection for downloading release tarballs and dependencies

  • git and wget installed

  • Bash shell environment

Step-by-Step Instructions:

  1. Create the setup script: Save the following script as setup_casa_pipeline.sh:

#!/bin/bash
# CASA + Pipeline Setup Script
# Downloads a base CASA distribution and installs a specified pipeline branch.

set -euo pipefail  # Exit on error, undefined vars, and pipe failures

# Specify base CASA version and pipeline git branch/tag (modify as needed)
casa_ver="casa-6.7.1-13-py3.10-gpu.el8"
pipe_ver="release/2026.1.0"

echo "Setting up CASA ${casa_ver} with pipeline branch ${pipe_ver}..."

# Download and extract base CASA distribution
echo "Downloading CASA tarball..."
wget "https://casa.nrao.edu/download/distro/casa/releaseprep/${casa_ver}.tar.xz"

echo "Extracting CASA tarball..."
tar xvf "${casa_ver}.tar.xz"

# Clone target pipeline repository
echo "Cloning pipeline repository..."
git clone --branch "${pipe_ver}" --single-branch \
    https://open-bitbucket.nrao.edu/scm/pipe/pipeline.git

# Install third-party pipeline requirements inside CASA's Python environment
echo "Installing pipeline requirements..."
PYTHONNOUSERSITE=1 "${casa_ver}/bin/pip3" install \
    --disable-pip-version-check \
    --upgrade-strategy=only-if-needed \
    --use-pep517 \
    -r pipeline/requirements.txt

# Install pipeline package into CASA
echo "Installing pipeline package..."
PYTHONNOUSERSITE=1 "${casa_ver}/bin/pip3" install \
    --disable-pip-version-check \
    --upgrade-strategy=only-if-needed \
    --use-pep517 \
    pipeline/.

echo "Setup complete! Pipeline package successfully installed inside ${casa_ver}."

  1. Customize target versions:

    • Modify casa_ver inside the script to select a different base CASA release.

    • Modify pipe_ver to target a specific git branch, release tag, or commit hash.

  2. Make executable and run:

chmod +x setup_casa_pipeline.sh
./setup_casa_pipeline.sh

  1. Package as a monolithic tarball (Optional): Once setup completes, archive and compress the resulting CASA directory for easy distribution:

tar czf casa-6.7.1-with-pipeline.tar.gz casa-6.7.1-13-py3.10-gpu.el8/

  1. Deploy to target systems: Transfer and unpack the standalone bundle on target environments (e.g., HPC clusters):

# Transfer archive to remote host
scp casa-6.7.1-with-pipeline.tar.gz user@cluster:/shared/software/

# Extract on target machine
tar xzf casa-6.7.1-with-pipeline.tar.gz

  1. Verify the installation: Test that CASA launches properly and loads the pipeline package revision:

./casa-6.7.1-13-py3.10-gpu.el8/bin/casa --version
./casa-6.7.1-13-py3.10-gpu.el8/bin/python3 -c "import pipeline; print(pipeline.revision)"

Tip

Editable Development Installation

If you are actively developing or modifying pipeline source code, replace the final package install command in Step 1 with editable mode (-e):

PYTHONNOUSERSITE=1 "${casa_ver}/bin/pip3" install \
    --disable-pip-version-check \
    --upgrade-strategy=only-if-needed \
    --use-pep517 \
    -e pipeline/.

Installing in editable mode links CASA directly to your local git checkout rather than copying files into site-packages. Any modifications you make to the pipeline code will take effect immediately upon restarting CASA, without needing to reinstall.

What are the different Pipeline installation and deployment use cases?

The pipeline can be deployed in several distinct configurations depending on your operational needs:

  • Monolithic CASA Distribution (Self-contained tarball) A complete CASA 6 release bundled with an embedded Python interpreter, all C++/Python libraries, and the Pipeline pre-installed.

    • Best for: Observatory production operations and standalone data reduction requiring a self-contained, reproducible environment with near-zero setup.

  • Modular Python Environment (Managed via Pixi or Conda) A standard Python environment where CASA 6 tools (casatools, casatasks) and Pipeline dependencies are installed dynamically according to the Pipeline project package specifications.

    • Best for: Scientific research, customized data workflows, and environments needing minimal initial disk size (requires internet access or a local package cache).

  • Editable Development Installation Built on top of either a monolithic CASA distribution or a modular Python environment, with the Pipeline source repository linked in editable mode (pip install -e).

    • Best for: Active software development, testing new algorithms, and debugging pipeline recipes without reinstalling after code changes.

  • Containerized Execution (Docker / Singularity / Apptainer) An encapsulated container image that packages the OS runtime, CASA suite, and Pipeline into an immutable, portable environment.

    • Best for: High-Performance Computing (HPC/HTC) clusters, cloud-native processing, and automated CI/CD workflows.

Configuration

How do I customize pipeline workflow executaion and behaviors?

Parameters are defined in:

  • PPR XML files — task-specific settings for production runs

  • Recipe procedures — task parameters passed to recipereducer

  • Environment variables — global configuration (see Interface Reference)

What environment variables can I set?

Key variables include:

  • FLUX_SERVICE_URL — ALMA flux density database URL

  • JYPERKDB_URL — Jy/K conversion database URL

  • SCIPIPE_ROOTDIR — pipeline data root directory

  • ENABLE_TIER0_PLOTMS — enable/disable plotms visualization

See Environment Variables for complete list.

Data Products

Where are the output products stored?

Pipeline outputs are written to the current working directory by default, managed by export tasks (hifa_exportdata, hsd_exportdata, etc.). Output structure:

./
  uid___*.ms/           # Processed measurement set
  calibration/          # Calibration tables
  products/             # FITS images and weblog
  pipeline_manifest.xml # Processing metadata
  pipeline_aquareport.xml # Quality assessment

What is in pipeline_manifest.xml?

The manifest contains:

  • Processing metadata (versions, timestamps, execution time)

  • Data product inventory (names, paths, sizes)

  • Task execution summary

  • Consumed by: ALMA Science Archive, NRAO Archive, AQUA system

How do I interpret the weblog?

The weblog is an HTML report with:

  • Per-stage summaries and diagnostic plots

  • Flag statistics and RFI detection results

  • Calibration solution plots

  • Image statistics and quality metrics

Open index.html in the weblog directory to browse results.

Execution

Troubleshooting

What if the flux service is unavailable?

The pipeline has fallback behavior:

  • Primary service is FLUX_SERVICE_URL (almascience.org)

  • Backup is FLUX_SERVICE_URL_BACKUP (asa.alma.cl)

  • Use local flux.csv override file for manual values

Set via environment or PPR configuration.

How do I handle antenna position correction failures?

By default, if the getantposalma task fails (e.g., due to ALMA web service issues or network offline states), the pipeline raises a PipelineException and halts execution.

If you want the pipeline to log a warning and continue processing without applying corrections, you can enable lenient mode by setting the ALLOW_GETANTPOSALMA_FAILURE environment variable to true before running the pipeline:

import os
# Lenient mode: Log warning and continue processing if getantposalma fails
os.environ['ALLOW_GETANTPOSALMA_FAILURE'] = 'true'

# Strict mode (default): Crash with PipelineException if getantposalma fails
os.environ['ALLOW_GETANTPOSALMA_FAILURE'] = 'false'

What if I need to run in a headless environment (HPC cluster)?

CASA tasks (mainly plotms and viewer tasks) require an active X11 display server. In a headless environment (such as an HPC cluster or CI runner), you should execute CASA using the system-level xvfb-run utility to redirect graphics to a virtual frame buffer:

xvfb-run -a casa --nogui --nologger -c pipeline/runpipeline.py ppr_file.xml

The pipeline also features an internal virtual display server (powered by the python package pyvirtualdisplay, enabled via configuration). If you are already running CASA with the external xvfb-run utility, you should manually set the environment variable XVFB_RUN=1. The pipeline detects this variable and skips starting its own nested virtual display session to avoid resource conflicts:

export XVFB_RUN=1
xvfb-run -a casa --nogui --nologger -c pipeline/runpipeline.py ppr_file.xml

References