Configuration Reference for Unified Notebook CI/CD¶

Canonical reference (maintainers)

These inputs mirror notebook-ci-unified.yml@v1 in notebook-ci-actions (the source of truth). If you just write notebooks, see the Quick Reference.

This document provides a comprehensive reference for configuring the unified notebook CI/CD system.

Workflow inputs¶

These mirror the workflow_call inputs of notebook-ci-unified.yml@v1 (the source of truth).

Input	Type	Required	Default	Description
`execution-mode`	string	yes	-	`pr`, `merge`, `scheduled`, `on-demand`
`trigger-event`	string	no	`all`	`validate`, `execute`, `security`, `html`, `deprecate`, `all` (on-demand)
`python-version`	string	no	`3.11`	Python version
`conda-environment`	string	no	-	Custom conda env (e.g. `hstcal`, `stenv`)
`custom-requirements`	string	no	-	Path to a custom requirements file
`crds-server-url`	string	no	`''`	Sets `CRDS_SERVER_URL` (JWST-style pipelines)
`crds-context`	string	no	`''`	Sets `CRDS_CONTEXT`
`crds-path`	string	no	`''`	Sets `CRDS_PATH`
`single-notebook`	string	no	-	Single notebook path for a targeted run
`affected-directories`	string (JSON)	no	`[]`	JSON array of directories (auto-detected or manual)
`enable-validation`	boolean	no	`true`	pytest nbval validation
`enable-security`	boolean	no	`true`	bandit security scan
`enable-execution`	boolean	no	`true`	Notebook execution
`enable-storage`	boolean	no	`true`	Store executed outputs to `gh-storage`
`enable-html-build`	boolean	no	`false`	JupyterBook HTML build
`pre-processing-script`	string	no	-	Script run before validation/execution
`post-processing-script`	string	no	-	Script run after execution
`deprecation-days`	number	no	`60`	Days until deprecation (deprecate event)
`custom-runner-config`	boolean	no	`false`	Select runners via `ci_config.txt`

Secrets (all optional): CASJOBS_USERID, CASJOBS_PW.

Core Configuration Parameters¶

Execution Mode¶

Controls the primary behavior of the workflow:

Value	Description	Use Case
`'pr'`	Pull request mode	Validates changed notebooks, selective execution
`'merge'`	Merge/deploy mode	Full processing and documentation deployment
`'scheduled'`	Scheduled maintenance	Weekly validation and deprecation management
`'on-demand'`	Manual execution	Flexible manual triggers with various options

Example:

with:
  execution-mode: 'pr'

Trigger Event¶

Fine-grained control over what actions to perform (used with on-demand mode):

Value	Description	Actions Performed
`'all'`	Full pipeline	Validation + Security + Execution + Storage + HTML
`'validate'`	Validation only	pytest nbval validation
`'execute'`	Execution only	Notebook execution without validation
`'security'`	Security only	bandit security scanning
`'html'`	HTML build only	JupyterBook documentation generation
`'deprecate'`	Deprecation management	Tag/manage deprecated notebooks

Example:

with:
  execution-mode: 'on-demand'
  trigger-event: 'validate'

Environment Configuration¶

Python Version¶

Specify the Python version for the execution environment:

with:
  python-version: '3.11'        # Default
  # python-version: '3.10'      # Alternative
  # python-version: '3.12'      # Latest

Conda Environment¶

Use predefined conda environments from conda-forge:

with:
  conda-environment: 'hstcal'   # For HST workflows
  # conda-environment: 'stenv'  # For JWST workflows
  # conda-environment: 'astropy' # For general astronomy

Available Environments: - hstcal - HST calibration tools - stenv - Space Telescope environment for JWST - astropy - General astronomy Python environment - Custom environments available on conda-forge

Custom Requirements¶

Specify custom requirements file path:

with:
  custom-requirements: 'requirements.txt'           # Root requirements
  # custom-requirements: 'environment/deps.txt'    # Custom path
  # custom-requirements: 'notebooks/requirements.txt' # Notebook-specific

Feature Control¶

Enable/Disable Features¶

Control which CI/CD features are active:

with:
  enable-validation: true      # pytest nbval validation
  enable-security: true        # bandit security scanning
  enable-execution: true       # notebook execution
  enable-storage: true         # store outputs to gh-storage
  enable-html-build: false     # JupyterBook HTML generation

Feature Combinations:

Validation Only¶

with:
  enable-validation: true
  enable-security: false
  enable-execution: false
  enable-storage: false
  enable-html-build: false

Execution with Storage¶

with:
  enable-validation: false
  enable-security: false
  enable-execution: true
  enable-storage: true
  enable-html-build: false

Full CI/CD Pipeline¶

with:
  enable-validation: true
  enable-security: true
  enable-execution: true
  enable-storage: true
  enable-html-build: true

Notebook Selection¶

Single Notebook Targeting¶

Execute or validate a specific notebook:

with:
  single-notebook: 'notebooks/example/demo.ipynb'
  execution-mode: 'on-demand'
  trigger-event: 'execute'

Directory-Based Selection¶

Automatically detected based on changed files in PR mode, or manually specified:

with:
  affected-directories: '["notebooks/hst", "notebooks/jwst"]'  # JSON array

Advanced Configuration¶

Post-Processing Scripts¶

Execute custom scripts after notebook processing:

with:
  post-processing-script: 'scripts/custom_processing.sh'

Example post-processing script:

#!/bin/bash
# scripts/custom_processing.sh

echo "Running custom post-processing..."

# Image optimization
find _build -name "*.png" -exec optipng {} \;

# Custom file processing
python scripts/process_outputs.py

# Cleanup
rm -rf temp_files/

echo "Post-processing complete"

Deprecation Management¶

Configure notebook deprecation settings:

with:
  deprecation-days: 60         # Days until notebook expires
  trigger-event: 'deprecate'   # For deprecation actions

Repository-Specific Examples¶

HST Notebooks Repository¶

name: HST Notebook CI
jobs:
  hst-ci:
    uses: spacetelescope/notebook-ci-actions/.github/workflows/notebook-ci-unified.yml@v1
    with:
      execution-mode: 'pr'
      python-version: '3.11'
      conda-environment: 'hstcal'
      enable-validation: true
      enable-security: true
      enable-execution: true
      enable-storage: true
      enable-html-build: false
      post-processing-script: 'scripts/hst_image_processing.sh'
    secrets:
      CASJOBS_USERID: ${{ secrets.CASJOBS_USERID }}
      CASJOBS_PW: ${{ secrets.CASJOBS_PW }}

JWST Notebooks Repository¶

name: JWST Notebook CI
jobs:
  jwst-ci:
    uses: spacetelescope/notebook-ci-actions/.github/workflows/notebook-ci-unified.yml@v1
    with:
      execution-mode: 'merge'
      python-version: '3.11'
      conda-environment: 'stenv'
      enable-validation: true
      enable-security: true
      enable-execution: true
      enable-storage: true
      enable-html-build: true
      post-processing-script: 'scripts/jwst_data_processing.sh'
    secrets:
      CASJOBS_USERID: ${{ secrets.CASJOBS_USERID }}
      CASJOBS_PW: ${{ secrets.CASJOBS_PW }}

Standard Python Repository¶

name: Standard Notebook CI
jobs:
  standard-ci:
    uses: spacetelescope/notebook-ci-actions/.github/workflows/notebook-ci-unified.yml@v1
    with:
      execution-mode: 'pr'
      python-version: '3.11'
      custom-requirements: 'requirements.txt'
      enable-validation: true
      enable-security: false       # Disable security for simple repos
      enable-execution: true
      enable-storage: true
      enable-html-build: false

Documentation-Heavy Repository¶

name: Documentation CI
jobs:
  docs-ci:
    uses: spacetelescope/notebook-ci-actions/.github/workflows/notebook-ci-unified.yml@v1
    with:
      execution-mode: 'merge'
      python-version: '3.11'
      enable-validation: false     # Skip validation for docs
      enable-security: false
      enable-execution: false      # Skip execution for docs
      enable-storage: false
      enable-html-build: true      # Focus on HTML generation
      post-processing-script: 'scripts/docs_optimization.sh'

Secrets Configuration¶

Required Secrets¶

Configure these secrets in your repository settings:

Secret	Required	Description	Example
`CASJOBS_USERID`	Optional	CasJobs database user ID	`'your_userid'`
`CASJOBS_PW`	Optional	CasJobs database password	`'your_password'`

Setting Secrets¶

Go to your repository settings
Navigate to "Secrets and variables" > "Actions"
Click "New repository secret"
Add the secret name and value

In workflow:

secrets:
  CASJOBS_USERID: ${{ secrets.CASJOBS_USERID }}
  CASJOBS_PW: ${{ secrets.CASJOBS_PW }}

Trigger Configuration¶

Pull Request Triggers¶

Configure which files trigger the workflow:

on:
  pull_request:
    branches: [ main ]
    paths:
      - 'notebooks/**'           # Any notebook changes
      - 'requirements.txt'       # Root requirements
      - 'pyproject.toml'         # Python project config
      - '*.yml'                  # YAML configuration
      - '*.yaml'                 # YAML configuration
      - '*.md'                   # Documentation
      - '*.html'                 # Web assets
      - '*.css'                  # Stylesheets
      - '*.js'                   # JavaScript

Push Triggers¶

Configure main branch deployment:

on:
  push:
    branches: [ main ]
    paths:
      - 'notebooks/**'
      - 'requirements.txt'
      - 'pyproject.toml'
      - '*.yml'
      - '*.yaml'
      - '*.md'
      - '*.html'

Scheduled Triggers¶

Configure maintenance schedules:

on:
  schedule:
    # Every Sunday at 2 AM UTC
    - cron: '0 2 * * 0'

    # Every day at midnight UTC (for high-activity repos)
    # - cron: '0 0 * * *'

    # Every Monday at 9 AM UTC (for work-week schedules)
    # - cron: '0 9 * * 1'

Manual Triggers¶

Configure on-demand workflows:

on:
  workflow_dispatch:
    inputs:
      action_type:
        description: 'Action to perform'
        required: true
        type: choice
        options:
          - 'validate-all'
          - 'execute-single'
          - 'build-html-only'
        default: 'validate-all'

      single_notebook:
        description: 'Notebook path for single actions'
        required: false
        type: string

      python_version:
        description: 'Python version override'
        required: false
        type: string
        default: '3.11'

Performance Optimization¶

Docs-Only Detection¶

Automatically detected for these file types: - *.md, *.rst - Documentation - *.html, *.css, *.js - Web assets - _config.yml, _toc.yml - Documentation config - *.yml, *.yaml - Configuration files (non-workflow)

Selective Execution¶

Automatically processes only: - Changed notebooks in PR mode - Affected directories when requirements change - Dependencies based on file change analysis

Caching Optimization¶

The unified workflow includes: - Python environment caching - Package installation caching - Dependency resolution caching

Resource Management¶

Configure timeouts and limits:

# In the unified workflow (advanced users)
timeout-minutes: 120           # Maximum workflow runtime
max-parallel: 5               # Maximum parallel jobs

Error Handling¶

Continue on Error¶

For non-critical steps:

# In custom implementations
continue-on-error: true

Conditional Execution¶

Skip steps based on conditions:

# Example: Skip security scan for docs-only changes
if: needs.setup-matrix.outputs.docs-only != 'true'

Retry Logic¶

The unified workflow includes automatic retry for: - Git operations - Package installations - Network-dependent operations

Monitoring and Debugging¶

Workflow Summary¶

The unified workflow provides detailed summaries including: - Configuration used - Execution strategy (selective/full) - Performance metrics - Error details

Debug Mode¶

Enable verbose logging:

# Set in repository variables
ACTIONS_STEP_DEBUG: true
ACTIONS_RUNNER_DEBUG: true

Performance Metrics¶

Monitor these metrics in workflow summaries: - Execution time per notebook - Number of notebooks processed - Cache hit/miss ratios - Resource usage

Best Practices¶

Configuration Management¶

Start simple: Begin with basic configuration, add complexity gradually
Document choices: Comment configuration decisions in workflow files
Test thoroughly: Validate configuration with test PRs
Monitor performance: Watch for execution time and resource usage

Repository-Specific Customization¶

Use appropriate environments: Choose conda env that matches your domain
Enable relevant features: Only enable features you actually need
Configure triggers carefully: Avoid unnecessary workflow runs
Customize post-processing: Add domain-specific processing as needed

Maintenance¶

Regular updates: Keep workflow references current (@v1 or specific tags)
Monitor deprecation: Watch for deprecated configuration options
Performance review: Periodically review and optimize configuration
Documentation updates: Keep repository docs in sync with configuration