Running Individual Pipeline Steps

🚨 DEPRECATED NOTEBOOK

This notebook is scheduled for deprecation on 2026-07-26.

Please migrate to newer alternatives or contact maintainers before using this notebook in production.

This warning was automatically generated during documentation build.

Running Individual Pipeline Steps#

The NIRISS WFSS pipeline notebook in the jwst-pipeline-notebooks repository shows similar material to this notebook, and will continue to be maintained after this notebook is deprecated.

This notebook walks through calibrating the data with individual pipeline steps (AssignWCS and FlatField) rather than running the entire pipeline stage.

Use case: When using a package outside of the standard JWST pipeline, there may be certain steps that are still helpful to utilize within the JWST pipeline. Here we show the most commonly run individual steps, AssignWCS and FlatField. AssignWCS determines and stores the WCS (World Coordinate System) information, and FlatField removes detector features.
Data: JWST/NIRISS images and spectra from program 2079.
Tools: astropy, crds, glob, jwst, matplotlib, numpy, os, urllib, zipfile
Cross-instrument: NIRISS

Content

Author: Rachel Plesha (rplesha@stsci.edu), Camilla Pacifici (cpacifici@stsci.edu)
Last modified: May 2024
Last tested: This notebook was last tested with JWST pipeline version 1.12.5 and the CRDS context jwst_1229.pmap.

Imports & Data Setup#

# Update the CRDS path to your local directory
%env CRDS_PATH=crds_cache
%env CRDS_SERVER_URL=https://jwst-crds.stsci.edu

import os
import glob
import urllib
import zipfile
import numpy as np

from astropy.io import fits

from matplotlib import pyplot as plt
%matplotlib inline

from jwst.assign_wcs import AssignWcsStep
from jwst.flatfield import FlatFieldStep

Check what version of the JWST pipeline you are using. To see what the latest version of the pipeline is available or install a previous version, check GitHub. Also verify what CRDS context you are using. CRDS documentation explains how to set a specific context to use in the JWST pipeline. If either of these values are different from the last tested note above there may be differences in this notebook.

import jwst
import crds
print('JWST Pipeliene Version:', jwst.__version__)
print('CRDS Context:', crds.get_context_name('jwst'))

Data setup#

Here we download and open the zip file that contains all of the rate files, and we also create an output directory for the calibrated files if it does not already exist

data_dir_in = 'data' # directory where the rate files should be
data_dir_out = 'data/calibrated_steps/' # directory where to save the calibrate files

# if the directory does not exist that you want to save out to, make that directory first
for datadir in [data_dir_in, data_dir_out]:
    if not os.path.exists(datadir):
        os.makedirs(datadir)

# Download uncalibrated data from Box into the data directory:
boxlink = 'https://data.science.stsci.edu/redirect/JWST/jwst-data_analysis_tools/niriss_wfss_advanced/niriss_wfss_extra_input.zip'
boxfile = os.path.basename(boxlink)
urllib.request.urlretrieve(boxlink, boxfile)

zf = zipfile.ZipFile(boxfile, 'r')
zf.extractall(path=data_dir_in)

# move the files downloaded from the box file into the top level data directory
box_download_dir = os.path.join(data_dir_in, boxfile.split('.zip')[0])
for filename in glob.glob(os.path.join(box_download_dir, '*')):
    if '.csv' in filename:
        # move to the current directory
        os.rename(filename, os.path.basename(filename))
    else:
        # move to the data directory 
        os.rename(filename, os.path.join(data_dir_in, os.path.basename(filename)))

# remove unnecessary files now
os.remove(boxfile)
os.rmdir(box_download_dir)

Running Individual Pipeline Steps#

While you could look at the rate images, instead consider running the files through the assign_wcs and flat_field steps of the pipeline to clean up detector artifacts.

Assign WCS Step#

The assign_wcs step of the pipeline is a critical part to obtaining the correct spectral trace cutouts for WFSS images. To read more about the step, visit the AssignWCS description page.

# Run assign_wcs; we are only running on one file for demonstration here
ratefile = os.path.join(data_dir_in, 'jw02079004002_02101_00001_nis_rate.fits')
result = AssignWcsStep.call(ratefile, output_dir=data_dir_out, save_results=True)

# A quick sanity check to ensure that the files were calibrated.
# if this is zero, check the log message above for any errors that may have occurred during the calibration
wcsstep_files = glob.glob(os.path.join(data_dir_out, '*assignwcsstep*'))
print(len(wcsstep_files), 'assignwcsstep files written') # 1 file should have been written

Flat Field Step#

After the assignwcs file is run, we then want to run the flat_field step of the pipeline which removes detector artifacts using the flat field reference files. To read more about the step, visit the FlatField description page.

# Run flat_field
for wcsfile in wcsstep_files:
    result = FlatFieldStep.call(wcsfile, output_dir=data_dir_out, save_results=True)

# A quick sanity check to ensure that the files were calibrated.
# if this is zero, check the log message above for any errors that may have occurred during the calibration
flatfield_files = glob.glob(os.path.join(data_dir_out, '*flatfieldstep*'))
print(len(flatfield_files), 'flatfieldstep files written') # 1 file should have been written (matching the wcs step)

Compare Rate vs. Flat fielded Data#

Running the cell below shows the same direct image from just the rate file versus when the flat_field step of the pipeline is run. Some detector artifacts are noticably gone such as the cross-hatching in the bottom right and middle of the detector.

There are remaining optical artifacts to be aware of on the NIRISS known issues page such as the 1/f noise.

test_rate_file = ratefile # look at a direct image for this comparision
test_flat_file = os.path.join(data_dir_out, os.path.basename(test_rate_file).replace('rate.fits', 'flatfieldstep.fits'))

plot_files = [test_rate_file, test_flat_file]
plot_titles = ['Rate File', 'Flat Corrected File']

fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(15, 15))
fig.suptitle(f"{os.path.basename(test_rate_file).split('_rate')[0]}\n{fits.getval(test_rate_file, 'PUPIL')}", x=0.5, y=0.72)

for filename, title, ax in zip(plot_files, plot_titles, [ax1, ax2]):
    with fits.open(filename) as hdu:
        # fill in the nan values from the bad pixels with zero; otherwise a single, non-dithered image is impossible to really see
        data = hdu[1].data
        data[np.isnan(data)] = 0
        
        ax.imshow(data, vmin=0.2, vmax=1.2, origin='lower')
        ax.set_title(title)
Space Telescope Logo