Understanding CRDS and How to Select Calibration Reference Files

Understanding CRDS and How to Select Calibration Reference Files#

Kernel Information and Read-Only Status#

To run this notebook, please select “Roman Research Nexus {VERSION}” kernel at the top right of your window. For example “Roman Research Nexus 2026.2”.

This notebook is read-only. You can run cells and make edits, but you must save changes to a different location. We recommend saving the notebook within your home directory, or to a new folder within your home (e.g. file > save notebook as > my-nbs/nb.ipynb). Note that a directory must exist before you attempt to add a notebook to it.

Introduction#

This notebook will help you understand how the Calibration Reference Data System (CRDS) works, how the Roman Calibration Pipeline interacts with this subsystem to get the best reference files for your data, and how to retrieve these files directly from CRDS

Local Run Settings#

If you want to run the notebook in your local machine, refer to the information in local installation instructions before proceeding with the notebook. The instructions provide important information about setting up your environment and installing dependencies.

Imports#

Libraries used:

  • os for operating system functions

  • copy for making copies of Python objects

  • crds for access to calibration reference files

  • numpy for array manipulation

  • roman_datamodels for opening Roman WFI ASDF files

import os
import copy
from pathlib import Path

import numpy as np
import roman_datamodels as rdm
#import s3fs

The Calibration Reference Data System (CRDS)#

The Roman Calibration Pipeline (RomanCal), uses calibration reference and parameter files from the CRDS. These reference files, developed and validated by STScI’s Science Operations Center, are continually updated as new WFI data become available. CRDS assigns the most appropriate reference file for each calibration step using metadata keywords and file-specific matching criteria. To use the best-available reference files for an observation, no action is needed as RomanCal will query for the best reference files for each calibration step in the pipeline.

To do this independently we will use the crds Python application programming interface (API); however, the CRDS webserver can also be accessed to browse calibration reference files in a tabular interface. Note that there are multiple CRDS servers, though most users will interact with the Operations (OPS) instance. Please be sure to navigate to the correct webserver for the instance in which you are interested.

For more details, see the RDox page on CRDS for Roman WFI and the CRDS documentation.

import crds
from crds.client import api

Check your environment. If running the notebook in Nexus, these CRDS environment variables would be already configured. If running locally and not currently set up, don’t worry, we will select the CRDS context later in the notebook.

print("CRDS_SERVER_URL:", os.environ.get('CRDS_SERVER_URL'))
print("CRDS_CONTEXT:", os.environ.get('CRDS_CONTEXT'))
print("CRDS_PATH:", os.environ.get('CRDS_PATH'))
print('CRDS_OBSERVATORY:|', os.environ.get('CRDS_OBSERVATORY'))

CRDS_SERVER_URL: https://roman-crds.stsci.edu
CRDS_CONTEXT: roman-edit
CRDS_PATH: /home/runner/crds_cache
CRDS_OBSERVATORY:| None

Retrieving Reference Files#

As you run the exposure or mosaic pipeline, the most up-to-date reference files will be automatically selected for each step. However, if you would like to use a specific reference file, these can be retrieved through the crds Python API and stored localy to be used later in a particular calibraton step (more on that later). Let’s begin with how to access reference files from CRDS.

First, let’s begin with how to access reference files from CRDS. For that we will use the crds.getrecommendations() function. This function returns a dictionary with the basenames of the reference files that match the criteria that you supply. In this case, the files are not downloaded.

In this function, the selection criteria are specified in a dictionary of key-value pairs, where eacb Roman WFI metadata keyword in the dictionary is all-caps and always begins with “ROMAN.META.”. The remaining parts of the string correspond to the metadata keyword locations in the science data file schema. To lear more about Science Products Schemas refer to the Roman Atribute Dictionary.

Different reference file types require different combinations of science metadata to match to the reference files. In general, all reference file types will require the instrument name (“INSTRUMENT.NAME”) and start time (“EXPOSURE.START_TIME”). Most file types require the detector name (“INSTRUMENT.DETECTOR”), and some file types require the exposure type (“EXPOSURE.TYPE”) or optical element (“INSTRUMENT.OPTICAL_ELEMENT”).

For example, for the mask and flat files, the required keywords are:

  • mask

    • ROMAN.META.INSTRUMENT.NAME

    • ROMAN.META.INSTRUMENT.DETECTOR

    • ROMAN.META.EXPOSURE.START_TIME

  • flat

    • ROMAN.META.INSTRUMENT.NAME

    • ROMAN.META.INSTRUMENT.DETECTOR

    • ROMAN.META.INSTRUMENT.OPTICAL_ELEMENT

    • ROMAN.META.EXPOSURE.START_TIME

These keywords may be combined into a single dictionary to find multiple reference file types using crds.getreferences(). This function returns full local paths and downloads or caches if missing. For example, if you would like to find the name of the dark and flat reference files used by the pipeline, and download them, you could run the following example:

meta = {'ROMAN.META.INSTRUMENT.NAME': 'WFI',
        'ROMAN.META.INSTRUMENT.DETECTOR': 'WFI01',
        'ROMAN.META.INSTRUMENT.OPTICAL_ELEMENT': 'F158',
        'ROMAN.META.EXPOSURE.START_TIME': '2026-01-01 00:00:00'
       }

ref_files = crds.getrecommendations(meta, reftypes=['mask', 'flat'], observatory='roman')

CRDS - INFO -  Syncing 30 files

CRDS - INFO -  Fetching  /home/runner/crds_cache/mappings/roman/roman_wfi_specpsf_0001.rmap      572 bytes  (1 / 30 files) (0 / 77.6 K bytes)

CRDS - INFO -  Fetching  /home/runner/crds_cache/mappings/roman/roman_wfi_skycells_0002.rmap      712 bytes  (2 / 30 files) (572 / 77.6 K bytes)

CRDS - INFO -  Fetching  /home/runner/crds_cache/mappings/roman/roman_wfi_sflat_0001.rmap      548 bytes  (3 / 30 files) (1.3 K / 77.6 K bytes)

CRDS - INFO -  Fetching  /home/runner/crds_cache/mappings/roman/roman_wfi_saturation_0003.rmap    2.4 K bytes  (4 / 30 files) (1.8 K / 77.6 K bytes)

CRDS - INFO -  Fetching  /home/runner/crds_cache/mappings/roman/roman_wfi_relflux_0001.rmap      555 bytes  (5 / 30 files) (4.3 K / 77.6 K bytes)

CRDS - INFO -  Fetching  /home/runner/crds_cache/mappings/roman/roman_wfi_refpix_0005.rmap    2.3 K bytes  (6 / 30 files) (4.8 K / 77.6 K bytes)

CRDS - INFO -  Fetching  /home/runner/crds_cache/mappings/roman/roman_wfi_readnoise_0007.rmap    3.4 K bytes  (7 / 30 files) (7.1 K / 77.6 K bytes)

CRDS - INFO -  Fetching  /home/runner/crds_cache/mappings/roman/roman_wfi_photom_0004.rmap    2.3 K bytes  (8 / 30 files) (10.5 K / 77.6 K bytes)

CRDS - INFO -  Fetching  /home/runner/crds_cache/mappings/roman/roman_wfi_optmodel_0001.rmap      586 bytes  (9 / 30 files) (12.9 K / 77.6 K bytes)

CRDS - INFO -  Fetching  /home/runner/crds_cache/mappings/roman/roman_wfi_matable_0004.rmap      690 bytes  (10 / 30 files) (13.4 K / 77.6 K bytes)

CRDS - INFO -  Fetching  /home/runner/crds_cache/mappings/roman/roman_wfi_mask_0003.rmap    2.3 K bytes  (11 / 30 files) (14.1 K / 77.6 K bytes)

CRDS - INFO -  Fetching  /home/runner/crds_cache/mappings/roman/roman_wfi_linearity_0005.rmap    2.4 K bytes  (12 / 30 files) (16.4 K / 77.6 K bytes)

CRDS - INFO -  Fetching  /home/runner/crds_cache/mappings/roman/roman_wfi_ipc_0003.rmap    3.3 K bytes  (13 / 30 files) (18.8 K / 77.6 K bytes)

CRDS - INFO -  Fetching  /home/runner/crds_cache/mappings/roman/roman_wfi_inverselinearity_0005.rmap    2.6 K bytes  (14 / 30 files) (22.1 K / 77.6 K bytes)

CRDS - INFO -  Fetching  /home/runner/crds_cache/mappings/roman/roman_wfi_integralnonlinearity_0002.rmap    2.8 K bytes  (15 / 30 files) (24.7 K / 77.6 K bytes)

CRDS - INFO -  Fetching  /home/runner/crds_cache/mappings/roman/roman_wfi_gain_0004.rmap    2.3 K bytes  (16 / 30 files) (27.5 K / 77.6 K bytes)

CRDS - INFO -  Fetching  /home/runner/crds_cache/mappings/roman/roman_wfi_flat_0006.rmap    8.5 K bytes  (17 / 30 files) (29.7 K / 77.6 K bytes)

CRDS - INFO -  Fetching  /home/runner/crds_cache/mappings/roman/roman_wfi_etc_0002.rmap    2.3 K bytes  (18 / 30 files) (38.2 K / 77.6 K bytes)

CRDS - INFO -  Fetching  /home/runner/crds_cache/mappings/roman/roman_wfi_epsf_0004.rmap   16.3 K bytes  (19 / 30 files) (40.5 K / 77.6 K bytes)

CRDS - INFO -  Fetching  /home/runner/crds_cache/mappings/roman/roman_wfi_dustmap_0003.rmap      855 bytes  (20 / 30 files) (56.8 K / 77.6 K bytes)

CRDS - INFO -  Fetching  /home/runner/crds_cache/mappings/roman/roman_wfi_distortion_0002.rmap    3.6 K bytes  (21 / 30 files) (57.7 K / 77.6 K bytes)

CRDS - INFO -  Fetching  /home/runner/crds_cache/mappings/roman/roman_wfi_detectorstatus_0002.rmap      773 bytes  (22 / 30 files) (61.2 K / 77.6 K bytes)

CRDS - INFO -  Fetching  /home/runner/crds_cache/mappings/roman/roman_wfi_darkdecaysignal_0002.rmap      783 bytes  (23 / 30 files) (62.0 K / 77.6 K bytes)

CRDS - INFO -  Fetching  /home/runner/crds_cache/mappings/roman/roman_wfi_dark_0011.rmap    4.4 K bytes  (24 / 30 files) (62.8 K / 77.6 K bytes)

CRDS - INFO -  Fetching  /home/runner/crds_cache/mappings/roman/roman_wfi_area_0002.rmap    2.3 K bytes  (25 / 30 files) (67.2 K / 77.6 K bytes)

CRDS - INFO -  Fetching  /home/runner/crds_cache/mappings/roman/roman_wfi_apcorr_0003.rmap    2.3 K bytes  (26 / 30 files) (69.5 K / 77.6 K bytes)

CRDS - INFO -  Fetching  /home/runner/crds_cache/mappings/roman/roman_wfi_abvegaoffset_0002.rmap    2.5 K bytes  (27 / 30 files) (71.8 K / 77.6 K bytes)

CRDS - INFO -  Fetching  /home/runner/crds_cache/mappings/roman/roman_wfi_absflux_0001.rmap      555 bytes  (28 / 30 files) (74.3 K / 77.6 K bytes)

CRDS - INFO -  Fetching  /home/runner/crds_cache/mappings/roman/roman_wfi_0059.imap    2.4 K bytes  (29 / 30 files) (74.8 K / 77.6 K bytes)

CRDS - INFO -  Fetching  /home/runner/crds_cache/mappings/roman/roman_0061.pmap       344 bytes  (30 / 30 files) (77.3 K / 77.6 K bytes)

The ref_files variable now contains a dictionary for each of the reference file types you requested (MASK and FLAT). These are the reference files that correspond to a science observation taken at midnight UTC on January 1, 2026 in the WFI imaging mode with optical element F158 and detector WFI01. Let’s take a look at the names of the files CRDS returned:

ref_files

{'mask': 'roman_wfi_mask_0030.asdf', 'flat': 'roman_wfi_flat_0166.asdf'}

We can also use crds.getreferences() to accomplish the same thing; however, getreferences() goes one step further beyond getrecommendations() and will download the reference files if they are not already in your local cache. Using the same example as above:

meta = {'ROMAN.META.INSTRUMENT.NAME': 'WFI',
        'ROMAN.META.INSTRUMENT.DETECTOR': 'WFI01',
        'ROMAN.META.INSTRUMENT.OPTICAL_ELEMENT': 'F158',
        'ROMAN.META.EXPOSURE.START_TIME': '2026-01-01 00:00:00'
       }

ref_files = crds.getreferences(meta, reftypes=['mask', 'flat'], observatory='roman')

CRDS - INFO -  Syncing 2 files

CRDS - INFO -  Fetching  /home/runner/crds_cache/references/roman/wfi/roman_wfi_mask_0030.asdf   67.1 M bytes  (1 / 2 files) (0 / 267.7 M bytes)

CRDS - INFO -  Fetching  /home/runner/crds_cache/references/roman/wfi/roman_wfi_flat_0166.asdf  200.5 M bytes  (2 / 2 files) (67.1 M / 267.7 M bytes)

And once again we can examine the output of ref_files:

ref_files

{'mask': '/home/runner/crds_cache/references/roman/wfi/roman_wfi_mask_0030.asdf',
 'flat': '/home/runner/crds_cache/references/roman/wfi/roman_wfi_flat_0166.asdf'}

This time, ref_files contains the path to the file in the local cache (controlled by the CRDS_PATH environment variable) since we did not simply ask for the file name but also checked if the file was in our cache, and if it was not then we downloaded it.

CRDS Mapping Files#

CRDS organizes the reference files using mapping files: the PMAP file, the IMAP files, and the RMAP files. The first and most important mapping file is the PMAP file (commonly referred to as the CRDS context). This file is set by the CRDS_CONTEXT environment variable. It provides a way to identify the full set of reference files that each step of the RomanCal pipeline will use at a given time. By using the reference files tied to the same PMAP, you ensure compatibility between reference files.

If not already set, we can select a particular CRDS context with the following command.

pmap = os.environ['CRDS_CONTEXT']='roman_0058.pmap'

Let’s check again our environment variables

print(f"CRDS server location: {os.environ.get('CRDS_SERVER_URL')}")
print(f"CRDS context file: {os.environ.get('CRDS_CONTEXT')}")

CRDS server location: https://roman-crds.stsci.edu
CRDS context file: roman_0058.pmap

We can see that the PMAP was correctly set.

The IMAP is the second tier of mapping files. It contains the full set of reference types applicable to a given instrument and available for use by RomanCal. For WFI, a particular version of the IMAP provides a list of all the reference file types used by RomanCal to calibrate that instrument, along with the RMAP versions part of the selected CRDS context.

The RMAP is the third tier of mapping files. It provides the full list of reference files active with a given CRDS context, along with the selection parameters for that particular reference file type, which matches them to science observations. An RMAP includes the versions of the reference file that is part of that CRDS context.

From the user point of view, only the PMAP or CRDS context file needs to be set. Note, however, that the CRDS context is also CRDS server-dependent and is set by the environment variable CRDS_SERVER_URL.

Note that in the above case, we selected a particular CRDS context. To revert the above selection to the default CRDS context, we use the crds.get_default_context function. We then check we have the correct CRDS context.

# The following line selects the default pmap for the Roman observatory.
pmap = crds.get_default_context("roman")

# We check we have the correct CRDS context
print("Selected pmap:", pmap)

# Typical location of the pmap in the cache
cache = Path(os.environ.get("CRDS_PATH", ""))
print(cache)
possible_paths = [
    cache / "mappings" / "roman" / pmap,
    cache / "mappings" / pmap,
]

local_pmap = any(p.exists() for p in possible_paths)
if local_pmap:
    print("Found at:", next(p for p in possible_paths if p.exists()))
else:
    print(f"Downloading mappings rules for {pmap} ...")
    api.dump_mappings(pmap)
    

Selected pmap: roman_0061.pmap
/home/runner/crds_cache
Found at: /home/runner/crds_cache/mappings/roman/roman_0061.pmap

Now let’s explore the set of mappings associated with our selected CRDS context.

ctx = crds.get_symbolic_mapping(pmap)

maps = ctx.mapping_names()

print(f"\nTotal mappings: {len(maps)}")
print("\nFirst 20 mappings:")
for m in sorted(maps)[:20]:
    print(m)

Total mappings: 30

First 20 mappings:
roman_0061.pmap
roman_wfi_0059.imap
roman_wfi_absflux_0001.rmap
roman_wfi_abvegaoffset_0002.rmap
roman_wfi_apcorr_0003.rmap
roman_wfi_area_0002.rmap
roman_wfi_dark_0011.rmap
roman_wfi_darkdecaysignal_0002.rmap
roman_wfi_detectorstatus_0002.rmap
roman_wfi_distortion_0002.rmap
roman_wfi_dustmap_0003.rmap
roman_wfi_epsf_0004.rmap
roman_wfi_etc_0002.rmap
roman_wfi_flat_0006.rmap
roman_wfi_gain_0004.rmap
roman_wfi_integralnonlinearity_0002.rmap
roman_wfi_inverselinearity_0005.rmap
roman_wfi_ipc_0003.rmap
roman_wfi_linearity_0005.rmap
roman_wfi_mask_0003.rmap

For more information on the mapping rules, see the readthedocs documentation.

Now let’s isolate the names of the MASK files in the associated RMAP.

for m in maps:
    if 'mask' in m:
        mask_rmap = m 
        break
    else:
        pass

masks = crds.rmap.load_mapping(mask_rmap)

Now that we have loaded up the list of MASK files in the applicable RMAP, let’s take a look at it using the todict() method to turn it into a dictionary we can more easily visualize:

masks.todict()

{'header': {'classes': ('Match', 'UseAfter'),
  'derived_from': 'roman_wfi_mask_0002.rmap',
  'file_ext': '.asdf',
  'filekind': 'mask',
  'filetype': 'mask',
  'instrument': 'wfi',
  'ld_tpn': 'wfi_mask.tpn',
  'mapping': 'reference',
  'name': 'roman_wfi_mask_0003.rmap',
  'observatory': 'roman',
  'parkey': (('ROMAN.META.INSTRUMENT.DETECTOR',),
   ('ROMAN.META.EXPOSURE.START_TIME',)),
  'sha1sum': '47d1c37517e5e40cd13240f9ca432cf644b3994a',
  'suffix': 'mask',
  'text_descr': 'bad pixel mask',
  'tpn': 'wfi_mask.tpn'},
 'text_descr': 'Bad Pixel Mask',
 'parameters': ('ROMAN.META.INSTRUMENT.DETECTOR', 'USEAFTER', 'REFERENCE'),
 'selections': [('WFI01', '2023-01-01 00:00:00', 'roman_wfi_mask_0030.asdf'),
  ('WFI02', '2023-01-01 00:00:00', 'roman_wfi_mask_0029.asdf'),
  ('WFI03', '2023-01-01 00:00:00', 'roman_wfi_mask_0026.asdf'),
  ('WFI04', '2023-01-01 00:00:00', 'roman_wfi_mask_0022.asdf'),
  ('WFI05', '2023-01-01 00:00:00', 'roman_wfi_mask_0021.asdf'),
  ('WFI06', '2023-01-01 00:00:00', 'roman_wfi_mask_0024.asdf'),
  ('WFI07', '2023-01-01 00:00:00', 'roman_wfi_mask_0032.asdf'),
  ('WFI08', '2023-01-01 00:00:00', 'roman_wfi_mask_0034.asdf'),
  ('WFI09', '2023-01-01 00:00:00', 'roman_wfi_mask_0020.asdf'),
  ('WFI10', '2023-01-01 00:00:00', 'roman_wfi_mask_0028.asdf'),
  ('WFI11', '2023-01-01 00:00:00', 'roman_wfi_mask_0023.asdf'),
  ('WFI12', '2023-01-01 00:00:00', 'roman_wfi_mask_0035.asdf'),
  ('WFI13', '2023-01-01 00:00:00', 'roman_wfi_mask_0036.asdf'),
  ('WFI14', '2023-01-01 00:00:00', 'roman_wfi_mask_0025.asdf'),
  ('WFI15', '2023-01-01 00:00:00', 'roman_wfi_mask_0033.asdf'),
  ('WFI16', '2023-01-01 00:00:00', 'roman_wfi_mask_0027.asdf'),
  ('WFI17', '2023-01-01 00:00:00', 'roman_wfi_mask_0019.asdf'),
  ('WFI18', '2023-01-01 00:00:00', 'roman_wfi_mask_0031.asdf')]}

As we can see, the parkey key in the RMAP tells us the matching criteria unique to this reference file type from the dictionary we created when we used crds.getrecommendations() and crds.getreferences(). Also notice that the parameters field tells us the column names for the selections part of the dictionary. In this case, we see they include:

"ROMAN.META.INSTRUMENT.DETECTOR"
"USEAFTER"
"REFERENCE".

The REFERENCE column is simply the name of the reference file that matches those critiera. The USEAFTER date is the date after which a file should be used for a science observation. CRDS will match the file with the closest preceding USEAFTER date to the science observation date (given by “ROMAN.META.EXPOSURE.START_TIME”).

Download a File by Name#

If you know the specific reference file name and would like to download it directly from the CRDS on-premise server, you can call it via a command line option:

crds sync --files <filename> --output-dir=<pathname>

where <filename> is the name of the reference file (e.g. “roman_wfi_mask_0022.asdf”) and <pathname> is the location where the file will be saved. Note: in the future, Roman reference files will also be available via an AWS S3 bucket, and these instructions will be updated to describe how to access them there.

To run this in a Jupyter notebook, we write out the command as a string and pass it to the command line script code:

cmd = f"crds.sync --files {ref_files['mask']}"
_ = crds.sync.SyncScript(cmd)()

CRDS - INFO -  Syncing explicitly listed files.

CRDS - INFO -  0 errors

CRDS - INFO -  0 warnings

CRDS - INFO -  1 infos

In this case, nothing happened since we had already previously downloaded this file into our cache using crds.getreferences(). But if you know the file name of a reference file that you want to retrieve from CRDS without using the matching criteria, the cell above would download the file to your cache. Simply replace {ref_files['mask']} in the cmd string with the name of the ASDF file.

mask = rdm.open(ref_files['mask'])
mask.info()

root (AsdfObject)
├─asdf_library (Software)
│ ├─author (str): The ASDF Developers
│ ├─homepage (str): http://github.com/asdf-format/asdf
│ ├─name (str): asdf
│ └─version (str): 3.3.0
├─history (AsdfDictNode)
│ └─extensions (AsdfListNode)
│   ├─0 (ExtensionMetadata) ...
│   ├─1 (ExtensionMetadata) ...
│   └─2 (ExtensionMetadata) ...
└─roman (MaskRef) # Mask Reference File Schema
  ├─meta (AsdfDictNode) # Common Reference File Metadata Properties
  │ ├─author (str): RFP # Author
  │ ├─description (str): Initial bad pixel masks for DQInit step in romancal.  Masks from NASA/GSFC # Description
  │ ├─instrument (AsdfDictNode) ...
  │ ├─origin (Origin): STSCI # Institution / Organization Name
  │ ├─pedigree (str): GROUND # Pedigree
  │ ├─reftype (str): MASK
  │ ├─telescope (Telescope): ROMAN # Telescope Name
  │ └─useafter (Time): 2023-01-01 00:00:00.000 # Use After Date
  └─dq (NDArrayType) # Mask Data Quality Array ...
Some nodes not shown.

What is Next#

  • Understand and Explore the Area Pixels Mask reference file.

  • Understand and Explore the Bad Pixels Mask reference file.

  • Understand and Explore the Dark reference file.

  • Understand and Explore the Flat Field reference file.

  • Understand and Explore the Distortion reference file.

  • Understand and Explore the Gain reference file.

  • Understand and Explore the Linearity reference file.

  • Understand and Explore the Photom reference file.

  • Understand and Explore the PSF/ePSF reference file.

  • Understand and Explore the Readnoise reference file.

  • Understand and Explore the Reference Pixels reference file.

  • Understand and Explore the Saturation reference file.

About this Notebook#

Author: R. Diaz, T. Desjardins.

Updated On: August 2026

Top of page roman_logo