Use Cases

The ReDCaT Delivery Tools are intended for use internally by STScI staff to deliver and verify reference files for use in supporting instrument pipelines or the ETC.

Instrument teams will use these tools to submit reference files to the ReDCaT Team.

The ReDCaT Team will use these tools to verify, certify and then submit the files for ingestion into CRDS or ETC.

Instrument Team Examples

In order to submit a delivery to the ReDCaT Team, instrument team users will typically need to do a few things:

  1. Create or modify a delivery form
  2. Verify and certify files according to fits, json, or asdf and CRDS standards
  3. Submit the files, form, and certification output to the ReDCaT Team

Creating or Modifying a Delivery Form

A blank copy of the delivery form that the ReDCaT Team uses to document and interpret reference file deliveries is included with the installation of the Delivery Tools. In an Ipython environment or script:

from reffile_delivery_tools.misc.read_deliveryform import DeliveryForm

form = DeliveryForm()  # blank form

form = DeliveryForm('/path/to/delivery_form/delivery_form_name.txt')  # Filled in form

The benefit of accessing the form in this way is that it ensures that your form is always compliant with the version of these tools that you currently have installed, and you will be notified whether or not there is a new version of the delivery tools available. This is useful because updates are mandatory when there are changes to the questions on the delivery form. This strictness is necessary to keep the tools in compliance with ReDCaT policy and procedures.

After ingesting the form, you can modify the answers to the questions via the data attribute:

d.data['1. Name of deliverer'] = John Doe

To see the changes that you’ve made:

print(d)

The delivery form data is a python OrderedDict object, which means that the questions are always in the same order. This means that you don’t need to necessarily know what the exact question is if you’ve arranged your answers in the right order:

answers = [answer_1, answer_1a, answer_2, ...]
for question, answer in zip(d.data.keys(), answers):
    d.data[question] = answer

Once the form is ready, you can save the results to a file:

d.write('/path/to/new/form/delivery_form_name.txt')  # The default is to overwrite the existing file

Note

Don’t fill out #16. Disk location and name of files. This will be automatically be filled in as part of the delivery process.

Certifying Files

Users can verify/certify pipeline reference files independently from the submission using certify_reffiles. This function is powered by crds.certify and will automatically set up your environment to the corrct CRDS configuration, run certify on the files, verify the file format and provide a certify log file. To certify your files:

from reffile_delivery_tools.instrument_team_tools.submit_delivery import certify_reffiles

files_to_certify = [path_to_file1, path_to_file2, ...]
observatory = 'hst'  # or 'jwst'
staging_location = 'ops'  # Staging location also determines which CRDS server you certify against
certify_reffiles(files_to_certify, observatory, staging_location)

Warning

Currently the JWST-TEST configuration is not supported.

Submitting Files to the ReDCaT Team

Instrument team users can execute a delivery in two ways:

Via a script or IPython session:

from reffile_delivery_tools.instrument_team_tools.submit_delivery import submit_to_redcat

submit_to_redcat(subject='New COS DISPTAB', staging_directory='ops')  # If you're in the directory with the files

Via the command-line:

submit_to_redcat -s "New COS DISPTAB" -d ops

These will perform the following functions:

  • Verify that the file format(s) adhere to standards
  • Certify the files according to CRDS standards
  • Create a staging area according to the instrument and delivery type (ops, test or etc)
  • Copy the files, delivery form and certification output to the staging area
  • Send an email notification to the ReDCaT Team
  • Create a JIRA issue on the Reference File Deliveries project for tracking the status of the delivery

Note that the ability to include this step in an automated “delivery” pipeline is currently fairly limited due to the JIRA issue creation step, which is done from the user’s JIRA account, and so requires login.

ReDCaT Team Examples

The installation for members of the ReDCaT Team includes everything listed above with the addition of redcat_tools and redcat_scripts.

As a part of the ReDCaT team, users will need to submit files to CRDS and store the results of the delivery in an “archive” area.

Typically these particular tools are done via the command line for ReDCaT members, so those are the cases that will be the focus here.

Submitting to CRDS

Submissions to CRDS are typically done from the staged delivery directory that’s located under /grp/redcat/staging/:

deliver_to_crds

And that’s it (apart from then entering your CRDS password when prompted)!

For most purposes, the default arguments will be sufficient. However, if you need to make the delivery from a location that’s not the staging area, you can specify the staging directory with the delivery_directory argument:

deliver_to_crds --delivery_directory /path/to/the/staged/delivery

Storing Delivery Results

Once a submission is completed, the results (which include the certify output, a log of the submission to CRDS and the delivery form) need to be stored in the appropriate “record-keeping” directory. This is executed from the command-line with:

store_delivery_results

Just like in the case of deliver_to_crds, this step is usually completed from the staged delivery location, but you can execute the delivery from elsewhere by specifying the delivery_directory argument:

store_delivery_results --delivery_directory /path/to/the/staged/delivery