********* Use Cases ********* The ReDCaT Delivery Tools are intended for use internally by STScI staff to deliver and verify reference files for use in supporting instrument pipelines or the ETC. Instrument teams will use these tools to submit reference files to the ReDCaT Team. The ReDCaT Team will use these tools to verify, certify and then submit the files for ingestion into CRDS or ETC. Instrument Team Examples ======================== In order to submit a delivery to the ReDCaT Team, instrument team users will typically need to do a few things\: #. Create or modify a delivery form #. Verify and certify files according to ``fits``, ``json``, or ``asdf`` and CRDS standards #. Submit the files, form, and certification output to the ReDCaT Team Creating or Modifying a Delivery Form ------------------------------------- A blank copy of the delivery form that the ReDCaT Team uses to document and interpret reference file deliveries is included with the installation of the Delivery Tools. In an Ipython environment or script:: from reffile_delivery_tools.misc.read_deliveryform import DeliveryForm form = DeliveryForm() # blank form form = DeliveryForm('/path/to/delivery_form/delivery_form_name.txt') # Filled in form The benefit of accessing the form in this way is that it ensures that your form is *always compliant* with the version of these tools that you currently have installed, and you will be notified whether or not there is a new version of the delivery tools available. This is useful because updates are *mandatory* when there are changes to the questions on the delivery form. This strictness is necessary to keep the tools in compliance with ReDCaT policy and procedures. After ingesting the form, you can modify the answers to the questions via the ``data`` attribute:: d.data['1. Name of deliverer'] = John Doe To see the changes that you've made:: print(d) The delivery form data is a python ``OrderedDict`` object, which means that the questions are *always* in the same order. This means that you don't need to necessarily know what the exact question is if you've arranged your answers in the right order:: answers = [answer_1, answer_1a, answer_2, ...] for question, answer in zip(d.data.keys(), answers): d.data[question] = answer Once the form is ready, you can save the results to a file:: d.write('/path/to/new/form/delivery_form_name.txt') # The default is to overwrite the existing file .. note:: *Don't* fill out #16. Disk location and name of files. This will be automatically be filled in as part of the delivery process. Certifying Files ---------------- Users can verify/certify pipeline reference files independently from the submission using ``certify_reffiles``. This function is powered by ``crds.certify`` and will automatically set up your environment to the corrct CRDS configuration, run ``certify`` on the files, verify the file format and provide a certify log file. To certify your files:: from reffile_delivery_tools.instrument_team_tools.submit_delivery import certify_reffiles files_to_certify = [path_to_file1, path_to_file2, ...] observatory = 'hst' # or 'jwst' staging_location = 'ops' # Staging location also determines which CRDS server you certify against certify_reffiles(files_to_certify, observatory, staging_location) .. warning:: Currently the JWST-TEST configuration *is not supported*. Submitting Files to the ReDCaT Team ----------------------------------- Instrument team users can execute a delivery in two ways\: Via a script or IPython session:: from reffile_delivery_tools.instrument_team_tools.submit_delivery import submit_to_redcat submit_to_redcat(subject='New COS DISPTAB', staging_directory='ops') # If you're in the directory with the files Via the command-line:: submit_to_redcat -s "New COS DISPTAB" -d ops These will perform the following functions\: - Verify that the file format(s) adhere to standards - Certify the files according to CRDS standards - Create a staging area according to the instrument and delivery type (ops, test or etc) - Copy the files, delivery form and certification output to the staging area - Send an email notification to the ReDCaT Team - Create a JIRA issue on the Reference File Deliveries project for tracking the status of the delivery Note that the ability to include this step in an automated "delivery" pipeline is currently fairly limited due to the JIRA issue creation step, which is done from the user's JIRA account, and so requires login. ReDCaT Team Examples ==================== The installation for members of the ReDCaT Team includes everything listed above with the addition of ``redcat_tools`` and ``redcat_scripts``. As a part of the ReDCaT team, users will need to submit files to CRDS and store the results of the delivery in an "archive" area. Typically these particular tools are done via the command line for ReDCaT members, so those are the cases that will be the focus here. Submitting to CRDS ------------------ Submissions to CRDS are typically done from the staged delivery directory that's located under ``/grp/redcat/staging/``:: deliver_to_crds And that's it (apart from then entering your CRDS password when prompted)! For most purposes, the default arguments will be sufficient. However, if you need to make the delivery from a location that's *not* the staging area, you can specify the staging directory with the ``delivery_directory`` argument:: deliver_to_crds --delivery_directory /path/to/the/staged/delivery Storing Delivery Results ------------------------ Once a submission is completed, the results (which include the certify output, a log of the submission to CRDS and the delivery form) need to be stored in the appropriate "record-keeping" directory. This is executed from the command-line with:: store_delivery_results Just like in the case of ``deliver_to_crds``, this step is usually completed from the staged delivery location, but you can execute the delivery from elsewhere by specifying the ``delivery_directory`` argument:: store_delivery_results --delivery_directory /path/to/the/staged/delivery