{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# DrizzlePac Initialization\n", "\n", "
This Jupyter notebook discusses the steps necessary to set up your computing environment to use DrizzlePac. This is the first step before using any of the other DrizzlePac tutorials. The code cells in this notebook can be used to partially confirm that your environment is properly configured for DrizzlePac before proceeding to the other tutorials.\n", "\n" ] }, { "cell_type": "markdown", "id": "3927e66a", "metadata": {}, "source": [ "This notebook currently fails to execute, use as reference only" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Introduction\n", "\n", "DrizzlePac is a Python software package developed at STScI that is designed to align and combine HST images and is a successor to the older MultiDrizzle software. Since July 2012, all drizzled data products obtained from MAST are produced with AstroDrizzle. \n", "\n", "An abbreviation for Astrometric Drizzle, AstroDrizzle was designed from the ground-up to substantially improve the handling of distortion in the image header World Coordinate System. AstroDrizzle removes geometric distortion, corrects for sky background variations, flags cosmic-rays, and combines images with optional subsampling. Drizzled data products from MAST are generated for single visit associations only.\n", "\n", "To combine data from additional visits, TweakReg may be used to update the image WCS using matched source lists. Once the full set of images of a given target are properly aligned, they may be combined using AstroDrizzle.\n", "\n", "While the DrizzlePac software has been optimized to work with Hubble Space Telescope (HST) data, it can work with other types of data so long as the images adhere to the FITS standards for multi-extension files and for describing the World Coordinate System (WCS). It assumes that all distortions have been properly described in the WCS of the image, e.g. via the SIP distortion coefficients. [More details may be found here](http://www.stsci.edu/scientific-community/software/drizzlepac/features.html#h3-3-61c90abe-2d25-4c81-b5e0-450b9a59b17b) under the section 'Aligning to Non-HST Image'. \n", "\n", "In this notebook, we will demonstrate how to set up your environment to analyze HST data with links to several resources, as well as a demonstration of how to download observations from the HST archive and their associated reference files." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "
Before working with HST data, users are advised to consult both the data handbooks and the instrument team websites for the instrument of interest. Additional useful discussion of the drizzling algorithm and of how distortion information is represented in the image header may be found in the DrizzlePac Handbook.
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Notes about HST files:**\n", "\n", "HST images are stored using the Flexible Image Transport System (FITS). In particular, HST data use a multi-extension FITS (MEF) format. In this format, science data, error arrays, and data quality information for a single observation are stored as different extensions within a single file. The data format for the detector you are working with may be found in your instrument's data handbook, e.g. for [WFC3](https://hst-docs.stsci.edu/display/WFC3IHB) and [ACS](https://hst-docs.stsci.edu/display/ACSIHB/ACS+Instrument+Handbook).\n", "\n", "\n", "\n", "**Recent changes to ACS/WFC and WFC3/UVIS data quality flags:**\n", "\n", "In early 2017, the ACS instrument team changed the definition of data quality (DQ) flags populated in the calibrated FLT/FLC files. New calibration techniques now make it possible discern between unstable and stable hot pixels, the later of which are corrected by ‘calacs’ when subtracting the dark. Thus, pixels identified as hot and stable (DQ flag=16) may now be treated as 'good' data when drizzling, and those identified as unstable (DQ flag=32) should be treated as 'bad'. A new MDRIZTAB reference table (16r12191j_mdz.fits) was delivered in June 2017 and contains a set of default parameters for combining exposures with AstroDrizzle. With changes to the DQ flag definitions, the parameters 'driz_sep_bits' and 'final_bits', which define DQ flags for drizzle to ignore (e.g. to treat as good), are now set to a value of 336 (the sum of 16+64+256) so that stable hot pixels, warm pixels, and full-well saturated pixels will not be rejected when combining exposures. For details, see [ACS ISR 2017-05](http://www.stsci.edu/hst/acs/documents/isrs/isr1705.pdf).\n", "\n", "The WFC3 instrument team implemented a similar change to the DQ flag definitions in December 2018, and an updated MDRIZTAB reference file (2ck18260i_mdz.fits) reflects the new recommended drizzle parameter settings such that DQ flag values 16, 64, and 256 are treated as good pixels. These new flags are valid for UVIS observations obtained after Nov 08 2012, when the dark calibration program began using post-flash to mitigate hot pixel trailing due to poor charge transfer efficiency at low background levels. A description of the new UVIS bad pixel tables is described in [WFC3 ISR 2018-15](http://www.stsci.edu/hst/wfc3/documents/ISRs/WFC3-2018-15.pdf).\n", "\n", "This new set of DrizzlePac notebooks takes into account the updated DQ parameter settings for processing both ACS/WFC and WFC3/UVIS data. Similar updates to WFC3/IR data quality flags will be implemented in early 2019." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "[Top of Page](#title_ID)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Required Software\n", "\n", "Anaconda is a method of managing python package installations in various environments and AstroConda adds STScI specific astronomy-related Python packages. Users unfamiliar with Anaconda/Astroconda should see the [documentation](https://astroconda.readthedocs.io/en/latest/). One must first install Anaconda and then they can use our Astroconda channel within Anaconda. **For most users**, the standard Python 3 software stack that **does not include IRAF/PyRAF** is appropriate and should be used for DrizzlePac. \n", "\n", "**Please be sure you have the latest version of AstroConda.** The AstroConda page on [updating your software stack](https://astroconda.readthedocs.io/en/latest/updating.html) goes into more detail, but you can update your base `conda` installation and *everything* installed in your AstroConda environment (assuming it is named \"astroconda\") by typing in a bash shell:\n", "```\n", "conda deactivate\n", "conda update --all\n", "conda update -n astroconda --all\n", "conda activate astroconda\n", "```\n", "\n", "**Please complete the previous step even if you have JUST installed AstroConda or Conda as it is necessary to ensure all updates.**\n", "\n", "For example, a newer change has made the `activate/deactivate` command above begin with `conda` instead of `source`. You should update everything to ensure you stay up-to-date with the software. \n", "\n", "In addition to the default AstroConda configuration, many DrizzlePac examples will use [astroquery](https://astroquery.readthedocs.io/en/latest/) to obtain data from the Mikulski Archive for Space Telescopes (MAST). To install this, type the following in your bash shell:\n", "```\n", "conda install -c conda-forge astroquery\n", "```\n", "The `astroquery.mast` API has [additional documentation](https://astroquery.readthedocs.io/en/latest/mast/mast.html) for reference.\n", "\n", "Many of the notebooks make use of `ImageFileCollections` in `ccdproc` to inspect the image header. To install this, type:\n", "```\n", "conda install -c conda-forge ccdproc\n", "```\n", "For each of the tutorials, a 'requirements.txt' file is present in the directory along with the notebook which lists any other package dependencies. " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "
It should be noted that these notebooks are not tutorials for conda or python, but all steps needed to work with DrizzlePac are explained herein. There are in-depth introductions to conda available here and to python available here.
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "[Top of Page](#title_ID)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Imports \n", "\n", "These imports are required for this particular notebook and are used for checking your system's setup.\n", "\n", "- `astroquery.mast Observations`: Establishes a connection to a server to query MAST. Please try re-running the cell if the connection fails.\n", "- `os`: Python interface to the operating system.\n", "- `shutil`: Python shell utilities.\n", "- `stwcs`: HST world coordinate system (WCS) updates." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "from astroquery.mast import Observations\n", "import os\n", "import shutil\n", "import stwcs\n", "import subprocess " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "[Top of Page](#title_ID)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Retrieving Data from MAST\n", "\n", "The `astroquery.mast` API can be used to programatically retrieve data from the HST archive with the same kinds of filtering available through the MAST Portal. Here we show an example of how to retrieve a WFC3/UVIS observation of NGC104 by searching for the specific dataset name (`obs_id` in the search below). Note that we have set `obstype='all'` as some datasets may be classified as calibration if they were taken as part of an instrument calibration program even though they are perfectly useable for science. The default behavior is to search only for science observations." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "obs_table = Observations.query_criteria(obs_id='ib2j02n5q', obstype='all')\n", "download_tab = Observations.download_products(obs_table['obsid'], mrp_only=False, \n", " productSubGroupDescription=['FLC'])" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now let's move all of the files we just downloaded to the current working directory and remove the \"mastDownload/\" directory:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "for file in download_tab['Local Path']:\n", " os.rename(file, os.path.basename(file))\n", " \n", "shutil.rmtree('mastDownload')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "[Top of Page](#title_ID)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Reference Files\n", "\n", "HST data require reference files that specify calibration information. `Drizzlepac` also needs various reference files in order to calibrate properly, so it's important to follow these instructions to allow `DrizzlePac` access to these needed calibration files. \n", "\n", "There are many types of reference files for each detector, so care should be taken to ensure that all necessary reference files for a particular dataset have been retrieved. The path to the reference files for each instrument (e.g., WFC3, ACS, WFPC2) is indicated with the appropriate environment variable. For the instruments supported by DrizzlePac, these are:\n", "\n", "- ACS = jref\n", "- WFC3 = iref\n", "- WFPC2 = uref\n", "\n", "The Calibration Reference Data System (CRDS) both stores the reference files and determines the mapping of reference files to observations. The `crds` tool included in AstroConda can find and download the best reference files for a particular observation. The [documentation](https://hst-crds.stsci.edu/static/users_guide/index.html) for `crds` describes many of the more advanced options, but we will demonstrate here how to obtain updated reference file information stored in the FITS header of an observation and also download those files to a local directory.\n", "\n", "First we need to set some environment variables:\n", "- CRDS_SERVER_URL: Location of the CRDS server.\n", "- CRDS_PATH: Path to where reference files will be downloaded." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "os.environ['CRDS_SERVER_URL'] = 'https://hst-crds.stsci.edu'\n", "os.environ['CRDS_PATH'] = os.path.abspath(os.path.join('.', 'reference_files'))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "While the `crds.bestrefs` tool is also accessible inside of Python, it was designed with a command line interface in mind, therefore it is easiest to use it this way:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "subprocess.check_output('crds bestrefs --files ib2j02n5q_flc.fits --sync-references=1 --update-bestrefs', shell=True, stderr=subprocess.DEVNULL)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now that we have the reference files for this observation downloaded to a local directory called reference_files/, we need to tell the DrizzlePac software how to find these files. Our example dataset \"ib2j02n5q\" comes from WFC3, therefore we indicate the path to the associated reference files with the \"iref\" environment variable:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "os.environ['iref'] = os.path.abspath(os.path.join('.', 'reference_files', 'references', 'hst', 'wfc3')) + os.path.sep" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now the DrizzlePac software will be able to locate the reference files it requires in the case that the user needs to update the geometric distortion information in the image header. These files are the IDCTAB (Instrument Distortion Correction Table, `*idc.fits`), the D2IMFILE (Column Correction Reference File, `*d2i.fits`), and the NPOLFILE (Non-polynomial Offsets Reference File, `*npl.fits`). " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "[Top of Page](#title_ID)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Update World Coordinate System\n", "\n", "Before combining images with `drizzlepac.astrodrizzle` to make mosaics, the data may first need to be processed with `stwcs.updatewcs`. This task is required for populating older (pre-AstroDrizzle Archive pipeline data) WFC3, ACS, STIS, and WFPC2 images with linear and polynomial distortion correction information in a format compatible with AstroDrizzle. It is also required for data that the user wishes to update to use a more recent (or custom) set of distortion reference files that were downloaded from MAST at previously. \n", "\n", "The `updatewcs` task will update the header keywords with new WCS information and apply several distortion corrections from reference files. In general, data recently retrieved from MAST will have already had this step performed and does **not** need to be run. \n", "\n", "**WARNING: If you re-run an instrument calibration pipeline on raw data, or your files were retrieved from MAST long ago, you must run `updatewcs` or you will encounter errors.**\n", "\n", "As an example, if it were necessary to run `updatewcs` on our WFC3/UVIS file, we can update the WCS as follows:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "stwcs.updatewcs.updatewcs('ib2j02n5q_flc.fits', use_db=False)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Note that we have set `use_db=False` in our call to `updatewcs`. The Astrometric Database referenced in this call is a work in progress, and astrometric solutions are not yet available for all parts of the sky. The Astrometry Working Group at STScI has created this infrastructure in `updatewcs` to include multiple astrometric solutions as additional extensions to HST FITS files. The default behavior of `updatewcs` is `use_db=True`. In cases where it is left with the default value, warnings may appear while using the Astrometric Database with `updatewcs`, but your data are still properly prepared for `astrodrizzle`." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## About this Notebook\n", "\n", " Author: T. Desjardins, STScI ACS Team \n", " Updated: December 14, 2018" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "[Top of Page](#title_ID)" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.5.6" } }, "nbformat": 4, "nbformat_minor": 2 }