{ "cells": [ { "cell_type": "markdown", "id": "b0493e3e", "metadata": {}, "source": [ "# Large Downloads in `astroquery.mast`\n", "***\n", "For some programs stored in the MAST Archive, you may encounter issues when downloading data via the MAST Portal due to a large number of files. This applies particularly to JWST programs using Wide-Field Slitless Spectroscopy. It is preferable — and often, necessary — to use an API to get this data instead. In this tutorial, we'll use seemingly innocuous observations that expand into a considerable number of related files.\n", "\n", "To that end, this notebook will demonstrate:\n", "* Searching the MAST Portal for observations using the [astroquery.mast](https://astroquery.readthedocs.io/en/latest/mast/mast.html) API\n", "* Retreiving associated data products, without causing a timeout error\n", "* Downloading the desired subset of data products\n", "\n", "## Table of Contents\n", "* [Imports](#Imports)\n", "* [Search the MAST Archive](#Search-the-MAST-Archive)\n", "* [Retrieve Associated Products](#Retreive-Associated-Products)\n", "* [Filter and Download Products](#Filter-and-Download-Products)\n", "* [Further Reading](#Further-Reading)\n", "\n", "## Imports\n", "In order to run this notebook, we need: \n", "* `astroquery.mast` to access the MAST Archive\n", "* `astropy.table` to hold the results of our queries, combine them, and then filter them for unique products" ] }, { "cell_type": "code", "execution_count": 1, "id": "944d0474", "metadata": { "execution": { "iopub.execute_input": "2024-02-20T21:00:22.866993Z", "iopub.status.busy": "2024-02-20T21:00:22.866827Z", "iopub.status.idle": "2024-02-20T21:00:23.262578Z", "shell.execute_reply": "2024-02-20T21:00:23.261964Z" } }, "outputs": [], "source": [ "from astroquery.mast import Observations\n", "from astropy.table import unique, vstack, Table" ] }, { "cell_type": "markdown", "id": "0c21754e", "metadata": {}, "source": [ "## Search the MAST Archive\n", "The first step to downloading the data is finding the observations we're interested in. This is easiest to do using `query_criteria`, which allows us to specify criteria such as RA/Dec, filters, exposure time, and any other fields listed [here](https://mast.stsci.edu/api/v0/_c_a_o_mfields.html). \n", "\n", "In this example, we use `query_criteria` to find NIRCam observations from [JWST Program 1073](https://www.stsci.edu/cgi-bin/get-proposal-info?id=1073&observatory=JWST). When querying for JWST data, using `obs_collection = 'JWST'` greatly inreases the speed of the search by decreasing the number of potential matches. This applies to all mission available in MAST, including HST." ] }, { "cell_type": "code", "execution_count": 2, "id": "3507f9d9", "metadata": { "execution": { "iopub.execute_input": "2024-02-20T21:00:23.265057Z", "iopub.status.busy": "2024-02-20T21:00:23.264869Z", "iopub.status.idle": "2024-02-20T21:00:23.648290Z", "shell.execute_reply": "2024-02-20T21:00:23.647707Z" } }, "outputs": [], "source": [ "matched_obs = Observations.query_criteria(\n", " obs_collection = 'JWST'\n", " , proposal_id = '1073'\n", " , instrument_name = 'NIRCAM/IMAGE' # Be sure to specify the full \"instrument/mode\" configuration!\n", " )" ] }, { "cell_type": "code", "execution_count": 3, "id": "84997008", "metadata": { "execution": { "iopub.execute_input": "2024-02-20T21:00:23.650633Z", "iopub.status.busy": "2024-02-20T21:00:23.650434Z", "iopub.status.idle": "2024-02-20T21:00:23.661368Z", "shell.execute_reply": "2024-02-20T21:00:23.660853Z" } }, "outputs": [ { "data": { "text/html": [ "Table length=15\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "
idxdataproduct_typefilterscalib_levelt_exptimeproposal_piintentTypeobsidinstrument_name
0imageF277W3171.788Koekemoer, Anton M.science83254329NIRCAM/IMAGE
1imageF150W3171.788Koekemoer, Anton M.science83254330NIRCAM/IMAGE
2imageF150W3515.364Koekemoer, Anton M.science76622656NIRCAM/IMAGE
3imageF277W3515.364Koekemoer, Anton M.science76622650NIRCAM/IMAGE
4imageF277W3773.046Koekemoer, Anton M.science83254500NIRCAM/IMAGE
5imageF070W3773.046Koekemoer, Anton M.science83254519NIRCAM/IMAGE
6imageF277W3343.576Koekemoer, Anton M.science75900624NIRCAM/IMAGE
7imageF277W3343.576Koekemoer, Anton M.science83254380NIRCAM/IMAGE
8imageF150W3343.576Koekemoer, Anton M.science118344942NIRCAM/IMAGE
9imageF150W3343.576Koekemoer, Anton M.science75914186NIRCAM/IMAGE
10imageF115W3343.576Koekemoer, Anton M.science83254391NIRCAM/IMAGE
11imageF277W3343.576Koekemoer, Anton M.science75884422NIRCAM/IMAGE
12imageF277W31374.304Koekemoer, Anton M.science83254838NIRCAM/IMAGE
13imageF150W3687.152Koekemoer, Anton M.science83254839NIRCAM/IMAGE
14imageF115W3687.152Koekemoer, Anton M.science83254840NIRCAM/IMAGE
\n", "\n" ], "text/plain": [ "" ] }, "execution_count": 3, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# This displays selected columns from the observation table, as a sanity check\n", "columns = ['dataproduct_type', 'filters', 'calib_level', 't_exptime', 'proposal_pi', 'intentType', 'obsid','instrument_name']\n", "matched_obs[columns].show_in_notebook(display_length=5)" ] }, { "cell_type": "markdown", "id": "40eebc32", "metadata": {}, "source": [ "The above search results in 15 observations. Keep this in the number in mind as we search for associated products.\n", "\n", "## Retreive Associated Products\n", "Each observation has associated data products. Which products are of interest to you depends on how you intend to use the data; more on this in the section below. For now, let's retreive all the products by requesting them in small \"chunks\".\n", "\n", "**Note: It is wise to avoid requesting all of the products simultaneously.** This is extremely likely to take an enormous amount of time, fail, or worse, do both, ultimately giving you a headache. MAST offers no medical advice, but we are decidedly anti-headache. Requesting products in groups of five offers the best balance between speed and reliability." ] }, { "cell_type": "code", "execution_count": 4, "id": "ef20ff5e", "metadata": { "execution": { "iopub.execute_input": "2024-02-20T21:00:23.663611Z", "iopub.status.busy": "2024-02-20T21:00:23.663323Z", "iopub.status.idle": "2024-02-20T21:00:28.660120Z", "shell.execute_reply": "2024-02-20T21:00:28.659561Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "There are 6768 unique files, which are 299.5 GB in size.\n" ] } ], "source": [ "# Split the observations into \"chunks\" of size five\n", "sz_chunk = 5\n", "chunks = [matched_obs[i:i+sz_chunk] for i in range(0,len(matched_obs), sz_chunk)]\n", "\n", "# Get the list of products for each chunk\n", "t = [Observations.get_product_list(chunk) for chunk in chunks]\n", "\n", "# Keep only the unique files\n", "files = unique(vstack(t), keys='productFilename')\n", "\n", "# How many files are there? How large are they?\n", "print(f\"There are {len(files)} unique files, which are {sum(files['size'])/10**9:.1f} GB in size.\")" ] }, { "cell_type": "markdown", "id": "7305a182", "metadata": {}, "source": [ "Now the issue with requesting all of the products simultaneously is clear: there are more than 6,000 unique files associated with our 15 observations.\n", "\n", "Running this search on the MAST Portal results in over 30,000 files since the Portal does not exclude duplicate results; that is nearing the limit of the what the Portal can load. One of the advantages of using the API is avoiding this large number of duplicates." ] }, { "cell_type": "markdown", "id": "6d6ee4ff", "metadata": {}, "source": [ "## Filter and Download Products\n", "\n", "If you are trying to download proprietary data, you will need to login. This requires a MAST token, which you can create at the [auth.mast](#https://auth.mast.stsci.edu/tokens) wesbite. If you have not set this as environment variable, you will have to enter it in the login prompt below.\n", "\n", "In this example, we are looking to download the uncalibrated products. We will filter those out below using the `productSubGroupDescription` field. You can find the other available product filters, including product type and file size, [here](https://mast.stsci.edu/api/v0/_productsfields.html). Examples are also included, but commented out, in the cell below.\n", "\n", "An additional option we make use of is the `curl_script` flag. Rather than downloading the data immediately, this method instead downloads a curl script. This is turned off by default, but is more robust than a direct download, and is highly recommended when downloading a large number of files. You can run this script using `bash mastDownload_dddd.sh`, changing `dddd` to reflect the actual name of your file." ] }, { "cell_type": "code", "execution_count": 5, "id": "544817bf", "metadata": { "execution": { "iopub.execute_input": "2024-02-20T21:00:28.662392Z", "iopub.status.busy": "2024-02-20T21:00:28.662071Z", "iopub.status.idle": "2024-02-20T21:00:31.739056Z", "shell.execute_reply": "2024-02-20T21:00:31.738585Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Downloading URL https://mast.stsci.edu/api/v0.1/Download/bundle.sh to ./mastDownload_20240220210028.sh ..." ] }, { "name": "stdout", "output_type": "stream", "text": [ " [Done]\n" ] } ], "source": [ "# Un-comment below if downloading data during its exclusive access period.\n", "#Observations.login()\n", "\n", "manifest = Observations.download_products(\n", " files\n", " , productSubGroupDescription='UNCAL'\n", " , curl_flag=True\n", " #, dataproduct_type='IMAGE'\n", " #, calib_level = [2]\n", " )" ] }, { "cell_type": "markdown", "id": "8098034b", "metadata": {}, "source": [ "All of the code in this notebook is available as a 'companion script', for further convenience." ] }, { "cell_type": "markdown", "id": "2c6e2adf", "metadata": {}, "source": [ "## Futher Reading\n", "* For a full explanation of product levels and the processing pipleline, see [Science Data Products](https://outerspace.stsci.edu/display/MASTDOCS/Science+Data+Products)\n", "* [JWST Archive Manual](https://outerspace.stsci.edu/display/MASTDOCS/JWST+Archive+Manual)\n", "* [Astropy](https://www.astropy.org) and the relevant [Table](https://docs.astropy.org/en/stable/api/astropy.table.Table.html?highlight=table) object" ] }, { "cell_type": "markdown", "id": "1404aa6d", "metadata": {}, "source": [ "## About this Notebook\n", "\n", "\n", "**Authors:** Thomas Dutkiewicz, Dick Shaw
\n", "**Keywords:** Downloads, astroquery, MAST
\n", "**Last Updated:** Aug 2022
\n", "**Next Review Date:** Feb 2023\n", "***\n", "[Top of Page](#top)\n", "\"Space " ] } ], "metadata": { "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.11.8" } }, "nbformat": 4, "nbformat_minor": 5 }