Intermediate: Search and Download GI Program Light Curves

Intermediate: Search and Download GI Program Light Curves#

Introduction#

This notebook uses the MAST Portal’s advanced search options to retrieve the light curves for a single guest investigator program. The notbook will show how to do an advanced query on the MAST’s database of holdings, determine which data files are associated with those observations and then download the files of interest.

For more information about the TESS mission and collected data, visit the MAST’s TESS homepage. To read more details about TESS Data Products, look in the TESS Science Product Description Document. A list of Guest Investigator programs can be found at the TESS GI List of Approved Programs.

Table of Contents#


Query MAST
Retrieve Product List
Download Data
Summarize Code
About this Notebook

Imports#

  • Observations module from astroquery.mast is needed to make the query and download.

  • astropy.io.fits module is needed to view the downloaded fits file.

For information on how to install astroquery see their [instructions](https://astroquery.readthedocs.io/en/latest/index.html). At the time of writing this requires the latest version on pip.

from astroquery.mast import Observations
from astropy.io import fits

Query the MAST CAOM Database#

We want to retrieve TESS timeseries data (lightcurve and target pixel files) for the Guest Investigator program G011183 from Sector 1 (PI: Stephen Kane). We will need to query the MAST holdings database for the observations. First we will simply count the number of observations and then we will request a table of those observations.

All the filter names are listed on the MAST CAOM Field Description Page. Or, can be found by hovering over the column names after doing a search in the MAST portal.

The filters we will need to use are,

  • obs_collection to specify that we want TESS data

  • dataproduct_type to specify that we want timeseries data

  • sequence_number to specify that we want sector 1

  • propsal_id to specify the GI program number.

Remember that more than one GI can propose for the same target so we need wild cards around the name of the program in our query.

obsCount = Observations.query_criteria_count(obs_collection = "TESS",
                                             dataproduct_type = ["timeseries"],
                                             sequence_number = 1,
                                             proposal_id = "*G011183*")
print("Number of Observations: %i" % obsCount)

Number of Observations: 42

obsTable = Observations.query_criteria(obs_collection = "TESS",
                                       dataproduct_type = ["timeseries"],
                                       sequence_number = 1,
                                       proposal_id = "*G011183*")
obsTable[0:5]['obsid','proposal_id','obs_id']
Table length=5
obsidproposal_idobs_id
str8str31str47
60829138G011112_G011183tess2018206045859-s0001-0000000025155310-0120-s
60835362G011112_G011183_G011132tess2018206045859-s0001-0000000029344935-0120-s
60840578G011112_G011183_G011132tess2018206045859-s0001-0000000038846515-0120-s
60839329G011112_G011183_G011132tess2018206045859-s0001-0000000097409519-0120-s
60843759G011112_G011183_G011132tess2018206045859-s0001-0000000149603524-0120-s

Retrieve the list of Data Products#

Next we use astroquery to retrieve the list of data products that are associated with each observation.

We will only ask for the data products associated with the first five. The [0:6] can be removed from the code below to get all the observations.

dataProducts = Observations.get_product_list(obsTable[0:6])
dataProducts.colnames

['obsID',
 'obs_collection',
 'dataproduct_type',
 'obs_id',
 'description',
 'type',
 'dataURI',
 'productType',
 'productGroupDescription',
 'productSubGroupDescription',
 'productDocumentationURL',
 'project',
 'prvversion',
 'proposal_id',
 'productFilename',
 'size',
 'parent_obsid',
 'dataRights',
 'calib_level',
 'filters']

dataProducts['obsID', 'productFilename', 'description']
Table length=72
obsIDproductFilenamedescription
str8str63str33
60829138tess2018206190142-s0001-s0001-0000000025155310-00106_dvr.pdffull data validation report
60829138tess2018206190142-s0001-s0001-0000000025155310-00106_dvr.xmlfull data validation report (xml)
60829138tess2018206190142-s0001-s0001-0000000025155310-00366_dvr.pdffull data validation report
60829138tess2018206190142-s0001-s0001-0000000025155310-00366_dvr.xmlfull data validation report (xml)
60829138tess2018206190142-s0001-s0001-0000000025155310-00106_dvm.pdfData validation mini report
60829138tess2018206190142-s0001-s0001-0000000025155310-00366_dvm.pdfData validation mini report
60829138tess2018206190142-s0001-s0001-0000000025155310-01-00106_dvs.pdfTCE summary report
60829138tess2018206190142-s0001-s0001-0000000025155310-01-00366_dvs.pdfTCE summary report
60829138tess2018206190142-s0001-s0001-0000000025155310-00106_dvt.fitsData validation time series
.........
60843759tess2018206190142-s0001-s0001-0000000149603524-00366_dvr.pdffull data validation report
60843759tess2018206190142-s0001-s0001-0000000149603524-00366_dvr.xmlfull data validation report (xml)
60843759tess2018206190142-s0001-s0001-0000000149603524-00106_dvm.pdfData validation mini report
60843759tess2018206190142-s0001-s0001-0000000149603524-00366_dvm.pdfData validation mini report
60843759tess2018206190142-s0001-s0001-0000000149603524-01-00106_dvs.pdfTCE summary report
60843759tess2018206190142-s0001-s0001-0000000149603524-01-00366_dvs.pdfTCE summary report
60843759tess2018206190142-s0001-s0001-0000000149603524-00106_dvt.fitsData validation time series
60843759tess2018206190142-s0001-s0001-0000000149603524-00366_dvt.fitsData validation time series
60843759tess2018206045859-s0001-0000000149603524-0120-s_lc.fitsLight curves
60843759tess2018206045859-s0001-0000000149603524-0120-s_tp.fitsTarget pixel files

Download Light curves#

We limit our list of data products to just those with the description set to “Light curves” because we just want this type of data file. We then download those products and print out the manifest to show the download location. Finally we use fits.info to show the structure of the fits file.

want = dataProducts['description'] == "Light curves"
print(dataProducts[want])

 obsID   obs_collection dataproduct_type ... dataRights calib_level filters
-------- -------------- ---------------- ... ---------- ----------- -------
60829138           TESS       timeseries ...     PUBLIC           3    TESS
60831534           TESS       timeseries ...     PUBLIC           3    TESS
60835362           TESS       timeseries ...     PUBLIC           3    TESS
60839329           TESS       timeseries ...     PUBLIC           3    TESS
60840578           TESS       timeseries ...     PUBLIC           3    TESS
60843759           TESS       timeseries ...     PUBLIC           3    TESS

manifest = Observations.download_products(dataProducts[want])
print(manifest)

Downloading URL https://mast.stsci.edu/api/v0.1/Download/file?uri=mast:TESS/product/tess2018206045859-s0001-0000000025155310-0120-s_lc.fits to ./mastDownload/TESS/tess2018206045859-s0001-0000000025155310-0120-s/tess2018206045859-s0001-0000000025155310-0120-s_lc.fits ...

 [Done]

Downloading URL https://mast.stsci.edu/api/v0.1/Download/file?uri=mast:TESS/product/tess2018206045859-s0001-0000000231663901-0120-s_lc.fits to ./mastDownload/TESS/tess2018206045859-s0001-0000000231663901-0120-s/tess2018206045859-s0001-0000000231663901-0120-s_lc.fits ...

 [Done]

Downloading URL https://mast.stsci.edu/api/v0.1/Download/file?uri=mast:TESS/product/tess2018206045859-s0001-0000000029344935-0120-s_lc.fits to ./mastDownload/TESS/tess2018206045859-s0001-0000000029344935-0120-s/tess2018206045859-s0001-0000000029344935-0120-s_lc.fits ...

 [Done]

Downloading URL https://mast.stsci.edu/api/v0.1/Download/file?uri=mast:TESS/product/tess2018206045859-s0001-0000000097409519-0120-s_lc.fits to ./mastDownload/TESS/tess2018206045859-s0001-0000000097409519-0120-s/tess2018206045859-s0001-0000000097409519-0120-s_lc.fits ...

 [Done]

Downloading URL https://mast.stsci.edu/api/v0.1/Download/file?uri=mast:TESS/product/tess2018206045859-s0001-0000000038846515-0120-s_lc.fits to ./mastDownload/TESS/tess2018206045859-s0001-0000000038846515-0120-s/tess2018206045859-s0001-0000000038846515-0120-s_lc.fits ...

 [Done]

Downloading URL https://mast.stsci.edu/api/v0.1/Download/file?uri=mast:TESS/product/tess2018206045859-s0001-0000000149603524-0120-s_lc.fits to ./mastDownload/TESS/tess2018206045859-s0001-0000000149603524-0120-s/tess2018206045859-s0001-0000000149603524-0120-s_lc.fits ...

 [Done]

                                                         Local Path                                                         ...
--------------------------------------------------------------------------------------------------------------------------- ...
./mastDownload/TESS/tess2018206045859-s0001-0000000025155310-0120-s/tess2018206045859-s0001-0000000025155310-0120-s_lc.fits ...
./mastDownload/TESS/tess2018206045859-s0001-0000000231663901-0120-s/tess2018206045859-s0001-0000000231663901-0120-s_lc.fits ...
./mastDownload/TESS/tess2018206045859-s0001-0000000029344935-0120-s/tess2018206045859-s0001-0000000029344935-0120-s_lc.fits ...
./mastDownload/TESS/tess2018206045859-s0001-0000000097409519-0120-s/tess2018206045859-s0001-0000000097409519-0120-s_lc.fits ...
./mastDownload/TESS/tess2018206045859-s0001-0000000038846515-0120-s/tess2018206045859-s0001-0000000038846515-0120-s_lc.fits ...
./mastDownload/TESS/tess2018206045859-s0001-0000000149603524-0120-s/tess2018206045859-s0001-0000000149603524-0120-s_lc.fits ...

fits.info(manifest['Local Path'][0])

Filename: ./mastDownload/TESS/tess2018206045859-s0001-0000000025155310-0120-s/tess2018206045859-s0001-0000000025155310-0120-s_lc.fits
No.    Name      Ver    Type      Cards   Dimensions   Format
  0  PRIMARY       1 PrimaryHDU      44   ()      
  1  LIGHTCURVE    1 BinTableHDU    167   20076R x 20C   [D, E, J, E, E, E, E, E, E, J, D, E, D, E, D, E, D, E, E, E]   
  2  APERTURE      1 ImageHDU        49   (11, 11)   int32

Summarize Code#

Here is a summary of the code required to do the problem described above, without all the outputs and investigations getting in the way. The query information is brought to the top so that it can easily be changed. For instance you might want to do a create a query on a different GI program number. One thing you will notice is that astroquery is smart enough to know that you have already downloaded the files and so you will get the message “Found cached file” instead of “Downloading URL”

# Query Information
mission = "TESS"
dataProdType = ["timeseries"]
GiProgram = "*G011183*"
fileType = "Light curves"
sector = 1

# Query Mast Holdings
obsTable = Observations.query_criteria(obs_collection = mission,
                                       dataproduct_type = dataProdType,
                                       sequence_number = sector,
                                       proposal_id = GiProgram)

# Get Product List
dataProducts = Observations.get_product_list(obsTable[0:2])
want = dataProducts['description'] == fileType

# Download Data
manifest = Observations.download_products(dataProducts[want])
print(manifest)

INFO: Found cached file ./mastDownload/TESS/tess2018206045859-s0001-0000000025155310-0120-s/tess2018206045859-s0001-0000000025155310-0120-s_lc.fits with expected size 2039040. [astroquery.query]
INFO: Found cached file ./mastDownload/TESS/tess2018206045859-s0001-0000000029344935-0120-s/tess2018206045859-s0001-0000000029344935-0120-s_lc.fits with expected size 2039040. [astroquery.query]
                                                         Local Path                                                         ...
--------------------------------------------------------------------------------------------------------------------------- ...
./mastDownload/TESS/tess2018206045859-s0001-0000000025155310-0120-s/tess2018206045859-s0001-0000000025155310-0120-s_lc.fits ...
./mastDownload/TESS/tess2018206045859-s0001-0000000029344935-0120-s/tess2018206045859-s0001-0000000029344935-0120-s_lc.fits ...

About this Notebook#

Author: Susan E. Mullally, STScI Archive Scientist
Updated On: 2018-11-29

Top of Page STScI logo

Contents