Intermediate: Search and Download GI Program Light Curves

Introduction

This notebook uses the MAST Portal's advanced search options to retrieve the light curves for a single guest investigator program. The notbook will show how to do an advanced query on the MAST's database of holdings, determine which data files are associated with those observations and then download the files of interest.

For more information about the TESS mission and collected data, visit the MAST's TESS homepage. To read more details about TESS Data Products, look in the TESS Science Product Description Document. A list of Guest Investigator programs can be found at the TESS GI List of Approved Programs.

Table of Contents


Query MAST
Retrieve Product List
Download Data
Summarize Code
About this Notebook


Imports

  • Observations module from astroquery.mast is needed to make the query and download.
  • astropy.io.fits module is needed to view the downloaded fits file.

    For information on how to install astroquery see their instructions. At the time of writing this requires the latest version on pip.

In [1]:
from astroquery.mast import Observations
from astropy.io import fits

Query the MAST CAOM Database

We want to retrieve TESS timeseries data (lightcurve and target pixel files) for the Guest Investigator program G011183 from Sector 1 (PI: Stephen Kane). We will need to query the MAST holdings database for the observations. First we will simply count the number of observations and then we will request a table of those observations.

All the filter names are listed on the MAST CAOM Field Description Page. Or, can be found by hovering over the column names after doing a search in the MAST portal.

The filters we will need to use are,

  • obs_collection to specify that we want TESS data
  • dataproduct_type to specify that we want timeseries data
  • sequence_number to specify that we want sector 1
  • propsal_id to specify the GI program number.

Remember that more than one GI can propose for the same target so we need wild cards around the name of the program in our query.

In [2]:
obsCount = Observations.query_criteria_count(obs_collection = "TESS",
                                             dataproduct_type = ["timeseries"],
                                             sequence_number = 1,
                                             proposal_id = "*G011183*")
print("Number of Observations: %i" % obsCount)
Number of Observations: 42
In [3]:
obsTable = Observations.query_criteria(obs_collection = "TESS",
                                       dataproduct_type = ["timeseries"],
                                       sequence_number = 1,
                                       proposal_id = "*G011183*")
obsTable[0:5]['obsid','proposal_id','obs_id']
Out[3]:
Table masked=True length=5
obsidproposal_idobs_id
str11str31str47
17000015852G011183tess2018206045859-s0001-0000000231716612-0120-s
17000006704G011183_G011025_G011048_G011250tess2018206045859-s0001-0000000231836962-0120-s
17000009900G011183_G011025_G011048tess2018206045859-s0001-0000000234501204-0120-s
17000003763G011183_G011025_G011048_G011250tess2018206045859-s0001-0000000099734092-0120-s
17000012356G011183_G011132_G011048_G011250tess2018206045859-s0001-0000000281461362-0120-s

Retrieve the list of Data Products

Next we use astroquery to retrieve the list of data products that are associated with each observation.

We will only ask for the data products associated with the first five. The [0:6] can be removed from the code below to get all the observations.

In [4]:
dataProducts = Observations.get_product_list(obsTable[0:6])
dataProducts.colnames
Out[4]:
['obsID',
 'obs_collection',
 'dataproduct_type',
 'obs_id',
 'description',
 'type',
 'dataURI',
 'productType',
 'productGroupDescription',
 'productSubGroupDescription',
 'productDocumentationURL',
 'project',
 'prvversion',
 'proposal_id',
 'productFilename',
 'size',
 'parent_obsid',
 'dataRights']
In [5]:
dataProducts['obsID', 'productFilename', 'description']
Out[5]:
Table masked=True length=12
obsIDproductFilenamedescription
str11str55str18
17000003763tess2018206045859-s0001-0000000099734092-0120-s_lc.fitsLight curves
17000003763tess2018206045859-s0001-0000000099734092-0120-s_tp.fitsTarget pixel files
17000015852tess2018206045859-s0001-0000000231716612-0120-s_lc.fitsLight curves
17000015852tess2018206045859-s0001-0000000231716612-0120-s_tp.fitsTarget pixel files
17000006704tess2018206045859-s0001-0000000231836962-0120-s_lc.fitsLight curves
17000006704tess2018206045859-s0001-0000000231836962-0120-s_tp.fitsTarget pixel files
17000009900tess2018206045859-s0001-0000000234501204-0120-s_lc.fitsLight curves
17000009900tess2018206045859-s0001-0000000234501204-0120-s_tp.fitsTarget pixel files
17000012356tess2018206045859-s0001-0000000281461362-0120-s_lc.fitsLight curves
17000012356tess2018206045859-s0001-0000000281461362-0120-s_tp.fitsTarget pixel files
17000010554tess2018206045859-s0001-0000000325178933-0120-s_lc.fitsLight curves
17000010554tess2018206045859-s0001-0000000325178933-0120-s_tp.fitsTarget pixel files

Download Light curves

We limit our list of data products to just those with the description set to "Light curves" because we just want this type of data file. We then download those products and print out the manifest to show the download location. Finally we use fits.info to show the structure of the fits file.

In [6]:
want = dataProducts['description'] == "Light curves"
print(dataProducts[want])
   obsID    obs_collection dataproduct_type ...   size  parent_obsid dataRights
----------- -------------- ---------------- ... ------- ------------ ----------
17000003763           TESS       timeseries ... 2039040  17000003763     PUBLIC
17000015852           TESS       timeseries ... 2039040  17000015852     PUBLIC
17000006704           TESS       timeseries ... 2039040  17000006704     PUBLIC
17000009900           TESS       timeseries ... 2039040  17000009900     PUBLIC
17000012356           TESS       timeseries ... 2039040  17000012356     PUBLIC
17000010554           TESS       timeseries ... 2039040  17000010554     PUBLIC
In [7]:
manifest = Observations.download_products(dataProducts[want])
print(manifest)
Downloading URL https://mast.stsci.edu/api/v0.1/Download/file?uri=mast:TESS/product/tess2018206045859-s0001-0000000099734092-0120-s_lc.fits to ./mastDownload/TESS/tess2018206045859-s0001-0000000099734092-0120-s/tess2018206045859-s0001-0000000099734092-0120-s_lc.fits ... [Done]
Downloading URL https://mast.stsci.edu/api/v0.1/Download/file?uri=mast:TESS/product/tess2018206045859-s0001-0000000231716612-0120-s_lc.fits to ./mastDownload/TESS/tess2018206045859-s0001-0000000231716612-0120-s/tess2018206045859-s0001-0000000231716612-0120-s_lc.fits ... [Done]
Downloading URL https://mast.stsci.edu/api/v0.1/Download/file?uri=mast:TESS/product/tess2018206045859-s0001-0000000231836962-0120-s_lc.fits to ./mastDownload/TESS/tess2018206045859-s0001-0000000231836962-0120-s/tess2018206045859-s0001-0000000231836962-0120-s_lc.fits ... [Done]
Downloading URL https://mast.stsci.edu/api/v0.1/Download/file?uri=mast:TESS/product/tess2018206045859-s0001-0000000234501204-0120-s_lc.fits to ./mastDownload/TESS/tess2018206045859-s0001-0000000234501204-0120-s/tess2018206045859-s0001-0000000234501204-0120-s_lc.fits ... [Done]
Downloading URL https://mast.stsci.edu/api/v0.1/Download/file?uri=mast:TESS/product/tess2018206045859-s0001-0000000281461362-0120-s_lc.fits to ./mastDownload/TESS/tess2018206045859-s0001-0000000281461362-0120-s/tess2018206045859-s0001-0000000281461362-0120-s_lc.fits ... [Done]
Downloading URL https://mast.stsci.edu/api/v0.1/Download/file?uri=mast:TESS/product/tess2018206045859-s0001-0000000325178933-0120-s_lc.fits to ./mastDownload/TESS/tess2018206045859-s0001-0000000325178933-0120-s/tess2018206045859-s0001-0000000325178933-0120-s_lc.fits ... [Done]
                                                         Local Path                                                         ...
--------------------------------------------------------------------------------------------------------------------------- ...
./mastDownload/TESS/tess2018206045859-s0001-0000000099734092-0120-s/tess2018206045859-s0001-0000000099734092-0120-s_lc.fits ...
./mastDownload/TESS/tess2018206045859-s0001-0000000231716612-0120-s/tess2018206045859-s0001-0000000231716612-0120-s_lc.fits ...
./mastDownload/TESS/tess2018206045859-s0001-0000000231836962-0120-s/tess2018206045859-s0001-0000000231836962-0120-s_lc.fits ...
./mastDownload/TESS/tess2018206045859-s0001-0000000234501204-0120-s/tess2018206045859-s0001-0000000234501204-0120-s_lc.fits ...
./mastDownload/TESS/tess2018206045859-s0001-0000000281461362-0120-s/tess2018206045859-s0001-0000000281461362-0120-s_lc.fits ...
./mastDownload/TESS/tess2018206045859-s0001-0000000325178933-0120-s/tess2018206045859-s0001-0000000325178933-0120-s_lc.fits ...
In [8]:
fits.info(manifest['Local Path'][0])
Filename: ./mastDownload/TESS/tess2018206045859-s0001-0000000099734092-0120-s/tess2018206045859-s0001-0000000099734092-0120-s_lc.fits
No.    Name      Ver    Type      Cards   Dimensions   Format
  0  PRIMARY       1 PrimaryHDU      44   ()      
  1  LIGHTCURVE    1 BinTableHDU    167   20076R x 20C   [D, E, J, E, E, E, E, E, E, J, D, E, D, E, D, E, D, E, E, E]   
  2  APERTURE      1 ImageHDU        49   (11, 13)   int32   

Summarize Code

Here is a summary of the code required to do the problem described above, without all the outputs and investigations getting in the way. The query information is brought to the top so that it can easily be changed. For instance you might want to do a create a query on a different GI program number. One thing you will notice is that astroquery is smart enough to know that you have already downloaded the files and so you will get the message "Found cached file" instead of "Downloading URL"

In [9]:
# Query Information
mission = "TESS"
dataProdType = ["timeseries"]
GiProgram = "*G011183*"
fileType = "Light curves"
sector = 1

# Query Mast Holdings
obsTable = Observations.query_criteria(obs_collection = mission,
                                       dataproduct_type = dataProdType,
                                       sequence_number = sector,
                                       proposal_id = GiProgram)

# Get Product List
dataProducts = Observations.get_product_list(obsTable[0:2])
want = dataProducts['description'] == fileType

# Download Data
manifest = Observations.download_products(dataProducts[want])
print(manifest)
INFO: Found cached file ./mastDownload/TESS/tess2018206045859-s0001-0000000231716612-0120-s/tess2018206045859-s0001-0000000231716612-0120-s_lc.fits with expected size 2039040. [astroquery.query]
INFO: Found cached file ./mastDownload/TESS/tess2018206045859-s0001-0000000231836962-0120-s/tess2018206045859-s0001-0000000231836962-0120-s_lc.fits with expected size 2039040. [astroquery.query]
                                                         Local Path                                                         ...
--------------------------------------------------------------------------------------------------------------------------- ...
./mastDownload/TESS/tess2018206045859-s0001-0000000231716612-0120-s/tess2018206045859-s0001-0000000231716612-0120-s_lc.fits ...
./mastDownload/TESS/tess2018206045859-s0001-0000000231836962-0120-s/tess2018206045859-s0001-0000000231836962-0120-s_lc.fits ...

About this Notebook

Author: Susan E. Mullally, STScI Archive Scientist
Updated On: 2018-11-29


In [ ]: