Ctrl+K

Beginner: Searching MAST using astroquery.mast

Beginner: Searching MAST using astroquery.mast#

Introduction and Goals:#

This is a beginner tutorial on accessing the MAST Archive using the Astroquery API. We’ll cover the major search features you might find useful when querying for observations. By the end of this tutorial, you will:

  • Understand how to search for observations hosted on the MAST Archive

  • Download data products corresponding to your observations of interest

  • Create a visual display of the downloaded data

Table of Contents#

  • Imports

  • Three ways to search for MAST observations

    • By Region

    • By Object Name

    • By Criteria

  • Getting Associated Data Products

    • Performing a Product Query

    • Filtering Data Products

  • Downloading Products

  • Displaying Data

  • Further Reading

Imports#

Let’s start by importing the packages we need for this notebook.

  • astroquery.mast to access the MAST API

  • astropy to create coordinate objects and access the data

  • matplotlib to plot the data

from astropy.coordinates import SkyCoord
from astroquery.mast import Observations
from astropy.io import fits
from matplotlib.colors import SymLogNorm

import matplotlib.pyplot as plt

%matplotlib inline

Three Ways to Search for MAST Observations#

1. By Region#

To search by coordinates (and a radius), you can use query_region.

The coordinates can be given as a string or astropy.coordinates object, and the radius as a string or float object. If no radius is specified, the default is 0.2 degrees.

Let’s try an example search with the coordinates (322.49324, 12.16683) and no radius.

# This will give a warning that the coordinates are being interpreted as an ICRS coordinate provided in degrees
obsByRegion = Observations.query_region("322.49324 12.16683")
len(obsByRegion)

WARNING: InputWarning: Coordinate string is being interpreted as an ICRS coordinate provided in degrees. [astroquery.utils.commons]

3043

Excellent! At the time of writing, this search returns 2240 results. As the MAST archive grows, this number increases; you might see a larger value. Let’s take a look a subset of the results.

# Preview the first 3 results
obsByRegion[:3].show_in_notebook()
Table length=3
idxintentTypeobs_collectionprovenance_nameinstrument_nameprojectfilterswavelength_regiontarget_nametarget_classificationobs_ids_ras_decdataproduct_typeproposal_picalib_levelt_mint_maxt_exptimeem_minem_maxobs_titlet_obs_releaseproposal_idproposal_typesequence_numbers_regionjpegURLdataURLdataRightsmtFlagsrcDenobsiddistance
0scienceTESSSPOCPhotometerTESSTESSOpticalTESS FFI--tess-s0055-1-1325.935119355519911.77496602082765imageRicker, George359796.601343796359823.76825269676475.199786600.01000.0--59841.0N/A--55POLYGON 329.837843 19.006512 333.753586 8.161861 322.106193 4.225655 318.039501 15.437198 329.837843 19.006512----PUBLICFalsenan951333210.0
1scienceTESSSPOCPhotometerTESSTESSOptical96703881--tess2022217014003-s0055-0000000096703881-0242-s322.49317112.166862timeseriesRicker, George359796.6041688888959823.76815293982120.0600.01000.0--59841.0G04046--55CIRCLE 322.493171 12.166862 0.00138889--mast:TESS/product/tess2022217014003-s0055-0000000096703881-0242-s_lc.fitsPUBLICFalsenan937705000.0
2scienceTESSSPOCPhotometerTESSTESSOptical96704587--tess2022217014003-s0055-0000000096704587-0242-s322.53518323129112.2706892440286timeseriesRicker, George359796.6041632870459823.76815065972120.0600.01000.0--59841.0G04103--55CIRCLE 322.53518323 12.27068924 0.00138889--mast:TESS/product/tess2022217014003-s0055-0000000096704587-0242-s_lc.fitsPUBLICFalsenan93770499397.01844231118105

The columns of the above table (there are many!) correspond to searchable fields in the MAST database. You can find the full list of criteria here and an example of performing a search by criteria down below.

If we want to avoid that pesky warning from the search above we can create a coordinates object and pass it to our search. Let’s try that now, and in addition, let’s add a radius of 1 arcsecond to this search.

# Set up our coordinates object
coord = SkyCoord(290.213503, -11.800746, unit='deg')

# Same search as above, now with a radius of 1 arcsecond
obsByRegion2 = Observations.query_region(coord, radius='1s')
len(obsByRegion2)

9

Sanity check: we expect that as the radius gets smaller, we get fewer results. In this case, only 15 results are returned.

Let’s take a look at the results again, but let’s limit the columns that get displayed.

# Let's limit the number of columns we see at once
columns = ['obs_collection', 'intentType', 'instrument_name', 
            'target_name', 't_exptime', 'filters', 'dataproduct_type']

# Show the results with the above columns only
obsByRegion2[columns].show_in_notebook()
Table length=9
idxobs_collectionintentTypeinstrument_nametarget_namet_exptimefiltersdataproduct_type
0TESSsciencePhotometerTESS FFI475.199781TESSimage
1PS1scienceGPC11126.007731.0gimage
2PS1scienceGPC11126.007900.0iimage
3PS1scienceGPC11126.007876.0rimage
4PS1scienceGPC11126.007580.0yimage
5PS1scienceGPC11126.007480.0zimage
6HLSPsciencePhotometerTICA FFI475.2TESSimage
7GALEXscienceGALEXAIS_246_1_35176.0NUVimage
8GALEXscienceGALEXAIS_246_1_35176.0FUVimage

This “streamlined” view is helpful: it avoids visual clutter and helps you to focus on the fields that are most relevant to your search.

2. By Object Name#

To search for an object by name, you can use the query.object function. This function also optionally allows you to specify a radius.

The object name is first resolved to coordinates by a call to the Simbad and NED archives. Then, the search proceeeds based on these coordinates.

obsByName = Observations.query_object("M51",radius=".005 deg")

print("Number of results:",len(obsByName))

# Here we just print the values, but you can use '.show_in_table' like we did in the previous cell
print(obsByName[:10])

Number of results: 1745
intentType obs_collection provenance_name ... srcDen  obsid   distance
---------- -------------- --------------- ... ------ -------- --------
   science           TESS            SPOC ...    nan 27545566      0.0
   science           TESS            SPOC ...    nan 27347170      0.0
   science           TESS            SPOC ...    nan 27386992      0.0
   science           TESS            SPOC ...    nan 83163378      0.0
   science          SWIFT              -- ...    nan  1628611      0.0
   science          SWIFT              -- ... 5885.0  1491575      0.0
   science          SWIFT              -- ... 5885.0  1491576      0.0
   science          SWIFT              -- ... 5885.0  1493598      0.0
   science          SWIFT              -- ... 5885.0  1493597      0.0
   science          SWIFT              -- ... 5885.0  1493982      0.0

For some special catalogs, like those used with Kepler, K2, and TESS, astroquery performs a direct lookup using the MAST catalog. It is important to include both the catalog identifier and number when searching one of these datasets; for example, to query the TESS Input catalog you would use “TIC 261136679”, rather than a plain “261136679”.

obsByTessName = Observations.query_object("TIC 261136679", radius="1s")

# As before, we use columns to limit the display
columns = ['obs_collection', 'wavelength_region', 'provenance_name', 't_min', 't_max', 'obsid']
obsByTessName[columns].show_in_notebook(display_length=10)
Table length=166
idxobs_collectionwavelength_regionprovenance_namet_mint_maxobsid
0TESSOpticalSPOC58324.8130439004658352.66690385416660865631
1TESSOpticalSPOC58410.4159353009258436.3323207291761132377
2TESSOpticalSPOC58516.8531245833458541.4992148726962468248
3TESSOpticalSPOC58596.2709659606558623.3754181481565153352
4TESSOpticalSPOC58624.4595443402858652.3765100115765180725
5TESSOpticalSPOC58653.4181698726858681.8553605555665462606
6TESSOpticalSPOC59035.779107759060.1399472327797249
7TESSOpticalSPOC59061.3474442459086.5971616927844486
8TESSOpticalSPOC59144.012856159169.4431255328131771
9TESSOpticalSPOC59228.248649259253.5614096328364165
10TESSOpticalSPOC59254.4847447916759279.4780542361160737942
11TESSOpticalSPOC59333.3539865277859360.0487042361161660222
12TESSOpticalSPOC59361.2717330092659389.2164645486162242849
13TESSOpticalSPOC59962.29213774305459987.72299065972118311751
14TESSOpticalSPOC59987.9333227893560013.651212337965126301646
15TESSOpticalSPOC60040.6081145833360067.529729918984140242238
16TESSOpticalSPOC60068.2341994444460096.08870777778152374421
17TESSOpticalSPOC60097.1693794444460125.92666236111167876913
18TESSOpticalSPOC60126.1372707754660153.89626454861172373724
19TESSOpticalSPOC60154.1060564467660181.6381984375178063767
20TESSOpticalSPOC58324.794076562558352.6769594675960835895
21TESSOpticalSPOC58324.7940992361158436.34875071759662470298
22TESSOpticalSPOC58324.7940992361158541.499119641263394803
23TESSOpticalSPOC58324.79409922453558681.8579177546365486302
24TESSOpticalSPOC58324.79407652777459253.56369871527661531517
25TESSOpticalSPOC58324.79407652777459389.2187876504662342022
26TESSOpticalSPOC58324.7940764930560096.12668114583160348376
27TESSOpticalSPOC58324.7940764930560181.633050034725198774334
28TESSOpticalSPOC58410.3992267013958436.3334372222261114378
29TESSOpticalSPOC58516.86113947916658541.4990966203762459906
30TESSOpticalSPOC58596.27672734953458623.39247916666565142109
31TESSOpticalSPOC58624.45498547453558652.39271228009665182732
32TESSOpticalSPOC58653.4204896643558681.8578947222265462812
33TESSOpticalSPOC59035.7786457459060.1432361327811411
34TESSOpticalSPOC59035.7786457459060.1423102327803712
35TESSOpticalSPOC59061.3506331459086.5976402727823276
36TESSOpticalSPOC59061.3506331459086.5974087827820792
37TESSOpticalSPOC59144.0127342959169.4437263528062277
38TESSOpticalSPOC59144.0127342959169.4428004428058401
39TESSOpticalSPOC59228.24810439814659253.564624687528289741
40TESSOpticalSPOC59228.2647712384359253.5636987384328306022
41TESSOpticalSPOC59333.35432055555459360.05311312561592898
42TESSOpticalSPOC59333.35432055555459360.0519557060261598395
43TESSOpticalSPOC59361.2714109490859389.2199450231562018551
44TESSOpticalSPOC59361.2714109490859389.2187876157462005566
45TESSOpticalSPOC59962.2964217129659987.72454979167117955554
46TESSOpticalSPOC59962.2964217129659987.71066060185117955549
47TESSOpticalSPOC59987.93844357638660013.65217048611124157589
48TESSOpticalSPOC59987.93844357638660013.648698171295124157576
49TESSOpticalSPOC60040.6132177314860067.53207005787139095591
50TESSOpticalSPOC60040.6132177314860067.53183857639139095580
51TESSOpticalSPOC60068.23879266203660096.12945888889151530181
52TESSOpticalSPOC60068.23879266203660096.12945888889151530185
53TESSOpticalSPOC60097.17391609953560125.92817840278173755271
54TESSOpticalSPOC60097.17391609953560125.92817840278173755263
55TESSOpticalSPOC60126.1420676620460153.896710891204171667548
56TESSOpticalSPOC60126.1420676620460153.88490546296171667556
57TESSOpticalSPOC60154.1112980324160181.63929988426176755217
58TESSOpticalSPOC60154.1154646527860181.62610565972176755222
59SWIFTUV;OPTICAL--57393.02883157393.88121531563562
60SWIFTUV--57393.023923657393.87598381563441
61SWIFTUV;OPTICAL--57393.026342657393.87761571563440
62SWIFTOPTICAL--57393.02796357393.87844911563561
63SPITZER_SHAInfraredSSC Pipeline53109.7472792553109.748213771737024
64SPITZER_SHAInfraredSSC Pipeline53109.745348153109.745772871737024
65SPITZER_SHAInfraredSSC Pipeline53109.744649153109.745073851737024
66SPITZER_SHAInfraredSSC Pipeline53109.743867853109.744098391737024
67SPITZER_SHAInfraredSSC Pipeline53109.7433816953109.743612231737024
68SPITZER_SHAInfraredSSC Pipeline53109.7433816953109.743612231737024
69SPITZER_SHAInfraredSSC Pipeline53109.743867853109.744098391737024
70SPITZER_SHAInfraredSSC Pipeline53109.7460703953109.747004921737024
71SPITZER_SHAInfraredSSC Pipeline53109.745348153109.745772871737024
72SPITZER_SHAInfraredSSC Pipeline53109.7472792553109.748213771737024
73SPITZER_SHAInfraredSSC Pipeline53109.7460703953109.747004921737024
74SPITZER_SHAInfraredSSC Pipeline53109.744649153109.745073851737024
75SPITZER_SHAInfraredSSC Pipeline53103.6357850153103.636628461634700
76SPITZER_SHAInfraredSSC Pipeline53103.6371017853103.642557111634700
77SPITZER_SHAInfraredSSC Pipeline53052.1855368553052.190406941739236
78SPITZER_SHAInfraredSSC Pipeline53052.1855368553052.190406941739236
79SPITZER_SHAInfraredSSC Pipeline53052.1855368553052.190406941739236
80SPITZER_SHAInfraredSSC Pipeline53052.1855368553052.190406941739236
81SPITZER_SHAInfraredSSC Pipeline54599.0837825254599.086101491699737
82SPITZER_SHAInfraredSSC Pipeline54599.0837825254599.086101491699737
83SPITZER_SHAInfraredSSC Pipeline54599.0837825254599.086101491699737
84SPITZER_SHAInfraredSSC Pipeline54599.0837825254599.086101491699737
85SPITZER_SHAInfraredSSC Pipelinenannan1695305
86SPITZER_SHAInfraredSSC Pipelinenannan1695305
87SPITZER_SHAInfraredSSC Pipelinenannan1695305
88SPITZER_SHAInfraredSSC Pipelinenannan1695305
89HLSPOpticalELEANOR58324.8138555558352.6677157332119741
90HLSPOpticalELEANOR58324.8138555558352.6677157332116730
91HLSPOpticalELEANOR58324.8138555558352.6677157332121978
92HLSPOpticalELEANOR58324.8138555558352.6677157332125013
93HLSPOpticalELEANOR58410.4167556258436.333145432156777
94HLSPOpticalELEANOR58410.4167556258436.333145432164982
95HLSPOpticalELEANOR58516.853935858541.5000263132204419
96HLSPOpticalELEANOR58516.853935858541.5000263132206316
97HLSPOpticalELEANOR58516.853935858541.5000263132212892
98HLSPOpticalELEANOR58516.853935858541.5000263132208188
99HLSPOpticalELEANOR58596.2717779258623.3762303532250860
100HLSPOpticalELEANOR58596.2717779258623.3762303532240741
101HLSPOpticalELEANOR58624.4603614258652.377327332256445
102HLSPOpticalELEANOR58653.4189871558681.8561780732266203
103HLSPOpticalELEANOR58653.4189871558681.8561780732277156
104HLSPOpticalELEANOR58653.4189871558681.8561780732267875
105HLSPOpticalELEANOR58653.4189871558681.8561780732272118
106HLSPOpticalGSFC-ELEANOR-LITE58324.8130439004658352.66690385416684704022
107HLSPOpticalGSFC-ELEANOR-LITE58410.4159353009258436.33232072917116010112
108HLSPOpticalGSFC-ELEANOR-LITE58516.8531245833458541.49921487269161104956
109HLSPOpticalQLP58324.8247605258352.6576430959455369
110HLSPOpticalQLP58410.4271411758436.3224678458459774
111HLSPOpticalQLP58516.8640451873758541.489502505455003357
112HLSPOpticalQLP58596.2824122758623.3662197149252805
113HLSPOpticalQLP58624.4703922358652.3664525647144300
114HLSPOpticalQLP58653.4289521958681.8455243145092053
115HLSPOpticalQLP59035.7829410694559060.1368837514963917481
116HLSPOpticalQLP59061.3521510376659086.5947603718464550682
117HLSPOpticalQLP59144.0170296896259169.4401516742166665691
118HLSPOpticalQLP59228.2523993449359253.558271506869866855
119HLSPOpticalQLP59333.3586162850359360.0465293885275350117
120HLSPOpticalQLP59361.27570598259389.2133607016977972093
121HLSPOpticalQLP59962.2937717870859987.72236311529163587542
122HLSPOpticalQLP59987.93533071782460013.65114138601164976661
123HLSPOpticalQLP60040.6101059112760067.529653193895173966129
124HLSPOpticalQLP60068.2356803957460096.08792077331183888484
125HLSPOpticalQLP60097.17126703029560125.92599268537186194451
126HLSPOpticalQLP60126.1389562487660153.895683024544199776093
127HLSPOpticalQLP60154.10864980658560181.63827206567200558578
128HLSPOpticalTASOC58324.814321113658352.6680385038980761526
129HLSPOpticalTASOC58410.4167022028358436.332862430489049638
130HLSPOpticalTASOC58324.7948770654358352.6777602447379395544
131HLSPOpticalTASOC58410.4000361432558436.3495285723988546893
132HLSPOpticalTESS-SPOC58324.8135207407458352.6672374768559804479
133HLSPOpticalTESS-SPOC58410.4158929745458436.3320483680660274783
134HLSPOpticalTESS-SPOC58516.85280603009658541.4990966203762618178
135HLSPOpticalTESS-SPOC58596.2920054166758623.37581231481563179060
136HLSPOpticalTESS-SPOC58624.459152187558652.3760456018563279269
137HLSPOpticalTESS-SPOC58653.43854521990658681.8551169791763547796
138HLSPOpticalTESS-SPOC59035.7786457459060.1395324953633452
139HLSPOpticalTESS-SPOC59061.3547997559086.5974087853779499
140HLSPOpticalTESS-SPOC59144.0127342939859169.44280043981461491828
141HLSPOpticalTESS-SPOC59228.24810439814659253.5609209143561833113
142HLSPOpticalTESS-SPOC59333.35432055555459360.0491778935265863007
143HLSPOpticalTESS-SPOC59361.2714109490859389.2160098263965990217
144HLSPOpticalTESS-SPOC59962.2964217129659987.72269789352178972056
145HLSPOpticalTESS-SPOC59987.9402954629660013.65147601852192672058
146HLSPOpticalTESS-SPOC60040.6150696296360067.52998668981199010078
147HLSPOpticalTESS-SPOC60068.2406445486160096.08825475694199351997
148HLSPOpticalTESS-SPOC60126.1439195138960153.89601645833201262105
149HLSPOpticalTESS-SPOC60154.1112980324160181.638605451386201544704
150HLSPOpticalTICA59035.7780821523659060.1391819911196842434
151HLSPOpticalTICA59061.3475206950759086.597509581699122009
152HLSPOpticalTICA59144.014150995359169.4446952710999403814
153HLSPOpticalTICA59228.2502249507259253.5627136263199516461
154HLSPOpticalTICA59254.4863246548959279.4793691295299617468
155HLSPOpticalTICA59333.3543450678759360.0487776575699941936
156HLSPOpticalTICA59361.2709933919859389.21542511648100034118
157HLSPOpticalTICA59962.2938747559759987.72441852046113534134
158HLSPOpticalTICA59987.9350665286260013.65264742589118319607
159HLSPOPTICALTICA60040.6086534308360067.52993744984128200817
160HLSPOPTICALTICA60068.2336350707360096.08778880769137243062
161HLSPOpticalTICA60097.168807011160123.74055396067147196500
162HLSPOpticalTICA60126.1363862529460153.89563273499150890395
163HLSPOpticalTICA60154.1062867371460181.63868136378165587028
164GALEXUVAIS53970.4571412037153970.45834490740538561
165GALEXUVAIS53970.4571412037153970.45834490740538561

It’s worth noting that even though we queried using the TESS Input Catalog, not all of the results are from the TESS mission. The TIC is a catalog like any other, and the MAST Archive resolves the input name to a location on the sky. Other missions that have observed the same region will also return results.

To search for results that are specific to TESS, or your mission of choice, see the ‘by other criteria’ section below.

3. By Other Criteria (with or without name/region) #

To search for observations based on additonal parameters, you can use query_criteria. In a sense, this is a more powerful version of the tools above, as you can still search by coordinates and objectname; however, you can inculde additional desired criteria. You may also run the search without specifying a name or coordinates.

To perform the search, give your critera as keyword arguments. Valid criteria and their descriptions are listed here. (Note that these are columns of the results tables we saw above!) Some relevant examples are: “filters”, “t_exptime” (exposure time), instrument_name, “provenance_name”, and “sequence_number”.

Let’s for TESS Sector 9 data with an exposure time between 1400 and 1500s.

obsByCriteria = Observations.query_criteria(obs_collection=["TESS"], sequence_number=9,
                                            t_exptime=[1400,1500])

columns = ['target_name','s_ra', 's_dec', 't_exptime', 'obsid']
obsByCriteria[columns].show_in_notebook(display_length=5)
Table length=16
idxtarget_names_ras_dect_exptimeobsid
0TESS FFI148.8381670809981-53.6401965875519051425.59941462895676
1TESS FFI173.83652057983417-10.277520139369341425.59941462892431
2TESS FFI162.4696250638618-5.4256527079199271425.59941462892977
3TESS FFI151.37494470193366-26.745954695297461425.59941462895165
4TESS FFI119.47374170204309-54.1110656462167941425.59941462896721
5TESS FFI143.74057778883315-36.890513963133131425.59941462893522
6TESS FFI157.48628713094715-43.225718556148741425.59941462894034
7TESS FFI169.0707180032819-21.3872269798947061425.59941462891912
8TESS FFI75.9695971279556-60.544387805883951425.59941462898889
9TESS FFI163.9337788078188-32.310803223636931425.59941462894594
10TESS FFI100.0080806433491-59.274941346736821425.59941462898353
11TESS FFI110.78962946825075-70.68695293804271425.59941462897785
12TESS FFI72.28880568223647-72.702289364485991425.59941462899417
13TESS FFI157.27070350969674-16.3217157826658761425.59941462891432
14TESS FFI135.13107661187195-63.352246631182561425.59941462897231
15TESS FFI133.67905448025425-46.182396030083241425.59941462896230

There’s no limit on the number of filters you can apply in a search. It may be an interesting exercise for the reader to go through the example below and figure out what exactly we’re searching for.

(Hint: check the field descriptions)

# Make sure to run this cell, as the data is used in the sections that follow

exByCriteria = Observations.query_criteria(obs_collection=["HLA"], s_dec=[50,60], 
                                            calib_level=[3], proposal_pi="Mould*", 
                                            dataproduct_type="IMAGE", t_max=[49800,49820])

columns = ['obs_collection', 'obs_id', 'target_name', 'filters', 'instrument_name', 'proposal_id']
exByCriteria[columns].show_in_notebook()
Table length=4
idxobs_collectionobs_idtarget_namefiltersinstrument_nameproposal_id
0HLAhst_05766_04_wfpc2_f555w_pcNGC5457-FLD2F555WWFPC2/PC5766
1HLAhst_05766_04_wfpc2_f555w_wfNGC5457-FLD2F555WWFPC2/WFC5766
2HLAhst_05766_04_wfpc2_total_pcNGC5457-FLD2DETECTIONWFPC2/PC5766
3HLAhst_05766_04_wfpc2_total_wfNGC5457-FLD2DETECTIONWFPC2/WFC5766

Getting Associated Data Products#

Performing a Product Query#

Each observation returned from a MAST query can have one or more associated data products. For example, a JWST observation might return an uncalibrated file, a guide-star file, and the science data you’re searching for.

You can input a table of observations or list of observation ids (“obs_id”) and get_product_list will return a table containing the associated data products.

Since we already have a list of observations, we can use that as the starting point for our query. To keep it simple, let’s look at only the last observation from our search above.

# Let's select a small subset from our critera search above
newObsList = exByCriteria[-1:]

# Now we get the list of products associated with that observation
dataProducts = Observations.get_product_list(newObsList)

dataProducts.show_in_notebook(display_length=5)
Table length=32
idxobsIDobs_collectiondataproduct_typeobs_iddescriptiontypedataURIproductTypeproductGroupDescriptionproductSubGroupDescriptionproductDocumentationURLprojectprvversionproposal_idproductFilenamesizeparent_obsiddataRightscalib_level
025153181HLAimagehst_05766_04_wfpc2_f555w_wf_02Preview-FullSmast:HLA/url/cgi-bin/preview.cgi?dataset=hst_05766_04_wfpc2_f555w_wf_02PREVIEW------HLA--5766hst_05766_04_wfpc2_f555w_wf_02_drz.jpg--25579950PUBLIC2
125153181HLAimagehst_05766_04_wfpc2_f555w_wf_02HLA simple fits science imageSmast:HLA/url/cgi-bin/getdata.cgi?dataset=hst_05766_04_wfpc2_f555w_wf_02_drz.fitsSCIENCE--DRZ--HLA--5766hst_05766_04_wfpc2_f555w_wf_02_drz.fits1028160025579950PUBLIC2
225579950HLAimagehst_05766_04_wfpc2_total_wfHLA DAOPHOT CatalogCmast:HLA/url/cgi-bin/getdata.cgi?download=1&filename=hst_05766_04_wfpc2_total_wf_daophot_trm.catCATALOGMinimum Recommended ProductsDAOPHOT--HLA--5766hst_05766_04_wfpc2_total_wf_daophot_trm.cat--25579950PUBLIC3
325579950HLAimagehst_05766_04_wfpc2_total_wfHLA SExtractor CatalogCmast:HLA/url/cgi-bin/getdata.cgi?download=1&filename=hst_05766_04_wfpc2_total_wf_sexphot_trm.catCATALOGMinimum Recommended ProductsSEXPHOT--HLA--5766hst_05766_04_wfpc2_total_wf_sexphot_trm.cat--25579950PUBLIC3
425579950HLAimagehst_05766_04_wfpc2_total_wfPreview-FullCmast:HLA/url/cgi-bin/preview.cgi?dataset=hst_05766_04_wfpc2_total_wfPREVIEW------HLA--5766hst_05766_04_wfpc2_total_wf_drz.jpg--25579950PUBLIC3
525579950HLAimagehst_05766_04_wfpc2_total_wfHLA simple fits science imageCmast:HLA/url/cgi-bin/getdata.cgi?dataset=hst_05766_04_wfpc2_total_wf_drz.fitsSCIENCEMinimum Recommended ProductsDRZ--HLA--5766hst_05766_04_wfpc2_total_wf_drz.fits3079872025579950PUBLIC3
624556184HSTimageu2ms0402tDADS C3M file - Calibrated exposure WFPC2Smast:HST/product/u2ms0402t_c3m.fitsAUXILIARY--C3M--CALWFPC22.5.3 (Sep 4, 2008)5766u2ms0402t_c3m.fits20736025579950PUBLIC2
724556184HSTimageu2ms0402tDADS C3T file - Calibrated data WFPC2Smast:HST/product/u2ms0402t_c3t.fitsAUXILIARY--C3T--CALWFPC22.5.3 (Sep 4, 2008)5766u2ms0402t_c3t.fits21024025579950PUBLIC2
824556184HSTimageu2ms0402tDADS CGR file - Calibrated data WFPC/FOCSmast:HST/product/u2ms0402t_cgr.fitsAUXILIARY--CGR--CALWFPC22.5.3 (Sep 4, 2008)5766u2ms0402t_cgr.fits3168025579950PUBLIC2
924556184HSTimageu2ms0402tDADS CMH fileSmast:HST/product/u2ms0402j_cmh.fitsAUXILIARY--CMH--CALWFPC2--5766u2ms0402j_cmh.fits1740825579950PUBLIC1
1024556184HSTimageu2ms0402tDADS CMI fileSmast:HST/product/u2ms0402j_cmi.fitsAUXILIARY--CMI--CALWFPC2--5766u2ms0402j_cmi.fits10086425579950PUBLIC1
1124556184HSTimageu2ms0402tDADS CMJ fileSmast:HST/product/u2ms0402j_cmj.fitsAUXILIARY--CMJ--CALWFPC2--5766u2ms0402j_cmj.fits255488025579950PUBLIC1
1224556184HSTimageu2ms0402tDADS DGR file - Raw data WFPC/FOCSmast:HST/product/u2ms0402t_dgr.fitsAUXILIARY--DGR--CALWFPC2--5766u2ms0402t_dgr.fits3168025579950PUBLIC1
1324556184HSTimageu2ms0402tDADS FLT_HLETSmast:HST/product/u2ms0402t_flt_hlet.fitsAUXILIARY--FLT_HLET--CALWFPC22.5.3 (Sep 4, 2008)5766u2ms0402t_flt_hlet.fits3456025579950PUBLIC2
1424556184HSTimageu2ms0402tDADS Q0F file - Raw data quality WFPC/WFPC2/FOC/FOS/GHRS/HSPSmast:HST/product/u2ms0402t_q0f.fitsAUXILIARY--Q0F--CALWFPC2--5766u2ms0402t_q0f.fits518400025579950PUBLIC1
1524556184HSTimageu2ms0402tDADS Q0M file - Raw data quality WFPC2Smast:HST/product/u2ms0402t_q0m.fitsAUXILIARY--Q0M--CALWFPC2--5766u2ms0402t_q0m.fits517824025579950PUBLIC1
1624556184HSTimageu2ms0402tDADS Q1F file - Calibrated data quality WFPC/WFPC2/FOC/FOS/GHRS/HSPSmast:HST/product/u2ms0402t_q1f.fitsAUXILIARY--Q1F--CALWFPC2--5766u2ms0402t_q1f.fits10368025579950PUBLIC1
1724556184HSTimageu2ms0402tDADS Q1M file - Calibrated data quality WFPC2Smast:HST/product/u2ms0402t_q1m.fitsAUXILIARY--Q1M--CALWFPC2--5766u2ms0402t_q1m.fits10944025579950PUBLIC1
1824556184HSTimageu2ms0402tDADS SHM file - Engineering telemetry WFPC2Smast:HST/product/u2ms0402t_shm.fitsAUXILIARY--SHM--CALWFPC2--5766u2ms0402t_shm.fits3456025579950PUBLIC1
1924556184HSTimageu2ms0402tDADS TRL file - Processing logSmast:HST/product/u2ms0402t_trl.fitsAUXILIARY--TRL--CALWFPC2--5766u2ms0402t_trl.fits2880025579950PUBLIC1
2024556184HSTimageu2ms0402tDADS X0F file - Raw bias data WFPC/WFPC2/FOSSmast:HST/product/u2ms0402t_x0f.fitsAUXILIARY--X0F--CALWFPC2--5766u2ms0402t_x0f.fits10368025579950PUBLIC1
2124556184HSTimageu2ms0402tDADS X0M file - Raw bias data WFPC2Smast:HST/product/u2ms0402t_x0m.fitsAUXILIARY--X0M--CALWFPC2--5766u2ms0402t_x0m.fits10944025579950PUBLIC1
2224556184HSTimageu2ms0402tDADS LOG fileSmast:HST/product/u2ms0402t_log.txtINFO--LOG--CALWFPC2--5766u2ms0402t_log.txt6262525579950PUBLIC1
2324556184HSTimageu2ms0402tPreview-FullSmast:HST/product/u2ms0402t_drw.jpgPREVIEW------CALWFPC22.5.3 (Sep 4, 2008)5766u2ms0402t_drw.jpg142982625579950PUBLIC3
2424556184HSTimageu2ms0402tDADS C0F file - Calibrated exposure WFPC/WFPC2/FOC/FOS/GHRS/HSPSmast:HST/product/u2ms0402t_c0f.fitsSCIENCE--C0F--CALWFPC22.5.3 (Sep 4, 2008)5766u2ms0402t_c0f.fits1030752025579950PUBLIC2
2524556184HSTimageu2ms0402tDADS C0M file - Calibrated exposure WFPC2Smast:HST/product/u2ms0402t_c0m.fitsSCIENCE--C0M--CALWFPC22.5.3 (Sep 4, 2008)5766u2ms0402t_c0m.fits1030752025579950PUBLIC2
2624556184HSTimageu2ms0402tDADS C1F file - Calibrated exposure WFPC/FOC/FOS/GHRS/HSPSmast:HST/product/u2ms0402t_c1f.fitsSCIENCE--C1F--CALWFPC22.5.3 (Sep 4, 2008)5766u2ms0402t_c1f.fits518400025579950PUBLIC2
2724556184HSTimageu2ms0402tDADS C1M fileSmast:HST/product/u2ms0402t_c1m.fitsSCIENCE--C1M--CALWFPC22.5.3 (Sep 4, 2008)5766u2ms0402t_c1m.fits517824025579950PUBLIC2
2824556184HSTimageu2ms0402tDADS D0F file - Raw exposure WFPC/WFPC2/FOC/FOS/GHRS/HSPSmast:HST/product/u2ms0402t_d0f.fitsSCIENCE--D0F--CALWFPC2--5766u2ms0402t_d0f.fits518400025579950PUBLIC1
2924556184HSTimageu2ms0402tDADS D0M file - Raw exposure WFPC2Smast:HST/product/u2ms0402t_d0m.fitsSCIENCE--D0M--CALWFPC2--5766u2ms0402t_d0m.fits517824025579950PUBLIC1
3024556184HSTimageu2ms0402tDADS DRW fileSmast:HST/product/u2ms0402t_drw.fitsSCIENCE--DRW--CALWFPC22.5.3 (Sep 4, 2008)5766u2ms0402t_drw.fits14580288025579950PUBLIC3
3124556184HSTimageu2ms0402tDADS FLT file - Calibrated exposure ACS/WFC3/STIS/COSSmast:HST/product/u2ms0402t_flt.fitsSCIENCE--FLT--CALWFPC22.5.3 (Sep 4, 2008)5766u2ms0402t_flt.fits2592000025579950PUBLIC2

Filtering the Data Products#

After the data products have been retrieved, you can use filter_products to download only data products that meet your given criteria. Available filters are listed here. Some examples are: “mrp_only” (Minimum Recommended Products) and “extension” (file extension).

A note on filtering: each listed filter is joined with an AND, but each option within that filter is joined with an OR. For example, the search below will return any products that are ‘science’ type and have a calibration level of 2 or 3.

scienceProducts = Observations.filter_products(dataProducts, productType=["SCIENCE"],
                                               calib_level=[2,3], mrp_only=False)

scienceProducts.show_in_notebook(display_length=5)
Table length=8
idxobsIDobs_collectiondataproduct_typeobs_iddescriptiontypedataURIproductTypeproductGroupDescriptionproductSubGroupDescriptionproductDocumentationURLprojectprvversionproposal_idproductFilenamesizeparent_obsiddataRightscalib_level
025153181HLAimagehst_05766_04_wfpc2_f555w_wf_02HLA simple fits science imageSmast:HLA/url/cgi-bin/getdata.cgi?dataset=hst_05766_04_wfpc2_f555w_wf_02_drz.fitsSCIENCE--DRZ--HLA--5766hst_05766_04_wfpc2_f555w_wf_02_drz.fits1028160025579950PUBLIC2
125579950HLAimagehst_05766_04_wfpc2_total_wfHLA simple fits science imageCmast:HLA/url/cgi-bin/getdata.cgi?dataset=hst_05766_04_wfpc2_total_wf_drz.fitsSCIENCEMinimum Recommended ProductsDRZ--HLA--5766hst_05766_04_wfpc2_total_wf_drz.fits3079872025579950PUBLIC3
224556184HSTimageu2ms0402tDADS C0F file - Calibrated exposure WFPC/WFPC2/FOC/FOS/GHRS/HSPSmast:HST/product/u2ms0402t_c0f.fitsSCIENCE--C0F--CALWFPC22.5.3 (Sep 4, 2008)5766u2ms0402t_c0f.fits1030752025579950PUBLIC2
324556184HSTimageu2ms0402tDADS C0M file - Calibrated exposure WFPC2Smast:HST/product/u2ms0402t_c0m.fitsSCIENCE--C0M--CALWFPC22.5.3 (Sep 4, 2008)5766u2ms0402t_c0m.fits1030752025579950PUBLIC2
424556184HSTimageu2ms0402tDADS C1F file - Calibrated exposure WFPC/FOC/FOS/GHRS/HSPSmast:HST/product/u2ms0402t_c1f.fitsSCIENCE--C1F--CALWFPC22.5.3 (Sep 4, 2008)5766u2ms0402t_c1f.fits518400025579950PUBLIC2
524556184HSTimageu2ms0402tDADS C1M fileSmast:HST/product/u2ms0402t_c1m.fitsSCIENCE--C1M--CALWFPC22.5.3 (Sep 4, 2008)5766u2ms0402t_c1m.fits517824025579950PUBLIC2
624556184HSTimageu2ms0402tDADS DRW fileSmast:HST/product/u2ms0402t_drw.fitsSCIENCE--DRW--CALWFPC22.5.3 (Sep 4, 2008)5766u2ms0402t_drw.fits14580288025579950PUBLIC3
724556184HSTimageu2ms0402tDADS FLT file - Calibrated exposure ACS/WFC3/STIS/COSSmast:HST/product/u2ms0402t_flt.fitsSCIENCE--FLT--CALWFPC22.5.3 (Sep 4, 2008)5766u2ms0402t_flt.fits2592000025579950PUBLIC2

Downloading Products#

Passing a table of products (like the one above) to download_products will download the entire table. You can also pass a list of Observation IDs (obs_id) if you know them.

download_products also allows you to filter data as you request the download. In the example below, we will only download the drizzled files (drz.fits).

Products will by default be downloaded into the current working directory, in a subdirectory called “mastDownload.”
The full local file paths will have the form “mastDownload/Mission/Observation ID/file.”

# This is the filtered download of the scienceProducts table
manifest = Observations.download_products(scienceProducts, extension=("drz.fits"))

# Uncomment below for "plain" download of the scienceProducts table
# manifest = Observations.download_products(scienceProducts)

Downloading URL https://mast.stsci.edu/api/v0.1/Download/file?uri=mast:HLA/url/cgi-bin/getdata.cgi?dataset=hst_05766_04_wfpc2_f555w_wf_02_drz.fits to ./mastDownload/HLA/hst_05766_04_wfpc2_f555w_wf_02/hst_05766_04_wfpc2_f555w_wf_02_drz.fits ...

 [Done]

Downloading URL https://mast.stsci.edu/api/v0.1/Download/file?uri=mast:HLA/url/cgi-bin/getdata.cgi?dataset=hst_05766_04_wfpc2_total_wf_drz.fits to ./mastDownload/HLA/hst_05766_04_wfpc2_total_wf/hst_05766_04_wfpc2_total_wf_drz.fits ...

 [Done]

Note: download_products includes caching by default. If you have downloaded the files before, they will not be downloaded again unless caching is turned off. This may cause issues if the data is updated and the filename remains the same!

manifest.show_in_notebook()
Table length=2
idxLocal PathStatusMessageURL
0./mastDownload/HLA/hst_05766_04_wfpc2_f555w_wf_02/hst_05766_04_wfpc2_f555w_wf_02_drz.fitsCOMPLETENoneNone
1./mastDownload/HLA/hst_05766_04_wfpc2_total_wf/hst_05766_04_wfpc2_total_wf_drz.fitsCOMPLETENoneNone

The manifest returns useful information about the status of the files. You can find the local path, along with a status. This will either be COMPLETE, SKIPPED, or ERROR. If the status is ERROR, there will be additional information in the ‘Message’ column. The URL field includes a link to directly download the data.

Displaying Data#

At this point the data is ready for analysis, and we are done querying the MAST Archive.

Below we take a look at the data files using astropy.fits and matplotlib.

# Get the filenames from the manifest
filename0 = manifest['Local Path'][0]
filename1 = manifest['Local Path'][1]

# Open the files using astropy.fits
file1 = fits.open(filename0)
file2 = fits.open(filename1)

f, (ax1, ax2) = plt.subplots(1, 2)
f.set_figheight(5)
f.set_figwidth(12)
ax1.imshow(file1[0].data, cmap="inferno", norm=SymLogNorm(linthresh=0.03,vmin=0, vmax=1.5))
ax2.imshow(file2['SCI'].data, cmap="inferno", norm=SymLogNorm(linthresh=0.03,vmin=-0.01, vmax=1.5))

<matplotlib.image.AxesImage at 0x7fcf7fbede50>
../../../_images/efc1a22a2a620d0aeaab86c67a97692be19a5ca0ebd01f52f94450cc440ad665.png

We see the same region of the sky, with subtle differences caused by the differing filters.

Further Reading#

Full documentation on astroquery.mast can be found here.

About this Notebook#

For additonal questions, comments, or feedback, please email archive@stsci.edu.

Authors: Thomas Dutkiewicz, Scott Fleming
Keywords: MAST, astroquery
Latest update Oct 2022
Next Review: Apr 2023

Top of Page Space Telescope Logo

Contents