Wildcard Handling with Astroquery.mast

Wildcard Handling with Astroquery.mast#

Learning Goals#

By the end of this tutorial, you will:

  • Use the wildcards available for astroquery.mast.Observations criteria queries

  • Broaden and refine astroquery.mast.Observations criteria queries

  • Fully utilize the instrument_name criteria, especially for JWST queries

  • Query for moving targets using target ephemeris and time criteria such as t_min and t_max

Introduction#

This Notebook demonstrates the use of wildcards in astroquery.mast.Observations criteria queries. The use of wildcards is encouraged for certain criteria types (namely, string object types) to ensure your query returns all results.

We will demonstrate 3 use-cases for wildcards when doing criteria queries and emphasize certain criteria where wildcard usage is highly encouraged, particularly for JWST queries. We will also use the last example to demonstrate the use of value ranges when working with float object criteria types.

The workflow for this notebook consists of:

Imports#

import astropy.units as u
import matplotlib.pyplot as plt

from astropy.coordinates import SkyCoord
from astropy.table import Table, unique, vstack
from astropy.time import Time
from astroquery.mast import Observations

Wildcards with astroquery.mast.Observations#

The use of wildcards when making astroquery.mast.Observations queries can help ensure you retrieve all observations without leaving anything out. The available wildcards are % and *: % replaces a single character, while * replaces more than one character preceding, following, or in between the existing characters, depending on its placement. See the Observation Criteria Queries section in the astroquery.mast documentation for more information on the wildcards.

Wildcards are only available for certain criteria. string type objects accept wildcards, but float, integer, or any other objects do not accept wildcards.

Users may call the get_metadata method to see the list of query criteria and their data types. The criteria listed as string objects under the Data Type column are criteria that can be called with wildcards:

Observations.get_metadata("observations")
Table length=34
Column NameColumn LabelData TypeUnitsDescriptionExamples/Valid Values
str21str25str7str10str72str116
intentTypeObservation TypestringWhether observation is for science or calibration.Valid values: science, calibration
obs_collectionMissionstringCollectionE.g. SWIFT, PS1, HST, IUE
provenance_nameProvenance NamestringProvenance name, or source of dataE.g. TASOC, CALSTIS, PS1
instrument_nameInstrumentstringInstrument NameE.g. WFPC2/WFC, UVOT, STIS/CCD
projectProjectstringProcessing projectE.g. HST, HLA, EUVE, hlsp_legus
filtersFiltersstringInstrument filtersF469N, NUV, FUV, LOW DISP, MIRROR
wavelength_regionWavebandstringEnergy BandEUV, XRAY, OPTICAL
target_nameTarget NamestringTarget NameEx. COMET-67P-CHURYUMOV-GER-UPDATE
target_classificationTarget ClassificationstringType of targetEx. COMET;COMET BEING ORBITED BY THE ROSETTA SPACECRAFT;SOLAR SYSTEM
..................
s_regions_regionstringICRS ShapeSTC/S FootprintWill be ICRS circle or polygon. E.g. CIRCLE ICRS 17.71740689 -58.40043015 0.625
jpegURLjpegURLstringPreview Image URLhttps://archive.stsci.edu/hst/previews/N4QF/N4QF18090.jpg
distanceDistance (")floatarcsecAngular separation between searched coordinates and center of obsevation
obsidProduct Group IDintegerDatabase identifier for obs_idLong integer, e.g. 2007590987
dataRightsData RightsstringData Rightsvalid values: public,exclusive_access,restricted
mtFlagMoving TargetbooleanMoving Target FlagIf True, observation contains a moving target, if False or absent observation may or may not contain a moving target
srcDenNumber of Catalog ObjectsfloatNumber of cataloged objects found in observation
dataURLData URLstringData URL
proposal_typeProposal TypestringType of telescope proposalEg. 3PI, GO, GO/DD, HLA, GII, AIS
sequence_numberSequence NumberintegerSequence number, e.g. Kepler quarter or TESS sector

Case 1: Wildcard Search with instrument_name#

For our first example we will search for all NIRISS observations taken by a certain proposal/program PI. Our two query criteria are proposal_pi and instrument_name, which are both string object criteria. As such, both can be wildcarded for ease of use.

In fact, it is sometimes necessary to use wildcards when searching on instrument_name. Both HST and JWST use instrument configurations in this field to allow for more precise advanced searches (e.g. NIRISS/IMAGE and STIS/FUV-MAMA). When performing a “generic” search, you must include a wildcard or these more detailed results will be excluded.

We will demonstrate this by looking at the results for the query below:

observations = Observations.query_criteria(proposal_pi="Espinoza, Nestor",
                                           instrument_name="NIRISS*")
observations
Table masked=True length=234
intentTypeobs_collectionprovenance_nameinstrument_nameprojectfilterswavelength_regiontarget_nametarget_classificationobs_ids_ras_decdataproduct_typeproposal_picalib_levelt_mint_maxt_exptimeem_minem_maxobs_titlet_obs_releaseproposal_idproposal_typesequence_numbers_regionjpegURLdataURLdataRightsmtFlagsrcDenobsidobjID
str7str4str7str12str4str13str8str14str54str50float64float64str10str16int64float64float64float64float64float64str70float64str4str3int64str132str80str81str6boolfloat64str9str10
scienceJWSTCALJWSTNIRISS/IMAGEJWSTCLEAR;GR700XDINFRAREDECLIPTIC-RA20Calibration; External flat fieldjw04479005004_02103_00001_nis27.65213333333333511.538361111111112imageEspinoza, Nestor260302.7240237847260302.72539072917107.368600.02800.0SOSS background measurements60302.969837914479CAL--POLYGON 27.630184308 11.515689331 27.617041473 11.550534444 27.652891551 11.563337498 27.666052471 11.528508455mast:JWST/product/jw04479005004_02103_00001_nis_trapsfilled.jpgmast:JWST/product/jw04479005004_02103_00001_nis_rateints.fitsPUBLICFalsenan199450721994897958
scienceJWSTCALJWSTNIRISS/IMAGEJWSTCLEAR;GR700XDINFRAREDECLIPTIC-RA20Calibration; External flat fieldjw04479005004_02101_00001_nis27.65213333333333511.538361111111112imageEspinoza, Nestor260302.7187734259360302.72014037037107.368600.02800.0SOSS background measurements60302.969814774479CAL--POLYGON 27.633430036 11.506963853 27.620287986 11.541809103 27.656137087 11.554611785 27.669297224 11.519782607mast:JWST/product/jw04479005004_02101_00001_nis_trapsfilled.jpgmast:JWST/product/jw04479005004_02101_00001_nis_rateints.fitsPUBLICFalsenan199450728994897990
scienceJWSTCALJWSTNIRISS/IMAGEJWSTCLEAR;GR700XDINFRAREDECLIPTIC-RA00Calibration; External flat fieldjw04479002007_03101_00001_nis9.3443875000000021.4307027777777779imageEspinoza, Nestor260265.7251107291760265.728714502315300.63600.02800.0SOSS background measurements60265.778159644479CAL--POLYGON 9.333529438 1.410136387 9.317466748 1.443635061 9.351259548 1.459645711 9.367341619 1.426166404mast:JWST/product/jw04479002007_03101_00001_nis_trapsfilled.jpgmast:JWST/product/jw04479002007_03101_00001_nis_rateints.fitsPUBLICFalsenan192100165994898025
scienceJWSTCALJWSTNIRISS/IMAGEJWSTCLEAR;GR700XDINFRAREDECLIPTIC-RA20Calibration; External flat fieldjw04479003003_02103_00001_nis27.65213333333333511.538361111111112imageEspinoza, Nestor260302.6355372106560302.63690415509107.368600.02800.0SOSS background measurements60302.968773044479CAL--POLYGON 27.642397157 11.510166881 27.629256076 11.545012535 27.665105997 11.557814112 27.678265165 11.522984528mast:JWST/product/jw04479003003_02103_00001_nis_trapsfilled.jpgmast:JWST/product/jw04479003003_02103_00001_nis_rateints.fitsPUBLICFalsenan199450466994898037
scienceJWSTCALJWSTNIRISS/IMAGEJWSTCLEAR;GR700XDINFRAREDECLIPTIC-RA20Calibration; External flat fieldjw04479005002_02101_00001_nis27.65213333333333511.538361111111112imageEspinoza, Nestor260302.69455937560302.695926319444107.368600.02800.0SOSS background measurements60302.96901614479CAL--POLYGON 27.651364802 11.513369385 27.638224654 11.548215432 27.674075384 11.561015938 27.687233617 11.526185961mast:JWST/product/jw04479005002_02101_00001_nis_trapsfilled.jpgmast:JWST/product/jw04479005002_02101_00001_nis_rateints.fitsPUBLICFalsenan199450467994898038
scienceJWSTCALJWSTNIRISS/IMAGEJWSTCLEAR;GR700XDINFRAREDECLIPTIC-RA20Calibration; External flat fieldjw04479003002_02101_00001_nis27.65213333333333511.538361111111112imageEspinoza, Nestor260302.6181505555660302.6195175107.368600.02800.0SOSS background measurements60302.968796194479CAL--POLYGON 27.651364823 11.513369371 27.638224681 11.54821542 27.674075412 11.561015922 27.68723364 11.526185942mast:JWST/product/jw04479003002_02101_00001_nis_trapsfilled.jpgmast:JWST/product/jw04479003002_02101_00001_nis_rateints.fitsPUBLICFalsenan199450464994898046
scienceJWSTCALJWSTNIRISS/IMAGEJWSTCLEAR;GR700XDINFRAREDECLIPTIC-RA20Calibration; External flat fieldjw04479005002_02103_00001_nis27.65213333333333511.538361111111112imageEspinoza, Nestor260302.6998223263960302.70118927083107.368600.02800.0SOSS background measurements60302.970520744479CAL--POLYGON 27.648119374 11.522095277 27.634978398 11.556941172 27.67083009 11.569742092 27.68398915 11.534912266mast:JWST/product/jw04479005002_02103_00001_nis_trapsfilled.jpgmast:JWST/product/jw04479005002_02103_00001_nis_rateints.fitsPUBLICFalsenan199450727994898071
scienceJWSTCALJWSTNIRISS/IMAGEJWSTCLEAR;GR700XDINFRAREDECLIPTIC-RA20Calibration; External flat fieldjw04479003002_02103_00001_nis27.65213333333333511.538361111111112imageEspinoza, Nestor260302.6233898148160302.62475675926107.368600.02800.0SOSS background measurements60302.968773044479CAL--POLYGON 27.648119428 11.522095296 27.634978481 11.556941202 27.670830183 11.569742094 27.683989214 11.534912257mast:JWST/product/jw04479003002_02103_00001_nis_trapsfilled.jpgmast:JWST/product/jw04479003002_02103_00001_nis_rateints.fitsPUBLICFalsenan199450468994898074
scienceJWSTCALJWSTNIRISS/IMAGEJWSTCLEAR;GR700XDINFRAREDECLIPTIC-RA20Calibration; External flat fieldjw04479003003_02101_00001_nis27.65213333333333511.538361111111112imageEspinoza, Nestor260302.6302749884360302.63164193287107.368600.02800.0SOSS background measurements60302.968784624479CAL--POLYGON 27.639151475 11.518892283 27.626009483 11.553737756 27.661860336 11.566539828 27.675020412 11.531710425mast:JWST/product/jw04479003003_02101_00001_nis_trapsfilled.jpgmast:JWST/product/jw04479003003_02101_00001_nis_rateints.fitsPUBLICFalsenan199450469994898082
...................................................................................................
scienceJWSTCALJWSTNIRISS/IMAGEJWSTCLEAR;GR700XDINFRAREDECLIPTIC-RA80Calibration; External flat fieldjw01541005001_0210j_00001_nis124.1008916666666819.23126111111111imageEspinoza, Nestor259688.6669157291759688.670519513886300.63600.02800.0NIRISS Sensitivity and Stability for Transiting Exoplanet Observations59774.85416661541COM--POLYGON 124.076467567 19.220645089 124.085310607 19.256843186 124.12385524 19.248275671 124.115033469 19.212082416mast:JWST/product/jw01541005001_0210j_00001_nis_trapsfilled.jpgmast:JWST/product/jw01541005001_0210j_00001_nis_rateints.fitsPUBLICFalsenan796616231057582763
scienceJWSTCALJWSTNIRISS/IMAGEJWSTCLEAR;GR700XDINFRAREDECLIPTIC-RA80Calibration; External flat fieldjw01541005001_02109_00001_nis124.1008916666666819.23126111111111imageEspinoza, Nestor259688.6409601967659688.644563981485300.63600.02800.0NIRISS Sensitivity and Stability for Transiting Exoplanet Observations59774.85416661541COM--POLYGON 124.081528099 19.216454622 124.090372057 19.25265247 124.128915435 19.244083871 124.120092748 19.207890864mast:JWST/product/jw01541005001_02109_00001_nis_trapsfilled.jpgmast:JWST/product/jw01541005001_02109_00001_nis_rateints.fitsPUBLICFalsenan796616591057582773
scienceJWSTCALJWSTNIRISS/IMAGEJWSTCLEAR;GR700XDINFRAREDECLIPTIC-RA80Calibration; External flat fieldjw01541005001_0210l_00001_nis124.1008916666666819.23126111111111imageEspinoza, Nestor259688.6721179513959688.67572173611300.63600.02800.0NIRISS Sensitivity and Stability for Transiting Exoplanet Observations59774.85416661541COM--POLYGON 124.075740844 19.217750235 124.084583604 19.253948359 124.123127586 19.245380963 124.114306096 19.20918768mast:JWST/product/jw01541005001_0210l_00001_nis_trapsfilled.jpgmast:JWST/product/jw01541005001_0210l_00001_nis_rateints.fitsPUBLICFalsenan796616911057582782
scienceJWSTCALJWSTNIRISS/IMAGEJWSTCLEAR;GR700XDINFRAREDECLIPTIC-RA80Calibration; External flat fieldjw01541005001_02107_00001_nis124.1008916666666819.23126111111111imageEspinoza, Nestor259688.6357690972259688.639372881946300.63600.02800.0NIRISS Sensitivity and Stability for Transiting Exoplanet Observations59774.85416661541COM--POLYGON 124.082255001 19.219349464 124.091099278 19.255547276 124.129643297 19.246978522 124.12082029 19.21078555mast:JWST/product/jw01541005001_02107_00001_nis_trapsfilled.jpgmast:JWST/product/jw01541005001_02107_00001_nis_rateints.fitsPUBLICFalsenan796617111057582792
scienceJWSTCALJWSTNIRISS/IMAGEJWSTCLEAR;GR700XDINFRAREDECLIPTIC-RA80Calibration; External flat fieldjw01541005001_02105_00001_nis124.1008916666666819.23126111111111imageEspinoza, Nestor259688.6305772453759688.634181030095300.63600.02800.0NIRISS Sensitivity and Stability for Transiting Exoplanet Observations59774.85416661541COM--POLYGON 124.085148743 19.218701569 124.093993665 19.254899234 124.132537371 19.246329833 124.123713719 19.21013701mast:JWST/product/jw01541005001_02105_00001_nis_trapsfilled.jpgmast:JWST/product/jw01541005001_02105_00001_nis_rateints.fitsPUBLICFalsenan796617271057582798
scienceJWSTCALJWSTNIRISS/IMAGEJWSTCLEAR;GR700XDINFRAREDECLIPTIC-RA80Calibration; External flat fieldjw01541005001_0210r_00001_nis124.1008916666666819.23126111111111imageEspinoza, Nestor259688.6877164467659688.69132023148300.63600.02800.0NIRISS Sensitivity and Stability for Transiting Exoplanet Observations59774.85416661541COM--POLYGON 124.07284721 19.21839799 124.081689382 19.25459625 124.120233664 19.246029444 124.111412762 19.209836026mast:JWST/product/jw01541005001_0210r_00001_nis_trapsfilled.jpgmast:JWST/product/jw01541005001_0210r_00001_nis_rateints.fitsPUBLICFalsenan796617461057582803
scienceJWSTCALJWSTNIRISS/IMAGEJWSTCLEAR;GR700XDINFRAREDECLIPTIC-RA80Calibration; External flat fieldjw01541005001_0210f_00001_nis124.1008916666666819.23126111111111imageEspinoza, Nestor259688.6565327777859688.6601365625300.63600.02800.0NIRISS Sensitivity and Stability for Transiting Exoplanet Observations59774.85416661541COM--POLYGON 124.078634483 19.217102483 124.087477844 19.253300468 124.126021524 19.244732468 124.117199434 19.208539324mast:JWST/product/jw01541005001_0210f_00001_nis_trapsfilled.jpgmast:JWST/product/jw01541005001_0210f_00001_nis_rateints.fitsPUBLICFalsenan796617631057582812
scienceJWSTCALJWSTNIRISS/IMAGEJWSTCLEAR;GR700XDINFRAREDECLIPTIC-RA80Calibration; External flat fieldjw01541005001_0210z_00001_nis124.1008916666666819.23126111111111imageEspinoza, Nestor259688.7084919907459688.71209577546300.63600.02800.0NIRISS Sensitivity and Stability for Transiting Exoplanet Observations59774.85416661541COM--POLYGON 124.069226757 19.216150803 124.078067971 19.252349246 124.116611924 19.243783236 124.10779198 19.207589636mast:JWST/product/jw01541005001_0210z_00001_nis_trapsfilled.jpgmast:JWST/product/jw01541005001_0210z_00001_nis_rateints.fitsPUBLICFalsenan796617811057582816
scienceJWSTCALJWSTNIRISS/IMAGEJWSTCLEAR;GR700XDINFRAREDECLIPTIC-RA80Calibration; External flat fieldjw01541005001_0210d_00001_nis124.1008916666666819.23126111111111imageEspinoza, Nestor259688.6513424189859688.6549462037300.63600.02800.0NIRISS Sensitivity and Stability for Transiting Exoplanet Observations59774.85416661541COM--POLYGON 124.077907696 19.214207662 124.086750759 19.250405679 124.125293793 19.241837815 124.116472001 19.205644639mast:JWST/product/jw01541005001_0210d_00001_nis_trapsfilled.jpgmast:JWST/product/jw01541005001_0210d_00001_nis_rateints.fitsPUBLICFalsenan796618041057582819
scienceJWSTCALJWSTNIRISS/SOSSJWSTCLEAR;GR700XDINFRAREDHAT-P-14Star; Exoplanet Systems; Exoplanets; F dwarfs; F starsjw01541001001_04102_00001-seg001_nis260.116177163368638.24215651042749timeseriesEspinoza, Nestor259738.553906296359738.55835982639329.64600.02800.0NIRISS Sensitivity and Stability for Transiting Exoplanet Observations59774.85416661541COM--POLYGON 260.071030425 38.2494554 260.119367244 38.246686364 260.11850644 38.23799955 260.070173176 38.240758866mast:JWST/product/jw01541001001_04102_00001-seg001_nis_ramp.jpgmast:JWST/product/jw01541001001_04102_00001-seg001_nis_rateints.fitsPUBLICFalsenan857002051057802420

Our query returned many NIRISS observations led by the PI Dr. Espinoza. Let’s get all the unique values under the instrument_name column to see what our * wildcard picked up.

set(observations['instrument_name'])

{np.str_('NIRISS/IMAGE'), np.str_('NIRISS/SOSS')}

Our observations have the advanced labeling; had we simply set instrument_name = "NIRISS", we would have missed several observations. For more details on this advanced labeling, see the JWST Instrument Names page.

A note of caution: There is such a thing as too many wildcards#

You can be too generous with the wildcards, so be sure to exercise caution in their use. Too much ambiguity can lead to unintended results. Let’s take a look at our example below.

observations = Observations.query_criteria(proposal_pi='Espinoza, Nestor',
                                           instrument_name='NIR*') # Surely only one instrument begins with 'NIR'
set(observations['instrument_name'])

{np.str_('NIRISS/IMAGE'), np.str_('NIRISS/SOSS'), np.str_('NIRSPEC/SLIT')}

This query returns NIRSPEC/SLIT observations in addition to the NIRISS ones, which is not what we intended.

Case 2: Wildcard Search with instrument_name and proposal_id#

Let’s add an additional string criterion and wildcard into the mix. We’ll do this with the proposal_id field which, despite its numeric content, is encoded as a string.

Let’s query for a four digit proposal/program IDs that begin with 15.

observations = Observations.query_criteria(proposal_pi='Espinoza, Nestor',
                                           instrument_name='NIRISS*',
                                           proposal_id=['15%%']) # Only a four digit result will match this

set(observations['proposal_id']), set(observations['instrument_name'])

({np.str_('1512'), np.str_('1541')},
 {np.str_('NIRISS/IMAGE'), np.str_('NIRISS/SOSS')})

Resources#

The following is a list of resources that were referenced throughout the tutorial, as well as some additional references that you may find useful:

Citations#

If you use any of astroquery’s tools for published research, please cite the authors. Follow this link for more information about citing astroquery:

About This Notebook#

If you have comments or questions on this notebook, please contact us through the Archive Help Desk e-mail at archive@stsci.edu.

Author: Jenny V. Medina
Keywords: astroquery, wildcards, moving target
Last Updated: Jun 2023

Contents