Wildcard Handling with Astroquery.mast

Wildcard Handling with Astroquery.mast#

Learning Goals#

By the end of this tutorial, you will:

  • Use the wildcards available for astroquery.mast.Observations criteria queries

  • Broaden and refine astroquery.mast.Observations criteria queries

  • Fully utilize the instrument_name criteria, especially for JWST queries

  • Query for moving targets using target ephemeris and time criteria such as t_min and t_max

Introduction#

This Notebook demonstrates the use of wildcards in astroquery.mast.Observations criteria queries. The use of wildcards is encouraged for certain criteria types (namely, string object types) to ensure your query returns all results.

We will demonstrate 3 use-cases for wildcards when doing criteria queries and emphasize certain criteria where wildcard usage is highly encouraged, particularly for JWST queries. We will also use the last example to demonstrate the use of value ranges when working with float object criteria types.

The workflow for this notebook consists of:

Imports#

import astropy.units as u
import matplotlib.pyplot as plt

from astropy.coordinates import SkyCoord
from astropy.table import Table, unique, vstack
from astropy.time import Time
from astroquery.mast import Observations

Wildcards with astroquery.mast.Observations#

The use of wildcards when making astroquery.mast.Observations queries can help ensure you retrieve all observations without leaving anything out. The available wildcards are % and *: % replaces a single character, while * replaces more than one character preceding, following, or in between the existing characters, depending on its placement. See the Observation Criteria Queries section in the astroquery.mast documentation for more information on the wildcards.

Wildcards are only available for certain criteria. string type objects accept wildcards, but float, integer, or any other objects do not accept wildcards.

Users may call the get_metadata method to see the list of query criteria and their data types. The criteria listed as string objects under the Data Type column are criteria that can be called with wildcards:

Observations.get_metadata("observations")
Table length=37
Column NameColumn LabelData TypeUnitsDescriptionExamples/Valid Values
str21str25str7str10str72str116
intentTypeObservation TypestringWhether observation is for science or calibration.Valid values: science, calibration
obs_collectionMissionstringCollectionE.g. SWIFT, PS1, HST, IUE
provenance_nameProvenance NamestringProvenance name, or source of dataE.g. TASOC, CALSTIS, PS1
instrument_nameInstrumentstringInstrument NameE.g. WFPC2/WFC, UVOT, STIS/CCD
projectProjectstringProcessing projectE.g. HST, HLA, EUVE, hlsp_legus
filtersFiltersstringInstrument filtersF469N, NUV, FUV, LOW DISP, MIRROR
wavelength_regionWavebandstringEnergy BandEUV, XRAY, OPTICAL
target_nameTarget NamestringTarget NameEx. COMET-67P-CHURYUMOV-GER-UPDATE
target_classificationTarget ClassificationstringType of targetEx. COMET;COMET BEING ORBITED BY THE ROSETTA SPACECRAFT;SOLAR SYSTEM
..................
obsidProduct Group IDintegerDatabase identifier for obs_idLong integer, e.g. 2007590987
dataRightsData RightsstringData Rightsvalid values: public,exclusive_access,restricted
mtFlagMoving TargetbooleanMoving Target FlagIf True, observation contains a moving target, if False or absent observation may or may not contain a moving target
srcDenNumber of Catalog ObjectsfloatNumber of cataloged objects found in observation
dataURLData URLstringData URL
proposal_typeProposal TypestringType of telescope proposalEg. 3PI, GO, GO/DD, HLA, GII, AIS
sequence_numberSequence NumberintegerSequence number, e.g. Kepler quarter or TESS sector
wave_minMin. WavelengthfloatnmMin.wavelength
wave_maxMax. WavelengthfloatnmMax.wavelength
wave_regionWavebandstringEnergy Band UCD

Case 1: Wildcard Search with instrument_name#

For our first example we will search for all NIRISS observations taken by a certain proposal/program PI. Our two query criteria are proposal_pi and instrument_name, which are both string object criteria. As such, both can be wildcarded for ease of use.

In fact, it is sometimes necessary to use wildcards when searching on instrument_name. Both HST and JWST use instrument configurations in this field to allow for more precise advanced searches (e.g. NIRISS/IMAGE and STIS/FUV-MAMA). When performing a “generic” search, you must include a wildcard or these more detailed results will be excluded.

We will demonstrate this by looking at the results for the query below:

observations = Observations.query_criteria(proposal_pi="Espinoza, Nestor",
                                           instrument_name="NIRISS*")
observations
Table masked=True length=234
intentTypeobs_collectionprovenance_nameinstrument_nameprojectfilterswave_regiontarget_nametarget_classificationobs_ids_ras_decdataproduct_typeproposal_picalib_levelt_mint_maxt_exptimewavelength_regionem_minem_maxobs_titlet_obs_releaseproposal_idproposal_typesequence_numbers_regionjpegURLdataURLdataRightsmtFlagsrcDenobsidobjIDwave_minwave_max
str7str4str7str12str4str13str8str14str54str50float64float64str10str16int64float64float64float64str8float64float64str70float64str4str3int64str136str80str81str6boolfloat64str9str10float64float64
scienceJWSTCALJWSTNIRISS/IMAGEJWSTCLEAR;GR700XDINFRAREDECLIPTIC-RA80Calibration; External flat fieldjw01541005001_0210h_00001_nis124.1008916666666819.23126111111111imageEspinoza, Nestor259688.6617246296359688.665328414354300.63INFRARED600.02800.0NIRISS Sensitivity and Stability for Transiting Exoplanet Observations59774.85416661541COM--POLYGON 124.079361272 19.219997306 124.088204918 19.256195264 124.126749247 19.247627141 124.117926871 19.211434025mast:JWST/product/jw01541005001_0210h_00001_nis_trapsfilled.jpgmast:JWST/product/jw01541005001_0210h_00001_nis_rateints.fitsPUBLICFalsenan796613761162083517600.02800.0
scienceJWSTCALJWSTNIRISS/IMAGEJWSTCLEAR;GR700XDINFRAREDECLIPTIC-RA80Calibration; External flat fieldjw01541005001_0210v_00001_nis124.1008916666666819.23126111111111imageEspinoza, Nestor259688.6981105208359688.70171430556300.63INFRARED600.02800.0NIRISS Sensitivity and Stability for Transiting Exoplanet Observations59774.85416661541COM--POLYGON 124.070680154 19.221940542 124.079522008 19.258138913 124.118067242 19.249572589 124.109246657 19.213379062mast:JWST/product/jw01541005001_0210v_00001_nis_trapsfilled.jpgmast:JWST/product/jw01541005001_0210v_00001_nis_rateints.fitsPUBLICFalsenan796614021162083558600.02800.0
scienceJWSTCALJWSTNIRISS/IMAGEJWSTCLEAR;GR700XDINFRAREDECLIPTIC-RA80Calibration; External flat fieldjw01541005001_0210b_00001_nis124.1008916666666819.23126111111111imageEspinoza, Nestor259688.6461513078759688.64975509259300.63INFRARED600.02800.0NIRISS Sensitivity and Stability for Transiting Exoplanet Observations59774.85416661541COM--POLYGON 124.080801154 19.213559815 124.089644763 19.249757705 124.128187507 19.241189289 124.119365169 19.20499624mast:JWST/product/jw01541005001_0210b_00001_nis_trapsfilled.jpgmast:JWST/product/jw01541005001_0210b_00001_nis_rateints.fitsPUBLICFalsenan796614331162083606600.02800.0
scienceJWSTCALJWSTNIRISS/IMAGEJWSTCLEAR;GR700XDINFRAREDECLIPTIC-RA80Calibration; External flat fieldjw01541005001_0210p_00001_nis124.1008916666666819.23126111111111imageEspinoza, Nestor259688.6825120138959688.68611579861300.63INFRARED600.02800.0NIRISS Sensitivity and Stability for Transiting Exoplanet Observations59774.85416661541COM--POLYGON 124.072120417 19.215503156 124.080962242 19.251701457 124.119505889 19.243134834 124.110685335 19.206941375mast:JWST/product/jw01541005001_0210p_00001_nis_trapsfilled.jpgmast:JWST/product/jw01541005001_0210p_00001_nis_rateints.fitsPUBLICFalsenan796614581162083654600.02800.0
scienceJWSTCALJWSTNIRISS/IMAGEJWSTCLEAR;GR700XDINFRAREDECLIPTIC-RA80Calibration; External flat fieldjw01541005001_0210n_00001_nis124.1008916666666819.23126111111111imageEspinoza, Nestor259688.6773216435259688.68092542824300.63INFRARED600.02800.0NIRISS Sensitivity and Stability for Transiting Exoplanet Observations59774.85416661541COM--POLYGON 124.075014085 19.214855403 124.083856539 19.25105356 124.122399877 19.242486306 124.113578694 19.206292991mast:JWST/product/jw01541005001_0210n_00001_nis_trapsfilled.jpgmast:JWST/product/jw01541005001_0210n_00001_nis_rateints.fitsPUBLICFalsenan796614901162083702600.02800.0
scienceJWSTCALJWSTNIRISS/IMAGEJWSTCLEAR;GR700XDINFRAREDECLIPTIC-RA80Calibration; External flat fieldjw01541005001_0210t_00001_nis124.1008916666666819.23126111111111imageEspinoza, Nestor259688.6929068171359688.696510601854300.63INFRARED600.02800.0NIRISS Sensitivity and Stability for Transiting Exoplanet Observations59774.85416661541COM--POLYGON 124.073573871 19.221292827 124.082416316 19.257491062 124.12096125 19.248924145 124.112140074 19.212730753mast:JWST/product/jw01541005001_0210t_00001_nis_trapsfilled.jpgmast:JWST/product/jw01541005001_0210t_00001_nis_rateints.fitsPUBLICFalsenan796615221162083750600.02800.0
scienceJWSTCALJWSTNIRISS/IMAGEJWSTCLEAR;GR700XDINFRAREDECLIPTIC-RA80Calibration; External flat fieldjw01541005001_0210x_00001_nis124.1008916666666819.23126111111111imageEspinoza, Nestor259688.7032890277859688.7068928125300.63INFRARED600.02800.0NIRISS Sensitivity and Stability for Transiting Exoplanet Observations59774.85416661541COM--POLYGON 124.069953497 19.219045681 124.078795054 19.255244082 124.117339642 19.246677893 124.108519354 19.210484334mast:JWST/product/jw01541005001_0210x_00001_nis_trapsfilled.jpgmast:JWST/product/jw01541005001_0210x_00001_nis_rateints.fitsPUBLICFalsenan796615421162083804600.02800.0
scienceJWSTCALJWSTNIRISS/IMAGEJWSTCLEAR;GR700XDINFRAREDECLIPTIC-RA80Calibration; External flat fieldjw01541005001_02101_00001_nis124.1008916666666819.23126111111111imageEspinoza, Nestor259688.6200654050959688.623669189816300.63INFRARED600.02800.0NIRISS Sensitivity and Stability for Transiting Exoplanet Observations59774.85416661541COM--POLYGON 124.08369426 19.212911861 124.092538264 19.249109658 124.131080754 19.240540834 124.122258022 19.204347878mast:JWST/product/jw01541005001_02101_00001_nis_trapsfilled.jpgmast:JWST/product/jw01541005001_02101_00001_nis_rateints.fitsPUBLICFalsenan796615751162083855600.02800.0
scienceJWSTCALJWSTNIRISS/IMAGEJWSTCLEAR;GR700XDINFRAREDECLIPTIC-RA80Calibration; External flat fieldjw01541005001_02103_00001_nis124.1008916666666819.23126111111111imageEspinoza, Nestor259688.62538688657559688.6289906713300.63INFRARED600.02800.0NIRISS Sensitivity and Stability for Transiting Exoplanet Observations59774.85416661541COM--POLYGON 124.084421799 19.215806744 124.093266409 19.252004442 124.131809472 19.243435191 124.122986133 19.207242333mast:JWST/product/jw01541005001_02103_00001_nis_trapsfilled.jpgmast:JWST/product/jw01541005001_02103_00001_nis_rateints.fitsPUBLICFalsenan796615981162083906600.02800.0
............................................................................................................
scienceJWSTCALJWSTNIRISS/IMAGEJWSTCLEAR;GR700XDINFRAREDJWST-NEP-TDFUnidentified; Blank fieldjw06658008002_02103_00001_nis260.865.76444444444445imageEspinoza, Nestor260621.0848671527860621.08623409722107.368INFRARED600.02800.0CAL-NIS-307: SOSS background measurements60621.295509276658CAL--POLYGON 260.819156955 65.735747111 260.745715464 65.757424648 260.799189758 65.787690727 260.872691811 65.766014184mast:JWST/product/jw06658008002_02103_00001_nis_trapsfilled.jpgmast:JWST/product/jw06658008002_02103_00001_nis_rateints.fitsPUBLICFalsenan2329476951166990995600.02800.0
scienceJWSTCALJWSTNIRISS/IMAGEJWSTCLEAR;GR700XDINFRAREDECLIPTIC-RA00Calibration; External flat fieldjw06658001006_02101_00001_nis9.3443875000000021.4307027777777779imageEspinoza, Nestor260530.8071092476860530.80847619213107.368INFRARED600.02800.0CAL-NIS-307: SOSS background measurements60530.887083276658CAL--POLYGON 9.353265571 1.459105935 9.366939477 1.424563876 9.332107585 1.410960808 9.318412147 1.445484488mast:JWST/product/jw06658001006_02101_00001_nis_trapsfilled.jpgmast:JWST/product/jw06658001006_02101_00001_nis_rateints.fitsPUBLICFalsenan2269857321166992058600.02800.0
scienceJWSTCALJWSTNIRISS/IMAGEJWSTCLEAR;GR700XDINFRAREDECLIPTIC-RA40Calibration; External flat fieldjw06658005003_02103_00001_nis59.056575-9.294311111111085imageEspinoza, Nestor260564.5675826157460564.568949560184107.368INFRARED600.02800.0CAL-NIS-307: SOSS background measurements60565.962870276658CAL--POLYGON 59.093441048 -9.269664443 59.095059218 -9.306778604 59.057204639 -9.308219169 59.055564257 -9.271115095mast:JWST/product/jw06658005003_02103_00001_nis_trapsfilled.jpgmast:JWST/product/jw06658005003_02103_00001_nis_rateints.fitsPUBLICFalsenan2298709621166992715600.02800.0
scienceJWSTCALJWSTNIRISS/IMAGEJWSTCLEAR;GR700XDINFRAREDECLIPTIC-RA40Calibration; External flat fieldjw06658005003_02101_00001_nis59.056575-9.294311111111085imageEspinoza, Nestor260564.5623433680560564.5637103125107.368INFRARED600.02800.0CAL-NIS-307: SOSS background measurements60565.95998836658CAL--POLYGON 59.093803992 -9.278944538 59.095422272 -9.316058696 59.05756669 -9.317499326 59.055926202 -9.280395255mast:JWST/product/jw06658005003_02101_00001_nis_trapsfilled.jpgmast:JWST/product/jw06658005003_02101_00001_nis_rateints.fitsPUBLICFalsenan2298709641166992757600.02800.0
scienceJWSTCALJWSTNIRISS/IMAGEJWSTCLEAR;GR700XDINFRAREDECLIPTIC-RA00Calibration; External flat fieldjw06658003010_02101_00001_nis9.3443875000000021.4307027777777779imageEspinoza, Nestor260530.92422913194660530.92559607639107.368INFRARED600.02800.0CAL-NIS-307: SOSS background measurements60530.957789266658CAL--POLYGON 9.370693979 1.465912026 9.384367597 1.431369837 9.349535473 1.417767099 9.33584032 1.452290909mast:JWST/product/jw06658003010_02101_00001_nis_trapsfilled.jpgmast:JWST/product/jw06658003010_02101_00001_nis_rateints.fitsPUBLICFalsenan2269857841166993727600.02800.0
scienceJWSTCALJWSTNIRISS/IMAGEJWSTCLEAR;GR700XDINFRAREDECLIPTIC-RA00Calibration; External flat fieldjw06658001001_02101_00001_nis9.3443875000000021.4307027777777779imageEspinoza, Nestor260530.7720048379660530.77337178241107.368INFRARED600.02800.0CAL-NIS-307: SOSS background measurements60530.86025456658CAL--POLYGON 9.339216571 1.443648859 9.352890097 1.409106686 9.318058323 1.395503909 9.304363269 1.430027705mast:JWST/product/jw06658001001_02101_00001_nis_trapsfilled.jpgmast:JWST/product/jw06658001001_02101_00001_nis_rateints.fitsPUBLICFalsenan2269411761166993859600.02800.0
scienceJWSTCALJWSTNIRISS/IMAGEJWSTCLEAR;GR700XDINFRAREDJWST-NEP-TDFUnidentified; Blank fieldjw06658008001_03107_00001_nis260.865.76444444444445imageEspinoza, Nestor260621.0726708680560621.0740378125107.368INFRARED600.02800.0CAL-NIS-307: SOSS background measurements60621.296249966658CAL--POLYGON 260.814240136 65.748770819 260.740756903 65.770445687 260.794251676 65.800713711 260.867795543 65.779039824mast:JWST/product/jw06658008001_03107_00001_nis_trapsfilled.jpgmast:JWST/product/jw06658008001_03107_00001_nis_rateints.fitsPUBLICFalsenan2329476861166995136600.02800.0
scienceJWSTCALJWSTNIRISS/IMAGEJWSTCLEAR;GR700XDINFRAREDJWST-NEP-TDFUnidentified; Blank fieldjw06658008001_03101_00001_nis260.865.76444444444445imageEspinoza, Nestor260621.0569449537160621.058311898145107.368INFRARED600.02800.0CAL-NIS-307: SOSS background measurements60621.291296276658CAL--POLYGON 260.827622468 65.75633845 260.754128893 65.778019731 260.807655094 65.808283086 260.881209297 65.786602772mast:JWST/product/jw06658008001_03101_00001_nis_trapsfilled.jpgmast:JWST/product/jw06658008001_03101_00001_nis_rateints.fitsPUBLICFalsenan2329476891166995249600.02800.0
scienceJWSTCALJWSTNIRISS/IMAGEJWSTCLEAR;GR700XDINFRAREDJWST-NEP-TDFUnidentified; Blank fieldjw06658008001_03105_00001_nis260.865.76444444444445imageEspinoza, Nestor260621.0674323495460621.06879929398107.368INFRARED600.02800.0CAL-NIS-307: SOSS background measurements60621.294120376658CAL--POLYGON 260.832534032 65.743314148 260.759082488 65.764998251 260.812588559 65.795259549 260.886100657 65.773576426mast:JWST/product/jw06658008001_03105_00001_nis_trapsfilled.jpgmast:JWST/product/jw06658008001_03105_00001_nis_rateints.fitsPUBLICFalsenan2329476911166995324600.02800.0
scienceJWSTCALJWSTNIRISS/IMAGEJWSTCLEAR;GR700XDINFRAREDJWST-NEP-TDFUnidentified; Blank fieldjw06658008001_03103_00001_nis260.865.76444444444445imageEspinoza, Nestor260621.0621938425960621.063560787035107.368INFRARED600.02800.0CAL-NIS-307: SOSS background measurements60621.291087946658CAL--POLYGON 260.845918438 65.750879419 260.772455995 65.772569603 260.825992669 65.802826471 260.899515662 65.781137254mast:JWST/product/jw06658008001_03103_00001_nis_trapsfilled.jpgmast:JWST/product/jw06658008001_03103_00001_nis_rateints.fitsPUBLICFalsenan2329476941166995380600.02800.0

Our query returned many NIRISS observations led by the PI Dr. Espinoza. Let’s get all the unique values under the instrument_name column to see what our * wildcard picked up.

set(observations['instrument_name'])

{np.str_('NIRISS/IMAGE'), np.str_('NIRISS/SOSS')}

Our observations have the advanced labeling; had we simply set instrument_name = "NIRISS", we would have missed several observations. For more details on this advanced labeling, see the JWST Instrument Names page.

A note of caution: There is such a thing as too many wildcards#

You can be too generous with the wildcards, so be sure to exercise caution in their use. Too much ambiguity can lead to unintended results. Let’s take a look at our example below.

observations = Observations.query_criteria(proposal_pi='Espinoza, Nestor',
                                           instrument_name='NIR*') # Surely only one instrument begins with 'NIR'
set(observations['instrument_name'])

{np.str_('NIRISS/IMAGE'), np.str_('NIRISS/SOSS'), np.str_('NIRSPEC/SLIT')}

This query returns NIRSPEC/SLIT observations in addition to the NIRISS ones, which is not what we intended.

Case 2: Wildcard Search with instrument_name and proposal_id#

Let’s add an additional string criterion and wildcard into the mix. We’ll do this with the proposal_id field which, despite its numeric content, is encoded as a string.

Let’s query for a four digit proposal/program IDs that begin with 15.

observations = Observations.query_criteria(proposal_pi='Espinoza, Nestor',
                                           instrument_name='NIRISS*',
                                           proposal_id=['15%%']) # Only a four digit result will match this

set(observations['proposal_id']), set(observations['instrument_name'])

({np.str_('1512'), np.str_('1541')},
 {np.str_('NIRISS/IMAGE'), np.str_('NIRISS/SOSS')})

Resources#

The following is a list of resources that were referenced throughout the tutorial, as well as some additional references that you may find useful:

Citations#

If you use any of astroquery’s tools for published research, please cite the authors. Follow this link for more information about citing astroquery:

About This Notebook#

If you have comments or questions on this notebook, please contact us through the Archive Help Desk e-mail at archive@stsci.edu.

Author: Jenny V. Medina
Keywords: astroquery, wildcards, moving target
Last Updated: Jun 2023