Wildcard Handling with Astroquery.mast#


Learning Goals#

By the end of this tutorial, you will:

  • Use the wildcards available for astroquery.mast.Observations criteria queries

  • Broaden and refine astroquery.mast.Observations criteria queries

  • Fully utilize the instrument_name criteria, especially for JWST queries

  • Query for moving targets using target ephemeris and time criteria such as t_min and t_max

Introduction#

This Notebook demonstrates the use of wildcards in astroquery.mast.Observations criteria queries. The use of wildcards is encouraged for certain criteria types (namely, string object types) to ensure your query returns all results.

We will demonstrate 3 use-cases for wildcards when doing criteria queries and emphasize certain criteria where wildcard usage is highly encouraged, particularly for JWST queries. We will also use the last example to demonstrate the use of value ranges when working with float object criteria types.

The workflow for this notebook consists of:

Imports#

import astropy.units as u
import matplotlib.pyplot as plt

from astropy.coordinates import SkyCoord
from astropy.table import Table, unique, vstack
from astropy.time import Time
from astroquery.mast import Observations

Wildcards with astroquery.mast.Observations#

The use of wildcards when making astroquery.mast.Observations queries can help ensure you retrieve all observations without leaving anything out. The available wildcards are % and *: % replaces a single character, while * replaces more than one character preceding, following, or in between the existing characters, depending on its placement. See the Observation Criteria Queries section in the astroquery.mast documentation for more information on the wildcards.

Wildcards are only available for certain criteria. string type objects accept wildcards, but float, integer, or any other objects do not accept wildcards.

Users may call the get_metadata method to see the list of query criteria and their data types. The criteria listed as string objects under the Data Type column are criteria that can be called with wildcards:

Observations.get_metadata("observations")
Table length=34
Column NameColumn LabelData TypeUnitsDescriptionExamples/Valid Values
str21str25str7str10str72str116
intentTypeObservation TypestringWhether observation is for science or calibration.Valid values: science, calibration
obs_collectionMissionstringCollectionE.g. SWIFT, PS1, HST, IUE
provenance_nameProvenance NamestringProvenance name, or source of dataE.g. TASOC, CALSTIS, PS1
instrument_nameInstrumentstringInstrument NameE.g. WFPC2/WFC, UVOT, STIS/CCD
projectProjectstringProcessing projectE.g. HST, HLA, EUVE, hlsp_legus
filtersFiltersstringInstrument filtersF469N, NUV, FUV, LOW DISP, MIRROR
wavelength_regionWavebandstringEnergy BandEUV, XRAY, OPTICAL
target_nameTarget NamestringTarget NameEx. COMET-67P-CHURYUMOV-GER-UPDATE
target_classificationTarget ClassificationstringType of targetEx. COMET;COMET BEING ORBITED BY THE ROSETTA SPACECRAFT;SOLAR SYSTEM
..................
s_regions_regionstringICRS ShapeSTC/S FootprintWill be ICRS circle or polygon. E.g. CIRCLE ICRS 17.71740689 -58.40043015 0.625
jpegURLjpegURLstringPreview Image URLhttps://archive.stsci.edu/hst/previews/N4QF/N4QF18090.jpg
distanceDistance (")floatarcsecAngular separation between searched coordinates and center of obsevation
obsidProduct Group IDintegerDatabase identifier for obs_idLong integer, e.g. 2007590987
dataRightsData RightsstringData Rightsvalid values: public,exclusive_access,restricted
mtFlagMoving TargetbooleanMoving Target FlagIf True, observation contains a moving target, if False or absent observation may or may not contain a moving target
srcDenNumber of Catalog ObjectsfloatNumber of cataloged objects found in observation
dataURLData URLstringData URL
proposal_typeProposal TypestringType of telescope proposalEg. 3PI, GO, GO/DD, HLA, GII, AIS
sequence_numberSequence NumberintegerSequence number, e.g. Kepler quarter or TESS sector

Case 1: Wildcard Search with instrument_name#

For our first example we will search for all NIRISS observations taken by a certain proposal/program PI. Our two query criteria are proposal_pi and instrument_name, which are both string object criteria. As such, both can be wildcarded for ease of use.

In fact, it is sometimes necessary to use wildcards when searching on instrument_name. Both HST and JWST use instrument configurations in this field to allow for more precise advanced searches (e.g. NIRISS/IMAGE and STIS/FUV-MAMA). When performing a “generic” search, you must include a wildcard or these more detailed results will be excluded.

We will demonstrate this by looking at the results for the query below:

observations = Observations.query_criteria(proposal_pi="Espinoza, Nestor",
                                           instrument_name="NIRISS*")
observations
Table masked=True length=234
intentTypeobs_collectionprovenance_nameinstrument_nameprojectfilterswavelength_regiontarget_nametarget_classificationobs_ids_ras_decdataproduct_typeproposal_picalib_levelt_mint_maxt_exptimeem_minem_maxobs_titlet_obs_releaseproposal_idproposal_typesequence_numbers_regionjpegURLdataURLdataRightsmtFlagsrcDenobsidobjID
str7str4str7str12str4str13str8str14str54str50float64float64str10str16int64float64float64float64float64float64str70float64str4str3int64str132str63str81str6boolfloat64str9str9
scienceJWSTCALJWSTNIRISS/IMAGEJWSTCLEAR;GR700XDINFRAREDLAT60-LONG170Calibration; Stray light testjw06658006003_02103_00001_nis209.8708333333333455.98163888888889imageEspinoza, Nestor260730.8630181365760730.86438508102107.368600.02800.0CAL-NIS-307: SOSS background measurements60730.882534676658CAL--POLYGON 209.884353009 56.010420829 209.911590978 55.976539549 209.850552771 55.961351186 209.823235887 55.995199858mast:JWST/product/jw06658006003_02103_00001_nis_trapsfilled.jpgmast:JWST/product/jw06658006003_02103_00001_nis_rateints.fitsPUBLICFalsenan243253919848149878
scienceJWSTCALJWSTNIRISS/IMAGEJWSTCLEAR;GR700XDINFRAREDLAT60-LONG170Calibration; Stray light testjw06658006001_03101_00001_nis209.8708333333333455.98163888888889imageEspinoza, Nestor260730.0428464814860730.04421342593107.368600.02800.0CAL-NIS-307: SOSS background measurements60730.170011526658CAL--POLYGON 209.860538216 55.994333641 209.887785953 55.960457671 209.826782557 55.94525742 209.79945601 55.979100775mast:JWST/product/jw06658006001_03101_00001_nis_trapsfilled.jpgmast:JWST/product/jw06658006001_03101_00001_nis_rateints.fitsPUBLICFalsenan243202118848149881
scienceJWSTCALJWSTNIRISS/IMAGEJWSTCLEAR;GR700XDINFRAREDLAT60-LONG170Calibration; Stray light testjw06658006001_05101_00001_nis209.8708333333333455.98163888888889imageEspinoza, Nestor260730.0484301851860730.04979712963107.368600.02800.0CAL-NIS-307: SOSS background measurements60730.169826346658CAL--POLYGON 209.85378515 56.002818453 209.881044559 55.968943916 209.820030345 55.953740453 209.792692106 55.987582367mast:JWST/product/jw06658006001_05101_00001_nis_trapsfilled.jpgmast:JWST/product/jw06658006001_05101_00001_nis_rateints.fitsPUBLICFalsenan243202122848149898
scienceJWSTCALJWSTNIRISS/IMAGEJWSTCLEAR;GR700XDINFRAREDLAT60-LONG170Calibration; Stray light testjw06658006003_02101_00001_nis209.8708333333333455.98163888888889imageEspinoza, Nestor260730.857780358860730.85914730324107.368600.02800.0CAL-NIS-307: SOSS background measurements60730.882650376658CAL--POLYGON 209.891095521 56.00193274 209.918320257 55.968049636 209.857292187 55.952865364 209.829988552 55.98671587mast:JWST/product/jw06658006003_02101_00001_nis_trapsfilled.jpgmast:JWST/product/jw06658006003_02101_00001_nis_rateints.fitsPUBLICFalsenan243253920848149904
scienceJWSTCALJWSTNIRISS/IMAGEJWSTCLEAR;GR700XDINFRAREDECLIPTIC-RA20Calibration; External flat fieldjw06658004001_02103_00001_nis27.65213333333333511.538361111111112imageEspinoza, Nestor260528.10223651620660528.10360346065107.368600.02800.0CAL-NIS-307: SOSS background measurements60528.376712876658CAL--POLYGON 27.643848626 11.560123528 27.657095897 11.525316236 27.621290101 11.512407714 27.608017119 11.547195632mast:JWST/product/jw06658004001_02103_00001_nis_trapsfilled.jpgmast:JWST/product/jw06658004001_02103_00001_nis_rateints.fitsPUBLICFalsenan226939326848176371
scienceJWSTCALJWSTNIRISS/IMAGEJWSTCLEAR;GR700XDINFRAREDECLIPTIC-RA20Calibration; External flat fieldjw06658004001_02101_00001_nis27.65213333333333511.538361111111112imageEspinoza, Nestor260528.0969979976960528.09836494213107.368600.02800.0CAL-NIS-307: SOSS background measurements60528.380659666658CAL--POLYGON 27.647121704 11.551407475 27.660368281 11.51660008 27.624563489 11.503691838 27.611291205 11.53847986mast:JWST/product/jw06658004001_02101_00001_nis_trapsfilled.jpgmast:JWST/product/jw06658004001_02101_00001_nis_rateints.fitsPUBLICFalsenan226939320848176390
scienceJWSTCALJWSTNIRISS/IMAGEJWSTCLEAR;GR700XDINFRAREDLAT-30-LONG320Calibration; External flat fieldjw06658009001_02103_00001_nis334.82304166666665-42.85699999999997imageEspinoza, Nestor260596.6619346064860596.66330155093107.368600.02800.0CAL-NIS-307: SOSS background measurements60597.017546286658CAL--POLYGON 334.841740819 -42.875752608 334.811840907 -42.845758267 334.85313257 -42.82383362 334.883063495 -42.853794315mast:JWST/product/jw06658009001_02103_00001_nis_trapsfilled.jpgmast:JWST/product/jw06658009001_02103_00001_nis_rateints.fitsPUBLICFalsenan230534643848176540
scienceJWSTCALJWSTNIRISS/IMAGEJWSTCLEAR;GR700XDINFRAREDLAT-30-LONG320Calibration; External flat fieldjw06658009001_02101_00001_nis334.82304166666665-42.85699999999997imageEspinoza, Nestor260596.6566960995460596.65806304398107.368600.02800.0CAL-NIS-307: SOSS background measurements60597.018229116658CAL--POLYGON 334.834306808 -42.868232112 334.804413474 -42.838236197 334.845702259 -42.816313724 334.875626605 -42.846275997mast:JWST/product/jw06658009001_02101_00001_nis_trapsfilled.jpgmast:JWST/product/jw06658009001_02101_00001_nis_rateints.fitsPUBLICFalsenan230534644848176556
scienceJWSTCALJWSTNIRISS/IMAGEJWSTCLEAR;GR700XDINFRAREDLAT-30-LONG320Calibration; External flat fieldjw06658009004_02103_00001_nis334.82304166666665-42.85699999999997imageEspinoza, Nestor260596.69544049768660596.69680744213107.368600.02800.0CAL-NIS-307: SOSS background measurements60597.023969896658CAL--POLYGON 334.803292901 -42.884682068 334.773407186 -42.854677812 334.814718334 -42.832766862 334.844635109 -42.862737478mast:JWST/product/jw06658009004_02103_00001_nis_trapsfilled.jpgmast:JWST/product/jw06658009004_02103_00001_nis_rateints.fitsPUBLICFalsenan230535824848176616
...................................................................................................
scienceJWSTCALJWSTNIRISS/IMAGEJWSTCLEAR;GR700XDINFRAREDECLIPTIC-RA80Calibration; External flat fieldjw01541005001_0210d_00001_nis124.1008916666666819.23126111111111imageEspinoza, Nestor259688.6513424189859688.6549462037300.63600.02800.0NIRISS Sensitivity and Stability for Transiting Exoplanet Observations59774.85416661541COM--POLYGON 124.077907696 19.214207662 124.086750759 19.250405679 124.125293793 19.241837815 124.116472001 19.205644639mast:JWST/product/jw01541005001_0210d_00001_nis_trapsfilled.jpgmast:JWST/product/jw01541005001_0210d_00001_nis_rateints.fitsPUBLICFalsenan79661804849743240
scienceJWSTCALJWSTNIRISS/SOSSJWSTCLEAR;GR700XDINFRAREDHAT-P-14Star; Exoplanet Systems; Exoplanets; F dwarfs; F starsjw01541-o001_t002_niriss_clear-gr700xd-substrip256260.1161771630843338.242156511096475timeseriesEspinoza, Nestor359738.2612149305659738.55248649305618855.408600.02800.0NIRISS Sensitivity and Stability for Transiting Exoplanet Observations59774.85416661541COM--POLYGON 260.16190084 38.23723653 260.11337623 38.23744814 260.11348554 38.24618148 260.16201952 38.24596704 260.16190084 38.23723653--mast:JWST/product/jw01541-o001_t002_niriss_clear-gr700xd-substrip256_x1dints.fitsPUBLICFalsenan213209384850390145
scienceJWSTCALJWSTNIRISS/SOSSJWSTCLEAR;GR700XDINFRAREDHAT-P-14Star; Exoplanet Systems; Exoplanets; F dwarfs; F starsjw01541001001_04102_00001-seg001_nis260.116177163368638.24215651042749timeseriesEspinoza, Nestor259738.553906296359738.55835982639329.64600.02800.0NIRISS Sensitivity and Stability for Transiting Exoplanet Observations59774.85416661541COM--POLYGON 260.071030425 38.2494554 260.119367244 38.246686364 260.11850644 38.23799955 260.070173176 38.240758866mast:JWST/product/jw01541001001_04102_00001-seg001_nis_ramp.jpgmast:JWST/product/jw01541001001_04102_00001-seg001_nis_rateints.fitsPUBLICFalsenan85700205850390362
scienceJWSTCALJWSTNIRISS/SOSSJWSTCLEAR;GR700XDINFRAREDCD-38-2551Star; G starsjw02113005001_04103_00001-seg001_nis94.33630544130959-38.323444815098135timeseriesEspinoza, Nestor260216.2528248379660216.257597488424329.64600.02800.0Exploring the morning and evening limbs of a transiting exoplanet60582.576261422113GO--POLYGON 94.343932683 -38.287734536 94.342195697 -38.325771711 94.331098247 -38.325419041 94.332829036 -38.287380656mast:JWST/product/jw02113005001_04103_00001-seg001_nis_ramp.jpgmast:JWST/product/jw02113005001_04103_00001-seg001_nis_rateints.fitsPUBLICFalsenan178895888851081576
scienceJWSTCALJWSTNIRISS/SOSSJWSTCLEAR;GR700XDINFRAREDCD-38-2551Star; G starsjw02113005001_04101_00001-seg001_nis94.33630544945105-38.3234448051187timeseriesEspinoza, Nestor260215.7743559722260215.774483148155.494600.02800.0Exploring the morning and evening limbs of a transiting exoplanet60582.584108712113GO--POLYGON 94.38208655 -38.32834702 94.33350101 -38.32815311 94.33361111 -38.31941977 94.38219434 -38.31961642 94.38208655 -38.32834702mast:JWST/product/jw02113005001_04101_00001-seg001_nis_ramp.jpgmast:JWST/product/jw02113005001_04101_00001-seg001_nis_x1dints.fitsPUBLICFalsenan178895938851081661
scienceJWSTCALJWSTNIRISS/SOSSJWSTCLEAR;GR700XDINFRAREDCD-38-2551Star; G starsjw02113-o005_t001_niriss_clear-gr700xd-substrip25694.33630544541155-38.323444810070136timeseriesEspinoza, Nestor360215.7568523958360216.2504900925932810.168600.02800.0Exploring the morning and evening limbs of a transiting exoplanet60582.963425882113GO--POLYGON 94.38208655 -38.32834702 94.33350101 -38.32815311 94.3336111 -38.31941978 94.38219433 -38.31961643 94.38208655 -38.32834702--mast:JWST/product/jw02113-o005_t001_niriss_clear-gr700xd-substrip256_x1dints.fitsPUBLICFalsenan188165781851098040
scienceJWSTCALJWSTNIRISS/SOSSJWSTCLEAR;GR700XDINFRAREDCD-38-2551Star; G starsjw02113003001_02101_00001-seg001_nis94.33630545011093-38.32344480430984timeseriesEspinoza, Nestor260215.7323431076460215.738556516204429.471600.02800.0Exploring the morning and evening limbs of a transiting exoplanet60581.985208212113GO--POLYGON 94.358674113 -38.288655679 94.35699272 -38.325780744 94.309378304 -38.324278409 94.3110501 -38.287162213mast:JWST/product/jw02113003001_02101_00001-seg001_nis_ramp.jpgmast:JWST/product/jw02113003001_02101_00001-seg001_nis_x1dints.fitsPUBLICFalsenan178895241988181339
scienceJWSTCALJWSTNIRISS/SOSSJWSTCLEAR;GR700XDINFRAREDCD-38-2551Star; G starsjw02113002001_02101_00001-seg001_nis94.33630545031605-38.323444804058404timeseriesEspinoza, Nestor260215.7202297858860215.72644319444429.471600.02800.0Exploring the morning and evening limbs of a transiting exoplanet60581.978854042113GO--POLYGON 94.372810846 -38.289090732 94.37114681 -38.326216279 94.323531402 -38.324727658 94.325185848 -38.287610984mast:JWST/product/jw02113002001_02101_00001-seg001_nis_ramp.jpgmast:JWST/product/jw02113002001_02101_00001-seg001_nis_x1dints.fitsPUBLICFalsenan178895222988181546
scienceJWSTCALJWSTNIRISS/SOSSJWSTCLEAR;GR700XDINFRAREDCD-38-2551Star; G starsjw02113004001_02101_00001-seg001_nis94.33630544990864-38.3234448045578timeseriesEspinoza, Nestor260215.7442897627360215.75050317129429.471600.02800.0Exploring the morning and evening limbs of a transiting exoplanet60581.986215162113GO--POLYGON 94.351610404 -38.28843683 94.349915827 -38.325561526 94.302302094 -38.324048775 94.30398707 -38.286932947mast:JWST/product/jw02113004001_02101_00001-seg001_nis_ramp.jpgmast:JWST/product/jw02113004001_02101_00001-seg001_nis_x1dints.fitsPUBLICFalsenan178895849988182195
scienceJWSTCALJWSTNIRISS/SOSSJWSTCLEAR;GR700XDINFRAREDCD-38-2551Star; G starsjw02113001001_02101_00001-seg001_nis94.33630545052999-38.32344480379617timeseriesEspinoza, Nestor260215.7075957233860215.71380913194429.471600.02800.0Exploring the morning and evening limbs of a transiting exoplanet60581.931585542113GO--POLYGON 94.379910812 -38.289302525 94.378273855 -38.326428813 94.330657223 -38.324961582 94.332284596 -38.287844171mast:JWST/product/jw02113001001_02101_00001-seg001_nis_ramp.jpgmast:JWST/product/jw02113001001_02101_00001-seg001_nis_x1dints.fitsPUBLICFalsenan178895197988192511

Our query returned many NIRISS observations led by the PI Dr. Espinoza. Let’s get all the unique values under the instrument_name column to see what our * wildcard picked up.

set(observations['instrument_name'])
{np.str_('NIRISS/IMAGE'), np.str_('NIRISS/SOSS')}

Our observations have the advanced labeling; had we simply set instrument_name = "NIRISS", we would have missed several observations. For more details on this advanced labeling, see the JWST Instrument Names page.

A note of caution: There is such a thing as too many wildcards#

You can be too generous with the wildcards, so be sure to exercise caution in their use. Too much ambiguity can lead to unintended results. Let’s take a look at our example below.

observations = Observations.query_criteria(proposal_pi='Espinoza, Nestor',
                                           instrument_name='NIR*') # Surely only one instrument begins with 'NIR'
set(observations['instrument_name'])
{np.str_('NIRISS/IMAGE'), np.str_('NIRISS/SOSS'), np.str_('NIRSPEC/SLIT')}

This query returns NIRSPEC/SLIT observations in addition to the NIRISS ones, which is not what we intended.

Case 2: Wildcard Search with instrument_name and proposal_id#

Let’s add an additional string criterion and wildcard into the mix. We’ll do this with the proposal_id field which, despite its numeric content, is encoded as a string.

Let’s query for a four digit proposal/program IDs that begin with 15.

observations = Observations.query_criteria(proposal_pi='Espinoza, Nestor',
                                           instrument_name='NIRISS*',
                                           proposal_id=['15%%']) # Only a four digit result will match this

set(observations['proposal_id']), set(observations['instrument_name'])
({np.str_('1512'), np.str_('1541')},
 {np.str_('NIRISS/IMAGE'), np.str_('NIRISS/SOSS')})

Resources#

The following is a list of resources that were referenced throughout the tutorial, as well as some additional references that you may find useful:

Citations#

If you use any of astroquery’s tools for published research, please cite the authors. Follow this link for more information about citing astroquery:

About This Notebook#

If you have comments or questions on this notebook, please contact us through the Archive Help Desk e-mail at archive@stsci.edu.

Author: Jenny V. Medina
Keywords: astroquery, wildcards, moving target
Last Updated: Jun 2023