JWST Duplication Checking

JWST Duplication Checking#

Introduction#

As with HST, JWST observations that duplicate existing, planned, or approved observations will be allowed only if investigators provide a scientific justification in their proposal and that request is approved. Consult the JWST Duplicate Observations Policy for details. Broadly speaking, observations you are considering might duplicate observations in the current or prior cycles if your target is of the same astrophysical source, or there is significant spatial overlap of fields, and the following apply:

  • Similar imaging passband, or overlapping spectral range

  • Similar (spectral) resolution

  • Similar exposure depth

Observations with a different scientific instrument than one you are considering might still duplicate if the observing configuration and capabilities are similar (e.g., NIRCam and NIRISS imaging, or NIRCam and NIRSpec spectroscopy).

This notebook illustrates how to use the python package astroquery.mast to search the Mikulski Archive for Space Telescopes (MAST) for potential duplicate observations. It is also possible to perform these searches with the MAST Portal, but often less efficiently.

Special Disclaimer#

The complete footprint of approved (but not executed) mosaicked or parallel observations is only approximate. That is, while the primary location is reported for an observation, the exact orientation. of the aperture is not known until the observation is scheduled for execution. Moreover metadata in MAST about planned/approved observations may not suffice to determine whether your intended observation is a genuine duplication, particularly for slit or MOS spectroscopy. You are responsible for evaluating the details of the planned observations by using the accepted program's APT file (and/or the Aladin display in APT, as appropriate) to determine if the potential duplications are genuine.

Table of contents#

Strategy#

The following strategy, applied to each target or field, will identify potentially duplicative observations.

  1. Search for common targets or fields. If no existing or planned observation coincides with your intended target, you are done.

  2. If there is a spatial overlap, determine if the instrument(s) and observing configurations you plan to use are the same or similar to the existing observations. If there is no commonality, you are done.

  3. If there is spatial overlap and common instruments/configuration, determine in detail the overlap in passband/spectral coverage, and exposure depth.

  4. If there is a likely duplication for an intended target, do one of the following:

    • Include in your proposal a scientific case for the duplicating observation(s)

    • Alter your intended observation(s) in a way that does not duplicate

    • Choose a different target

Setup#

Begin by importing some essential python packages: general utilities in astropy, and query services in astroquery.mast.

# Give the notebook cells more of the available width
from IPython.display import display, HTML
display(HTML("<style>.container { width:99% !important; }</style>"))

from astropy.table import Table, unique
from astropy.coordinates import Angle
from astroquery.mast import Mast
from astroquery.mast import Observations

Utility Routines#

The following utility routine will create URLs to the parent programs of matching observations.

APT_LINK = '(http://www.stsci.edu/cgi-bin/get-proposal-info?id={}&observatory=JWST)'


def get_program_URL(program_id):
    '''
    Generate the URL for program status information, given a program ID. 
    '''
    return APT_LINK.format(program_id)

Queries by Target/Postion#

All of the queries below search for JWST observations, using a search radius larger than fields of view (FoV) of JWST apertures, to allow for the possibility that the FoV may be rotated when approved-but-unexecuted observations are actually scheduled. Your queries will typically be more efficient if you restrict your search to JWST data by including the parameter: obs_collection = "JWST".

Searching for source names is not recommended when checking for potential duplications with fixed targets because the matches are not reliable, consistent, or complete. The exception would be for matching Object names of well known solar system bodies.

The best approach is to use an independent source for the coordinates of your desired target, including existing JWST or HST images if any exist. Failing that, the Mast.resolve_object() method will return coordinates in the ICRS reference frame, which you can use to verify whether a target name resolves to the coordinates you intend. See the Appendix for details.

Search by Target Name#

This example shows how to query for a single target with a standard name, HD 104237 which is a T-Tauri star. We first resolve the name and verify that the coordinates are correct, using the Mast class and the resolve_object() method.

# Resolve the coordinates if necessary
coords = Mast.resolve_object('HD 104237')
print(coords)

<SkyCoord (ICRS): (ra, dec) in deg
    (180.02119525, -78.19293492)>

In this case the resolved coordinates agree with those in Simbad, so it is safe to execute a simple cone search (a search within a specified radius of a place on the sky) to find JWST existing or planned observations. The astroquery.mast package provides the method Observations.query_criteria() to specify the parameters for the search; provide them as key=value pairs. The full set of query parameters for this method may be found on CAOM Field Descriptions.

# If the coordinates are ok
obs = Observations.query_criteria(
        obs_collection='JWST',
        objectname='HD 104237',
        radius='30s'
        )
print('Number of matching observations: {}'.format(len(obs)))

Number of matching observations: 0

WARNING: NoResultsWarning: Query returned no results. [astroquery.mast.discovery_portal]
If the number of matches is zero, there are no potential JWST duplications with this target.

Single Moving Target#

Moving targets, by definition, do not lend themselves to searches by position. This kind of search in MAST is limited to a modest set of solar system bodies with recognized names. Note the use of a wildcard character (*) to trigger a match, even when the target name includes other text.

obs = Observations.query_criteria(
        obs_collection="JWST",
        target_name="Io*")
    
print(f'Number of matching observations: {len(obs)}')

Number of matching observations: 39

There are several JWST observations of Io, including some that are planned but have not executed as of the beginning of Cy-2 (i.e., calib_level = -1). The details are in the returned table of results, the most essential of which can be viewed below.

out_cols = ['target_name', 'instrument_name', 'filters', 'calib_level', 't_exptime', 'proposal_id']
obs[out_cols]
Table length=39
target_nameinstrument_namefilterscalib_levelt_exptimeproposal_id
str2str11str12int64float64str4
IOMIRI/IFUCH232109.0324565
IOMIRI/IFUCH132109.0324565
IOMIRI/IFUCH432109.0324565
IOMIRI/IFUCH332109.0324565
IOMIRI/IFUCH332109.0324565
IOMIRI/IFUCH232109.0324565
IOMIRI/IFUCH132109.0324565
IOMIRI/IFUCH432109.0324565
IOMIRI/IFUCH34-LONG23194.0714565
..................
IOMIRI/IFUCH432109.0324078
IOMIRI/IFUCH132109.0324078
IOMIRI/IFUCH232109.0324078
IOMIRI/IFUCH332109.0324078
IOMIRI/IFUCH12-LONG23194.0714078
IOMIRI/IFUCH12-LONG23194.0714078
IOMIRI/IFUCH12-LONG23194.0714078
IOMIRI/IFUCH34-LONG23194.0714078
IOMIRI/IFUCH34-LONG23194.0714078
IOMIRI/IFUCH34-LONG23194.0714078

You may need to examine the specifics of the programs that obtained the observations to know whether your intended observations would duplicate. Here extend the results table with the unique Program Titles and URLs for the program status pages, which may offer clues:

obs['title'] = [x[:80] for x in obs['obs_title']]
unique(obs['proposal_id', 'title']).pprint(max_width=-1)
obs['proposal_URL'] = [get_program_URL(x) for x in obs['proposal_id']]
unique(obs['proposal_id', 'proposal_URL']).pprint(max_width=-1)

proposal_id                                      title                                      
----------- --------------------------------------------------------------------------------
       1373 ERS observations of the Jovian System as a demonstration of JWST capabilities fo
       4078 Mass-loss from Ios volcanic atmosphere: A unique synergy with the Juno Io fly-by
       4565              HST-Juno Io Campaign: Connecting Volcanos to the Plasma Environment
proposal_id                                proposal_URL                              
----------- -------------------------------------------------------------------------
       1373 (http://www.stsci.edu/cgi-bin/get-proposal-info?id=1373&observatory=JWST)
       4078 (http://www.stsci.edu/cgi-bin/get-proposal-info?id=4078&observatory=JWST)
       4565 (http://www.stsci.edu/cgi-bin/get-proposal-info?id=4565&observatory=JWST)
There are images, spectra, coronagraphic, and IFU observations of this target that could conflict with your intended proposal. Proposed images in passbands that overlap the IFU data would have to be examined closely for duplication.

Checking a List of Targets#

It is often useful to search for individual targets with the MAST Portal because the results are easily visualized. But it is more efficient to search over a large number of targets using astroquery.mast.

Loading Targets from CSV#

For efficiency, make a CSV (comma-separated variable) list of your targets, one per row. The list can alternatively be read from a local file: just substitute the file name for targ_table in the first argument of the Table.read() method. The list can contain many fields, but at a minimum must contain the target name and the equitorial coordinates.

For stellar targets, it is best to use a large enough radius to make your search robust against position uncertainties and proper motion between J2000 and the current epoch.

The first row of the file will be interpreted as a column name in the table. This is important.

targ_table = '''
target_name, RA, DEC, Radius, Description
Trappist-1, 23:06:29.3684948589, -05:02:29.037301866, 30s, Exoplanet host star
V* XX Cha, 11:11:39.559, -76:20:15.04, 30s, Variable star with debris disk
M 31, 00:42:44.330, +41:16:07.50, 12.0m, Andromeda Galaxy
M 57, 18:53:35.0967659112, +33:01:44.883287544, 2m, Planetary nebula
'''
targets = Table.read(targ_table, format='ascii.csv')

Next, make new columns to hold the coordinates and ranges in units of degrees.

targets['ra_deg'] = [Angle(x+' hours').degree for x in targets['RA']]
targets['dec_deg'] = [Angle(x+' degree').degree for x in targets['DEC']]
targets['radius_deg'] = [Angle(x).degree for x in targets['Radius']]
targets['N_obs'] = 0  # field to hold the count of matched observations 
targets
Table length=4
target_nameRADECRadiusDescriptionra_degdec_degradius_degN_obs
str10str19str19str5str30float64float64float64int64
Trappist-123:06:29.3684948589-05:02:29.03730186630sExoplanet host star346.62236872857875-5.0413992505183330.0083333333333333330
V* XX Cha11:11:39.559-76:20:15.0430sVariable star with debris disk167.91482916666666-76.337511111111110.0083333333333333330
M 3100:42:44.330+41:16:07.5012.0mAndromeda Galaxy10.68470833333333141.268750.20
M 5718:53:35.0967659112+33:01:44.8832875442mPlanetary nebula283.3962365246299633.0291342465399960.033333333333333330

It can save some effort to determine the number of JWST Observations that match each of your targets using the query_criteria_count() method. Note that the coordinate parameters must be specified as lower and upper bounds of a range (i.e., a python list), so use the coord_ranges() utility function.

It is good practice to first check the number of matching observations before fetching the results themselves, in case the number of results is very large. This is more important when querying MAST missions with a very large number of observations, such as HST. This type of search is typically fairly fast. 
# get the counts of each target
for t in targets: 
    t['N_obs'] = Observations.query_criteria_count(
        obs_collection='JWST',
        coordinates=f"{t['ra_deg']} {t['dec_deg']}",
        radius=t['radius_deg']
        )
targets['target_name', 'N_obs']

WARNING: InputWarning: Coordinate string is being interpreted as an ICRS coordinate provided in degrees. [astroquery.utils.commons]
Table length=4
target_nameN_obs
str10int64
Trappist-1335
V* XX Cha14
M 310
M 57467
For targets where no JWST Observations exist, there is no potential for duplication.

Evaluating Potential Duplications#

To examine the matching Observations for the other targets in more detail, use the query_criteria() method. Also, use a cone search (i.e., specify a coordinate pair and a radius), rather than an area search over a region in RA and Dec; see the Appendix for details. Note that a new column called “obs” is first added to the table to contain the results of the Observation search.

targets['obs'] = None
for t in targets: 
    t['obs'] = Observations.query_criteria(
        obs_collection='JWST',
        coordinates=f"{t['ra_deg']} {t['dec_deg']}",
        radius=t['radius_deg']
        )

WARNING: NoResultsWarning: Query returned no results. [astroquery.mast.discovery_portal]

Targets with matching JWST Observations are examined in the following sub-sections.

Trappist-1#

Trappist-1 is a well known exo-planet host star. If the intended observations are timeseries spectroscopy, the search would naturally be limited to a small area of sky. There are a several unique JWST observations of this target, so it is important to examine the relevant details.

out_cols = ['target_name', 'instrument_name', 'filters', 't_exptime', 'proposal_id']
obs = targets[0]['obs']
unique(obs[out_cols])
Table length=28
target_nameinstrument_namefilterst_exptimeproposal_id
str11str12str13float64str4
TRAPPIST-1MIRI/IMAGEF1280W8727.7135191
TRAPPIST-1MIRI/IMAGEF1280W11092.7235191
TRAPPIST-1MIRI/IMAGEF1280W13823.7475191
TRAPPIST-1MIRI/IMAGEF1280W15625.6595191
TRAPPIST-1MIRI/IMAGEF1500W11535.8412304
TRAPPIST-1MIRI/IMAGEF1500W11574.6922304
TRAPPIST-1MIRI/IMAGEF1500W74151.93077
TRAPPIST-1MIRI/IMAGEF1500W138234.5373077
TRAPPIST-1NIRISS/SOSSCLEAR;GR700XD988.922589
...............
TRAPPIST-1NIRSPEC/SLITCLEAR;PRISM13851.8482420
TRAPPIST-1NIRSPEC/SLITCLEAR;PRISM15039.646456
TRAPPIST-1NIRSPEC/SLITCLEAR;PRISM15085.3242589
TRAPPIST-1NIRSPEC/SLITCLEAR;PRISM19042.6726456
TRAPPIST-1NIRSPEC/SLITCLEAR;PRISM20897.1846456
TRAPPIST-1NIRSPEC/SLITCLEAR;PRISM24956.7566456
TRAPPIST-1NIRSPEC/SLITCLEAR;PRISM26426.7966456
TRAPPIST-1BMIRI/IMAGEF1280W13983.4271279
TRAPPIST-1BMIRI/IMAGEF1500W15690.0761177
UNKNOWNNIRSPEC/SLITOPAQUE;MIRROR0.032742

You will need to examine the specifics of the programs that obtained the observations to know for certain which of the 7 known exoplanets was being observed, and therefore whether your intended observations would duplicate. Here are the unique Program Titles, which may offer clues:

obs['title'] = [x[:80] for x in obs['obs_title']]
unique(obs['proposal_id', 'title'])
Table length=12
proposal_idtitle
str4str80
1177MIRI observations of transiting exoplanets
1201NIRISS Exploration of the Atmospheric diversity of Transiting exoplanets (NEAT)
1279Thermal emission from Trappist-1 b
1331Transit Spectroscopy of TRAPPIST-1e
1981Tell Me How Im Supposed To Breathe With No Air: Measuring the Prevalence and Div
2304Hot Take on a Cool World: Does Trappist-1c Have an Atmosphere?
2420Probing the Terrestrial Planet TRAPPIST-1c for the Presence of an Atmosphere
2589Atmospheric reconnaissance of the TRAPPIST-1 planets
2742NIRSpec darks for detector reconfiguration
3077TRAPPIST-1 Planets: Atmospheres Or Not?
5191Bare rocks are not supposed to do that
6456Using stellar contamination proxy TRAPPIST-1 b to search for an atmosphere on TR

The Program Status Pages offer the complete specifications for each existing program:

obs['proposal_URL'] = [get_program_URL(x) for x in obs['proposal_id']]
unique(obs['proposal_id', 'proposal_URL'])
Table length=12
proposal_idproposal_URL
str4str73
1177(http://www.stsci.edu/cgi-bin/get-proposal-info?id=1177&observatory=JWST)
1201(http://www.stsci.edu/cgi-bin/get-proposal-info?id=1201&observatory=JWST)
1279(http://www.stsci.edu/cgi-bin/get-proposal-info?id=1279&observatory=JWST)
1331(http://www.stsci.edu/cgi-bin/get-proposal-info?id=1331&observatory=JWST)
1981(http://www.stsci.edu/cgi-bin/get-proposal-info?id=1981&observatory=JWST)
2304(http://www.stsci.edu/cgi-bin/get-proposal-info?id=2304&observatory=JWST)
2420(http://www.stsci.edu/cgi-bin/get-proposal-info?id=2420&observatory=JWST)
2589(http://www.stsci.edu/cgi-bin/get-proposal-info?id=2589&observatory=JWST)
2742(http://www.stsci.edu/cgi-bin/get-proposal-info?id=2742&observatory=JWST)
3077(http://www.stsci.edu/cgi-bin/get-proposal-info?id=3077&observatory=JWST)
5191(http://www.stsci.edu/cgi-bin/get-proposal-info?id=5191&observatory=JWST)
6456(http://www.stsci.edu/cgi-bin/get-proposal-info?id=6456&observatory=JWST)
There are MIRI imaging observations (likely Time-series) in multiple bands, and both NIRISS and NIRSpec spectroscopic observations of this target that could conflict with your intended proposal.

V* XX Cha#

This pre-main-sequence star has a debris disk, and might be worth a coronagraphic observation. The search area is small (20 arcsec radius) to exclude nearby targets.

out_cols = ['target_name', 'instrument_name', 'filters', 't_exptime', 'calib_level', 'proposal_id']
obs = targets[1]['obs']
unique(obs[out_cols])
Table length=7
target_nameinstrument_namefilterst_exptimecalib_levelproposal_id
str10str11str12float64int64str4
V-XX-CHA-2MIRI/IFUCH13696.34831282
V-XX-CHA-2MIRI/IFUCH23696.34831282
V-XX-CHA-2MIRI/IFUCH33696.34831282
V-XX-CHA-2MIRI/IFUCH43696.34831282
V-XX-CHA-2MIRI/IMAGEF1280W3696.34831282
V-XX-CHA-2NIRSPEC/IFUF290LP;G395H32.2121282
V-XX-CHA-2NIRSPEC/IFUF290LP;G395H1288.41631282
While JWST spectra from MIRI and NIRSpec exist, there is no potential for duplication with a coronagraphic observation.

M 57#

The Ring Nebula (NGC 6720) is well observed, particularly with HST and, recently, with JWST. We searched a 4-arcmin region around the source to include potential spectral Observations in the nebular periphery. The following table shows the unique combinations of instrument configurations.

out_cols = ['target_name', 'instrument_name', 'filters', 't_exptime', 'proposal_id']
obs = targets[3]['obs']
unique(obs[out_cols])
Table length=69
target_nameinstrument_namefilterst_exptimeproposal_id
str26str12str12float64str4
NGC6720MIRI/IMAGEF1000W444.00799999999991558
NGC6720MIRI/IMAGEF1130W444.00799999999991558
NGC6720MIRI/IMAGEF1280W444.00799999999991558
NGC6720MIRI/IMAGEF1500W444.00799999999991558
NGC6720MIRI/IMAGEF1800W444.00799999999991558
NGC6720MIRI/IMAGEF2100W444.00799999999991558
NGC6720MIRI/IMAGEF2550W444.00799999999991558
NGC6720MIRI/IMAGEF560W444.00799999999991558
NGC6720MIRI/IMAGEF770W444.00799999999991558
...............
NGC6720-NIRSPEC-POSITION-2NIRSPEC/IFUF290LP;G395H102.1226640
NGC6720-NIRSPEC-POSITION-2NIRSPEC/IFUF290LP;G395H437.6676640
NGC6720-NIRSPEC-POSITION-2NIRSPEC/IFUF290LP;G395H7878.0066640
NGC6720-NIRSPEC-POSITION-2NIRSPEC/IFUF290LP;G395M145.8891558
NGC6720-NIRSPEC-POSITION-2NIRSPEC/IFUF290LP;G395M145.8896640
NGC6720-NIRSPEC-POSITION-2NIRSPEC/IFUF290LP;G395M583.5561558
NGC6720-NIRSPEC-POSITION-2NIRSPEC/IFUF290LP;G395M1313.00100000000026640
NGC6720-NIRSPEC-POSITION-2NIRSPEC/IFUOPAQUE;G140M58.3566640
NGC6720-NIRSPEC-POSITION-2NIRSPEC/IFUOPAQUE;G235M58.3566640
NGC6720-NIRSPEC-POSITION-2NIRSPEC/IFUOPAQUE;G395M58.3566640
There are both MIRI and NIRCam images in many passbands, and MIRI and NIRSpec spectroscopic IFU Observations of this target that could conflict with the intended proposal.

To retain this target it would be necessary to obtain spectra in a different location, or to use a different instrument/configuration/data-taking mode (e.g., time-series of the central star). It may be helpful to see the footprints of these Observations in the MAST Portal, or to view the observation specifications in Program GO-1558.

Appendix: Caveats#

Source Name#

It is possible to search MAST for individual sources by name, in one of two ways: Object Name or by Target Name.

  • Object Name: MAST invokes an astrophysical name resolver (e.g., NED or Simbad) to look up source coordinates. Not all object names are recognized by resolvers. Also, different resolvers sometimes return substantially different coordinates; MAST will pick one of them (but not tell you which one). Sometimes the coordinates returned by the resolvers differ by substantial amounts, often because they refer to different sources.

    • A classic case is the bright star μ Eri from the Bayer Greek letter system, and the star MU Eri in the general catalog of variable stars. in MAST,

      • Searching for * mu. eri (or just plain mu eri matches μ Eri (RA = 4:45:30.15, Dec = -3:15:16.777)

      • Searching for V* mu eri matches MU Eri (RA = 2:48:10.566, Dec = -15:18:04.03)

  • Target Name: This search matches to target names specified by Investigators in observing proposals to refer to sources. These are not guaranteed to match standard names for astrophysical sources, and often do not.

Searching by source names is not recommended when checking for potential duplications with fixed targets because the matches are not reliable, consistent, or complete. The exception would be for matching Object names of well known solar system bodies.

The best approach is to use an independent source for the coordinates of your desired target. Failing that, the Mast.resolve_object() method will return coordinates in the ICRS reference frame, which you can use to verify whether a target name resolves to the coordinates you intend.

Additional Resources#

About this notebook#

This notebook was developed by Archive Sciences Branch staff. For support, please direct questions to the Archive HelpDesk at archive@stsci.edu, or through the JWST HelpDesk Portal. Space Telescope Logo

Last update: 2024 Jan

Contents