JWST Duplication Checking#

Introduction#

As with HST, JWST observations that duplicate existing, planned, or approved observations will be allowed only if investigators provide a scientific justification in their proposal and that request is approved. Consult the JWST Duplicate Observations Policy for details. Broadly speaking, observations you are considering might duplicate observations in the current or prior cycles if your target is of the same astrophysical source, or there is significant spatial overlap of fields, and the following apply:

  • Similar imaging passband, or overlapping spectral range

  • Similar (spectral) resolution

  • Similar exposure depth

Observations with a different scientific instrument than one you are considering might still duplicate if the observing configuration and capabilities are similar (e.g., NIRCam and NIRISS imaging, or NIRCam and NIRSpec spectroscopy).

This notebook illustrates how to use the python package astroquery.mast to search the Mikulski Archive for Space Telescopes (MAST) for potential duplicate observations. It is also possible to perform these searches with the MAST Portal, but often less efficiently.

Special Disclaimer#

The complete footprint of approved (but not executed) mosaicked or parallel observations is only approximate. That is, while the primary location is reported for an observation, the exact orientation. of the aperture is not known until the observation is scheduled for execution. Moreover metadata in MAST about planned/approved observations may not suffice to determine whether your intended observation is a genuine duplication, particularly for slit or MOS spectroscopy. You are responsible for evaluating the details of the planned observations by using the accepted program's APT file (and/or the Aladin display in APT, as appropriate) to determine if the potential duplications are genuine.

Table of contents#

Strategy#

The following strategy, applied to each target or field, will identify potentially duplicative observations.

  1. Search for common targets or fields. If no existing or planned observation coincides with your intended target, you are done.

  2. If there is a spatial overlap, determine if the instrument(s) and observing configurations you plan to use are the same or similar to the existing observations. If there is no commonality, you are done.

  3. If there is spatial overlap and common instruments/configuration, determine in detail the overlap in passband/spectral coverage, and exposure depth.

  4. If there is a likely duplication for an intended target, do one of the following:

    • Include in your proposal a scientific case for the duplicating observation(s)

    • Alter your intended observation(s) in a way that does not duplicate

    • Choose a different target

Setup#

Begin by importing some essential python packages: general utilities in astropy, and query services in astroquery.mast.

# Give the notebook cells more of the available width
from IPython.display import display, HTML
display(HTML("<style>.container { width:99% !important; }</style>"))
import astropy
from astropy import table
from astropy.table import Table, unique
from astropy import units as u
from astropy.coordinates import Angle
from astroquery.mast import Mast
from astroquery.mast import Observations

Utility Routines#

The following utility routine will create URLs to the parent programs of matching observations.

APT_LINK = '(http://www.stsci.edu/cgi-bin/get-proposal-info?id={}&observatory=JWST)'

def get_program_URL(program_id):
    '''
    Generate the URL for program status information, given a program ID. 
    '''
    return APT_LINK.format(program_id)

Queries by Target/Postion#

All of the queries below search for JWST observations, using a search radius larger than fields of view (FoV) of JWST apertures, to allow for the possibility that the FoV may be rotated when approved-but-unexecuted observations are actually scheduled. Your queries will typically be more efficient if you restrict your search to JWST data by including the parameter: obs_collection = "JWST".

Searching for source names is not recommended when checking for potential duplications with fixed targets because the matches are not reliable, consistent, or complete. The exception would be for matching Object names of well known solar system bodies.

The best approach is to use an independent source for the coordinates of your desired target, including existing JWST or HST images if any exist. Failing that, the Mast.resolve_object() method will return coordinates in the ICRS reference frame, which you can use to verify whether a target name resolves to the coordinates you intend. See the Appendix for details.

Search by Target Name#

This example shows how to query for a single target with a standard name, HD 104237 which is a T-Tauri star. We first resolve the name and verify that the coordinates are correct, using the Mast class and the resolve_object() method.

# Resolve the coordinates if necessary
coords = Mast.resolve_object('HD 104237')
print(coords)
<SkyCoord (ICRS): (ra, dec) in deg
    (180.02119525, -78.19293492)>

In this case the resolved coordinates agree with those in Simbad, so it is safe to execute a simple cone search (a search within a specified radius of a place on the sky) to find JWST existing or planned observations. The astroquery.mast package provides the method Observations.query_criteria() to specify the parameters for the search; provide them as key=value pairs. The full set of query parameters for this method may be found on CAOM Field Descriptions.

# If the coordinates are ok
obs = Observations.query_criteria(
        obs_collection='JWST'
        ,objectname='HD 104237'
        ,radius='30s'
        )
print('Number of matching observations: {}'.format(len(obs)))
Number of matching observations: 0
WARNING: NoResultsWarning: Query returned no results. [astroquery.mast.discovery_portal]
If the number of matches is zero, there are no potential JWST duplications with this target.

Single Moving Target#

Moving targets, by definition, do not lend themselves to searches by position. This kind of search in MAST is limited to a modest set of solar system bodies with recognized names. Note the use of a wildcard character (*) to trigger a match, even when the target name includes other text.

obs = Observations.query_criteria(
        obs_collection="JWST"
        ,target_name="Io*"
        )
    
print(f'Number of matching observations: {len(obs)}')
Number of matching observations: 34

There are several JWST observations of Io, including some that are planned but have not executed as of the beginning of Cy-2 (i.e., calib_level = -1). The details are in the returned table of results, the most essential of which can be viewed below.

out_cols = ['target_name','instrument_name','filters','calib_level','t_exptime','proposal_id']
obs[out_cols].show_in_notebook()
Table length=34
idxtarget_nameinstrument_namefilterscalib_levelt_exptimeproposal_id
0IOMIRI/IFU---12664.0384565
1IOMIRI/IFU---1222.0034565
2IOMIRI/IFU---1222.0034565
3IOMIRI/IFU---1222.0034565
4IOMIRI/IFU---1222.0034565
5IOMIRI/IFU---1222.0034565
6IOMIRI/IFU---1222.0034565
7IOMIRI/IFU---1222.0034565
8IOMIRI/IFU---1222.0034565
9IOMIRI/IFUCH12-LONG23194.0714078
10IOMIRI/IFUCH12-LONG23194.0714078
11IOMIRI/IFUCH12-LONG23194.0714078
12IOMIRI/IFUCH34-LONG23194.0714078
13IOMIRI/IFUCH34-LONG23194.0714078
14IOMIRI/IFUCH34-LONG23194.0714078
15IOMIRI/IFUCH432109.0324078
16IOMIRI/IFUCH332109.0324078
17IOMIRI/IFUCH432109.0324078
18IOMIRI/IFUCH132109.0324078
19IOMIRI/IFUCH232109.0324078
20IOMIRI/IFUCH232109.0324078
21IOMIRI/IFUCH332109.0324078
22IOMIRI/IFUCH132109.0324078
23IOMIRI/IFUCH43790.88700000000011373
24IOMIRI/IFUCH33790.88700000000011373
25IOMIRI/IFUCH23790.88700000000011373
26IOMIRI/IFUCH13790.88700000000011373
27IONIRISS/AMIF430M;NRM31697.41373
28IONIRSPEC/IFUF290LP;G395H3343.5761373
29IONIRSPEC/IFUF170LP;G235H3687.1521373
30IONIRSPEC/IFUF100LP;G140H34668.4481373
31IONIRSPEC/IFUF170LP;G235H3687.1521373
32IONIRSPEC/IFUF290LP;G395H3343.5761373
33IONIRSPEC/IFUF100LP;G140H35135.2881373

You may need to examine the specifics of the programs that obtained the observations to know whether your intended observations would duplicate. Here extend the results table with the unique Program Titles and URLs for the program status pages, which may offer clues:

obs['title'] = [x[:80] for x in obs['obs_title']]
unique(obs['proposal_id','title']).pprint(max_width=-1)
obs['proposal_URL'] = [get_program_URL(x) for x in obs['proposal_id']]
unique(obs['proposal_id','proposal_URL']).pprint(max_width=-1)
proposal_id                                      title                                      
----------- --------------------------------------------------------------------------------
       1373 ERS observations of the Jovian System as a demonstration of JWST capabilities fo
       4078 Mass-loss from Ios volcanic atmosphere: A unique synergy with the Juno Io fly-by
       4565              HST-Juno Io Campaign: Connecting Volcanos to the Plasma Environment
proposal_id                                proposal_URL                              
----------- -------------------------------------------------------------------------
       1373 (http://www.stsci.edu/cgi-bin/get-proposal-info?id=1373&observatory=JWST)
       4078 (http://www.stsci.edu/cgi-bin/get-proposal-info?id=4078&observatory=JWST)
       4565 (http://www.stsci.edu/cgi-bin/get-proposal-info?id=4565&observatory=JWST)
There are images, spectra, coronagraphic, and IFU observations of this target that could conflict with your intended proposal. Proposed images in passbands that overlap the IFU data would have to be examined closely for duplication.

Checking a List of Targets#

It is often useful to search for individual targets with the MAST Portal because the results are easily visualized. But it is more efficient to search over a large number of targets using astroquery.mast.

Loading Targets from CSV#

For efficiency, make a CSV (comma-separated variable) list of your targets, one per row. The list can alternatively be read from a local file: just substitute the file name for targ_table in the first argument of the Table.read() method. The list can contain many fields, but at a minimum must contain the target name and the equitorial coordinates.

For stellar targets, it is best to use a large enough radius to make your search robust against position uncertainties and proper motion between J2000 and the current epoch.

The first row of the file will be interpreted as a column name in the table. This is important.

targ_table = '''
target_name, RA, DEC, Radius, Description
Trappist-1, 23:06:29.3684948589, -05:02:29.037301866, 30s, Exoplanet host star
V* XX Cha, 11:11:39.559, -76:20:15.04, 30s, Variable star with debris disk
M 31, 00:42:44.330, +41:16:07.50, 12.0m, Andromeda Galaxy
M 57, 18:53:35.0967659112, +33:01:44.883287544, 2m, Planetary nebula
'''
targets = Table.read(targ_table, format='ascii.csv')

Next, make new columns to hold the coordinates and ranges in units of degrees.

targets['ra_deg'] = [Angle(x+' hours').degree for x in targets['RA']]
targets['dec_deg'] = [Angle(x+' degree').degree for x in targets['DEC']]
targets['radius_deg'] = [Angle(x).degree for x in targets['Radius']]
targets['N_obs'] = 0  # field to hold the count of matched observations 
targets
Table length=4
target_nameRADECRadiusDescriptionra_degdec_degradius_degN_obs
str10str19str19str5str30float64float64float64int64
Trappist-123:06:29.3684948589-05:02:29.03730186630sExoplanet host star346.62236872857875-5.0413992505183330.0083333333333333330
V* XX Cha11:11:39.559-76:20:15.0430sVariable star with debris disk167.91482916666666-76.337511111111110.0083333333333333330
M 3100:42:44.330+41:16:07.5012.0mAndromeda Galaxy10.68470833333333141.268750.20
M 5718:53:35.0967659112+33:01:44.8832875442mPlanetary nebula283.3962365246299633.0291342465399960.033333333333333330

It can save some effort to determine the number of JWST Observations that match each of your targets using the query_criteria_count() method. Note that the coordinate parameters must be specified as lower and upper bounds of a range (i.e., a python list), so use the coord_ranges() utility function.

It is good practice to first check the number of matching observations before fetching the results themselves, in case the number of results is very large. This is more important when querying MAST missions with a very large number of observations, such as HST. This type of search is typically fairly fast.
# get the counts of each target
for t in targets: 
    t['N_obs'] = Observations.query_criteria_count(
        obs_collection='JWST'
        ,coordinates=f"{t['ra_deg']} {t['dec_deg']}"
        ,radius=t['radius_deg']
        )
targets['target_name','N_obs'].show_in_notebook()
WARNING: InputWarning: Coordinate string is being interpreted as an ICRS coordinate provided in degrees. [astroquery.utils.commons]
Table length=4
idxtarget_nameN_obs
0Trappist-1295
1V* XX Cha14
2M 310
3M 57143
For targets where no JWST Observations exist, there is no potential for duplication.

Evaluating Potential Duplications#

To examine the matching Observations for the other targets in more detail, use the query_criteria() method. Also, use a cone search (i.e., specify a coordinate pair and a radius), rather than an area search over a region in RA and Dec; see the Appendix for details. Note that a new column called “obs” is first added to the table to contain the results of the Observation search.

targets['obs'] = None
for t in targets: 
    t['obs'] = Observations.query_criteria(
        obs_collection='JWST'
        ,coordinates=f"{t['ra_deg']} {t['dec_deg']}"
        ,radius=t['radius_deg']
        )
WARNING: NoResultsWarning: Query returned no results. [astroquery.mast.discovery_portal]

Targets with matching JWST Observations are examined in the following sub-sections.

Trappist-1#

Trappist-1 is a well known exo-planet host star. If the intended observations are timeseries spectroscopy, the search would naturally be limited to a small area of sky. There are a several unique JWST observations of this target, so it is important to examine the relevant details.

out_cols = ['target_name','instrument_name','filters','t_exptime','proposal_id']
obs = targets[0]['obs']
unique(obs[out_cols]).show_in_notebook()
Table length=17
idxtarget_nameinstrument_namefilterst_exptimeproposal_id
0TRAPPIST-1MIRI/IMAGEF1500W11535.8412304
1TRAPPIST-1MIRI/IMAGEF1500W11574.6922304
2TRAPPIST-1MIRI/IMAGEF1500W74151.93077
3TRAPPIST-1MIRI/IMAGEF1500W138234.5373077
4TRAPPIST-1NIRISS/SOSSCLEAR;GR700XD988.922589
5TRAPPIST-1NIRISS/SOSSCLEAR;GR700XD11965.9321201
6TRAPPIST-1NIRISS/SOSSCLEAR;GR700XD15130.4762589
7TRAPPIST-1NIRISS/SOSSCLEAR;GR700XD15723.8282589
8TRAPPIST-1NIRSPEC/SLITCLEAR;PRISM9770.1121201
9TRAPPIST-1NIRSPEC/SLITCLEAR;PRISM11855.6982420
10TRAPPIST-1NIRSPEC/SLITCLEAR;PRISM11870.0081331
11TRAPPIST-1NIRSPEC/SLITCLEAR;PRISM11929.941981
12TRAPPIST-1NIRSPEC/SLITCLEAR;PRISM13751.5562420
13TRAPPIST-1NIRSPEC/SLITCLEAR;PRISM15085.3242589
14TRAPPIST-1BMIRI/IMAGEF1280W13983.4271279
15TRAPPIST-1BMIRI/IMAGEF1500W15690.0761177
16UNKNOWNNIRSPEC/SLITOPAQUE;MIRROR0.032742

You will need to examine the specifics of the programs that obtained the observations to know for certain which of the 7 known exoplanets was being observed, and therefore whether your intended observations would duplicate. Here are the unique Program Titles, which may offer clues:

obs['title'] = [x[:80] for x in obs['obs_title']]
unique(obs['proposal_id','title']).pprint(max_width=-1)
proposal_id                                      title                                      
----------- --------------------------------------------------------------------------------
       1177                                       MIRI observations of transiting exoplanets
       1201  NIRISS Exploration of the Atmospheric diversity of Transiting exoplanets (NEAT)
       1279                                               Thermal emission from Trappist-1 b
       1331                                              Transit Spectroscopy of TRAPPIST-1e
       1981 Tell Me How Im Supposed To Breathe With No Air: Measuring the Prevalence and Div
       2304                  Hot Take on a Cool World:  Does Trappist-1c Have an Atmosphere?
       2420     Probing the Terrestrial Planet TRAPPIST-1c for the Presence of an Atmosphere
       2589                             Atmospheric reconnaissance of the TRAPPIST-1 planets
       2742                                       NIRSpec darks for detector reconfiguration
       3077                                          TRAPPIST-1 Planets: Atmospheres Or Not?

The Program Status Pages offer the complete specifications for each existing program:

obs['proposal_URL'] = [get_program_URL(x) for x in obs['proposal_id']]
unique(obs['proposal_id','proposal_URL']).pprint(max_width=-1)
proposal_id                                proposal_URL                              
----------- -------------------------------------------------------------------------
       1177 (http://www.stsci.edu/cgi-bin/get-proposal-info?id=1177&observatory=JWST)
       1201 (http://www.stsci.edu/cgi-bin/get-proposal-info?id=1201&observatory=JWST)
       1279 (http://www.stsci.edu/cgi-bin/get-proposal-info?id=1279&observatory=JWST)
       1331 (http://www.stsci.edu/cgi-bin/get-proposal-info?id=1331&observatory=JWST)
       1981 (http://www.stsci.edu/cgi-bin/get-proposal-info?id=1981&observatory=JWST)
       2304 (http://www.stsci.edu/cgi-bin/get-proposal-info?id=2304&observatory=JWST)
       2420 (http://www.stsci.edu/cgi-bin/get-proposal-info?id=2420&observatory=JWST)
       2589 (http://www.stsci.edu/cgi-bin/get-proposal-info?id=2589&observatory=JWST)
       2742 (http://www.stsci.edu/cgi-bin/get-proposal-info?id=2742&observatory=JWST)
       3077 (http://www.stsci.edu/cgi-bin/get-proposal-info?id=3077&observatory=JWST)
There are MIRI imaging observations (likely Time-series) in multiple bands, and both NIRISS and NIRSpec spectroscopic observations of this target that could conflict with your intended proposal.

V* XX Cha#

This pre-main-sequence star has a debris disk, and might be worth a coronagraphic observation. The search area is small (20 arcsec radius) to exclude nearby targets.

out_cols = ['target_name','instrument_name','filters','t_exptime','calib_level','proposal_id']
obs = targets[1]['obs']
unique(obs[out_cols]).show_in_notebook()
Table length=7
idxtarget_nameinstrument_namefilterst_exptimecalib_levelproposal_id
0V-XX-CHA-2MIRI/IFUCH13696.34831282
1V-XX-CHA-2MIRI/IFUCH23696.34831282
2V-XX-CHA-2MIRI/IFUCH33696.34831282
3V-XX-CHA-2MIRI/IFUCH43696.34831282
4V-XX-CHA-2MIRI/IMAGEF1280W3696.34831282
5V-XX-CHA-2NIRSPEC/IFUF290LP;G395H32.2121282
6V-XX-CHA-2NIRSPEC/IFUF290LP;G395H1288.41631282
While JWST spectra from MIRI and NIRSpec exist, there is no potential for duplication with a coronagraphic observation.

M 57#

The Ring Nebula (NGC 6720) is well observed, particularly with HST and, recently, with JWST. We searched a 4-arcmin region around the source to include potential spectral Observations in the nebular periphery. The following table shows the unique combinations of instrument configurations.

out_cols = ['target_name','instrument_name','filters','t_exptime','proposal_id']
obs = targets[3]['obs']
unique(obs[out_cols]).show_in_notebook()
Table length=43
idxtarget_nameinstrument_namefilterst_exptimeproposal_id
0NGC6720MIRI/IMAGEF1000W444.00799999999991558
1NGC6720MIRI/IMAGEF1130W444.00799999999991558
2NGC6720MIRI/IMAGEF1280W444.00799999999991558
3NGC6720MIRI/IMAGEF1500W444.00799999999991558
4NGC6720MIRI/IMAGEF1800W444.00799999999991558
5NGC6720MIRI/IMAGEF2100W444.00799999999991558
6NGC6720MIRI/IMAGEF2550W444.00799999999991558
7NGC6720MIRI/IMAGEF560W444.00799999999991558
8NGC6720MIRI/IMAGEF770W444.00799999999991558
9NGC6720NIRCAM/IMAGEF150W2;F162M483.155999999999841558
10NGC6720NIRCAM/IMAGEF212N483.155999999999841558
11NGC6720NIRCAM/IMAGEF300M483.155999999999841558
12NGC6720NIRCAM/IMAGEF335M483.155999999999841558
13NGC6720-MRS-POSITION-1MIRI/IFUCH12763.9361558
14NGC6720-MRS-POSITION-1MIRI/IFUCH22763.9361558
15NGC6720-MRS-POSITION-1MIRI/IFUCH32763.9361558
16NGC6720-MRS-POSITION-1MIRI/IFUCH42763.9361558
17NGC6720-MRS-POSITION-1MIRI/IMAGEF1000W921.3121558
18NGC6720-MRS-POSITION-1MIRI/IMAGEF1130W921.3121558
19NGC6720-MRS-POSITION-1MIRI/IMAGEF770W921.3121558
20NGC6720-MRS-POSITION-2MIRI/IFUCH12763.9361558
21NGC6720-MRS-POSITION-2MIRI/IFUCH22763.9361558
22NGC6720-MRS-POSITION-2MIRI/IFUCH32763.9361558
23NGC6720-MRS-POSITION-2MIRI/IFUCH42763.9361558
24NGC6720-MRS-POSITION-2MIRI/IMAGEF1000W921.3121558
25NGC6720-MRS-POSITION-2MIRI/IMAGEF1130W921.3121558
26NGC6720-MRS-POSITION-2MIRI/IMAGEF770W921.3121558
27NGC6720-NIRSPEC-POSITION-1NIRSPEC/IFUF070LP;G140M145.8891558
28NGC6720-NIRSPEC-POSITION-1NIRSPEC/IFUF070LP;G140M583.5561558
29NGC6720-NIRSPEC-POSITION-1NIRSPEC/IFUF100LP;G140M145.8891558
30NGC6720-NIRSPEC-POSITION-1NIRSPEC/IFUF100LP;G140M583.5561558
31NGC6720-NIRSPEC-POSITION-1NIRSPEC/IFUF170LP;G235M145.8891558
32NGC6720-NIRSPEC-POSITION-1NIRSPEC/IFUF170LP;G235M583.5561558
33NGC6720-NIRSPEC-POSITION-1NIRSPEC/IFUF290LP;G395M145.8891558
34NGC6720-NIRSPEC-POSITION-1NIRSPEC/IFUF290LP;G395M583.5561558
35NGC6720-NIRSPEC-POSITION-2NIRSPEC/IFUF070LP;G140M145.8891558
36NGC6720-NIRSPEC-POSITION-2NIRSPEC/IFUF070LP;G140M583.5561558
37NGC6720-NIRSPEC-POSITION-2NIRSPEC/IFUF100LP;G140M145.8891558
38NGC6720-NIRSPEC-POSITION-2NIRSPEC/IFUF100LP;G140M583.5561558
39NGC6720-NIRSPEC-POSITION-2NIRSPEC/IFUF170LP;G235M145.8891558
40NGC6720-NIRSPEC-POSITION-2NIRSPEC/IFUF170LP;G235M583.5561558
41NGC6720-NIRSPEC-POSITION-2NIRSPEC/IFUF290LP;G395M145.8891558
42NGC6720-NIRSPEC-POSITION-2NIRSPEC/IFUF290LP;G395M583.5561558
There are both MIRI and NIRCam images in many passbands, and MIRI and NIRSpec spectroscopic IFU Observations of this target that could conflict with the intended proposal.

To retain this target it would be necessary to obtain spectra in a different location, or to use a different instrument/configuration/data-taking mode (e.g., time-series of the central star). It may be helpful to see the footprints of these Observations in the MAST Portal, or to view the observation specifications in Program GO-1558.

Appendix: Caveats#

Source Name#

It is possible to search MAST for individual sources by name, in one of two ways: Object Name or by Target Name.

  • Object Name: MAST invokes an astrophysical name resolver (e.g., NED or Simbad) to look up source coordinates. Not all object names are recognized by resolvers. Also, different resolvers sometimes return substantially different coordinates; MAST will pick one of them (but not tell you which one). Sometimes the coordinates returned by the resolvers differ by substantial amounts, often because they refer to different sources.

    • A classic case is the bright star μ Eri from the Bayer Greek letter system, and the star MU Eri in the general catalog of variable stars. in MAST,

      • Searching for * mu. eri (or just plain mu eri matches μ Eri (RA = 4:45:30.15, Dec = -3:15:16.777)

      • Searching for V* mu eri matches MU Eri (RA = 2:48:10.566, Dec = -15:18:04.03)

  • Target Name: This search matches to target names specified by Investigators in observing proposals to refer to sources. These are not guaranteed to match standard names for astrophysical sources, and often do not.

Searching by source names is not recommended when checking for potential duplications with fixed targets because the matches are not reliable, consistent, or complete. The exception would be for matching Object names of well known solar system bodies.

The best approach is to use an independent source for the coordinates of your desired target. Failing that, the Mast.resolve_object() method will return coordinates in the ICRS reference frame, which you can use to verify whether a target name resolves to the coordinates you intend.

Additional Resources#

About this notebook#

This notebook was developed by Archive Sciences Branch staff. For support, please direct questions to the Archive HelpDesk at archive@stsci.edu, or through the JWST HelpDesk Portal. Space Telescope Logo

Last update: 2024 Jan