Roman Research Nexus Catalog Database access

Roman Research Nexus Catalog Database access#

Kernel Information and Read-Only Status#

To run this notebook, please select “Roman Research Nexus {VERSION}” kernel at the top right of your window. For example “Roman Research Nexus 2026.2”.

This notebook is read-only. You can run cells and make edits, but you must save changes to a different location. We recommend saving the notebook within your home directory, or to a new folder within your home (e.g. file > save notebook as > my-nbs/nb.ipynb). Note that a directory must exist before you attempt to add a notebook to it.

Introduction#

This notebook demonstrates how to access and query Roman catalogs held at the Mikulski Archive for Space Telescopes (MAST), through (1) astroquery as well as (2) advanced query patterns using Astronomical Data Query Language (ADQL) queries via a Table Access Protocol (TAP) service at MAST.

MAST will provide access to a range of Roman catalog products (such as multiband photometric source catalogs as detected from WFI imaging; spectral catalogs derived from WFI grism and prism spectroscopy; microlensing event variability and light curve catalogs; and Project Infrastructure team created catalogs). This notebook demonstrates methods of querying these catalogs from MAST databases, enabling efficient filtering and sample selection across both spatial location and miriad other properties.

Defining terms#

  • SQL: Structured Query Language, a programming language used for relational databases.

  • ADQL: Astronomical Data Query Language, a SQL-like programming language that includes functionality for spatial searches.

  • TAP: Table Access Protocol, an International Virtual Observatory standard for providing programmatic access to catalogs.

Imports#

Here we import the required packages for our catalog database access examples including:

from astroquery.mast import Catalogs
import pyvo as vo

from astropy.coordinates import SkyCoord
import astropy.units as u

import matplotlib.pyplot as plt
import numpy as np

Accessing Roman Catalogs#

In this section, we will demonstrate how to obtain information about the catalog schemas (covering descriptions of tables, as well as column units, datatypes, and descriptions), and then methods of connecting to and querying Roman catalogs.

Obtaining information about Roman Catalogs#

Roman catalogs are Level 4 (L4) extracted products, and are produced by multiple Roman data pipelines. For full details about the data processing and detailed contents of Roman catalogs, please see RDox.

Users can also search and explore information about Roman catalogs using MAST’s Catalog Schema Browser. The schema browser provides searchable listings of column names, units, and descriptions for a number of catalogs available at MAST. As Roman catalogs become available, their schemas will be made available through this Schema Browser.

The MAST Catalog Schema Browser is integrated into RRN, and can be launched from the JupyterLab Launcher:

Open the Launcher from the JupyterLab menu using File -> New Launcher.

Launcher panel
In the future, user-contributed catalogs (L5) are anticipated to be made available through similar methods; these will be documented separately.

Querying Roman Catalogs using astroquery#

To begin, we will query Roman catalogs using astroquery.

For demonstration purposes (as Roman catalogs are not yet available), our example will query the PanSTARRS DR2 catalog near NGC 6535. This PanSTARRS example provides an overview of the access methods that will be available for Roman catalogs (astroquery, and ADQL/TAP below). In particular, both PanSTARRS and Roman (will) have large numbers of rows, which requires database queries for access; it is infeasible to load the full tables into memory for analysis.

In addition to a cone search, we will also filter to select only bright, blue stars. The specific selection criteria we will apply are:

  • Located within 1.5 arcminutes of the NGC 6535

  • Are point sources (not extended objects)

  • Detected flux in the $g$, $i$, $y$ bands (quality cuts)

  • Have $i$ band magnitudes brighter than 18

  • Have $g-y$ colors less than 0.8 (the bluest, reddest filters for PanSTARRS)

Though the specifics of the PanSTARRS and forthcoming Roman catalogs differ (in terms of filters, specific columns stemming from different processing pipelines, etc.), similar queries combining spatial, point vs. resolved source, and multiband color/magnitude criteria will be possible with Roman WFI multiband catalogs.

Based on documentation about PanSTARRS catalogs — the PanSTARRS DR2 catalogs documentation, the astroquery.mast PanSTARRS catalog information, and the MAST Catalog Schema Browser — we will use the MeanObject catalog provided by astroquery.mast. This table provides relevant photometric information in the columns gMeanPSFMag, iMeanPSFMag, yMeanPSFMag (magnitudes optimized for point sources), and iMeanKronMag (used to distinguish between point and resolved sources), as well as position information (raMean, decMean).

Also, with astroquery, we can directly filter for some of these constraints. However, we will need to post-process the returned results to apply other constraints.

We first obtain the sky coordinates for NGC 6535.

# Resolve the position of NGC 6535
c0 = SkyCoord.from_name("NGC 6535")
c0

<SkyCoord (ICRS): (ra, dec) in deg
    (270.96045833, -0.29763889)>

We then extract photometric and position data (by selecting specific columns to return) for all objects in the MeanObject catalog that are within a 1.5 arcminute radius around NGC 6535. We also include constraints to limit the results to only objects with measured magnitudes for gMeanPSFMag, iMeanPSFMag, yMeanPSFMag, iMeanKronMag.

Next we apply post-processing constraints to refine our sample.

As we are interested only in stars, we compare the Kron and PSF magnitudes in the $i$ band. Stars (point sources) typically have iMeanPSFMAG-iMeanKronMag <= 0.05 (as shown in the PanSTARRS documentation).

Lastly, we apply the color and magnitude cuts, and then display the table with our final sample.

# Query MAST for matching observations in 
# the PanSTARRS DR2 Mean catalog
results_aq = Catalogs.query_criteria(
    collection="ps1_dr2",
    catalog="mean_object",
    coordinates=c0,
    radius = 1.5*u.arcmin,
    select_cols=["objID", "raMean","decMean",
             "gMeanPSFMag","iMeanPSFMag","yMeanPSFMag",
             "iMeanKronMag"],    # List of columns to return
    gMeanPSFMag=">-999",   # mag > -999; missing magnitudes are -999
    iMeanPSFMag=">-999",   # mag > -999; missing magnitudes are -999
    yMeanPSFMag=">-999",   # mag > -999; missing magnitudes are -999
    iMeanKronMag=">-999",  # mag > -999; missing magnitudes are -999
)

# Note: the returned table column names are all lower case, 
# which we use below.

# Post-process cuts:
# 1. Distinguish stars from extended objects using
#    difference of PSF, Kron magnitudes
results_aq = results_aq[
    (results_aq['imeanpsfmag'] - results_aq['imeankronmag']) <= 0.05
]

# 2. Apply color, magnitude cuts
# Copy original catalog for reference:
results_aq_all = results_aq.copy()
results_aq = results_aq[
    ((results_aq['gmeanpsfmag'] - results_aq['ymeanpsfmag']) < 0.8) 
    & (results_aq['imeanpsfmag'] < 18)
]

# Final sample selection
results_aq
Table length=11
objidrameandecmeangmeanpsfmagimeanpsfmagymeanpsfmagimeankronmag
degdegmagmagmagmag
int64float64float64float32float32float32float32
107662709583301962270.9583027798802-0.2820789391894300617.117516.9916.948917.0684
107642709703759594270.9703642717264-0.292385934762494816.654516.470316.359516.4361
107632709534144766270.95338546879503-0.304745564201800917.076516.768316.635216.7509
107622709573665384270.957335353332-0.312566895194870817.447217.333217.336617.4682
107652709505976571270.95056941908393-0.286571058820547916.366416.102715.921416.1581
107632709733451952270.97331765199516-0.307089439772228616.925416.762816.702316.8846
107622709526829524270.9526486988225-0.3091180083520196416.464516.194316.032516.2402
107642709674579676270.9674327466154-0.2923132350920778516.25716.015315.856516.0009
107642709539600777270.95393138778394-0.299738383120972416.105815.742515.5315.7387
107662709570245695270.9569986424153-0.278962515691736616.86216.69316.611216.7813
107642709657944948270.9657688899816-0.2962528120512241716.441316.227416.039416.186

We visualize our results by plotting a $g-y$ vs $i$ color-magnitude diagram.

The selected bright, blue stars are shown as large red points, while all stars within 1.5 arcminutes of NGC 6535 are shown in gray for comparison.

f, ax = plt.subplots()
f.set_size_inches((5,5))

# Stars meeting the color/magnitude cuts
ax.scatter(
    results_aq['gmeanpsfmag']-results_aq['ymeanpsfmag'],
    results_aq['imeanpsfmag'], 
    s=15,lw=0, color='red',
    label='Bright blue stars',
)

# Plot the full set of stars for comparison
ax.scatter(
    results_aq_all['gmeanpsfmag']-results_aq_all['ymeanpsfmag'],
    results_aq_all['imeanpsfmag'], 
    s=5,lw=0,zorder=-1, color='gray',
    label='All stars in region',
)

ax.legend()

ax.set_xlabel("g-y, PSF Mag")
ax.set_ylabel("i, PSF Mag")

ax.set_ylim(ax.get_ylim()[::-1])    # Invert for correct magnitude order

(21.140229082107545, 12.911569738388062)
../../_images/361bab9d7b5f1c264b45b127d93dbaf7c5975c066c2fb8d2d21e657520836393.png

Querying Roman Catalogs using MAST’s TAP service using pyvo#

It is also possible to access Roman catalogs via a TAP service at MAST, by submitting queries written in ADQL. A benefit of constructing queries in the SQL-like language ADQL is that we can specify advanced query patterns, such as comparison constraints and derived constraints (i.e., as needed to apply the star/resolved and color criteria in this example), directly in the query. This makes it possible get the sample of interest in one query, avoiding a two step process where we load a larger sample into memory and then perform final cuts in our notebook.

For this second demonstration, we will apply the same selection criteria as above to select bright, blue stars near NGC 6535 from the PanSTARRS DR2 catalog.

First, we connect to the MAST PanSTARRS DR2 TAP service.

(Note: A full list of MAST TAP services is available via https://mast.stsci.edu/vo-tap/.)

# Use `pyvo` to connect to the MAST PanSTARRS TAP service:
TAP_service = vo.dal.TAPService(
    "https://mast.stsci.edu/vo-tap/api/v0.1/mast_catalogs/"
)

Again, we will access the mean object catalog (mean_object here; see the PanSTARRS DR2 catalogs documentation and MAST Catalog Schema Browser), which contains all the columns we will need.

Note: As a reference when constructing queries, an overview description of methods, capabilities, and maximum result set sizes is available via:

TAP_service.describe()

We now construct our query, including the color/magnitude constraints and star/extended object separation. As an overview, our query is constructed as follows.

We specify the table, mean_object, with the “FROM” statement. The subset of columns to be returned (objid, ramean, decmean, gmeanpsfmag, imeanpsfmag, ymeanpsfmag, imeankronmag) are specified with the “SELECT” statement. We then specify the query constraints in the “WHERE” statement, with multiple constraints linked using “AND”. This includes:

  • the cone search (as in the “CONTAINS(…)” clause; note that for ADQL, the radius must be expressed in degrees),

  • requiring the $g$, $i$, $y$ mean PSF magnitudes and $i$ mean Kron magnitude to be present (cutting objects that have any of these values missing),

  • the PanSTARRS-specific cut to select stars (imeanpsfmag - imeankronmag <= 0.05), and

  • the $g-y$ color and $i$ magnitude cuts.

We then submit this query to the TAP service, and retrieve and display the table with the query results.

(Note: This MAST PanSTARRS TAP service, and the Roman catalog TAP service, are case-insensitive for column names.)

# Query MAST for matching observations in 
# the PanSTARRS DR2 mean catalog

# Note: ADQL comments are preceded by "--"
adql_query = f"""
SELECT objid, ramean, decmean, 
    gmeanpsfmag, imeanpsfmag, ymeanpsfmag, 
    imeankronmag
FROM mean_object
WHERE CONTAINS(
    POINT('ICRS', ramean, decmean),
    CIRCLE('ICRS',{c0.ra.deg}, {c0.dec.deg}, {(1.5*u.arcmin).to_value(u.deg)})          
)=1                                       -- 1.5 arcmin around NGC 6535
AND gmeanpsfmag > -999                    -- mag > -999; missing magnitudes are -999
AND imeanpsfmag > -999                    -- mag > -999; missing magnitudes are -999
AND ymeanpsfmag > -999                    -- mag > -999; missing magnitudes are -999
AND imeankronmag > -999                   -- mag > -999; missing magnitudes are -999
AND (imeanpsfmag - imeankronmag) <= 0.05  -- distinguish stars from extended objects
AND (gmeanpsfmag-ymeanpsfmag) < 0.8       -- color cut
AND imeanpsfmag < 18                      -- magnitude cut
"""

# Submit query to the MAST TAP service:
job = TAP_service.run_async(adql_query)

# Retrieve results:
results_tap = job.to_table()

# Final sample selection
results_tap

/home/runner/micromamba/envs/ci-env/lib/python3.13/site-packages/pyvo/dal/query.py:403: DALOverflowWarning: Results truncated due to server limits. Consider setting a maxrec value.
  warn("Results truncated due to server limits. Consider "
Table length=11
objidrameandecmeangmeanpsfmagimeanpsfmagymeanpsfmagimeankronmag
degdegmagmagmagmag
int64float64float64float32float32float32float32
107662709583301962270.9583027798802-0.2820789391894300617.117516.9916.948917.0684
107652709505976571270.95056941908393-0.286571058820547916.366416.102715.921416.1581
107622709573665384270.957335353332-0.312566895194870817.447217.333217.336617.4682
107642709703759594270.9703642717264-0.292385934762494816.654516.470316.359516.4361
107642709674579676270.9674327466154-0.2923132350920778516.25716.015315.856516.0009
107622709526829524270.9526486988225-0.3091180083520196416.464516.194316.032516.2402
107632709733451952270.97331765199516-0.307089439772228616.925416.762816.702316.8846
107632709534144766270.95338546879503-0.304745564201800917.076516.768316.635216.7509
107642709539600777270.95393138778394-0.299738383120972416.105815.742515.5315.7387
107642709657944948270.9657688899816-0.2962528120512241716.441316.227416.039416.186
107662709570245695270.9569986424153-0.278962515691736616.86216.69316.611216.7813

By leveraging the advanced usage patterns possible with AQDL, this single TAP query returns the same results as our two-step astroquery example where we first queried and then applied the final cuts.

(For this simple example, applying such cuts in memory are not prohibitive, but for cases where the selected sample may be large, performing all cuts server-side can be much more performant.)

We confirm that both methods yield the same set of objects by sorting the two result tables by object ID, and checking that all object IDs are the same.

# Sort both by objid
results_aq.sort('objid')
results_tap.sort('objid')
# Check the ID lists are the same:
print(np.all(results_aq['objid'] == results_tap['objid']))

True

Additional Resources#

Additional information can be found at the following links:

Other RRN notebooks#

  • Data Discovery and Access: The next step in this workflow, covering how to search for and access (particularly cloud streaming access) of file-based Roman products.

MAST notebooks#

Other resources#

Citations and acknowledgements#

About this Notebook#

The information about TAP access is based on the TIKE MAST PanSTARRS DR2 TAP notebook by Rick White & Theresa Dower, with edits from T. Dutkiewicz.

Author: Sedona Price
Updated On: 2026-08-03

Top of page roman_logo