MAST Table Access Protocol Hubble Source Catalog Demo#
This tutorial demonstrates how to use astroquery to access the Hubble Source Catalog (HSC) via a Virtual Observatory standard Table Access Protocol (TAP) service at MAST, and work with the resultant data.
Table of Contents#
HSC TAP Service Introduction
Imports
Connecting, Finding, and Displaying Table Information
Connecting to a TAP Service
Querying for Table Schema Information
Querying for Data
Use Case: Plotting a light curve for the most variable object in a field
Use Case: Create a color magnitude diagram for the Small Magellanic Cloud
Additional Resources
About This Notebook
HSC TAP Service Introduction#
Table Access Protocol (TAP) services allow more direct and flexible access to astronomical data than the simpler types of IVOA standard data services. Queries are built with the SQL-like Astronomical Data Query Language (ADQL), and can include geographic/spatial queries as well as filtering on other characteristics of the data. This also allows the user fine-grained control over the returned columns, unlike the fixed set of coumns returned from cone, image, and spectral services.
For this example, we’ll be using the astropy affiliated PyVO client, which is interoperable with other valid TAP services, including those at MAST. PyVO documentation is available at ReadTheDocs: https://pyvo.readthedocs.io
We’ll be using PyVO to call the most recent version (3) of the Hubble Source Catalog TAP service at MAST. The schema is described within the service, and we’ll show how to inspect it. The schema is also the same as the one available via the CasJobs interface, with an additional view added for the most common positional queries. CasJobs has its own copy of the schema documentation, which can be accessed through its own site: http://mastweb.stsci.edu/hcasjobs/
Imports#
import pyvo as vo
## For handling ordinary astropy Tables in responses
from astropy.table import Table
# For displaying and manipulating some types of results
%matplotlib inline
import requests
import astropy
import numpy as np
import time
from matplotlib import pyplot as plt
# For the second example: kernel density estimates
from scipy.stats import gaussian_kde
# suppress unimportant unit warnings from many TAP services
import warnings
warnings.filterwarnings("ignore", module="astropy.io.votable.*")
Connecting, Finding, and Displaying Table Information#
Connecting to a TAP Service#
The PyVO library is able to connect to any TAP service, given the “base” URL as noted in metadata registry resources describing the service. This is the URL for the newest version of the Hubble Source Catalog TAP service.
HSC_service = vo.dal.TAPService("https://mast.stsci.edu/vo-tap/api/v0.1/hsc/")
Querying for Table Schema Information#
TAP services are self-describing, which means the service itself can be asked for its schema and documentation about it. Since the Hubble Source Catalog does not follow a data model described by a standard, this is the best way to see what tables and columns we have available to then query based on geometry or other filters.
Note that several views for HSC, including the main science table SumMagAper2CatView, are extremely wide, containing columns for all potential filters, each of which may have null data. The ability to filter queries based on only columns we want, or that aren’t null, makes TAP services more flexible and potentially faster.
HSC_service.describe()
Capability ivo://ivoa.net/std/TAP
Interface vs:ParamHTTP
https://mast.stsci.edu/vo-tap/api/v0.1/hsc
Language ADQL
Output format application/x-votable+xml
Also available as votable
Output format text/csv;header=present
Also available as csv
Maximum size of resultsets
Default 100000 row
Maximum 100000 row
Capability ivo://ivoa.net/std/DALI#examples
Interface vr:WebBrowser
https://mast.stsci.edu/vo-tap/api/v0.1/hsc/examples
HSC_tables = HSC_service.tables
for tablename in HSC_tables.keys():
if not "tap_schema" in tablename:
# No descriptions are available for these tables; this will just print the name
HSC_tables[tablename].describe()
print("Columns={}".format(sorted([k.name for k in HSC_tables[tablename].columns ])))
print("----")
dbo.detailedcatalog
Detailed list of source catalog parameters
Columns=['abscorr', 'aperture', 'catid', 'ci', 'd', 'det', 'detector', 'dsigma', 'exposuretime', 'filter', 'flags', 'fluxaper2', 'imageid', 'imagename', 'instrument', 'kronradius', 'magaper2', 'magauto', 'matchdec', 'matchid', 'matchra', 'memid', 'mode', 'propid', 'sourcedec', 'sourceid', 'sourcera', 'startmjd', 'starttime', 'stopmjd', 'stoptime', 'targetname', 'wavelength', 'ximage', 'yimage']
----
dbo.hcvdetailedview
Detailed list of Hubble Catalog of Variables parameters
Columns=['ci', 'corrmag', 'd', 'filter', 'imagename', 'mag', 'magerr', 'matchid', 'mjd']
----
dbo.hcvsummaryview
Summary list of Hubble Catalog of Variables parameters
Columns=['autoclass', 'chi2', 'dec', 'expertclass', 'filter', 'filterdetflag', 'groupid', 'mad', 'matchid', 'meancorrmag', 'meanmag', 'numlc', 'ra', 'subgroupid', 'varqualflag']
----
dbo.propermotionsview
List of proper motion information
Columns=['ci', 'ci_sigma', 'decmean', 'decmeanerr', 'dsigma', 'epochend', 'epochmean', 'epochstart', 'htmid', 'kronradius', 'kronradius_sigma', 'latmean', 'latmeanerr', 'lonmean', 'lonmeanerr', 'numfilters', 'numimages', 'numsources', 'numvisits', 'objid', 'pmdec', 'pmdecdev', 'pmdecerr', 'pmlat', 'pmlatdev', 'pmlaterr', 'pmlon', 'pmlondev', 'pmlonerr', 'pmra', 'pmradev', 'pmraerr', 'ramean', 'rameanerr', 'x', 'y', 'z']
----
dbo.sourcepositionsview
List of source position information
Columns=['ddec', 'dec', 'dlat', 'dlon', 'dra', 'dt', 'epochmean', 'lat', 'lon', 'matchid', 'objid', 'ra', 'sourceid']
----
dbo.summagaper2catview
Summary list of source catalog with Aper2 magnitudes
Columns=['a_f435w', 'a_f435w_mad', 'a_f435w_n', 'a_f475w', 'a_f475w_mad', 'a_f475w_n', 'a_f502n', 'a_f502n_mad', 'a_f502n_n', 'a_f550m', 'a_f550m_mad', 'a_f550m_n', 'a_f555w', 'a_f555w_mad', 'a_f555w_n', 'a_f606w', 'a_f606w_mad', 'a_f606w_n', 'a_f625w', 'a_f625w_mad', 'a_f625w_n', 'a_f658n', 'a_f658n_mad', 'a_f658n_n', 'a_f660n', 'a_f660n_mad', 'a_f660n_n', 'a_f775w', 'a_f775w_mad', 'a_f775w_n', 'a_f814w', 'a_f814w_mad', 'a_f814w_n', 'a_f850lp', 'a_f850lp_mad', 'a_f850lp_n', 'abscorr', 'ci', 'ci_sigma', 'dsigma', 'extinction', 'kronradius', 'kronradius_sigma', 'matchdec', 'matchid', 'matchra', 'numfilters', 'numimages', 'numvisits', 'spectrumflag', 'startmjd', 'starttime', 'stopmjd', 'stoptime', 'targetname', 'w2_f1042m', 'w2_f1042m_mad', 'w2_f1042m_n', 'w2_f122m', 'w2_f122m_mad', 'w2_f122m_n', 'w2_f160bn15', 'w2_f160bn15_mad', 'w2_f160bn15_n', 'w2_f160bw', 'w2_f160bw_mad', 'w2_f160bw_n', 'w2_f170w', 'w2_f170w_mad', 'w2_f170w_n', 'w2_f185w', 'w2_f185w_mad', 'w2_f185w_n', 'w2_f218w', 'w2_f218w_mad', 'w2_f218w_n', 'w2_f255w', 'w2_f255w_mad', 'w2_f255w_n', 'w2_f300w', 'w2_f300w_mad', 'w2_f300w_n', 'w2_f336w', 'w2_f336w_mad', 'w2_f336w_n', 'w2_f343n', 'w2_f343n_mad', 'w2_f343n_n', 'w2_f375n', 'w2_f375n_mad', 'w2_f375n_n', 'w2_f380w', 'w2_f380w_mad', 'w2_f380w_n', 'w2_f390n', 'w2_f390n_mad', 'w2_f390n_n', 'w2_f410m', 'w2_f410m_mad', 'w2_f410m_n', 'w2_f437n', 'w2_f437n_mad', 'w2_f437n_n', 'w2_f439w', 'w2_f439w_mad', 'w2_f439w_n', 'w2_f450w', 'w2_f450w_mad', 'w2_f450w_n', 'w2_f467m', 'w2_f467m_mad', 'w2_f467m_n', 'w2_f469n', 'w2_f469n_mad', 'w2_f469n_n', 'w2_f487n', 'w2_f487n_mad', 'w2_f487n_n', 'w2_f502n', 'w2_f502n_mad', 'w2_f502n_n', 'w2_f547m', 'w2_f547m_mad', 'w2_f547m_n', 'w2_f555w', 'w2_f555w_mad', 'w2_f555w_n', 'w2_f569w', 'w2_f569w_mad', 'w2_f569w_n', 'w2_f588n', 'w2_f588n_mad', 'w2_f588n_n', 'w2_f606w', 'w2_f606w_mad', 'w2_f606w_n', 'w2_f622w', 'w2_f622w_mad', 'w2_f622w_n', 'w2_f631n', 'w2_f631n_mad', 'w2_f631n_n', 'w2_f656n', 'w2_f656n_mad', 'w2_f656n_n', 'w2_f658n', 'w2_f658n_mad', 'w2_f658n_n', 'w2_f673n', 'w2_f673n_mad', 'w2_f673n_n', 'w2_f675w', 'w2_f675w_mad', 'w2_f675w_n', 'w2_f702w', 'w2_f702w_mad', 'w2_f702w_n', 'w2_f785lp', 'w2_f785lp_mad', 'w2_f785lp_n', 'w2_f791w', 'w2_f791w_mad', 'w2_f791w_n', 'w2_f814w', 'w2_f814w_mad', 'w2_f814w_n', 'w2_f850lp', 'w2_f850lp_mad', 'w2_f850lp_n', 'w2_f953n', 'w2_f953n_mad', 'w2_f953n_n', 'w3_blank', 'w3_blank_mad', 'w3_blank_n', 'w3_f098m', 'w3_f098m_mad', 'w3_f098m_n', 'w3_f105w', 'w3_f105w_mad', 'w3_f105w_n', 'w3_f110w', 'w3_f110w_mad', 'w3_f110w_n', 'w3_f125w', 'w3_f125w_mad', 'w3_f125w_n', 'w3_f126n', 'w3_f126n_mad', 'w3_f126n_n', 'w3_f127m', 'w3_f127m_mad', 'w3_f127m_n', 'w3_f128n', 'w3_f128n_mad', 'w3_f128n_n', 'w3_f130n', 'w3_f130n_mad', 'w3_f130n_n', 'w3_f132n', 'w3_f132n_mad', 'w3_f132n_n', 'w3_f139m', 'w3_f139m_mad', 'w3_f139m_n', 'w3_f140w', 'w3_f140w_mad', 'w3_f140w_n', 'w3_f153m', 'w3_f153m_mad', 'w3_f153m_n', 'w3_f160w', 'w3_f160w_mad', 'w3_f160w_n', 'w3_f164n', 'w3_f164n_mad', 'w3_f164n_n', 'w3_f167n', 'w3_f167n_mad', 'w3_f167n_n', 'w3_f200lp', 'w3_f200lp_mad', 'w3_f200lp_n', 'w3_f218w', 'w3_f218w_mad', 'w3_f218w_n', 'w3_f225w', 'w3_f225w_mad', 'w3_f225w_n', 'w3_f275w', 'w3_f275w_mad', 'w3_f275w_n', 'w3_f280n', 'w3_f280n_mad', 'w3_f280n_n', 'w3_f300x', 'w3_f300x_mad', 'w3_f300x_n', 'w3_f336w', 'w3_f336w_mad', 'w3_f336w_n', 'w3_f343n', 'w3_f343n_mad', 'w3_f343n_n', 'w3_f350lp', 'w3_f350lp_mad', 'w3_f350lp_n', 'w3_f373n', 'w3_f373n_mad', 'w3_f373n_n', 'w3_f390m', 'w3_f390m_mad', 'w3_f390m_n', 'w3_f390w', 'w3_f390w_mad', 'w3_f390w_n', 'w3_f395n', 'w3_f395n_mad', 'w3_f395n_n', 'w3_f410m', 'w3_f410m_mad', 'w3_f410m_n', 'w3_f438w', 'w3_f438w_mad', 'w3_f438w_n', 'w3_f467m', 'w3_f467m_mad', 'w3_f467m_n', 'w3_f469n', 'w3_f469n_mad', 'w3_f469n_n', 'w3_f475w', 'w3_f475w_mad', 'w3_f475w_n', 'w3_f475x', 'w3_f475x_mad', 'w3_f475x_n', 'w3_f487n', 'w3_f487n_mad', 'w3_f487n_n', 'w3_f502n', 'w3_f502n_mad', 'w3_f502n_n', 'w3_f547m', 'w3_f547m_mad', 'w3_f547m_n', 'w3_f555w', 'w3_f555w_mad', 'w3_f555w_n', 'w3_f600lp', 'w3_f600lp_mad', 'w3_f600lp_n', 'w3_f606w', 'w3_f606w_mad', 'w3_f606w_n', 'w3_f621m', 'w3_f621m_mad', 'w3_f621m_n', 'w3_f625w', 'w3_f625w_mad', 'w3_f625w_n', 'w3_f631n', 'w3_f631n_mad', 'w3_f631n_n', 'w3_f645n', 'w3_f645n_mad', 'w3_f645n_n', 'w3_f656n', 'w3_f656n_mad', 'w3_f656n_n', 'w3_f657n', 'w3_f657n_mad', 'w3_f657n_n', 'w3_f658n', 'w3_f658n_mad', 'w3_f658n_n', 'w3_f665n', 'w3_f665n_f6', 'w3_f665n_f6_mad', 'w3_f665n_f6_n', 'w3_f665n_mad', 'w3_f665n_n', 'w3_f673n', 'w3_f673n_mad', 'w3_f673n_n', 'w3_f680n', 'w3_f680n_mad', 'w3_f680n_n', 'w3_f689m', 'w3_f689m_mad', 'w3_f689m_n', 'w3_f763m', 'w3_f763m_mad', 'w3_f763m_n', 'w3_f775w', 'w3_f775w_mad', 'w3_f775w_n', 'w3_f814w', 'w3_f814w_mad', 'w3_f814w_n', 'w3_f845m', 'w3_f845m_mad', 'w3_f845m_n', 'w3_f850lp', 'w3_f850lp_mad', 'w3_f850lp_n', 'w3_f953n', 'w3_f953n_mad', 'w3_f953n_n', 'w3_fq232n', 'w3_fq232n_mad', 'w3_fq232n_n', 'w3_fq243n', 'w3_fq243n_mad', 'w3_fq243n_n', 'w3_fq378n', 'w3_fq378n_mad', 'w3_fq378n_n', 'w3_fq387n', 'w3_fq387n_mad', 'w3_fq387n_n', 'w3_fq422m', 'w3_fq422m_mad', 'w3_fq422m_n', 'w3_fq436n', 'w3_fq436n_mad', 'w3_fq436n_n', 'w3_fq437n', 'w3_fq437n_mad', 'w3_fq437n_n', 'w3_fq492n', 'w3_fq492n_mad', 'w3_fq492n_n', 'w3_fq508n', 'w3_fq508n_mad', 'w3_fq508n_n', 'w3_fq575n', 'w3_fq575n_mad', 'w3_fq575n_n', 'w3_fq619n', 'w3_fq619n_mad', 'w3_fq619n_n', 'w3_fq634n', 'w3_fq634n_mad', 'w3_fq634n_n', 'w3_fq672n', 'w3_fq672n_mad', 'w3_fq672n_n', 'w3_fq674n', 'w3_fq674n_mad', 'w3_fq674n_n', 'w3_fq727n', 'w3_fq727n_mad', 'w3_fq727n_n', 'w3_fq750n', 'w3_fq750n_mad', 'w3_fq750n_n', 'w3_fq889n', 'w3_fq889n_mad', 'w3_fq889n_n', 'w3_fq906n', 'w3_fq906n_mad', 'w3_fq906n_n', 'w3_fq924n', 'w3_fq924n_mad', 'w3_fq924n_n', 'w3_fq937n', 'w3_fq937n_mad', 'w3_fq937n_n', 'w3_g102', 'w3_g102_mad', 'w3_g102_n', 'w3_g141', 'w3_g141_mad', 'w3_g141_n', 'w3_g280', 'w3_g280_mad', 'w3_g280_n']
----
dbo.summagautocatview
Summary list of source catalog with MagAuto magnitudes
Columns=['a_f435w', 'a_f435w_mad', 'a_f435w_n', 'a_f475w', 'a_f475w_mad', 'a_f475w_n', 'a_f502n', 'a_f502n_mad', 'a_f502n_n', 'a_f550m', 'a_f550m_mad', 'a_f550m_n', 'a_f555w', 'a_f555w_mad', 'a_f555w_n', 'a_f606w', 'a_f606w_mad', 'a_f606w_n', 'a_f625w', 'a_f625w_mad', 'a_f625w_n', 'a_f658n', 'a_f658n_mad', 'a_f658n_n', 'a_f660n', 'a_f660n_mad', 'a_f660n_n', 'a_f775w', 'a_f775w_mad', 'a_f775w_n', 'a_f814w', 'a_f814w_mad', 'a_f814w_n', 'a_f850lp', 'a_f850lp_mad', 'a_f850lp_n', 'abscorr', 'ci', 'ci_sigma', 'dsigma', 'extinction', 'kronradius', 'kronradius_sigma', 'matchdec', 'matchid', 'matchra', 'numfilters', 'numimages', 'numvisits', 'spectrumflag', 'startmjd', 'starttime', 'stopmjd', 'stoptime', 'targetname', 'w2_f1042m', 'w2_f1042m_mad', 'w2_f1042m_n', 'w2_f122m', 'w2_f122m_mad', 'w2_f122m_n', 'w2_f160bn15', 'w2_f160bn15_mad', 'w2_f160bn15_n', 'w2_f160bw', 'w2_f160bw_mad', 'w2_f160bw_n', 'w2_f170w', 'w2_f170w_mad', 'w2_f170w_n', 'w2_f185w', 'w2_f185w_mad', 'w2_f185w_n', 'w2_f218w', 'w2_f218w_mad', 'w2_f218w_n', 'w2_f255w', 'w2_f255w_mad', 'w2_f255w_n', 'w2_f300w', 'w2_f300w_mad', 'w2_f300w_n', 'w2_f336w', 'w2_f336w_mad', 'w2_f336w_n', 'w2_f343n', 'w2_f343n_mad', 'w2_f343n_n', 'w2_f375n', 'w2_f375n_mad', 'w2_f375n_n', 'w2_f380w', 'w2_f380w_mad', 'w2_f380w_n', 'w2_f390n', 'w2_f390n_mad', 'w2_f390n_n', 'w2_f410m', 'w2_f410m_mad', 'w2_f410m_n', 'w2_f437n', 'w2_f437n_mad', 'w2_f437n_n', 'w2_f439w', 'w2_f439w_mad', 'w2_f439w_n', 'w2_f450w', 'w2_f450w_mad', 'w2_f450w_n', 'w2_f467m', 'w2_f467m_mad', 'w2_f467m_n', 'w2_f469n', 'w2_f469n_mad', 'w2_f469n_n', 'w2_f487n', 'w2_f487n_mad', 'w2_f487n_n', 'w2_f502n', 'w2_f502n_mad', 'w2_f502n_n', 'w2_f547m', 'w2_f547m_mad', 'w2_f547m_n', 'w2_f555w', 'w2_f555w_mad', 'w2_f555w_n', 'w2_f569w', 'w2_f569w_mad', 'w2_f569w_n', 'w2_f588n', 'w2_f588n_mad', 'w2_f588n_n', 'w2_f606w', 'w2_f606w_mad', 'w2_f606w_n', 'w2_f622w', 'w2_f622w_mad', 'w2_f622w_n', 'w2_f631n', 'w2_f631n_mad', 'w2_f631n_n', 'w2_f656n', 'w2_f656n_mad', 'w2_f656n_n', 'w2_f658n', 'w2_f658n_mad', 'w2_f658n_n', 'w2_f673n', 'w2_f673n_mad', 'w2_f673n_n', 'w2_f675w', 'w2_f675w_mad', 'w2_f675w_n', 'w2_f702w', 'w2_f702w_mad', 'w2_f702w_n', 'w2_f785lp', 'w2_f785lp_mad', 'w2_f785lp_n', 'w2_f791w', 'w2_f791w_mad', 'w2_f791w_n', 'w2_f814w', 'w2_f814w_mad', 'w2_f814w_n', 'w2_f850lp', 'w2_f850lp_mad', 'w2_f850lp_n', 'w2_f953n', 'w2_f953n_mad', 'w2_f953n_n', 'w3_blank', 'w3_blank_mad', 'w3_blank_n', 'w3_f098m', 'w3_f098m_mad', 'w3_f098m_n', 'w3_f105w', 'w3_f105w_mad', 'w3_f105w_n', 'w3_f110w', 'w3_f110w_mad', 'w3_f110w_n', 'w3_f125w', 'w3_f125w_mad', 'w3_f125w_n', 'w3_f126n', 'w3_f126n_mad', 'w3_f126n_n', 'w3_f127m', 'w3_f127m_mad', 'w3_f127m_n', 'w3_f128n', 'w3_f128n_mad', 'w3_f128n_n', 'w3_f130n', 'w3_f130n_mad', 'w3_f130n_n', 'w3_f132n', 'w3_f132n_mad', 'w3_f132n_n', 'w3_f139m', 'w3_f139m_mad', 'w3_f139m_n', 'w3_f140w', 'w3_f140w_mad', 'w3_f140w_n', 'w3_f153m', 'w3_f153m_mad', 'w3_f153m_n', 'w3_f160w', 'w3_f160w_mad', 'w3_f160w_n', 'w3_f164n', 'w3_f164n_mad', 'w3_f164n_n', 'w3_f167n', 'w3_f167n_mad', 'w3_f167n_n', 'w3_f200lp', 'w3_f200lp_mad', 'w3_f200lp_n', 'w3_f218w', 'w3_f218w_mad', 'w3_f218w_n', 'w3_f225w', 'w3_f225w_mad', 'w3_f225w_n', 'w3_f275w', 'w3_f275w_mad', 'w3_f275w_n', 'w3_f280n', 'w3_f280n_mad', 'w3_f280n_n', 'w3_f300x', 'w3_f300x_mad', 'w3_f300x_n', 'w3_f336w', 'w3_f336w_mad', 'w3_f336w_n', 'w3_f343n', 'w3_f343n_mad', 'w3_f343n_n', 'w3_f350lp', 'w3_f350lp_mad', 'w3_f350lp_n', 'w3_f373n', 'w3_f373n_mad', 'w3_f373n_n', 'w3_f390m', 'w3_f390m_mad', 'w3_f390m_n', 'w3_f390w', 'w3_f390w_mad', 'w3_f390w_n', 'w3_f395n', 'w3_f395n_mad', 'w3_f395n_n', 'w3_f410m', 'w3_f410m_mad', 'w3_f410m_n', 'w3_f438w', 'w3_f438w_mad', 'w3_f438w_n', 'w3_f467m', 'w3_f467m_mad', 'w3_f467m_n', 'w3_f469n', 'w3_f469n_mad', 'w3_f469n_n', 'w3_f475w', 'w3_f475w_mad', 'w3_f475w_n', 'w3_f475x', 'w3_f475x_mad', 'w3_f475x_n', 'w3_f487n', 'w3_f487n_mad', 'w3_f487n_n', 'w3_f502n', 'w3_f502n_mad', 'w3_f502n_n', 'w3_f547m', 'w3_f547m_mad', 'w3_f547m_n', 'w3_f555w', 'w3_f555w_mad', 'w3_f555w_n', 'w3_f600lp', 'w3_f600lp_mad', 'w3_f600lp_n', 'w3_f606w', 'w3_f606w_mad', 'w3_f606w_n', 'w3_f621m', 'w3_f621m_mad', 'w3_f621m_n', 'w3_f625w', 'w3_f625w_mad', 'w3_f625w_n', 'w3_f631n', 'w3_f631n_mad', 'w3_f631n_n', 'w3_f645n', 'w3_f645n_mad', 'w3_f645n_n', 'w3_f656n', 'w3_f656n_mad', 'w3_f656n_n', 'w3_f657n', 'w3_f657n_mad', 'w3_f657n_n', 'w3_f658n', 'w3_f658n_mad', 'w3_f658n_n', 'w3_f665n', 'w3_f665n_f6', 'w3_f665n_f6_mad', 'w3_f665n_f6_n', 'w3_f665n_mad', 'w3_f665n_n', 'w3_f673n', 'w3_f673n_mad', 'w3_f673n_n', 'w3_f680n', 'w3_f680n_mad', 'w3_f680n_n', 'w3_f689m', 'w3_f689m_mad', 'w3_f689m_n', 'w3_f763m', 'w3_f763m_mad', 'w3_f763m_n', 'w3_f775w', 'w3_f775w_mad', 'w3_f775w_n', 'w3_f814w', 'w3_f814w_mad', 'w3_f814w_n', 'w3_f845m', 'w3_f845m_mad', 'w3_f845m_n', 'w3_f850lp', 'w3_f850lp_mad', 'w3_f850lp_n', 'w3_f953n', 'w3_f953n_mad', 'w3_f953n_n', 'w3_fq232n', 'w3_fq232n_mad', 'w3_fq232n_n', 'w3_fq243n', 'w3_fq243n_mad', 'w3_fq243n_n', 'w3_fq378n', 'w3_fq378n_mad', 'w3_fq378n_n', 'w3_fq387n', 'w3_fq387n_mad', 'w3_fq387n_n', 'w3_fq422m', 'w3_fq422m_mad', 'w3_fq422m_n', 'w3_fq436n', 'w3_fq436n_mad', 'w3_fq436n_n', 'w3_fq437n', 'w3_fq437n_mad', 'w3_fq437n_n', 'w3_fq492n', 'w3_fq492n_mad', 'w3_fq492n_n', 'w3_fq508n', 'w3_fq508n_mad', 'w3_fq508n_n', 'w3_fq575n', 'w3_fq575n_mad', 'w3_fq575n_n', 'w3_fq619n', 'w3_fq619n_mad', 'w3_fq619n_n', 'w3_fq634n', 'w3_fq634n_mad', 'w3_fq634n_n', 'w3_fq672n', 'w3_fq672n_mad', 'w3_fq672n_n', 'w3_fq674n', 'w3_fq674n_mad', 'w3_fq674n_n', 'w3_fq727n', 'w3_fq727n_mad', 'w3_fq727n_n', 'w3_fq750n', 'w3_fq750n_mad', 'w3_fq750n_n', 'w3_fq889n', 'w3_fq889n_mad', 'w3_fq889n_n', 'w3_fq906n', 'w3_fq906n_mad', 'w3_fq906n_n', 'w3_fq924n', 'w3_fq924n_mad', 'w3_fq924n_n', 'w3_fq937n', 'w3_fq937n_mad', 'w3_fq937n_n', 'w3_g102', 'w3_g102_mad', 'w3_g102_n', 'w3_g141', 'w3_g141_mad', 'w3_g141_n', 'w3_g280', 'w3_g280_mad', 'w3_g280_n']
----
dbo.catalog_image_metadata
Summary list of Image processing metadata
Columns=['image_filedate', 'image_name', 'image_published', 'objid', 'pa_aperture', 'sexcat_aper1', 'sexcat_aper2', 'sexcat_filedate', 'sexcat_nobj', 'sexcat_ntot', 'sexcat_published']
----
Querying for Data#
As noted above, this view contains every filter known in the HSC, and can return each even if it is NULL for the given match (this can cause warnings in astroquery). In order to narrow results, one could query on individual filters where their value is not null, or only return certain of them.
Here we are searching for every row with data within a .1 degree circle of RA=129.23 and Dec=7.95, and returning a few columns to get an idea of what we have available.
results = HSC_service.run_async("""
SELECT TOP 10 MatchRA, MatchDec, TargetName, StartTime, StopTime
FROM dbo.SumMagAper2CatView
WHERE CONTAINS(POINT('ICRS', MatchRA, MatchDec),CIRCLE('ICRS',129.23,7.95,0.1))=1
""")
results.to_table()
Table length=10
| matchra | matchdec | targetname | starttime | stoptime |
|---|---|---|---|---|
| deg | deg | |||
| float64 | float64 | object | object | object |
| 129.2301415929065 | 7.950695449211841 | J083655+075712 | 2009-12-07 16:10:34.380000 | 2009-12-07 16:53:36.620000 |
| 129.230634402777 | 7.950503873826787 | J083655+075712 | 2009-12-07 16:10:34.380000 | 2009-12-07 16:53:36.620000 |
| 129.2306298970064 | 7.9506171736020175 | J083655+075712 | 2009-12-07 16:10:34.380000 | 2009-12-07 16:53:36.620000 |
| 129.230916722967 | 7.950108687927433 | J083655+075712 | 2009-12-07 16:10:34.380000 | 2009-12-07 16:53:36.620000 |
| 129.22898220642438 | 7.95042469130075 | J083655+075712 | 2009-12-07 16:10:34.380000 | 2009-12-07 16:53:36.620000 |
| 129.22911087256767 | 7.94912639772422 | J083655+075712 | 2009-12-07 16:10:34.380000 | 2009-12-07 16:53:36.620000 |
| 129.2287543834665 | 7.950874479922319 | J083655+075712 | 2009-12-07 16:10:34.380000 | 2009-12-07 16:53:36.620000 |
| 129.23021294536036 | 7.9516294527782065 | J083655+075712 | 2009-12-07 16:10:34.380000 | 2009-12-07 16:53:36.620000 |
| 129.22832014297236 | 7.949703958229378 | J083655+075712 | 2009-12-07 16:10:34.380000 | 2009-12-07 16:53:36.620000 |
| 129.22904424114446 | 7.951706894402595 | J083655+075712 | 2009-12-07 16:10:34.380000 | 2009-12-07 16:53:36.620000 |
We can also filter by start/stop time or any other column in the view:
results = HSC_service.run_async("""
SELECT TOP 10 MatchID, MatchRA, MatchDec, TargetName, StartTime, StopTime
FROM dbo.SumMagAper2CatView
WHERE
CONTAINS(POINT('ICRS', MatchRA, MatchDec),CIRCLE('ICRS',129.23,7.95,0.1))=1
AND StartTime > '2015-01-01' AND StopTime < '2015-04-01'
""")
results.to_table()
Table length=10
| matchid | matchra | matchdec | targetname | starttime | stoptime |
|---|---|---|---|---|---|
| deg | deg | ||||
| int64 | float64 | float64 | object | object | object |
| 25671973 | 129.23292107651497 | 7.994530012714008 | SDSSJ0837+0801 | 2015-03-15 14:22:48.950000 | 2015-03-15 14:55:27.980000 |
| 27864401 | 129.23498639426631 | 7.994779169533944 | SDSSJ0837+0801 | 2015-03-15 14:22:48.950000 | 2015-03-15 15:07:16.973000 |
| 26083531 | 129.24040045414554 | 7.994206594193051 | SDSSJ0837+0801 | 2015-03-15 14:22:48.950000 | 2015-03-15 15:07:16.973000 |
| 26469969 | 129.23797839073094 | 7.994726362108929 | SDSSJ0837+0801 | 2015-03-15 14:22:48.950000 | 2015-03-15 15:07:16.973000 |
| 85404139 | 129.2219574095833 | 7.995008272788658 | SDSSJ0837+0801 | 2015-03-15 14:22:48.950000 | 2015-03-15 15:07:16.973000 |
| 88918217 | 129.22048740597603 | 7.994958674698499 | SDSSJ0837+0801 | 2015-03-15 14:22:48.950000 | 2015-03-15 15:07:16.973000 |
| 10629948 | 129.2310662829644 | 7.996037541307772 | SDSSJ0837+0801 | 2015-03-15 14:22:48.950000 | 2015-03-15 15:07:16.973000 |
| 20950138 | 129.21896921562035 | 7.995096579949863 | SDSSJ0837+0801 | 2015-03-15 14:22:48.950000 | 2015-03-15 15:07:16.973000 |
| 51540145 | 129.2346595020903 | 7.996304692203746 | SDSSJ0837+0801 | 2015-03-15 14:22:48.950000 | 2015-03-15 15:07:16.973000 |
| 62500828 | 129.23596310108877 | 7.996289301829676 | SDSSJ0837+0801 | 2015-03-15 14:34:37.940000 | 2015-03-15 15:07:16.973000 |
Use Case: Plotting a light curve for the most variable object in a field#
A use case example: search for objects with 10 or more ACS F475W magnitudes in a crowded field near IC 1613 (see HSC Use Case 3). Then get the individual A_F475W measurements for the most variable object in the list and plot the light curve. Note we must use asynchronous query mode for this example rather than synchronous, because it has a longer allowed timeout, which can be useful for large or complex queries.
results = HSC_service.run_async("""
SELECT MatchID, MatchRA, MatchDec, TargetName, NumImages, NumVisits, A_F475W, A_F475W_MAD, A_F475W_N
FROM dbo.SumMagAper2CatView
WHERE
A_F475W_N >= 10
AND
CONTAINS(POINT('ICRS', MatchRA, MatchDec),CIRCLE('ICRS',16.117562,2.162183,0.1))=1
""")
HSC_results = results.to_table()
HSC_results
/opt/hostedtoolcache/Python/3.11.10/x64/lib/python3.11/site-packages/pyvo/dal/query.py:325: DALOverflowWarning: Partial result set. Potential causes MAXREC, async storage space, etc.
warn("Partial result set. Potential causes MAXREC, async storage space, etc.",
Table length=21720
| matchid | matchra | matchdec | targetname | numimages | numvisits | a_f475w | a_f475w_mad | a_f475w_n |
|---|---|---|---|---|---|---|---|---|
| deg | deg | mag | mag | |||||
| int64 | float64 | float64 | object | int32 | int32 | float64 | float64 | int32 |
| 2773895 | 16.117561379987404 | 2.162183863356067 | IC1613 | 26 | 14 | 23.48359966278076 | 0.13920021057128906 | 12 |
| 42821437 | 16.117473416546922 | 2.1623009639591975 | IC1613 | 24 | 12 | 25.8966007232666 | 0.07639884948730469 | 12 |
| 43395956 | 16.11753891193645 | 2.162450000553497 | IC1613 | 26 | 14 | 24.524649620056152 | 0.010099411010742188 | 12 |
| 99348914 | 16.117879067449117 | 2.1623185494588673 | IC1613 | 26 | 14 | 23.855900764465332 | 0.020750045776367188 | 12 |
| 67769235 | 16.11728289292717 | 2.161947424653218 | IC1613 | 26 | 14 | 21.676349639892578 | 0.0062999725341796875 | 12 |
| 52609170 | 16.117391770743467 | 2.161720153122062 | IC1613 | 26 | 14 | 25.514650344848633 | 0.04720020294189453 | 12 |
| 15101151 | 16.1176256174641 | 2.1627492019526584 | IC1613 | 26 | 14 | 25.166099548339844 | 0.012300491333007812 | 12 |
| 78393093 | 16.1178932196339 | 2.162674625053314 | IC1613 | 20 | 12 | 26.46500015258789 | 0.06009960174560547 | 12 |
| 97698117 | 16.117786462889626 | 2.1616046057098774 | IC1613 | 24 | 12 | 25.29955005645752 | 0.026999473571777344 | 12 |
| ... | ... | ... | ... | ... | ... | ... | ... | ... |
| 48130354 | 16.099508259805827 | 2.1242547489396593 | IC1613 | 22 | 12 | 26.72819995880127 | 0.03849983215332031 | 12 |
| 89005808 | 16.098797153012097 | 2.124595528281905 | IC1613 | 24 | 12 | 26.587400436401367 | 0.06390094757080078 | 12 |
| 27400872 | 16.100707004346656 | 2.123581759229902 | IC1613 | 22 | 11 | 25.32710075378418 | 0.04450035095214844 | 11 |
| 65424064 | 16.10038127958583 | 2.123437806823172 | IC1613 | 21 | 11 | 26.598100662231445 | 0.03560066223144531 | 11 |
| 59920509 | 16.100242518180476 | 2.1234727017059543 | IC1613 | 24 | 12 | 26.65369987487793 | 0.03789997100830078 | 12 |
| 22890588 | 16.09901310855383 | 2.1236732912908427 | IC1613 | 17 | 11 | 26.685699462890625 | 0.037601470947265625 | 11 |
| 47730425 | 16.098964825939724 | 2.1234575027278906 | IC1613 | 24 | 12 | 25.13914966583252 | 0.014949798583984375 | 12 |
| 102012433 | 16.098991048151095 | 2.1233534249841197 | IC1613 | 22 | 11 | 26.762100219726562 | 0.05010032653808594 | 11 |
| 2227961 | 16.099174195576534 | 2.123200074969115 | IC1613 | 24 | 12 | 24.69809913635254 | 0.009099960327148438 | 12 |
plt.rcParams.update({'font.size': 16})
plt.figure(1,(10,6))
plt.scatter(HSC_results['a_f475w'], HSC_results['a_f475w_mad'])
plt.xlabel('A_F475W')
plt.ylabel('A_F475W_MAD')
Text(0, 0.5, 'A_F475W_MAD')
madvalues = HSC_results['a_f475w_mad']
i = np.argmax(madvalues)
print()
print(HSC_results[i])
matchid = HSC_results['matchid'][i]
details = HSC_service.run_async("""
SELECT SourceID, ImageID, SourceRA, SourceDec, D, Filter, Detector, MagAper2, StartMJD
FROM dbo.DetailedCatalog
WHERE
MatchID={}
AND Detector='ACS/WFC' AND Filter='F475W' AND Det='Y'
ORDER BY StartMJD
""".format(matchid))
HSC_details = details.to_table()
HSC_details
plt.rcParams.update({'font.size': 16})
plt.figure(1,(10,6))
plt.scatter(HSC_details['StartMJD'], HSC_details['MagAper2'])
plt.xlabel('MJD')
plt.ylabel('A_F475W')
matchid matchra matchdec targetname numimages numvisits a_f475w a_f475w_mad a_f475w_n
deg deg mag mag
--------- ------------------ ------------------ ---------- --------- --------- ------------------ ------------------ ---------
102334412 16.130003898898327 2.1655152822114365 IC1613 26 14 20.970849990844727 0.5844001770019531 12
Text(0, 0.5, 'A_F475W')
Use Case: Create a color magnitude diagram for the Small Magellanic Cloud#
For another example of using data from a TAP service, we start by doing a search around the SMC with a .25 degree radius for objects with ACS F555W and F814W measurements. HSC TAP will limit us to 100k responses by default. Note this is a large query that can take over a minute to run. See HSC Use Case 2 for more details.
Since our query exceeds the 100k response limit, we will get a warning that we are only seeing a partial result set. This partial set is sufficient for our purposes, so you can ignore it!
t0 = time.time()
results = HSC_service.run_async("""
SELECT MatchID, MatchRA, MatchDec, CI, A_F555W, A_F814W
FROM dbo.SumMagAper2CatView
WHERE A_F555W_N > 0 and A_F814W_N > 0
AND CONTAINS(POINT('ICRS', MatchRA, MatchDec),CIRCLE('ICRS',13.1866,-72.8286,0.25))=1
""")
HSC_results = results.to_table()
print("Query completed in {:.1f} sec".format(time.time()-t0))
HSC_results
Query completed in 95.3 sec
Table length=100000
| matchid | matchra | matchdec | ci | a_f555w | a_f814w |
|---|---|---|---|---|---|
| deg | deg | mag | mag | mag | |
| int64 | float64 | float64 | float64 | float64 | float64 |
| 87193895 | 13.024000412138577 | -72.92968859326582 | 1.443580271285257 | 24.840599060058594 | 24.58449935913086 |
| 35773699 | 13.025035586383748 | -72.9299794739034 | 1.027870362187609 | 24.61240005493164 | 24.349300384521484 |
| 104017528 | 13.023555729947981 | -72.9297926015843 | 1.0253703888551688 | 21.873899459838867 | 22.162900924682617 |
| 40535007 | 13.0257520380231 | -72.93018196050252 | 1.4502315057648554 | 25.07379913330078 | 24.825300216674805 |
| 61032592 | 13.02421440278526 | -72.93009558926467 | 1.5189506389476632 | 22.529699325561523 | 22.386699676513672 |
| 89324003 | 13.02272444537175 | -72.92990471713435 | 1.0160493629950065 | 24.373300552368164 | 24.10099983215332 |
| 101057597 | 13.026053482371552 | -72.93037135486455 | 1.0277777910232544 | 25.468599319458008 | 25.322099685668945 |
| 80208700 | 13.024679741921718 | -72.93025288056081 | 0.9643518814334162 | 25.877300262451172 | 25.44540023803711 |
| 81158369 | 13.023183887830848 | -72.93007760854097 | 1.0523456850169615 | 22.43899917602539 | 22.407800674438477 |
| ... | ... | ... | ... | ... | ... |
| 31291564 | 13.216750037744728 | -73.06546128146896 | 1.0450463052149173 | 22.649200439453125 | 22.56599998474121 |
| 1950079 | 13.183951460664364 | -73.06562609839335 | 1.1067592435412936 | 24.14150047302246 | 23.97480010986328 |
| 34415391 | 13.22645919127738 | -73.06533939164694 | 1.0614197379277075 | 20.332799911499023 | 19.793099403381348 |
| 105025275 | 13.297977267668726 | -73.06336678542297 | 1.120401201424775 | 24.06730079650879 | 23.66629981994629 |
| 2700817 | 13.305409173846048 | -73.06305483464266 | 1.3234876556161008 | 23.963300704956055 | 23.725099563598633 |
| 27490802 | 13.208846313783647 | -73.06554049347696 | 1.0196296705140009 | 25.318199157714844 | 25.075700759887695 |
| 79314878 | 13.129966526893558 | -73.06504859318578 | 1.01114195070149 | 24.7810001373291 | 24.54400062561035 |
| 78785853 | 13.245981289154832 | -73.06499094023793 | 1.143641972247465 | 24.362300872802734 | 24.17329978942871 |
| 59974181 | 13.989040704560631 | -72.82224209119366 | 0.8882870000821572 | 24.282699584960938 | 23.828100204467773 |
Next, plot the color-magnitude diagram for the ~100k points retrieved from the database. This uses kernel density estimate for the crowded plot. As a preview for the demo, we are only working with 1/25th of the data so that the notebook executes quickly. You can switch the commented-out lines to call gaussian_kde for the full plot, which can take a few minutes to complete.
f555w = HSC_results['a_f555w']
f814w = HSC_results['a_f814w']
VminusI = f555w-f814w
CI = HSC_results['ci']
w = np.where((CI>0.9) & (CI<1.6) & (VminusI > -1.5) & (VminusI < 1.5))
print(len(w[0]),"points remaining after CI and V-I filtering")
# Calculate the point density
x = np.array(VminusI[w])
y = np.array(f555w[w])
xy = np.vstack([x,y])
t0 = time.time()
z = gaussian_kde(xy[:, ::25])(xy) #to do the KDE on only the full dataset, comment out this and uncomment below:
#z = gaussian_kde(xy)(xy) #uncomment this line to do the KDE on the full dataset
print("kde took {:.1f} sec".format(time.time()-t0))
87914 points remaining after CI and V-I filtering
kde took 2.8 sec
# Sort the points by density, so that the densest points are plotted last
idx = z.argsort()
x, y, z = x[idx], y[idx], z[idx]
plt.rcParams.update({'font.size': 16})
plt.figure(1,(12,10))
plt.scatter(x, y, c=z, s=2, cmap='plasma')
plt.autoscale(tight=True)
plt.xlabel('V-I')
plt.ylabel('V')
plt.gca().invert_yaxis()
plt.colorbar()
plt.text(.17,.93,'{:d} stars in SMC'.format(len(x)),
horizontalalignment='center',
transform=plt.gca().transAxes)
#plt.savefig("smc_colormag.png")
Text(0.17, 0.93, '87914 stars in SMC')
Additional Resources#
Table Access Protocol#
IVOA standard for RESTful web service access to tabular data
http://www.ivoa.net/documents/TAP/
Hubble Source Catalog v3#
Catalog created at MAST by combining the tens of thousands of visit-based source lists in the Hubble Legacy Archive (HLA) into a single master catalog.
https://archive.stsci.edu/hst/hsc/
Astronomical Query Data Language (2.0)#
IVOA standard for querying astronomical data in tabular format, with geometric search support
http://www.ivoa.net/documents/latest/ADQL.html
PyVO#
an affiliated package for astropy
find and retrieve astronomical data available from archives that support standard IVOA virtual observatory service protocols.
https://pyvo.readthedocs.io/en/latest/index.html
About this Notebook#
Authors: Rick White & Theresa Dower, STScI Archive Scientist & Software Engineer. Additional edits by Thomas Dutkiewicz.
Last Updated: May 2024
