Downloading COS Data

Learning Goals

This Notebook is designed to walk the user (you) through: Downloading existing Cosmic Origins Spectrograph (COS) data from the online archive

0. Introduction

- 0.1. A one-cell summary of this Notebook's key points

1. Using the web browser interface

- 1.1. The Classic HST Web Search

- 1.2. Searching for a Series of Observations on the Classic Web Search

- 1.3. The MAST Portal

- 1.4. Searching for a Series of Observations on the MAST Portal

2. Using the Python module Astroquery

- 2.1. Searching for a single source with Astroquery

- 2.2. Narrowing Search with Observational Parameters

- 2.3. Choosing and Downloading Data Products

- 2.4. Using astroquery to find data on a series of sources

Choosing how to access the data

This Notebook explains three methods of accessing COS data hosted by the STScI Mikulski Archive for Space Telescopes (MAST). You may read through all three, or you may wish to focus on a particular method which best suits your needs. Please use the table below to determine which section on which to focus.

The Classic HST Search (Web Interface) The MAST Portal (Web Interface) The Astroquery (Python Interface)
- User-friendly point-and-click searching - Very user-friendly point-and-click searching - Requires a bit of Python experience
- Advanced mission-specific search parameters, including: central wavelength, detector, etc. - Lacks some mission-specific search parameters - Allows for programmatic searching and downloads
- Can be difficult to download the data if not on the STScI network - Easy to download selected data - Best for large datasets
Use this method if... ...You're unfamiliar with Python and need to search for data by cenwave ...You're exploring the data and you don't need to search by cenwave ...You know Python and have an idea of what data you're looking for, or you have a lot of data
Described in... Section 1.1 Section 1.3 Section 2.1

Note that these are only recommendations, and you may prefer another option. For most purposes, the writer of this tutorial recommends the Astroquery Python interface, unless you are not at all comfortable using python or you are primarily interested in exploring the available data.

0. Introduction

The Cosmic Origins Spectrograph (COS) is an ultraviolet spectrograph on-board the Hubble Space Telescope(HST) with capabilities in the near ultraviolet (NUV) and far ultraviolet (FUV).

This tutorial aims to prepare you to access the existing COS data of your choice by walking you through downloading a processed spectrum, as well as various calibration files obtained with COS.

0.1. A one-cell summary of this Notebook's key points:

While the rest of this Notebook will walk you through each step and decision made when downloading COS data, the following code cell serves as a summary for the Notebook. It contains the key material condensed into a single code cell, without much explanation. If this is all the help you need, great! If you still have questions, read on!

Now, returning to our more detailed walkthrough...

We will define a few directories in which to place our data.

And to create new directories, we'll import pathlib.Path:

1. Downloading the data through the browser interface

One can search for COS data from both a browser-based Graphical User Interface (gui) and a Python interface. This Section (1) will examine two web interfaces. Section 2 will explain the Python interface.

Note, there are other, more specialized ways to query the mast API not discussed in this Notebook. An in-depth MAST API tutorial can be found here.

A browser gui for searching specifically through HST archival data can be found here. We will be discussing this HST search in the section below. As of September, 2021, two other portals also allow access to the same data:

The search page of the HST interface is laid out as in fig. 1.1:

Fig 1.1

where here we have indicated we would like to find all archival science data from the COS far-ultraviolet (FUV) configuration, taken with any grating while looking at Quasi-Stellar Objects (QSO) within a 3 arcminute radius of (1hr:37':40", +33d 09m 32s). The output columns we have selected to see are visible in the bottom left of Fig 1.1.

Note that if you have a list of coordinates, Observation ID(s), etc. for a series of targets you can click on the "File Upload Form" and attach your list of OBSIDs or identifying features. Then specify which type of data your list contains using the "File Contents" drop-down menu.

Figure 1.2 shows the results of our search shown in Fig 1.1.

Fig 1.2

We now choose our dataset. We rather arbitrarily select LCXV13050 because of its long exposure time, taken under an observing program described as:

"Project AMIGA: Mapping the Circumgalactic Medium of Andromeda"

This is a Quasar known as 3C48, one of the first quasars discovered.

Clicking on the dataset, we are taken to a page displaying a preview spectrum (Fig 1.3).

Fig 1.3

We now return to the search page and enter in LCXV13050 under "Dataset" with no other parameters set. Clicking "search", now we see a single-rowed table with just our dataset, and the option to download datasets. We mark the row we wish to download and click "Submit marked data for retrieval from STDADS". See Fig 1.4.

Fig 1.4

Now we see a page like in Fig 1.5, where we can either sign in with STScI credentials, or simply provide our email to proceed without credentials. Generally, you may proceed anonymously, unless you are retrieving proprietary data to which you have access. Next, make sure to select "Deliver the data to the Archive staging area". Click "Send Retrieval Request to ST-DADS" and you will receive an email with instructions on downloading the data.

Fig 1.5

Now the data is "staged" on a MAST server, and you need to download it to your local computer.

Downloading the staged data

We demonstrate three methods of downloading your staged data:

  1. If your terminal supports it, you may use the wget tool.
  2. However if that does not work, we recommend using a secure ftp client application.
  3. Finally, if you would instead like to download staged data programmatically, you may use the Python ftplib package, as described here in STScI's documentation of the MAST FTP Service. For your convenience, we have built the download_anonymous_staged_data function below, which will download anonymously staged data via ftps.

Downloading the staged data with wget

If you are connected to the STScI network, either in-person or via a virtual private network (VPN), you should use the wget command as in the example below:

wget -r --ftp-user=anonymous --ask-password ftps://archive.stsci.edu/stage/anonymous/anonymous<directory_number> --directory-prefix=<data_dir>

where directory_number is the number at the end of the anonymous path specified in the email you received from MAST and data_dir is the local directory where you want the downloaded data. You will be prompted for a password. Type in the email address you used, then press enter/return.

Now all the data will be downloaded into a subdirectory of data_dir: "./archive.stsci.edu/stage/anonymous/anonymous<directory_number>/"

Downloading the staged data with a secure ftp client application (CyberDuck)

CyberDuck is an application which allows you to securely access data stored on another machine using ftps. To download your staged data using Cyberduck, first download the Cyberduck application (free, with a recommended donation). Next, open a new browser window (Safari, Firefox, and Google Chrome have all been shown to work,) and type in the following web address: ftps://archive.stsci.edu/stage/anonymous<directory_number>, where directory_number is the number at the end of the anonymous path specified in the email you received from MAST. For example, if the email specifies:

"The data can be found in the directory... /stage/anonymous/anonymous42822"

then this number is 42822

Your browser will attempt to redirect to the CyberDuck application. Allow it to "Open CyberDuck.app", and CyberDuck should open a finder window displaying your files. Select whichever files you want to download by highlighting them (command-click or control-click) then right click one of the highlighted files, and select "Download To". This will bring up a file browser allowing you to save the selected files to wherever you wish on your local computer.

Downloading the staged data with ftps

To download anonymously staged data programmatically with ftps, you may run the download_anonymous_staged_data function as shown here:

download_anonymous_staged_data(email_used="my_email@stsci.edu", directory_number=80552, outdir="./here_is_where_I_want_the_data")

Which results in:

Downloading lcxv13050_x1dsum1.fits
  Done
...
...
Downloading lcxv13gxq_flt_b.fits
  Done

Well Done making it this far!

Attempt the exercise below for some extra practice.

Exercise 1: Searching the archive for TRAPPIST-1 data

TRAPPIST-1 is a cool red dwarf with a multiple-exoplanet system.

What is the dataset ID, and how long was the exposure?

Place your answer in the cell below.

Now let's try using the web interface's file upload form to search for a series of observations by their dataset IDs. We're going to look for three observations of the same object, the white dwarf WD1057+719, taken with three different COS gratings. Two are in the FUV and one in the NUV. The dataset IDs are

So that we have an example list of datasets to input to the web search, we make a comma-separated-value txt file with these three obs_ids, and save it as obsId_list.txt.

Then we link to this file under the Local File Name browse menu on the file upload form. We must set the File Contents term to Data ID, as that is the identifier we have provided in our file, and we change the delimiter to a comma. Because we are searching by Dataset ID, we don't need to specify any additional parameters to narrow down the data.

Fig 1.6

We now can access all the datasets, as shown in Fig. 1.7:

Fig 1.7

Now, to download all of the relevant files, we can check the mark box for all of them, and again hit "Submit marked data for retrieval from STDADS". This time, we want to retrieve all the calibration files associated with each dataset, so we check the following boxes:

(See Fig. 1.8)

Fig 1.8

The procedure from here is the same described above in Section 1.1. Now, when we download the staged data, we obtain multiple subdirectories with each dataset separated.

1.3. The MAST Portal

STScI hosts another web-based gui for accessing data, the MAST Portal. This is a newer interface which hosts data from across many missions and allows the user to visualize the target in survey images, take quick looks at spectra or lightcurves, and manage multiple search tabs at once. Additionally, it handles downloads in a slightly more beginner-friendly manner than the current implementation of the Classic HST Search. This guide will only cover the basics of accessing COS data through the MAST Portal; you can find more in-depth documentation in the form of helpful video guides on the MAST YouTube Channel.

Let's find the same data we found in Section 1.1, on the QSO 3C48:

Navigate to the MAST Portal at https://mast.stsci.edu/portal/Mashup/Clients/Mast/Portal.html, and you will be greeted by a screen where the top looks like Fig. 1.9.

Fig 1.9

Click on "Advanced Search" (boxed in red in Fig. 1.9). This will open up a new search tab, as shown in Fig. 1.10:

Fig 1.10

Fig 1.10 (above) shows the default search fields which appear. Depending on what you are looking for, these may or may not be the most helpful search fields. By unchecking some of the fields which we are not interested in searching by right now (boxed in green), and then entering the parameter values by which to narrow the search into each parameter's box, we generate Fig. 1.11 (below). One of the six fields (Mission) by which we are narrowing is boxed in a dashed blue line. The list of applied filters is boxed in red. A dashed pink box at the top left indicates that 2 records were found matching all of these parameters. To its left is an orange box around the "Search" button to press to bring up the list of results

Here we are searching by:

Search Parameter Value
Mission HST
Instrument COS/FUV
Filters G160M
Target Name 3C48
Observation ID LCXV* (the star is a "wild card" value, so the search will find any file whose obs_id begins with LCXV)
Product Type spectrum

Fig 1.11

Click the "Search" button (boxed in orange), and you will be brought to a page resembling Fig. 1.12.

Fig 1.12

Above, in Fig 1.12:

1.4. Searching for a Series of Observations on the MAST Portal

To download multiple datasets: The MAST portal acts a bit like an online shopping website, where you add your data products to the checkout cart/basket, then open up your cart to checkout and download the files.

Using the checkboxes, mark all the datasets you wish to download (in this case, we'll download both LCXV13040 and LCXV13050). Then, click the "Add data products to Download Basket" button (circled in a dashed-purple line), which will take you to a "Download Basket" screen resembling Fig 1.13:

Fig 1.13

Each dataset contains many files, most of which are calibration files or intermediate processing files. You may or may not want some of these intermediate files in addition to the final product file. In the leftmost "Filters" section of the Download Basket page, you can narrow which files will be downloaded (boxed in red). By default, only the minimum recommended products (mrp) will be selected. In the case of most COS data, this will be the final spectrum x1dsum file and association asn file for each dataset. The mrp files for the first dataset (LCXV13040) are highlighted in yellow. These two mrp filetypes are fine for our purposes here; however if you want to download files associated with specific exposures, or any calibration files or intermediate files, you can select those you wish to download with the checkboxes in the file tree system (boxed in dashed-green).

For this tutorial, we simply select "Minimum Recommended Products" at the top left. With this box checked, all of the folders representing individual exposures are no longer visible. Check the box labelled "HST" to select all files included by the filters, and click the "Download Selected Items" button at the top right (dashed-black circle). This will bring up a small window asking you what format to download your files as. For datasets smaller than several Gigabytes, the Zip format will do fine. Click Download, and a pop-up window will try to open to download the files. If no download begins, make sure to enable this particular pop-up, or allow pop-ups on the MAST page.

Your files should now be downloaded as a compressed Zip folder. If you need help uncompressing the Zipped files, check out these links for: Windows and Mac. There are numerous ways to do this on Linux, however we have not vetted them.

2. The Python Package astroquery.mast

Another way to search for and download archived datasets is from within Python using the module astroquery.mast. We will import one of this module's key submodules: Observations.

Please note that the canonical source of information on this package is the astroquery docs - please look there for the most up-to-date instructions.

We will import the following packages:

2.1. Searching for a single source with Astroquery

There are many options for searching the archive with astroquery, but we will begin with a very general search using the coordinates we found for WD1057+719 in the last section to find the dataset with the longest exposure time using the COS/FUV mode through the G160M filter. We could also search by object name to have it resolved to a set of coordinates, with the function Observations.query_object(objectname = '3C48').

This command has generated a table of objects called "query_1". We can see what information we have on the objects in the table by printing its keys, and see how many objects are in the table with len(query_1).

2.2. Narrowing Search with Observational Parameters

Now we narrow down a bit with some additional parameters and sort by exposure time. The parameter limits we add to the search are:

Caution!

Please note that these queries are Astropy tables and do not always respond as expected for other data structures like Pandas DataFrames. For instance, the first way of filtering a table shown below is correct, but the second will consistently produce the wrong result. You must search and filter these tables by masking them, as in the first example below.

2.3. Choosing and Downloading Data Products

Now we can choose and download our data products from the archive dataset.

We will first generate a list of data products in the dataset: product_list. This will generate a large list, but we will only show the first 10 values.

Now, we will download just the minimum recommended products (mrp) which are the fully calibrated spectrum (denoted by the suffix _x1d or here x1dsum) and the association file (denoted by the suffix _asn). We do this by setting the parameter mrp_only to True. The association file contains no data, but rather the metadata explaining which exposures produced the x1dsum dataset. The x1dsum file is the final product summed across all of the fixed pattern noise positionsGratingOffsetPositions(FP-POS)) (FP-POS). The x1d and x1dsum<n> files are intermediate spectra. Much more information can be found in the COS Instrument Handbook.

We would set mrp_only to False, if we wanted to download all the data from the observation, including:

However, use caution with downloading all files, as in this case, setting mrp_only to False results in the transfer of 7 Gigabytes of data, which can take a long time to transfer and eat away at your computer's storage! In general, only download the files you need. On the other hand, often researchers will download only the raw data, so that they can process it for themselves. Since here we only need the final x1dsum and asn files, we only need to download 2 Megabytes.

Exercise 2: Download the raw counts data on TRAPPIST-1

In the previous exercise, we found an observation COS took on TRAPPIST-1 system. In case you skipped Exercise 1, the observation's Dataset ID is LDLM40010.

Use Astroquery.mast to download the raw TIME-TAG data, rather than the x1d spectra files. See the COS Data Handbook Ch. 2 for details on TIME-TAG data files. Make sure to get the data from both segments of the FUV detector (i.e. both RAWTAG_A and RAWTAG_B files). If you do this correctly, there should be five data files for each detector segment.

Note that some of the obs_id may appear in the table as slightly different, i.e.: ldlm40alq and ldlm40axq, rather than ldlm40010. The main obs_id they fall under is still ldlm40010, and this will still work as a search term. They are linked together by the association file described here in section 2.3.

2.4. Using astroquery to find data on a series of sources

In this case, we'll look for COS data around several bright globular clusters:

We will first write a comma-separated-value (csv) file objectname_list.csv listing these sources by their common name. This is a bit redundant here, as we will immediately read back in what we have written; however it is done here to deliberately teach both sides of the writing/reading process, and as many users will find themselves with a csv sourcelist they must search.

Excellent! You've now done the hardest part - finding and downloading the right data. From here, it's generally straightforward to read in and plot the spectrum. We recommend you look into our tutorial on Viewing a COS Spectrum.

Congratulations! You finished this Notebook!

There are more COS data walkthrough Notebooks on different topics. You can find them here.


About this Notebook

Author: Nat Kerman nkerman@stsci.edu

Updated On: 2022-02-25

This tutorial was generated to be in compliance with the STScI style guides and would like to cite the Jupyter guide in particular.

Citations

If you use astropy, matplotlib, astroquery, or numpy for published research, please cite the authors. Follow these links for more information about citations:


Top of Page Space Telescope Logo







Exercise Solutions:

Note, that for many of these, there are multiple ways to get an answer.

We will import: