wiki:DC3bWebDataAccess
Last modified 7 years ago Last modified on 02/24/2012 11:51:24 PM

Accessing DC3b Data Via the Web

The raw input data and processed data products from DC3b processing is available over the Web. Access by most users is expected to be through the Gator User Interface at https://osiris.ipac.caltech.edu. This page specifically describes the URL-based web service interface to the data. This is document is not intended for end users but rather for developers that wish to programmatically access the data.

HTTP Interface

The server lsst-web.ncsa.illinois.edu provides the access to the data. The web service is in the form of a HTTP GET request in which the identifier for the desired data item is embedded in the URL. The data is delivered synchronously: that is, the service will block until the data is ready for delivery, and the response is an FITS image file, a tar file, or other data item. A returned MIME-type indicates the type. Collections of data can also be retrieved as tar files.

The entire namespace can be explored by pointing your favorite browser to:

http://lsst-web.ncsa.illinois.edu/lsstdata

To request a data product, a client accesses a URL of the following form:

http://lsst-web.ncsa.illinois.edu/lsstdata/dc3product/pt1.2/3000<id>

where <id> is the canonical identifier as described on the wiki:DC3bDataOrganization page, starting with the collection name obs (for raw and calibration input data) or datarel (for processed output products).

Any directory in the entire hierarchy can be downloaded as a single tar file.

When using the wget utility, specifying the option "--content-disposition" is recommended. That will tell wget to create the output file using a more descriptive filename than it otherwise would.

Examples

ImSim raw data

The ImSim raw data is at:

http://lsst-web.ncsa.illinois.edu/lsstdata/dc3product/pt1.2/3000/obs/ImSim/raw

Raw ImSim data as individual amplifier (channel) images:

http://lsst-web.ncsa.illinois.edu/lsstdata/dc3product/pt1.2/3000/obs/ImSim/raw/v886053981-fy/E001/R34/S22/imsim_886053981_R34_S22_C04_E001.fits.gz

All of these are valid URLs for downloading the entire subdirectory specified as a single tar file:

http://lsst-web.ncsa.illinois.edu/lsstdata/dc3product/pt1.2/3000/obs/ImSim/raw/v886053981-fy/E001/R34/S22.tar
http://lsst-web.ncsa.illinois.edu/lsstdata/dc3product/pt1.2/3000/obs/ImSim/raw/v886053981-fy/E001/R34.tar
http://lsst-web.ncsa.illinois.edu/lsstdata/dc3product/pt1.2/3000/obs/ImSim/raw/v886053981-fy/E001.tar
http://lsst-web.ncsa.illinois.edu/lsstdata/dc3product/pt1.2/3000/obs/ImSim/raw/v886053981-fy.tar

ImSim calibration data

The calibration data is at:

http://lsst-web.ncsa.illinois.edu/lsstdata/dc3product/pt1.2/3000/obs/ImSim/bias
http://lsst-web.ncsa.illinois.edu/lsstdata/dc3product/pt1.2/3000/obs/ImSim/dark
http://lsst-web.ncsa.illinois.edu/lsstdata/dc3product/pt1.2/3000/obs/ImSim/flat

Examples:

http://lsst-web.ncsa.illinois.edu/lsstdata/dc3product/pt1.2/3000/obs/ImSim/bias/v0/R01.tar
http://lsst-web.ncsa.illinois.edu/lsstdata/dc3product/pt1.2/3000/obs/ImSim/bias/v0/R01/S10/imsim_0_R01_S10_C06.fits
http://lsst-web.ncsa.illinois.edu/lsstdata/dc3product/pt1.2/3000/obs/ImSim/dark/v1/R43.tar
http://lsst-web.ncsa.illinois.edu/lsstdata/dc3product/pt1.2/3000/obs/ImSim/dark/v1/R43/S02/imsim_1_R43_S02_C10.fits
http://lsst-web.ncsa.illinois.edu/lsstdata/dc3product/pt1.2/3000/obs/ImSim/flat/v2-fg/R22.tar
http://lsst-web.ncsa.illinois.edu/lsstdata/dc3product/pt1.2/3000/obs/ImSim/flat/v2-fg/R22/S11/imsim_2_R22_S11_C04.fits

ImSim output data

The calibrated ImSim data is at:

http://lsst-web.ncsa.illinois.edu/lsstdata/dc3product/pt1.2/3000/datarel/ImSim/calexp
http://lsst-web.ncsa.illinois.edu/lsstdata/dc3product/pt1.2/3000/datarel/ImSim/apCorr
http://lsst-web.ncsa.illinois.edu/lsstdata/dc3product/pt1.2/3000/datarel/ImSim/psf

Examples:

http://lsst-web.ncsa.illinois.edu/lsstdata/dc3product/pt1.2/3000/datarel/ImSim/apCorr/v887676270-fg/R21/S02.pickle
http://lsst-web.ncsa.illinois.edu/lsstdata/dc3product/pt1.2/3000/datarel/ImSim/apCorr/v887676270-fg.tar
http://lsst-web.ncsa.illinois.edu/lsstdata/dc3product/pt1.2/3000/datarel/ImSim/psf/v887676270-fg/R21/S02.boost
http://lsst-web.ncsa.illinois.edu/lsstdata/dc3product/pt1.2/3000/datarel/ImSim/psf/v887676270-fg.tar

http://lsst-web.ncsa.illinois.edu/lsstdata/dc3product/pt1.2/3000/datarel/ImSim/calexp/v887676270-fg/R21/S02.fits
http://lsst-web.ncsa.illinois.edu/lsstdata/dc3product/pt1.2/3000/datarel/ImSim/calexp/v887676270-fg/R21.tar
http://lsst-web.ncsa.illinois.edu/lsstdata/dc3product/pt1.2/3000/datarel/ImSim/calexp/v887676270-fg.tar

Note that the last one will take up to 20 minutes (30.8GB) using a browser, less if using wget.