2 Data Discovery (Catalog Services API)

Catalog Web Services

The catalog web services interact with inventory databases for each dataset. This provides a uniform interface for getting access details to each dataset without needing to know the technical details of the database for each dataset.

These web services are used by the Next engines that handle dataset processing, but could be useful to any system building orders outside the Next system.

There are a few guidelines that govern the design of the API. Knowing these will make use of the API more intuitive. Not all datasets currently follow all aspects of these guidelines at present, but adherence will grow over time.

1) All URLs contain a base 'rest/dataset/catalog'.
2) The endpoints built upon this add a singular resource type.
   catalog/file   -- queried list of file objects
   catalog/survey -- queried list of survey objects
3) Cases where lists of IDs or names are returned use endpoints
   that specifically identify the type of information returned.
   In some cases, these return info limited by query criteria.
   In other cases, they list available values.
   catalog/filename  -- queried list of file names (e.g.: dscovr)
   catalog/surveyid  -- queried list of survey IDs
   catalog/datatype  -- list of available datatypes
   catalog/platform  -- list of available platforms
4) 'A catalog/extents' endpoint returns extent information that, to
   the extent possible, reuses a set of labels describing a
   query's extent along various dimensions.
   count   -- number of files (if no other counts present)
   size    -- size of order (if no other sizes present)
   fileSize, pointSize -- when file and point sizes are present
   fileCount, pointCount -- when file and point counts are present
   geometry  -- geographical extent
   earliestTime, latestTime -- data time range

Queries are typically of two forms. An 'available-value' query might simply list all available values that a query parameter might take. An 'inventory-query' will return a list of data items that conform with specified filter criteria. The 'available-value' queries might be used by a client to display available selections for particular filter criteria.

Filter criteria are used to narrow one's view of data within NCEI holdings down to the point where enough information is known to make an order. While the particular criteria supported vary from dataset to dataset, most at least accept criteria that limit the time or spatial location of interest. These specifics are detailed by dataset below.

2.1 Archive Catalog Service

Search for Archive Data

The Archive catalog is a generic catalog that accesses files associated with any dataset through use of generic tracking IDs. No scientific criteria, such as geometry or times are accepted. Searching for Archive catalog/metadata entails issuing a GET using a URL with this pattern:

https://www.ngdc.noaa.gov/next-catalogs/rest/archive/catalog/file<param-list>

This pattern ends with "rest/archive/catalog/file", followed by a parameter list of the form "?trackingIds=value,value...".

Tracking IDs consist of arbitrary hexadecimal strings of the following form: "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx", where each 'x' is a lowercase hexadeximal character (0-9a-f).

Examples:

https://www.ngdc.noaa.gov/next-catalogs/rest/archive/catalog/file?trackingIds=7df058b8-4409-4c33-8c42-943fa14d3cdb,20d66032-224f-4dd3-bb0a-f6c409196ca9

{"count":2,
"items":[
	{"dirPath": "satellite/dscovr/data/2015/12", "size": "79303707", "trackingId": "7df058b8-4409-4c33-8c42-943fa14d3cdb", "name": "it_vc0_dscovr_s20151229000000_e20151229235959_p20151229003000_emb.nc.gz", "dateArchived": "2014-03-06T00:00:00"},
	{"dirPath": "satellite/dscovr/data/2015/12", "size": "33485123", "trackingId": "20d66032-224f-4dd3-bb0a-f6c409196ca9", "name": "ot_mg1_dscovr_s20151229000000_e20151229235959_p20151229003000_pub.nc.gz", "dateArchived": "2014-03-06T00:00:00"}]
}

An example of an empty result:

{"count":0,"items":[]}

Catalog requests have no limitations regarding the number of files returned. Actual orders are limited in total size.

Get an Estimate of the Size/Extent of a DSCOVR Request

An 'extents' query provides the total size of the result set associated with a particular Archive query.

To query Archive extents, issue a GET using this URL pattern:

https://www.ngdc.noaa.gov/next-catalogs/rest/archive/catalog/extents<param-list>

This pattern ends with the string "rest/archive/catalog/extents", followed by a parameter list of the form: "?trackingIds=value,value...". The parameter list is identical to that used to make a catalog search.

Example:

https://www.ngdc.noaa.gov/next-catalogs/rest/archive/catalog/extents?trackingIds=7df058b8-4409-4c33-8c42-943fa14d3cdb,20d66032-224f-4dd3-bb0a-f6c409196ca9

{"count": 2, "size":112788830}

2.2 DMSP Catalog Service

Search for DMSP Data

Obtaining DMSP catalog/metadata entails issuing a GET using a URL with this pattern:

https://www.ngdc.noaa.gov/next-catalogs/rest/dmsp/catalog/file<param-list>

This pattern ends with "rest/dmsp/catalog/file", followed by a parameter list of the form "?name1=value&name2=value...".

The DMSP catalog supports the following values

Notes:

The satellite parameter is case-sensitive, i.e. f17 and F17 are not equivalent.

The orbitStartTime and orbitEndTime define a time range, and only results within that range will be returned. Both are required in conjunction with one another. Dates must be written in ISO 8601 format (e.g. yyyy-MM-dd).

The geometry is specified in the format longitude,latitude using decimal degrees.

Examples:

https://www.ngdc.noaa.gov/next-catalogs/rest/dmsp/catalog/file?satellite=F17&orbitStartTime=2010-12-01&orbitEndTime=2010-12-02

{"count":14,
"items":[
	{"filePath":"/stornext/ngdc/archive/satellite/dmsp/processed/data/F17/2011/ols/F17201106060008.OIS.gz", "orbitTime":"2010-12-01T01:00:00Z","size":"1000000"},
	{"filePath":"/stornext/ngdc/archive/satellite/dmsp/processed/data/F17/2011/ols/F17201106060150.OIS.gz","orbitTime":"2010-12-01T03:00:00Z","size":"1000000"},
  …
	{"filePath":"/stornext/ngdc/archive/satellite/dmsp/processed/data/F17/2011/ols/F17201106062355.OIS.gz","orbitTime":"2010-12-02T21:00:00Z","size":"1000000"}]
}
    (deprecated attribute 'dataFile' may also be present)

Empty result

{"count":0,"items":[]}

Requests require filter parameters, an error is returned in the event of insufficient parameters.

{"message":"listing ionosphere granules requires satellite and geometry or satellite, orbitStartTime and orbitEndTime"}

Get an Estimate of the Size of a DMSP Request

An 'extents' query provides the temporal extent and total size of the result set associated with a particular DMSP query.

To query DMSP extents, issue a GET using this URL pattern:

https://www.ngdc.noaa.gov/next-catalogs/rest/dmsp/catalog/extents<param-list>

This pattern ends with the string "rest/dmsp/catalog/extents", followed by a parameter list of the form: "?name1-value&name2=value...". The parameter list is identical to that used to make a catalog search. At least one parameter must be specified.

Example:

https://www.ngdc.noaa.gov/next-catalogs/rest/dmsp/catalog/extents?satellite=F17

{"orbitStartTime":"2003-11-04T00:08:58Z","orbitEndTime":"2012-07-18T22:55:51Z","count":29099,"size":732300312576}

2.3 DSCOVR Catalog Service

Search for DSCOVR Data

The primary entity searched for within the DSCOVR dataset is a file that is associated with a particular time range. Searching for DSCOVR catalog/metadata entails issuing a GET using a URL with this pattern:

https://www.ngdc.noaa.gov/next-catalogs/rest/dscovr/catalog/file<param-list>

This pattern ends with "rest/dscovr/catalog/file", followed by a parameter list of the form "?name1=value&name2=value...".

The DSCOVR catalog supports the following values

At the present time, the following 'processEnvs' values are supported:

At the present time, the following 'dataTypes' values are supported:

Notes:

The satellite, processEnvs, dataTypes parameters are case sensitive, and are all lowercase. The satellite parameter is included primarily for future use just in case a client may eventually be requesting data from among multiple satellites.

The time parameters (dataStartTime, dataEndTime, processStartTime and processEndTime) define time ranges (in UTC time). Any data file whose start-to-end time range or processing time, respectively, intersect the specified range will be found by the query. Dates must be written in ISO 8601 format (e.g. yyyy-MM-dd T HH:mm Z).

Examples:

https://www.ngdc.noaa.gov/next-catalogs/rest/dscovr/catalog/file?processEnvs=oe&dataTypes=vc0,vc1,mg1&dataStartTime=2015-12-29T00:00Z&dataEndTime=2015-12-30T23:59Z

(Whitespace added for readability)

{"count":14,
"items":[
  {"id": 103, "satellite": "dscovr", "fileName": "oe_vc0_dscovr_s20151229000000_e20151229235959_p20151229003000.nc.gz",
              "processEnv": "oe", "dataType": "vc0", "dataTypeDescription": "Real-Time Observatory Telemetry",
              "processTime": "2015-12-30T01:00:00Z", "startTime": "2015-12-29T00:00:00Z", "endTime": "2015-12-29T23:59:59Z",
              "fileSize": 9999, "trackingId": "93d3f757-0082-41f0-a7b5-f3030a8b6b05"},
  {"id": 104, "satellite": "dscovr", "fileName": "oe_vc1_dscovr_s20151229000000_e20151229235959_p20151229003000.nc.gz",
              "processEnv": "oe", "dataType": "vc1", "dataTypeDescription": "Stored Observatory Telemetry",
              "processTime": "2015-12-30T01:00:00Z", "startTime": "2015-12-29T00:00:00Z", "endTime": "2015-12-29T23:59:59Z",
              "fileSize": 9999, "trackingId": "0044037b-8922-4ef2-98ba-b27a6b61799e"},
  …
  {"id": 105, "satellite": "dscovr", "fileName": "oe_mg1_dscovr_s20151229000000_e20151229235959_p20151229003000.nc.gz",
              "processEnv": "oe", "dataType": "mg1", "dataTypeDescription": "Magnetometer L1 data",
              "processTime": "2015-12-30T01:00:00Z", "startTime": "2015-12-29T00:00:00Z", "endTime": "2015-12-29T23:59:59Z",
              "fileSize": 9999, "trackingId": "73d9164b-db16-4b95-8606-d9ce72bcba5e"}]
}

An example of an empty result:

{"count":0,"items":[]}

Catalog requests have no limitations regarding the number of files returned. Actual orders are limited in total size.

Get a List of DSCOVR Filenames

A filename query supports the alternative of simply fetching a list of filenames matching query criteria, rather than fetching SatFile objects. Issue a GET request using a URL with this pattern:

https://www.ngdc.noaa.gov/next-catalogs/rest/dscovr/catalog/filename<param-list>

This pattern ends with "rest/dscovr/catalog/filename", followed by a parameter list of the form "?name1=value&name2=value...".

The same filter criteria used for the 'file' query are supported for this 'filename' query.

Examples:

https://www.ngdc.noaa.gov/next-catalogs/rest/dscovr/catalog/filename?processEnvs=oe&dataTypes=vc0,vc1,mg1&dataStartTime=2015-12-29T00:00Z&dataEndTime=2015-12-30T23:59Z

{"count":14,
"items":[
  "oe_vc0_dscovr_s20151229000000_e20151229235959_p20151229003000.nc.gz",
  "oe_vc1_dscovr_s20151229000000_e20151229235959_p20151229003000.nc.gz",
  …
  "oe_mg1_dscovr_s20151229000000_e20151229235959_p20151229003000.nc.gz"]
}

An example of an empty result:

{"count":0,"items":[]}

Get a List of Available DataTypes

A 'datatype' query provides a list of supported datatypes that are available within the DSCOVR dataset. Issue a GET request using a URL with this pattern:

https://www.ngdc.noaa.gov/next-catalogs/rest/dscovr/catalog/datatype<param-list>

This pattern ends with "rest/dscovr/catalog/datatype". This query supports no filter criteria.

Examples:

https://www.ngdc.noaa.gov/next-catalogs/rest/dscovr/catalog/datatype

{"count":15,
"items":[
  { "name:" "vc0", "description": "Real-Time Observatory Telemetry"},
  { "name:" "vc1", "description": "Stored Observatory Telemetry"},
  …
  { "name:" "att", "description": "Spacecraft Attitude"}
}

Get an Estimate of the Size/Extent of a DSCOVR Request

An 'extents' query provides the temporal extent and total size of the result set associated with a particular DSCOVR query. There is no geographical extent for this dataset.

To query DSCOVR extents, issue a GET using this URL pattern:

https://www.ngdc.noaa.gov/next-catalogs/rest/dscovr/catalog/extents<param-list>

This pattern ends with the string "rest/dscovr/catalog/extents", followed by a parameter list of the form: "?name1=value&name2=value...". The parameter list is identical to that used to make a catalog search.

Example:

https://www.ngdc.noaa.gov/next-catalogs/rest/dscovr/catalog/extents?processEnvs=oe&dataTypes=vc0,vc1,mg1&dataStartTime=2015-12-29T00:00Z&dataEndTime=2015-12-30T23:59Z

{"startTime":"2015-12-29T00:08:58Z","endTime":"2015-12-30T00:08:58Z","count":4,"size":7323003}

2.4 Ionosphere Catalog Service

Search for Ionosphere Data

Obtaining ionospheric catalog/metadata entails issuing a GET using a URL with this pattern:

https://www.ngdc.noaa.gov/next-catalogs/rest/ionosphere/catalog<param-list>

This pattern ends with the string "rest/ionosphere/catalog", followed by a parameter list of the form: "?name1-value&name2=value...". At least one parameter must be specified.

The Ionospheric catalog supports the following values

Notes:

The stationName parameter is case-sensitive.

The obsStartTime and obsEndTime define a time range, and only results within that range will be returned. Dates must be written in ISO 8601 format (e.g. yyyy-MM-dd).

Examples:

https://www.ngdc.noaa.gov/next-catalogs/rest/ionosphere/catalog?stationName=BC840&obsStartTime=2010-12-01&obsEndTime=2010-12-02

{ "items":[
    {"stationName":"BC840","dataFile":"BC840_2010335000005.MMM","contentType":"ionogram",
            "observationTime":"2010-12-01T00:00:05Z","path":"/nfs/MIDS/data/public/BC840/individual/2010/335/ionogram"},
    {"stationName":"BC840","dataFile":"BC840_2010335000005.SAO","contentType":"scaled",
            "observationTime":"2010-12-01T00:00:05Z","path":"/nfs/MIDS/data/public/BC840/individual/2010/335/scaled"},
    …
    {"stationName":"BC840","dataFile":"BC840_2010335234505_IO.PNG","contentType":"image",
            "observationTime":"2010-12-01T23:45:05Z","path":"/nfs/MIDS/data/public/BC840/individual/2010/335/image"}],
 "count":576
}

Empty result

{"items":[],"count":0}

Requests require filter parameters, an error is returned in the event of insufficient parameters.

{"message":"at least one filter criteria is required: stationName, geometry, obsStartTime, obsEndTime"}

Get an Estimate of the Size of an Ionospheric Data Request

An 'extents' query provides the temporal extent and total download size (in bytes) associated with a particular Ionosphere query. Geographical extent is not available for this dataset.

To query Ionospheric extents, issue a GET using this URL pattern:

https://www.ngdc.noaa.gov/next-catalogs/rest/ionosphere/catalog/extents<param-list>

This pattern ends with the string "rest/ionosphere/catalog/extents", followed by a parameter list of the form: "?name1-value&name2=value...". The parameter list is identical to that used to make a catalog selection. At least one parameter must be specified.

Example:

https://www.ngdc.noaa.gov/next-catalogs/rest/ionosphere/catalog/extents?obsStartTime=2010-12-01&obsEndTime=2010-12-02&stationName=BC840

which will result in a response like this:

{"obsStartTime":"2010-12-01T00:00:05Z","obsEndTime":"2010-12-01T23:45:05Z","size":127692484,"count":576}

With ionosphere, some filter criteria are required because the result is too expensive to calculate when sizing the entire collection. Therefore, a request for extents with no filter criteria like:

https://www.ngdc.noaa.gov/next-catalogs/rest/ionosphere/catalog/extents

will result in the message:

{"message":"invalid request: obsStartTime, obsEndTime and one of geometry or stationName are required"}

2.5 Hydrographic Sounding Catalog Service

List Available File Categories

Hydrographic soundings files are categorized. File statistics (counts and sizes) are provided in terms of these categories. Queries and orders can filter on these categories, giving some degree of control over the content of an order. The most basic query provided for sounding datasets is the ability to get a list of these categories. To obtain a list of all file categories issue a GET request having the following URL pattern:

https://www.ngdc.noaa.gov/next-catalogs/rest/sounding/catalog/category

This pattern ends with "rest/sounding/catalog/category". It accepts no parameters.

JSON Response: The response JSON contains a list of names and a total count.

{ "items":["Bathymetry","Documentation","Metadata", "Point Data"...],
  "count":7 }

List Available Surveys (by name)

Each survey is identified by a survey name. To obtain a list of all sounding survey names, or those corresponding to specified filtering parameters, issue a GET or POST request having the following URL pattern:

https://www.ngdc.noaa.gov/next-catalogs/rest/sounding/catalog/surveyid

This pattern ends with "rest/sounding/catalog/surveyid", followed by an optional parameter list of the form: "?name1-value&name2=value...".

In this format, the valid parameters are

JSON Response: The response JSON contains a list of names and a total count.

{ "items":["B00001","B00002","B00003"...], "count":12855 }

List Available Platforms (by name)

Each vessel that supports surveys is identified by a platform name. To obtain a list of all sounding platform names, or those corresponding to specified filtering parameters, issue a GET or POST request having the following URL pattern:

https://www.ngdc.noaa.gov/next-catalogs/rest/sounding/catalog/platform

This pattern ends with "rest/sounding/catalog/platform", followed by an optional parameter list of the form: "?name1-value&name2=value...".

In this format, the valid parameters are

JSON Response: The response JSON contains a list of names and a total count.

{ "items":["BRISTOL ENDEAVOR","Barge746","Beachcraft King Air 200"...],
  "count":258 }

Search for Soundings Surveys

To select Soundings bathymetry surveys, issue a GET or POST using this URL pattern:

https://www.ngdc.noaa.gov/next-catalogs/rest/sounding/catalog/survey<param-list>

This pattern ends with the string "rest/sounding/catalog/survey", followed by a parameter list of the form: "?name1-value&name2=value...". If the request involves many (>980) parameter values, then use of a POST is required.

In this format, the valid parameters are

Notes:

The platforms and surveys are case-sensitive, and must match precisely.

The startYear and endYear define a time range, and only data recorded within that range are selected.

The max and offset parameters are designed to support a paged response.

Note that specified format affects the size estimates returned with responses.

A typical response looks like the example shown below. Notice that the count returned indicates the total number of results available, while the items list typically only holds one page worth of results.

The JSON result of https://www.ngdc.noaa.gov/next-catalogs/rest/sounding/catalog/survey?surveys=B00026 will look like this:

{
  "count": 1,
  "items": [
    {
      "surveyId": "B00026",
      "platform": "NOAA Ship SURVEYOR (S132)",
      "startTime": "1985-09-01T00:00:00Z",
      "endTime": "1985-09-02T00:00:00Z",
      "geometry": "-125.5164,44.4718,-125.2659,45.0181",
      "modifyTime": "2014-09-01T15:02:22Z",
      "approvalTime": null,
      "surveyYear": 1985,
      "locality": "Yaquina",
      "sublocality": "54 NM West-Southwest of Yaquina Head",
      "hblockDir": "B00001-B02000",
      "fileStatsByCat": [
        {
          "name": "Bathymetry",
          "count": 3,
          "size": 427533
        },
        {
          "name": "Metadata",
          "count": 1,
          "size": 71680
        },
        {
          "name": "Point Data",
          "count": 1,
          "size": 710176
        }
      ],
      "fileCount": 5,
      "fileSizeSum": 1209389,
      "pointCount": 22193
    }
  ]
}

The example above has been shortened and reformatted for readability. The actual response will be returned as a long single line.

If there are no items selected, the result will look like this:

{"items":[],"count":0}

It is likely that if you specify no parameters that the response will be larger than can be ordered. Examination of the results may provide an indication of how the query can be tightened up to provide a more reasonably sized result.

Get an Estimate of the Size of a Soundings Data Request

An 'extents' query provides the geographical and temporal extent of the results, and file count and total download size (in bytes) associated with a particular Soundings query. A fileStatsByCat array lists file counts and sizes by category.

To query Soundings data extents, GET from this URL:

https://www.ngdc.noaa.gov/next-catalogs/rest/sounding/catalog/extents

This pattern ends with the string "rest/sounding/catalog/extents", followed by a parameter list of the form: "?name1-value&name2=value...". The parameter list is identical to that used to make a survey catalog query.

The extents result will look like this:

{
  "startYear": 1979,
  "endYear": 1991,
  "geometry": "-158.0151,18.4038,-121.8892,59.6266",
  "count": 556,
  "size": 409199477,
  "pointCount": 3625659,
  "fileStatsByCat": [
    {
      "name": "Bathymetry",
      "count": 330,
      "size": 69309241
    },
    {
      "name": "Descriptive Reports",
      "count": 2,
      "size": 5759794
    },
    {
      "name": "Images",
      "count": 4,
      "size": 210224554
    },
    {
      "name": "Metadata",
      "count": 110,
      "size": 7884800
    },
    {
      "name": "Point Data",
      "count": 110,
      "size": 116021088
    }
  ]
}

The extents catalog query provides no information about individual surveys. The extents reflect the totals across all surveys matched as part of the query. As you add criteria to limit the selection, such as platforms, then the values returned will reflect the smaller volume of data matched.

Get Information about Sounding Files

The survey query provides statistics regarding categorized numbers and sizes of files associated with surveys. The Next system uses a file query internally to acquire more detailed information about these files. This information is typically not useful to clients, but is documented herein as a REST endpoint is used to access the information. To query sounding files, issue a GET or POST using the following URL pattern:

NOTE: If the request involves many (>980) parameter values, then use of a POST is required.

https://www.ngdc.noaa.gov/next-catalogs/rest/sounding/file<param-list>

This pattern ends with "rest/sounding/file", optionally followed by a parameter list of the form "?name1-value&name2=value...". Parameters are supplied to limit the list of files to those associated with particular surveys. Note that sounding files do NOT have file-specific dates and geometries, making it impossible to filter results these criteria.

In this format, the valid parameters are

JSON Response

A typical response looks like the example shown below. Notice that the count returned indicates the total number of results available, while the items list typically only holds one page worth of results.

{
  "items": [
    {
      "surveyId": "B00001",
      "filePath": "coast/B00001-B02000/B00001/trackline/B00001.a93.gz",
      "fileSize": 663017,
      "accessProtocol": true,
      "formatId": 12,
      "canPublish": true,
      "category": "Bathymetry"
    },
    {
      "surveyId": "B00001",
      "filePath": "coast/B00001-B02000/B00001/trackline/B00001.xyz.gz",
      "fileSize": 711620,
      "accessProtocol": true,
      "formatId": 14,
      "canPublish": true,
      "category": "Bathymetry"
    },
    {
      "surveyId": "B00001",
      "filePath": "coast/B00001-B02000/B00001/trackline/B00001_h93.htm",
      "fileSize": 14291,
      "accessProtocol": true,
      "formatId": 13,
      "canPublish": true,
      "category": "Bathymetry"
    },
    {
      "surveyId": "B00002",
      "filePath": "coast/B00001-B02000/B00002/trackline/B00002.a93.gz",
      "fileSize": 357139,
      "accessProtocol": true,
      "formatId": 12,
      "canPublish": true,
      "category": "Bathymetry"
    },
    {
      "surveyId": "B00002",
      "filePath": "coast/B00001-B02000/B00002/trackline/B00002.xyz.gz",
      "fileSize": 374798,
      "accessProtocol": true,
      "formatId": 14,
      "canPublish": true,
      "category": "Bathymetry"
    },
    {
      "surveyId": "B00002",
      "filePath": "coast/B00001-B02000/B00002/trackline/B00002_h93.htm",
      "fileSize": 14306,
      "accessProtocol": true,
      "formatId": 13,
      "canPublish": true,
      "category": "Bathymetry"
    },
    {
      "surveyId": "B00003",
      "filePath": "coast/B00001-B02000/B00003/trackline/B00003.a93.gz",
      "fileSize": 482138,
      "accessProtocol": true,
      "formatId": 12,
      "canPublish": true,
      "category": "Bathymetry"
    },
    {
      "surveyId": "B00003",
      "filePath": "coast/B00001-B02000/B00003/trackline/B00003.xyz.gz",
      "fileSize": 524268,
      "accessProtocol": true,
      "formatId": 14,
      "canPublish": true,
      "category": "Bathymetry"
    },
    {
      "surveyId": "B00003",
      "filePath": "coast/B00001-B02000/B00003/trackline/B00003_h93.htm",
      "fileSize": 14289,
      "accessProtocol": true,
      "formatId": 13,
      "canPublish": true,
      "category": "Bathymetry"
    }
  ],
  "count": 9
}

If the query contains a mistake, the JSON response will contain a message with details about usage.

{"message": "invalid request: Dates and times must be written in ISO 8601 format (e.g. yyyy-MM-dd)"}

2.6 Multibeam Catalog Service

List Available File Categories

Multibeam files are categorized. File statistics (counts and sizes) are provided in terms of these categories. Queries and orders can filter on these categories, giving some degreee of control over the content of an order. The most basic query provided for multibeam datasets is the ability to get a list of these categories. To obtain a list of all file categories issue a GET request having the following URL pattern:

https://www.ngdc.noaa.gov/next-catalogs/rest/multibeam/catalog/category

This pattern ends with "rest/multibeam/catalog/category". It accepts no parameters.

JSON Response: The response JSON contains a list of names and a total count.

{ "items":["Bathymetry","Documentation","Metadata"...], "count":7 }

List Available Surveys (by name)

Each survey is identified by a survey name. To obtain a list of all multibeam survey names, or those corresponding to specified filtering parameters, issue a GET or POST request having the following URL pattern:

https://www.ngdc.noaa.gov/next-catalogs/rest/multibeam/catalog/surveyid

This pattern ends with "rest/multibeam/catalog/surveyid", followed by an optional parameter list of the form: "?name1-value&name2=value...".

In this format, the valid parameters are

JSON Response: The response JSON contains a list of names and a total count.

{ "items":["09CQ01_Saipan","09CQ02_Tinian","2006_07_FMG"...], "count":2060 }

List Available Platforms (by name)

Each vessel that supports surveys is identified by a platform name. To obtain a list of all multibeam platform names, or those corresponding to specified filtering parameters, issue a GET or POST request having the following URL pattern:

https://www.ngdc.noaa.gov/next-catalogs/rest/multibeam/catalog/platform

This pattern ends with "rest/multibeam/catalog/platform", followed by an optional parameter list of the form: "?name1-value&name2=value...".

In this format, the valid parameters are

JSON Response: The response JSON contains a list of names and a total count.

{ "items":["AeroCommander","Ahi","Atlantic Surveyor"...], "count":70 }

Search for Multibeam Surveys

To select Multibeam bathymetry surveys, issue a GET or POST using this URL pattern:

https://www.ngdc.noaa.gov/next-catalogs/rest/multibeam/catalog/survey<param-list>

This pattern ends with the string "rest/multibeam/catalog/survey", followed by a parameter list of the form: "?name1-value&name2=value...".

NOTE: If the request involves many (>980) parameter values, then use of a POST is required.

In this format, the valid parameters are

Notes:

The platforms and surveys are case-sensitive, and must match precisely.

The startYear and endYear define a time range, and only data recorded within that range are selected.

The max and offset parameters are designed to support a paged response.

A typical response looks like the example shown below. Notice that the count returned indicates the total number of results available, while the items list typically only holds one page worth of results.

The JSON result of https://www.ngdc.noaa.gov/next-catalogs/rest/multibeam/catalog/survey?surveys=AT4L02 will look like this:

{ "count": 1,
  "items": [
    {
    "id": "78700",
    "ngdcId": "02130063",
    "surveyId": "AT4L02",
    "platform": "Atlantis",
    "chiefScientist": null,
    "instrument": "SeaBeam 2112",
    "nav1": "GPS",
    "startTime": "2001-02-11T00:00:00Z",
    "endTime": "2001-02-18T00:00:00Z",
    "geometry": "-79.2166150481706,31.5775122120196,-77.6774699529622,33.1431898525343",
    "fileStatsByCat": [
      {
      "name": "Bathymetry",
      "count": 280,
      "size": 575374905
      }
    ],
    "fileCount": 280,
    "fileSizeSum": 575374905
    }
  ]
}

The example above has been shortened and reformatted for readability. The actual response will be returned as a long single line.

If there are no items selected, the result will look like this:

{"items":[],"count":0}

It is likely that if you specify no parameters that the response will be larger than can be ordered. Examination of the results may provide an indication of how the query can be tightened up to provide a more reasonably sized result.

Get an Estimate of the Size of a Multibeam Data Request

An 'extents' query provides the geographical and temporal extent of the results, and file count and total download size (in bytes) associated with a particular Multibeam query. A fileStatsByCat array lists file counts and sizes by category.

To query Multibeam data extents, GET or POST from this URL:

https://www.ngdc.noaa.gov/next-catalogs/rest/multibeam/catalog/extents

NOTE: If the request involves many (>980) parameter values, then use of a POST is required.

This pattern ends with the string "rest/multibeam/catalog/extents", followed by a parameter list of the form: "?name1-value&name2=value...". The parameter list is identical to that used to make a survey catalog query.

The extents result will look like this:

{
  "startYear": 2001,
  "endYear": 2001,
  "geometry": "-79.2166150481706,31.5775122120196,-77.6774699529622,33.1431898525343",
  "fileStatsByCat": [
    {
      "name": "Bathymetry",
      "count": 280,
      "size": 575374905
    }
  ],
  "count": 280,
  "size": 575374905
}

The extents catalog query provides no information about individual surveys. The extents reflect the totals across all surveys matched as part of the query. As you add criteria to limit the selection, such as platforms, then the values returned will reflect the smaller volume of data matched.

Get Information about Multibeam Files

The survey query provides statistics regarding categorized numbers and sizes of files associated with surveys. The Next system uses a file query internally to acquire more detailed information about these files. This information is typically not useful to clients, but is documented herein as a REST endpoint is used to access the information. To query multibeam files, issue a GET or POST using the following URL pattern:

NOTE: If the request involves many (>980) parameter values, then use of a POST is required.

https://www.ngdc.noaa.gov/next-catalogs/rest/multibeam/file<param-list>

This pattern ends with "rest/multibeam/file", optionally followed by a parameter list of the form "?name1-value&name2=value...". Parameters are supplied to limit the list of files to those associated with particular surveys and/or dates. Note that multibeam files have file-specific dates and geometries, making it possible to filter results these criteria.

In this format, the valid parameters are

JSON Response

A typical response looks like the example shown below. Notice that the count returned indicates the total number of results available, while the items list typically only holds one page worth of results.

{
  "items": [
    {
      "id": 124205,
      "surveyId": "TN182",
      "startTime": "2005-08-08T00:00:00Z",
      "endTime": "2005-08-08T00:00:00Z",
      "filePath": "/stornext/ngdc/archive/insitu_ocean/trackline/thomas_g._thompson/tn182/multibeam/data/version1/MB/0036_20050808_231018p.mb57.gz",
      "fileSize": 6196815,
      "accessProtocol": "ARCHIVE",
      "formatId": 57,
      "mostCurrent": true,
      "version": 2,
      "canPublish": true,
      "category": "Processed Data"
    },
    {
      "id": 124206,
      "surveyId": "TN182",
      "startTime": "2005-08-14T00:00:00Z",
      "endTime": "2005-08-14T00:00:00Z",
      "filePath": "/stornext/ngdc/archive/insitu_ocean/trackline/thomas_g._thompson/tn182/multibeam/data/version1/MB/0036_20050814_102538p.mb57.gz",
      "fileSize": 5102645,
      "accessProtocol": "ARCHIVE",
      "formatId": 57,
      "mostCurrent": true,
      "version": 2,
      "canPublish": true,
      "category": "Processed Data"
    },
    {
      "id": 124207,
      "surveyId": "TN182",
      "startTime": "2005-07-24T00:00:00Z",
      "endTime": "2005-07-24T00:00:00Z",
      "filePath": "/stornext/ngdc/archive/insitu_ocean/trackline/thomas_g._thompson/tn182/multibeam/data/version1/MB/0038_20050724_095507p.mb57.gz",
      "fileSize": 9839280,
      "accessProtocol": "ARCHIVE",
      "formatId": 57,
      "mostCurrent": true,
      "version": 2,
      "canPublish": true,
      "category": "Processed Data"
    },
    {
      "id": 124208,
      "surveyId": "TN182",
      "startTime": "2005-07-22T00:00:00Z",
      "endTime": "2005-07-22T00:00:00Z",
      "filePath": "/stornext/ngdc/archive/insitu_ocean/trackline/thomas_g._thompson/tn182/multibeam/data/version1/MB/0037_20050722_202111p.mb57.gz",
      "fileSize": 7181024,
      "accessProtocol": "ARCHIVE",
      "formatId": 57,
      "mostCurrent": true,
      "version": 2,
      "canPublish": true,
      "category": "Processed Data"
    },
    {
      "id": 124209,
      "surveyId": "TN182",
      "startTime": "2005-07-31T00:00:00Z",
      "endTime": "2005-07-31T00:00:00Z",
      "filePath": "/stornext/ngdc/archive/insitu_ocean/trackline/thomas_g._thompson/tn182/multibeam/data/version1/MB/0037_20050731_091947p.mb57.gz",
      "fileSize": 11206673,
      "accessProtocol": "ARCHIVE",
      "formatId": 57,
      "mostCurrent": true,
      "version": 2,
      "canPublish": true,
      "category": "Processed Data"
    },
    {
      "id": 124210,
      "surveyId": "TN182",
      "startTime": "2005-08-08T00:00:00Z",
      "endTime": "2005-08-08T00:00:00Z",
      "filePath": "/stornext/ngdc/archive/insitu_ocean/trackline/thomas_g._thompson/tn182/multibeam/data/version1/MB/0037_20050808_233750p.mb57.gz",
      "fileSize": 3847880,
      "accessProtocol": "ARCHIVE",
      "formatId": 57,
      "mostCurrent": true,
      "version": 2,
      "canPublish": true,
      "category": "Processed Data"
    },
    {
      "id": 124211,
      "surveyId": "TN182",
      "startTime": "2005-08-14T00:00:00Z",
      "endTime": "2005-08-14T00:00:00Z",
      "filePath": "/stornext/ngdc/archive/insitu_ocean/trackline/thomas_g._thompson/tn182/multibeam/data/version1/MB/0037_20050814_161330p.mb57.gz",
      "fileSize": 10342389,
      "accessProtocol": "ARCHIVE",
      "formatId": 57,
      "mostCurrent": true,
      "version": 2,
      "canPublish": true,
      "category": "Processed Data"
    },
    {
      "id": 124212,
      "surveyId": "TN182",
      "startTime": "2005-07-22T00:00:00Z",
      "endTime": "2005-07-22T00:00:00Z",
      "filePath": "/stornext/ngdc/archive/insitu_ocean/trackline/thomas_g._thompson/tn182/multibeam/data/version1/MB/0038_20050722_212111p.mb57.gz",
      "fileSize": 6645181,
      "accessProtocol": "ARCHIVE",
      "formatId": 57,
      "mostCurrent": true,
      "version": 2,
      "canPublish": true,
      "category": "Processed Data"
    },
    {
      "id": 124213,
      "surveyId": "TN182",
      "startTime": "2005-08-09T00:00:00Z",
      "endTime": "2005-08-09T00:00:00Z",
      "filePath": "/stornext/ngdc/archive/insitu_ocean/trackline/thomas_g._thompson/tn182/multibeam/data/version1/MB/0038_20050809_060549p.mb57.gz",
      "fileSize": 13387263,
      "accessProtocol": "ARCHIVE",
      "formatId": 57,
      "mostCurrent": true,
      "version": 2,
      "canPublish": true,
      "category": "Processed Data"
    },
    {
      "id": 124214,
      "surveyId": "TN182",
      "startTime": "2005-07-22T00:00:00Z",
      "endTime": "2005-07-22T00:00:00Z",
      "filePath": "/stornext/ngdc/archive/insitu_ocean/trackline/thomas_g._thompson/tn182/multibeam/data/version1/MB/0039_20050722_222111p.mb57.gz",
      "fileSize": 4320474,
      "accessProtocol": "ARCHIVE",
      "formatId": 57,
      "mostCurrent": true,
      "version": 2,
      "canPublish": true,
      "category": "Processed Data"
    }
  ],
  "count": 1719
}

If the query contains a mistake, the JSON response will contain a message with details about usage.

{"message": "invalid request: Dates and times must be written in ISO 8601 format (e.g. yyyy-MM-dd)"}

2.7 Trackline Catalog Service

Trackline Catalog Use

The trackline catalog is responsible for providing information to both client and engine regarding available data and its size. Externally accessible methods all return a map that can be rendered as JSON. The exact content of these maps varies across endpoint, but a 'message' parameter is always present if and only if a request has failed. Unlike most other datasets, 'trackline' can return both trackline point and ancillary file data. The point data is dynamically generated into files from a point database, while the ancillary files are pulled from disk. The includeAncillary parameter is key to determining catalog behavior, as follows:

If includeAncillary is false, then the engine generates only the survey request to the catalog. The order includes point data for all matching surveys (without ancillary files). The engine uses the catalog to get a list of surveyIds that can be used to fetch point data.

If includeAncillary is true, then the engine generates both 'survey' and 'files' requests to the catalog, receiving a list of all matching surveys and all associated ancillary files. The client typically uses any supported search criteria initially to acquire an overall list of surveys with which to work. The actual order may reflect a subset of the original survey list, specified using the surveyIds parameter in conjunction with 'categories' to fine-tune the content of an order. The resulting order includes point data and ancillary files for all selected surveys.

The survey request returns a list of surveys, including information about the size and count of both point data and ancillary files. Ancillary file information is summarized by category.

The presence of an optional 'categories' parameter with an order indicates that ancillary file results should be filtered, using the 'categories' parameter, which contains a list of "category:sub-category" combinations that are of interest.

The catalog supports the use of any of the criteria defined for the survey endpoint with files and extents endpoints.

List Available Platforms

A platform is a vehicle that gathers survey data, such as a ship or an airplane. To obtain a list of all platform names, issue a GET request having the following URL pattern:

https://www.ngdc.noaa.gov/next-catalogs/rest/trackline/platform

This pattern ends with "rest/trackline/platform". It accepts no parameters.

JSON Response: The response JSON contains a list of names and a total count.

{ "items":["Moana Wave","Polarstern","Heincke"...], "count":417 }

List Survey IDs

To obtain a list of survey IDs, issue a GET request having the following URL pattern:

https://www.ngdc.noaa.gov/next-catalogs/rest/trackline/surveyid

This pattern ends with "rest/trackline/surveyid". It accepts no parameters.

JSON Response

{ "items":["U-3-65-NP","79000212","82002711"...], "count":6150 }

Search for Surveys

A survey search is available by issuing a GET request having the following URL pattern:

https://www.ngdc.noaa.gov/next-catalogs/rest/trackline/survey<param-list>

This pattern ends with "rest/trackline/survey", optionally followed by a parameter list of the form "?name1=valeu&name2=value...". Parameters are supplied to provide criteria for selecting surveys of interest. The following parameters are supported. All are optional. If surveyTypes is not specified, then it defaults to 'All Parameters'.

Datatype (also called survey type) restricts results to surveys having particular types of data, with 'All Parameters' (the default) signifying an interest in all available types. The following data/survey types can be specified:

'All Parameters', 'Bathymetry', 'Gravity', 'Magnetics', 'Multi-Channel Seismics', 'Single-Channel Seismics', 'Side Scan Sonar', 'Subbottom Profile', 'Seismic Refraction', 'Shot-Point Navigation'

JSON Response

{
  "items":[
    {
      "surveyId":"CK77-1",
      "platform":"Acheron",
      "institution":"Hawaii, U of",
      "startYear":1977,
      "endYear":1977,
      "firstDateAdded":"1983-03",
      "lastDateAdded":"1983-04",
      "bbox":[-159.91,-21.18,-156.51,-18.74],
      "pointCounts":{
        "All Parameters":1025,
        "Bathymetry":1025,
        "Magnetics":1000,
        "Gravity":500,
        "Single-Channel Seismics":0,
        "Multi-Channel Seismics":0,
        "Subbottom Profile":0,
        "Side Scan Sonar":0,
        "Seismic Refraction":0,
        "Shot-Point Navigation":0
      },
      "dataForms":{
        "All Parameters":"ANA+DIG",
        "Bathymetry":"DIGITAL",'
        "Magnetics":DIGITAL,
        "Gravity":ANA+DIG,
        "Single-Channel Seismics":null,
        "Multi-Channel Seismics":null,
        "Subbottom Profile":null,
        "Side Scan Sonar":null,
        "Seismic Refraction":null,
        "Shot-Point Navigation":null
      }
    }
  ],
  "count":1
}

If the query contains a mistake, the JSON response will contain a message with details about usage.

{"message": "invalid request: Dates and times must be written in ISO 8601 format (e.g. yyyy-MM-dd)"}

Get an Estimate of the Size of a Trackline Request

An 'extents' query provides total counts and download sizes (in bytes) for points and files associated with a particular Trackline query. Geographical extent is not provided for this dataset at the current time.

To query trackline extents, issue a GET using the following URL pattern:

https://www.ngdc.noaa.gov/next-catalogs/rest/trackline/catalog/extents<param-list>

This pattern ends with "rest/trackline/catalog/extents", optionally followed by a parameter list of the form "?name1=value&name2=value...". Parameters are supplied to provide criteria for selecting surveys of interest. The parameters are supported are identical to those supported when searching for surveys of interest (the rest/trackline/survey endpoint)". All are optional. If surveyTypes is not specified, then it defaults to 'All Parameters'.

JSON Response

Example:

{ pointCount: 20205, pointSize: 1033617, fileCount: 12, fileSize: 304667 }

If the query contains a mistake, the JSON response will contain a message with details about usage.

{"message": "invalid request: Dates and times must be written in ISO 8601 format (e.g. yyyy-MM-dd)"}

Get Information about Trackline Ancillary Files

The survey query provides statistics regarding categorized numbers and sizes of ancillary files associated with surveys. The Next system uses a file query internally to acquire more detailed information about ancillary files. This information is typically not useful to clients, but is documented herein as a REST endpoint is used. To query trackline files, issue a GET using the following URL pattern:

https://www.ngdc.noaa.gov/next-catalogs/rest/trackline/file<param-list>

This pattern ends with "rest/trackline/file", optionally followed by a parameter list of the form "?surveyids=value,value...". Parameters are supplied to limit the list of ancillary files to those associated with particular surveys.

JSON Response

Example:

{ count: 2,
    items:
	[{surveyId: mgln02mv, filePath: melville/mgln02mv/gravity/raw/100033.txt,
	  fileSize: 2115, cat: Gravity, subcat: Raw},
	{surveyId: mgln02mv, filePath: melville/mgln02mv/gravity/raw/mgln02mv_grav.tar.gz,
	  fileSize: 88655, cat: Gravity, subcat: Raw}]
  }

If the query contains a mistake, the JSON response will contain a message with details about usage.

{"message": "invalid request: Dates and times must be written in ISO 8601 format (e.g. yyyy-MM-dd)"}