(Quick Reference)

1 Introduction

The "Next" (NCEI Extract) system provides access to data within NCEI archives, using a simple and scalable retrieval architecture. NCEI users often want to be able to make a single request for multiple kinds of data that, behind the scenes, is retrieved from multiple systems. Something like: “Just give me all the data from this area.” One of the goals of the Next architecture is to support this type of use case by providing a unified ordering interface.

The Next web API can RESTfully receive requests for data, divide and delegate the work of fulfilling the request to the appropriate subsystems, collect and package the results, and then deliver them to the user.

This document targets programmers who are developing applications that are intended to use the Next data delivery system. The ordering process involves several steps. An overview of these steps is provided here. Various sections of this documentation delve into the individual steps in more detail.

Overview

Data within the NCEI archive is organized by dataset, each of which might be expected to be associated with a separate catalog-level metadata record. Each query or order made to the Next system is organized in terms of a list of items, with each item being associated with one of these datasets.

Interaction with the Next system involves the steps listed below. Each of these steps is supported by particular REST requests. This document focuses on describing this REST interface. Although there are numerous underlying programs running to support this web service by retrieving data and packaging them for delivery, an application utilizing the REST interface need not typically be aware of these.

Steps to Making a Next Order

• querying catalog services to find data with search criteria,
• requesting data as orders for one or more data items,
• checking order status
• downloading the order

'''Querying Catalog Services''': The catalog interface is used to determine the nature and amount of data that is available for ordering. This is done by specifying catalog requests with specified selection criteria. Separate REST requests are available for each dataset. The purpose of catalog services are to find selection criteria that are desirable for placing an order.

Some of these criteria, such as time and geographical location, are consistent across datasets, while others are specific to particular datasets. The amount of information provided within responses to catalog queries varies significantly between datasets. The Trackline dataset, for example, is organized by survey, and provides a significant amount of information about available surveys, as well as their sizes if ordered. It emphasizes providing a significant amount of feedback to users regarding what is available. Other datasets support a simpler approach of simply listing filenames and associated sizes, assuming that the filenames themselves are meaningful to users.

'''Placing An Order''': A request to order data provides the same type of criteria that were used in catalog requests to specify which data is being ordered. Additional ordering parameters are also typically available for specifying other characteristics of an order, such as desired file format. Each order consists of a list of items. The attributes associated with each item vary from dataset to dataset.

'''Checking Order Status''': A prompt response form the Next system is typically prorvided, confirming receipt of the order. The process of processing an order is asynchronous. The status associated with the response indicates its current status. Additional requests can be made thereafter, with the client application basically waiting for NOTIFIED status. The order is then available for retrieval via a specified URL. If the order fails for any reason, then the status response indicates why the order failed.

'''Downloading the Order''': The status response provides a URL link that can be used to directly download the order. Behind the scenes, the Next system keeps track of such access, updating status to PICKED UP once the order has been downloaded at least once. The order then remains available for a short time, just in case a user wants to download it more then once.

Reference Links

See the Next Project page in the NCEI-CO intranet for an overview, or view technical details in Next Development Notes