NMMR Services

From NGDCWiki

(Difference between revisions)
Jump to: navigation, search
Current revision (10:20, 30 April 2013) (view source)
m (Replacing page with 'NMMR is deprecated.')
 
Line 1: Line 1:
-
==NMMR Services: Getting metadata records out of the NMMR==
+
NMMR is deprecated.
-
 
+
-
This is a list of the ways we support (or might be able to support) for getting metadata records out of the NMMR. It is roughly in order of increasing access functionality from a programmer's point of view, but also in order of increasing number of dependencies (thus decreasing reliability).
+
-
 
+
-
==Summary Diagram==
+
-
[[Image:NMMRMetadataAccessOptions.gif]]
+
-
 
+
-
Note: Not all options are shown for clarity. See table below for all options.
+
-
 
+
-
==Summary Table==
+
-
 
+
-
* Service: Name of service used to retrieve metadata records
+
-
* Search: If and how searching for records can be performed
+
-
* Listing: If and how a list of available records can be obtained
+
-
* Access: mechanism for retrieving a complete record
+
-
* Dependencies: hardware and software systems at NGDC that need to be running for access
+
-
* Sync w/ NMMR: whether records are in sync with those currently in the NMMR
+
-
* Record States: which record states (Published, Draft, Validated, Approved) can be retrieved
+
-
* CRUD: which operations are available on records (Create, Read, Update, Delete)
+
-
<br /><br />
+
-
{|cellspacing="0" border="1" align="center"
+
-
!Service
+
-
!Search
+
-
!Listing
+
-
!Access
+
-
!Dependencies
+
-
!Sync w/ NMMR
+
-
!Record States
+
-
!CRUD
+
-
|- align="center"
+
-
![[#Geospatial One Stop| Geospatial One Stop]]
+
-
|date, geospatial extent, keyword, ...
+
-
|None (other services?)
+
-
|Interactive (other services: HTTP, Z39.50, OAI, and CS-W 2.0?)
+
-
|No dependencies, redundant servers
+
-
|up to a week out of date
+
-
|Published
+
-
|Read only
+
-
 
+
-
|- align="center"
+
-
![[#Web Accessible Folder (WAF)| Web Accessible Folder]]
+
-
|Google site: (may not be up to date)
+
-
|URL of directory
+
-
|URL to XML file
+
-
|'www', 'metadata1'; independent of NMMR
+
-
|in sync
+
-
|Published
+
-
|Read only
+
-
 
+
-
|- align="center"
+
-
![[#Unix rsync| Unix rsync]]
+
-
|None
+
-
|'-n, --dry-run' option, or rsh/ssh
+
-
|see 'man rsync'
+
-
|'metadata1'; independent of the NMMR
+
-
|in sync
+
-
|Published
+
-
|Read only
+
-
 
+
-
|- align="center"
+
-
![[#Z39.50 Clearinghouse Protocol| Z39.50]]
+
-
|date, geospatial extent, keyword, ...
+
-
|all-inclusive search, e.g. any lat/lon
+
-
|Z39.50 client (e.g. zclient)
+
-
|'metadata1: z3950 server', 'kodkod:oracle'; independent of NMMR
+
-
|in sync
+
-
|Published
+
-
|Read only
+
-
 
+
-
|- align="center"
+
-
![[#NMMR Public URL| NMMR Public URL]]
+
-
|None
+
-
|None
+
-
|URL using "BAT UID"
+
-
|'metadata1: tomcat', 'kodkod:oracle'; dependent on NMMR
+
-
|in sync
+
-
|Published, Draft, Validated, and Approved
+
-
|Read only
+
-
 
+
-
|- align="center"
+
-
![[#SRU/SRW: URL or SOAP to Z39.50 (in progress)| SRU/SRW: URL or SOAP to Z39.50 (in progress)]]
+
-
|date, geospatial extent, keyword, ...
+
-
|all-inclusive search, e.g. any lat/lon
+
-
|CQL query in URL or wrapped with SOAP
+
-
|'metadata1: z3950 server, SRW proxy', 'kodkod:oracle'; independent of NMMR
+
-
|in sync
+
-
|Published
+
-
|Read only
+
-
 
+
-
|- align="center"
+
-
![[#OAI-PMH (planned)| OAI-PMH (planned)]]
+
-
|?
+
-
|?
+
-
|?
+
-
|?
+
-
|?
+
-
|Published
+
-
|Read only
+
-
 
+
-
|- align="center"
+
-
![[#NMMR SOAP (custom)| NMMR SOAP (custom)]]
+
-
|None
+
-
|MMRAccess.getRecordList()
+
-
|metadata XML wrapped with SOAP
+
-
|'metadata1: tomcat', 'kodkod:oracle'; dependent on NMMR
+
-
|in sync
+
-
|Published, Draft, Validated, and Approved
+
-
|Create, Read, Update, Delete
+
-
 
+
-
|- align="center"
+
-
![[#NMMR XMLDB SOAP (custom, alpha?)| NMMR XMLDB SOAP (custom, alpha?)]]
+
-
|?
+
-
|?
+
-
|<nowiki>http://www/nmmr/services/nmmrdb</nowiki> ??
+
-
|'metadata1: tomcat', 'kodkod:oracle'; dependent on NMMR
+
-
|in sync
+
-
|?
+
-
|?
+
-
|}
+
-
 
+
-
<br /><br />
+
-
 
+
-
==Details==
+
-
===Geospatial One Stop===
+
-
 
+
-
Part of the FGDC Clearinghouse network, this site (http://geodata.gov) harvests Published records from the NMMR database each week. [Included here as item (0) -- even though we have no direct control over the hardware or software -- since it is the most reliable access to NGDC metadata: multiple site redundancy and failover, and no dependencies on our local servers.]
+
-
 
+
-
* Search is available by date, geospatial extent, keyword, and a variety of other parameters.
+
-
* No list is available from the GOS web portal of all the Published NGDC records. A search for 'NGDC' approximates this, but isn't complete or exclusive. (There is an accurate list available when logging in as the metadata "creator", but that isn't easily accessed.) Services-based access might be able to provide this; see Access, below.
+
-
* Access is by interactive web page. Programmatic or services-based access has been or is intended to be implemented for HTTP, Z39.50, OAI, and CS-W 2.0. We have not tested any of these services, and don't know the state of GOS's implementations.
+
-
* Access is completely independent of the NMMR and NGDC; only the GOS portal needs to be running.
+
-
* Records may be up to a week out of date from those in the NMMR.
+
-
* Only records in the Published state are available; Draft, Validated, and Approved records are not visible.
+
-
* Read-only; no records may be created, updated or deleted.
+
-
 
+
-
===Web Accessible Folder (WAF)===
+
-
 
+
-
This is simply a directory of XML files of Published metadata records, accessible from anywhere on the Web. An index to the various record sets available is at http://www.ngdc.noaa.gov/metadata/published/. Thus, a simple URL retrieves the text/xml of a given record.
+
-
 
+
-
* Google site search can be used to search individual folders, e.g. 'water site:www.ngdc.noaa.gov/metadata/published/NGDC'. But this is not reliably up to date, since Google makes no promises about how often they index a site. We could purchase a Google service, such as the Appliance or Mini to ensure timely indexing.
+
-
* A list of available records is visible with a URL of the record set's directory, e.g. http://www.ngdc.noaa.gov/metadata/published/NESDIS_Products/.
+
-
* Access is by URL using the file name. In some record sets, this name is identical to the Dataset ID field (or other unique field) in the metadata record (e.g. http://www.ngdc.noaa.gov/metadata/published/NESDIS_Products/AVHRR.xml). In others, this is simply a "BAT UID", an internal (Blue Angel Technologies) database primary key (e.g. http://www.ngdc.noaa.gov/metadata/published/NGDC/1/1467.xml).
+
-
* Access is completely independent of the NMMR; if the NMMR or its database are down, these are still accessible, dependent only on the main 'www' NGDC web server and the 'metadata1' machine being turned on.
+
-
* Records are in sync with those in the NMMR.
+
-
* Only records in the Published state are available; Draft, Validated, and Approved records are not visible.
+
-
* Read-only; no records may be created, updated or deleted.
+
-
 
+
-
===Unix rsync===
+
-
 
+
-
An IP/unix utility for synchronizing files and directories remotely. This can be made very secure by requiring ssh as the transport used by rsync. Currently, this is configured to allow only particular IP addresses to access the NMMR metadata directories.
+
-
 
+
-
* No search is available; you must know the name of the file or directory (i.e. record set) you wish to retrieve.
+
-
* A list of available records is visible using the '-n, --dry-run' option, or the underlying transport shell (rsh/ssh) on the record set's directory.
+
-
* Access is by the unix (or cygwin) rsync utility, see 'man rsync' for details.
+
-
* Access is completely independent of the NMMR; if the NMMR or its database are down, these are still accessible, dependent only on the 'metadata1' machine being turned on.
+
-
* Records are in sync with those in the NMMR.
+
-
* Only records in the Published state are available; Draft, Validated, and Approved records are not visible.
+
-
* Read-only; no records may be created, updated or deleted.
+
-
 
+
-
===Z39.50 Clearinghouse Protocol===
+
-
 
+
-
An old and established protocol for accessing metadata. We are running the Blue Angel Z39.50 server in concert with the NMMR to make Published records available to any Z39.50 client from anywhere on the Internet.
+
-
 
+
-
* Search is available by date, geospatial extent, keyword, and a variety of other parameters.
+
-
* A list of available records is visible using an all-inclusive search, e.g. for records with bounding coordinates of any valid lat/lon values: zclient metadata.ngdc.noaa.gov 2210 "NGDC" "90 -180 -90 180[1,2060,2,7,4,201]"
+
-
* Access is by Z39.50 client (e.g. http://clearinghouse4.fgdc.gov/ftp/) using the (rather cryptic) RPN syntax (e.g. http://isite.awcubed.com/KWAQS-tutorial.html). For example, to retrieve XML records with "water" in the Title field: zclient metadata.ngdc.noaa.gov 2210 "NGDC" "water[1,4]" xml
+
-
* Access is independent of the NMMR application, but depends on its database (the machine 'kodkod') and the separate Z39.50 server running on 'metadata1'.
+
-
* Records are in sync with those in the NMMR.
+
-
* Only records in the Published state are available; Draft, Validated, and Approved records are not visible.
+
-
* Read-only; no records may be created, updated or deleted.
+
-
 
+
-
===NMMR Public URL===
+
-
 
+
-
Direct access via URL to any metadata record in the NMMR from anywhere on the Internet.
+
-
 
+
-
* No search is available; you must know the 'BAT UID' and record set for the record you wish to retrieve (see Access, below).
+
-
* No list of available records is possible with this.
+
-
* Access is by URL using the "BAT UID", an internal (Blue Angel Technologies) database primary key (e.g. http://www.ngdc.noaa.gov/nmmr/public/viewRecord.do?xmlstyle=plainxml&recuid=1467&recordset=NGDC&view=text/plain). This URL can be accessed from anywhere on the web. This URL can be constructed programmatically, or copy/pasted from a browser session when logged into the NMMR. Note that the BAT UID for a given metadata record is NOT constant. It may change based on underlying database changes or upon export/import to the NMMR.
+
-
* Access is dependent of the NMMR application (on the machine 'metadata1') and its database (on 'kodkod').
+
-
* Records are in sync with those in the NMMR.
+
-
* Records in all states are available: Published, Draft, Validated, and Approved.
+
-
* Read-only; no records may be created, updated or deleted.
+
-
 
+
-
===SRU/SRW: URL or SOAP to Z39.50 (in progress)===
+
-
 
+
-
It may be possible to send queries to the existing Z39.50 server in a URL or over SOAP using a Search/Retrieve Web Service (SRW) gateway (http://www.loc.gov/standards/sru/srw/). One such gateway is the YazProxy (http://www.indexdata.dk/yazproxy). It may also be possible to install a different Z39.50 or OAI server that directly supports SRW, though we would prefer not to add Yet Another Server to maintain.
+
-
 
+
-
We have not yet installed this capability, and cannot confirm its usability. The details here are based on how we think things should work.
+
-
 
+
-
* Search is available by date, geospatial extent, keyword, and a variety of other parameters.
+
-
* A list of available records is visible using an all-inclusive search, e.g. search for records with bounding coordinates of any valid lat/lon values.
+
-
* Access is by Common Query Language (CQL, see http://www.loc.gov/standards/sru/cql/) queries in a URL or wrapped in SOAP. CQL queries are much less cryptic than the traditional Z39.50 RPN syntax, e.g (title any "water marine")
+
-
* Access is independent of the NMMR application, but depends on its database (the machine 'kodkod') and the separate Z39.50 server (and SRU/SRW proxy) running on 'metadata1'.
+
-
* Records are in sync with those in the NMMR.
+
-
* Only records in the Published state are available; Draft, Validated, and Approved records are not visible.
+
-
* Read-only; no records may be created, updated or deleted.
+
-
 
+
-
===OAI-PMH (planned)===
+
-
 
+
-
Open Archives Initiative Protocol for Metadata Harvesting (OAI-PMH, see http://www.openarchives.org/), this is an improved protocol based on experience with Z39.50. Several of our partners have requested that we make our metadata available via OIA: GOS, GCMD, and Lamont/Doherty.
+
-
 
+
-
We have not yet installed this capability, and cannot confirm its usability. We don't know yet what capabilities OAI will provide.
+
-
 
+
-
===NMMR SOAP (custom)===
+
-
 
+
-
There is currently a custom SOAP interface to the NMMR for NODC's Accession Tracking system. We could develop something similar for other custom purposes by implementing the Java interface gov.noaa.mm.access.MMRAccess.
+
-
 
+
-
We have not yet implemented this capability, and cannot confirm its usability. The details here are based on how we think things should work.
+
-
 
+
-
* No search is available; you must know the 'BAT UID' and record set for the record you wish to retrieve (see Access, below).
+
-
* A list of available records can be obtained using the MMRAccess.getRecordList() method.
+
-
* Access is by wrapping metadata XML with SOAP, using MMRAccess.saveRecord(), getRecord(), updateRecord(), deleteRecord().
+
-
* Access is dependent of the NMMR application (on the machine 'metadata1') and its database (on 'kodkod').
+
-
* Records are in sync with those in the NMMR.
+
-
* Records in all states are available: Published, Draft, Validated, and Approved.
+
-
* Records may be created, read, updated or deleted.
+
-
 
+
-
===MMR XMLDB SOAP (custom, alpha?)===
+
-
 
+
-
There was apparently some development going on a long time ago (2005?) on an XMLDB/Axis-based SOAP access to the NMMR. It doesn't appear to work right now [need to ask Travis and Eric K?]
+
-
 
+
-
We have not yet implemented this capability, and cannot confirm its usability. The details here are based on how we think things should work.
+
-
 
+
-
* Search ??
+
-
* Listing ??
+
-
* Access: The URL http://www.ngdc.noaa.gov/nmmr/services/nmmrdb seems to be involved ??.
+
-
* Access is dependent of the NMMR application (on the machine 'metadata1') and its database (on 'kodkod').
+
-
* Records are in sync with those in the NMMR.
+
-
* which record states (Published, Draft, Validated, Approved) can be retrieved ??
+
-
* which operations are available on records (Create, Read, Update, Delete) ??
+
-
 
+
-
[[Category:Metadata]]
+

Current revision

NMMR is deprecated.

Personal tools