SAWG MTG Minutes 2011-02-11
- Dan Kowal
- Anna Milan
- Phil Jones
- Scott McCormick
Submission Agreement Review
- Dan would like to use part of the SAWG mtgs as review of an SA in progress to evaluate if the current template is being used as expected and if any tweaks are needed in it. He'll propose one for next time.
- Phil: What does CLASS need out of the SA?
- Scott: it clarifies requirements even though it's not a requirements document. We use it to build documents in CLASS. The ICDs on the otherhand mainly focus on ingest arrangments. Search and Access details covered elsewhere - usually the SA. We may want to re-think of the ingest side.
- Phil described the CCR(?) model data SA and how Lea wanted particular descriptions captured.
- Dan: ICD vs. SA? How different, how to reference between them?
- Scott: The ICDs should cover the arrangements between the provider and CLASS.
- Phil: The SA is between the provider and data center.
- Dan/Phil: Reviewed documentation that's pertinent for the PDI section in the SA, especially giving context and lineage info for the data.
ISO Metadata in CLASS Update
- Anna: Work request for getting support for ISO metadata has been approved and slated to go into CLASS rel 5.5.1 in July.
- CLASS will support iso standard metadata.
- Will support ISO prestation in HTML view.
- The search-collection interface will look the same but underlying metadata model will change.
- Harvest will point to the ISO WAF maintained at NGDC. Once transition is done, metadata will no longer be displayed in FGDC.
- Talking a lot with NODC to convert to ISO. Will need to pick up that discussion again.
- All of the records have been converted to ISO.
- CLASS has an ISO schema.
- Are there CLASS specific mandatory elements? Not really. Each data type family is different - AERO 100 example given.
- Baseline elements in ISO will be necessary, and information is represented consistently across all records at this time.
- They are used for CLASS search metadata.
- When you get into the data type metadata pages, there's title, abstract, start and end dates, other values that vary across the data sets.
- Anna performed a complete mapping of all of the data families - sent to Martin and Shelly. Anna has a spreadsheet. Will send out an updated version.
- Stacy Zeigler is the go-between. Not sure what the test process will look like.
- Scott: This will go through official testing; integration/test enviroment. pulling in the ISO metadata. Pull Anna in the test round in June.
- Anna: feels a little disconnected from the CLASS process.
- Phil: If people are writing metadata for CLASS, they need to know. At what level do we write the metadata - lowest level, the data type level...the web display is another?
- Anna: Will create a wiki page for transitioning legacy stuff. How CLASS can support data type level metadata is an outstanding question. Perhaps we proceed on just a Campaign by Campaign basis?
- Scott: Wouldn't have to be; perhaps can be handled in a general way.
- Anna: let's not rock the boat too much. Go step by step; transition first and then tackle more involved transitioning of metadata.
- Phil: How do you define a family; hasnt' always been consistent in CLASS like GOES SST. NCSAT. DMSP families. Anna: That data type collection and groups can be more open ended.
- Scott: CLASS SE team moving away from data families. Definitely some limitations for how data families are handled.
- Anna: would need some system wide change for handling granular metadata. This would be great, but not sure how that gets going.
- Phil: GOES-R metadata; having everything in one place as opposed to having it separated in a companion file... where does this stand? Heard from Tino about his concerns for file management; leaning towards one file with metadata combined in the netCDF file. ISO-compliant metadata in netCDF; Do we still need the XML? If we add the global CF discovery attributes, that could work.
- Anna: Ted proposed urls with unique IDs to external sources. But there's risk involved albeit more efficient. Need to know how the information is coming from the ground segment to CLASS.
- Scott: Dependency on supporting the system longterm. What happens when the urls change?
- Phil: Is Ted leaning on having everything in one file?
- Dan: DEWG discussion?
- Scott: There's a Face to Face at the end of March for DEWG. March 29th - 31st. * Phil: Would be good to discuss things on a more frequent basis.
- Scott: There is an analysis going on about the tradeoffs. 2.5 mil files a day is what proposed right now. If we get an XML metadata record, double it to 5 mil files. If we don't get them, then we have to read every NetCDF file; still looking into that.
- Phil: Running NC operators besides XML processing shouldn't be very computer processing intensive. Avoid the ingest risk of having to process a set of two files.
- Anna: How would this affect the display metadata requirements?
- Scott: The assumption in the analysis is that we wouldn't lose any of the metadata for search and display; it would be accessible in the NetCDF file. More discussion needed that it's coming in a usable way.
Phil: You can add as many attribute extensions in the CF global attributes. Could use the ISO element names so that it's standardized. The GHRSST people have done something similar; outline attributes and how they map to specific attributes; have documents for how to pull in information. It's on the NODC web site GDF-2, Section 8.
- Something come up recently, GCOM Data coming in IDPS. One of the questions is whether the data is going to be stored in CLASS. Hasn't been officially directed to store the data. Good to participate to know what's going on, but need official guidance.
- ROM estimate for Jason-3 data coming to CLASS. New person, Deirdre Byrne from NODC working on this; This is similar to Jason-2. Jason-3 coming to CLASS in 2013.
- NPP - Big test in April (NCPT part 2), run the whole ground segment; 3 day run in operational mode. Some hardware upgrades to complete. NC-4 test in June. End to end tests for replications. Draft of test plan going around; Kelly P from NGDC has seen it and commented.
- Deepika Dhuria working onsite at NODC. Senior Systems Engineer - working requirements. Like ASCPO. Stacey equivalent.
- Anna: Data Type families for NPP will need better metadata.
- Scott: Will have Steve from CLASS contact Anna. Al Bibbero, at NGDC, is mostly supporting external interfaces.
- Phil: Presented TPW overview to COWG this week. Operational vs. high quality product. COWG is on board; however, the next step is to identify funding before requesting it to COPB. This is a new twist.
- Scott: Task objective statement from the data centers for Year 4 Contract to be worked on in the near future.
- Dan: Would be great to work in the tasks or work in the objectives to support unfunded datasets.
Announcements, Next Meeting
- CLASS All Hands Workshop: June 7-8, 2011 in Morgantown, WV.
- Next Meeting: Friday, March 11.