SAWG MTG Minutes 2011-03-11
From NGDC Wiki
- Dan Kowal
- Anna Milan
- Phil Jones
- Scott McCormick
- Jeremy Throwe
- CLASS (Scott)
- Informed the group about the FGDC -> ISO transition at CLASS will be supported in the 5.5 release in the end of June.
- Working through a lot of Project Charters dealing with CDRs from NCDC:
- Blended Rain Products
- Discussed who the main POC should be for each of these requests.
- Ingest Activities:
- Planning for two NODC data sets in rel. 5.5
- NPP activities consuming a lot of time.
- GOES-R PDR requiring a lot of energy for next week.
- NCDC (Phil)
- New Dataset Requests:
- OSDPD: veg products from NPP and AVHRR, and Surf. Temp from GOES to be operational in 2012.
- No funding for the three, so still need to know how to move forward in these scenarios.
- River Forecast Centers.
- Grid of precipitation. POR goes back to mid 1990s.
- Will begin with one Center in Arkansas. Working on NetCDF conversion, but may decide to keep the original record as well.
- Tech. Mtg. with NDE on SA and ICD. Focused on the NewCaps (NUPS?) (Sp?) products, legacy IAZI, MIRS and SST products.
- New Dataset Requests:
- NGDC (Dan)
- Aiming to wrap up VIPIR archive recommendation for next COWG Mtg. Will be an interesting one to present with no budget.
- Entertaining another non-funded request called MultiLens which is aerial photography from the former Coast and Geodetic Survey and is being digitized through CDMP.
- There's interest in using ATRAC for SA development, especially for existing SAs that are currently going through modifications.
- Phil gave an overview of the new version of this Archive Project Tracking tool.
- Ken Casey has some input regarding some more tie into the Appraisal/Approval Guidelines in the intro. Still need to discuss this further, but intro spells out the types of information collected and tracked in the system.
- Tasks. You can now add as many as you want.
- When you initiate Requests to Archive (R_to_A) or Submission Agreement process in the system, a task is automatically created.
- Metadata, information from the R_to_A will feed into the SA development.
- If R_to_A is not needed (working on existing SA's like in the case of NGDC), you can add all of the info in the SA form and not have to first enter info in the R_to_A form -- there's no dependency of information from one to the other.
- Non-NOAA providers can have accounts and give input to a particular project.
- Metadata Task is focused on Collection Level development. You can add additional tasks for other metadata development.
- Dan will review the SA form in particular and compare with the existing template we're working with now to see if everything is captured.
- ATRAC is not a document management system. Therefore, any additional reference info that has been put in Appendices will need to reside elsewhere and referenced from the SA with some citation.
Companion Metadata Issue
- CLASS seemed to be the only group calling for it. It became clear that we had a large number of files generated in this scenario that has management implications.
- DEWT said we had misinterpreted the data volume statistics. Instead of 2.5 MIL files/day (Products, Metadata, Manifests), it was more like 1.5 MIL files/day.
- It was reported that the refresh rate from the MAG was incorrect. Instead of .5 sec refresh, it was more like .5 min. The volume is then reduced to 1/60 of the original estimate.
- There will be a metadata TIM at the end of the month. Data Center reps will be there to hash out the issue of whether or not a companion metadata file is still needed.
- If we go the embedded route, CLASS does not want to have an NPP situation where metadata locations moved around. It has to be easy to extract. They have assurances from Carlomusto that it will be easy, but want more than just assusurances.
- Phil: If ISO compliant, this should not be a difficult process. Use ncdump to generate NcML and then extract the metadata from there.
- There's an engineering review board that will analyze this further to understand the processing implications - meeting week from Tues.
- The original MAG instrument scenario was driving this, but now given the major correction, it's probably a non-issue. However, CLASS would still like to get Data Centers to weigh in and evaluate from the end user perspective how best to package data.
Submission Agreement Review
- It is the CLASS understanding that once the SA matures, we'll need to revisit the functional requirements and updated accordingly.
- The next review cycle is coming up, 3/22.
Future Considerations for CLASS Metadata (Anna)
- Anna will set up a meeting in the near future to get full data center participation and CLASS to discuss future directions of metadata management for CLASS.
- Main question at hand is can CLASS deal with harvesting from individual Data Center WAFs as opposed to just one?
- Jeremy said that reconfiguration calls for a requirements change and Shelly Briscoe and/or Martin Peschy will need to be involved.
- Anna said this decentralized model may be the way of the future if CLASS slides more into the background supporting archival storage, and the Data Centers are the gateway for the data. This is what she'd like to discuss. She brought up an example of a metadata records for reanalysis data at NCDC. This record is not in the CLASS metadata system, but only in the NCDC system.
- Phil: reconfiguration should not be so hard.
- Ken Casey continues to look at the one WAF concept.
- Anna expressed that there could be some danger in doing the distributed approach. Conflicts may occur. Jeremy said that Ownership/data type will have to be worked out. Anna's concern was with the responsiveness of CLASS once campaigns have ended and metadata has to be updated beyond that.
- Anna said that the topic of "Enterprise vs. Distributed" needs discussion and will try to get a meeting going the week of Mar. 21.
- Jeremy said they no longer have requirement teams per se, but have integrated groups per project. Keep him in the loop.
- Friday, April 8.