An Introduction to the NOAA/NASA SSM/I PATHFINDER Pentad and Monthly Atmospheric Moisture Data Sets Table of Contents * 1.0 Introduction * 2.0 Pentad and Monthly Pathfinder HDF File Structure o 2.1 Pentad (Monthly) Cloud/Rain Liquid Water Grid o 2.2 Sum of the Squared Cloud/Rain Liquid Water Values o 2.3 Number of Pixels per Bin o 2.4 Pentad (Monthly) Columnar Water Vapor Grid o 2.5 Sum of the Squared Columnar Water Vapor Values o 2.6 Number of Pixels per Bin o 2.7 File Description * 3.0 HOW TO GET THE HDF LIBRARY AND TOOLS * 4.0 HOW TO COMPILE THE PROGRAMS EXTRACTRG and GETFILEDESC o 4.1 EXTRACTMG o 4.2 GETFILEDESC 1.0 INTRODUCTION This README contains information on the structure of the Hierarchical Data Format (HDF) Pentad (5 days) and Monthly Atmospheric Moisture (AM) grid files, instructions on how to get and install HDF on your machine, and a description of two utilities-- extractmg which extracts specified items from the pentad or monthly file, and getfiledesc which pulls out the file description and prints it to the screen. Attachment 1 is a brief discussion of the technical/scientific aspects of the data set, including pertinent references. The Marshall Space Flight Center (MSFC) Pathfinder Pentad and Monthly Atmospheric Moisture data sets were created using the MSFC SSM/I Pathfinder daily HDF Atmospheric Moisture files. A discussion of daily atmospheric moisture files is available from MSFC User Services at 205-922-5932 or e-mail at msfc@eos.nasa.gov. The daily data were grouped into a 1 degree longitude by 1 degree latitude array. See Attachment 1 for a discussion. The array is 360 x 180 where the first dimension is longitude and the second is latitude. Array location (1,1) contains data between 180 and 179 degrees west longitude, and 89 and 90 degrees north latitude. Array location (2, 2) contains data for the 1 degree square bounded by 179-178 degrees west longitude and 88-89 degrees north latitude. Data for each bin is accumulated for either 5 days (pentad) or one month, as appropriate. The first pentad begins with January 1 and includes data through January 5. Each succeeding pentad is generated using 5 day increments. During leap year, the pentad beginning on February 26 contains 6 days, including February 29; therefore, the other pentads are consistent with non-leap year pentads. The monthly files include all available data in the month. Attachment 2 is a list of the pentad or monthly atmospheric moisture files which were not generated because of instrument or data problems. Each pentad or monthly file contains grids of cloud/rain liquid water (LWG) in mg/cm2, columnar water vapor (WVG) in g/cm2, the sum of the squared values of cloud/rain liquid water (LWS), the sum of the squared values of columnar water vapor (WVS), the number of valid cloud/rain liquid water values within the bin (LWN), and the number of valid columnar water vapor values within the bin (WVN), and a file description. The monthly file is similar to the pentad file. Section 2 contains a description of the file structure. The MSFC Pathfinder team performed quality control (Q/C) on each file, and the details of this Q/C are available from MSFC User Services. HDF 8-bit raster images of the atmospheric moisture objects are available via ftp from the MSFC server. The 1 degree by 1 degree values were linearly interpolated to 0.5 degrees to generate the image, composed of 720 by 360 pixels. A color palette is attached for ease in viewing. One way to view the browse images is with the HDF Collage tool. The files were created on a Silicon Graphics VGX class computer with version 3.3, release 4 of the HDF/netCDF library. If you are receiving these files on tape, they were transferred to tape with the UNIX tar facility. All files have been compressed with the IRIX UNIX compress command. The file size for an uncompressed file is 0.8 megabytes. NOTE: All discussions in this text are for row major applications written in c. If you are using the HDF FORTRAN interface to read the HDF objects, the arrays will be transposed. For example, array A(5,3) in an HDF c interface would become A(3,5) in an HDF FORTRAN interface. This will apply to all HDF data array discussions in this text. 2.0 Pentad and Monthly Pathfinder HDF File Structure Each pentad or monthly atmospheric moisture file has the following contents: Table 1. Objects in the Pathfinder HDF Pentad (Monthly) Atmospheric Moisture File ITEM HDF OBJECT TYPE HDF REF No. Version Descriptor N/A 1 Pentad (or Monthly) Scientific Data Set 2 Cloud/rain liquid water (LWG) Sum of Squared Cloud/rain Scientific Data Set 3 liquid water values Number of Cloud/rain Scientific Data Set 4 liquid water pixels per bin Pentad (or Monthly) Scientific Data Set 5 Columnar water vapor (WVG) Sum of Squared Columnar Scientific Data Set 6 water vapor values Number of Columnar water Scientific Data Set 7 vapor pixels per bin File Description Annotation 8 Table 2 shows the flag used in the pentad and monthly atmospheric moisture objects. Valid calculated values are scaled before being stored. That is, the pentad or monthly cloud/rain liquid water values are multiplied by 10 and then stored as integers; so to retrieve a pentad or monthly cloud/rain liquid water values, you must divide the stored number by 10. Columnar water vapor values are multiplied by 100; perform a similar procedure by dividing by 100. Table 2 Flags For Atmospheric moistures ------------------------- | | FLAG 1 | ------------------------- | LWG | -10 | | WVG | -10 | ------------------------- where: LWG = Pentad or Monthly Cloud/Rain Liquid Water in mg/cm2 WVG = Pentad or Monthly Columnar Water Vapor in g/cm2 FLAG 1 = No atmospheric moisture data was accumulated in the bin. The following sections provide further details on each of the HDF atmospheric moisture objects. 2.1 Pentad (Monthly) Cloud/Rain Liquid Water Grid (LWG) This HDF object contains the pentad or monthly cloud/rain liquid water grid. The grid values are multiplied by 10 before storing to retain a precision of 0.1 mg/cm2. For example, 31.5 mg/cm2 is stored as 315. The minimum value is 0, and the maximum unscaled value is 1000. Flagged values, shown above, are stored in locations where the cloud/rain liquid water could not/should not be calculated. The array size of the grid is 360 x 180. Each value is stored as a 4 byte integer. The HDF data type used to store cloud/rain liquid water is DFNT_INT32. It will be necessary to use this type in any software written to access the data. The HDF reference number is 2. 2.2 Sum of the Squared Cloud/Rain Liquid Water Values (LWS) This object contains a grid of the sum of the squared cloud/rain liquid water values which are included in each bin of the grid. Daily valid values within a bin for the specified number of days were squared and summed. Flagged values, shown above, are stored in locations where the cloud/rain liquid water could not/should not be calculated. The array size is 360 x 180. The data are stored as a 4-byte integers and HDF data type DFNT_INT32. The HDF reference number is 3. 2.3 Number of Pixels per Bin (LWN) This object contains the number of cloud/rain liquid water pixels which have been grouped into each bin in the LWG object; only valid cloud/rain liquid water values( i.e., no flag values) are counted. The daily cloud/rain liquid water pixels are counted in each bin for the specified number of days and stored in this object. The array size is 360 x 180. The values are stored as 4-byte integers and HDF data type DFNT_INT32. The HDF reference number is 4. 2.4 Pentad (Monthly) Columnar Water Vapor Grid (WVG) This HDF object contains the pentad or monthly columnar water vapor grid. The grid values are multiplied by 100 before storing to retain a precision of 0.01 g/cm2. For example, 3.15 g/cm2 is stored as 315. The minimum value is 0, and the maximum unscaled value is 10. Flagged values, shown above, are stored in locations where the cloud/rain liquid water could not/should not be calculated. The array size of the grid is 360 x 180. Each value is stored as a 4 byte integer. The HDF data type used to store the columnar water vapor is DFNT_INT32. It will be necessary to use this type in any software written to access the data. The HDF reference number is 5. 2.5 Sum of the Squared Columnar Water Vapor Values (WVS) This object contains a grid of the sum of the squared columnar water vapor values which are included in each bin of the grid. Daily valid columnar water vapor within a bin for the specified number of days were squared and summed. Flagged values, shown above, are stored in locations where the cloud/rain liquid water could not/should not be calculated. The array size is 360 x 180. The data are stored as a 4-byte integers and HDF data type DFNT_INT32. The HDF reference number is 6. 2.6 Number of Pixels per Bin (WVN) This object contains the number of columnar water vapor pixels which have been grouped into each bin in the WVG object; only valid columnar water vapor values ( i.e., no flag values) are counted. The daily columnar water vapor pixels are counted in each bin for the specified number of days and stored in this object. The array size is 360 x 180. The values are stored as 4-byte integers and HDF data type DFNT_INT32. The HDF reference number is 7. 2.7 File Description This object is an ASCII description of the pentad or monthly AM grid file. The description can be read with the program "getfiledesc" supplied by MSFC, after you compile it with the HDF library. Section 3.0 contains instructions on how to get the HDF library and section 4 contains information on how to compile and create "getfiledesc". The HDF object reference number is 8. An example of a file description is: Wentz Atmospheric Moisture Products File ID = Moist.pen_87241_87245.hdf This is a LEVEL 3 product. This product is a 5-day composite grid, including Julian day 87241 through Julian day 87245. This grid includes 5 days of data. The grid is a 1-degree by 1-degree longitude/latitude grid; grid location (1,1), in the upper left corner, is located at 90 deg N latitude, 180 deg longitude. The gridbox covers 1 degree (90-89 N, 180-179 W) from that location. SSM/I PATHFINDER Software Version Number 1.0 MSFC File Structure Version Number 1.0 HDF Version Number 3.3, Release 3 MSFC Tool Set Version Numbers: extractmg 1.0 getfiledesc 1.0 This product was produced by NASA Marshall Space Flight Center. For more information on the technical content please contact Marshall Space Flight Center: Earth Observing System - Distributed Active Archive Center User Services; Phone: 205-922-5932, or via e-mail at: msfc@eos.nasa.gov 3.0 HOW TO GET THE HDF LIBRARY AND TOOLS This section gives the instructions for getting a UNIX version of HDF. 1. First connect to the NCSA ftp server. ftp.ncsa.uiuc.edu 2. Login as anonymous giving your Internet e-mail address as password. Get the README.FIRST and README files. They have most of the information referred to below plus other useful information. 3. Change to the HDF directory ftp> cd HDF There you will find several directories such as: HDF3.3r4 4. Change to the directory with the latest version, or the one you trust most. Any ALPHA release may not be your best choice. ftp> cd HDF3.3r4 5. Now you are almost there. Change directories once again to the tar directory. ftp> cd tar There you will find numerous files beginning with "HDFx.xrx.tar.Z.part-??". You can ftp the entire file from the directory using "HDFx.xrx.tar.Z.part". 6. Making sure you are in binary mode get all of these files. ftp> binary ftp> prompt ftp> mget HDF3.3r4.tar.Z.part* NOTE: You will want to come back to the top level directory and go to the Unix/Collage directory and change to the subdirectory which contains the version of collage you want to use. 7. Now return to your own system. ftp> bye 8. The files you just got contain the entire HDF library, examples, and utilities. You must concatenate them into one big file. cat HDF3.3r4.tar.Z.part*> hdf.tar.Z 9. Now uncompress the whole thing by typing. uncompress hdf.tar.Z This will yield a tar file called hdf.tar 10. Extract the contents of the tar file with the tar utility. tar -xvf hdf.tar This will create several directories including an HDF directory. 11. After the tar finishes, there will be a series of Make files in the /HDFx.xrx/hdf subdirectory. There should be one that corresponds to your type of machine. You will still have to do a little editing of the makefile to tell it where some files are, but there are ample comments to help you through. Also read the INSTALL.TOP file for further help. When you have completed the editing, run the makefile. This will install everything creating the lib/libdf.a directory and file. (Note: All utilities may not make. The team has had trouble with a few of them. You probably will not need them anyway. Just be sure that you get the libdf.a file.) When you have the lib/libdf.a file created you have the HDF library. This can then be used to compile the extractrg.c, which is included with this release. 4.0 HOW TO COMPILE THE PROGRAMS EXTRACTMG and GETFILEDESC Two programs are included with this distribution. They are extractmg.f and getfiledesc.c. Both run on the Silicon Graphics but should port easily to other platforms. A "makefile", named "Make.am.3", is included with the distribution. It can be used to compile the programs on a UNIX platform. After making changes to the directory paths and flags, use the following UNIX command to run the make-file program: make -f Make.am_3 all This will compile both programs, and two executables will result: "extractmg" and "getfiledesc" 4.1 EXTRACTMG The program, extractmg.c, extracts an HDF object from the pentad or monthly file. It produces a new HDF file with the name of the HDF object selected. The format for using the utility is: extractmg For example, the command extractmg Moist.pen_87241_87245.hdf LWG will produce a file call LWG_pen.87241. Run the program with no arguments, and it will list all of the options as shown below: ACRONYM FILE Object Type Variable Type --------------- ------ ------------ -------------- LWG CLoud Liquid Water Grid SDS INT32 LWS LWG Sum Of Squares Grid SDS INT32 LWN LCG Number of Pixels in Bin SDS INT32 WVG Columnar Water Vapor Grid SDS INT32 WVS WVG Sum Of Squares Grid SDS INT32 WVN WVG Number of Pixels in Bin SDS INT32 where SDS is Scientific Data Set and INT32 is a 32 bit integer. Use the code when selecting an object to extract. You may select several codes at once, separated by a space. 4.2 GETFILEDESC The getfiledesc program prints the text of the HDF annotation contained in the pentad or monthly file. Its usage is getfiledesc < Pentad or Monthly AM filename>. The result should look like the example file description found in section 2.7, File Description. --------------------------------------------------------------------------- Attachment 1 Atmospheric Moisture Atmospheric moisture products contain cloud/rain liquid water and columnar water vapor. The correction to cloud/rain liquid water identified by Wentz (1992) was applied to the Pathfinder data sets. A value of cloud/rain liquid water equal to or greater than 18 mg/cm2 is a qualitative indicator of rain. These parameters are calculated over water areas (plus coastal, sea ice, and possible sea ice areas) using the algorithms developed by Wentz (1989, 1992). Because Wentz cautions that parameters calculated over fresh water lakes are suspect as are those within 50-100 km of the coast, values over the open ocean or seas are the most reliable. Values over coastal, possible sea ice, and sea ice areas are calculated, but are flagged because of the questionable validity of the algorithm over these areas. The liquid water and water vapor products are stored together in a daily file, along with a browse image of each product and low resolution latitudes and longitudes. Non-flagged daily Pathfinder atmospheric moisture parameters were accumulated into separate 1 degree longitude and latitude rectangles for the defined period, pentads or monthly. These binned amounts were divided by the number of non-flagged good values in each bin to give the average for each rectangle. The atmospheric moisture values were also squared and summed. Bins in which no valid pixels are located are set to -10. Wentz, Frank J., 1992, Measurement of Oceanic Wind Vector Using Satellite Microwave Radiometers, IEEE Transactions on Geoscience and Remote Sensing, vol. 30, pp. 960-972. Wentz, Frank J., 1992, Revision-1 Update For SSM/I Geophysical Tapes User's Manual, RSS Technical Report 040792, Remote Sensing Systems, Santa Rosa, CA, 11pp. Wentz, Frank J., 1989, User's Manual SSM/I Geophysical Tapes, RSS Technical Report 060989, Remote Sensing Systems, Santa Rosa, CA, 16pp. --------------------------------------------------------------------------- Attachment 2 Anomalous Pentad and Monthly Atmospheric Moisture Files 1987 DMSP F8 Satellite The pentad for July 30 through August 3, 1987 was not generated because Pathfinder daily precipitation rate information began on August 1, 1987. Pentad for the period October 3-7,1987 (87276-87280) has 1 day of data missing. The December 1987 monthly file was not generated because the SSM/I instrument was turned off on December 3, 1987 due to overheating. Also, pentads starting on Dec 2, 1987 (day 336) through December 31, 1987 were not generated for the same reason. 1988 DMSP F8 Satellite The pentads for the periods January 1- January 5, January 6-January 10, January 11 - January 15, 1988 was not generated because the instrument was not turned on until January 13, 1988. The January 1988 monthly file was not generated. The following pentads contained less than 5 days of data: May 6-10,1988 (88127-88131) has 3 days of data missing. September 23-27, 1988 (88267-88271) has 1 day of data missing. December 22-26, 1988 (88357-88361) has 2 days of data missing. December 27-31, 1988 (88362-88366) has 1 day of data missing. [Image] [Image] Disclaimer [Image] E-mail to User Services [Image] MSFC DAAC Home Page NASA Information Contact: Cathy Lapenta, DAAC Manager MSFC DAAC Web Curator: Danny M. Hardin, DAAC WebMaster Author: Beverly Burns Last update: (August 15, 1995) Accesses Since (August 15, 1995): 14