EarthData has developed an in depth production process
for processing GeoSAR data to meet NOAA's
specifications. The following outlines the GeoSAR
acquisition and processing workflow that was
designed to produce the final product deliverables.
1. PROJECT PLANNING.
The success of this project depended on effective
management of all components of the project. To support
this effort EarthData identified an experienced
management team to oversee all aspects of the project.
This team was responsible for ensuring that all project
requirements outlined in the SOW and any subsequent
amendments were met and that an open
communication/reporting line was maintained with all
participants.
2. GeoSAR ACQUISITION AND PROCESSING WORKFLOW.
This section describes the general workflow for the
GeoSAR system for the planning, collection, processing,
and mosaicking of GeoSAR dual band IFSAR products.
2.1 Collection & Processing Requirements Production Steps
Six major activities take place in this production step.
A. Ground Control requirements are derived from the
Project accuracy and QC specification in combination with
the data take flight lines. The need for in situ corner
reflectors, kinematic GPS profiles, and mosaicking control
drive this task.
B. Kinematic GPS is routinely specified for collection
(where cost effective) to provide an independent means of
quickly verifying the end mosaic product. This is an
important part of the overall QA process, since this data is
sequestered from the mosaicking staff until after the mosaic
is completed. Kinematic GPS data is frequently collected
prior to deployment in conjunction with the site
survey (see item-D, below).
C. GPS Base Station Location candidate sites are
selected from the existing High Accuracy Regional
Networks (HARN) in the project area, and the High
Precision Geodetic Net (HPGN) in California. One or more
such locations are usually occupied as the base stations
during the flight mission. If such a station is not available
within or close to the project site, then either an existing
HPGN densification station is used or a new station is
established through static GPS survey connected to
HPGN stations and meeting the FGCS Order-C, Class-I
relative positioning standard.
D. Site Surveys are usually conducted several weeks to
several months prior to deployment to identify reflector
ground control locations, identify and resolve deployment
issues (obtain permission or permits), and to verify that the
GPS base station receives a high quality signal at the
selected primary and backup locations. Kinematic dGPS
traces are frequently obtained at this time using one of the
base station locations.
E. Detailed Acquisition Planning generates a complete
acquisition plan including final notched frequency
waveforms based upon coordination with the Army
Spectrum Office. Flight lines are fully specified and the
associated run-time data packets finalized and checked by
the Project Manager and the Radar Operators for accuracy
and completeness.
F. Sortie Packing is the final planning step prior to
initiating the Deployment & Acquisition of the raw radar
data. In this step the various data take lines are bundled
into a flight (sortie) and the corresponding logistical support
requirements are determined (e.g., ground control, media,
contingency supplies, etc.). The sortie schedule is
specified, crews hired, personnel reserved, suppliers hired,
and support material inventoried.
2.2 Deployment & Acquisition of Raw Data Production
Steps
Three major activities take place in this production step.
A. Mobilization initiates the acquisition process. If the
aircraft requires alternate basing, then it is moved to this
base. The aircrew, the Radar operators, and the In-field
Acquisition Manager secure local lodging and prepare for
the initial sortie. The GPS base station and ground crews
are deployed to the region, where upon they install the
specified reflector ground control, check GPS equipment
for proper operation, and secure local lodging. All supplies
required to support the acquisition are propositioned.
B. Sortie Generation consists of five basic steps, each
requiring close coordination by the In-field Acquisition
Manager. Some of these steps may not pertain to every
sortie.
a. Reflector Deployment-if not already in place, the
specified corner reflectors are positioned and orientated
per the sortie plan.
b. Ground Station Deployment-the GPS ground crew
deploy to their specified locations, install the monitoring
equipment, and start collecting local GPS data at least 30
minutes prior to "wheels-up" and continue recording until
notified of "wheels down".
c. Preflight Logistics-sortie flight plan is filed, sortie supplies
loaded, aircraft and radar equipment preflight checked,
verify GPS ground crew in position.
d. Data Takes-fly the specified plan. If problems arise, use
best judgment to extract the maximum value from the sortie.
e. Post Flight Logistics-notify GPS ground crew of landing,
offload recorded data takes and auxiliary sensor data files
and media, update operators' log, debrief In-field
Acquisition Manager on mission, ship (as appropriate) data
takes to production facility.
This process is repeated for every sortie. Radar hardware
failures and/or aircraft maintenance issues are resolved as
quickly as possible and the sortie schedule is adjusted
accordingly.
C. Demobilization commences upon completion of the last
sortie. All staff and aircraft return to home base. Post
deployment inspection and maintenance are performed.
2.3 X-band and P-band Data Processing Production Steps
Seven major activities take place in this production step.
A. Data Take Ingestion is the process of incoming
reception and logging of the data tapes and auxiliary data
sent from the field into the production database. The base
station GPS data is combined with the aircraft GPS data to
create a dGPS location of the aircraft relative to the base
station ground control point. This data is entered into the
production database.
B. Motion Measurement Processing (MMP) examines the
auxiliary data for motion quality and prepares the
parameters necessary for transferring the raw data off of
the tapes. MMP is the beginning step of the Ground
Processor. The motion data is sent through a QA process
to verify its quality. The MMP inputs the dGPS aircraft
position location data and combines it with the auxiliary
antenna motion data to generate a Time Varying Parameter
(TVP) file for the data take, which is used by the X-band
and P-band processors to motion compensate the raw
data.
C. Tape Transfer is based upon data obtained from the
MMP process, which identified what data is located where
on the tape. The Tape transfer process requires 3 to 4
times real time to complete the transfer from the
high-density Sony tapes into a format suitable for
processing.
D. X-band Swath Processing inputs operator or MMP
specified parameters and outputs a coregistered X-band
reflectance image and DEM at the specified ground
sample distance. This is a computationally intense process,
limited to about 4-quads/hour throughput.
E. P-band Swath Processing inputs operator or MMP
specified parameters and X-band DEM (or other suitable
DEM) and outputs a coregistered P-band reflectance
image and DEM at the specified ground sample distance.
This is a computationally intense process, limited to about
2-quads/hour throughput.
Steps 2, 3, and 4, while sequential, can be executing in
parallel on different swaths. Up to a dozen different swaths
may be in processing at single time.
F. QC Swath occurs when the radar processors have
completed their reduction of the raw data into reflectance
and DEM swaths. This is a manual operation where each
swath is examined for processing anomalies, such as
phase-unwrapped regions, ambiguity jumps, of noisy data.
Approximately 20% of this data will be reprocessed with
different input parameters to mitigate the anomalies.
G. Ready to Mosaic is the final QC check of the
processed swaths to ensure that all the available data has
been processed correctly and enough data is on hand to
generate a composite mosaic for (a large portion of) the
project area.
2.4 Wide Area Mosaicking Production Steps
Three major activities take place in this production step.
a. Segmentation-chop the swaths into segments
containing usable, eliminating sub spec data (usually a
result of severe motion artifacts due to turbulent weather
during the data take).
b. Swath De-Tilt-remove residual linear tilt in range for the
swath (this is minimized by calibration, but might be required
on wide swaths).
c. Region Match Exclusion-mask out large water regions
so they do not generate spurious interswath match points
(this region is usually defined during the flight planning
process).
d. Control Point Extraction-using ground control
information, extract the position of the radar corner
reflectors to a fraction of a pixel.
e. Swath Point Matching-in the overlap regions of any two
swaths, find all reflectance (and/or DEM) points which
correspond to the same point on the ground. This process
generates hundreds to thousands of match points per
swath-pair.
f. Match Point Culling-severely cull the match points to
retain only the very best, highly correlated points.
B. Mosaic Affine Transformations is the process of
combining the match points (produced by item-f above)
and the ground control points in a weighted lease mean
squared error swath-by-swath affine transformation
for geographic registration to generate a composite mosaic of
the region. Data from overlapping swaths are averaged in
common areas and feathered at the boundaries. The same
transformation parameters are applied for both the
reflectance and height data. Null values in one swath are
replaced by non-null values in overlapping swaths.
Orthogonal tie lines provide powerful near range
constraints to remove residual tilts. Ground control points
remove residual systematic vertical bias (z-bump).
C. QC and Mosaic Statistics are collected for the
composite mosaic. Independent ground control is used to
estimate the overall quality. Once the mosaic is declared
completed by the mosaicking staff compare the kinematic
GPS data values collected during acquisition against the
mosaicked values to assess independently of the
mosaicking statistics the overall quality of the final mosaic.
If the mosaic is satisfactory, i.e., there are no major
blunders, then optionally the kinematic data can be used
as additional constraints to improve further the final
product. If the result is not satisfactory, then the anomalies
are noted and the composite mosaic is redone (usually from
the swath point matching or from the affine transformation
step.
2.5 Product Finishing and Packaging Steps.
Seven major activities take place in this production step.
A. Crop to Quads chops the composite mosaic into units
suitable for ingestion by the DEM and Image editing
workstation.
B. Transfer data to product finishing team.
C. DEM Edit is the manual process for removing residual
radar artifacts from the DEM according to NOAA's
specifications.
D. Image Edit is cosmetic smoothing or removal of radar
artifacts to enhance the photo-like quality of the imagery.
E. QC is the final check on inherent data quality. Data,
which fails to meet the delivery spec, is sent back to
the processing team for remediation.
F. Formatting established the data onto NOAA's specified
media and formats.
G. Delivery is the process of shipping the data to NOAA
and following up that the data is found to be satisfactory.
3 DEM AND IMAGE CLEANING AND PRODUCT FINISHING
The following provides an in depth description of the steps
involved in this process.
A Pre-Processing Data Review.
Once the data acquisition and processing/mosaicking
phase has been completed the data will be provided to the
product development and finishing team to continue
the processing and delivery development phase.
B Data Preparation.
Prior to beginning the processing and delivery development
phase, all data will be processed into the workflow
structure designed to support the delivery requirements.
Data will be downloaded and placed into a folder or
database. The downloaded data will then be transferred
into a job folder where it will be extracted and cut into tiles
roughly the size of a 7.5-minute quad. Automated data
validation software routines will be run on the data to
check for gross errors and anomalies. Once all of the data
is validated it will be processed through an additional
control check. A series of supplemental control points
gathered during data acquisition will be utilized to provide
an independent accuracy assessment and RMSE
validation prior to beginning the final production phase.
Once all of the data has been accepted, it is placed in a
secure folder/database location to be used for the final
processing and delivery development phase.
C Image/DEM Edit
The delivery development phase in broken into the Image
edit and DEM edit phase. Each phase is made up of a
series of steps designed to meet the delivery requirements
outlined in the project SOW. The following provides an
outline of the processes used to develop each set of
deliverables.
a. Image Edit.
Our image edit process has been developed to meet the
requirements for imagery deliverables outlined in the SOW.
EarthData shall provide orthorectified radar
reflectance imagery for the area of the main task order.
(1) Imagery shall be in GeoTiff format and tiled with
no compression to fit the 7.5-minute USGS quads.
(2) Imagery shall have a horizontal resolution of 3
meters and a horizontal accuracy of 2.5 meters.
(a) Translate to Tiff format.
Data is withdrawn from the secure accepted database file
location as needed and prepared for translation.
EarthData utilizes proprietary software to coverts the
existing Magnitude Image (or reflectance image) files to Tiff
format. The converted data is visually viewed for
completeness and prepared for the next step.
(b) Merge tiles into final sheets.
Based on the project limits EarthData will develop a tile
structure that conforms to the limits of 7.5-minute USGS
quads. This tile structure will be used as the template for
merging tiles into the final sheets.
To support this effort EarthData will utilize existing
commercial off-the-shelf (COTS) and proprietary software to
convert the Tiff format data into clipped and merged final
7.5-minute quad limits. The converted data is then
reviewed for completeness and prepared for the next step.
(c) Edit voids/PhotoShop.
The GeoSAR preliminary flight planning and processing
steps are intentionally robust to reduce and minimize the
occurrence of void areas within the data. However, some
limited void areas are typical for IFSAR-generated data.
EarthData will utilize COTS and proprietary software to
identify and fill identified voids. Each void will be
individually reviewed and corrected during this process. At
the end of this stage the base image product is complete
and ready for input into the database.
(d) Update index or database.
A final QC of the data is performed on the completed data
prior to input into the index or database. Based on
acceptance, the final data is merged into the existing
database structure. At this point the data is ready to start
the next DEM edit phase of production.
b DEM Edit.
Our DEM edit process has been developed to ensure
adherence to requirements outlined in the SOW. These
include.
(1) EarthData shall deliver DEM data in an ESRI
floating point grid format with a 3.0-meter cell size. The
data shall be split into tiles corresponding to USGS 7.5
minute quads.
(2) Delivered reflective surface DEM data shall have a
vertical accuracy of +/- 1.5 meter RMSEz or better.
EarthData shall provide our "best effort" to meet vertical
accuracy requirements within urban corridors (dense
buildings of greater than 2-3 stories) due to shadowing and
"layover" within these areas.
(3) Elevation points used to generate the DEM shall
have a horizontal resolution of 3 meters and a horizontal
accuracy of 2.5 meters.
(4) Data shall be delivered on DVD.
(a) Digitize water polygons from Tiffs
Based on the acceptance of the Tiff formatted data,
EarthData technicians will interactively identify water
polygons that require digitizing. All stream channels will be
digitized for leveling if they exceed approximately twenty
(20) meters in width. Open water bodies five (5) acres or
larger will be digitized to level water in these areas.
(b) Drape water polygons on DEMs.
Using COTS and proprietary software the 2D collected
water polygons are draped over generated DEMs. This
process assigns each element with a "Z" value.
(c) Fill water polygons with lowest elevation
Using proprietary developed software the draped water
polygons are imported and grids are dropped in to level to
the lowest elevation within that polygon. A void map is
generated for all edited voids. Color codes are used to
identify void type, i.e. water, terrain.
(d) Generate void polygons and fill.
Using our proprietary developed software routines any
remaining voids are identified and a polygon is generated
around each void. Using the poly fill routine the software
fills any remaining voids with an average of the surrounding
data. A void map is generated for all edited voids. Color
codes are used to identify void type, i.e. water, terrain.
(e) Using TIN, Image and contours fix wells and spikes.
During this step we generate a TIN and contours. Using
the TIN and contours along with the magnitude image we
identify and correct any wells, spikes and other anomalies.
(f) Final Q/C of DEM using contours/TIN and Magnitude
image
An independent review of the data is conducted at this
point to ensure that all wells, spikes and other anomalies
were identified and corrected. This independent review is
conducted by someone other than the technician who
worked on that area.
(g) Generate final DEM deliverable.
Based on the acceptance of data from the final QC
process the final DEM is generated.
4 DELIVERABLE GENERATION
The Image edit and DEM edit phase prepare data for final
deliverable generation. Based on final data acceptance,
the data flowing from each phase will be formatted to an
acceptable data delivery format and prepared for delivery.
Final deliverables include.
(1) Digital Elevation Model (DEM),
(2) Raw Magnitude radar imagery (Mag).
Supplemental data consists of
(1) Height error layer (HEL),
(2) Void mask,
(3) Water mask, and
(4) TIFF (8 bit raster of Magnitude imagery)
Final deliverable formats consists of
(1) DEM - ESRI Float Grid Format
(2) Mag - IEEE 32 bit Float Format
(3) Void Mask - ESRI Shape File Format
(4) Water Mask - ESRI Shape FIle Format
(5) Mag - 8 bit Raster TIFF Format
5 METADATA
Project level metadata is provided for each phase of the
project.
