From Chris Lindsley at ORNL:

I am working on inputing metadata for the EVS-1 CARVE project from the ORNL DAAC into CMR.  For CARVE we have the flight path for each granule.  I was hoping to input that flight path into the CMR as a polyline.  I don't see polyline as an option in the UMM documentation.  Do you know if there is a way I can do that?

I was looking at some metadata from icebridge and it looks like they input a very thin polygon which at lower zoom levels looks like a polyline.  Unfortunately this kind of solution requires double the points.

Do you happen to know what I should use as an upper limit of points in either a polyline or polygon?  Our raw data has 30,000 or so points for each flight.  That's obviously too much, I'd like to know what kind of subsetting I'll need to do.

  • No labels

17 Comments

  1. user-7b92a

    I would use 100 points as a maximum for a polygon or a line. We haven't done extensive testing on the performance implications of adding granules with lots of points but I do know that this is more performance intensive as a lot of our intersection algorithms are in O(n) time or worse. If 100 seems too few to properly represent your data let us know and we can do some testing to determine a more exact number. You want to choose just enough points so that if someone is spatially searching for data in that area they'll get accurate results. 

    We do support defining granules with multiple lines. If you're using ECHO10 you would specify the spatial metadata with multiple lines:

    <Spatial>
      <HorizontalSpatialDomain>
        <Geometry>
          <Line>
            <Point><PointLongitude>0</PointLongitude><PointLatitude>1</PointLatitude></Point>
            <Point><PointLongitude>180</PointLongitude><PointLatitude>-90</PointLatitude></Point>
          </Line>
          <Line>...</Line>
          <Line>...</Line>
        </Geometry>
      </HorizontalSpatialDomain>
    </Spatial>

  2. Chris Lindsley

    Thanks for the information Jason.  I don't think 100 points will be enough for most of our flights.  Here is a visualization we're working on that shows one example of a flight.   https://carve.ornl.gov/visualize/

    I wonder if there might be some other way to visualize the flight path in the Earth Data Search Client rather than trying to use the spatial CMR information.  Any thoughts? 

    1. user-7b92a

      I've filed a new issue, CMR-2505 - Getting issue details... STATUS , for us to investigate this. I think we're going to need to make our spatial indexing smarter such that a granule can have hundreds or thousands of points without impacting spatial search performance. Can you give us some details so we can properly plan for this? How many granules do you have to represent? Do they all have about 30K points each?

  3. Chris Lindsley

    It looks like we'll be ingesting about 5000 granules for the CARVE project and probably a similar number for the AirMOSS project.  Almost all of the granules will have flight path information.  A single flight path will be repeated across several granules.  I can't imagine that we would ever want to send you all 30,000 points for each flight.  We're researching tools that would let us trim down the number of points in the most logical way.  One option that we're looking at is NSIDC's MetGEN tool.

    I'm going to include a little of my research here just so we have an example of how other flight related data is handled.

    I've been searching for IceBridge data in EDSC and I see that they have broken their flights up into several granules based on time.  The search below shows a granule for every 4 minutes of the flight.  I'm guessing they broke the flights up for reasons other than CMR, but it does have the side affect of making the polygons much simpler.

    https://search.earthdata.nasa.gov/search/granules?p=C187690352-NSIDC_ECS&g=G190363779-NSIDC_ECS&m=75.66733534887643!-58.78090824493091!4!0!0!&tl=1440453591!6!!&qt=2009-04-05T00%3A00%3A00.000Z%2C2009-04-06T23%3A59%3A59.000Z&q=icebridge+atm

    Thanks,

    Chris

     

    1. user-7b92a

      Can you give us a timeline for when you want to ingest this data so we can prioritize the issue? Are you ready to do it now?

      1. Chris Lindsley

        It will probably be a couple months before we're ready to start ingesting into the production CMR.  We're still working on getting all the metadata loaded in our database.  Next will be to start sending to the UAT CMR so I can work out the kinks in that delivery.

        If inputting the points in CMR doesn't end up being an option, would having a GIBS image of the flight path overlayed onto EDSC results be an option?  That wouldn't help with the search result accuracy, but would be a nice visualization.

        1. user-7b92a

          I think Patrick Quinn is the right person to answer your EDSC visualization question. I've forwarded it to him to take a look.

  4. user-7b92a

    Thanks for the information. Please let me know when you get a chance approximately how many points you'll be using per granule so we can test with that if necessary.

  5. user-7b92a

    I came up with an idea for how to improve performance for polygons with many points while working on some performance improvements for Ice Bridge granules.  CMR-2681 - Getting issue details... STATUS  We'd simplify the polygons in a way that maintains spatial search accuracy to improve spatial intersection performance.

  6. user-6e18a

    Hi Chris and Jason

     

     The SIPSMetGen utility that I helped create, and Michael Stanley programed is used for aircraft missions for Icebridge and has the capability to reduce millions of data points into concave hull polygons usually with under 100 points. The utility has configuration parameters that can  be adjusted that will allow you to do some tuning of the output to balance the number of output polygon points vs how closely your data are represented in the resulting polygon.

     Chris, can you send me the spatial coordinates for one of your files and I'll send you the output to see if it meets your requirements? If it does then I can help you get the utility running for your own use.  The polygon is spit out into a ODL or XML metadata container file,  but it would be easy to harvest just the polygon out of those files if you don't want the rest of the metadata output from our utilities.

     

    -Mark Schwab (schwabm@nsidc.org)

    1. user-6e18a

       

      I'll even help script up a shell utility to harvest the resultant polygon out of the odl output files if you just want to

      use our tool for polygon generation.

       

    2. user-7b92a

      Thanks Mark. I'll email you the data.

      1. user-6e18a

        Chris just asked me how to format it to send, so unless you're sending it for him. Hold off, It's probably on its way already.

        1. user-7b92a

          I just sent you an email. I wouldn't worry about involving Chris on this particular issue as he's not working on search performance or spatial code at the moment.

          1. user-7b92a

            I'm sorry. I thought you were talking about Chris Durbin. I realize now you meant a different Chris. That was my mistake.

  7. Chris Lindsley

    I sent the sample to Mark.  If this works well, I would like to take Mark up on the offer to get the resulting points into a format that can easily be imported into our database. That will then be used to create the ECHO10 granule metadata for CMR.

     

  8. Chris Lindsley

    I just realized that I never updated this thread with what I ended up using for our CARVE flight paths.  A conversation with Patrick Quinn led me to this algorithm:  https://en.wikipedia.org/wiki/Ramer%E2%80%93Douglas%E2%80%93Peucker_algorithm

    I found a perl implementation at http://www.usnaviguide.com/douglas-peucker.htm

    This code reduced my points from ~30,000 to around 500 with no visual loss in accuracy.  Performance in the CMR is quite good.