Page History

David Landis

Dan Slayback

Greg Ederer

Fritz Pollicelli

Diane Davies

• Cloud shadow mask layer
• Continuing to incorporate new code in to the existing code.
  • ran some compiler tests. bunch of errors that need to be fixed.
  • need to talk about existing flood code // variables that haven't been declared properly
  • not sure if it is just sub-routines need to be in specific orders. Greg thinks it might be an environment issue. missing some library.
  • other than that things need to be doing well.
  • will need to talk to Dan about the logic in the code. David has been simplifying the code as perl can do more than the IDL code.
  • NEXT STEPS: testing the code on full MODIS products. within a month or two should have testing complete and ready
    • unit testing - what David is doing now. small tests on algorithms to make sure they are performing the right operation
    • integrate well - larger testing at test file level:
    • integration testing - production testing: test it on a handful of time selected intervals to test over a range of inputs. this usually involves making a release to production. the production team runs it and keeps output private for evaluating.
• Then it will be Flood Detection - is the water we detect flood or not. Usually done by comparing to a water mask. Look for differences.
  • eventual output will produce geotiff images
  • part of the compositing.
  • need to decide what you want as output products // want to replicate current product which is a geoTIFF with 4 raster values. talked about modifying / improving this but that is in the future.
  • Fritz? what about shp files? takes 95% of processing time. don't provide all the information. want to loose them.
• Follow up question from Dan:
  • another month or two for testing on the cloud shadow?
  • fixing problems that crop up initially
    
  • how long until it is in production? Next big piece is compositing code. Should be straight forward in terms of code - it is probably more of a data management issue, that is handled by the production system. Just a matter of writing the perl wrapper code. Hard to estimate. optimistically it is a week or two but nothing has gone smoothly. As we go through the test process will be good for David to see how the mechanics of the production system work. Hopefully he will have a better sense of how it works for compositing part.
  • Summer?
• Follow up from last time:
  
  • Concluded that it should be grid data. are there different grids for different time periods? using own grid which is not standard MODIS grid.
  • Whatever grid - it would be done before flood code implements.
  • Grid - projection or Grid - physical size array? Projection will always be the same
  • currently 10 x 10 code but moving forward need to decide. this will be larger than a 10 x 10
    • there are different ways to do this.
    • Fritz - one goal was to avoid waiting for 10 x10 deg grids. it should operate on granules as they come in.
    • if final out put is a 10 x 10 tile when do you say it is complete? is there another granule coming along?
    • 10 x 10 grids are currently at 250m. these could be same grid as GIBS uses. Action for Dan to send these notes to Dan.
    • what makes it more efficient in the system?
      • Decision for David and Greg - it is a bit beyond where we are.
      • need cloud shadow mask first
        • then how do we compare what we have with history to tell if there is a flood there?
        • that is when geolocation comes in to play to be able to determine if there is a change
        • at that point we can adjust things on both sides so comparison is as simple as possible
        • also need to line up with mod44 water mask? - Fritz / could be a different water reference layer.

David Landis

Dan Slayback

Greg Ederer

Gang Ye

Diane Davies

• With the shutdown. David was not allowed on to GSFC and not able to access his computer. No VPN set up
  • he was able to go in yesterday and get some files, so should be able to carry on.
  • Greg said he can work at SSAI Aerospace building. They have connections that don't require VPN set up
  • away next week in Florida but after that David should be able to start testing again.
    
    

PROGRESS since last meeting

• Testing of PDS commands to replace the IDL commands
  • • some of the perl algorithms enables you to do more with less code, for example by dealing with image border issues automatically.– This is allowing David to streamline some of the code.
    • looks like all files that go in to cloud shadow are 1km apart from SR data, which is only being loaded to get the dimension.
    • one reason to process cloud shadow at 250 m resolution is that it would  then be simpler to drop in an improved cloud mask in the future, such as might be developed from the 250m reflectance 250m data (down the road)

Question on whether the code should operate on swath or gridded data:

• • • Current algorithm takes gridded data for SR, cloud, cloud properties – not swath.
      • when writing code David should assume he will get gridded inputs.
      • will be working on gridded swath imagery, so it will be larger than current 10 x 10 box
    • Currently set up so the output for given day is produced on the grid for Terra SR file for that day. Thus Aqua and files for previous day are re-gridded to match the Terra SR grid.
      • For LANCE processing, we could do the same, OR we can use a fixed global grid for all products.  
      • Greg: code allows you to do either, so that option is available.
    • Dan asking if Greg could run some test data to map out test granules at native vs fixed grid
      • action @ Dan to send a tile / AOI and Greg will order the data

Question re: funding and the shutdown.

• • • Greg said the relevant HBS project tasks were under review by GSFC, but given the shutdown this has not occurred. Works is progressing following a letter saying the work was approved through Feb 1st. Not clear if they will extend it again.

Next meeting 20 Feb 2019

David Landis

Dan Slayback

Fritz Pollicelli

Greg Ederer

Gang Ye

Diane Davies

• Almost done converting code - cloud mask. taking flood code using it as a shell and using it to covert idl → perl
• (cloud layer is from MOD35 = mask)
• Working on data structures
  • Talking to greg about how data arrays are handled (rather than the perl format) - the standard would
  • David is lookin up perl data language (PDL) options on line. Functions that read in the HDF data - read it in a PDL language.
  • He is looking at the official PDL website. Not ventured out to look at other sites.
• Hoping for a few months. Depends on how quickly he can get the dilation code working (filling in the holes). At each place you have a cloud get height from cloud top temperature. So need complete arrays and it is patchy. Need to smooth it out to make is spatially complete. Erosion and dilation are Dan's attempt to expand the cloud top temperature. is a simpler way to do this? looking at patching functions in a 2d array - nearest neighbor averaging for non-bad pixels. some handle bad data values and some don't.
• Issue is no-one knows how the functions work and so it is trial and error. Documentation is not good so it is a slow process.
• If python was allowed this would be much quicker however for now the SA will not allow this. Currently the MODAPS machines only use code written in C, Perl or Fortran.
• This maybe an issue for LANCE moving forward if all science code has to be re-written.
• It will be a few months until this is finished.

Meet again in 2 weeks. Need to prepare an update for LANCE UWG.

Dan Slayback

Fritz Pollicelli

David Landis

Greg Ederer

Gang Ye

Diane Davies

• David converting IDL code to Perl. Good headway. some snags in trying to find the proper function for some of the more sophisticated IDL routines. (congrid, morph, dilate, quadrat)
• Working with Greg yesterday and some questions for Dan.
  • may be some work arounds. trying to fill holes in all layers / to project the cloud.
  • several routines in perl that do similar task
• Move to testing specific routines to see if they do similar things – perl equivalent code. Still more code conversion.
• Next steps: to get IDL routines mimicked in Perl. or writing simplified version that can do the hole filling. then get it on to the server to test the code – running it through the interpreters.

Next meeting 24 October 2018

Dan Slayback

Fritz Pollicelli

David Landis

Greg Ederer

Gang Ye

Diane Davies

Dan Slayback

Fritz Pollicelli

Greg Ederer

Diane Davies

Gang Ye

Brief meeting. David Landis has been hired to work on transitioning the code to Pearl.

Dan to drop by and see him in Building 33 G316

Dan and Greg to try and set up weekly meetings with David to touch base on any issues that might arise.

Plan to meet next Wednesday 29th if David is free.

Dan Slayback

Fritz Pollicelli

Ed Masuoka

Diane Davies

• Current Status: Step 1 is complete. The initial swath based water product is running as a PGE and Dan has checked the out put product and is happy with it.

Next steps

• Greg is going to amend the PGE to include a cloud shadow mask. Dan and Greg are discussion whether to use the existing cloud shadow mask (likely) or whether to use a different one.
  • Dan needs to provide Greg with the code for the existing cloud shadow mask that uses MOD35.
    • the current code Dan uses has IDL operators. this will need to be re-written.
  • Diane and Dan to meet with Greg this week to discuss next steps this.
    
    
• Dan to provide an outline for code to composite and tile the initial product. Dan to work with Greg who will write the code.
• Terrain shadow filter (monthly product computed as part of this process) is to be applied to the tiled product
• HAND (Height Above Nearest Drainage) filter, which identifies areas where flood water should not be present, will also be applied to the tiled product.

Dan Slayback

Greg Ederer

Diane Davies

• Greg amended PGE code and has completed step 1. Dan happy with the product
• Discussion on how to implement the cloud mask.
  • looking at options
  • this should be done as part of the PGE running in step 1
  • Diane to contact Eric Vermote to see if there was an improved cloud shadow mask. He said
  • "There is a cloud cloud shadow mask in the MOD09 product (at 1km resolution), at this resolution it is very difficult to evaluate cloud shadow mask and I am not sure about the performance. 
    

    Cloud shadow masks are currently being evaluated for Landsat 8 and Sentinel 2 data (using an operator derived  validation mask) we can probably gain a lot of experience from this and then transition the “best” algorithm to MODIS the best way we can. That will not be right away unfortunately but more like in a year timeframe,

    As far as terrain shadow it is an interesting problem quite feasible if you have a good Digital Terrain Model but could be computationally expensive because my guess is that you have to do it  at a finer spatial resolution than what your actual resolution but that can be studied too once again with Landsat 8/Sentinel 2 data.