Page History

Dan Slayback

David Landis

Greg Ederer

Diane Davies

David - Greg and David made good progress and got through various problems in the code.

About to start tweaking to produce the first 3:2:1 Day composites so they have a library of them to generate the final products.

However, Dan said This would be different to what is being generated. Need to read MODWater every day - this makes things simpler.

Code will be updated to use water product from all three days.

Next steps: 

• David and Greg will get prototype products generated. just at the point where starting to put this together.
• within a week should have prototype products coming out that Dan can look at. 
• Greg asked what Dan uses to define standard water: MOD44W - single static layer. Needs to be updated but should remain a static layer e.g. generated from last 5 years?
  • action @Gregto look at MOD44W.
  • action @Dan to check his notes on what he did.
• Greg to work on PGE production rules with Neal and Sadashiva
  • talking with Neal so have narrowed this down.
  • todays info will simplify this
• Dan's asked how the process would be organized it so granules processed as soon as possible. If time stamping these, is there value in holding on to older version once newer version has been generated? Do we overwrite the old with the new? or save the old as it takes up some more space. Agreed we should overwrite the older file

Greg will be working with Neal on production rules: take as many inputs as you give it and create tile as output. 

Robert talking about every 3rd orbit. others talking about sessions. as soon as granule becomes available. So need to work on production rules or use separate task. RR task - in order to update the fire data. Either way - would work. Probably opt to stick with PGE. Modify code so PGE processes granules as often as they comes in.

Dan Slayback

David Landis

Greg Ederer

Fritz Pollicelli

Diane Davies

Flood Notes - edited by Dan and Greg

Ryan Boller

Karen Michael

Dan Slayback

Fritz Pollicelli

Minnie Wong

Greg Ederer

Otmar Olsina

Matt Cechini

• Dan provided an overview of existing products – these can be found at:  https://floodmap.modaps.eosdis.nasa.gov/
  • presenting 1,2 and 3 day products without it being confusing.
  • Consider adding all 3 as toggle-able options when layer is loaded in Worldview. In the existing application there is also a 14 day product which was to enable users to see flood in recent past without having to go to calendar. This is probably not required in Worldview. could drop 1 day of historical reprocessing.
  • native resolution is ~250m (product is in geographic (lat/lon))
    
  • NRT or just historical data? Hopefully both. TBD - there is interest in back processing the archive as this would be useful for users. Robert Wolfe suggested it be considered for a standard product - this would require a proposal.
    
• Points to consider when visualizing the flood product:
• would need to let users know that it is an experimental product

• Output options:
  • 0- no/missing/saturated data or clouds
  • 1 – no water detected
    
  • 2 - water detected, which overlaps with ref water (eg, normal water)
    
  • 3* – flood
• transparent PNGs with a layer where there is water. Matt asked if they would you be able to bake in different RGB values. Currently there are 4 values.
• There is a plan to update 3 to distinguish between recurring (seasonal) flood and non-seasonal flooding.
• If it was a GeoTIFF you would be able to see the values
• Is there anything to visualize as a vector?
  • Dan doesn't think it adds much to the vector, and actually reduces information provided (4-5 raster values are difficult to efficiently vectorize into a user-friendly package). But some users may want SHP files? Still talking about it with Bob Brackenridge.
  • Ryan mentioned a planned Worldview / GIBS feature that will enable users to toggle transparent values but this is currently scheduled for mid (?) 2020
    

• Status of product: still in progress. Code being written for second PGE – in terms of getting imagery to GIBS, it will follow standard procedure for other MODAPS products
  • Timeline: few months out but could get sample imagery prior to that
    
    
• Fritz asked how would users access the flood imagery - from a webpage? straight to WV?
  • possibly from hazards and disasters page?
  • There could still be a flood webpage
  • In the Worldview Flood category it was suggested to put flood product at the top and add other layers as supporting products. It was agreed this could also be useful for other layers (e.g. various options for active fire).This may require updates to the WV code to support.
    
• There was a question on the archive / download: it is an experimental product would it still be stored at MODAPS. Hopefully yes but this and the ongoing QA and support of the product need to be addressed.
• If GIBS / Worldview gets access to the RTC SAR product (from Sentinel 1) via ASF (which may be done on a disaster basis), this could be shown in conjunction with the flood product. 
  • ESA has some restrictions on RTC products so there may be some issues with this.
    
    
• From a GIBS perspective
  • Need to nail down RGB values in imagery (action @Dan Slayback)
  • Get sample geoTIFF and PNG (action @Dan Slayback)
  • Then the color map will need to be sorted.
  • so for now color all values apart from no-water 
• Once this is done and the product is ready, then the formal process can happen
• toggle hidden values - should be ready by summer 2020: enable users to toggle between cloud cover values
• LANCE MODIS will need to finalize output to GIBS. 
• LANCE – MODIS would ship 3 different layers to GIBS (one for each flood product, 1-day, 2-day, 3-day) but GIBS would bundle them in to one
• LANCE-MODIS package all 3 layers in HDF for end users - and then users can separate out imagery 

·       NEXT STEPS

• Dan to send sample of current products as RGBA GEOTiff or palleted PNG (with metadata) to Matt. 
  • Note the current flood products are10 x 10-degree tiles which is not what the final output will be. Matt says that is not a problem for preliminary viewing.
• imager values now are 1- 4 but not colored so needs to send RGB files.
• They need to be RGBA images. Dan needs to add extra value for seasonal flood.

David Landis

Fritz Policelli

Dan Slayback

Greg Ederer

Diane Davies

David Landis

Fritz Policelli

Dan Slayback

Greg Ederer

Diane Davies

David Landis

Dan Slayback

Greg Ederer

Diane Davies

• Single HDF file of cloud shadow and original MOD09 sent to Dan. It looks reasonable. Might be useful to look and see other layers and see if there are issues but that can be done later. Good enough to move to the next step
• Now need to take algorithm for cloud shadow and merge it with the water detection. These components will be the input for the PGE which can be run globally and science tested.
• The next step to combine the cloud shadow code back in to water mask code so it puts it in as separate HDF layer.
  • David will probably run a bunch of images and view them as the test image had a lot of small cloud - so it is hard to see how effective it is. Want examples of clearly defined cloud edges.
• Once the water detection and cloud shadow mask are combined the next step will be flood detection which uses rolling 3 days of data:
  • Dan may need to provide more details on the compositing. 

Action 

• Diane to send David links to WV /RR swath imagery
• Greg to send Dan information on the terrain flag.

David Landis

Dan Slayback

Greg Ederer

Diane Davies

Fritz Pollicelli

• Good progress on the code David wrote to extract the cloud shadow from MO/YD09. The code seems to be working. The next step is to get it to write the output HDF file so it can be checked.  The first PGE is already in production producing the water layer. This second step will add the cloud shadow mask.

Action for David/Greg to send sample HDF files to Dan

• Once Dan approves the output, the next step will be to integrate it in to the PGE. Test it and then put it in to production.
• Rather than have two separate files (water and cloud mask) they will be bundled in to one and be re-gridded to 250m so everything should be consistent. 
  
  
• Should we apply masks / or not? It would be possible to put the water data set with mask in to an output file. 
  • We discussed applying cloud shadow masks to the water output in the this dataset, but agreed it would be better to simply include the mask, and apply it in the second PGE where flood products are created.
• The next step is building the second PGE, to composite the water detections from the first PGE over multiple Terra and Aqua datasets (over 1 to 3 days) to create the final water and flood maps.
  
  • The cloud shadow dataset will include the original MOD09 cloud mask QA so we have that for reference
• There will be 2 PGEs: the first creates the water detection product (applying water detection algorithm to a single Terra or Aqua granule), and the second composites these water products over several observations (1 to 3 days) to create the final water and flood maps.
• Fritz asked about terrain shadow and HAND mask. They will come in at the compositing stage. Greg thinks MOD09 has a terrain shadow flag.

Action: Dan to take a look at the MOD09 terrain shadow to see if it looks suitable -  it could be an improvement on what is being done now. 

• Is the plan to combine the terrain shadow with the cloud shadow? No, they are not combined into a separate layer, but will generally both be applied when generating the final water and flood maps. 
• The HAND layer is meant to deal with the limitations of the terrain shadow layer. it is an elevation mask that shows where flood water is unlikely to accumulate e.g. on the side of a mountain – other examples too. 

Action: Dan to assess to see if the HAND layer still required

David Landis

Dan Slayback

Greg Ederer

Diane Davies

Flood mapping transition to LANCE document provided by Dan has been circulated and agreed by all. See attached

Module A: water detection

• Action: David to send samples of cloud mask comparing MOD09 and examples of dialation at 250m to be sent to Dan for QA
• Request by Dan to keep some examples for documentation
• Greg has estimated completion of Module A as mid-May

Module B

• David asked about the process of removing cloud shadow pixels from the water mask. How is this done? simply removing any detected water if it falls under cloud shadow. In future could look to see if it falls under reference water.
• Are we still planning multi-day composites? yes - it provides a longer view.

C. Final product generation and distribution

This module would simply take the final products generated from the compositing module, and reformat (if necessary) for distribution sites (legacy plus standard LANCE channels), and for GIBS/Worldview ingest.

Options to consider:

• Continued distribution of MWP files via legacy site: https://floodmap.modaps.eosdis.nasa.gov/
  • To do so will require extending (or recoding) the current ArcGIS-based processing that generates the MFM graphic map products per tile (MFM = MODIS Flood Map). These are currently generated by an automated process running in arcpy on a virtual windows machine (using an ArcGIS Desktop map file template).
    • The MWP is a GeoTIFF – snapshot for a tile. It is not very helpful as is. Needs to be color coded for visualization
    • Need to determine how this will be done. Worldview? mapserver?
  • We could continue generating these products on our  OAS2 server, but this will require some modification. — agreed this should be done for an interim period.
  • Alternately, we could update this processing to run on an ArcGIS Server instance hosted elsewhere.
  • Alternately, we could write new code to generate a similar graphic map product independent of ArcGIS.
    • Should we get rid of graphic map products?
    • Diane to ask Tian Yao what formats the users would the data?
    • Disasters portal are currently just display the data but don't display it correctly - use value of 1 (don't think there is water). Should display it as transparent so folks can see underlying map.
    • they have the same products Dan had in terms of making the product useful – allowing user to manipulate dates etc. They just simple maps. not done anything beyond the basics. Dan has mentioned this - suggest they build a custom web application.
    • Suggest we ask GIBS/Worldivew start working with current products. Note there will be an additional pixel value "ephemeral"

David Landis

Dan Slayback

Greg Ederer

Fritz Pollicelli

Diane Davies

• • Cloud shadow mask layer
  • David is continuing to incorporate new code in to the existing code.
    • ran some compiler tests. there are a bunch of errors that need to be fixed.
    • need to talk about existing flood code // variables that haven't been declared properly
      • not sure if it is just sub-routines need to be in specific orders. Greg thinks it might be an environment issue. missing some library.
    • other than that things seem to be doing well.
    • will need to talk to Dan about the logic in the code. David has been simplifying the code as perl can do more than the IDL code.
  • Next steps for cloud shadow mask: testing the code on full MODIS products. within a month or two should have testing complete and ready
    • Greg explained the levels of testing:
      
      • unit testing - what David is doing now. small tests on algorithms to make sure they are performing the right operation
      • integration testing - larger testing at test level
      • production testing: test it on a handful of time selected intervals to test over a range of inputs. this usually involves making a release to production. the production team runs it and keeps output private for evaluating.
  • Once cloud shadow mask is ready, the next step will be Flood Detection: is the water we detect flood or not.
    • This is usually done by comparing the water results with a water mask and identifying differences.
    • The eventual output will be GeoTIFF images.
      • it was agreed that we want to replicate current product which is a geoTIFF with 4 raster values. This could be modified / improved in the future.
      • Fritz asked about SHP files, but it was agreed to not provide these. They currently take 95% of processing time and don't provide all the information layers.
    • This is part of the compositing process.
      
      
  • Follow up question from Dan:
    
    • how long until complete code is in production?
      • Summer time frame
      • Next big piece is compositing code. Dan thinks this should be straight forward in terms of code - more of a data management issue, Greg said this is handled by the production system. Just a matter of writing the perl wrapper code. Hard to estimate. optimistically it is a week or two but nothing has gone smoothly so far! As we go through the test process will be good for David to see how the mechanics of the production system work. Hopefully he will have a better sense of how it works for compositing part.
        
        
  • Follow up from last time: Question re: grid swath
    
    • Concluded that it should be grid data. Using own grid which is not standard MODIS grid.
    • Whatever grid - it will be done before flood code is run so this should't impact David's code.
    • Projection will always be the same
    • Grid - physical size array? currently 10 x 10 code but moving forward need to decide. this will be larger than a 10 x 10
      • there are different ways to do this.
      • Fritz - one goal was to avoid waiting for 10 x10 deg grids. the code should run on granules as they come in.
      • if final out put is a 10 x 10 tile when do you say it is complete? if is another granule is processed will it be an incremental product?
      • 10 x 10 grids are currently at 250m. these could/should be same grid as GIBS uses. Action @Dan Slayback to send these notes to Greg
      • In deciding what grids should be used we should look at what is most efficient in the system?
        • This will be a decision for David and Greg - it is a bit beyond where we are.
        • Need cloud shadow mask first
          • then how do we compare what we have with history to tell if there is a flood there?
          • that is when geolocation comes in to play to be able to determine if there is a change
          • at that point we can adjust things on both sides so comparison is as simple as possible
          • also need to line up with mod44 water mask? - Fritz / could be a different water reference layer.

David Landis

Dan Slayback

Greg Ederer

Gang Ye

Diane Davies

• With the shutdown. David was not allowed on to GSFC and not able to access his computer. No VPN set up
  • he was able to go in yesterday and get some files, so should be able to carry on.
  • Greg said he can work at SSAI Aerospace building. They have connections that don't require VPN set up
  • away next week in Florida but after that David should be able to start testing again.
    
    

PROGRESS since last meeting

• Testing of PDS commands to replace the IDL commands
  • • some of the perl algorithms enables you to do more with less code, for example by dealing with image border issues automatically.– This is allowing David to streamline some of the code.
    • looks like all files that go in to cloud shadow are 1km apart from SR data, which is only being loaded to get the dimension.
    • one reason to process cloud shadow at 250 m resolution is that it would  then be simpler to drop in an improved cloud mask in the future, such as might be developed from the 250m reflectance 250m data (down the road)

Question on whether the code should operate on swath or gridded data:

• • • Current algorithm takes gridded data for SR, cloud, cloud properties – not swath.
      • when writing code David should assume he will get gridded inputs.
      • will be working on gridded swath imagery, so it will be larger than current 10 x 10 box
    • Currently set up so the output for given day is produced on the grid for Terra SR file for that day. Thus Aqua and files for previous day are re-gridded to match the Terra SR grid.
      • For LANCE processing, we could do the same, OR we can use a fixed global grid for all products.  
      • Greg: code allows you to do either, so that option is available.
    • Dan asking if Greg could run some test data to map out test granules at native vs fixed grid
      • action @ Dan to send a tile / AOI and Greg will order the data

Question re: funding and the shutdown.

• • • Greg said the relevant HBS project tasks were under review by GSFC, but given the shutdown this has not occurred. Works is progressing following a letter saying the work was approved through Feb 1st. Not clear if they will extend it again.

Next meeting 20 Feb 2019

David Landis

Dan Slayback

Fritz Pollicelli

Greg Ederer

Gang Ye

Diane Davies

• Almost done converting code - cloud mask. taking flood code using it as a shell and using it to covert idl → perl
• (cloud layer is from MOD35 = mask)
• Working on data structures
  • Talking to greg about how data arrays are handled (rather than the perl format) - the standard would
  • David is lookin up perl data language (PDL) options on line. Functions that read in the HDF data - read it in a PDL language.
  • He is looking at the official PDL website. Not ventured out to look at other sites.
• Hoping for a few months. Depends on how quickly he can get the dilation code working (filling in the holes). At each place you have a cloud get height from cloud top temperature. So need complete arrays and it is patchy. Need to smooth it out to make is spatially complete. Erosion and dilation are Dan's attempt to expand the cloud top temperature. is a simpler way to do this? looking at patching functions in a 2d array - nearest neighbor averaging for non-bad pixels. some handle bad data values and some don't.
• Issue is no-one knows how the functions work and so it is trial and error. Documentation is not good so it is a slow process.
• If python was allowed this would be much quicker however for now the SA will not allow this. Currently the MODAPS machines only use code written in C, Perl or Fortran.
• This maybe an issue for LANCE moving forward if all science code has to be re-written.
• It will be a few months until this is finished.

Meet again in 2 weeks. Need to prepare an update for LANCE UWG.

Dan Slayback

Fritz Pollicelli

David Landis

Greg Ederer

Gang Ye

Diane Davies

• David converting IDL code to Perl. Good headway. some snags in trying to find the proper function for some of the more sophisticated IDL routines. (congrid, morph, dilate, quadrat)
• Working with Greg yesterday and some questions for Dan.
  • may be some work arounds. trying to fill holes in all layers / to project the cloud.
  • several routines in perl that do similar task
• Move to testing specific routines to see if they do similar things – perl equivalent code. Still more code conversion.
• Next steps: to get IDL routines mimicked in Perl. or writing simplified version that can do the hole filling. then get it on to the server to test the code – running it through the interpreters.

Next meeting 24 October 2018

Dan Slayback

Fritz Pollicelli

David Landis

Greg Ederer

Gang Ye

Diane Davies

Dan Slayback

Fritz Pollicelli

Greg Ederer

Diane Davies

Gang Ye

Brief meeting. David Landis has been hired to work on transitioning the code to Pearl.

Dan to drop by and see him in Building 33 G316

Dan and Greg to try and set up weekly meetings with David to touch base on any issues that might arise.

Plan to meet next Wednesday 29th if David is free.

Dan Slayback

Fritz Pollicelli

Ed Masuoka

Diane Davies

• Current Status: Step 1 is complete. The initial swath based water product is running as a PGE and Dan has checked the out put product and is happy with it.

Next steps

• Greg is going to amend the PGE to include a cloud shadow mask. Dan and Greg are discussion whether to use the existing cloud shadow mask (likely) or whether to use a different one.
  • Dan needs to provide Greg with the code for the existing cloud shadow mask that uses MOD35.
    • the current code Dan uses has IDL operators. this will need to be re-written.
  • Diane and Dan to meet with Greg this week to discuss next steps this.
    
    
• Dan to provide an outline for code to composite and tile the initial product. Dan to work with Greg who will write the code.
• Terrain shadow filter (monthly product computed as part of this process) is to be applied to the tiled product
• HAND (Height Above Nearest Drainage) filter, which identifies areas where flood water should not be present, will also be applied to the tiled product.

Dan Slayback

Greg Ederer

Diane Davies

• Greg amended PGE code and has completed step 1. Dan happy with the product
• Discussion on how to implement the cloud mask.
  • looking at options
  • this should be done as part of the PGE running in step 1
  • Diane to contact Eric Vermote to see if there was an improved cloud shadow mask. He said
  • "There is a cloud cloud shadow mask in the MOD09 product (at 1km resolution), at this resolution it is very difficult to evaluate cloud shadow mask and I am not sure about the performance. 
    

    Cloud shadow masks are currently being evaluated for Landsat 8 and Sentinel 2 data (using an operator derived  validation mask) we can probably gain a lot of experience from this and then transition the “best” algorithm to MODIS the best way we can. That will not be right away unfortunately but more like in a year timeframe,

    As far as terrain shadow it is an interesting problem quite feasible if you have a good Digital Terrain Model but could be computationally expensive because my guess is that you have to do it  at a finer spatial resolution than what your actual resolution but that can be studied too once again with Landsat 8/Sentinel 2 data.