Page History

NRT vs End of day: data and imagery

• Jihad has back end, Greg the front end. Matching the two up. Should be ready on staging by the end of the day.
  • security scan then production
• Maki is working on the imagery but no new news
  • Diane to follow up.
  • According to Greg she should have everything she needs to finish up.
  • She sometimes has a hard time with nasa email - use gst email

Progress on OPS Code

• David has started writing up the NRT code.
  • Need to work on this
• Greg going to document how the current legacy works so we understand the differences between the two systems.
• David has started implementing new logic on zenith code. and fixed bugs that were stopping decompression
• Sadashiva found that to compile full Format mode is just one small piece of code that needs to be changed
  • Sanjeeb needs to connect with Greg on this as soon as he is ready
  • Need special requirement to make sure the full format version doesn't go in to production

NRT vs End of day: data and imagery

• Neil looked at this. can list on web server node but were not mounted in to kupernetes. so web service couldn't see this
• Jihad says the infrastructure team are updating the service and it won't be ready for 10 - 14 days.
• on the imagery side: Maki has code ready to test on NRT4

Progress on OPS Code

• Exchange of emails between Greg, David and Dan to figure out how best to select pixels and thresholding
• action @ Greg and David to document exactly what the NRT algorithm is going -> doing
  • @ david will write up the current code and Greg the legacy to show in detail how the algorithm handles pixel selection
  • there is a slight difference between legacy and current code and it is not clear what the source of the difference is
• action @ Sadashiva to email Sanjeeb / LDOPE re: tool to uncompact layers (rather than re-running the PGE)
  • enable Greg/Dan to pull all pixels from files and thus test different pixel selection ideas.  

Fritz has been looking at ways to improve the cloud shadow mask.

2 September 2021

Dan, Fritz, Sadashiva, David, Greg

NRT vs End of day: data and imagery

• file names are showing up in NRT3 (in development) but they can't be found on LVFS
  • Greg has a ticket in to Jihad as this was working a while ago. 
  • various reboots have been implemented so it could just be a reboot is required
• Maki still working on the imagery.
  • Greg needs to reach out to Maki - this week hopefully.

David - progress on the code

• since last week: code is cleaned up and producing the proper metadata files.
  • been doing some testing
• NEXT STEPS
  • Greg needs to look at this and decide what to do with the zenith angle
    • Greg to write up
    • needs Dan's input too.
• David can merge GeoTIFF code ready prior to algorithm

NRT vs End of day: data and imagery

• some de-bugging going on
• Maki still working on the imagery.
  • the major difference between CR and flood is swath vs gridded

David - progress on the code

• working on methods for pixel selection. 3 methods sent Greg
  • Greg needs to look at this and decide what to do with the zenith angle
  • needs Dan's input too.
    • Greg to respond to email and then decide whether
• David can merge GeoTIFF code ready prior to algorithm
• Current code in NRT already has the geoTIFF stuff. not sure adding in ability to process pointer files will add to the NRT.

Fritz - putting together something on cloud shadow for Eric.

Sadashiva - asked for hurdles and meeting with Robert.

NRT vs End of day

• Greg's test code shows the update files. 
• So this is working on Greg's local machine. 
  • Next steps to put the code on NRT3 staging server for testing. 
  • additional tests need to be done as the code could impact LAADS files.
    • Once testing is done (mid next week) - it should be quite quick to stage it on NRT3
  • This also works for the TIFF products

NRT GIBS imagery

• Maki had a plan for implementing this
  • code will run PGE 159 then image code will run
    • not sure if next step is Greg's or Makki's

David - progress on the code

• all seems to be working fine. various test files. 
• David can merge code and put it in to subversion and get it ready to run as a PGE
  • closest to nadir.
    • Dan would like David to implement this on flood code.
    • check it has all components - including TIFF products
      
      • Dan would like David to implement this on flood code.
  • Still to be done .
  • then get ready for subversion then run PGE

Greg:  There will be differences in how we run code in OPS vs NRT

• may or may not be significant. Run David's code in OPS first then NRT.

Greg wrote up a document to describe some other measurements besides brightness that have an effect on deciding whether a pixel is showing water. These other measurements, solar zenith angle, sensor zenith angle, and pixel/grid-cell overlap area, can be used to improve the likelihood that a dim or dark pixel is, in fact, due to water and not some other factor.

• currently the code uses sensor zenith. So close to zenith gets higher weighting.
• ops code picks most recent pixel it sees.
• MOD09 uses nearest neighbor: closest to nadir.
  • Dan would like David to implement this on flood code.
• Dan would like to give the options Greg came up with some consideration

No weighting out until PGE 159 (apart from bad pixels). PGE 155 doesn't do any selection just maps pixels to grid cells. 

29 July 2021

Grid to swath / zenith angle /  interpretation of PGE155 Zenith Angle files

David - progress on the code

• Testing with additional tiles. seems to be working okay. 
  • NEXT: send code to Greg to put it in to the OPS system and run a 3 day test. 
  • if it looks okay run larger test.
  • It will be a couple of weeks when back from leave
• Equator to 30-40 N runs in 15 -20 mins
  • David can try the code with some NRT inputs (one swath) and see how quickly that runs
  • David can merge code and put it in to subversion and get it ready to run as a PGE
    • check it has all components

NRT vs End of day

• Greg has the files showing up
  • next add it to the staging service and see if it can be downloaded

22 July 2021

David, Greg, Sadashiva, Dan, Fritz,

8 July

Greg, David, Sadashiva

Grid to swath / zenith angle /  interpretation of PGE155 Zenith Angle files

• L2G discussion still to take place. with Sudipta, Sadashiva, Greg and Dan
  • for standard MOD09GA is already aggregated at the end of the day: pixels picked out on best quality 
  • could be used for water detection and flood detection. 
  • wouldn't need 155 as it would be a use the MOD09GA gridded product
    • David wouldn't need to decompress the zenith angles. they could just be read.
  • Sadashiva suggests we don't un-compact everything.
  • When you run a granule in L2G in NRT: first layer nearest neighbour observation
    • pick up geo angle at that time
    • when layer next comes in you can see the observation from the previous observation. 
    • some work to get solar zenith.
•  NRT vs Daily delivery of flood product.
  • Greg working on website. Jihad has done his changes for now.  A week or two away
  • Maki working on getting the imagery in to GIBS
  • Legacy system running until end of August
    
    
• David has made a lot of progress on the code
  • old version was using interim files to take the 1km zenith angle files and scale them to 250m
    • being done pixel by pixel (1km → 500 →250m) - it was taking 70 mins to run
    • that has been replaced with new code so it runs in 11 mins
  • currently being run on end of the day water files
    • would run on NRT: existing end of the day + level 

1 July 

Greg, David, Sadashiva

Grid to swath / zenith angle /  interpretation of PGE155 Zenith Angle files

• L2G lite is a daily product. Greg spoke to Sudipta, so wouldn't work for NRT
  • Sadashiva: doing the development for OPS processing not NRT?
  • Greg: need clarification from Dan: the issue shows up in NRT as well as standard.
    • NRT is not using the zenith angle
  • SD: running L2G process in NRT as granules come in. generate gridded tiles as soon as they come in
  • as granules come in process L2G = that is processed for that granule. So already doing it for NRT
  • If you are doing it that way you are processing once granule // what are we trying to acheive. 
    
    
• Are we trying to implement the zenith angle for NRT or OPS?
  • both - to avoid false positives
  • maybe use different approaches. we have been trying not to. Need to do:

1 . refinement to NRT PGE and implement some improvements

2. development of OPS PGE using L2G lite

archive all NRT and use that for the development and testing.

• need to see about archiving the pointer files

TAKE THIS OFF LINE: DISCUSS BETWEEN DAN, SADASHIVA, SUDIPTA, GREG - Greg to call the meeting to determine the way forward. 

Put code development on hold to see if we use L2G lite

NRT vs Daily delivery of flood product.

• Greg not got to this. NRT bug = needs to be fixed.
• Greg needs to follow up with Maki on imagery

Code that David is working on is going well. 

User guide and website are updated.

24 June

Greg, David, Fritz, Dan

Grid to swath / zenith angle /  interpretation of PGE155 Zenith Angle files

• good progress on speeding up the algorithm.
• still need to look at L2G lite

 NRT vs Daily delivery of flood product.

• Partial implementation. Greg thinks it look
• Jihad needs to do a bit more. Greg to start working on the web side
  • possible just a week or two away
• Greg to follow up on NRT imagery

NEXT STEPS

• modify compositing rules and assess the impact – DAVID
  • currently using the fixed threshold (2)
  • probably a 50% threshold would work.
  • Dan then to assess how well it works by looking at a couple of days of global data using the TEST set up that Greg has.
  • 3 levels of testing for both changes.
    • CHANGE OF WHICH SR OBSERVATIONS ARE ANALYZED
    • CHANGE THRESHOLD RULE 
    • before that run on a few files (before and after), then run on larger. then on a science test on a production system (couple of months or weeks scattered around the year).
  • Don't update the operational algorithm until both are done. 
  • re-run dates in March or go back to Jan

Legacy system: June 30 is the date for cut off.

• Dan to send a note with user guide. keep it running for at least 30 days after NRT is running in NRT

17 June

Greg, Dan, Sadashiva

• PG 13 MOD09 L2G lite. All in one file
  • spend some time looking at this as an alternative. 
  • 1km - 500 and 250m: Action Greg / David to look at it to start 
• User guide
  • few comments for Dan.
    • page 1: reformat to look like other user guides
    • revision to be moved to second page
    • section 6.4 add figure to illustrate para.
    • Dan to give Greg a tile and he will get this.
      
      
• Imagery showing up in GIBS at the end of the day
  • Greg to contact Maki

 NRT vs Daily delivery of flood product.

• Jihad has finished his part to now it is up to Greg. It will still be a week or two. 

Grid to swath / zenith angle /  interpretation of PGE155 Zenith Angle files

• Up date from David: I'm making progress in testing my files. One file is made with my old program (that takes 70 minutes to run 1 file) and the other with my new (tweaked) program (that takes 15 minutes to run). My hope is that the two files will be sufficiently equal that I can go with the newer code. So I'm working on a way to compare them directly, layer by layer, pixel by pixel. I hope to have that functioning soon. 
• There maybe other ways Greg can help. 

10 June 

Diane, Greg, Dan, Sadashiva, Fritz, David

NRT vs Daily delivery of flood product.

• Ceph is close to ready on nrt systems. Once it is ready, Greg can work to incorporate it into nrt web sites for NRT update of Flood files. Greg will follow up with Jihad (see the ticket at https://gitlab.modaps.eosdis.nasa.gov/infrastructure/help/-/issues/95 )

• Robert decided, after talking with infrastructure group, that Ceph sounded close enough that it didn't make sense to re-implement PGE159 using the DNB production rules

• Jihad was just back from vacation. 

  • • Greg to ping Jihad and ask about timeline

Grid to swath / zenith angle /  interpretation of PGE155 Zenith Angle files

• David got a version working that takes 1km to 500m and scales to 250m for the zenith values.
  • It took over 1 hour for each file/per satellite. So D worked on code so now it is 15 mins. 
  • everyone els has written their programs in C. this is in Perl. so slower. If it can process the entire array it would be okay but this is pixel by pixel.
  • Sadashiva said it has to be done pixel by pixel even in C.
  • Greg to work with David on this in an advisory capacity
  • Spend another week or so on this and if there isn't much progress in terms of the speed of the code, consider writing it in C.
• Dan should consider L2G lite reflectance which has a "best observation" – what about just using this? 
  • it would make it more simple as there would just be 2D data to deal with.
  • direct correspondence between 1km - 500.
  • would just be using one observation instead of full resampling ( or nearest neighbor in NRT).
  • PG 13 MOD09 L2G lite. All in one file
    • spend some time looking at this as an alternative. 

User Guide

• Diane to remind Sadashiva to finish reviewing the User Guide

Flood alerts

• on the legacy system there is a JPEG map 
• on LANCE it can be viewed in Worldview

27 May

David, Dan, Greg, Sadashiva, Diane

NRT vs Daily delivery of flood product.

• infrastructure team have implemented a directory server that allows discs to be seen on other machines = first step
  • now the system programs are available everywhere
  • installed on moddev-1
  • so making progress but not yet available for production
  • Greg can start working on the instance on mod-dev1 to check code
• Sadashiva will write to Carol and cc Neal & Robert to see if we can move this forward - by asking Carol's group to re-write the production rules. It already works for DNB. 
  • it will give the developers more breathing room to implement the new system.

Pixels: grid to swath / zenith angle

• David has code from Sudipta to assist with the compositing. Should be sufficient information in the SR guide to follow. 

User Guide

• Sadashiva almost done reviewing this.
  • looks good. Dan to sent word file.

VIIRS continuation flood product

• There will be future calls to try and get the flood product as a science product.

20 May 2021

David, Sadashiva, Greg, Diane

VIIRS flood - need to put it in the plan. Important to demonstrate good quality products from MODIS first. 

discussion about how we want to use the historical data. 

Think about how we can make it a standard product?

NRT vs Daily

Neal has done his part. now with Jihad.

Pixels: grid to swath / zenith angle

Suggestion by Sadashiva to consider another approach 250m: grid to swath / zenith angle

13 May 2021

David, Dan, Greg, Fritz, Sadashiva

Pixels: grid to swath / zenith angle - David still working on this.

• Need some input. Maybe Sanjeeb? He has worked a lot with the gridding code. 
• Email questions and cc Sadashiva

Earthdata flood article. this will wait until the data is available in NRT

• NRT vs Daily
  • Neal has a piece he needs to do and then 
  • Greg will need to do something on his end

Qualitative comparison

• this is finished. almost finished updating the user guide
• looked good. 
  • impacts: increase cloud shadow false positives in the N but capturing more surface water (lakes and rivers) when the thresholds are update for the cloud shadow it should be an improved product.
  • Do we then release the updated user guide for the NRT?

https://modis-land.gsfc.nasa.gov/MODLAND_grid.html 

• Sadashiva to follow up on this. 

NEXT STEPS

• once the looks are sorted
  • implement thresholds
• run a year of data just to verify
• then 10 years

6 May 2021

David, Dan, Greg, Fritz, Sadashiva

view zenith angle is 1km (@the edge of swath they are given less weight).

need to get no of water layers to match zenith angles.

need to see what % of grid pixels overlap swath pixels to decide flood count determination.

either case need to map from pieces of swath pixel to grid pixels whatever the resolution.

interim products that are produced that could be used to de-code.

GeoTIFFs and NRT issue

Qualitative comparison - going well. better delineation of water bodies. false positives are an issue. otherwise products are the same.

User guide - being updated. Finish up qualitative evaluation and add that to the guide. 

Greg made data available for Dan 

Mailing list – non-NASA users won't be able to go to the web interface. they will need to just send  and email to subscribe and unsubscribe. 

https://modis-land.gsfc.nasa.gov/MODLAND_grid.html lat long grid missing . Dan to ask Sadashiva.

Geotiffs look fine; communicated their availability to Disasters group.

Evaluation

Working on qualitative evaluation (of the 100 sites over 3 years), which for as far as I've gotten (not too far...) looks great. Minor pixel-level differences, but nothing at all concerning.

Found an issue in the current product - volcanic areas in Hawaii are showing up as flood. We had this issue several months ago because the original terrain shadow files I delivered to Greg did not have the fix that we'd implemented in the legacy system. So I provided the updated files way back in December, and I believe I verified way back then that the issue was fixed. However, all the data in Worldview (so since Mar 23 or so) have this problem. I suspect the set of terrain shadow files got reverted to the original set or something. 
Should be fixed now. Greg has implemented the files from Dan in OPS and NRT

GeoTIFFS 

running on NRT3 and NRT4

• Diane and Dan to update the landing page and user guide

NRT vs end of the day issues

Robert suggested a meeting. Not yet scheduled.

OPS Code

David has been working on the pointer files and pixel sampling.  

Earthdata article

Fritz and Dan working on flood article 

Fritz's Alerts: using planet scope - making good progress. idea that when get an alert - can make a more detailed map using high resolution data. as a derived product this could be distributed. 

22 April 2021

No meeting last week. Notes for week before not available on the wiki

GeoTIFFS

• running and in production on NRT4
• Dan to check they look okay
• next step to make it run on NRT3
• Diane - to let the Disasters Portal know they are available

NRT vs Daily

OPS Code

• Greg and David still working on pointer files and Pixel sampling tests using LDOPE tools
  • some have 4 and some have 6
  • when water file pge152 detects water = swaith run pge155 to get grid. then there are many swath pixels that overlap a grid pixels. need to decide which are useful and which to keep. read pointer files to give view / zenith angle. zenith pixels are 1km but grid = 250m
  • Sadashiva says there is there a process
    • Sudipta has given information on how it works
    • still need to learn and understand how that works
• Updates to OPS code will be added to NRT code. This should be straight forward.

Earthdata article

• Joseph has been in touch with Dan and Fritz

Evaluation

Currently on version BETA

Qualitative comparison with legacy MWP: 100 events of 3 years.

• Dan has this data and has started the comparison.
• Should be done by Mid-May

Next step will be to test the thresholds 35, 65 etc. // need OPS code running first before starting that. 

of 3 years to be done by Dan – then go to provisional 

Database

• 225 days and 850 million flood pixels. uptick.

Flood viewer

• Diane to set up a meeting with Albert and Bob Brackenridge to look at requirements for Flood viewer.

Greg 

• to limit legacy flood product to 3 days instead of 7 
• then add cloud shadow layer from MOD09 for Fritz – this will come out a TIFF.

• 8 April.

Greg, Dan, Fritz, Sadashiva, David

GeoTIFFS can be pushed to NRT code now

• trying to integrate David's code to Gregs (issues with memory allocation)
• will continue without David's code for. will wait until Monday to push to production  
  • David's code handles more than one pixel // this has implications for thresholding.
  • Dan thinks we need a test for the appropriate threshold which may take some testing
• DOIs have been registered:
  10.5067/MODIS/MCDWD_L3_F1_NRT.061
  10.5067/MODIS/MCDWD_L3_F1C_NRT.061
  10.5067/MODIS/MCDWD_L3_F2_NRT.061
  10.5067/MODIS/MCDWD_L3_F3_NRT.061
• once Greg makes these available - Diane needs to update the landing page and Greg update the user guide.

Worldview - imagery now running in production - https://go.nasa.gov/3sWIDCx

OPS code

• David has this in hand. Greg integrating it in to his code.
• David started doing the 3-day tests. hit a problem that Greg is now correcting. 
• Supposed to be first 3 days of 6 months. Got 2 days then snag.

Pixel sampling tests using LDOPE tools

• Dan needs to say how he wants to process pixels from different swaths vs orbits
  • 50% threshold? Request Sadashiva and LDOPE folks provide guidance.
    • Same orbit: aggregate by coverage and cloudiness. → separate cloud and clear pixels. partial cloud is of concern
      • nearerst to pixel center first option but if cloudy choose non-cloudy as long as it covers certain percentage of grid
      • currently count number of cloud and water pixels. 
      • Is the fraction significant enough to be included? 
    • Different orbit: harder as different time. need to look at geometry. which one do you trust the one close to nadir (geometryr) or % cloud, or height of cloud
      • Action for Sadashiva / LDOPE provide pointer to core segment for initial development. then dan can take a closer look on how to tweak this.
• when pixels overlap a grid cell it counts each of the pixels - hence the work on the threshold. it will depend on the number of pixels for that cell from overlaps and scans. adds them all up for each category (water, shadow etc) then apply threshold rules. This is what David's algorithm.
• need more guidance from Dan
• Greg has a better idea of the operations

Database

211 days and 800 million flood pixels

No meeting next week

Fritz's questions:

• will need to switch from experimental system to the new system. 
• size of files? 10 x 10 degree but pixel size is different
• in a week the GeoTIFFs will be available
• when will we process the whole archive? Not for a while. want to implement the algorithm changes first. 

Images are now being produced on NRT3

• can we check images are produced in NRT (not at the end of the day)
• Neal has to make sure it is available. We need to let the GIBS team know

 Archive

• we have the okay from Robert to do provide an "experimental" archive

OPS Code

• Refining the OPS code and testing it with water files that David has been generating.
  • Seems to be working well. 
  • Output looks good
  • Need to merge with changes for GTIFF files
  • The Greg needs to make it run as a PGE in OPS.
    • Will do a 3 day test.   
    • Get Dan to check it then do a 3 year run

DAN 

• C6 vs C61: looks good.
• Evaluate 3 year run in OPS - will take approx one month // there is a small chance we will see differences (unlikely)

- STILL TO DO

• Pixel sampling from scans using LDOPE tools - Greg thinks David might be able to help with this.
  Dan to send a list of things to be tested to David and Greg
• Look at thresholds from OPS algorithm from David // writing that up

GTIFFS 

• Disasters group looked at the GTIFF.
  • They would like both tiling and internal pyramids. It is a simple GDAL command to do on their end. 
    • suggest we go ahead as is now. Albert from DFO is happy with it.
    • it would increase the file size
• Look at thresholds from OPS algorithm from David // writing that up
• Evaluate 3 year run in OPS - will take approx one month // there is a small chance we will see differences (unlikely)
• Will be available in NRT when LVFS issue is sorted (Neal, Jihad and Navid working on this) Greg to follow up to see where they are at.

To be revisited 

Add pyramids to GTIFFs

• PGEs are almost running on NRT3 so that should happen in the next day or so. 
  • once confirmed they will start sending the imagery to GIBS and WV will start their process.
• Still waiting on LVFS for NRT granules - this will also have an impact on GeoTIFF availability
  
  
• GTIFFs
  • reviewed by Dan and look good. 
  • sent some to Jeremy (NASA Disasters) and DFO for approval
  • just finalizing whether they have tiles vs pyramids
  • metadata - minor updatesC6 vs C61
  • will be created in NRT - trying to get them staged on the website at the same time.
    • currently will only get them at the end of the day like the HDF products
  • Check doi registration for GeoTIFFs 

DAN - STILL TO DO

• C6 vs C61
• Look at thresholds from OPS algorithm from David
• Evaluate 3 year run in OPS - will take approx one month // there is a small chance we will see differences (unlikely)
• Pixel sampling from scans using LDOPE tools - Greg thinks David might be able to help with this.
  Dan to send a list of things to be tested to David and Greg

OPS Code

• commenting code to make it clearer
• David waiting for thresholding from Dan
• then more testing before code gets turned over to Greg to integrate in to the system
• this is almost ready but the recent changes in the NRT weren't included. PGE 155 updates and PGE 159 (changes that David)
  • due for a big PGE release when establish current NRT is working on 3 and 4. then do a new release with the changes. can roll back to this version of there are any issues in the new PGE release.

Database 

• 190 days 751 million global pixels
• need to think about how to manage this.

11 March 2021

Priority for Dan

1. GeoTIFFS
2. C6 vs C61
3. Look at thresholds from OPS algorithm from David
4. Evaluate 3 year run in OPS - will take approx one month
5. Pixel sampling from scans using LDOPE tools - Greg thinks David might be able to help with this.
   Dan to send a list of things to be tested to David and Greg

Diane to initiate discussions with Karen and Robert re: archiving the flood product

GeoTIFFS

• wrote and tested code as part of PGE 159- on products in test products in OPS
  • These are on moddev-c7
    • /SSTG/gederer/PGE159-TEST-LOTS-GEOTIFFS/MCDWD_L3_F1
      /SSTG/gederer/PGE159-TEST-LOTS-GEOTIFFS/MCDWD_L3_F1C
      /SSTG/gederer/PGE159-TEST-LOTS-GEOTIFFS/MCDWD_L3_F2
      /SSTG/gederer/PGE159-TEST-LOTS-GEOTIFFS/MCDWD_L3_F3 .
  • Now have about 10k GTIFF images 
  • Greg doesn't have a good GTIFF viewer
  • @ Dan Slayback to sign off on these and then if happy send them to disasters for their approval.
• formal PGE release process needs to be undertaken. 
  • CM would need info and then deploy it 
  • at this point the main concern is that it does what we want
  • already added new products in to dbase on NRT4
    • this would need to be done dbase on NRT3 and LAADS
  • currently no mdata other than std geolocation 
  • using GDAL command
• Asad is registering these as DOIs
  • Dan to send Greg an example of how to add DOI in to metadata tags 

CMR & EMS - diane to follow up with Asad

Flood imagery

• Imagery is going to GIBS correctly.
• Production imagery only comes from April 1st. 
• waiting for Worldview team to schedule when it will go in to Worldview
• Sadashiva to email Gang and Matt

3 year Run in OPS

• complete
  • Dan has the data and needs to take a look

C6 v C61 comparison

• Greg finished this and data is available for Greg to look at
  • Dan needs to take look 

David and Greg working on OPS algorithm

• developed code that can look at the number of observations and determine threshold
• Question is what is a good threshold %? 50% - extension of what was done before 
  • David to run the products for Dan to take a look at.
  • single file is ready for viewable.
  • David needs more to run more tests. Greg will get these to David.
• When done run in OPS as separate archive so can compare. 
• will keep this separate from GTIFF code. as change to thresholds is more complex
  • Greg was hoping to get the GTIFF code added
  • wanted to check C6 vs C61 is okay 

PGE 155

• had some changes too that need to go in to an OPS test (3 day followed by 3 year)

LDOPE tools 

• Dan to take a look at brightness temperature - as this might have less cloud
• This is a re-sampling process.  looks at max observation coverage and takes that one. 
  • if other observation covers the grid cell it is ignored so you are only seeing partial coverage.
  • do we want max observation coverage or aggregate?
  • sometimes weighted average of observations
  • can you take minimum brightness? yes 
    • for flood. if diff surface areas the one with lower brightness would have less cloud. 
    • ? from Dan. are there tools to enable Dan to test the different options?
  • yesterday Greg was trying to get to the bottom of this. via Sudipta Greg found out that LDOPE has tools that enables separate SDS to be read and look at outcomes. sounds like a research thing - not a quick and easy answer. once figured out could back port in to NRT

3 March 2021

LANCE requirements:

• Data - on redundant servers
  • add website updates to
    • earthdata.nasa.gov
    • lance3 and lance 4
• User guide
• DOI and landing page
• Register in CMR - enable earthdata search
  • Asad is doing this 
• Register metrics with EMS 
• imagery in Worldview

GeoTIFFS

• shortnames
• MCDWD_L3_F1_NRT
• MCDWD_L3_F1C_NRT
• MCDWD_L3_F2C_NRT
• MCDWD_L3_F3_NRT
• doi - CMR - EMS registration
• add information to user guide and landing page

GIBS Imagery

• next sprint starts next week - need to get this included!
• Greg to contact Sudipta to check all good.

GeoTIFFS

• added code. not been able to test it yet. 
• will be able to test later today or yesterday 

Latency

• PGE 159 runs every 10 mins to check everything is coming in. 5 mins to process. send results to imagery producing PGE. 
• PGEs that process MOD09 - immediately 
• maybe additional padding in the scheduler - 15-20 mins

3 year Run in OPS

• complete
• Dan has the data and needs to take a look

C6 v C61 comparison

• Greg finished this and data is available for Greg to look at

Database  - is going well 

177 days

David working on OPS algorithm

• looking good.
• now a question of threshold for flood vs non-flood
• in the ST if we want to replicate what is happening in NRT can we do that?
  • current code in 3 year is exactly the same
  • integrate Sanjeeb and our changes - may see some differences.
  • Dan should do evaluation. so step 1 is done.
    
    
• almost same code as for NRT but running in OPS. Not quite - same as it just accounts for pixel from first scan. if there are additional overlaps it won't include them.
• 10 lines at 1km. as you go further from nadir you get more pixels overlapping. these are counted and retained by PGE 155_LG code. takes all the swaths and grids the pixels
  • in NRT look at one swath at a time. just look at one layer. not at overlapping pixels. first pixel in any overlapping scan. those pixels are in the 155 but just look at the first one the rest are thrown away.
  • scan and grid don't align. so go at an angle to the grid cell. even at the center if there is a scan line overlapping 2 grid cells
  • This is a re-sampling process.  looks at max observation coverage and takes that one. 
    • if other observation covers the grid cell it is ignored so you are only seeing partial coverage.
    • do we want max observation coverage or aggregate?
    • sometimes weighted average of observations
    • can you take minimum brightness? yes 
      • for flood. if diff surface areas the one with lower brightness would have less cloud. 
      • ? from Dan. are there tools to enable Dan to test the different options?
      • yesterday Greg was trying to get to the bottom of this. via Sudipta Greg found out that LDOPE has tools that enables separate SDS to be read and look at outcomes. sounds like a research thing - not a quick and easy answer. once figured out could back port in to NRT

Variable (per pixel) threshold modifications will require changes to NRT. This could be done a the same time. 

Action @ Dan to think about this before integrating it in to the OPS code. 

• Dan asked Greg to consider how could the 50% threshold be implemented?
  • thinks it will be straight forward

24 February 2021

discuss GeoTIFFs and where to store them

Revisit: date for removing legacy product

GIBS Imagery

• diane to contact GIBS
• dan to look at imagery in GIBS

GeoTIFFS

• Need to add code to PGE 159 to create GTIFF for each 10 x 10 tiles (4 of each)
  • daily cumulative GTIFFs per tile - updated throughout the day 
  • 1 x file for each tile throughout the day. the more overpasses the more information you see in that tile.
  • each file name has a different
• If we want them to be downloadable we need product names / ESDTs and added to the database tables as we do with other products. 
  • this means they will be staged to the website and put in the download directories on the NRT systems
  • in ops - they would get archived
  • it requires some table edits and scripts
  • NAMES: MCDWD - contain tile and date but not time. 
  • action @ Dan Slayback to come up with names 

Diane to contact Robert about the end of day product - so he is aware and contact Neal and Asad re: GTIFF

User Guide

• Almost finished. Question on the improvements. Should it be done in 2 phases:
  • 1 - getting this one out plus evaluation comparing to legacy and evaluation (keep as beta)
  • 2 - rest of changes (release 1)

3 Day Run in OPS

issue with three day production rules

• days where no data were produced due to sensor calibration etc
• production stops at that point as it can't find previous day 
• need to amend production rules to handle that situation

NRT processing

OPS Processing

• in general working well. compressed water layers decompress properly. 
• weird problem combining them with a ref water mask
  • need to find a way to test this
  • same code that used to work

Database

• 169 days and still running. 700 million pixels 
  • 1 billion records= 1 year - a 40% drop after new terrain shadow masks are in place.

Diff between C6 (OPS processing) and C61 (NRT)

• Greg has been working on a script for this on his dev server

Terrain Shadow zip files

• some terrain shadow zip files that David has received has zero data.
• Dan uploaded them again for David to download but he is still having issues // suggest command line utilities to unpack them

18th February 2021

Dan, Fritz, Greg, Sadashiva, David and Diane

User Guide

• Sadashiva to review today / tomorrow
• Please look at last section wrt who we term the versions.
  • for MODAPS usually beta, provisional, validated
  • in this case Dan is thinking:
    • PHASE 1: close to old system
    • PHASE 2: improved terrain shadow etc

3 Day Run in OPS

• OPS has about 100 recipes.
• Kurt is running the date ranges that Dan sent.
  • Dan can start looking at the data on MOD_DEV_C7
  • Files should remain until we ask them to be removed

NRT processing

•  Still waiting on Jihad and Navid.
  • looks like solution for DNB is not enough
  • not sure how they are looking in to it.
• There is a virualization that happens that makes it look like there is no production time. 
• LVFS has to do work to find them. We need to do work to manipulate the file name.
• Can we do it the old way? No not at this point.
• Agree to provide products at the end of the day to avoid further delays. Imagery will be NRT

OPS Processing

• Greg has been working on getting test files to David
• found bug in L2 G that Sanjeeb has fixed.
  • not impacting NRT
• David has things almost working - seems to be an issue with the reference water mask.
• Test files for T & A seem to be work 

Diff between C6 (OPS processing) and C61 (NRT)

• Greg can check this on his dev server
• some terrain shadow zip files that David has received has zero data.
  • Dan to upload them again for David to download 

Database

• 162 days and still running. 685 million pixels 
  • 1 billion records= 1 year - a 40% drop after new terrain shadow masks are in place.

11th February 2021

User Guide

• Dan to upload to One Drive
  • Sadashiva and Diane to review

3 Day Run in OPS

• 3 year test run. plans are entered for this. currently not running
  • asked for a list of date ranges. these were sent. need to follow up to see if will run 3 years or date ranges.
  • This will need to be run in C6
    • Could to check MOD09 differences between C6 and C61
    • Greg could run flood as c6 and c61

Top 2 row issue

• ignore for now

NRT Processing 

• re: data not getting put on the NRT download site when complete
• Needs changes from Jihad to make them show up in LVFS (end of the month) it is an input and write it back out. original system wasn't set up for this. 
  • Sadashiva to follow with Neal / Jihad

David working on OPS algorithm

• good progress. need better test files
  • Greg to send these to David. Creating these via the dev server. Almost done.

Worldview / GIBS

• GIBS imagery is being produced but they are ignoring it
  • Dan - to give Minnie brief overview
  • once we hear about NRT data we can give GIBS the go ahead

3 day run in OPS

• ran okay
• Dan looked at it briefly and it looks okay.
  • needs to look more rigorously before asking for 2 years. 

NRT Processing

• There is an issue where it not putting the data on to the download site until the end of the day.
• Greg to contact Neal about this
  • Not sure if it is MODAPS issue or PGE issue?

David is working on OPS algorithm

• layers are decompressing properly for the new structure of the data - based on the internal storage structure

Validation plan

• Dan has been collecting data. needs to summarize it still. should have something in the next couple 

User guide

• waiting for validation 

Removing the northern tiles 

• No terrain shadow file for Dec - Jan 22. 
• The algorithm should be using the new TS file so it should be better. 
  • excessive 

Worldview Diane to contact WV and GIBS about possibly going live at end of Feb and adding flood imagery to UAT - GIBS

Database

• 141 days and still running. 630 million pixels (3-day product)
  • images look good
  • if n. terrain shadow is more effective we should be seeing a drop in the number of daily pixels.
    • There has been a 25% reduction in the number of flood pixels.

GMU and UWM have a global flood mapping product. Fritz has been applying his flood alert. They claim to have solved the problem of terrain shadows. Fritz is cutting out Landsat shaped tiles. Looking at the flood pixels (0-20%) high number for Landsat areas. If only look at 100% pixels still think it will be hard to determine which ones are important floods. 

21 January 2021

• David Landis
• Sadashiva 
• Dan Slayback
• Greg Ederer

3 day run in OPS

• Issues: production rules and output in production imagery
  • out put for the test. for all the days it is missing the previous L3  (MCDWD_L3) file. 
  • production rules don't wait to make sure the previous day is available. 
    • not an issue in in NRT but it is in OPS
  • if you look at the 3 day flood, there is often no data but it should have the current days data.
    • most have these narrow pieces of swath so that sounds like a code issue
    • Greg ran the code on his own system - the production algorithm should be resolved next week
    • also need to check the algorithm part re: previous days data. Use current L3 file but use previous day L3 file. Production rules allow you to skip it as they are set up now.
    • Composite is today plus L2 from previous day
  • Suggest address production rule issue first? But there is something going on that needs looking at - consistent error (seeing wedges of data). 
  • Dan will look again at the NRT outputs to make sure they look okay and confirm that he didn't miss this issue. 
    • when previous days L3 is available, don't see the wedge shape.
    • Wedge suggests overlap from swaths and only getting sufficient observations from wedge.
      
      
• David is working on OPS algorithm
  • Greg has been helping 
  • trying to decode the additional compressed layers in the file
    • tweaking algorithm. reading the algorithm slowly as it is building 1 pixel at a time. 

• Database
  • 135 days and still running. 6 million pixels.
    • images look good
    • oval shape flood – eastern Siberia 18th Jan
• Email alerts
  • running daily
  • getting some cloud shadow anomalies
  • 24 x 14 km - 360k of them so bound to have anomalies.
    • starting to work with Landsat grid
• Validation plan
  • release the product without doing the validation just summarizing the statistical
  • in N. getting more cloud shadow issues
    • release before or after this is fixed? N. latitude - will need variable threshold which will need work.
      • could ask GIBS to not put those to rows in to GIBS. NEED TO UPDATE THE TILE SCHEME IN NRT
      • SD: recommendation would be to not produce the high latitude tiles.
        • any attempt to communicate the issues is fraught
        • so what is the best way to implement this?
          • don't produce them? different tiles scheme. requires no adjustments elsewhere
          • can run tests with the tiles in as needed.
    • do we want geoTIFF outputs?
      • we can use WVS

• 3 day run in OPS
  • no updates
  • Kurt is running this but doesn't know he is working with
  • Greg is coordinating it
• Schedule
  • no update
• GIBS imagery
  • no update
• GeoTIFF
  • no update
• Database
  • 128 days and still running.
  • 1.7 billion records - estimate for a year.
    • may change as we move away from the winter solstice and terrain shadow 
• PGE 159 Updates
  • David fixing it to work with PGE 155 inputs. he ran in to a few issues reading the HDF metadata. 
    • Greg working on this
• Dan could come up with a different plan for evaluation - compare legacy and NRT
  • this would enable us to get the flood product
• Email alerts
  • 24 x 14 km in Fritz's
  • we want 10 x 10 degree blocks
  • makes sense to bin things by cloud cover percentage. 
    • need to look at this more carefully as not a linear relationship 100% cloud
    • interesting to incorporate this in to the flood algorithm or flood alert detection
• User Guide
  • Dan needs to work on this - as a beta version
  • and a document comparing LANCE with heritage system 
• Set release date for 14th Feb

7 January 2021

Greg, Sadashiva, Dan, Frtiz, 

• 3 day data in OPS had some issues - missing files
  • hopefully just production rules. Greg will take a look at this.
• GeoTIFF - action for Greg
• Database
  • thresholds from Dan.
  • once these are in they will be ready to test with the pre-composited water products.
  • continuing to run files in to the database: 121 days - 556 million flood pixels (seem to include a bunch of terrain shadows especially in the Himalayas).
• Dan managed to get the legacy system to the new server

17 December

Fritz, Dan, Greg, David, Diane

•  3 day run in OPS
• Dan has collected data
• Action @ Dan to take a look at the data - to be done by 23.
• Schedule
• run 2 years of OPS data.
• This could start this next week but need input from Dan first.
• Action @ Greg to wait to hear from Dan and when he does – to initiate the 2 year run.
• GIBS imagery
• will update code to prevent yellow recurring flood showing for now.
• release late Jan
• GeoTIFF 
  • currently using  GDAL
  • specs: tile by tile just for the flood products
  • 2&3 day
  • naming? in the past never put more than one dataset in GTIFF file* Dan to give this to Greg
  • not all readers can handle multi-layer GTIFF
  • start with a single band.
  • get some user feedback.
  • archiving GTIFFS : need to have names for each file
• needs to be added to PGE 159 - Action @  Greg to do for NRT code
• Database
• Dan sent thresholds to David as a placeholder
• PGE 159 Updates
  • Could get them from Curt but he will need a test plan 
  • Action @ Greg and David. get our own samples by running them in MOD_DevC7   
  • if this doesn’t work ask Curt
  • Sudipta and Sanjeeb running PGE 155 when running in OPS had to remove old files (from their test) as their files wouldn't run wtih PGE 159. 
  • need new test plan that doesn't overwrite. 
• David working on this. He has a sample format. working with the code.
• hope to have multiple samples. 
• User Guide
• Ongoing
• Flood Alerts
• Dummy thresholds sent to David
• Work plan - Ed has submitted this ESDIS
  • ? 1 day = one water detection, 2 day = two water detections etc
  • could change threshold and make it dependent on the number of observations. this would get rid of some of the false positives. 
  • F: will look at compositing 
• Ed, Robert and Sadashiva had a conversation. Fritz will put together a limited proposal for ROSES A33 to do some post processing of the initial flood mapping using fuzzy logic to reduce false positives. also details on the flood alerts.
• Question about NOAA VIIRS Flood product – how does it differ? Should LANCE produce a VIIRS product if NOAA already has one?
  • they claim 95% success in terrain shadow problems
  • use a geometry approach for cloud
  • for terrain they use surface roughness via DEM
  • NOAA doesn't produce a surface water product. they mask and only deliver flood.
  • not sure about archive and long term distribution - only go back to 2019.
• GMU and UWM have a system running and a 2018 paper. In process of transitioning it to NOAA
• they did a comparison with the MODIS product in the 2018 paper.  
• Fritz is still trying to figure their flood alert. 
• VIIRS has SW band - good for water
• What about Sentinel 3 – will we get NRT data?
• Fritz asked Dan how long for experimental server to move to LANCE
• Action @ Dan to add this to the schedule

• 3 day run in OPS. Plan sent  by Greg.
  • Greg to send a reminder and Sadashiva to check 
• Schedule - Diane sent around and would appreicate feedback.
• Flood alerts: 
  • Dan to send sample threshold data to Greg and David. To add in to dbase.
• David adding data to dbase 93 days. Sept 8 to now.
  • 430 million flood pixels
  • looking at images seem to be static floods in a bunch of places that are likely terrain shadow. as sun gets lower they will likely get bigger 
  • Fritz is having to go to a high no of SD to get a reasonable number of alerts. 
• Dan has been looking at GIBS / WV. 
  • terrain shadow were not as up to date as they should be. gave to Greg to update. 
    • only affects Hawaii
• Due to technical problems on Terra will not get as much data on the N. latitudes. 
  • on legacy system there are missing tiles on 70 N.
  • seems to be a difference between NRT3 & 4 for legacy files (not the LANCE).
  • MODIS went in to assembly anomaly on Oct 5: one of the superset went bad. and can't be used for storing onboard data. 
    • to avoid overwrite of data. reduce the day rate and increase the night portion. 
      • 46: 54 day:night ratio
      • Oct 7 - Nov 8: then field operations changed it as you get overlap at the poles so now alternative orbits. You will see something of a zigzag in the imager.
      • to recover the memory it would need to reboot the solid state recorder. too much of a risk.  
• Modify PGE 159
  • David has information from Sudipta
    • needs some end of the day water files – David to get this from Sanjeeb 
• Dan to contact WV / GIBS

•  Still waiting on OPS code for 3 day processing. Greg to follow up
• Do we need to wait for forward processing of C6.1 - No.
  • In Jan may run C6 and C6.1 on separate servers 
• Imagery should keep going to GIBS for test instance of WV.
  • Currently it is the only C6.1 product running on NRT4.
  • The CR imagery from nrt3 is also being sent to the test instance of GIBS.
  • Dan monitoring this - flag if there are any issues.
    • 721 CR - blue color set for surface water looks like snow and ice.

• Still Need to modify PGE 159 – flood detection to run in OPS (to run so fewer output files) and to use outputs from 155.  

  • This will not impact getting the 2 years of data for Dan. as the OPS version of PGE 155 will run with the existing  PGE 159.
  • David is writing the update to PGE 159 - This will be done after new year. This will need to be tested before running in OPS (3 day test) and then we can decide whether to re-run the 2 years worth of data. 

• LAADS production - looks like nothing is running. 
  • Greg to contact Curt to see what is happening
  • Sadashiva thinks Sanjeeb and Sudipta were perhaps confused about which version to run.
    • Seems like a communication error? Greg has asked for it to run. Questions come back.
    • Greg to ask Gang to run it cc'ing Curt
• Dan: product evaluation - waiting for run.
  • working on the user guide
• Flood product in Worldview
  • looks good. useful for pointing out areas where there are problems
• GeoTIFF
  • generate GeoTIFFs
  • there is a tool that works with HDF4 to convert to GeoTIFF
  • this could be done via processing in the PGE
  • need to look at the extra volume it would take to archive GeoTIFFs as well as HDF
• David Landis:  1.5 - 1.8 million flood pixels/records per year in the database. Lots of false positives. As for size - we can deal this as we go forward and think about designing the alerts and web displays. Sounds like we might need a different approach to the FIRMS approach. What is the requirement to view the data? Could use the database for the last month and if you want longer.

• Status of OPS production (Greg):
  • if OPS runs using NRT rule it generates more files it would if the standard files (land L3G) were used. The difference is a factor of 4 in the number of files created
    • some concern re: disk space
    • agreement will run with the current production rules to get Dan's test data done but this could be modified for other OPS running
  • running wrong version of PGE 155 (still failing and not processing all granules overlapping one tile). maybe need correct version in recipe or 
  • need to re-do PGE 159 to use the combined rather than 1 datasets (last granule that came in). may take a month or two.
  • Dan asked about the c-datasets that PGE 159 needs. Does it allow you to see separate granules? Yes its a 1 dimensional array that allows you to identify these.
  • Sadashiva: even in the current implementation the way the L2G works - it finds max  coverage observation and puts it in first layer then puts it in compact layer
  • Name conflict vs production rule issue. Greg to follow up with Sadashiva
  • Go ahead with test for validation data for now, so we can get back on schedule then look at how to update PGE for future exercise.
  • Could run new version of 155 as long as the output goes to another archive. 
• Product evaluation (Dan): almost complete.
  • working on the user guide
• How will the product be advertised?
  • communication all needs to be done via the LANCE website. 
  • Private vs public archive (via authorization)
  • Need to get permission from project scientist Hal Marring to determine if this can be archived as an applications product. Need to revisit this before Ed leaves.
  • Need to coordinate archive NRT forward processing. Should be routine.
  • Greg to talk to Neal about this and Diane to contact Ed re: permission from Hal Marring. 
• GIBS/Worldview
• No word yet from GIBS on whether the data is being ingested for testing. Diane to follow up

• Status of OPS production (Greg):
• No word yet from Kurt, who is needed to run the test. Has been somewhat MIA; Greg will check in with Gang to track him down and see where this is.
• Product evaluation (Dan):
• After more examination of differences, it is apparent that *most* differences are due to the larger number of looks available in the LANCE implementation, where swaths overlap (towards the poles). Which is letting in more cloud shadow false positives, esp in the 2-day product. A second order (after Nov 30 release) project will be to examine alternate compositing thresholds to see if these false positives can be reduced without also significantly reducing true water detections.
• Currently finalizing summary graphs illustrating differences.
• Next will develop User Guide.
• Will focus transitioning legacy system to new server only after User Guide and Evaluation Report are completed, assuming 617 Sys Admins will allow the existing server to continue operating a few weeks past the CentOS 6 EOL on Nov 30.
• Joe Nigro was able to find his raw database of specific dates for the flood and non-flood events used in the existing evaluation report; this will make it much less time consuming to re-evaluate those sites with the LANCE product.
• Current NRT production issue: current day products not reaching nrt site until end of day (Greg):
• Greg needs to work this with Neal
• Release schedule - Nov 30 still feasible? (Dan)
• possibly, if we can get OPS into production to generate the evaluation data.
• GIBS/Worldview
• No word yet from GIBS on whether the data is being ingested for testing.
• Database ingest (David)
• Continues, and data is looking good
• Alerts (Fritz)
• Initial attempt was generating ~5000 alerts per day, based on default parameters...working on ways to revise and filter.

22 October 

Greg, Dan, Sadashiva, David and Fritz

PGE Code

• all PGEs have the most recent version.
  • these are running in NRT but not yet in OPS
  • Greg wanted to re-run the 3 days in OPS to check all working
  • Actions
    • Sadashiva to check with Sudipta and LDOPE to confirm that they should check out the recent version so the 3 day test can be run in OPS 
    • Greg to make sure Curt and Gang run the 3-days
    • Greg to let Dan know when this is running
    • Dan to check the output from the 3-day test

• Greg still waiting to hear back from Navid on hardware. 
  • Greg to follow up

Evaluation

• Dan has developed some tools to look at the NRT product in more detail. Still looking but overall looks good. When will this assessment be complete? Should be next week.
  • the main issue is the one day product looks different due to the different cloud mask. And the far north looks wrong but aware of this and will remove the 70 N tiles. 
• Bhaskar is developing pages in the LAADS website to include one for flood. He was asking about what products will be public. Dan shared the table to clarify.
  • Greg sent a link of a similar page: https://ladsweb.modaps.eosdis.nasa.gov/missions-and-measurements/applications/nacp/
• Dan getting original study from Joe Nigh. 50 flood events and 50 non-flood events. 

Removing 70N tiles

• Dan has confirmed top row need to be removed - this should be done once we have a weeks worth of imagery in Worldview.

Imagery

• Files are being produced. 
  • Diane checking with GIBS: they will be setting up processing in UAT in the next couple of days. So sometime next week it should be viewable for the science team to start their review of it.

Database and flood alert program

• David has written a tool to load the data more easily
• Fritz has looked at 174,000 AOIs - which led to several thousand alerts
• Discussions with David ongoing

Next week: Diane on leave - meeting to go ahead anyway

PGE Code

• All PGEs were updated to NRT 4 and to OPS. 
• Limited name changes in 155 but 152 and 159 the names and metadata are updated.
• Fill values are fixed and consistent.
• ingested all mask files in to OPS
• asked Curt to re-run test days in OPS. once that works will run the year or two years required.
• Greg sent email to Navid

Evaluation 

• Dan is looking at the differences. No big surprises. Wants to look more - trying to quantify it in a meaningful way.
  • Nothing worrisome.
  • would like to look at the specific dates from OPS

Imagery 

• file are being produced

Removing 70N tiles

• Dan to confirm which ones need to be removed - 
  • top row
• will remove in NRT so imagery doesn't show up in GIBS, but not in OPS

Database

• too many rows to use like we do in FIRMS.
• Greg hasn't had time to think about alternatives
  • rolling update?
  • not sure how it fits in with what Fritz is doing with different thresholding.
  • only keeping flood pixels but sounds like we might need all pixels.
  • David to start looking at the flood alerts and how it might work. Fritz to send David what he has already done.

Fritz - running statistics for floods

• wants to run the stats on a chron job - once every 15 mins or so?
• New server - Dan will need SA

8th October 2020

Dan, Greg, David, Sadashiva, Fritz, Diane

PGE Code and Evaluation

Dan still working on the evaluation. Greg discovered terrain shadow files were not there for all tiles and that was a problem for the production rules.

• Dan had to populate empty files to provide to Greg get the code to run. A complete new list of mask files have been ingested to NRT4 and OPS systems. The NRT4 looks like it is processing them.

6 files over the course of the year were missing or incorrect in the 70 N row. That row is a mess and the files need to be updated. It is generally frozen in the winter so there is an argument that we don't need to run those in the winter.
action @ Dan to consider adding that as a caveat for the End User guide

Suggested solution

• remove the 70N from the tile scheme until March action for Greg
• in OPS run all tiles for the dates required for the evaluation
• need to talk to Maki about running a different tile scheme and not produce those images for GIBS

More on PGE Code

• Gang, Neal and Greg have been working on finalizing the production rules. Some issues still re: files not showing up until the end of the day. 
• action @ Greg to solve problem of why files are not showing up as soon as updated. and check to see the imagery will get processed when 
• File name changes requested by Dan. Greg has these and has made changes to the code. He needs to run through the tests to make sure they are okay. will go to production on Monday. 
• action @ Greg needs to notify Sudipta know about the file name changes. To do that today cc'ing Dan to make sure it is correct.  

Imagery 

• changing the file names. Need to get these finalized, then should be okay. Looks like next week before we get the first image tiles out next week.

Database

• managed to load 29 days worth of data (3-day flood layer). 146 million flood pixels for 30 days. that would be (x 12) 1.7 billion pixels for a year.
• purpose of the database: to render images, manage alerts, could be used for statistics (not currently done for FIRMS).
• Based on the information provided by David - the number of pixels will be too big to keep them all. maybe we need a rolling archive? TBD

Alerts

• Need to understand the logic for the alerts.
• Set up a prototype.
• Look at design (data files vs database)

Note: Ed has said we can proceed with the alerts. We don't have a mandate to do this from the UWG but they didn't object when Ed said this was the plan. However we do have a mandate to  get the flood product out so any work on the alerts is fine as long as it doesn't detract from getting the flood product into LANCE.

• Statistics that Fritz was running is complete. Looks at planet tiles that fit in planet-scope size areas.
  • Machine to machine alerts would be useful for the flood community
  • Fritz is going to be looking for new alerts in the data. 

Old Flood system

• Existing Server becomes obsolete at the end of November
• Push the data archive to NCCS. Most of the product data is on the MODAPS distribution server. 

Need to re-visit:

Plans for improving the terrain shadow and cloud shadow. 

1st October 2020

Dan, Greg, David, Sadashiva, Fritz, Diane

PGE Code

• Ed gave the go ahead to keep the code running on NRT4 (C61)
• Dan wrote some code to come up with statistics based on the products being produced in NRT - the results should be available in a few days.
• OPS code: Greg is in the process of updating the code to require the mask files. Should be done later today. Once done - he will run the 3 day and make sure they are being used correctly. Once done we can start on the larger test towards the evaluation.

Question from Dan: can we rename some of the HDF layer files? Yes - it is quite a bit of work.

• action @ Dan to send Greg the list of names that need to be changed. 
• action @ Greg to keep LDOPE in the loop as GIBS layers will change too

Imagery

• PGE is baselined and looking to get this in to operations on NRT4. Not seen anything coming out yet.
  waiting to hear from Neal - action @ Sadashiva to check the status of this

Database

• loading data in to the database. 20 days loaded of the 3-day
  • likely to get more terrain false positives in the winter
  • action @ David to let Dan know how many positive hits on a specific day over the past week so Dan can compare it with his stats – to give us an idea of what the database size will be

Flood hardware

• the hardware is there. There is a need to install mapserver if we are going to generate flood imagery using the dbase.
• need a website. no docker container yet that holds this. 
  • will be setting this up as part of the FIRMS - to docker website. Estimate 1 month. Asen and Otmar will work on this at the same time as FIRMS. 
• will need to put together a docker container for python. SA will not be responsible. Navid's group will likely be responsible for deployment. There is a formal process where approval comes from management (e.g. Robert, Bhaskar for MODAPS). For flood this could be Dan / Sadashiva

User Guide - looking at this.

• Sadashiva: user guide to cover only the products that are released to the public

Stats for the Alerts

8 scripts finished. 5 still running. When done will bring in new datasets and can start issuing warnings when exceed certain thresholds.

Are we still on track for November release? Likely

• biggest issue will be getting the operational code running for the evaluation.  
• action @Dan to send the years required for historical processing in OPS Greg

PGE Code 

OPS code: Greg has a ticket in to update the code to require the mask files 

Dan: quantifying the new vs old product over a couple of weeks.

• the apparent diff is the MOD09 being used for cloud shadow instead of MOD35.
• Then will need the data from OPS to do the full evaluation

Imagery - Sadashiva

• The code is on NRT 4. Neal was trying to figure the streams for GIBS. (one for test and one for OPS).
• need CR from NRT 4

Flood Database - David

• reset to add in the 3 day product only 1/10th of the data. Screenshots of the 12 days → movie.
• flood showing in N Canada and Siberia. huge improvement over what we had a month ago
• Next steps:
  • David to continue loading the data:
  • Get Navid to set up another database as they are on FIRMS2
  • Focus on figuring out the size
  • Greg checked the hardware monitoring system and they have some assigned. need to figure out how to get stuff on to it. prototype flood on FIRMS would need to move it across to new server. action for Greg to figure next steps

Stats code still running

• 13 scripts running one for each latitude band. 
• Breaking things in to the Planet size boxes Quite a few that don't meet the 80% cloud free requirement. Some 110 meet the requirement but had zero % flood coverage

User Guide - Dan working on this.

PGE code

Greg: No changes to the NRT code for 2 weeks. NRT is running well. There was an issue in OPS where they had forgotten to stage the mask files – Greg has been working with the OPS manager to help get them staged. The code runs without the mask files. Sadashiva suggested we make it a requirement (not optional). Greg agreed

@action for Greg to check with the OPS manager to see if he has staged the mask files.

@action to modify the production rules to make it a requirement to have the mask files in order to run.

Dan: Not looked at the NRT yet. Will do later today or tomorrow.

GIBS image product.

Sadashiva: NRT 3 & 4. currently running on NRT 4 (test), NRT3 is operational and visible. Flood map has been running in C6.1. C6.1 is still running in test. Can run every day on NRT4. If we want it visible to very user need on NRT3 but that is running C6.

If you baseline as C6 it can use C6 SR. So the question is – are we ready to go public.

NRT 3 be moved to C61 in October (polarization fix for Terra). Close to approving C61.

Keep it on NRT4 for now as an Alpha. From Matt - If MODAPS and GIBS added the flood product as an operational product, we could process it and just not add it to the public WV. But we would have a test WV instance that would let us see the imagery.

Flood Database

David is seeing some pixels that aren't flood. A lot are clustered along the edge of cloud. so there are cloud shadows that are not being removed. Is the MOD09 good enough? May need to screen out areas where we are not going to report alerts. This should be okay in the 3-day product.

@action for David

Next steps: reset to add in the 3 day product.
Give examples of size of data base for 1-day

May struggle with the size of the database. It is much bigger than the fire data. Dan suggested we could have a rolling archive. Need to think about the design.

Scripts for Alerts

Fritz: running scripts again.

User Guide - documentation - still to be done by Dan when time permits.

• This should also include FAQs and information for the Earthdata LANCE pages

10th September

PGE 

• Running in NRT with both A&T and set up so it can run with just one or the other.
• Ran 3 days in OPS looks like there is an issue with the tiles around border of ocean /land
  • probably as mask file not staged properly as it works fine in NRT
• Visually the data from NRT looks good. 
  • next step will be a statistical comparison
  • once OPS are running we can run the dates for the evaluation

Imagery

• No updates. we were waiting for it to be running in NRT
  • it has been running since the beginning of the week so action for Sadashiva to follow up and transfer GIBS imagery to NRT system.
  • PGE was ready and tested in house. hopefully can transition to NRT and then on to GIBS.

Flood Database

• Loading test data. Seems to be running fine. Noticed the problem of flood 'ringing' bodies of water.
  • visually it still looks like the whole world is flooded. 
  • Action for Otmar

User Guide - documentation - still to be done by Dan when time permits.

• This should also include FAQs and information for the Earthdata LANCE pages

3 September

PGE

NRT Production

• Yesterday production rules to allow code with just one instrument. It seemed to run okay.
• Algorithm code seems to be okay. No issues found since the pan-sharpened product.
• Now that Aqua is back - 

OPS 

• Fixed operation rules. Issue caused by additional PGE (11P - a night time PGE) that runs in OPS but not NRT.
• Some stuck pieces - something that operations needs to deal with. Hope to be finished today.
• Sanjeeb was making changes to handle issues he was dealing with in PGE 152. Preparing a release for that. 

Imagery

• as no products being made due to Aqua loss
• on OPS side same issue - PGE wasn't 

BRDF and Albedo MAIAC. won't come back on as multi-day products using both Aqua and Terra. Working to make combined products with whatever data is available. After 8 days everything will be stable and usable.

Flood Database

• program that loads the flood data has been re-written and runs well. 
• one day worth of data - August 22. except large number of flood pixels along the shore of any large body of water
• From 19 million to 2 million pixels when you go from 1 to 2 day product. 
• want to do more test

Alerts Prototyping activity

• Tiles and river basins to start with. Can we prototype both?
  • would need statistics for the river basins 
  • aim to do this manually / run the stats and see what the alert would look like 

Fritz

• server - end of life.
• checking for funds. not enough to cover a new server until new fiscal year.
• In Nov will need to update the operating system

Found another bug on the flood alerts. Need to start again. for statistics.

20 August

PGE code

• Greg fixed issue with false positives at the edges of some tiles. This was due to an error in band sharpening code (similar to PGE 100 written by Jeff Schmaltz). This was fixed in Jeff's code in 2016 but other code was not updated. Should be resolved now.

Standard Processing

• Code ran for 3 days. Some issues - probably in the production code.
  • Action@Greg to look at this

Flood imagery in GIBS

• Dan sent final color map scheme to Sudipta
• Sadashiva will check on progress. it was agreed it would be great to have a demonstration ready for the LANCE UWG but while desirable realistically there probably isn't enough time. 

Flood Database

• David got the ingest code working so it is ingesting in to the database.

• Otmar put together a prototype: https://firms2.modaps.eosdis.nasa.gov/floods/

  • comments / feedback welcome - to Diane

LANCE UWG

• should be able to show some imagery from Dan and from the prototype

Flood Statistics

• Fritz is running flood statistics for the archive, based on the Planet tiling grid (25 x 25km overlaid on the MODIS grid). Fritz has access to Planet imagery so is able to compare with the high resolution imagery.

Email alerts

• will we use the Planet tiles or watersheds or both? To be confirmed. Possibly both.
  • action @ Dan to confirm how the watershed statistics will be determined and how often they need to be updated. This should be automated.

Hardware for floods

• Greg to check with SAs

13 August 2020

PGE Code

-- Issue with terrain shadow mask not being applied is fixed. It was an issue with production rules not finding the correct file to load.

-- Dan has identified a possible issue with false positives at the edges of some tiles.

-- Dan has identified a possible issue with totally empty L2G files.

Standard Processing

-- we have loaded all the necessary information for standard production into ops, and will run a preliminary test of three days to make sure everything runs correctly.

-- Dan will identify a list of test dates to use for the evaluation test

Flood Alerts Database

-- David has been testing the database code, and it looks good. He has added several days of data and a reasonable number of data points are getting added.

-- Greg will see if Otmar can mock up something that can view the data from the database.

Flood Imagery to GIBS

-- Dan received feedback from Matt on color tables.

-- Dan fill forward color table information to Sudipta.

-- Image generation code is mostly complete, ready for final testing once final lookup table is integrated

6 August 2020

PGE Code

• Data looking good. Dan still checking it. 
• What would happen if we were missing data? 

• it would be a manual process to correct this. the production team are used to re-processing data.
• for the most part - the data are stable. sometimes there are maneuvers.
  
  
• there are differences in the way NRT vs OPS handles granules during maneuvers.
  • NRT 3 steps to check if quality is impacted. not perfect but suspect manuals will be removed as part of the pre-processing. (Automated process).
  • for the OPS system the maneuver data are processed off line and QA checked by LDOPE and a decision made as to whether it should be removed.  
• If the Flood team notices bad data is going through then they should let LDOPE know and they maybe able to fine tune it.
• The missing data issue Dan saw was related to files that overwrote each other. Neal and Greg have fixed this.
  
  

what should we call this product? 

• Alpha after Dan has looked at it. 

• Beta after the Evaluation report of 100 sites (same as existing product). 
  

• Release at the Beta Stage and get it into Worldview. 

• Beta stage timeline? Depends on how long it takes to set up the standard processing. 

• Beyond Beta - recurring flood over 10 years. updating the terrain shadow etc. Essentially a new / better product. 

Evaluation Process

• Dan's plan is evaluation rather than validation; it is mostly inter-comparison where neither is validated using truth data. 100 sites

Standard Processing

• Sadashiva has emailed Gang but not heard back yet. L2 should be straight forward. L3 more work.
  • They won't need to deal with challenges of NRT - can fall back to how it is implemented in OPS. 
  • Gang needs to assign someone. 
  • Action @ Sadashiva to contact Gang to see if we can get the evaulation done by the LANCE UWG on 9 September

GIBS imagery

• Action @ Dan to ping Matt to see if his suggestion is okay 

Flood Alerts Database

•  David got access to the dbase yesterday so will test program later today.
  • previous version was putting out the proper data in to a TXT file.
  • Once data are there:
    • start looking at displaying the map images like we do with FIRMS and start on alerts. There are still some issued with the alerting system. Fritz has been looking at.
    • Fritz has been working on the statistics by latitude bands. This will probably take a week. Based on the planet image size. Software can be changed to do it by 1 x 1 degree

User Guide - documentation - still to be done by Dan when time permits.

• This should also include FAQs and information for the Earthdata LANCE pages

Dan,Greg, David, Sadashiva

PGE Code

1. Something is going on in the northern tiles - the reflectance is low and detecting all as water. Greg thinks the real issue is in the input from MOD09 files. Greg is taking a look at this and will email Sudipta.
   There is a band at high lat that gets classified as water. it is obviously artificial. Maybe a quality flag needs to be turned off
2. Current production includes more tiles than the original product. Option to turn these off for the initial NRT processing. Once the standard processing has been run for some time (long enough to produce additional reference water) then these tiles could be added back in. If they are left in now without reference water mask they will show up as flood when they shouldn't.

Tile schemes list all tiles to be processed. so removing of tiles can be done here.  Sandeep and Neal can help with this.

Sadashiva discussed standard processing with Ed. 

• action for Sadashiva to send follow up email to Ed  and action for Diane to contact Karen to see if she has heard anything

GIBS imagery

• Dan sent email to Matt to check what he needs to provide to Sudipta. 
• action for Dan to provide Sudipta with the final color scheme

Flood Alerts Database

•  David was wondering if he was getting too many pixels for the tip of S. America
  • Greg checked code and thinks it looked reasonable
  • need to test by running code and populating the database

User Guide - documentation - still to be done by Dan when time permits.

• This should also include FAQs and information for the Earthdata LANCE pages

16 July 2020

PGEs

Greg sent Sudipta the issue about PGE 152 about the LPGE today. They are looking at it now. Sounds like the problem maybe to do w/ the deep ocean flag.

Sadashiva discussed standard processing with Ed. There are a couple of points that need to be considered.

1. MODAPS will do the necessary processing for the validation but before that we need to be sure the PGE is generating quality data for validation
   Dan - had planned to do a qualitative comparison between the new and current data. For Dan to do this we need the PGE issues finalized. Need issue with PGE 152 finalized and data checked by Dan
2. Come up with a plan for what we need to do the validation. How much data? Global vs specific sites.
   1. Dan - would not need global for the 50+ validation sites. That will enable the validation to be updated. If all okay then run 10 years of data to update the flood water product. Again could do this strategically. Do certain areas for 10 years then expand

Action @ Dan to come up with proposed plan for Ed and Sadashiva

GIBS imagery

• in progress. they have tested the PGE locally in LDOPE
• Action @ Dan to provide the LUT 

User Survey

• almost finalized.
• hopefully ready to send next week
• Dan is liaising with DFO

Flood Alerts Database

• program is pulling the data out from the images but seems to be pulling too many points. 
• Greg was going to review the code but didn't get to it this week.
• Fritz is doing some tests on alerts on the old system.

Action@ Greg to look at David's code

User Guide - documentation - still to be done by Dan when time permits.

• This should also include FAQs and information for the Earthdata LANCE pages

9 July 2020

PGE 159: small changes made

PGE 152: more significant changes made. some failures due to PERL code. Running since Tuesday. Data should be good. and PGE 159 should be using that data correctly.

• There was still a question over data values in the cloud mask
  • action for Dan to look at this and see if it needs updating
  • not showing today's data on the website. Neal needs to look at this when back from leave - data will appear on the website at the end of the day but it should be available as granules are processed.

Putting the code in to standard processing

• waiting to get the okay from Ed about what level of processing should be supported.
• will support what is needed for validation but maybe looking for something more.
• action for Sadashiva to send follow up email to Ed 

GIBS imagery

• LDOPE has confirmed with Dan that he recommends all three ( 1,2 and 3 day) products.
• Code being written.
• action for Dan to provide Sudipta with the final color scheme

User Survey

• DFO - to run the survey. Can't run this through NASA easily
• Small updates to be finalized by Diane and confirmed by Dan
• then Dan should run by Bob Brackenridge, Albert and Fritz.

Flood Alerts Database

•  David is working on SQL commands. Not hooked in to test table yet. 
  • want to check the right values will be populated (900K entries)
  • currently generating CSV files for testing
  • wondering if getting multiple listings for the same pixel. 
  • action for Greg to help check PDL code.

User Guide - documentation - still to be done by Dan when time permits.

• This should also include FAQs and information for the Earthdata LANCE pages.

1 July 2020

Discussed feedback from LAND MODIS - VIIRS meeting

• A question was asked:  Can Flood Map process generate annual land water mask product similar to the one that Mark Carroll is generating using the MODIS 16-day composite reflectance product?
• re: historic archive for flood (and for other orphaned products). This is a challenge as
  • resources to do the QA are greater than the resources just to archive the product.
  • ideally need a science team to take care of any changes in the instrument calibration.
  • larger issue of re-processing for different collections
• For short term solution: ESDIS funding. fine
• Long term solution: ROSES call.

Dan sent a document sent to Ed about the next steps. Waiting for feedback.

PGE 159:

Greg installed fixes which broke everything in production! The fixes were to do with the cloud and cloud shadow processing. The way it was originally done - it was mixing up pixels. some areas in the image where pixels got shifted. Greg found the error and fixed in in PGE 152. Right now the previous version of the code is running until the new version ready.

Putting the code in to standard processing

• waiting to get the okay from Ed. 
• once we get the okay there will be some work to be done to test the code that Greg has written
• Sadashiva to follow up on his

GIBS imagery

• no progress this week. Sadashiva will follow up.

Flood Alerts Database

• the database has been mostly done by David.
• Greg had given David the code to do the inserts. He had started to do this. not finished or tested

User Survey

• Diane waiting for okay from Karen/Robert - - action Diane to finalize and check okay to move ahead.
• Once we have this - run it by Bob Brackenridge and Fritz.

User Guide - documentation - still to be done by Dan when time permits.

• This should also include FAQs and information for the Earthdata LANCE pages. 

25 June 2020

• Dan has had a closer look at the data and is happy with the overall product.
  • still looking to confirm everything is okay. Action Dan - by next week should have finished.
• Greg did contact Gang to see about standard processing for historical data for validation (STIG). 
  • Action - Greg to follow up with Gang
    
    
• GIBS imagery: still waiting for the test. Not heard back yet. Sudipta is working with Maki
  
  
• Greg emailed Ed, Robert and Navid re: hardware for alerts
  • Navid suggested this should be developed for Kubernetes like deployment using the archive for data storage instead of relying on any local disk
    • No development environment for Kubernetes
    • Action - Greg to check where Navid is expecting the development to take place. perhaps on local machines but as there is no data on the local machines this would involve data having to be copied across. 
      
      
• DBASE table - David has been working on this and made good progress. Code used to take data from the images → the database
  • it will be a lot of data!
  • not connecting to anything yet 
  • prelim database on FIRMS2 for now
    
    
• User Survey - Dan and Diane are working on this. 
• User Guide - documentation - still to be done by Dan when time permits.
  • This should also include FAQs and information for the Earthdata LANCE pages. 

11 June 2020

• Update from Sadashiva
  • LDOPE now has global browses of L3 water composite, and two different cloud masks that Dan wanted to review for suspected differences.
  • LDOPE will soon start working on the GIBS image for the Flood map.
    
    
• Operational PGE –  Greg contacted Gang to have STIG integrate the PGE to MODAPS following different set of production rules for operational processing at MODAPS. Greg to follow up again
  
  
•  Dan has been writing proposals and unable to get to the flood work.
  • needs to check the output from PGE 159
  • draft strawman for flood alerts thresholds

• Otmar has been looking at the alerts.
  • tabled: waiting to understand how the threshold works
    
    
• User Guide - documentation - still to be done by Dan when time permits.

May 28 2020

PGE 159:

• Dan found a few issues with the logic. Won't be too big a deal to fix them. 
  • need to change where threshold is applied
  • no data not showing up - seems to be a threshold issue. Greg reviewing them. 
  • hopefully the changes will be in production on Monday. so Dan could take another look on Wednesday.
  • datasets look mostly there so we can proceed with imagery

Imagery 

• Greg still needs to talk to Maki
• GIBS team (correspondence) only need updated tiles - not whole days data each time
  • they want to modify one of the no data values. Dan to send info to Dan so it can be coded.

Question from Dan

• once you get the correction done - how is the imagery coded?
  • David will do this and stick it in with PGE 159 → image files
    
    • timeline for this? David has a program running that works. it is a bit slow and needs speeding up.
• Re: Imagery Greg said they still need to understand how they get to GIBS and how that interacts with other PGEs (another weeks worth of work). Maki and Sadashiva work with this so can help on this.  Sadashiva says once we have the product we agree on then they can augment a process to generate the GIBS image. Core code is there. Just a matter of adding correct information in to the PGE. Should be straight forward.
  • Sadashiva will aim to start working with LDOPE on this. 

• User Guide - documentation - still to be done by Dan when time permits.

May 21 2020

PGE 159: Looks good.

• Dan has one day downloaded and needs to look more closely at other days.
  • possibly showing more cloud shadow in Day 3 & 2. than it should be
• PGE was running on NRT4 which is currently down 
  
  
• Sadashiva has been making L3 and L2G images but stopped as code was being re-worked. they can start again with LDOPE
  • what images would you like to highlight to enable a better look at what is going on.
  • Sadashiva to work with Sudipta on this
  • Greg mentioned the dataset names have changed again. Dan to verify they are okay

GIBS imagery

• Matt has a prototype in a UAT instance of Worldview. Looks great although possibly a lot of cloud shadow
• need to determine colors of flood vs regular water. 

David was working on PNG or JPEG images from composite files for easier validation. Really close

Next meeting 28th via TEAMS

David, Greg, Sadashiva, Dan, Otmar, Diane

May 14th 2020

PGE 159: Dan reviewed the data in current production and found some issues in the code. Hope to have another release today or tomorrow which (if all goes well) would mean the 2 day product would be ready for Monday. 

GIBS imagery: Could give GIBS the old (ongoing) product, or maybe it would be better to wait for the new code? Dan to get in touch with some 1,2 & 3 day products. Greg is still not sure what format the GIBS folks want the data in despite corresponding with Maki. Dan thinks Paletted PNG.
Maki commented that it might be a lot of imagery to ship to GIBS. MODAPS send the whole world every time it is updated.
Could we ship just the changed files? Dan to ask Matt

David to work on 10 x 10 images which can then convert to 9x9 as expected by GIBS. 

Moving to standard production: wait until PGE 159 is working properly

Alerts:

1. data to the database
2. thresholding - need to determine the steps
   1. statistics thresholding which needs to be derived from standard production

Next steps:

1. finalize PGE 159 
2. move to standard processing environment to enable: 
   1. validation of the NRT product
   2. with 10 years of data it will be possible to generate a new reference water mask 
   3. once the new reference water mask is in place, 10 years of data should be re-processed to generate the statistics for the alerts. this isn't a huge volume of data but need to keep in mind that at MODAPS there is a lot of re-processing already going on.  
3. work on the alert system
   1. determine water shed region
   2. process for determining statistics
   3. backend process for alerts
   4. UI interface

Sadashiva, Greg, David, Fritz, Greg

May 7th 2020

Funding for Dan - seems to be sorted.

Flood alerts: Preliminary discussion. We don't want to alerts for false positives which are in the product. so if we know how many routinely occur in the area we can set the alert accordingly - with MOD44W rivers etc have moved plus terrain shadow. So need threshold based on running stats over time. How many false positives do you expect. Challenge to dev threshold for each area of interest. Could be 10 x 10 tiles initially. Ideally users to submit AOI but then we would need to develop stats based on this. We have 8 years of data but LANCE product will be different, so ideally we would process 10 years of data via LANCE so we can then generate new reference water.

More complex than FIRMS! If we have thresholds - could apply them to incoming data. open question - where can we generate the statistics. Could be done in IDL, python but not acceptable to MODAPS. The layers could be done monthly or yearly.

The threshold will change throughout the year as terrain shadow false positives will change between seasons. 

Possibly develop regions of interest. Need to give this more consideration.

PGE 159: Greg got code from David this morning. Hoping to get it in to production over the next day or two. Will take a few days to see the impact due to the sliding window. Sometime next week can review data. At that point it will be worth spending time to make sure it does what it is supposed to. Can then talk about what we need to do to run it in the operational system to accumulate statistics over the period of the mission. 

Imagery for GIBS: Greg has been in touch with Maki and Dan. A few more things to clear up but it shouldn't be a big deal to create the issue. It will be a lot of imagery and sending it back and forth to GIBS every 10 mins might be problematic. Currently Maki sends entire world as a set of tiles (600 every so often). It is a lot of data to process. We are at 250 tiles - can we just send updates vs just the latest imagery. This needs further discussion with Maki to see how it can be done without impacting other NRT imagery.

Sadashiva: global products. gibs process put its in to the 9x9 tiles. Matt doesn't do any choosing of the data / processing. It has to be done at NRT to convert approx 250 → 9x9 tiles. 

29th April 

PGE 159: running for 3 weeks. getting ready to add final masking stages where we check ref water masks and set flood flags.

• ingested all masks: part of archive set 1 (ancillary data) done this way so if masks are updated they can be added and kept track of.
• wrapper code for PGE is pretty much ready to go. not yet installed.
• Dan found issues with current product. turns out we weren't using the correct thresholds for 2-3 day counts. 
• found issues w/ some of the logic for the counting in the algorithm, so David, Dan and Greg have been clarifying and fixing these issues.
  • hopefully later this week a new version will be in production. then we should get the complete products with masks and flags

• One remaining issue: still don't have the hand masks converted to ENVI files. Dan was to work on that but ran out of money. Greg could try. 
  • issue with the HAND - there is just one file per tile but Dan would like to look at the threshold to generate the HAND mask and decide what threshold to apply. HAND is an index and in the past 30m has been used but Dan would like to take the opportunity to review this before it goes in to production.  It should only impact mountainous areas so not a huge concern but would like the chance to do that. 
  • Greg: can put logic in there with threshold as zero until have the file. Logic should be if file doesn't exist - it should continue without e.g. for flat areas.
  • Will want to run it without the HAND mask to enable Dan to cross compare with existing product. 

Way to handle different masks

• will go in to different archive sets. probably won't change much. discussed this w/ Neal - seems the best way 

GIBS Imagery

• Dan was unable to follow up on this as he ran out of funding for the month
  • SSAI needs a new sub-task to what was the old 6026. 

Funding

• Ed and Frtiz talked about budgets. Ed will talk to Keith / Ernie to see if Dan can start charging to this now. 
• Future? Possibly a MODIS / VIIRS combined product? 

Flood Alerts

• experiment with the alerts / next steps.
• thresholds need to be run for each user AOI
• would need approx 10 years of archive data to come up with the thresholds. Have the existing flood data base from 2012. 
  • depends what mask is used. 
    • improve terrain shadow mask to reduce false positives would change the statistics. so need an archive for current product. 
    • using Mark Carrolls land water mask
      • Dan mentioned they would prefer to derive one from this product so it is more consistent.
  • initially historic processing would be archived at MODAPS. could be delivered elsewhere (LP DAAC or Hydrology DAAC) 
  • Ed thinks we are at the stage where ESDIS seems okay with it

• Ed will let us know when he has talked to SSAI to let us know the plan for charging.

22nd April

Production of PGE 159. Greg has been working on getting the mask file in to the code. Following some back and forth with the CM and DB folks it looks like they have a workable solution.  The masks will be filed like regular files so they can be downloaded (binary and header envi files) by users. DBase are tables ready to go. Greg thinks the ingest will start today or tomorrow.

Once done the next step is for Greg to take the code David has given him (with ref water mask) and put that in to production. 

Dan: looked at the data and thinks like the wrong threshold was used on the 2 & 3 day products.

• Greg to look at these thresholds and see what is going on.

David: Terrain shadow - had some questions for Dan. Cleared up. It turns out the test file being used had no terrain shadow in! 

How to handle and mask and versions of the mask files: Greg said there is no easy solution but suggests the best way forward is to use a "recipe" - this is a description of inputs necessary to run PGEs. the recipes run the production rules to decide what inputs  will get used. so there could be a recipe with mask "on" and another with the mask "of". Once in the recipe it will be easy.  You can also run the recipes simultaneously should you want to compare outputs.  

• Dan asked how the different products would be identified. Different archives, and the metadata would list the input files. Since this is standard input data (just like any other in the production system) the input files could be referenced from the archive system (metadata field = input pointer). 
• Sadashiva confirmed that these alternative runs would be limited for validation / comparison.

GIBS imagery

• Dan has received a reply from Matt. needs to look at this in more detail.
• Greg will talk to Maki about image generation. She can look at what it will take to generate imagery.  It is likely to be a transparent layer with the data points on it. Need to make sure we have a "no data value" available (that can be turned on / off or made opaque).
• Greg to contact Maki

Consider LOE flood alerts.

• there are system issues to discuss:  would it run on FIRMS? a separate system? should it be run it like FIRMS, or we come up with a more modern approach? there are a lot of system questions that we haven't begun to think about.
  • add Otmar to next week's discussion if he is available to look at options.

Fritz needs to tag up with Dan to finalize costs to finish up costs for phase 1 including water mask, user guide etc That didn't include the alerts. The alerts were under the follow on work. Fritz needs to put more money on the task. 

Question on running validation data

• Once running this provisionally, we will need to run some historical data, so this will need to be tested and run on LAADS. 
  • if this is a short period this shouldn't be a problem but it will still need to be set up
    • it would take some time to set up (a couple of weeks). Greg has not put any effort in to getting it running there - but it should be similar to the process on NRT so he doesn't envisage any problems.
  • Sadashiva asked if it would run as in NRT or at the end of the day (as if in standard processing). This means they will run slightly differently so we will need to look at the differences and identify the cause of them.
    • end of the day 
    • is the PGE set up for that? It will have all end granules rather than as they come in. Greg thinks it will cope; it is designed to take any number of input files. if it sees water it notes it, if on a subsequent pixel there is no water the PGE still keeps the water in there. 
    • L3 process will see one file per tile. it is till requires L2 PGE to get water product and something to grid it. 
      • Sadashiva said gridding will be one file for entire day. Greg thinks this is okay.

Production of PGE 159

Looked at the data. from Greg's perspective it looks good. Accumulating data over the 3 days from both Terra and Aqua. Production seems to be stable (approx 200 files per day as expected). Neil and Greg were looking at the files chosen: 6 are over the ocean so after reviewing with Dan it has been decided to remove those, and a few others as recommended by Dan.

Dan - said the 1,2 &3 day all look quite similar. maybe showing up too much cloud shadow. Not sure why...

• Action Dan/Greg Need to dig to see what is happening.
• Non-observations file. not used. this is generated and used by the gridding algorithm but Greg is not sure what it is used for. Stripped images. It is not generated by the L2G nor used by the L3.

Yesterday David sent an update which includes the reference water mask processing. so the next step will be to add this in and run a release.

David still to add terrain shadow to code 

For follow up

HAND mask  Is there an on/off switch could be applied? how could the products be differentiated in the file name? 

• whether HAND mask was applied and what version should be added to the metadata to file. 
• there are standard metadata fields in the HDF

have we had an answer from Matt / GIBS

Greg will talk to Maki about image generation. She can look at what it will take to generate imagery. 

Consider LOE flood alerts

Production of PGE 159

Greg said 2 days of data have run well. That equates to approximately 250 files for days 98 and 99. So looks like the production rules are okay.

• next thing to check the data looks reasonable (same link as before) - action for Dan and Greg
• figure out how to get in process files to show on website (show in production if you know where to look) 

Dan has completed the Terrain Shadow grid.

• HAND - not yet done. validation needs be done first. before it is applied so there is no rush.

David - is close to getting the land/water mask added in to the code. He is still getting errors so will talk to Greg.

• once the land water mask is in the code the terrain shadow (and HAND) mask should be easy to add.

User guide & validation

• Dan has looked at the user guide and doesn't see any issues. 

re: validation:  the plan is to look at concurrent files and processing historical data for certain areas.

• Dan asked how processing the old files would be done?
• Sadashiva: suggests this should be done once the system has been running well for a short time.
  • have it run on the current system. do a thorough review of the data (can get help from LDOPE) including compare it with the existing products coming out of the existing system. then determine what historical data is required to enable the validation 
• Greg mentioned that they would also need to set up production rules as we don't have historical data in NRT. Rules for science processing are slightly different. The inputs would be science quality SR products. Should be a simpler process - run once a day as don't need to keep track of incoming files.
• need to come up with an approach: will have the whole days data so in terms of the product that comes out - it may be different to the NRT product and we will need to understand why it is different. If it is then we need to account for this as part of the validation
  • could run it up as a science test. there are dedicated test machines. historical mod09 data is already there. 
  • NRT vs Standard is done regularly as part of LANCE

HAND mask  Is there an on/off switch could be applied? how could the products be differentiated in the file name? 

• whether HAND mask was applied and what version should be added to the metadata to file. 
• there are standard metadata fields in the HDF

Next meeting 

have we had an answer from Matt / GIBS

Greg will talk to Maki about image generation. She can look at what it will take to generate imagery. 

Consider LOE flood alerts 

Sadashiva, Diane, Greg, David, Dan Fritz

2 APRIL 2020

Greg: production rule fixes seemed okay on 091 - got 250 tiles (expected), but day 092 only got 6 tiles. Looking for problems. 

Dan sent out his converted binary files for the reference water. others still to be done.

Action: David needs to look at the binary file. Greg to send link to Greg.

L2G gridded: 256 tiles in flood tile scheme. of those 250 are being processed consistently. need to see which 6 are missing. maybe in N. hemisphere. 

Ocean tiles are not being produced in Level3

David: got code for water mask - working for  3 day composite to product the flood. Need to compare to 1 day and 2 day and 3 day. 
he has the binary mask files - won't be able to read them in with the HDF reader. maybe PDL or Perl.

AOB:

Validation and the HAND mask. it occurred to Dan that it would be nice to get validation sites reprocessed and compare to make sure we are getting the same date. It would be useful to have a product without the HAND mask applied. Is there an on/off switch could be applied? how could the products be differentiated in the file name? or combined? or don't apply the HAND mask at the get go and then add it in once the validation has been done. 

• Greg would like to get it working correctly first. Then make changes.

Dan will hopefully have updates to the masks when improvements are made. Can we make sure the metadata for the files keeps track of which masks are applied - at some point. Action Greg / David to add this in.

NRT Flood Tag up: 25 March 2020

PGE 11: All files except for Aqua missing when doing production rules. Running now for a week (SR)

PGE 152 and 155: process granules and turn them in to tiles – seem ok

PGE 159: production rules were failing for flood PGE: update files were not being stored correctly. Output files are being produced but not staged to the NRT server. Greg working with Neal on this.  Then will be able to see if the issues Dan saw are resolved. On Greg’s local system the counts show up correctly.

• We agreed no data value from 0 to -99. Still remains to be done. But this doesn’t affect the flood detection itself.

 Action: for Greg to let Dan know when they are on NRT4

Gridding the masks

Dan decided he would do this. Dan has reference water will be done today. The others will be easier – should be this week.
David will then need to update the code to include these masks. David has been working to get the flood code ready to read the state of water map. Once the additional files are in he will add these.
Question: Can they sit on the system and be referenced? Greg thinks the masks will be an array. Since the grids will line up then just need to read in the mask array and overlay it on the composite data or whatever it is masking out. As for where the files are stored – Greg thinks the PGE wrapper code will take care of that. Will need to come up with a number for the different mask files. Greg will need to look at the notes. Either production rules or PGE wrapper. Read them in like how we deal with other masks.

L2G Data
Dan looked at L2G data: When looked at 2 dates (a week ago), the global grid – a lot more tiles being generated that we need (e.g. in the middle of the ocean) probably because they have a spec of land in them. We don’t need them. It is not a problem to keep them.

Level 3 data: there are no ancillary files for these additional tiles. Greg noticed this and is filtering them out and in L3 they will only producing files for the tiles that Dan currently gets. 2 tiles at the tip of South America weren’t generated for a couple of days. Greg to check the files. It could be because the land tile scheme switches as the poles go from spring to summer. Greg said that for some areas MOD09 has tiles for the north in the summer but not in the winter. Sadashiva also mentioned no tiles are produced if there is no data.

 Action: Greg to figure why they are missing.

User guides
Sadashiva sent examples to Dan for him to look at.

AOB
Dan mentioned that the current product – includes ocean map: 5-10km off shore. We can drop this. Fritz asked if that complicate would counting flood pixels? Dan said it shouldn’t alarms should be based on flood not on surface water. Greg said he thought ocean is being masked out as not data. Need to check this. Could it be due to the MOD09 product. Does this generate over oceans.

Action: Greg/Dan to check this

Frtiz: Once up and running will it be available for internal testing. Greg said the data are on NRT4.

How close are we to finishing up?

Greg – 2-4 weeks. Depending on how long it takes to get the masks done. It shouldn’t affect production too much. Talking about functional readiness only. 

Next meeting – Thursday 2^nd April at 11 AM

David, Dan, Greg, Sadashiva, Fritz, Diane

18 March

There have been some issues setting up the production system but it looks like PGE 11 has been fixed and installed. It will be a few days before we have a reasonable set of data. Up to now only 3 x tiles of Aqua data (so a lot missing). It also seems to be processing more tiles than we need.

Need to figure out how to update files throughout the day - currently as each new granule is added it updates the existing water composite file but this doesn't show until the end of the day - due to the way the production system keeps track of them. Neal Devine and Greg to figure out how to make this available as soon as it comes in rather than at the end of the day. So still some work to do.

It doesn't impact the work David is working on getting masks integrated in to the code.

Gridding the masks (hand, terrain shadow and land/water)

• David is discussing possible ideas with Greg. looks like GDAL command is best way forward.
  • once converted to MODIS grid, the terrain shadow needs to be composited. working on ideas.
  • could use Dan's help to come up with proper GDAL commands
  • Dan said the output should just be a 10x10 file 
  • For terrain shadow files these are computed once a month for Aqua and Terra? Ideally when applying the terrain shadow mask we should use the appropriate mask for that month.
    • when doing water count need to mask out observations using the terrain shadow mask. this is outlined in the updated document that Dan circulated. Dan said he could include IDL code for doing this if it would be helpful - yes please. Action@Dan to provide IDL code to David/in document.
  • Dan said the terrain shadow works okay but the problem is the terrain shadow is different each day as the MODIS tracks are different each day.
    • currently the process is compositing different orbits in to one 10 x 10 file. The outcome is the solar geometery will be different for some tiles.
    • in this new processing - you could do create a better terrain shadow for each. is it worth the improvement? don't know, Plus we are not funded to do this.
      • the right way would be to use the actual solar geometry. could be done....
  • The terrain shadow is approximate - it works reasonably well. 
  • David to read through the updated document and let Dan know if he has questions.
  • This information is useful for Greg to figure out how to get them in to the production system.
    • need to decide how to store them. various options: use perl to call them or use co-efficient files (like LUT) or put them directly in the database. or combine them in to one file for each month.
  • need geometry info to see how to convert them in to the right grid information - e.g. corner points and EPSG code for projection. Dan could help
• Naming convention - typical MODIS naming convention or flood naming convention? Sadashiva would prefer standard MODAPS tile format e.g HxxVxx
  • there is a translation that can convert to Lat Long so users can look for a tile of interest. Most search pages do this. It is not in LANCE yet but should be coming soon. (It wasn't done in the past as NRT hardware was limited. With  cuperneides there will be more servers running. Not done yet but coming. 
• GEOTiff conversion: LAADS has a way you search for a file you want. Tell it what conversion you want and then it sends an email saying where to get it.  Aim to have it as an output in GeoTIFF as users are used having the data this way.
  •  Take existing HDF files - stage them and make new products as GEOTIFF files. At this point you would only need to composite the final water layers for 1,2,3 

Changes to the flood map document:

• starting at next steps. modify modwater - done. updated tables. changes are update table 13
• clearer - that Terrain shadow needs to be applied early on. MST1
• code how it would be done in python

User guides. Sadashiva sent examples to Dan for him to look at.

March 11

David, Greg, Dan, Sadashiva, Fritz

The preliminary water composite product is now available at https://nrt4.modaps.eosdis.nasa.gov/archive/allData/61/MCDWD_L3_NRT/2020/.There is still some production rule debugging going on, but the outputs can be checked to see if they look reasonable. Note that this is not a full three days so the three day composite data will most likely look identical to the 2 day composite data, etc. We still need to add in the flood detection using the various masks. 

• The masks have been supplied but need to be regridded and algorithms updated.
• Are they running continuously? Greg said they are running but update files are not yet showing on the website - so some teething issues (file synchronization issue); production rules are still being debugged (Greg and Neal). the impact of this is that all processing for day 68 should be done but in production system 283 PGEs are waiting to run - they are waiting for ocean tiles which won't be coming so need to update production rules. 
• it looks like the files that are running are running correctly but not sure. need to confirm
• Dan briefly looked at one products: looks like there maybe an issue with false positives on the 2 day product. Need to look in more detail. ACTION DAN

Question re: naming convention. Currently using h-v format consistent with other MODAPS science products: 10 degree grid

• This could possibly be changed to the flood tile naming if that is more desirable for end users. Existing flood tile naming scheme uses lat/lon of upper left corner, eg 100E020N.
• Dan and users are used to current naming scheme, but may not be necessary to maintain. . Will be helpful to have conversion page or map. 
• One option to leave as is. There presumably will be a geotiff conversion at the end, and if desired, the lat/lon naming scheme could be reintroduced there (or that may just add more confusion and disconnect between the HDF and geotiff products).  

Action @Greg to check what the options are for this.

• if renaming the file would it cause problems on the database ingest? Gets this from the metadata in the file so this is probably okay. need to make sure the tile metadata works. 

GEOTIFF conversion

• are there standard ESDIS-provided tools? no
• an ongoing issue in building 32 HDF 5 files 
• various groups do have tools - plus modaps have their own tools (e.g. land have linux tools in local repository - written in C, build for linux systems. uses 20 year old technology). Unclear if that can be easily updated and made easily available and usable for Windows, Mac, and Linux end-users.
• Need to figure out recommended tool for GeoTIFF
• potential PGE can spit out GeoTIFF. This is on hold pending a decision by the UWG about how to move forward.
• GIBS GeoTIFF will not be a problem as MODAPS can use their existing code to convert as needed by GIBS.

David working on converting masks to grid: by calling GDAL from Perl (Dan suggests gdalwarp command). Need to give it the grid corners for each 10 deg tile and projection ref. Write a script to generate them based on the L2G. Projection presumably fixed so should be okay. export bounding coordinates from existing L2G products files. test result to ensure pixel boundaries overlap precisely with L2G/L3 products. Some software uses pixel centers, some uses pixel UL corner as the reference point – can be hard to know (or trust) so best to just try and test.  Existing L2G/L3 grid is extractable from the metadata in the L2G file, or presumably is available from within MODAPS?

Funding. Not enough for Dan to finish the transition !!!

• need to figure out validation cost. 
• include updating reference water or not
• can proceed on different levels of funding
• phase 1 = doesn't include user guide (for use by user to know how to use the product. can refer to some of the MODIS user guides. LDOPE can help with that)
• phase 2 = new mask, alerts, other improvements
• Action @ sadashiva  to send a couple of user guide examples to Dan
• Road map document is pretty close to an ATBD. Maybe needs updating.
• Dan currently has 11 hours per month to keep the existing system up and running, and any other tasks
• David G. has said it is pretty much the end of the funding. 
• Action @ Fritz to check situation with Bob and send an email to Diane. 
• Action @ Diane to flag this with Karen / Kevin 

Flood meeting 4 March

Dan, Greg, David, Sadashiva and Diane

on Friday David sent Greg an email regarding the compositing code which David finished.

• Greg has looked at it. To release PGE need to take David's code and put wrapper around it. And to add production rules (when to run).
  • Step 1 - done. Tested and works with all the different combinations of inputs that we get
  • step 2 - working on production rules today (what to run and when to run it). Should be ready to run tests later this afternoon
    • Greg will check it runs without errors and finds things it is looking for. Once validated. will release to production to NRT4 system - on files already there so everyone can look at it.
    • Still waiting for water mask file from Dan. That is not part of the algorithm code yet.
      • Can still run without to show compositing steps but won't show flood.
      • When get this global water file from Dan - can add it in.
      • Greg to move forward with the production anyway so he can test production rules by the end of the week.
        
        
• Dan - not working on this as he has not had time 
  • Ref water file, Hand Mask and Terrain shadow mask (date dependent).
  • Algorithm code doesn't look for them so production rules will need to be modified.
•  David to re- grid Hand Mask and Terrain shadow mask and ref water layer. Action @ Dan to provide these to David 
  • Action @ David to send them to Dan and Greg
    to check 
  • to Greg to update the code
• re: ref water file. Dan was hoping to re-do this. 
  •  Need to check with Fritz to see if there is funding for this.  Meanwhile carry on with old one.

• once David has done these.
  • Send to Greg. could be co-efficient or maybe a query.

• not yet gone in to production.
• needs to be tested.

PGE 155 Gridded data is available on NRT4:

• Dan asked about the fill value. this will be changed when PGE 159 is released. 

FEB 26 2020

Fritz

Dan

Greg

Flood Meeting Notes – Wed 26 Feb

Progress update from Greg

• Greg has started work on the water compositing PGE. He downloaded a bunch of the gridded data which looks good - although he didn't check this for geolocation.
• The data are on NRT4 if anyone else wants to look in the folder for C6.1. File names: MODWD_LG_NRT and MYDWD_LG_NRT.
• MODAPS is producing approx 3000 files a day. 

Dan asked if  this includes polar imagery that we don't need?

Greg: Yes. These are all 10x10 land tiles.

Dan brought up question of end-product pixel resolution and tile size.

• The current operational system uses a pixel size of 00219683655536 degrees, equaling to about 244.55 m at the equator, and resulting in output 10x10 tiles of about 4552x4552 pixels, but this varies dependent on the particular input data; pixels from different products for a given tile will not line up exactly with one another. (the product is not on a fixed global grid).
• The current/testing L2G products are output at a pixel size of 0.00208333 (~231.92 m at equator), resulting in 10x10 degree tiles of 4800x4800 dimension (on a fixed global grid).
• GIBS operates at a pixel size of 0.002197265625 (~244.60 m at equator), but in 9x9 degree tiles, resulting in tiles of 4096x4096.

Thus, if the current L2G resolution is maintained, the flood product will need to be resampled for display in GIBS/Worldview. This can be done, but resampling is never optimal, especially for a thematic product like this.

One alternative would be to instead generate the L2G product on the same grid as GIBS. The downside to this is that would require either: (1) the output product also be delivered in 9x9 tiles (this would allow those tiles to all be the same exact size of 4096x4096, and have integer tile degree boundaries); or (2) if the output product is kept at 10x10 degree tiles (more convenient for users to reference, and continues current product tradition), these output tiles would vary in size (most would be 4551x4551 pixels, but every 9^th would be 4552x4552), and would not have integer degree tile boundaries.

Greg said we would need to work through this. Can change PGE 155 to the required tile size. Greg couldn't recall how Sudipta and Sanjeeb came up with their information.

Action@Greg to figure out options for tile size and which would be best from MODAPS → GIBS perspective.

Action@Dan to consider what would be best for end users.

Gridded data are available (PGE 155).. Next step to get the algorithm using the data.

Progress update from David:

Need to be sure the algorithm is implemented properly. David and Greg have been trying to make sure the code matches road map document. Dan provided updated instructions.  David is happy with the new information from Dan and working on the code. He thinks this will be ready for  testing at the end of the week.

Action@ David to let Dan know when the test data are ready

Action @Dan to look at the outputs at the beginning of next week.

Dan

Greg

David

Landis

Actions from last week with updates:

1. Dan took a look at what it would take to create our own water mask. He looked at the existing land/water mask from Mark Carroll and Alfred. He conclude it would be preferable to create a land/water masks from our water product - as this would make it more consistent and reduce errors. 
   Update: Dan is out of funding so couldn't do this. 
   Frtiz and Dan have come up with a budget of work still to be done. This includes
   i) what they have already committed to do (but is not funded) and
   ii) what they would like to do if funding were made available. 
   This budget was sent by email to Bob Brackenridge who should contact NASA HQ
   
   
2. Re-gridding Hand layer and Terrain Shadow mask. 
   Update: Need to re-visit this. Could be something David does once compositing code is complete.
   
   
3. Dan to look at the updated PGE 153 (land water) output (L2 product) available on NRT4 and provide feedback to Sudipta - who is waiting for the green light to put this in production.
   Update: Dan found a few fill values that needed updating. These were corrected and PGE 152 went in to production today.
   PGE 155 (gridded product) also went in to production but some minor errors were found. Sanjeeb is fixing these and it should be ready today.
   
   Action: Dan, David and Greg all need to check the outputs from PGE 152 and 155
   
   
4. Compositing Code - David to continue working on this following a discussion on Table 13 from the Roadmap. 
1. It was suggested David use a config file for the values in case they change.
   Update: this is in progress. David and Greg to meet tomorrow (20 Feb) to go through Dan's roadmap.  They will send their plan (of proposed values) to Dan to check it is okay. Dan will be available if there are any questions.
   
   
5. Dan and Fritz to revisit the values listed in Table 13 of the road map with Bob Brackenridge and end users. It was suggested the 0 value that = insufficient data could be changed to 255 to be more consistent with other MODAPS products.
   Update: Dan agreed the values should be changed to 255. Just need to make sure users are aware. Ideally they will have new and old products for comparisson 
6. Diane to add Flood product to LANCE UWG agenda - need to make sure the following items are discussed: 

1. production of GeoTIFFS
2. reprocessing and archive of the historic data
3. implementation of an alert system
   Update: Discussion of Flood Product is on the UWG agenda

We also discussed Dan's funding - or lack of funding and the need to communicate status to NASA HQ. Diane to follow up with Ed re: his email to me and David.

NEXT MEETING  - Wednesday 29 Feb at 10 AM

11 Feb 2020

Dan Slayback

Greg Ederer

David Landis

Fritz Policelli

Land/water mask

• Dan would prefer to use our own land/water mask as it will be more consistent
• Looking at our product and see how easy it would be to spit out water reference mask - threshold could be adjusted
• Fritz agrees but not funded. Dan is hoping it won't take to much effort....  
• Alternative is to use Mark Carroll's product.  This could cause issues. This is what we are currently doing and it does end in some erroneous flood detections.
• Action for Dan to look at this some more

Re-gridding 

• Dan has not done that yet.

PGE 153

• Code that creates the L2 MODWATER is in production on NRT4. Greg notified everyone where to get this from
• Greg thinks it looks good but he has not validated it.
• Sudipta ran the L2G (PGE 155/156)on one of the outputs from Greg. Greg looked at results - seems good. Waiting for feedback from Dan. Waiting on this before it goes in to production
• ACTION - Dan to take a look and provide feedback to Sudipta. 

Composite Code

• going well. putting together counting layers that keeps track of the number of layers - need to talk with Dan about roadmap information. having trouble figuring it out. There are some intermediate working layers that could be excluded.
• Looking at latest version of Road map (20200206) - table 13.
  • David questioned the values. Dan to consider changing 0 (insufficient data) to 255. This would make it more consistent with the other MODAPS products. 
  • In table Dan refers to "name" an Abbrev - this could be changed to be more consistent with MODAPS. Greg says you have up to 64 characters so it is ok. you can have a name and description in an HDF file. Name is what shows up in a program like HDF look – so that is the important one. David updated some of the names to make it clear (e.g by adding 250m) in HDF look. There is also an attribute button in HDF look. 

ACTION @Dan to consider values in Table 13 of table and discuss with Bob Brackenridge and Alfred for end user perspective.

ACTION@ DAVID to continue as is in Table 13 suggestion from Greg to use config file (.DAT) that sets values. so if you need to adjust values it can be done in the config file and you don't have update the code. 

Fritz said WFP uses these values so they would need to change their code. 

Question from Dan. Will it possible to create GeoTIFF derived files from HDF? 
this would be very useful for end users. 

Greg said that would be the next step - dealing with the imagery part. 

• initial thoughts replicate what we have in FIRMS. FIRMS generates various files 

4th February

Dan Slayback

David Landis

Greg Ederer

Sadashiva 

PGE 152 MODWATER Product

Greg: On Friday was all set to go with ESTD names in dbase, code integrated with new names etc. Ran a test file for Sudipta, Sanjee and Dan. The file is complete in that it has all metadata, layers, correct names BUT cloud layer only has 2 bits - missing one bit. So they can still process as it is. (David reminded him that the cloud layer was missing one of the bits so need to be modified before releasing the PGE).

David updating code - a day or two to get 2 bit. once done this can do the next step which is to propogate the changes in cloud/cloud shadow through to the water composite product.

Sudipta says they want to keep the original L2G name which is fine.

Re: Flood / water mask

- currently using MOD 44W (now 10 years old). Dan to look at what Alfred did based on Mark's product- which would be through to 2015. It would need to be re-gridded

• can't use Marks product as is. it is conservative so work needs to be done. plus it is less straight forward
• could produce your own using existing flood / water product for last 10 years. how long ?
• Quickest to look at what Alfred has done and use that.

Terrain Layer and Hand Shadow

Terrain layer and hand shadow – need to be included. These are static layers that need to be re-gridded to L2G - David can do this. David to send to David.

There are intermediate flood products - Greg needs to verify this that everyone is on the same page.

Other

Greg - had to run tiles for Dan. Special request for a user who needed old tile. Done. Started running first 3 columns of tiles in LAADs. Finishing this/next week

Dan Slayback

Greg Ederer

Sadashiva

Dave Landis

Diane Davies

Greg gave an update:

• ESTD updates have been done and available for PGE developers to obtain as needed.
• They are working on PGE 152 - water detection PGE (MODWATER). Still going back and forth with Dan on fields. It already includes new product names and uses David's most recent code with cloud bit included as new layer. So cloud mask and cloud shadow mask will be included. Once sorted - will be posted for ingest in to NRT4
• Greg will organize an update of PGE 156 L2G - can see how they work together for final testing for changes David has made in L3 products.

PGE 152 should be ready early next week. 

→ Dan to take a look at a sample file of PGE 152 now and when ready.

→ Sadashiva to also take a look at a sample file of PGE 152 so he can update the L2G code (PGE 155).

MODWATER PRODUCT Now includes

1. water
2. water ratio
3. cloud shadow 
4. cloud (clear / cloudy and mixed - need to make sure these are all brought through)

*David to confirm these values.

PGE 155: L2G should be done later next week.

PGE 152: includes the additional data sets from David. L2G will need to change a bit to accommodate those. Greg will be in touch with Sudipta and Sanjeeb to make those changes.

Samples provide for L2G are with Dan - he needs to take a look at the format of the operational product to help  figure out water / land mask

Sadashiva to coordinate with LDOPE to reflect changes in L2 and integrate with L2G.

• one sample file from Greg will be enough to update the L2G
• Greg can get a sample file to Sadashiva by tomorrow. but developer is on vacation so it won't be integrated in to the L2G until next week.

Next for David: needs to propagate changes through to water composite products. then change count. 

Dan - had a question on archiving. Can we archive the product?

Sadashiva - NRT is different to standard so if we want to archive we may need to consider how to do this.  Sadashiva to talk to Ed / Robert.

Greg said implementation of 152 could be run in LAADS as well. if they decide to reprocess this historic data it would be possible.

if improvements in geo-location come with standard that would be good to reprocess it in standard quality as this would make it more useful for long-term trends. Log Out

13 January 2020

David Landis

Greg Ederer

Dan Slayback

Fritz Policelli

Sadashiva Devadiga

David reworked the water code to add in the cloud layer extracted from MOD09 so in same format but additional layer with a 0 or 1 cloud mask. 

it is interpolated to 250m (like the cloud shadow). David has compared it to the original MOD09 image and it looks good.

So Greg to test that before it is approved 

Still need to change water layer value:  to get rid of 10,11 and 12 from the water image. need to look at code as Dave didn't write that code.

Data - did detect water

No data - didn't detect water

Currently swath data. Did some tests with gridded data put in 100 for all data in the swath but not in the grid. This is interim code. Need to make sure handle no data in L2G.  

Sadashiva - do you have sample L2G data?

L2 Water code running on NRT - not yet released but being run. Greg not seen it - should be there.

Regarding what Neale has accomplished. Even for L3 progress, can ask for L2G to be run so you have some samples to work with.

Greg to deliver revised PGE then can run this week on L2 Swath

Make sure L2G gets these.

Production rules for L3 are written by Neal. Not been tested yet.

Sadashiva: are we using a static land water mask? which one?

Dan - now using MOD44W from 10 years ago. 

Using C6 data they have made annual MOD44W for up to 2015 and then some concern after that. Working with mark Carroll to update these. Maybe interested in using the latest land water mask. 

Alfred Hubbard looked at this with the older data. Dan would ideally take last 3 years and combine that to make a mask, or use their own product to generate an updated land water mask.

Need land water mask in approx 3-4 weeks.

Greg will get code water code in to PGE and released on NRT 4 and science test for a week  then Dan can take a look. Steps to get it done.

Once that is done. Connect it with the L2G PGE and then we will have the output from both phases. Then we can take L2G data and use it with David's new flood algorithm and compositing code to make sure it looks good. Not yet tested it from data in the production code.

Will need land water mask to finish flood algorithm and compositing code. 

GREG owes Dan an email about L2G gridding. Greg needs to look at the code for this. Sadashiva says PGE is already there. integrating it to the existing L2G version now is possible - this could be done by next week. Sadashiva to do this. Then it can be updated with the new code next week.

Will need some revisions - there are at least two different ways to do this.  

Question from Fritz: some faulty data from MODIS.... it was a science test issue. when they started testing C61 the code that was running the current flood process was using data from C6 and C61 - the program had a bug and was writing 0 in the file. It was in the NRT system. Not in the LAADS system - never run as SQ so as far as he is concerned no data archived with that false data.

mid Nov to new year.

So the issue is that they produce flood product from NRT so they archived that. If it is off 2 x orbits in a row it is very obvious. So Dan needs to find days and tiles re-run.

Greg could re-run for the full 3 months in LAADS. 

8 January 2020

Diane

Dan

Greg

David

David and Dan have exchanged emails on the logic of the code - following questions from Dan, David will amend the MODWATER code.

• Dan to send David an updated document outlining in more detail how the code should work - specifically focusing extracting from MOD09 cloud and water pixels (counts of the latter are needed for thresholding), and removing unnecessary layers.
• As the MODWATER code is already running in production Greg needs to look at how easily this can be updated. Usually if the code is in production there are a number of steps that need to be gone through to update code. As this code is not yet being used (and there are not ATBDs) the process should be straightforward. 
  
  
• The status on the Level 2G composition needs to be clarified. Diane to contact Sadashiva get an update. 
  • He said "We have the L2G process ready for integration. Late last year we agreed on a production rule for L2G and L3, that required some implementation at MODAPS at system level (production DB). I believe Neal was working on it or has already finished it. I wasn’t able to fully grasp his last email on this and so we asked Gang to have STIG move forward integrating the L2G PGE based on what Neal had in place to make that happen."
• Dan to provide water mask – still pending

4 December 2019

Greg Ederer

Dan Slayback

David Landis

Neal has started to work on production rules. 

• Greg needs to connect with Sadashiva re: tiled L2G product. Who is putting together the PGE for this? They said they are done.
• David / Greg have inputs. 
  
  • @Greg to find out who will write the PGE wrapper
• Second process will feed in to David's code. Can't test it as we don't have the tiled L2G code. He can simulate it but we need to figure out if there are difference.
• David - has code pulls in all of the new MODIS water products in a day and pulls in the composite water product from the day before. And it pulls apart product from yesterday to create composites for next day. If composite exists today, if it contains yesterday and day before. Take composite files for the day before so using 1 + 2 day so making 2 + 3 composite files for today. This includes information on whether clouds are present (but not the number of times - if making todays 2 day product need to know the number of valid pixels. how should this be stored? separate layer like a book keeping layer?) Need a count of cloudy / no data/ bad values. This helps understand where do we not have enough to know. For each day - book keeping layer would sum value valid observations. 3 layers 
  • 0 land
  • 1 water 
  • 11 cloud
  • 12 cloud bright pixel
  • 13 cloud shadow
  • 100 for no data i.e. outside of swath  
• individual water files have a 1 value for water. these get overlaid last. so have an array that pulls out each value for each new product. Flood water overwrites land if there is a flood.
• action @David Landis to write up logic and send it to Dan.
• aim to have book keeping layer in final file.

What is left: still not generating flood product

We have the water composite. But there is a separate process that compares water product with standard reference to get final flood product.

• aim was to update what we have been using (MOD44W). single layer - in time at that point. This is being generated by Mark Carroll? Should / could be updated each year.
• Can't just use the previous year as that won't be available reliably.
• Global layer to be provided by Dan
  • will it be on the MODIS grid?
• Where should comparisson be done? pull it in to the code being worked on now.

Also need TERRAIN SHADOW mask and HAND SHADOW mask. These will need to be pulled in and may change the value from flood to not flood.

• label the shadow? or make it no data? if it is shadow it is not useful data.

NOTES from Sadashiva

Attendees

In person: Robert Wolfe, Sadashiva Devadiga, Gregory Ederer, Dan Slayback, Sudipta Sarkar, Zhousen Wang

By Phone: Diane Davies, David Landis

Summary:

-          The NRT DNB radiance (VNP46A1) and the Flood Map process will include two PGEs – first PGE that will generate the L2G product followed by the second that will generate the L3 tiled product. The end product from the second PGE is delivered to public and is used to make GIBS images displayed on worldview.

-          The first PGE (L2G tiled process) is to be scheduled and run for every input L2 granule (5 minutes for MODIS or 6 minutes for VIIRS) generating one file per tile intersecting the granule. The PGE will generate new file(s) for the tile(s) and not update the existing file generated by an earlier run of the PGE. So there could be multiple versions of the L2G file for a tile at any moment.

-          This will require a new file naming convention for the L2G gridded  product. Proposed plan is to use granule time in addition to the data day and tile id.   

-          Second process will run at interval of granule time for the tiles that have new L2G files since the last run of the PGE generating a new L3 file for the tile. Similar to the L2G process, the L3 process will create a new file for the tile at every run of the PGE. So there will be multiple files for the same tile and so will need to follow the same file naming convention used for the L2G files. Rule could be to use the current granule id field from the L2G file used as input in the L3 process.

Action Items:

-          Neal will have to come up with necessary updates to database to support the processing of the PGE and new filenaming convention.

-          L2G process for MOD_WATER has been delivered. L2G for DNB will be same as the OPS: - integration can proceed only after Neal has finished his updates.

-          L3 process – Flood map team is working on their PGE. The operational version of the L3 DNB PGE (VNP46A1) cannot be used for NRT since this is an incremental assimilation of observations. A new NRT variation of the PGE is needed.

Dan Slayback

David Landis

Greg Ederer

Diane Davies

David - Greg and David made good progress and got through various problems in the code.

About to start tweaking to produce the first 3:2:1 Day composites so they have a library of them to generate the final products.

However, Dan said This would be different to what is being generated. Need to read MODWater every day - this makes things simpler.

Code will be updated to use water product from all three days.

Next steps: 

• David and Greg will get prototype products generated. just at the point where starting to put this together.
• within a week should have prototype products coming out that Dan can look at. 
• Greg asked what Dan uses to define standard water: MOD44W - single static layer. Needs to be updated but should remain a static layer e.g. generated from last 5 years?
  • action @Gregto look at MOD44W.
  • action @Dan to check his notes on what he did.
• Greg to work on PGE production rules with Neal and Sadashiva
  • talking with Neal so have narrowed this down.
  • todays info will simplify this
• Dan's asked how the process would be organized it so granules processed as soon as possible. If time stamping these, is there value in holding on to older version once newer version has been generated? Do we overwrite the old with the new? or save the old as it takes up some more space. Agreed we should overwrite the older file

Greg will be working with Neal on production rules: take as many inputs as you give it and create tile as output. 

Robert talking about every 3rd orbit. others talking about sessions. as soon as granule becomes available. So need to work on production rules or use separate task. RR task - in order to update the fire data. Either way - would work. Probably opt to stick with PGE. Modify code so PGE processes granules as often as they comes in.

Dan Slayback

David Landis

Greg Ederer

Fritz Pollicelli

Diane Davies

Flood Notes - edited by Dan and Greg

Ryan Boller

Karen Michael

Dan Slayback

Fritz Pollicelli

Minnie Wong

Greg Ederer

Otmar Olsina

Matt Cechini

• Dan provided an overview of existing products – these can be found at:  https://floodmap.modaps.eosdis.nasa.gov/
  • presenting 1,2 and 3 day products without it being confusing.
  • Consider adding all 3 as toggle-able options when layer is loaded in Worldview. In the existing application there is also a 14 day product which was to enable users to see flood in recent past without having to go to calendar. This is probably not required in Worldview. could drop 1 day of historical reprocessing.
  • native resolution is ~250m (product is in geographic (lat/lon))
    
  • NRT or just historical data? Hopefully both. TBD - there is interest in back processing the archive as this would be useful for users. Robert Wolfe suggested it be considered for a standard product - this would require a proposal.
    
• Points to consider when visualizing the flood product:
• would need to let users know that it is an experimental product

• Output options:
  • 0- no/missing/saturated data or clouds
  • 1 – no water detected
    
  • 2 - water detected, which overlaps with ref water (eg, normal water)
    
  • 3* – flood
• transparent PNGs with a layer where there is water. Matt asked if they would you be able to bake in different RGB values. Currently there are 4 values.
• There is a plan to update 3 to distinguish between recurring (seasonal) flood and non-seasonal flooding.
• If it was a GeoTIFF you would be able to see the values
• Is there anything to visualize as a vector?
  • Dan doesn't think it adds much to the vector, and actually reduces information provided (4-5 raster values are difficult to efficiently vectorize into a user-friendly package). But some users may want SHP files? Still talking about it with Bob Brackenridge.
  • Ryan mentioned a planned Worldview / GIBS feature that will enable users to toggle transparent values but this is currently scheduled for mid (?) 2020
    

• Status of product: still in progress. Code being written for second PGE – in terms of getting imagery to GIBS, it will follow standard procedure for other MODAPS products
  • Timeline: few months out but could get sample imagery prior to that
    
    
• Fritz asked how would users access the flood imagery - from a webpage? straight to WV?
  • possibly from hazards and disasters page?
  • There could still be a flood webpage
  • In the Worldview Flood category it was suggested to put flood product at the top and add other layers as supporting products. It was agreed this could also be useful for other layers (e.g. various options for active fire).This may require updates to the WV code to support.
    
• There was a question on the archive / download: it is an experimental product would it still be stored at MODAPS. Hopefully yes but this and the ongoing QA and support of the product need to be addressed.
• If GIBS / Worldview gets access to the RTC SAR product (from Sentinel 1) via ASF (which may be done on a disaster basis), this could be shown in conjunction with the flood product. 
  • ESA has some restrictions on RTC products so there may be some issues with this.
    
    
• From a GIBS perspective
  • Need to nail down RGB values in imagery (action @Dan Slayback)
  • Get sample geoTIFF and PNG (action @Dan Slayback)
  • Then the color map will need to be sorted.
  • so for now color all values apart from no-water 
• Once this is done and the product is ready, then the formal process can happen
• toggle hidden values - should be ready by summer 2020: enable users to toggle between cloud cover values
• LANCE MODIS will need to finalize output to GIBS. 
• LANCE – MODIS would ship 3 different layers to GIBS (one for each flood product, 1-day, 2-day, 3-day) but GIBS would bundle them in to one
• LANCE-MODIS package all 3 layers in HDF for end users - and then users can separate out imagery 

·       NEXT STEPS

• Dan to send sample of current products as RGBA GEOTiff or palleted PNG (with metadata) to Matt. 
  • Note the current flood products are10 x 10-degree tiles which is not what the final output will be. Matt says that is not a problem for preliminary viewing.
• imager values now are 1- 4 but not colored so needs to send RGB files.
• They need to be RGBA images. Dan needs to add extra value for seasonal flood.

David Landis

Fritz Policelli

Dan Slayback

Greg Ederer

Diane Davies

David Landis

Fritz Policelli

Dan Slayback

Greg Ederer

Diane Davies

David Landis

Dan Slayback

Greg Ederer

Diane Davies

• Single HDF file of cloud shadow and original MOD09 sent to Dan. It looks reasonable. Might be useful to look and see other layers and see if there are issues but that can be done later. Good enough to move to the next step
• Now need to take algorithm for cloud shadow and merge it with the water detection. These components will be the input for the PGE which can be run globally and science tested.
• The next step to combine the cloud shadow code back in to water mask code so it puts it in as separate HDF layer.
  • David will probably run a bunch of images and view them as the test image had a lot of small cloud - so it is hard to see how effective it is. Want examples of clearly defined cloud edges.
• Once the water detection and cloud shadow mask are combined the next step will be flood detection which uses rolling 3 days of data:
  • Dan may need to provide more details on the compositing. 

Action 

• Diane to send David links to WV /RR swath imagery
• Greg to send Dan information on the terrain flag.

David Landis

Dan Slayback

Greg Ederer

Diane Davies

Fritz Pollicelli

• Good progress on the code David wrote to extract the cloud shadow from MO/YD09. The code seems to be working. The next step is to get it to write the output HDF file so it can be checked.  The first PGE is already in production producing the water layer. This second step will add the cloud shadow mask.

Action for David/Greg to send sample HDF files to Dan

• Once Dan approves the output, the next step will be to integrate it in to the PGE. Test it and then put it in to production.
• Rather than have two separate files (water and cloud mask) they will be bundled in to one and be re-gridded to 250m so everything should be consistent. 
  
  
• Should we apply masks / or not? It would be possible to put the water data set with mask in to an output file. 
  • We discussed applying cloud shadow masks to the water output in the this dataset, but agreed it would be better to simply include the mask, and apply it in the second PGE where flood products are created.
• The next step is building the second PGE, to composite the water detections from the first PGE over multiple Terra and Aqua datasets (over 1 to 3 days) to create the final water and flood maps.
  
  • The cloud shadow dataset will include the original MOD09 cloud mask QA so we have that for reference
• There will be 2 PGEs: the first creates the water detection product (applying water detection algorithm to a single Terra or Aqua granule), and the second composites these water products over several observations (1 to 3 days) to create the final water and flood maps.
• Fritz asked about terrain shadow and HAND mask. They will come in at the compositing stage. Greg thinks MOD09 has a terrain shadow flag.

Action: Dan to take a look at the MOD09 terrain shadow to see if it looks suitable -  it could be an improvement on what is being done now. 

• Is the plan to combine the terrain shadow with the cloud shadow? No, they are not combined into a separate layer, but will generally both be applied when generating the final water and flood maps. 
• The HAND layer is meant to deal with the limitations of the terrain shadow layer. it is an elevation mask that shows where flood water is unlikely to accumulate e.g. on the side of a mountain – other examples too. 

Action: Dan to assess to see if the HAND layer still required

David Landis

Dan Slayback

Greg Ederer

Diane Davies

Flood mapping transition to LANCE document provided by Dan has been circulated and agreed by all. See attached

Module A: water detection

• Action: David to send samples of cloud mask comparing MOD09 and examples of dialation at 250m to be sent to Dan for QA
• Request by Dan to keep some examples for documentation
• Greg has estimated completion of Module A as mid-May

Module B

• David asked about the process of removing cloud shadow pixels from the water mask. How is this done? simply removing any detected water if it falls under cloud shadow. In future could look to see if it falls under reference water.
• Are we still planning multi-day composites? yes - it provides a longer view.

C. Final product generation and distribution

This module would simply take the final products generated from the compositing module, and reformat (if necessary) for distribution sites (legacy plus standard LANCE channels), and for GIBS/Worldview ingest.

Options to consider:

• Continued distribution of MWP files via legacy site: https://floodmap.modaps.eosdis.nasa.gov/
  • To do so will require extending (or recoding) the current ArcGIS-based processing that generates the MFM graphic map products per tile (MFM = MODIS Flood Map). These are currently generated by an automated process running in arcpy on a virtual windows machine (using an ArcGIS Desktop map file template).
    • The MWP is a GeoTIFF – snapshot for a tile. It is not very helpful as is. Needs to be color coded for visualization
    • Need to determine how this will be done. Worldview? mapserver?
  • We could continue generating these products on our  OAS2 server, but this will require some modification. — agreed this should be done for an interim period.
  • Alternately, we could update this processing to run on an ArcGIS Server instance hosted elsewhere.
  • Alternately, we could write new code to generate a similar graphic map product independent of ArcGIS.
    • Should we get rid of graphic map products?
    • Diane to ask Tian Yao what formats the users would the data?
    • Disasters portal are currently just display the data but don't display it correctly - use value of 1 (don't think there is water). Should display it as transparent so folks can see underlying map.
    • they have the same products Dan had in terms of making the product useful – allowing user to manipulate dates etc. They just simple maps. not done anything beyond the basics. Dan has mentioned this - suggest they build a custom web application.
    • Suggest we ask GIBS/Worldivew start working with current products. Note there will be an additional pixel value "ephemeral"

David Landis

Dan Slayback

Greg Ederer

Fritz Pollicelli

Diane Davies

• • Cloud shadow mask layer
  • David is continuing to incorporate new code in to the existing code.
    • ran some compiler tests. there are a bunch of errors that need to be fixed.
    • need to talk about existing flood code // variables that haven't been declared properly
      • not sure if it is just sub-routines need to be in specific orders. Greg thinks it might be an environment issue. missing some library.
    • other than that things seem to be doing well.
    • will need to talk to Dan about the logic in the code. David has been simplifying the code as perl can do more than the IDL code.
  • Next steps for cloud shadow mask: testing the code on full MODIS products. within a month or two should have testing complete and ready
    • Greg explained the levels of testing:
      
      • unit testing - what David is doing now. small tests on algorithms to make sure they are performing the right operation
      • integration testing - larger testing at test level
      • production testing: test it on a handful of time selected intervals to test over a range of inputs. this usually involves making a release to production. the production team runs it and keeps output private for evaluating.
  • Once cloud shadow mask is ready, the next step will be Flood Detection: is the water we detect flood or not.
    • This is usually done by comparing the water results with a water mask and identifying differences.
    • The eventual output will be GeoTIFF images.
      • it was agreed that we want to replicate current product which is a geoTIFF with 4 raster values. This could be modified / improved in the future.
      • Fritz asked about SHP files, but it was agreed to not provide these. They currently take 95% of processing time and don't provide all the information layers.
    • This is part of the compositing process.
      
      
  • Follow up question from Dan:
    
    • how long until complete code is in production?
      • Summer time frame
      • Next big piece is compositing code. Dan thinks this should be straight forward in terms of code - more of a data management issue, Greg said this is handled by the production system. Just a matter of writing the perl wrapper code. Hard to estimate. optimistically it is a week or two but nothing has gone smoothly so far! As we go through the test process will be good for David to see how the mechanics of the production system work. Hopefully he will have a better sense of how it works for compositing part.
        
        
  • Follow up from last time: Question re: grid swath
    
    • Concluded that it should be grid data. Using own grid which is not standard MODIS grid.
    • Whatever grid - it will be done before flood code is run so this should't impact David's code.
    • Projection will always be the same
    • Grid - physical size array? currently 10 x 10 code but moving forward need to decide. this will be larger than a 10 x 10
      • there are different ways to do this.
      • Fritz - one goal was to avoid waiting for 10 x10 deg grids. the code should run on granules as they come in.
      • if final out put is a 10 x 10 tile when do you say it is complete? if is another granule is processed will it be an incremental product?
      • 10 x 10 grids are currently at 250m. these could/should be same grid as GIBS uses. Action @Dan Slayback to send these notes to Greg
      • In deciding what grids should be used we should look at what is most efficient in the system?
        • This will be a decision for David and Greg - it is a bit beyond where we are.
        • Need cloud shadow mask first
          • then how do we compare what we have with history to tell if there is a flood there?
          • that is when geolocation comes in to play to be able to determine if there is a change
          • at that point we can adjust things on both sides so comparison is as simple as possible
          • also need to line up with mod44 water mask? - Fritz / could be a different water reference layer.

David Landis

Dan Slayback

Greg Ederer

Gang Ye

Diane Davies

• With the shutdown. David was not allowed on to GSFC and not able to access his computer. No VPN set up
  • he was able to go in yesterday and get some files, so should be able to carry on.
  • Greg said he can work at SSAI Aerospace building. They have connections that don't require VPN set up
  • away next week in Florida but after that David should be able to start testing again.
    
    

PROGRESS since last meeting

• Testing of PDS commands to replace the IDL commands
  • • some of the perl algorithms enables you to do more with less code, for example by dealing with image border issues automatically.– This is allowing David to streamline some of the code.
    • looks like all files that go in to cloud shadow are 1km apart from SR data, which is only being loaded to get the dimension.
    • one reason to process cloud shadow at 250 m resolution is that it would  then be simpler to drop in an improved cloud mask in the future, such as might be developed from the 250m reflectance 250m data (down the road)

Question on whether the code should operate on swath or gridded data:

• • • Current algorithm takes gridded data for SR, cloud, cloud properties – not swath.
      • when writing code David should assume he will get gridded inputs.
      • will be working on gridded swath imagery, so it will be larger than current 10 x 10 box
    • Currently set up so the output for given day is produced on the grid for Terra SR file for that day. Thus Aqua and files for previous day are re-gridded to match the Terra SR grid.
      • For LANCE processing, we could do the same, OR we can use a fixed global grid for all products.  
      • Greg: code allows you to do either, so that option is available.
    • Dan asking if Greg could run some test data to map out test granules at native vs fixed grid
      • action @ Dan to send a tile / AOI and Greg will order the data

Question re: funding and the shutdown.

• • • Greg said the relevant HBS project tasks were under review by GSFC, but given the shutdown this has not occurred. Works is progressing following a letter saying the work was approved through Feb 1st. Not clear if they will extend it again.

Next meeting 20 Feb 2019

David Landis

Dan Slayback

Fritz Pollicelli

Greg Ederer

Gang Ye

Diane Davies

• Almost done converting code - cloud mask. taking flood code using it as a shell and using it to covert idl → perl
• (cloud layer is from MOD35 = mask)
• Working on data structures
  • Talking to greg about how data arrays are handled (rather than the perl format) - the standard would
  • David is lookin up perl data language (PDL) options on line. Functions that read in the HDF data - read it in a PDL language.
  • He is looking at the official PDL website. Not ventured out to look at other sites.
• Hoping for a few months. Depends on how quickly he can get the dilation code working (filling in the holes). At each place you have a cloud get height from cloud top temperature. So need complete arrays and it is patchy. Need to smooth it out to make is spatially complete. Erosion and dilation are Dan's attempt to expand the cloud top temperature. is a simpler way to do this? looking at patching functions in a 2d array - nearest neighbor averaging for non-bad pixels. some handle bad data values and some don't.
• Issue is no-one knows how the functions work and so it is trial and error. Documentation is not good so it is a slow process.
• If python was allowed this would be much quicker however for now the SA will not allow this. Currently the MODAPS machines only use code written in C, Perl or Fortran.
• This maybe an issue for LANCE moving forward if all science code has to be re-written.
• It will be a few months until this is finished.

Meet again in 2 weeks. Need to prepare an update for LANCE UWG.

Dan Slayback

Fritz Pollicelli

David Landis

Greg Ederer

Gang Ye

Diane Davies

• David converting IDL code to Perl. Good headway. some snags in trying to find the proper function for some of the more sophisticated IDL routines. (congrid, morph, dilate, quadrat)
• Working with Greg yesterday and some questions for Dan.
  • may be some work arounds. trying to fill holes in all layers / to project the cloud.
  • several routines in perl that do similar task
• Move to testing specific routines to see if they do similar things – perl equivalent code. Still more code conversion.
• Next steps: to get IDL routines mimicked in Perl. or writing simplified version that can do the hole filling. then get it on to the server to test the code – running it through the interpreters.

Next meeting 24 October 2018

Dan Slayback

Fritz Pollicelli

David Landis

Greg Ederer

Gang Ye

Diane Davies

Dan Slayback

Fritz Pollicelli

Greg Ederer

Diane Davies

Gang Ye

Brief meeting. David Landis has been hired to work on transitioning the code to Pearl.

Dan to drop by and see him in Building 33 G316

Dan and Greg to try and set up weekly meetings with David to touch base on any issues that might arise.

Plan to meet next Wednesday 29th if David is free.

Dan Slayback

Fritz Pollicelli

Ed Masuoka

Diane Davies

• Current Status: Step 1 is complete. The initial swath based water product is running as a PGE and Dan has checked the out put product and is happy with it.

Next steps

• Greg is going to amend the PGE to include a cloud shadow mask. Dan and Greg are discussion whether to use the existing cloud shadow mask (likely) or whether to use a different one.
  • Dan needs to provide Greg with the code for the existing cloud shadow mask that uses MOD35.
    • the current code Dan uses has IDL operators. this will need to be re-written.
  • Diane and Dan to meet with Greg this week to discuss next steps this.
    
    
• Dan to provide an outline for code to composite and tile the initial product. Dan to work with Greg who will write the code.
• Terrain shadow filter (monthly product computed as part of this process) is to be applied to the tiled product
• HAND (Height Above Nearest Drainage) filter, which identifies areas where flood water should not be present, will also be applied to the tiled product.

Dan Slayback

Greg Ederer

Diane Davies

• Greg amended PGE code and has completed step 1. Dan happy with the product
• Discussion on how to implement the cloud mask.
  • looking at options
  • this should be done as part of the PGE running in step 1
  • Diane to contact Eric Vermote to see if there was an improved cloud shadow mask. He said
  • "There is a cloud cloud shadow mask in the MOD09 product (at 1km resolution), at this resolution it is very difficult to evaluate cloud shadow mask and I am not sure about the performance. 
    

    Cloud shadow masks are currently being evaluated for Landsat 8 and Sentinel 2 data (using an operator derived  validation mask) we can probably gain a lot of experience from this and then transition the “best” algorithm to MODIS the best way we can. That will not be right away unfortunately but more like in a year timeframe,

    As far as terrain shadow it is an interesting problem quite feasible if you have a good Digital Terrain Model but could be computationally expensive because my guess is that you have to do it  at a finer spatial resolution than what your actual resolution but that can be studied too once again with Landsat 8/Sentinel 2 data.