NRT Flood Meeting notes

Ask Sadashiva about quote for archiving the historic flood data

Jonathon:

• sent example code to Carole. R
• esponding to her request for clarifications.
• not sure we have  GPUs 
• testing to see how to pipeline should be configured
• Beth has agreed that Alex Saunders (PhD student) will pick up the validation work in April (when Beth is back from maternity leave).
• Need some additional information for Jonathon to finalize his code.
  
  • PGE 559 have tool to go from HDF5 to US5
  • jonathon has not been able to convert the data. 
    • J and Rui to meet.

Dan:

• has been working on the mask files edge tile effects
• ready to hand over to Rui
• run small test to make sure everything is staged correctly ASAP
  • OPS and NRT
  • then Dan can update user guide and switch to ops production with the new code.

Rui has tweaked loader module and PGE 559 tweaks - to correct handling granules of the day + revisions to loader module etc.

• code has been baselined
• will run it for a few days to make sure it runs okay
• waiting for MODIS version of mask file so can update MODIS 159

Ranjay:

• Where to access Sentinel 3 SR data to start looking at it for flood 
  • this is Eric Vermote's SR but he needs to confirm it is okay first. Carole let LDOPE and Jim Ray and Eric know - needs to confirm this is good. 
  • OLCI SR only at this point. no cloud mask.

Asen: 

• set up new table in exsiting database
• SA's are not keen to set up new dbase at this point.

Jonathon:

• plan to make model in to PGE  
• Present next week an overview of what has been done. trained using dynamic world S2 data. plans for evaluation for it. looked at high resoultion bands. looking at incorportating lowere resolution bands
  • using XX fusion to mergespatial resolution of  bands
• with ROSES - they write that this code will be delivered to MODAPS
  • if Fritz has proposed it will be delivered to NRT LANCE then it will be a formality.

22 Feb 2024

Updated new ref water mask with new MOD44W data

• 5 year window so for 2024 it will be from 1 March 2023 and last 5 years.
  • will do the same for the historical record.
• dataset looks good from year to year
• will take 3years out of 5 positive water pixels for the new water mask. 

more complex process will be to be burn in to the HAND layer for each year

• needs to be dilated before burning in
• Dan is updating the code to cope with edge of effect when it crosses the deadline 

PGE 132 to GIBS - action from last week (when update MODIS NRT product name and order of files changes, so need to make sure this still works).

• Maki sent test results - look good after updating layer names that PGE 132 uses.

Trying to get rid of SSTG area / disk 

• new area available under STG / scratch
• Dan has requested David's directories
• deadline for this ASAP / but July 2024 is drop dead date (it is is a CENTOS disk) 

Rui

• started work on PGE 159.
• Perl loader module will be updated for PGE 559
• waiting on Dan for ancillary files
• need to confirm what ESTD to use - Action@Dan to get this to him

Ranjay

• tests of 559 focusing on same areas as previous test run
• seems like there are differences? 
• capturing input files used. looks okay
• spoke to Rui - looks like it is doing what it is supposed to do 
• sent latest code to Rui then will do a science test run
• will do another test run

Asen

• added insufficient data layer to the flood viewer
• all need to investigate the flood viewer more.
• No update on database?  Carol posted a question on it. 
• Sadashiva & Carol to take this up at the Tuesday meeting. 
  • Sudipta 
  • Do we really need a new one?

Schedule Link on wiki page to the schedule (keep in onedrive). Revisit every couple of weeks

Archive can we archive the historic/NRT flood product

for science test PGE runs and sends data to LAADS. 

Sadashiva needs to come up with a cost estimate for archive. Neal or Asad could help confirm the metadata is suitable for distribution as well as going in to the archive.

Ranjay can take a look at the CR S3 for flood. 

MODIS

edge effect spotted by Asen was an edge effect from reference water so this is fixable

• aim for yearly MOD44W water mask.
• Plan to update NRT code using the 2023 mask.
  • Rui will need an update in the loader modular - should be straight forward
    • Dan to provide the mask to Rui either tomorrow (Friday) or Monday.
  • will need to test to make sure it is running the right file
  • first Dan will check the new mask looks okay

then OPS can proceed - will need yearly masks

• end of year code will look at persistent water over last three years to mark it as permanent.

VIIRS

• going back to water mask layer. 
  • for VIIRS will need to do something similar? take MODIS water mask and generate HDF 
  • when update MODIS NRT product name and order of files changes, so need to make sure this still works
    • CAROL TO CHECK WITH MAKI
  • didn't have global tiles for NRT test.
    • Carol needs to see what happened.
    • initial test: flood looks okay 
      • Dan suggested check for total count. looked like 2 & 3 were identical to day 1
      • maybe input file on science test not ingesting?/able to read from loader module
      • need to update science code so it lists all the files used (including mask, yesterday's file etc) as this will clarify any issues.
  • So Rui hold off on this until new code from Ranjay

8 Feb 2024

VIIRS product test complete. Ranjay has looked at test results after setting it up to have a global mosaic.

• looks okay for 1 day but for 2 & 3 for Feb 2nd no 2/3 day flood pixels.
  • date range 31 Jan - 4-5 Feb.
  • first couple of dates might be off (as no previous data) + the second day of the test there was an SNPP outage.
  • once verified it looks good. next step:
    • can it be run using N20/J1
    • J2 SLR are not yet running operationally - only in extended science test. Determining if they need to do cross-calibration for L1B. Eric Vermote is currently doing an analysis. Need to know if this needs sorting as it feeds all products down stream. 
  • Carol to work on setting up on J1 test.
    • is there anything that needs tweaking first??
  • Dan asking if the code could be used to run with MODIS? PGE loader module is not ready for this for NRT (okay for standard).
    • adding MODIS is not hard but Rui is not sure if MODAPS back end is ready as never had this type of combined product.
    • Carol will look at it and get back to the group. 
  • Sadashiva: why combined?
    • can't keep shifting the target. need to prioritize. Stick with MODIS historical product first.
    • Need plan for what needs to be done first

PRIORITY - to run the historical. MODIS PGE is ready why aren't we finished? – Sadashiva

• • • Dan to write up short plan to clarify goals - for discussion. resources are not unlimited and need to be planned / scheduled
    • Dan says waiting to update the permanent water mask when MOD44W is run (Mark Caroll thought this would be ready for 2023) but Sadashiva says Mark is not obligated to deliver this on schedule so we might not want to wait for this. 
  • MODIS code is ready to go historically. the only question is the permanent water mask. currently using the original MOD44W
    • would this need tests?
    • original published in 2009 and based on a couple of years data.
      • aim to use annual product. would need to change the mask for each year.
      • GET A PROCESSING PLAN to Carol for this. depending on resources this can be plugged in.
• Ranjay: these should also be used for VIIRS product? Eventually but not for testing purposes.

NRT PGE 159

• test done.
• OPS code in progress - should be another day or two
• added in additional granules (some additional islands, including 1 row north). Dan needs to evaluate these tiles - wants to check summer months.

David - completed notes on the code - Dan to post here.

Ranjay 559 (viirs code) ready to test in STIG. Rui - has done everything to set this up

Asen: was working on the database side. 

• automated
• making new dbase so not FIRMS

From Carol: We finally are seeing the new PGE159N run as it should, and storing the outputs at the end of the day as it should.  Data is in AS91 in nrt4.  So far, just one day, yesterday, but we will let it run through the weekend.  Note that the data only stays online for 7-days.

File names: need to be finalized

Rui: sent Dan something on cloud positives. Glint reflection within swath causes issues - means water is not detected

Asen:

Asen:

• transitioned to working on Flood from STREAM. Started looking in to flood map viewer. Aim to make it appear with a grid at higher levels and when zoom in the points. Wrapping up that development now.
• archive data to make data available to users:
  • date range and tiles
  • package this in zip files that are not too large.
  • one file per year 
  • most people ask for HDF files
  • A-SIPS pull data : sends data metadata to CMR so data can be pulled.
• attributes
  • Dan to work with David and let Asen know what attributes are needed.
    • acq date time, type etc
    • may be opportunities to compress the database
    • Dan to email David and cc copy Ranjay

Carol: lots of issues running the test has run in to issues mimicking the NRT environment.

• think we have figured this out. hopefully it will run over the next day or two.
  • Greg and Neal were so smart when they developed the loader modules that it makes it hard for Carol etc to implement
  • Issues are because it is specialized. Writing note
  • 1st run NRT first then 2nd run OPS for the same dates
    • aim to make the NRT available for Dan to approve
    • then will run OPS
• Maki - responded re: making imagery for GIBS and GeoTIFF (same)

Ceph should be working but there were some file ingest problems.

Pulling data from the archive

• Diane to contact Sadashiva, Asen Neal about how to get data in to CMR / Earthdata Search

Rui to deliver code he has been working on next week. 

• needs to work on conversion tool first then loader module etc. for 159.

Dan: Probably want to run VIIRS with MODIS and run bigger tests combining one of the VIIRS + Terra

• ability to run mixed MODIS and VIIRS products. needs to be done later.
• PGE to run mixed is not ready
  • set up database in background
  • currently MODIS and VIIRS set up seperately. Not tested this combination. Next step after 159 is sorted. 
  • in terms of science code it should be able to handle MODIS and VIIRS

Ranjay - VIIRS 

• back and forth with Rui on metadata. deliver ready to deliver science code
• no compact layers for VIIRS 

David has a newer version of the 159 code: carol needs to pull this in. Change in the ordering and naming of the flood file.

• pass to Rui. May delay implementing until after the science test has run.

Tian:  continue to work on case study. results between MODIS and VIIRS = 91 % lower than case study before. Maybe some geolocation error in there. Compare water detection image from MODIS and S3. The pixel didn't match well. maybe due to the reprojection? Florida study case 97% similarities. Using MOD09. Plus register the two images together. 

S3 - band 8 and 14

next step to compare CR with MODIS SR for selected pixels to see the range of each instrument. 

• • 18 October 2023

Dan was not able to finish ancillary tiles as server was full but should be ready early next week

DS to Carol.: want to add tiles to production.

• Want to run a test set and confirm they are all worth keeping.
• Not yet tested if David’s code can work wth updated reference water with recurring flood. He had included the code and is running a test with this
  • Still in process
• David wanted about the designation for the NRT PGE for documentation
• Carol confirmed this is 170N for NRT)
• PGE = 170 for OPS
• Need to check whether there are any compressed layers in the file. If there are these are the suboptimal pixels so they would be ignored in NRT.
• David thinks the code assumes there are NONE. If there are code change required.
• Dan thinks lite version should get rid of these. Check w/ Rui when he is back and David to confirm what his code does.
• Ranjay has it set up that if NRT has compact layers, his code might use them.
• Ranjay and Dan assumed there would be no compact layers in 170N.
  • NRT 155 would have some
  • Dan to check data to see
• Changing MODIS file names to match VIIRS file names (getting rid of spaces and dashes).
  • Will this impact GIBS imagery and Tiff files?
  • YES - most likely. Usually look for dataset names.
  • Carol to check with Maki

Ranjay

• Got feedback on metadata component from Rui.
  • Ranjay will send updated outputs so Rui can look at it when back next week
• Sudipta was trying to set up a test on NRT 4 last week. He had some questions: Rui needs to respond to some of these questions so will need to wait until he is back.

Standardizing file names so consistent between MODIS and VIIRS - ongoing. and won't be an issue to implement. Dan to make a final decision

PGE 159 is in but need to wait for testing - discuss with Carol when back from leave next week.

Ranjay: sent sample outputs and science code to Rui. Once he has the time to look at it will work on conversion from SDF5? and finalizing the metadata. 

Dan asked David for documentation file 

• Dan wants to add additional tiles in to production. 
  • before run in OPS need updated set of tile.
  • David suggests there are one or two files up in Greenland (Row 0) would be useful for mapping flooding ice

• Dan wants to add additional tiles in to production. 
  • before run in OPS need updated set of tile.
  • Timeline?
• Tian 
  • S3 CR data 4 bands 468 and 17
  • need 8 and 17 for water detection
  • chose NY state study area in September
  • applied MODIS water detection and compared results with Opera product (from Landsat - high spatial resolution). Results look promising but some false positives - possibly due to terrain effects.
  • No clear day in Florida so had to shift study site.
  • will send slide to Dan.
• Next week out again - so will meet again on Wednesday

Dan - brief overview of interest from the Global Flood Partnership. 

Google has 40 people working on flood (scientist who used to be at GSFC was a skeptical about the product). Nominally it is global. flood model incorporates MET data and others. Some let Google have the data but won't let them publish it. They have a flood model and an inundation model - mapped using topography (not observed, modeled).

The problem is no-one doing a rigorous comparison. A student has a paper coming out ....using diff statistical metrics to compare different flood products. showing what you should be doing. charts on metrics for validating flood?

Rui

• PGE is ready and delivered. For science test need to ask Carol / Sudipta as operators will run test.
• @Dan to email Carol
• This PGE can run NRT and OPS
• once in production Dan needs to update the user guide
• Dan wants to add additional tiles in to production. 
  • before run in OPS need updated set of tile.
  • David suggests there are one or two files up in Greenland (Row 0) would be useful for mapping flooding ice

David

• finishing updates to the documentation. not sure when funding runs out but coming up soon. 
• Is PGE 155, 170 and 159 run on a specific list of tiles? Y = via file telling it
• PGE 152 runs on granules

Ranjay

• Running test data on VIIRS
• Identical to David's output, so working as it is supposed to
• Initially V7 - v10. Switch to V7 and V8 for OPS version
  • Next step: Rui to write modules once he has the science code so it can be run on the system
  • Ranjay to send some sample data
  • Need to standardize file names so they look similar to the MODIS file names
  • Rui has time to work on this

Diane

• Spoke to Karen re: archiving the flood product
• if we go through review process - NASA HQ then decides on DAAC
• ST vs collaborator (this would not be public, ask for data)
• Diane to get NASA HQ sponsor
• Bring this up at the LANCE UWG
• Google interested in pulling the flood product - nothing to stop them doing this.
  • Can we reprocess OPS mode to replace the NRT. Not tied to the collection reprocessing. 
  • it is being archived for evaluation / improvement. 
    • if there is an improvement that needs to be made, it can be done but not inside the regular cycle
  • that is the intent // enables checks

NRT vs OPS

• when calibration change Fire products may not have changed
• need to archive NRT
• historical reprocessing 

Asen - Flood viewer

• brief overview of existing flood viewer
• David is ingesting data in to the database
• highlighted the fact that there is a lot of data in the database in the N latitudes
  • this should be improved when the code updates are made.
  • looks like there is a lot flood but when zoomed in it is not so bad
• suggest aggregated grid approach so when you zoom in you get more detailed
  • aggregates are done for specific days / per ingest 
• Will need to decide what product to use
  • one / two or three day product?

20 September 2023

(Day early as Dan going to Singapore for Global Flood Partnership Meeting)

Tian

• was unable to download S3 Cr data to test against MODIS.
• looking on NRT4 and can see the last 7 days data.

No meeting next week

Question on: https://nasa.maps.arcgis.com/home/item.html?id=150b7375dee749bcb2914b0e32ce2659

David has been running his code but found different numbers of layers between NRT and standard. 

• he has been de-bugging the feeder program
• once de-bugged need fresh run of dates for NRT and standard
  • maybe an issue at high latitudes

Rui -

• has finished the original modaps test for pge 159 ops version
  • all looks good
  • dan suggests we run data with this version and then compare it with the new version when ready. 
  • suggests 2 x days globally
    • Dan needs to ask Sudipta for this
• waiting for combined code for NRT and OPS

Ranjay

• for the VIIRS finished processing science tests
  • sample outputs: S-NPP / N20 and T & A (couple of days globally for different combinations of sensors) 
    • are there specific outputs / tests that need to be looked at. 
    • Dan not yet looked at the data yet.
  • when code is checked. Rui will take it and convert it to version 5

Dan - has been working with the legacy datasets to improve the reference water mask. this is an alternative as we don't yet have the 10 years of data. 

Sadashiva

• not much capacity yet on CEPH
• IGARSS paper due March 2024

Jonathon (from Univ of Arizona)

• working with Fritz and Beth Tellman on the ML approach for VIIRs
• not sure how to go about developing ground truth to match granules
  • if working on standard L2
  • water detection only and then plan to apply cloud and shadow mask 

FLOOD APPLICATION TAG UP 

Sadshiva, Otmar, Dan, Asen, David

• Action for Otmar: to check to on connections as viewer is not currently connected to the DB
• David has been keeping the db updated.
• There was concern about the number of point sin the database – could look at options for restricting data points
• Otmar said he could look at algorithms to visualize the data: these would create static image when the user is zoomed out before switching to database when zoomed in
• Viewer not connected to database but it should work.
• Asen will work on the flood viewer - enough to provide a demo at the upcoming LANCE UWG.

David tested his code on NRT data but not on OPS data

• Dan sent David some OPS data to test the code. 
  • there is data for OPS and NRT for the same days. David to use these to test his code.
• David, Rui and David had a discussion about geotiffs that are produced by NRT code but not for OPS code (do this as a flag so an option to run this for OPS in the future). would need to change code?
  • if you want to generate geotiffs for OPS it would need to be hard coded (according to Rui a switch is not ideal).
  • it is possible to do as a global variable - co-efficient file.
  • code would stay same but have a text file that would enable the update to be done easily. DOCUMENT THIS
• Question about what to do with the geotiff code

CEPH

progress. 

• able to use gitlab and build code. This was enabled yesterday. executables are now able to be built and put to a named space. CM can build PGEs

Ranjay: testing using both MODIS and VIIRS in the flood product. Will let Dan know when ready.

Potential for some science testing next week but still TBD

Database - giving 3-4 million pixels per day. 4.5 billion pixels in a postgres database. three years of data. Hand mask is active in NRT so that will reduce the number of pixels.

• perhaps limit the areas to where we have more reliable coverage and then expand to global.
• how could we limit the database
• Alert?
• set up user interaction to get alerts.
• to make sure dbase doesn't exceed a limit. constrain geographically? reliable limited info. demonstrate prototype. 
  • ingest in to db can be restrictive according to the geographic constraints.
  • as a test could use David's database
  • could let the dbase generate for whole globe. see how much is generated. show distribution across globe to get a feel for where the high volume of detections are geographically. 
  • David Lance not sure where dabase is stored but he can access it via mod-dev c7
    
    
• Ask Otmar and Asen to join next week's call to discuss this further. 

Ranjay unavailable this week: Regarding the VIIRS flood status, I had prepared and provided several test cases (OPS/NRT) on overlapping dates for MODIS and VIIRS for Dan to test before my trip to Canada. However, I haven't had the opportunity to receive his feedback or confirmation regarding any potential additional test runs since my return. I'll make sure to reach out to him on this.

• David has been working on NRT code.  This needs to be compared with OPS data. but can't do any science testing after outage.
  
  • Dan will look to see if he can find Test OPS data to compare.
• Rui has started to take code from David and make it operational
  • he will be able to resume work on this when ceph is restored
  • he had taken ops code and was ready to deploy it vs taking David's updated code an deploying that.
    • Rui back on Monday - might be just as easy to use David's code and have once code for NRT and Std
  • Dan to send him an email to give him a heads up.

not a lot we can do on STIG in terms of the development work. no access to Gitlab. Access is still limited. 

• limiting creation of tags and deploys
  • so even if we put things in gitlab trying to access it and test it on the standard machines is not so easy.
  • normal way of doing things is not feasible
  • can't get code to CM
  • can't do command lines.
• Rui working on codes regardless. just can't do normal checkouts and deliveries
  • Loader module is almost finished.
    • offline test = good
    • online test needs file system (in recovery)
    • once loader is finished will we be able to run data through OPS?
      • need this for the historical data processing to do the updates
• Dan: PGE 159 10 years of archive - once run, Dan will want to do some analysis on the data. 
  • should he do it on modev7 or on his own machine
  • is there a set place where it will be sitting?
  • CD: don't know. it depends how much space is needed. those machines can access anything in LAADS to use the inputs including mask files. Dan will read those thousands of files and generate some summary metrics. The output won't be big - but the dataset itself might be large.
• David: 
  • made the first round of code changes to the flood program to support processing NRT files.
    • sent Dan files that were not right (due to an issue in the PERL script for the loader). That is now fixed.
  • code that Dan saw was processing all NRT files as once as a water layer and then processing them all. now loading them one at a time. load current flood file and then augment it. Cumbersome to test. So far looks good. 
    • currently using modev for testing. 
      • manually replicating what will happen in NRT production in STIG later on
• Ranjay
  • Using metadata list of all input files - done
  • run test outputs SNPP and N20 and MODIS Aqua + VIIRS.
    • Dan looking at outputs: looking at 2 things
      • checking the maths 
      • how it performs compared to MODIS. It is not detecting as much water even if we use just Aqua. This could be due to the resolution (375m vs 250m), so maybe the thresholds on the water detection algorithm needs to be tweaked.
        • compare how MODIS vs VIIRS detects water and change VIIRS to match MODIS
  • Dan asked R - have you tested code as if it is in NRT mode?
    • • wrapper script. will only take what hasn't been processed by looking at the metadata and adding to the current day. 
      • doing this for NRT and OPS
• Dan submitted an abstract to AGU
• Tian - no updates on S3.
  • attended meeting A49 Roses led by Applied Sciences Capacity Building. 
    • 10 projects mention flooding.
    • different to science projects - working with local communities to ask their needs.
    • needs id - local monitoring.
    • many of the projects only last for one year. Guess potential solicitations in the future. 
      • disasters roses call
• CDC meeting: black marble

Rui and Ranjay discussed loader module

• how to know what is the new unprocessed granules
• hybrid approach
  • loader provides all granules
  • track all inputs but process only those that haven't been processed. keep track in metadata and exclude those that have been processed this already
  • VIIRS L3 code has been changed by Rui - for NRT
    • running version of it. so will be able to compare this and the older version

MODIS

• David has coded something and ready to test
  • been having trouble running tests due to outage
  • has been updating ** program with PCF image at a time instead of all of them (what is currently does). 

Carole: OPS only version ingested?

• Rui has been working on this
• hindered by impacts on the system since last Friday
• Ceph disks were having a problem. still recovering. LAADS still not doing forward processing. no science testing. still trying to recover ceph.
  • refrain from using gitlab until all sorted as gitlab is connected to ceph
  • just now starting to stp for L0 data
• Option for flood NRT4 for GIBS

SNPP VIIRS in safe mode. VIIRS it will be over the next few days before it goes in to operations

• already not considered the primary satellite
• need 

Rui working on PGE 159 OPS code. will resume when system back up

Tian - IGARSS

• sentinel work mostly based on S1 & S2 

Sadashiva presented flood at IGARSS. Went well 

Overall it was agreed that the same code should run for both NRT and OPS but the production rules will vary for NRT vs OPS.

Currently the NRT trigger is handled via recipes.

-as soon as new granules come in to NRT then it looks for and gets the SR data and creates a gridded file which is processed by PGE 159.

David asked what happens what happens if there are more than one new granule coming in to the system. Greg says it only processes one at a time.

PGE 159 is set up to use 'update files' as suggested by Neal Devine, as this is how other PGEs handle processing in order to avoid having to go back to the beginning and process all files. The UPDATE file is a L3 gridded file for the day.

So Greg's NRT code takes 3 input files and processes them. inputs are:

1. Update file (L3 gridded file for the day so far)
2. Previous days L3 file 
3. gridded tile from the new granule

OPS code does not have an update file

PGE 159N.pl file = wrapper file. it takes executable that runs the algorithm and wraps it in PERL to run and process the data. (According to Greg, all PGEs have 2 perl scripts a loader and a wrapper (=that stages and runs algorithm code). 

David's code can be set up to process new flood files, hold the outputs as an 'update file' which is stored as a final product at the end of the day.

Ranjay's code is different in that it takes all available inputs regardless and processes them in the same way. This approach is good for multiple inputs (e.g. from SNPP and N20 and potentially for MODIS inputs too).  This approach works fine but in principle will be slower. Rui suggests Ranjay creates some kind of metadata file to keep track of what files have been processed so they are not re-processed. 

Further discussion is needed as to whether Ranjay has to modify his code so it runs more like the MODIS option 2 below OR keep to option 1 but somehow track input files so they are not processed more than once. 

_

MODIS OPS code to NRT: aim to create a single piece of code that would operate on OPS and NRT. 

• Quesiton of how the data is loaded. NRT has multiple input files, one for each swath overpass. OPS has it in daily composite.
• idea would be to change loading code.
• NRT code has no de-compression
• Input file name for NRT code has "nrt" in the name so then move it in to the NRT loading code 
• issue: timing - only 2 weeks left on 

VIIRS

• TESTING of OPS code is good
  • working on implementing the NRT (same code for both - just the input will vary and sometimes the production rules are different but it calls the same executable). so when the NRT is final will submit the same code. 
• Also developing single code for OPS and NRT VIIRS
• SSTG provide clarification on best way forward
  • use all avail water detection inputs and then publish results vs
  • use each swath data as it becomes available as it becomes avail
• if there is a glitch w/ L0 may cause an issue - but don't think it is a problem. another pge produces products on swath products so think it is okay
• could take list of files vs 
• PGE 159 is tile based. everything is in tiles. if we gradually process each granule will get a lot of tiles. 
  • only process newly arrived tile
  • reason to do list of tiles would make it easier for PGE 155 so it doesn't need to look for existing inputs.
  • for VIIRS N20 and SNPP could use same list . need to avoid reprocessing some data. need some way to tell if granule generated tile has been processed 
  • for ops = daily process so don't need daily input.
  • would want to combine both VIIRS 
  • For NRT run it every 10-15 mins as new data arrives. for OPS = daily process
    • Tiles combined in earlier stage
  • 520 output is in tiles
  • as tiles become available. they add to the list of input files and process them to get output.
  • because nrt will run ea 15 mins, some files in the list may repeat and you don't want to re-process old files
  • usually rules say anything new since last time it ran. 
    • current MODIS version uses output file
    • Rui thinks both approaches are fine. possibly Ranjay's method is better. Will meet with Dan next week and go from there. 

Fritz: ML for VIIRS

Kick off meeting. Need to see who from MODAPS will work on this. 

idea is that ML will use same infrastructure as for VIIRS and plug in ML module for defining water. run as a parallel system. seperate look using ML algorithm

• who to keep in the loop
  • Ranjay / Diane / Sadashiva
• expect to have ML model done and evaluated by March 2024. being done by Univ of Arizona (Jonathon)
• suggest an offline meeting. 
  • proposed to have it as a separate offering under LANCE 
  • question from a user perspective                          

LANCE UWG

Flood code:

• Dan mentioned that there is a code fix in David's code for missing data over ocean (not essential but it made the product look ugly).
  • this needs to be run in test by DL and reviewed by Dan
• Is the code ready for STIG? DL is cleaning up and adding documentation and getting rid of code that is no longer needed
• Carole et al ran the VIIRS NRT test data (PGE 170).
  • appeared yesterday. Ranjay will take a look at this.

Ranjay

• combined mask files in to one HDF file . shared with Rui who will make sure it meets the required standard. 
  • if it does then will update the code to use the HDF rather than what is currently in the code.
  • this HDF file is just for VIIRS. not for MODIS
  • most of initial layers look identical to MODIS but final flood looks different. Ranjay is reviewing why this is the case. Ranjay wants to wait and make sure he has the latest version of the product. 
  • Dan was only looking at third day
    • If no product from yesterday - then 1,2,3 will look the same?
    • If no data for day one then this won't lead to a good three day product so this should be commented out.
  • once done, Ranjay can compare the results. when he is satisfied they match he will send it on to Dan.

Rui

• combined mask files ingested. all looks good. waiting for code delivery?

Tian

• with prelim OLCI can compare SR between OLCI and MODIS for specific events. 
• could also compare MODIS, VIIRS and OLCI. 
  • not sure if she should wait?

Annual water mask - from Mark Caroll

Dan suggested Tian could do a confusion matrix for this work. 

Ranjay  - attended CDC collaboration meeting.

• • • May 11

David 

• Updating code PGE 159 for operational. Rui would integrate and get ready for use. Go through Science Test. Review results - if OK it moves to operations. 
• When DL ready - add code to GitLab (Carol can help if needs be as various options are feasible).
• Ideally next week, Dan will have reviewed output from David.
  • Dan will be out for 3 weeks starting 18th
  • DL will need to clean up code. Currently there are screenshots etc.
  • DL will work with Carol. 
• If DS can give the okay early next week. DL run same test data for 22 when in Git Lab to make sure it is okay.

Ranjay 

• with VIIRS need to think about issues
  • if just one instrument there will still be overlaps in the north, so need to decide how to deal with overlaps
    • Dan suggested some ways forward
    • Ranjay will run those tests: 2 variations of one sensor and variation with both sensors.
      • orbital times are close
      • is there value in using two VIIRS as they are so close together.
    • should be ready for when Dan comes back
    • Even while T&A are functioning can get rid of equatorial gaps and then should we get rid of A if we have VIIRS overpass.

Rui is on vacation. he is doing NRT version of 655 and 670

Tian

• updates from OLCI. previously applied GSFC alg to an area over San Francisco and reported many false positive values.
  • so chose another date a few days after to see if results had change. 
  • compare water detections with water mask = similar but WD has more false+values
    • want to ID land cover as guess mountain shadow is impacting.
  • so, plan to chose another area that is flat.
• Dan has been looking at DS (decision tree) water algorithm - developed for Landsat but now tried for MODIS for CONUS. Aqua has a problematic band but it works for Terra. Maybe it would work for VIIRs.
  • other ideas for getting rid of cloud shadow.
  • Chris Soulard (USGS) - has a paper out for US.
  • their use case is not NRT: looking at 5-10 day composites. 
  • Suggest Indus area, or Cambodia, S. India and Bangladesh

When Rui comes back Ranjay will talk to him about how to set up the mask files

Rui

• 655 and 670 OPS code delivered.
  • Dan to ask Carol / Sudipta to run this - specific dates
  • This can be run and Dan should look at results.
• NRT may take a while.
  • initial 659 results. looks fine.
  • will

Ranjay

• code ready to test data. will use data from 655 and 670 OPS code
• option of combining MODIS and VIIRS products
• working on PGE 659 so can combine M&V. VL2 outputs shoud be same as MODIS
  • doing a 250m resolution so may see some blockiness.
  • native resolution is 375m
    • decided to stream VIIRS ONLY and then combined.
    • if Carol can run dates from June last year then can look at blockiness issue vs MODIS to see if it is an issue, and if it is to quantify it.
    • there might be some edge effects.
  • plan to do a science test: MODIS vs VIIRS vs combined.

Dan

Thresholding even for some of the one day products with overlapping swaths, may allow threshold of 2

Dan

• written code looking at number of observations to determine optimal number of observations at most northerly latitudes (rows 1 and 2) - using ref water as truth
  • this optimal threshold should be finished in next week so David and Ranjay can implement it
  • it will be based on the total number of observations
    • Ranjay has already put a threshold variable option in his code
    • David has been looking at water layers in Lite code.
      • some cases there are 13 water layer with at least 50% empty. so can't just use the number of overpasses.

Rui

• making great progress
• working on 670 
• will need NRT 652 - should be identical to standard version with a few modifications 
• 655 and 670 NRT versions will be quite different. 
  • C code read
  • perl script almost done
  • not yet tested
  • still need some revision
• OPS version of 655 and 670 are easier to test. 
  • should be able to test in 2-3 weeks.
  • close to finishing
  • priority plan to finish the standard version first then work on testing NRT (the latter is difficult)
• MODIS versions are complicated
• done initial local testing

Standard version will be a few months out. followed by NRT

Timeline for MODIS. If senior review accepted then we can expect data through 2025.

Fritz

• still working on the planet alerts. just presented at NASA HQ (disasters)
  • mentioned GNSS and S3 
  • no roses call to do disasters type work
  • langley are doing something with planet using ML USDA to support FEMA
  • univ of arizona has a AI/ML model using planet. have talked about putting their model in to Fritz's system
  • need evaluation of various products!
    • which algorithms when etc
  • using legacy system? no
    • could have the flood watch product stop

Dan 

• found issue of double counting when 5 min granule (along track) edges between granules contain the same data.
  • causes some issues - on Aqua MODIS and VIIRS. 
  • code - smart enough that it knows orbit numbers so it can deal with it, but at the equator the orbit numbers are incremented and so the code thinks it is a parallel orbit so it includes both 
  • causing horizontal artifacts if hitting cloud shadow. 
    • couting pixel x2
    • for MODIS: for equatorial tiles don't look at double counts as you would not have overlaps
    • for VIIRS you would have overlaps.
  • known issue - Ranjay was waiting for VIIRS data as for Aqua it will be easy to fix. 
• Sudipta and Carol ran Gitlab version of NRT but it is not showing hand mask (via loader file).
  • this is a switch in the database. 
    • so the fix should be quick. Carol to do this.
  • output (without hand mask) between lite and  heavy looks similar 
• David's code:
  • reason for lite code -
  • test data David had didn't show the issue.
  • asked Sudipta to run data from June 22 – this has the issue
  • DL found that when he tried to run it the Terrain Shadow masks weren't there!
    • just running Greg's script, Carol doesn't get full set of granules
      • resulting list of files is not the same number
      • Carol needs to go in and look at the code.
      • Dan is curious to know if Greg's NRT code if these files aren't found.  Old legacy code - there is not always a terrain shadow file. if shadow didn't exist it would run in the legacy. Greg said his code woudn't do that so Dan created empty hand files.

Rui 

• • • • • Dan asking Rui when he can run PGE 170 for VIIRs
          • Needs to do 655 first
          • L
        • Rui has worked on updating PGE 165 adding coarse resolution to heavy files
        • tar ball files

David ready to start running tests on new data once we have the Terrain Shadows ready. only 6 tiles so it should be quick.

Ranjay - update via email

• Assist Dan in examining the orbit pointer and number of observations layers within a VIIRS L2G lite produce, specifically the surface reflectance VNP09GA. After investigating, we identified an issue where the changing orbit pointer at the equator affects the number of observation layers, similar to MODIS products. We are currently considering implementing logic to disregard compact layers for tiles at the equator. However, we will wait until we receive data from PGE670 before making a final decision.
• I am currently testing a Python script for the MODIS NRT input data. During testing, I discovered a slight mismatch when comparing the output to the actual NRT outputs, particularly in areas where the water mask is applied. To address this issue, I plan to run additional tests on the script to identify and resolve any discrepancies.

Dan heard back from Albert re: hand mask values. ambivalent but said if you offer users more data they rarely say no.

GIBS imagery is generated by separate set of PGEs that require co-efficient files etc. if we changed the hand mask values it would need to update the PGE for GIBS.

Dan working on adding new tiles. Ref water and terrain shadow. 

David and Dan exchanging emails about the OPS code. Bugs seem to be fixed. Close to doing a formal test with the OPS code

Talked about changing the thresholds to a variable esp at the high latitudes. The light version of the code might also eliminate some of this. Running some tests to see impact.

Plan to expand the tiles to include some coastal flooding e.g. Gabon, S. Alaska plus some islands such as Fiji AND the top row Alaska, Siberia. 

This could be valuable for showing how things change over time especially with global warming.

• this requires additional ancillary files to be generated including the HAND files - Dan will do this
  • Timeline - a few weeks
  • CDC interested in this

Coding to percent thresholds. there will be 3 ver of lite product

• hard coded to 123
• @25% threshold
• 50% threshold
  
  
• assume 1 and 2 should be the same.  50% is likely to be a bit different.
• tiles selected on day 51 are cloudy so Dan might need to ask Sudipta to run alternative test data.
• there is one tile over the Carolina's over 4 days.
  • need clear observations at high latitude
  • Dan to look at what David has done and then do another test on some additional data that Dan has from before.

Rui

Running code for L2G heavy and L2G lite. now running 

• Dan to take a look at the data
• when Greg delivered a previous version it introduced a logic change which may have caused an issue. 

Ranjay 

• Following emails between D and D.
  • helped with VIIRS python script.
  • ran tests and compared output. looks consistent
  • confident to do further testing. 
  • Seen NRT outputs from Dan but not had chance to work on them
• Currently R replicating ops version of flood data
  • talked about testing the nrt version too. 
  • L2G lite with all compact layers 
    • changed something with the gridding
    • CD script was processing all sinusoidal tiles not whole earth. Current one in NRT revision - uses the linear lat long tiles specifically. now processing less data and removed some errors.
• While investigating what was going on with David's code, Dan noted another issue re: horizontal streak.Need to ask Sudipta / Sadashiva

Looked at re-coding the HAND mask. Analysis shows most are genuinely HAND pixels so Dan not keen on changing this. He has sent this to Albert Setzer and Bob (key data users).  Will wait to hear from them.

LINK TO SLIDES FROM DAN

Ranjay

• processed same tiles as David is working on.
• compared the results and shared them to Dan and David. look similar - so code seems to be working
• Will do some more testing on additional tiles
  • could also do some comparison with NRT mode (Dan to point Ranjay to these files)

Rui

• no updated for MODIS. waiting for OPS version of PGE 159.
  • NRT has been delivered
• finished VIIRS 655 ran local tests and all fine
  • haven't added coarse resolution code
  • need to ask Sudipta if this is needed. MODIS has this so probably need it for VIIRS.
  • not started PGE lite 670
• Progress on VIIRS?

Carol

• Sudipta kicked of the  loader module (152 and 155) test this morning. waited for Dan to be back
  • tests: repeat of what has been done for NRT but with corrections in loader module for 152 and 155 - two chains run though 159 w/ PGE lite (170) and one where it runs 155 as the input 
  • Sudipta should let Dan know when it is available
  • will use 2 archive sets

Dan tiles in Alaska = Q from CDC

There are tiles missing. Hand mask and Ref water mask need to be made available for this. Dan planning to do to this. 

Ranjay

Dan pointed R to multi day outputs from 170. R is using them to test multi-day logic being worked on.
Ranjay met with David earlier to day to clarify some of the logic in the algorithm esp to do with pixel count and thresholds. This will help Ranjay implement the logic in python.

Question re: operational vs NRT code - what changes in terms of science code? separate code or same code.

CD: Code should be the same as long as it code can accept one granule as they come in vs doing tiled product at end of the day. End of day L2G lite will have a compressed layer whereas NRT will have just one layer. 
If metadata says only one layer then code should be able to process one layer or one granule. 

Rui     working on loader module (152 and 155). Carole is tweaking that. 

Rui is trying to test 159 in MODAPS. having problems ingesting files as it is busy. Perl script to do this but EEPH system is busy - over 1000 files are uploaded every second. Rui asked Jihad (working with Navid) for assistance.

More challenging when it is the NRT data as it has to be manually moved whereas if it is in STIG or MODAPS it can just be staged. There are issues as these are physical files.  Trying to move from Head Nodes (local) to EEPH disks (not as quick to move or to clean up) - due to it being new. (Part of LVFS but not LVFS - it is a infinite disk system which is managed by a database. Not disk volume or cross mounting so whenever you put a file there is a dbase - it enters where that file is, it makes an entry in. need to know the name of the file to retrieve it.

SD: Is this on the NRT system? files copied from Dan's disk. need NRT data in MODAPS - having to move the data manually.

CD: Rui is manually copying NRT to verify his scripts within an instance of MODAPS. Ingesting data so he can do a check out before sending to CM. Done all the time. Rui trying to ingest 3 days of Terra and Aqua so it is not a lot of data. but it is very slow. 

SD: has Rui talked to Jihad to explain what needs to be done? NRT and MODAPS are 2 different systems. Carol: that's why is manually copying it over. Now with the way it is configured it has to be done w/ the EEPH instance. SD: not sure if Jihad can help on this. Rui asked Neal and Jihad - all okay it is just the system is busy and Rui needs to be patient. 

Tian shared preliminary results from OLCI - suggesting to wait for SLSTR pilot for the next phase of that study. No action? 

OLCI - not useful for flood as no data in there. maybe when they go from L0 to L2. If id as land there is no data. Tried L1 radiance and process it in to SR. Prelim results. 

Will check to see if there is L2 land SR 

Fritz mentioned that OLCI has NRT SR products

FP: thinks they have R and NIR for SR

Fritz - working on training data for AI component.

David Landis

• code is working. outputs look good. 
  • not done additional threshold tests as having problem in different area
  • some of the high lat files have compressed layers which means there are additional measurements. thought point of 17o data was that it is pickign the best pixel so there wouldn't be compressed layers. Only true for NRT. For operational should be getting one tile per sensor per T/A per day. so compressed layers when there are additional swaths. 
• Rui 
  • L2G late always just use top layer - as best has been selected. Don't touch the others.
  • For the operational code we want one observation per swath so if there is swath overlap do want other swath's data but not the in-swath observations. Is that doable?
  • 159 Code: 
    • NRT is operating on all mapped granules
    • OPS has one input from Terra and one from Aqua
  • in the heavy got all (incl fractional pixels) but just want best observations for that pixel. But if interesecting another swath want the best pixel for that swath too. Dan thought we would have one observation per swath 
  • One observation per xx for PGE 170
  • NRT and OPS products will be v different as no of inputs will be different
    • outputs would be different too as NRT on all swaths and OPS on just one swath
    • is it feasible to change PGE 170 so output file still has compact layers but only propogating single best observation per swath? Difference came from PGE 155.
      • in ops stage multiple swaths in but output one 
        • aren't they kept in compact layers? yes
          • PGE 170 could drop poor observations if want to as top layer is best.
          • depending on location some tiles won't have additional observations (only when you have swath overlaps).
          • science code - should be able to handle compact layers if they are there or not.
          • PGE 159 needs best observation
        • on NRT multiple swath for same tile
        • on OPS it has been gridded in to the tile 
        • 159 distinguish swaths in compact layer. this has been an issue
        • For ops - one day = one tile not multiple observations. this is what the legacy product does. thought was the additional products would be helpful.
        • if we do this we need to change NRT to be more similar!! would need to select the best and ignore the others. Or change the OPS so it pixel
        • 170 too late. as gridding done in 155. multiple swath has been gridded. no swath observation preserved. 
        • So need to modify 155 
          • stage one swath or multiple tiles
          • ops designed for one tile
  • 170 output puts best observation in top layer but Dan thought it was populating best values from swath layers in the compact layers. 
    • not. re-arranging 155 and putting best at top
    • 155 supposed to put best in top layer
    • 170 - 1. takes one observation per layer and then put best in the first one. in remaining swath if any observations they will be put one per orbit. compact layer is only 
      • so selecting best single observation per swath?
      • Rui thinks it is not doing that 
        • do you follow observation pointer so can see which orbit it is coming from. Should only see one observation per orbit.
          • along with observation try to find out orbit number. if you find there are 2 x observations from same orbit PGE 170 is doing something wrong. Dan to check - look at granule pointer.
        • LDOPE has a tool for this - granule pointer
  • NEED TO CHECK TOP LAYER IS GETTING BEST DATA - ACTION FOR RUI
  • LSR BEING USED IS SIMILAR TO ONE USED FOR SNOW PRODUCT
    • see exchange with Kingsong

• Some of the of CA flood in central valley is being captured. 
• Rui
  • updating PGE 152 as found errors in science test.
    • older ver xx ubuntu 20
    • still doing updating so not merged
    • 155 and 170 working on adding coarse resolution to the code
  • VIIRS 655 - reviewed 512. there aer some products in 250m resolution so it is possible to generate high res but because there is no hi res cloud mask it might be a problem.
• how do you plan to get cloud mask at that resolution? MODIS current product uses mod09. 
• Does VIIRS SR have a cloud mask analogous to MODIS?
  • maybe merge resolution at L3 level. 512 should be able to do that - 
  • There is a cloud flag at 750m resolution.
  • In MODIS the cloud mask is resampled to 250m
  • CD - snow uses cloud mask and output is 375m so looks like 
  • Rui- in MODIS 152 there is cloud mask geometry but not cloud mask. Gets created when run the lat long projection 
  • cloud mask was created in 652 and 152 so should be in first part not in 655
  • Rui to look in to VIIRS resampling - so we know how it works. uses num py. no smoothing (like with MODIS)
• There will be separate flood.dats for now but eventually they will converge.
  • coefficients easy to change

David 

• working on 152 and 155 running.
  • Carol helping. run in to the same issues with permissions and file system. Some might be some standard pointers that point to archives of code that have been removed. DL has been upgrading the error messages in the code to better understand how/when/why it fails. 

HAND implemented and in production

• about to send Minnie an update.

Consider how MODIS and VIIRS could be integrated - VIIRS resampled to match VIIRS

Lat/long geolocation can be resampled

• Rui to look at the 350m pointer code, Carol will share, to see if is something that will be useful.
• 375 could be a better way to go as MODIS will eventually go away (could change PGE 650 to VIIRS resolution - take 5-6 weeks)
• PGE 670 us L2G lite. similar to MODIS 270. PGE was designed around 500m or 1km resolution so no support for 375m. 
• Concept of L2G lite is simple. Untag L2G heavy re-tag with L2G lite
  • Legacy PGE code is old and difficult to adapt
• Need to decide how to combine products.

Ranjay

• PGE 659. uses output from PGE 170 (MODIS) as no outputs from 670 so these are 250. Once have inputs for 659 will need to update this and create masks at 375m.
• 659 incorporated terrain shadow mask. ended up creating a separate file... 
• Next step single and multi-day composites.

5 January 2023

Hand mask

• meeting with Greg and Carol after the last Flood Call. 
  • Carol got the new masks ingested and running on NRT4 - this looks good. ready to turn on in production but Dan wants to update the user guide first and send a notification. 
  • @action Dan to add a note on the website and let users know
  • will need a PCR to document change. Production Change Request. Carol will share the form - simple to complete → Sadashiva who will sign off on it and forward it to anyone else that needs to sign off on it.
  • in the user guide - the updates - want a short name for URL (a re-direct e.g. earthdata.nasa.gov/globalflood) in the hands of Kevin Ward.
  • Dan hoping Greg can fix the API issue - currently if you query NRT sites and want an API listing it doesn't work for the current day. It works for previous days. Dan would like to see production lag for each tile. 

Emails with Sudipta and Carol on testing OPS product for the new workflow with PGE 170 with the lite files. 

• think we are ready to move forward with this
• CD: looks like Sudipta is setting up test runs (PGE 152 to 170 in both standard and NRT)
• PGE 159 on outputs. Once they have run the above - David to take that output and run PGE 159 to make sure it runs smoothly.
  • Once have the test complete. CD will let Dan know where to locate the datasets and Dan can share with David.
• On NRT side to run PGE 159 will need adapted PGE 159 to work with the lite code.
  • Rai has not received the updated version of 159.
  • Rai thinks the flood group is updating the 159 code // David has updated the OPS code for 159 to run with the new output from PGE 170 however he has not tested it on PGE170. Greg wrote the NRT version and he is not available. 
  • Could David work on updating the NRT version of PGE 159? Dan to email Greg and David to explore if Greg can do it or not.
• MODIS version has historical baggage
  • In the VIIRS version of things it would be better if it was one piece of code that can run both NRT and standard.

Ranjay - VIIRS

• • finalizing pge 659 (the last PGE). Doc from David was v helpful to understand how to process from 
  • also using latest reference files.
  • need to work on terrain shadow input (changes per date / location). need more L2G (PGE 170) lite files for testing
  • Ranjay's code can work on MODIS outputs. No 670 outputs so using MODIS outputs.
    • Talking about Water detection L2G lite
  • Rui said format for 670 is same as for MODIS 

Rui - nothing to report today. 655 and 670 - don't have anyone working on those PGEs.

• 652 is working in Giblab hub
• Rui says 655 and 670 are hard to work on as code base is old. he will take a look.
• L2G lat long tiles. Do we have L2G lat long projection tiles. Need baseline output. 652 output should be used for 655
  • take route of snow product. CD we have an LSR that is produced in lat long. SD in terms of SR should do similar. input for 655
  • is there a L2G lite in Lat long projection? Yes
  • Rui: PGE 512 that deals with all L2G lite
  • pointer file for L2G
  • have diff executables that use PGE 512.
    • rai: MODIS PGE 512 is based on 12
    • easier to start on PGE 512 to create 655 rather than adapt 155 - which is cleaner??
    • Rai: start from 512 as data and structure is different it is for VIIRS
    • Rai will ask Ginny for details on how 512 is updated.
    • take a look at 512 source code and 518??
    • lat long vs sinusoidal.
    • Rai - resolution in side L2
  • Now it might be easier to start from VIIRS variation with elements of 170 but has VIIRS resolution and sizing etc. 
    • Rai can work on this. 
    • hopefully can confirm this approach will work next week - look at source code first.

David mentioned 

• • Swath.PM in EOS HDF library - is giving an error
  • Permissions have all changed. David to send Carol a summary of what is needed - Carol can help find it.

HAND Mask

• Sadashiva suggests Dan sends a memo to Carol (STIG) with the information provided from Greg as to where the HAND file should be stored and which flag should be turned on and what ESTDs will be impacted
  • Do we need Carole to run tests?
  • DS: Greg has run some tests. Can we run it on NRT4 - check it and then move to NRT3
  • Greg has uploaded the hand mask files 

Emails from Rui re: PGE 170

• • • Rui sent a sample file to David
      • David got program to read it - it has the minimum amount of data 
      • David has also been working in documentation 
      • ready to start testing the files from Rui
      • David needs input from Greg to update 152 and 155 scripts that pull all files that are required to run PGE 159. These scripts grab all files needed for the process control file. These files include HAND mask, input files etc  
        • the code is hitting glitches
        • David to send an email to Diane

Ranjay:

• next step will be a large scale science test for PGE 652 - once all the tests are done.
• Then PGE 655 and 170 - he has some sample data to work on.
  • then will start working on PGE 159 (L3 product). hopes to have a framework 
• pointed Ranjay to PGE 170 for the final PGE 1
  • Rui will work on PGE 655 and 670 (VIIRS variations) once the MODIS variations have been sorted.

OPS Code

Greg is needed to put in the HAND masks –

• we also need to document how to do this and find permanent home for ancillary files e.g land/water amask

UWG

• clear support for historical flood data 
• comment from Robert re: wanting Flood version of FIRMS
• comment from MOM that they would like VIIRS flood product

David has Finished code new format for PGE. Now updating documentation file to explain how it works. 

Kingsong working on PGE 170 - making progress. discussions are ongoing with Rui and Sudipta

170 needs its own PGE number - this will be a new piece that is used to create PGE 155 lite

Rui working on PGE 155 so it can take input from 170 and run the whole day at once. PGE I55 will uses L2G lite input

similar for VIIRS DNB gridded: there are 2 versions of the code one for NRT and one for standard operations 

Ranjay:

Feedback from Rui. Ranjay delivered PGE 652 (had to make some changes to format and metadata). Rui will test. 

Chain: PGE 652 → 655 → 670

Rui is more familiar with VIIRS 

Greg - out 

• Maki: STIG / Rui is working on it with King Son (LDOPE)
  • Carole said they handed it to Kingsong is more familiar with the PGE and is helping write the L2G lite version of PGE 155
  • (Flood is not a priority as they are getting ready for JPSS-2 and wrapping up a lot of VIIRS stuff)
    • and it will probably take another couple of weeks. need to mobilize testing and database testing

• David: working on the flood code. revising it for the new PGE 170 formatted files. 
  • guessing what they are like

Ranjay

xx will be looking at the code and making sure it all checks out

• next step = swath to grid
• need to figure out PGE 170 (PGE 155 lite equivalent) for VIIRS. Who will do that?
  • Rui is helping
• currently starting to develop PGE 659 - the L3 product.

Dan

• Didn't get hand mask to Greg before he was sequestered. Still need to upload it.
• No one else knows how to do this. Need to get Greg to document this.
• David said ... Greg's script for running PGE 155 grabs all files - puts them in a working directory that David's code pulls.
  • David's script runs PGE 152 which generates  155 heavy creating everything 159 = final code
    • Hand files are only referenced in DL's 159 code
      • DL doesn't know where they are stored. 
      • eventually will be added to NRT code too - then all locations need to change.

Hand mask

• Dan was supposed to give Greg information
  • decided to use new mask which will be cleaner in places.
    • should help reduce noise when it is applied
    • working on generating that. Should be ready for Greg early next week.
    • Greg then needs to ingest and test it

Ray is Rui Zhang 

• Actively working on what will be the new PGE 155 lite. He emailed Dan and David about what layers to include. This is his priority now, although he's just started. 
• David working on modifying the OPS code to run with fewer layers.
• David to ask Rui to give a prototype (when available) to check it runs with his code

Greg 

• answered question on accessing flood data from EDSearch
• when you log in to EDSearch they create a session cookie to say you have logged in. however LAADS don't recognize that session cookie, so users have to log in again (using the same user id).
• Issue has been brought up several times with EDSearch and infrastructure group (Sadashiva and Navid). It will take some negotiation to sort this.
  • Byron Peters leads activity assigning tasks that need to be brought in to agenda
  • Has to be taken up by ESDIS. 
    • monthly meeting - agile process to go over this and how to resolve this

Flood update - Dan looking at impact of lack of MODIS 

WV tutorial - Dan to look at this with Minnie

MODIS Terra flood pixels are low. Data should be back. NRT3 no MODIS files (with 80% fewer pixels) 

• standard processing is waiting for new LUT
• not sure why it is on hold
• will ask for NRT to start processing 

LANCE UWG

OPS code

• There have been a number of changes made to the code on meeting with Sudipta.
  • The code now re-constructs swath overpasses and picks best pixel base on swath coverage.
    • This has added 1.5 hours to the process
    • discussing possibly using PGE 155 lite (rather than heavy) as it has done all of this processing
    • if we go with this - David could massively simply code.
      • Still investigating: would need access to lite and get structure
      • MODAPS already runs a similar code for a special product for snow - could do this for flood.
      • PGE 155 would need to be re-written to implement the lite code. Sudipta, Sadashiva and Dan were discussing what features would be needed to do that.  Who would implement the PGE code?
        • Sanjeeb originally wrote 155 = resource issue
        • Sadashiva says PGE 155 will stay the same. We need a new version of PGE 155 lite.
        • Carole and Sudipta think Ray will help with PGE lite.
        • Ray is going through what Sudipta went through and will help develop an approach.
        • DL: thinks PGE 155 will need to be re-written to some extent (to do with the control file).
        • GE: not sure if PGE 155 would be modified or a new file for 155 lite written. 
        • SD: no changes to PGE 155 just to 155 lite.
      • PGE 155 lite used for snow is sinusoidal projection - it would need to be changed to lat long
      • We know what is needed to get the product Dan wants.
      • VIIRS PGE will be independently done. Don't need to wait for MODIS stuff. 
      • Ranjay can independently do his work on 2 VIIRS PGEs 
        • MODIS 152, 155 <insert level 2 G lite before >159 (Name TBD 153? or 157>
        • For VIIRS same sequence of PGEs but are different so he can start work independently. Deliver to Carole. Will need to be a L2G lite. In VIIRS have this (same projection used for DNB).

Status of L2 VIIRS flood map

• Talking to Rui about final outputs - issue with python. Looking in to python bindings but so far dead ends. Looking at creating XX with C program and use python to fill in the data and any metadata updates that need to be done. 
• For final output - how similar would the work flow be to MODIS? Need some sample outputs - and PGE 159 so he can see how to get the outputs.
  • Greg PGE 152 and 159 are in GitLab and subversion. These are synched and are identical. Still working with CM to get production version.
  • Scripts for Tests are available in modcdev7.
    • take inputs from LVFS and produce outputs to simulate what the production system does. 
    • Greg to work with Ranjay to show how these scripts work. 

OPS code 

• meeting with Sudipta tomorrow 
  • David, Sadashiva, Ranjay, Sudipta
• Heard back from Greg re: production time stamp and time stamp in file name. 
  • there is no easy solution for when file was meaningfully updated, but this is generally the last file of the day - Dan to update in user guide

HAND Dan has been looking at HAND masks vs Terrain Shadow - It gets rid of false positives on bottom of shaded ridge. It is minor but won't hurt to have them in. will experiment with making it from Copernicus DEM which is not hydrologically corrected. If it is better might re-use it. 

Ranjay

• last week talked about output format that python wasn't able to do
  • c libraries avail for HDF outputs
  • working on python binding scripts - they enable python to call C function
  • working well but binders only work for dynamic C library functions and the outputs are static C library functions
    • hoping Ray/Rui from Stig will be able help.

OPS Code

• Sadashiva to set up meeting on PGE 155 so we fully understand what is going on in PGE 155 to better deal with overlapping observations
  • Currently getting excess false positives
  • David has come up with a method 
  • Ginny Kalb might be able to talk to this. She has done this for VIIRS but it started from the MODIS package.
    • @Carol to contact Ginny: Dan to put a summary in chat. Carol to cc Ranjay too
  • Ray is the member of stig that will support this. he may be familiar with the code but he is on vacation until next week. 

NRT code

• Greg is talking to Neal looking at taking out the production run at the end of the day but it is not using any additional data.
  • same as previous file but with a different time stamp. something to do with house keeping. during most of the day they live in the "update" table. allows more simple ingest of data that is not the final product. At the end of the day it has to be moved out of the update table. To close it out you have to run a code to move it from one area to another. Alternate solution is to just look in update table but might lose 11.15 production. 
  • This is one of the few codes where updates are made available throughout the day.
    • Alternative to treat them as product files rather than update
    • DNB each granule gets processed and gridded granular file is created then smaller gridded files are staged to create a full days product. They are made available to the public. Diff processing approach between NRT and Standard.
    • When this originally came up, Neal was reluctant to do it the way it is done for the DNB product but if it doesn't meet the needs of the users we might need to re-consider. 
      • End of day all get new time stamp

S3 OLCI

• most papers focus on water quality and not on flood.
• Fritz thinks there might be a missed opportunity to do more on this.
• Tian offered to take a study area and compare an NDWI and compare this with MODIS. Take a look at Pakistan.

OPS code

• Question  about swath to granule process. One problem with OPS is it is getting too many observations.
  • each output is recorded even if it is a fraction of a granule
  • NRT is not looking at full stack. Just most recent, so it has avoided this problem to some extent.
  • Need to fix PGE 155 so it is selecting intelligently on which observations to include.
  • then PGE 159. LG files
• Issue if just look at top layer: latest in array not necessarily best pixel.
  • filter is supposed to pull the optimal outputs but it is not doing what we want it to do. 
• PGE ?? aggregates every grid cell regardless of overpass - takes 
• PGE 159 needs to do this
  • Dan laid out a 2 step process from 155. need to treat them independently and then 
  • all observations have to be sorted to pick the best

VIIRS 

• Looking in to the C library functions and tests on how it works.
• Last week discussed using these with python to produce HDF EOS-5
  • this week worked on night time lights impact of Hurricane Fiona

S3 : continue to look at documents and plan to process the data.

Dan: Greg sent permutations with DEM and HAND and terrain shadow mask. Considering removing that. Dan thinks we should maybe get rid of it at least in some areas. Talk of other DEMs - main side of HAND mask built by Japanese guy created hydro DEM - but it is based on SRTM.  Dan would prefer not to change it again. HydroTX is a 12m resolution (based on TandemX same as Copernicus). Contact at Alaska SAR facility. 

OPS code - 

• NRT and legacy products have simple thresholds to determine if there are enough observations to say it is water. 
  • towards the poles there are more overlaps so more chances for cloud shadow to come in to the product
  • try a percentage approach. if there is 50% water observations - then it counts. 
    • results mixed. so still looking at what is happening and why.
  • Dan has been emailing Greg with questions
    • Greg hasn't had chance to look at the code (due to other LAADS work)
      • needs to look at production rules to answer Dan's questions
    • Also finishing up 3-day tests for Dan and code for on particular tile.
  • David: re invalid pixels showing up is to do with 155 and 152 processing as once decompressed there are 16 layers.
    • so the issue is in the data before it gets to David's code? in combined A&T files there are 16 water layers.
    • Dan getting different numbers from David
      • there is a discrepency between what Dan sees and the 16 layers that David is receiving
      • David is looking at the code
    • So many valid observations that even with >50% threshold it was not returning flood water observations.
    • Dan requesting individual files for granules intersecting one place/date to test
  • In the north where mapped pixels are narrow. how do you decide which pixel goes in to that mapped pixel swath. How do you determine which ones go in? If it intersects 4 observation pixels.
  • Greg to take a look at the code.
  • Sadashiva said to Dan:  you can decide to take one observation per orbit
    
    Action for Greg to set up a meeting with Sanjeeb, David, Sadashiva and Dan - to determine if it is the L2G lite code being used. If it is you will get 16 observations at the pole. If we are seeing more than 16 observations there is something wrong with the code.  If not L2G lite - need to decide how to do the composite. 

Update from Ranjay 

- Working on VIIRS version of the 152 PGE wrapper scripts to generate corresponding PCF files

- I  had some issue understanding on how to access the database table to query the viirs product attributes to identify the corresponding VNP09 products based on the given date and tile# (bounding box). Greg helped me to clarify some of the issues and I tested it and now I was able to query the database properly to access these attributes.

- Next step is to implement them in my script to generate runtime pcf files, which will eventually be used as an input for the PGE 152 script to generate the water detection outputs.

- For the metadata – in my current VIIRS version, I had copy the entire metadata from the vnp09, but after consulting with Greg, as some of the metadata comes from the PGE itself through .pcf files, and that info should be part of the metadata included in the hdf file, I will also work on generating mcf templates for VIIRS - similar to MODIS to generate the metadata.

- Once all these parts are completed - next step would be to put them all together and finalize it.

Can Ranjay work with Carole directly? so it is aligned with how Land SIPS does things  Greg agrees this makes sense. 

New Dev server - 

• David waiting for permissions on new server
• but modev c7 seems to be working okay now - partly because Greg has moved his tests off it.

Dan and David have been de-bugging the code

• When Dan looked at the recent run there were some issues.
  • found issues in 3 day with the reference water issue.
    • fix that
  • are there differences in 2 day too?
  • suggest testing code 
    • for internal consistency
    • will restrict testing to a particular day and tile in Alaska (16 overpasses) 
    • Then once fixed run the set of 80 tiles.
  • David has other questions on how the layer should be built up 
    • Greg may need to provide input on NRT code
  • ref for 2 day product was showing flood in regions
  • David still needs to fix some things
  • not applying the correct land/water mask
• Once we have it right need to consult with Greg.

New Dev server - 

• David waiting for permissions on new server

Ranjay: Working on VIIRS Code

• VIIRS land water mask: Sadashiva clarified it was based on static input from L1 geoloc files (so dead end unless & not useful for us). Ranjay indicated last week that the values were changing - SD clarified that it would only change on the edge if compositing
• For the PGE made progress in understanding how wrapper works
  • write python to mimic what PERL is doing
  • next steps to modify wrappers to work with VIIRS 
• Questions: how the PCF look up keys/tables are assigned? Are they fixed? Will it be different for VIIRS? LUT key files. Each variable has a numeric key : input files, directories and variable names. 
  • are they pre-defined
  • Maki said key value for input are specified in loader module then it will populate in the PCF file
  • there is a master table somewhere  

    Teng-Kui Lim TLim@gst.com

    • in VIIRS this is usually a Word scheme that describes the standardization.
    • Is it an input file used elsewhere.
    • for developing the script he doesn't need to know right now.

Fritz: working on ML version of VIIRS 

- looked at effect of RS with VIIRS water detection results. Picked high mountain areas to show TS - shows false positive as expected. Experimented with Digital elevation to see if that helps

Ranjay: Working on VIIRS Code

- looked at effect of RS with VIIRS water detection results. Picked high mountain areas to show TS - shows false positive as expected. Experimented with Digital elevation to see if that helps.

Compared with Land/Water layer

• Land / Water Background. It is a layer in VNP09 // but pulls QA from VNP35
  • Gives inland water
  • 750m resolution 
  • it is not a static layer 
• Sudipta confirmed it is from the VNP35 quality flag for Collection 1. For C2 they are trying to derive land water layer in the same way they did for MODIS. C2 trying to improve cloud mask for how it responds to snow & ice.
  • Ranjay has contacted Sudipta
• VNP have their own water detection algorithm that they are running on MODIS  
• Colleagues at Night lights were suggesting this layer is not good. 
  • it would be interesting to see how they derive.

Mark Caroll - looking at yearly land/water product. for MODIS/VIIRS

Ranjay is looking at PGE wrappers - to undertand how it works for PGE 159

• Could he run it for a full global day? not at that level yet. needs to set up production PGEs so he can run it globally.
• Could we consider a different development server for VIIRS testing when we get there. 

David: Made changes to the code based on Dan's feedback. 

First, the NOHAND data products appear problematic. They do not appear to have the correct reference water mask being applied, or something else may be going on. In one example, the 3-day product is showing values 1 (surface water) in areas outside of the reference water mask. So those pixels should be tagged flood (value 3). They are however correctly labelled as flood in the HAND version of the product. So perhaps there were other issues with the NOHAND code?  Sorted

Second, it looks like the 50% thresholding is not working well as is, when the valid counts are low. For example, if the valid count is 1 on a 2 or 3 day product, then because 50% of 1 is presumably being computed as 1, this will propagate any single false-positive observation into the 2 and 3 day products. The purpose of moving to a 50% threshold instead of fixed values was to increase the threshold to count something as water, but in such a case, it is actually decreasing it. We dont want that, so a better solution for the compositing thresholds may be:

1 day: 1 water observation required (so no 50% rule here) -- this may well be what is implemented already. 
2-day:  50% of valid, but no less than 2. So if there are 6 valid observations at a pixel, then the threshold would be 3. In the normal (lower latitude) case, there will be 4 observations, and so 50% = 2 = like the current NRT and legacy products.
3-day: 50% of valid, but no less than 3. So if 6 valid obs, need 3 water detections. If 12, need 6. 

Can that be done? YES and one of the scripts from Greg had an error (David's error) so it wasn't finding the flood data from the day before. So now it seems to be running better. In the process of running 4th day of data. Now has 3-4 directories of data with the 2 and 3 day products. 

Also can you clarify: if the valid count is 0, can water be reported if there is a water count? Or would the threshold still apply, and 50% * 0 = 0 so no water would get tagged there?  If so, we probably want to change that also to again insert a minimum threshold (1, 2, or 3), so that doesnt go to zero. The cloud layers are far from perfect, so its possible we will be detecting water when they say its cloud, and we generally do want to keep those. 

Third, it appears that the valid counts layers may not being correctly computed. In this file:

MCDWD_L3.A2022010.h26v05.061.2022205004917.hdf

the valid counts layers are identical for the 1, 2, and 3 day periods (however, it is slightly different for the 1-day CS). So something is wrong here.

I checked in detail at one of the cloud-shadow sites where the valid value is 1 in the 2-day layer (and in the 1-day and 3-day as well).  In this case, there should be one other valid observation from the previous day, where that product's QA flag shows no issues over this site. So it appears the valid layers for 2 and 3 day products are not getting generated correctly (or saved into the file).  Or, things are not getting generated in the correct order and so only the valid pixels from the current day is available and thus nothing is getting added to it from the previous days?

Finally - can you let me know where the L2G and L2 layers you have generated are, that go into the PGE159? (and give me read-access if I dont have it already). I might want to look at some directly to play around with the compositing. 

Fritz working on ML flood VIIRS

Greg - Delivered data to Dan before he went on leave but it has problems. 

• wondering if you can run the hand version on NRT4?
• CD : if it is a PGE and gets baselined it can be run to go to a test archive and run in parallel with the operational NRT code.
• this is a minimal change.
  • Greg would submit it as a new PGE version and the CM would implement.

New Dev server - 

• David waiting for permissions on new server

Boundary Granules

• seem to be working okay

Flood Alerts

• Fritz would like these to continue running? Bob and Albert requested this.
• These are from the 2-day product.
  • Dan turned off the windows side but the IDL side is still running that populates a text file
  • If grabbing the data product
    
    

Greg - Testing land mask in NRT

• Completed PGE 152 and 155 for 3 days for whole world
  • not started 159 as testing from Greg and David brought everything to a crawl 
  • been asked to find another server to do the testing on.
    • can use alternative but David won't have permissions 
  • still seeing some errors in the scripts - need to get to that.
  • Diane suggested we might use the cloud.
    • might be quicker to use one of the other compute modes

Re: python

• Maki working on installing python libraries
• If new flood VIIRS is running in python will it be in a containerized environment?
  • building using GitLab so it can be containerized or on LVFS
  • they are re-writing MODAPS to run on containers
    • there is a prototype system but it looks like it is still 6 months out 
  • getting new server set up for Greg = one day
  • getting new server set up for David =~two day

David should have a bunch of selected tiles almost ready for Dan.

Boundary Granules

• updates to the code for science PGEs have to go through an approval process.
• Gang let the changes go through but Sudipta was not aware of the exceptions

Greg - Testing of land mask in NRT processing

• Dan looked at data. seems to be an issue
  • some days/files seems to be no Aqua data or processing Aqua data fails
  • previous day has more pixels than
    • if A input missing there are few enough pixels for current day so it might drop
    • looks like data are there in file system but they might be corrupted
  • Hand mask seems to be being applied correctly
    • in terms of implementing in production it looks good to go
    • wanted a couple of days with and without hand mask to give statistics that will in to the user guide.
  • Greg has been running a limited set of tiles to save time
    • will need to run this globally
  • Change to the boundary granules seems to be working

NEXT STEPS

• • • fix bugs
    • 10 days ran in 1.5 - 2 days of data of 80 tiles

David - Testing of the OPS code

• Processing data. Dan added additional days that he would like data for.
  • PGE 155 
  • system is slow.
  • processing 10 days of flood data - one job has been running for 6 days
    • seems to be running at half speed on development machine
    • added 8 days of water data and 5 will be done
  • David's is taking longer.
    • 152 one day separately
    • 10 x 159 PGE days running
    • he is keeping all the water data 

Ranjay - VIIRS

• finished initial version that includes metadata.
• it is in HDF5 (MODIS is HDF4)
  • is this output acceptable for PGE 155 and 159?
    • no need a VIIRS version
• Next step: validation 
  • depending on results may need to do some tweaks
  • will compare it with MODIS Aqua for the same dates

Dan S. 

• Flood product is still behind in Worldview even though the NRT product was there.
  • GIBS are notified they are available.
  • They send back a receipt.
  • NRT3 system is having disc issues.
    • could just be the webserver
    • MODNRT3 has a disk issue

Ranjay - progressing (out much of the week).

Version on NRT and vs on Production data.

the way the pixels are handled is diff. Production can wait until end of day and process all pixels at once. NRT does it throughout the day. The tests

Greg - runs NRT algorithm on SR from production but as NRT algorithm would run it and gives you output at end of day

• changes to production rules were installed but are not yet running. new version of PGE not set to default. Greg/CM needs to set it to default
• test scripts - multiple days now showing. however date calculation was returning wrong answer so restarted test. Results are showing flooding so seems to be working. Need to finish running all test dates - 3 days of test data took 3 days to run. Week to finish up. Should be results at the weekend.
• David had same test days to test (multi-day script may have same issues. looked like bug in cpan installation). David needs to download it again to get the newest version 

David - runs the OPS algorithm with the same SR & geolocation (same inputs) but processes all pixels at the ends of day

• in process of testing data for Dan
  • will make sure he has latest changes from Greg's code
  • DL has small fixes to go to Greg too
  • should have final version of code to give to Greg, whenever Dan says the data looks okay.
    • create new PGE
    • install it in OPS
    • run Science Test
    • All this can be done in STIG / by Greg
  • one issue is that it is still in subversion - they are perl scripts requiring MODIS perl libraries. it is not in gitlab yet.
  • CD / Stig can take it in subversion. not ready to run NRT in gitlab.
    • This is standard PGE in terms of how it runs.
       
    • David plans to run code for next two weeks or until Dan says is okay. 
    • with this done we will be able to compare between OPS and NRT code.
  • HAND mask is being applied properly now.

Need to compare NRT and OPS.

SD Could do it now? Need to keep NRT data - save them somewhere and then run using the OPS data for the same period.

1. Greg to write script to download NRT data and save it. 
2. get David's algorithm to the point where it is producing outputs that aren't buggy and then compare it with the NRT algorithm
3. once satisfied ops algorithm is working properly then can compare OPS with data that we download from NRT production system.

GE - question to CD. is there a place in production for test scripts. Not really but CD will ping Maki so she can start thinking about this. 

For CDC

• they would like VIIRS as they are concerned that MODIS is going away in Aug 2023
• what would a PM only product look like?
• once Ranjay sorts our VIIRS -we need to sort out options

Tian - S3 flood product

Ranjay needs to create a VIIRS PGE 152

• Ranjay is doing this in python - testing and debugging now. 
  • next step is to evaluate the results
  • later on David Landis could take over the maintenance of all the flood map PGE
    • L2G PGE for MODIS 
    • next L2 G pointers
    • then VIIRS L2G PGE
      • SD thinks we have this for DNB (this is reflectance)
    • then PGE 159 = different code. doesn't rely on MODIS or VIIR it is to do with what comes out of 152 so if Ranjay's output product follows the same output it should be similar.
  • CD commented on how PGE 554 works for vnp461ai calls pointer and angle code from that PGE. All done my Ginny Kalb
  • need to look at how to transition PGE 155 to VIIRS
• Greg says Sanjeeb wrote PGE 155 (same code as under 12 different scripts). should be able to do similar thing for VIIRS quite easily. 
• NRT is currently running MODIS PGE 152 in C
  • Carol thinks we will be able to adapt the code to run this
    • land/water mask is currently in python.

Cloud shadow mask

• according to Ranjay the existing VIIRS CSM looks good.
• in MODIS not so good. If MODIS is going away consider using VIIRS CSM

Land water mask

• hoping to have something from the legacy product by the fall.
• if OPS code is working then can re-do the mask 
• Merit (Japan) has a global product for a land / water mask but this would only be an interim. made from landsat data. Still based on data that is 15 years old. 
• is there a more recent dataset?
• David L
• OPS code: going over things with the hand mask to make sure it is working properly
• issues with modevc7 server
• in process of switching OS from Centos to Ubuntu (dev server modevC20)
  • SA working to switch OS
  • David to check in mountainous areas
  • once done run the code for XX test dates testing different pixel selection methods
    • there 2 options (counts vs observed and zenith angle to select one with highest weight) // needs to be run on data in the north.
    • have threshold depend on no of observations. set to 50% now (Dan thinks it should be an improvement on what we have)
    • Greg has a list of dates and will put together a list of combinations we need to test. Greg to send the list to Dan for approval. 
    • Running OPS code still needs scripts. Greg needs to check David's scripts are up to date esp re: day/night granunles.  Friday & Tuesday
      • Greg has been debugging scripts based on Dan's comments. 
    • NRT code doesn't change dynamically
    • STIG - Sudipta for MODIS , Ray for VIIRS
    • once code is pushed over - we will be treated like a ST // updates done on source code in GitLab. note sent over. checks and cmd line test. once all okay changes go to CM (via subversion (centos) or gitlab integration (ubuntu)). There are guidelines. 
    • Flood team have not done any gitlab yet.
      • may need it in 

Greg

- made changes to the production rules re: day granules only. It will need to be approved by CM. In process. asked for it to run on NRT4 only so we can compare with NRT3. 

• fixed more test bug scripts.
  • pge 152 are now correctly handled and moved around
  • will be running pge 159 with nrt code to make sure it is running correctly.

what about increase in flood pixels

Greg - summary for last week discussed the test for the hand mask. it ran. Dan verified it is doing what it should for NRT code

• however bug in script code for PGE 152 and 155 output - inputs for 159. The PGE 152 and 155 was dying after processing files for 2 days, so some days missing input for 159 so when following days processed there was no previous day so they didn't show up in composite. Greg working on fixing the code. might be an issue in the virtual file system.

HAND MASK

• Based on topography (Height Above Nearest Drainage) - requires the DEM is hydrologically corrected (uses one from Japan) mostly based on SRTM
  • issue areas where reservoirs etc change
  • if there are updates to DEM would need to update the hand mask
  • could manually correct some areas but it would be an endless task
  • considering annual HAND rather than seasonal at this point
• working on adding this to the user guide
• Aim to make HAND mask available to users so they can look independently
• Dan has burned in reference water in to the HAND mask. If not in reference layer will be reported as flood.
• Fritz mentioned some lakes in East Africa have got bigger
• Dan will start work on new water mask soon and thinks this will need to be updated yearly so the reference water will be more accurate
• New version of hand mask has been sent to Greg. Run tests on this to make sure it works on MOD-DEV C7 as the masks are not yet on the production machine.
  • once validated - pass it on to Carol
  • Greg to do all the mask stuff (ingest to archive set 1- add to 3 other mask files + these are ENVI files)
  • current hand mask is commented out in XX system
  • for production rules: change D/N flag to avoid using night granules 
  • do this along with someone from Carole's group - WHO
  • could be a period (1-2 months) of PGE freeze once it goes to Carole's group.
  • Currently PGE is not in Gitlab just in subversion - no compiled code. all written in Perl. tested on ubuntu 20 and it runs. need to check it runs with ubuntu version of PERL code.

GIBS imagery 

• shows a strip across the top of N America 
• this needs to be removed - action Greg. Need to turn off day night flag to use only day granules. 

OPS code

• Dan thinks it is not showing HAND and terrain shadow mask
• waiting on Greg to fix the bugs
• STIG can help if it is the processing of the running the algorithm
  • we need to do tests and validate first

Ranjay

• cloud mask from VIIRS looks good compared to GMU and MODIS
• working on getting VIIRS water detection streamlined. similar to PGE 152
• Greg commented that 
  • on the bowtie stuff - PGE 152 only works on swath 
  • PGE 155 does the gridding and correcting for the bowtie distortion
    
    
• Metadata guidance? who should advise? Land QA folks // Sudipta
  • Greg has just taken PGE metadata and copy / modify it

Dan has some MODIS sites that has QA that would be great to send Ranjay.

Sadashiva 

Web interface for flood. Waiting for requirements. Prototype instance on FIRMS 2

• Fritz to draw up requirements at the end of the month
  

Not display above boundary. would need to change PGE to remove boundary granules. should be fairly straight forward.

Alerts 

• issues in FIRMS where users move on from their email address. Get in trouble with the NASA enterprise IT as they get bounces piling up.
  • need to deal with this for FIRMS and STREAM and flood in the future.  

May 26th 2022

Carol, Ranjay, Greg and Dan

Query from Fritz about VIIRS Flood product

Plan – spectral re-mapping.

• Cloud mask may not be as good as MODIS
  • Ranjay will look at this
• Fritz mentioned there would need to be an adjustment of the threshold
  • SD agrees:
  • There is a cross calibration that will need to be applied to make it compatible with MODIS. This is being looked at by the science team
  • Once we have this applied we need to compare and then see if we need an adjustment to the threshold.
• Re: SENTINEL 3
  • There are some ESA products that we could distribute
  • Could we do anything better? If we could then we would need mandate and funds
  • Ideally we should look at the flood products coming out of ESA and compare them with the NASA products
  • Eric says we can do better than ESA products, but NASA would have to fund this.

Dan

• has not been able to work on the HAND mask.
• needs to adjust the threshold after looking at more data (out next week so 2 weeks)
• The HAND mask ran without breaking anything
  • Greg has generated data with and without HAND mask
    • Dan to take a look at these data
  • Bugs in code seem to be fixed
  • Greg started running the time ranges that Dan gave him
    • Not sure if it finished as all servers were re-booted
    • Greg to let Dan know where the data are
    • Ran the data with David’s algorithm
    • Needs to be run with NRT too to be able to create a direct comparison

Ranjay

• Followed up with Tian re: GMU
  • Looked at NOAA product and how the cloud mask worked
  • Some issues downloading the data. Tian has reached out to Sanmei Li
  • Looked at NRT flood map images
    • Looked better than VIIRS Quality flag (they were overestimating cloud shadow). They use geometric based method to remove cloud shadow.
    • NEXT STEP to try and replicate the geometric approach and see how it compares with MODIS?
    • Comment from SD that the cloud shadow was hard to see in the slides. VIIRS SR algorithm was written by a contractor (NG) so not the same as MODIS & didn’t keep up with MODIS changes. So VIIRS algorithm doesn’t reflect the latest MODIS version so maybe improvements that could be done. Need to understand what is missing and what needs to be plugged in. Water absorption band is not there in MODIS.
      Work together to see compare LAND SIPS and UW-M cloud products. (Land vs Atmosphere SIPS).

David

• Code PGE 159 running 3 jobs and extra days of water data. More than Dan requested. When done they will be in 3-day groups so 3^rd day will have the third day of flood product.
  • NEXT STEP: David to run NRT and OPS data in test (without the hand mask) to make sure the performance is comparable between the two.
    • We were getting worse performance with OPS than NRT
    • Goal was to get both performing as well as each other.
    • Compare NRT with OPS data
    • Scripts can now process inputs with either NRT or OPS algorithm so can run test cases from both.
    • Test results we ran before were running the wrong days and mask files. That is all fixed.
  • Still taking 5 days to run 1 x day
  • Once NRT and OPS is comparable
    • Can test various algorithm variations

Update from Tian on meeting with Sanmei Lei on 4/22/22

• GMU have plans to go back and reprocess to 2012
• Looking at Tian's slides -
  • would like to know what cloud algorithm they use
  • no cloud shadow showing up in daily and 5 day products. only in NRT product
  • paper makes claims that it is superior to the MODIS global legacy system
    • Need a means of evaluating these products

HAND MASK

• doing some more sanity checking.
  • update user guide

OPS CODE

• making progress. close to having things running. question for Greg.

HAND Mask

• running a test now. one day complete.

OPS Code

• David working through it. some changes required due to a diff directory structure.
• Greg's code does more than the code David wrote. Needs to look for files etc so 
• Greg fixed some bugs that Dan pointed out. Need correct dates for input files and correct masks. 
  • can run with or without HAND masks enabled. 
  • some running in test - the days that Dan asked for 
  • one date range running with HAND mask.
  • hopefully more days will be available with multi-day filters soon. 
  • Changes are on David's area apart from hand 

Ranjay

• looked at cloud mask for VIIRS to compare with MODIS 
• Looked at MOD09 and VNP09
  • in VIIRS confident clear and probably clear - clear
    • and same for cloud: confident cloudy and probably cloudy - 
    • used to be percentage 
    • plan to do more testing next week
    • using python to do this work  - still okay.

Fritz

• Overhauling ROSES schedule to work on VIIRS
• Links to VIIRS NRT products 

re: Discrepancies in ops vs legacy and NRT code

• LDOPE can assist check the input files are the same (use pointer files to check) - production QA
  • Looks like there is a bug in the script that calls the data.
  • Ask LDOPE to be involved when tests are running - they can provide a sanity check
  • Greg to contact Sudipta
• Then once that is okay look the science aspect

Dan has been working on HAND mask - close to having this finished.

• • then send tiles to Greg 

Ranjay has been able to access the code and do a spectral re-map to VIIRS

• Used VIIRS data for algorithm and the simply water / no water product good but seems to match closer to Terra than Aqua
• spectral re-mapping of PGE 152
• Cloud flag was not included - (that is done in PGE 159)
• Threshold sensitivity could influence results but Sadashiva suggests leaving that for now as Land ST are looking at cross-calibration of  MODIS and VIIRS. Once that is done it may change results. 
• NEXT STEPS: Suggest a larger test. Run a few more granules to see how it performs then look at how to get a PGE for VIIRS - to be worked with STIG to transition to operational so we run it globally for test purposes. Different spatial areas and different times.
  • SNPP and J1 are not identical in band 2 after that once hand to STIG they will hep with PGE then Ranjay can look at cloud mask
    • so far spectral re-map
    • how to determine cloud mask. check to see if VIIRS has same cloud flag as MODIS and Cloud Shadow flag.
      • need to think about how to handle this in VIIRS //  how mitigating 
  • might be interested to compare Aqua and Terra to VIIRS may find difference in water pixels 
  • LDOPE have statistic tools that can be used

David - testing OPS code in preparation for testing options for PGE159 

• Has the code running on his area and ready to start testing PGE 159 but Greg will need to figure out the input issue first.
• maybe an issue with the object store - Greg to check 

7 April 2022

OPS Code:

Dan to sent additional dates to Greg to enable him to see how the algorithm perform for cloud shadow - that's starting to show on modev7

• one third done. greg out next week. so could could be another week. 
  • check water mask is not shifted

• David will be running the same process (PGE 152, 155, 159)

HAND shadow mask code

• some minor issues. tweaking code. looked more at the impact of HAND vs improved terrain shadow mask.
• once ready this can be slotted in to the code.
• might want to publish the HAND mask as a stand alone / ancillary data set. This could be made available as a GeoTIFF once it is in production.
  • Sadashiva says we would need documentation to accompany it
  •  initial source is from a researcher at Univ of Tokyo
  • it will go in archive sat 1 along with other land/water masks - publicly available but not advertised.

Ranjay has been looking at PGE 152. He has been trying to run this on the MODAPS server - still pending approvals.

• investigating VIIRS layers. once has access to the dbase files and run it. the plan is to generate a similar product for VIIRS code.

@Diane to ask Maki about the capability for using python - cc everyone. is there an example in sub-version or Gitlab / Github

Fritz - will be doing his code in python so he is also interested if this is an option. He is using specific machine learning code in python. Greg thinks there. isa new production (APS) that is being rolled out. It has containers to make this feasible. 

AQUA

Flood pixels have dropped by a quarter with missing Aqua (PM). Space craft electronics have an issue (not the instruments). trying to bring it back on the 14th April. It is complex as instruments use electronics from A&B. A side went down so trying to use electronics from B side. Might try to acquire data from some instruments - not necessarily for science.

As crossing times starts to go beyond 30 mins there will be coverage issues.

Action items from today:

@Dan to send additional dates to Greg /Dan to enable him to see how the algorithm perform for cloud shadow

@Dan revisit/regenerate HAND shadow mask / code.

PGE 159 already has the hand mask as part of the code but it doesn’t do anything because the mask is not there. It is in the PGE production rules. Right now the code says if it there use.
Estimated time:  a couple of weeks. Dan has already downloaded global DEM to make this happen.

@Ranjay – to look at PGE 152 and decide how to do this with VIIRS code.

He can use python

@David – to start test PGE 159 options.

                Copy over the files already created from Greg’s area OR copy and modify scripts to create more files in his area.

Is there documentation on how this is allowed?  Maki has figured this out. Stig and LDOPE have figured this out. Now running in production. In future it will be more flexible

Dan - Existing algorithm will still report small flood in terrain shadow areas.

Need to develop algorithm to exclude mountainous areas. Need to develop a mask – based on DEM

OPS Coder

• 8/10 days of input files
  • output of PGE 152 and 155 files available .
  • day 2 and 9 are not running completely - looks like there are some corrupt files in the system. 
  • These global input files are on moddevc7 and available to run PGE 150
    • Day 290 - 299 of 2020 (October)
    • not yet run PGE 159 on thee files
    • David has 2-3 diff versions of the code to try out
  • Dan requests a few at the end of the year for max terrain shadow - to send dates to Greg
  • action@David /Greg to run the tests

Ranjay 

• still gaining access to the code.
  • in final steps

Terrain Shadow

• thinks we will need to apply the HAND mask
• also figured why terrain shadow mask didn't work. It was because the imagery was not geocorrected/orthorectified. this was driving a good portion of the misses.
• gross mountain mask - like hand mask

• CDC requirements for flood products. 
  • GSFC disasters have a small amount of money from CDC - they would like RT flood application in their portal https://ephtracking.cdc.gov/DataExplorer/
  • are we happy for them to use the flood product
• Latency for flood product

OPS Code

• when LVFS goes down the processing stops. need notification to get it going
  • this is a problem for all production. but they can restart from where they leave off.
• 2 full days worth of data in 6 days excluding PGE 159
• idea to have 152 and 155 ready then can run PGE 159 as many times as we want
• David has run test data. he things 2/3 are not used by PGE159
  • add to script some generic deletes for 155 to save space but Greg is not worried about space for now
  • Other options for streamlining the processing. 
  • action@David /Greg to run the tests

Ranjay - reviewed documentation for high level summary.

• Ranjay is looking for an ATBD for PGE 152. (ATBD is the science concept that is peer reviewed).
  • There isn't one but there is a user guide  https://earthdata.nasa.gov/files/MCDWD_UserGuide_RevB.pdf
• where to access the latest code. Ranjay will need permissions to access Github / subversion. Not in Gitlab yet
• may need permissions to run tests on the development server
• Action @Greg to set Ranjay up with the various permissions.

Fritz 

• Discussion with Mark Carroll: he will be doing a land/water mask - this will be available in 2-3 months
  • Existing code = MOD44
  • This is a requirement to run the Land process at SIPS
  • Alfred worked a lot on this 5 years ago - conclusion was it was too conservative. might work for annual
  • advantage to using our own product 

Flood mapping viewer?

• • • Asen will be assigned .5 FT to do this
    • Database size is an issue. currently inserting 3-5 million pixels per day.
    • need to figure out a smarter way of handling the data
    • Diane to discuss with Asen and Otmar to start moving forward with this
    • he has a concept for a new cloud shadow mask using a combination of machine learning and geometry

3 March 2022

Greg, Sadashiva, Fritz, David Ranjay

OPS code

• Greg updating his scripts to be able to queue files for PGE 159
• David has updated his code - as per the request from Dan. There are two versions waiting to be tested once Greg's scripts are updated. 

Ranjay -will start <0.5 FT and build up to FT

• start by familiarizing himself with how MODIS code works by getting in touch with Greg and David
  • 152 does SR – main PGE that needs converting
  • after that compositing - should remain 
• calibrate how VIIRS SR is different from MODIS
• look at differences - especially cloud

Suggestion on OLCI

Sadashiva suggested that Fritz could take the SR and run an algorithm using OLCI Sentinel-3A data and possibly fuse it with MODIS / VIIRS 

• green in addition to Red and NIR so use diff water index from 
• take data available and show feasibility

17 Feb 2022

Dan, Fritz, Greg, Sadashiva, David and Diane

Running code

• 2 days of data. took almost a week to run.
  • in production it will be much faster. this system has other people using it
  • same directory as last time. old files have gone in to an old folder. 89 tiles per day.

Terrain Shadow

David - documentation:

• new version last week. waiting on comments from Dan. 

Legacy system:

• April 30 end date. posted on the legacy site. 
• Dan to end out a note to all users?

10 Feb 2022

Fritz, Greg, Sadashiva, David

PGE 159

• Tests stopped running on day 3-4
• Dan found inconsistencies and thinks the reference water masks match correctly to the right tile
• David found a bug in the code. Greg still needs to implement the fix (hopefully today /tomorrow)

Terrain Shadow

• no progress
• plan to run tests looking at TS at different solar geometry and combine those to come up with improved TS 

Update on meeting with Maggi Glasscoe (Guy Schumann gjpschumann@gmail.com, Karb karb@ornl.gov, Chris Chiesa and dbausch@pdc.org) 

• Met with Ranjay - provided overview slides. Plan to catch up with him again towards the end of February.  
• Meeting on 31 with Maggi Glasscoe and team on Flood Alerts - for the Pacific Disaster Centre - using a Model of Models approach 
• Next week Diane  not available
• Terrain shadow - no progress
• PGE 159 - running but with some issues

PGE 159

Greg: Running the code - After an initial blip it is running, 47 tiles processed for the first day.

• It will take approx. 10 days to get 10 days of data (as it is running 3 PGEs one at a time). It is slow as it is on MODDEV C-7.
• Expect better performance when in OPS
• Test result files available on moddev-c7 in /SSTG/gederer/TEST-FLOOD-ALL-PGES-2022-01-10/C61-RESULTS

David: Updated the documentation for PGE159. With Dan for review. May need another iteration.

Dan:

• Terrain shadow – no progress this week.
• Switch to new server: set end of April as the deadline.
• Dan to email everyone and let them know and put an update on the Legacy site.

Frtiz: getting a grant in place with Bob and DFO. Bob will be working on interfacing with new users.

14 January 2022

Dan, Greg, Sadashiva

Greg has been working on running David's code and it is now running. Hope to have some results on Tuesday!

• setting up the environment to run the new code and test the NRT code over multiple days (just like they would in production).
• Some changes David made in the code meant Greg had to tweak the code to run everything correctly.

Terrain Shadow – Dan has a plan to work on this. Time derailed by Tonga Volcano erruption.

NRT flood schedule – priorities set. aim to be more specific once Ranjay is onboard.

Ranjay - Sadashiva has spoken to Ginny and this seems to be a good fit.

• Diane to invite him to the meeting on the 27th Jan and prepare a slide outlining proposed input

Jan 6, 2022

Sadashiva, Greg, Dan, David

• • Sadashiva to talk to Robert and DNB team about onboarding Ranjay
    • at that point we need to have a meeting about how we plan to transition from MODIS to VIIRS.
      • Need to re-map spectral bands, geometry and process interface is different (HDF5 format, metadata different, no toolkit generated process control). Who will be the best person to handle all this. Greg? Someone else?
      • Then need some science input to handle spectral re-mapping, ensure same quality flags etc. Maybe Dan? Frtiz? Guidance can be from Sadashiva.

PGE 159

• Code going well. David still fixing bugs. Running test file and seems to working fine
• Now it is ready to run – need to pick some dates & places.

            What has been changed?

            David needs to update the document – by end of the week.

•  Would be useful to have comparison between last time it was run and this time so we can quantify if it was improved. Greg produced a series of files from 2020. David ran a test for that time and grid. These are in a directory and can be compared directly. Had a test in OPS that was run. Probably want to do that again.

David made some changes to the code:

• He found an error that was adding zeros. This led to a major change in the output – it now looks good.
• He is checking on the rest of the rules.
• Will do a bunch of test runs for 5-6 days. There is a file to check it against using some other code. Hopefully next week.

Imagery

• following an email exchange. Opt for #1.  That way there will be no product for the current day until at least one granule overlaps the tile. Then a product file will be generated. This initial product will be incomplete, as it will, at best, have only a full Terra swath. (At worse, it may just have a tiny corner covered by that first Terra swath, waiting for better coverage on the next orbit). 
• Aim to run it for a week to make sure results are what we expect.
• Currently in GIBS UAT.

(Note from Dan that we may want to re-consider later: The obvious way to know this is for the user to flip the flood layer on/off in comparison to the Aqua base layer (corrected reflectance). If the CR layer is there, then likely the flood product has been updated as well (although...I wonder what the largest time gap might be there, or if possibly the CR might get updated later than the flood sometimes??). 

So...I think this is ok for now to get things moving. The alternative would be to wait until Aqua data has been received to either generate or publish the GIBS image. This would then only provide the user with the full product (Terra + Aqua). The issue then is how the system can "know" when all Aqua swaths that will intersect the tile have actually been received. (and we probably dont actually want to wait for a small corner to come in 90 min later if the main intersecting swath has already been processed...).  And its quite possible in clear areas, the pre-Aqua product will still have valuable flood information, so personally I dont like waiting for Aqua. We just may need to update the user notes in WV about this - to check against CR to know which data have been incorporated and thus if the user should check back in a few hours).

Fritz received his funding today for the ROSES call.

• Current method: look at histogram NIR over Red with fixed threshold that BR came up with. Now with Planet using a dynamic threshold: 2 distributions. Land and Water – where they intersect that is the threshold. Works well. Used on Sentinel 1 data and it has not worked as well. Now looking at machine learning approaches. Refine water pixel using fuzzy logic and other inputs e.g. CS maks, Slope.
• Need to do some more experimenting – machine learning. Many models to pick from. Currently going with support vector C.
• Dan said the key is to find S/W with sufficient diagnostic capabilities that you can figure out what is going on.
• Fritz said at AGU folks are leaning towards ML. DS: Terrain data is key.
• Diane asked if others in MODAPS do this. Greg suggested perhaps BinTan? Did some work on deforestation.

Greg needs to finish reviewing David’s code and comparing results.

• Algorithms are a bit different but Greg needs more time to come up with plan.

Delayed by LAADS issues

Sadashiva wants to take another look: viewing geometry and footprint are dependent parameters and can’t be used as independent parameters. Sadashiva will have another look at the code and provide comments in a few days.

Greg: the whole thing needs a careful going over. Greg also needs a few days to go through this more

Status of cloud shadow:  In process of re-assigning staff to help with this. Ranjay – at the suggestion of Miguel. Need to make sure he has time re: DNB

Fritz wants to be on that email.

Fritz has an alert in the proposal – that uses blob detection. Instead of using a grid. It scans a scene and looks for blobs – a cluster of pixels that are identified as flood and then look to see if it is in somebody’s area of interest.

• Current system looks at 24 x 24km blocks: this is something that Fritz already runs.

He has written code for the new method.

1.5 billion points per year.

If new code reduces those further – it may come down

Documentation - Dan has provided some input. There are some issues with the code - which is why we are seeing some issues at higher latitudes.

action Greg to sit with David. make corrections and then compare it with the current test. 

Diane to follow with Maki

Fritz: could work on interface for alerts - but we can't call them that according to USAID

Greg mentioned that we now have a flood project in JIRA and a wiki for it. 

Along with schedule: Fritz would like to add some hooks in to the schedule. Diane to add these in. 

December 2, 2021

Dan, Sadashiva, David, Greg, Fritz

David: has been testing the code and tinkering with the formula that produces output. not entirely happy

• whole chunks of the ocean are listed as 0 = no water. not much missing data.
• think the layers are correct but data outputs aren't correct
• the question is how to manipulate the layers. Greg thinks it is an issue with the pixel selection.
• the code is selecting a single number that it thinks is best - so possibly loosing data. 

Dan needs to look at the code and provide feedback to David/Greg.

Maki provided an update on the NRT imagery.

• the first approach they tried didn't work.
• they are now looking at a second approach which will run the imagery every 30 mins

Fritz asked about LDOPE and the SOW that he drafted. Sadashiva has had some discussion - this is ongoing and will depend on resources.

Jared will be the manager for Fritz's ROSES call. No funding yet. Allocated .25 to work on LANCE side of things 

Sadashiva said Miguel and he had a meeting with NASA HQ (Mike F) about the transition to VIIRS. Mentioned that funding was request for this in FY22 - based on what Ed was anticipating. Need to make sure NASA HQ is onboard.

Fritz: been back and looked at his proposal. Below is the text in the proposal with regard to the user interface.  The calculation of the “potential flood event” (?) is a little more extensive, and we can discuss this separately.  Here is the information I proposed to collect from the users:  (1) region(s) of interest, (2) minimum population density, (3) minimum flood extent and (4) anomaly threshold (x sigma).  I am certainly interested in other ideas.

.. we will process the historical MODIS data following the above steps to create a data set for use in determining anomalous conditions and for product evaluation. Then (5) the potential flood alert product will be created from near real time data using the computer vision technique of “blob detection” to locate candidate potential flood AOIs (including center point coordinates and radius) in the flood map data, (6) we will filter the candidate potential flood alerts for a user-set population threshold (default value to be set through experimentation in collaboration with disaster management colleagues), (7) we will calculate the mean and standard deviation of flood indications for the AOI from the historical data for the day of year +/- ~ 2 weeks and cloud cover percentage of the scene of interest +/- ~10 percent for anomaly detection, (8) we will filter the candidate potential flood alerts to include only those with a statistically significant number of samples and which are anomalous conditions, defined as having a threshold number of standard deviations above the mean plus the margin of error for the mean (threshold to be user-set, with a default value to be determined through experimentation in collaboration with disaster management colleagues).  We expect that filtering for anomalies to be especially useful for eliminating “normal” seasonal flooding and false positives caused by inaccuracies in the water mask we use from consideration for the potential flood alerts.  It should also provide some filtering for false positives from shadows. An adjustable filter for minimum flood size along with a selectable area of interest will also be useful to international disaster management organizations to focus on large floods, while still enabling more local monitoring of smaller floods. Next (9), we will filter the remaining potential flood alerts for a default minimum flood size (again user-set, with a default value established by the proposal team), and finally (10) issue resulting potential flood alerts in both human and machine readable format, including a link to the MODIS flood map which includes the AOI. 

… the user interface to gather input on regions of interest, population density threshold, flood extent and anomaly threshold will be coded directly in LANCE without a precursor experimental set of code. It is our understanding that LANCE has acquired this capability through other similar efforts. This work is part of the LANCE effort which can be started very soon after project award.

Miguel suggested Sangjay work on Cloud shadow - he is already in SSAI task. won't need any budget changes.

Otmar and Asen to work on the UI

David could work on the spectral mapping to VIIRS once the operational code is ready

Meeting with ESDIS: Provisional time line. Aim to get the transition to VIIRS by May/ June.

• Diane to update the schedule / revamp the schedule

November 18, 2021

Fritz, Sadashiva, 

David: testing code. Greg sent links to test data for 2020.

• running same day through the code but not happy with the results. 
• pieces of code are good but the last piece that determines the 'best' pixel isn't delivering the results he would like
• not a big job: need to re-formulate the code and re-run tests
  • could be due to % coverage 
• Comments from Dan: looks like cloud layer is being used to screen out detections. in old system didn't do that as can see flood through thin cirrus cloud.
• Dan to send additional comments on code write up

Maki has delivered code and it is due to go in to testing. Then need to get it working in Worldview.

Fritz: written a SOW to improve cloud shadow mask.

• action to review email /SOW from Fritz (including looking at the paper cited)
• PGE 132 got baselined. Greg might want to follow through - ready for testing an operational processing
  • Sudipta and Maki discussing this
• David has adjusted the write up of the code // some more work to do on this.
  • Dan needs to provide more input
  • Comparison of detections from new vs old algorithm?
    • Greg to send David how to access the archive.

•  David will revise his draft algorithm description to address Dan's and Greg's comments
• Greg will contact Maki to see how she is doing. Would be nice to have imagery available for LANCE UWG meeting.
• Fritz will write a draft proposal of what work needs to be done for dealing with cloud shadow false positives. If we want to make progress on that at this time, will probably need more resources.

NRT vs End of day: data

• NRT flood is now available in NRT.
  • working on NRT4 yesterday. Hope to add flood products there on Monday.

Progress on OPS Code

• David to generate both versions of the output (old and new thresholding method) and then they can be compared by Dan.
• good progress
  • testing going well.
  • output between old count and new weighted version are v similar
  • NEXT STEP
    • Dan would like to see a write up of each method (should be done in a day or two)
    • Provide testing file outputs to Dan
    • Greg to take a look at the code 

Once OPS code is ready

• Dan thinks we make sure NRT and OPS process are doing the same thing and the results are the same at the end of the day
  • probably not. need to make sure NRT code is updated.
  • Need a coordinated approach to enable a test framework. Should be able to do it on the MOD-Dev C7 - action David and Greg

Fritz 

• followed up with Eric Vermote about cloud shadow. His response was that it has issues
  • Sadashiva thinks Eric doesn't have staff to work on this
    • Dan's suggestion
      1. make sure there isn't already an algorithm for this
      2. look at our own cloud shadow algorithm 
      3. produce a new algorithm and publish
    • Fritz
      
      • geometry approach
      • spectral  approach
      • Dan thinks there is also a time series approach

Progress on OPS Code

• hit a logic snag re: count layers
  • looking for a solution. issue is in the past kept track of each satellite seperately as it made a pass and created a file. Counted pixels. If it reached a certain threshold = water. Now a diff way to accept pixel or not based on compressed layer.
  • if we pick best pixel using zenith angle we don't have the 'counts' so need to figure out the logic for the multi-day product.
  • On the legacy product - each satellite has one 'vote' per day. So this would be compatible.
    • Maybe more complex where you have overlapping swaths?
  • David to generate both versions of the output (old and new thresholding method) and then they can be compared by Dan.

NRT vs End of day: data

• NRT flood is now available in NRT.
  • working on NRT4 yesterday. Hope to add flood products there on Monday.

NRT vs End of day: data and imagery

• Still some testing going on. Should be released on Monday.
• Maki is still working on this. 

Progress on OPS Code

• David is working on the code: still some bugs
• Running tests. Few things on Greg's list to implement. Should be done next week. 
• Greg's write up of the old code is done and David is using that as a template. 
  • Dan was able to review what Dan wrote. Greg preparing responses.
• Greg waiting to contact Sanjeeb - waiting to see David's results.

LDOPE Tools for testing

• Dan still looking to see if this works for him to be able to pull out the swath / granule

Worldview interface

• if you add the flood product why can't you download

Global Flood Partnership

• Fritz mentioned the LANCE NRT Flood. Questions: have we selected a DEM for upgrading the terrain shadow? Not yet.  Dan will need to look in to this.
• He also mentioned VIIRS and S3 or OLCI data

NRT vs End of day: data and imagery

• still testing. seems to be working. Still needs a security scan
  • so hopefully early next week
• Maki is still working on this. There was another

Progress on OPS Code

• David is working on the code: still some bugs
• He is still writing up the NRT code.
• Greg's write up of the old code  is done and David is using that as a template. 
• Greg waiting to contact Sanjeeb - waiting to see David's results.

Sadashiva summed up:

• Greg wrote out implementation of code that is currently running
• Dan to take an easy set of data / come up with an easy strategy for compiling the data 
• Dan needs to
• Access to LDOPE tools that can decompress the data 
• Action  Dan needs to see what he can do on his own with the LDOPE tools. Should have time next week.

Website & announcement from Earthdata 

NRT vs End of day: data and imagery

• Jihad has back end, Greg the front end. Matching the two up. Should be ready on staging by the end of the day.
  • security scan then production
• Maki is working on the imagery but no new news
  • Diane to follow up.
  • According to Greg she should have everything she needs to finish up.
  • She sometimes has a hard time with nasa email - use gst email

Progress on OPS Code

• David has started writing up the NRT code.
  • Need to work on this
• Greg going to document how the current legacy works so we understand the differences between the two systems.
• David has started implementing new logic on zenith code. and fixed bugs that were stopping decompression
• Sadashiva found that to compile full Format mode is just one small piece of code that needs to be changed
  • Sanjeeb needs to connect with Greg on this as soon as he is ready
  • Need special requirement to make sure the full format version doesn't go in to production

NRT vs End of day: data and imagery

• Neil looked at this. can list on web server node but were not mounted in to kupernetes. so web service couldn't see this
• Jihad says the infrastructure team are updating the service and it won't be ready for 10 - 14 days.
• on the imagery side: Maki has code ready to test on NRT4

Progress on OPS Code

• Exchange of emails between Greg, David and Dan to figure out how best to select pixels and thresholding
• action @ Greg and David to document exactly what the NRT algorithm is going -> doing
  • @ david will write up the current code and Greg the legacy to show in detail how the algorithm handles pixel selection
  • there is a slight difference between legacy and current code and it is not clear what the source of the difference is
• action @ Sadashiva to email Sanjeeb / LDOPE re: tool to uncompact layers (rather than re-running the PGE)
  • enable Greg/Dan to pull all pixels from files and thus test different pixel selection ideas.  

Fritz has been looking at ways to improve the cloud shadow mask.

2 September 2021

Dan, Fritz, Sadashiva, David, Greg

NRT vs End of day: data and imagery

• file names are showing up in NRT3 (in development) but they can't be found on LVFS
  • Greg has a ticket in to Jihad as this was working a while ago. 
  • various reboots have been implemented so it could just be a reboot is required
• Maki still working on the imagery.
  • Greg needs to reach out to Maki - this week hopefully.

David - progress on the code

• since last week: code is cleaned up and producing the proper metadata files.
  • been doing some testing
• NEXT STEPS
  • Greg needs to look at this and decide what to do with the zenith angle
    • Greg to write up
    • needs Dan's input too.
• David can merge GeoTIFF code ready prior to algorithm

NRT vs End of day: data and imagery

• some de-bugging going on
• Maki still working on the imagery.
  • the major difference between CR and flood is swath vs gridded

David - progress on the code

• working on methods for pixel selection. 3 methods sent Greg
  • Greg needs to look at this and decide what to do with the zenith angle
  • needs Dan's input too.
    • Greg to respond to email and then decide whether
• David can merge GeoTIFF code ready prior to algorithm
• Current code in NRT already has the geoTIFF stuff. not sure adding in ability to process pointer files will add to the NRT.

Fritz - putting together something on cloud shadow for Eric.

Sadashiva - asked for hurdles and meeting with Robert.

NRT vs End of day

• Greg's test code shows the update files. 
• So this is working on Greg's local machine. 
  • Next steps to put the code on NRT3 staging server for testing. 
  • additional tests need to be done as the code could impact LAADS files.
    • Once testing is done (mid next week) - it should be quite quick to stage it on NRT3
  • This also works for the TIFF products

NRT GIBS imagery

• Maki had a plan for implementing this
  • code will run PGE 159 then image code will run
    • not sure if next step is Greg's or Makki's

David - progress on the code

• all seems to be working fine. various test files. 
• David can merge code and put it in to subversion and get it ready to run as a PGE
  • closest to nadir.
    • Dan would like David to implement this on flood code.
    • check it has all components - including TIFF products
      
      • Dan would like David to implement this on flood code.
  • Still to be done .
  • then get ready for subversion then run PGE

Greg:  There will be differences in how we run code in OPS vs NRT

• may or may not be significant. Run David's code in OPS first then NRT.

Greg wrote up a document to describe some other measurements besides brightness that have an effect on deciding whether a pixel is showing water. These other measurements, solar zenith angle, sensor zenith angle, and pixel/grid-cell overlap area, can be used to improve the likelihood that a dim or dark pixel is, in fact, due to water and not some other factor.

• currently the code uses sensor zenith. So close to zenith gets higher weighting.
• ops code picks most recent pixel it sees.
• MOD09 uses nearest neighbor: closest to nadir.
  • Dan would like David to implement this on flood code.
• Dan would like to give the options Greg came up with some consideration

No weighting out until PGE 159 (apart from bad pixels). PGE 155 doesn't do any selection just maps pixels to grid cells. 

29 July 2021

Grid to swath / zenith angle /  interpretation of PGE155 Zenith Angle files

David - progress on the code

• Testing with additional tiles. seems to be working okay. 
  • NEXT: send code to Greg to put it in to the OPS system and run a 3 day test. 
  • if it looks okay run larger test.
  • It will be a couple of weeks when back from leave
• Equator to 30-40 N runs in 15 -20 mins
  • David can try the code with some NRT inputs (one swath) and see how quickly that runs
  • David can merge code and put it in to subversion and get it ready to run as a PGE
    • check it has all components

NRT vs End of day

• Greg has the files showing up
  • next add it to the staging service and see if it can be downloaded

22 July 2021

David, Greg, Sadashiva, Dan, Fritz,

8 July

Greg, David, Sadashiva

Grid to swath / zenith angle /  interpretation of PGE155 Zenith Angle files

• L2G discussion still to take place. with Sudipta, Sadashiva, Greg and Dan
  • for standard MOD09GA is already aggregated at the end of the day: pixels picked out on best quality 
  • could be used for water detection and flood detection. 
  • wouldn't need 155 as it would be a use the MOD09GA gridded product
    • David wouldn't need to decompress the zenith angles. they could just be read.
  • Sadashiva suggests we don't un-compact everything.
  • When you run a granule in L2G in NRT: first layer nearest neighbour observation
    • pick up geo angle at that time
    • when layer next comes in you can see the observation from the previous observation. 
    • some work to get solar zenith.
•  NRT vs Daily delivery of flood product.
  • Greg working on website. Jihad has done his changes for now.  A week or two away
  • Maki working on getting the imagery in to GIBS
  • Legacy system running until end of August
    
    
• David has made a lot of progress on the code
  • old version was using interim files to take the 1km zenith angle files and scale them to 250m
    • being done pixel by pixel (1km → 500 →250m) - it was taking 70 mins to run
    • that has been replaced with new code so it runs in 11 mins
  • currently being run on end of the day water files
    • would run on NRT: existing end of the day + level 

1 July 

Greg, David, Sadashiva

Grid to swath / zenith angle /  interpretation of PGE155 Zenith Angle files

• L2G lite is a daily product. Greg spoke to Sudipta, so wouldn't work for NRT
  • Sadashiva: doing the development for OPS processing not NRT?
  • Greg: need clarification from Dan: the issue shows up in NRT as well as standard.
    • NRT is not using the zenith angle
  • SD: running L2G process in NRT as granules come in. generate gridded tiles as soon as they come in
  • as granules come in process L2G = that is processed for that granule. So already doing it for NRT
  • If you are doing it that way you are processing once granule // what are we trying to acheive. 
    
    
• Are we trying to implement the zenith angle for NRT or OPS?
  • both - to avoid false positives
  • maybe use different approaches. we have been trying not to. Need to do:

1 . refinement to NRT PGE and implement some improvements

2. development of OPS PGE using L2G lite

archive all NRT and use that for the development and testing.

• need to see about archiving the pointer files

TAKE THIS OFF LINE: DISCUSS BETWEEN DAN, SADASHIVA, SUDIPTA, GREG - Greg to call the meeting to determine the way forward. 

Put code development on hold to see if we use L2G lite

NRT vs Daily delivery of flood product.

• Greg not got to this. NRT bug = needs to be fixed.
• Greg needs to follow up with Maki on imagery

Code that David is working on is going well. 

User guide and website are updated.

24 June

Greg, David, Fritz, Dan

Grid to swath / zenith angle /  interpretation of PGE155 Zenith Angle files

• good progress on speeding up the algorithm.
• still need to look at L2G lite

 NRT vs Daily delivery of flood product.

• Partial implementation. Greg thinks it look
• Jihad needs to do a bit more. Greg to start working on the web side
  • possible just a week or two away
• Greg to follow up on NRT imagery

NEXT STEPS

• modify compositing rules and assess the impact – DAVID
  • currently using the fixed threshold (2)
  • probably a 50% threshold would work.
  • Dan then to assess how well it works by looking at a couple of days of global data using the TEST set up that Greg has.
  • 3 levels of testing for both changes.
    • CHANGE OF WHICH SR OBSERVATIONS ARE ANALYZED
    • CHANGE THRESHOLD RULE 
    • before that run on a few files (before and after), then run on larger. then on a science test on a production system (couple of months or weeks scattered around the year).
  • Don't update the operational algorithm until both are done. 
  • re-run dates in March or go back to Jan

Legacy system: June 30 is the date for cut off.

• Dan to send a note with user guide. keep it running for at least 30 days after NRT is running in NRT

17 June

Greg, Dan, Sadashiva

• PG 13 MOD09 L2G lite. All in one file
  • spend some time looking at this as an alternative. 
  • 1km - 500 and 250m: Action Greg / David to look at it to start 
• User guide
  • few comments for Dan.
    • page 1: reformat to look like other user guides
    • revision to be moved to second page
    • section 6.4 add figure to illustrate para.
    • Dan to give Greg a tile and he will get this.
      
      
• Imagery showing up in GIBS at the end of the day
  • Greg to contact Maki

 NRT vs Daily delivery of flood product.

• Jihad has finished his part to now it is up to Greg. It will still be a week or two. 

Grid to swath / zenith angle /  interpretation of PGE155 Zenith Angle files

• Up date from David: I'm making progress in testing my files. One file is made with my old program (that takes 70 minutes to run 1 file) and the other with my new (tweaked) program (that takes 15 minutes to run). My hope is that the two files will be sufficiently equal that I can go with the newer code. So I'm working on a way to compare them directly, layer by layer, pixel by pixel. I hope to have that functioning soon. 
• There maybe other ways Greg can help. 

10 June 

Diane, Greg, Dan, Sadashiva, Fritz, David

NRT vs Daily delivery of flood product.

• Ceph is close to ready on nrt systems. Once it is ready, Greg can work to incorporate it into nrt web sites for NRT update of Flood files. Greg will follow up with Jihad (see the ticket at https://gitlab.modaps.eosdis.nasa.gov/infrastructure/help/-/issues/95 )

• Robert decided, after talking with infrastructure group, that Ceph sounded close enough that it didn't make sense to re-implement PGE159 using the DNB production rules

• Jihad was just back from vacation. 

  • • Greg to ping Jihad and ask about timeline

Grid to swath / zenith angle /  interpretation of PGE155 Zenith Angle files

• David got a version working that takes 1km to 500m and scales to 250m for the zenith values.
  • It took over 1 hour for each file/per satellite. So D worked on code so now it is 15 mins. 
  • everyone els has written their programs in C. this is in Perl. so slower. If it can process the entire array it would be okay but this is pixel by pixel.
  • Sadashiva said it has to be done pixel by pixel even in C.
  • Greg to work with David on this in an advisory capacity
  • Spend another week or so on this and if there isn't much progress in terms of the speed of the code, consider writing it in C.
• Dan should consider L2G lite reflectance which has a "best observation" – what about just using this? 
  • it would make it more simple as there would just be 2D data to deal with.
  • direct correspondence between 1km - 500.
  • would just be using one observation instead of full resampling ( or nearest neighbor in NRT).
  • PG 13 MOD09 L2G lite. All in one file
    • spend some time looking at this as an alternative. 

User Guide

• Diane to remind Sadashiva to finish reviewing the User Guide

Flood alerts

• on the legacy system there is a JPEG map 
• on LANCE it can be viewed in Worldview

27 May

David, Dan, Greg, Sadashiva, Diane

NRT vs Daily delivery of flood product.

• infrastructure team have implemented a directory server that allows discs to be seen on other machines = first step
  • now the system programs are available everywhere
  • installed on moddev-1
  • so making progress but not yet available for production
  • Greg can start working on the instance on mod-dev1 to check code
• Sadashiva will write to Carol and cc Neal & Robert to see if we can move this forward - by asking Carol's group to re-write the production rules. It already works for DNB. 
  • it will give the developers more breathing room to implement the new system.

Pixels: grid to swath / zenith angle

• David has code from Sudipta to assist with the compositing. Should be sufficient information in the SR guide to follow. 

User Guide

• Sadashiva almost done reviewing this.
  • looks good. Dan to sent word file.

VIIRS continuation flood product

• There will be future calls to try and get the flood product as a science product.

20 May 2021

David, Sadashiva, Greg, Diane

VIIRS flood - need to put it in the plan. Important to demonstrate good quality products from MODIS first. 

discussion about how we want to use the historical data. 

Think about how we can make it a standard product?

NRT vs Daily

Neal has done his part. now with Jihad.

Pixels: grid to swath / zenith angle

Suggestion by Sadashiva to consider another approach 250m: grid to swath / zenith angle

13 May 2021

David, Dan, Greg, Fritz, Sadashiva

Pixels: grid to swath / zenith angle - David still working on this.

• Need some input. Maybe Sanjeeb? He has worked a lot with the gridding code. 
• Email questions and cc Sadashiva

Earthdata flood article. this will wait until the data is available in NRT

• NRT vs Daily
  • Neal has a piece he needs to do and then 
  • Greg will need to do something on his end

Qualitative comparison

• this is finished. almost finished updating the user guide
• looked good. 
  • impacts: increase cloud shadow false positives in the N but capturing more surface water (lakes and rivers) when the thresholds are update for the cloud shadow it should be an improved product.
  • Do we then release the updated user guide for the NRT?

https://modis-land.gsfc.nasa.gov/MODLAND_grid.html 

• Sadashiva to follow up on this. 

NEXT STEPS

• once the looks are sorted
  • implement thresholds
• run a year of data just to verify
• then 10 years

6 May 2021

David, Dan, Greg, Fritz, Sadashiva

view zenith angle is 1km (@the edge of swath they are given less weight).

need to get no of water layers to match zenith angles.

need to see what % of grid pixels overlap swath pixels to decide flood count determination.

either case need to map from pieces of swath pixel to grid pixels whatever the resolution.

interim products that are produced that could be used to de-code.

GeoTIFFs and NRT issue

Qualitative comparison - going well. better delineation of water bodies. false positives are an issue. otherwise products are the same.

User guide - being updated. Finish up qualitative evaluation and add that to the guide. 

Greg made data available for Dan 

Mailing list – non-NASA users won't be able to go to the web interface. they will need to just send  and email to subscribe and unsubscribe. 

https://modis-land.gsfc.nasa.gov/MODLAND_grid.html lat long grid missing . Dan to ask Sadashiva.

Geotiffs look fine; communicated their availability to Disasters group.

Evaluation

Working on qualitative evaluation (of the 100 sites over 3 years), which for as far as I've gotten (not too far...) looks great. Minor pixel-level differences, but nothing at all concerning.

Found an issue in the current product - volcanic areas in Hawaii are showing up as flood. We had this issue several months ago because the original terrain shadow files I delivered to Greg did not have the fix that we'd implemented in the legacy system. So I provided the updated files way back in December, and I believe I verified way back then that the issue was fixed. However, all the data in Worldview (so since Mar 23 or so) have this problem. I suspect the set of terrain shadow files got reverted to the original set or something. 
Should be fixed now. Greg has implemented the files from Dan in OPS and NRT

GeoTIFFS 

running on NRT3 and NRT4

• Diane and Dan to update the landing page and user guide

NRT vs end of the day issues

Robert suggested a meeting. Not yet scheduled.

OPS Code

David has been working on the pointer files and pixel sampling.  

Earthdata article

Fritz and Dan working on flood article 

Fritz's Alerts: using planet scope - making good progress. idea that when get an alert - can make a more detailed map using high resolution data. as a derived product this could be distributed. 

22 April 2021

No meeting last week. Notes for week before not available on the wiki

GeoTIFFS

• running and in production on NRT4
• Dan to check they look okay
• next step to make it run on NRT3
• Diane - to let the Disasters Portal know they are available

NRT vs Daily

OPS Code

• Greg and David still working on pointer files and Pixel sampling tests using LDOPE tools
  • some have 4 and some have 6
  • when water file pge152 detects water = swaith run pge155 to get grid. then there are many swath pixels that overlap a grid pixels. need to decide which are useful and which to keep. read pointer files to give view / zenith angle. zenith pixels are 1km but grid = 250m
  • Sadashiva says there is there a process
    • Sudipta has given information on how it works
    • still need to learn and understand how that works
• Updates to OPS code will be added to NRT code. This should be straight forward.

Earthdata article

• Joseph has been in touch with Dan and Fritz

Evaluation

Currently on version BETA

Qualitative comparison with legacy MWP: 100 events of 3 years.

• Dan has this data and has started the comparison.
• Should be done by Mid-May

Next step will be to test the thresholds 35, 65 etc. // need OPS code running first before starting that. 

of 3 years to be done by Dan – then go to provisional 

Database

• 225 days and 850 million flood pixels. uptick.

Flood viewer

• Diane to set up a meeting with Albert and Bob Brackenridge to look at requirements for Flood viewer.

Greg 

• to limit legacy flood product to 3 days instead of 7 
• then add cloud shadow layer from MOD09 for Fritz – this will come out a TIFF.

• 8 April.

Greg, Dan, Fritz, Sadashiva, David

GeoTIFFS can be pushed to NRT code now

• trying to integrate David's code to Gregs (issues with memory allocation)
• will continue without David's code for. will wait until Monday to push to production  
  • David's code handles more than one pixel // this has implications for thresholding.
  • Dan thinks we need a test for the appropriate threshold which may take some testing
• DOIs have been registered:
  10.5067/MODIS/MCDWD_L3_F1_NRT.061
  10.5067/MODIS/MCDWD_L3_F1C_NRT.061
  10.5067/MODIS/MCDWD_L3_F2_NRT.061
  10.5067/MODIS/MCDWD_L3_F3_NRT.061
• once Greg makes these available - Diane needs to update the landing page and Greg update the user guide.

Worldview - imagery now running in production - https://go.nasa.gov/3sWIDCx

OPS code

• David has this in hand. Greg integrating it in to his code.
• David started doing the 3-day tests. hit a problem that Greg is now correcting. 
• Supposed to be first 3 days of 6 months. Got 2 days then snag.

Pixel sampling tests using LDOPE tools

• Dan needs to say how he wants to process pixels from different swaths vs orbits
  • 50% threshold? Request Sadashiva and LDOPE folks provide guidance.
    • Same orbit: aggregate by coverage and cloudiness. → separate cloud and clear pixels. partial cloud is of concern
      • nearerst to pixel center first option but if cloudy choose non-cloudy as long as it covers certain percentage of grid
      • currently count number of cloud and water pixels. 
      • Is the fraction significant enough to be included? 
    • Different orbit: harder as different time. need to look at geometry. which one do you trust the one close to nadir (geometryr) or % cloud, or height of cloud
      • Action for Sadashiva / LDOPE provide pointer to core segment for initial development. then dan can take a closer look on how to tweak this.
• when pixels overlap a grid cell it counts each of the pixels - hence the work on the threshold. it will depend on the number of pixels for that cell from overlaps and scans. adds them all up for each category (water, shadow etc) then apply threshold rules. This is what David's algorithm.
• need more guidance from Dan
• Greg has a better idea of the operations

Database

211 days and 800 million flood pixels

No meeting next week

Fritz's questions:

• will need to switch from experimental system to the new system. 
• size of files? 10 x 10 degree but pixel size is different
• in a week the GeoTIFFs will be available
• when will we process the whole archive? Not for a while. want to implement the algorithm changes first. 

Images are now being produced on NRT3

• can we check images are produced in NRT (not at the end of the day)
• Neal has to make sure it is available. We need to let the GIBS team know

 Archive

• we have the okay from Robert to do provide an "experimental" archive

OPS Code

• Refining the OPS code and testing it with water files that David has been generating.
  • Seems to be working well. 
  • Output looks good
  • Need to merge with changes for GTIFF files
  • The Greg needs to make it run as a PGE in OPS.
    • Will do a 3 day test.   
    • Get Dan to check it then do a 3 year run

DAN 

• C6 vs C61: looks good.
• Evaluate 3 year run in OPS - will take approx one month // there is a small chance we will see differences (unlikely)

- STILL TO DO

• Pixel sampling from scans using LDOPE tools - Greg thinks David might be able to help with this.
  Dan to send a list of things to be tested to David and Greg
• Look at thresholds from OPS algorithm from David // writing that up

GTIFFS 

• Disasters group looked at the GTIFF.
  • They would like both tiling and internal pyramids. It is a simple GDAL command to do on their end. 
    • suggest we go ahead as is now. Albert from DFO is happy with it.
    • it would increase the file size
• Look at thresholds from OPS algorithm from David // writing that up
• Evaluate 3 year run in OPS - will take approx one month // there is a small chance we will see differences (unlikely)
• Will be available in NRT when LVFS issue is sorted (Neal, Jihad and Navid working on this) Greg to follow up to see where they are at.

To be revisited 

Add pyramids to GTIFFs

• PGEs are almost running on NRT3 so that should happen in the next day or so. 
  • once confirmed they will start sending the imagery to GIBS and WV will start their process.
• Still waiting on LVFS for NRT granules - this will also have an impact on GeoTIFF availability
  
  
• GTIFFs
  • reviewed by Dan and look good. 
  • sent some to Jeremy (NASA Disasters) and DFO for approval
  • just finalizing whether they have tiles vs pyramids
  • metadata - minor updatesC6 vs C61
  • will be created in NRT - trying to get them staged on the website at the same time.
    • currently will only get them at the end of the day like the HDF products
  • Check doi registration for GeoTIFFs 

DAN - STILL TO DO

• C6 vs C61
• Look at thresholds from OPS algorithm from David
• Evaluate 3 year run in OPS - will take approx one month // there is a small chance we will see differences (unlikely)
• Pixel sampling from scans using LDOPE tools - Greg thinks David might be able to help with this.
  Dan to send a list of things to be tested to David and Greg

OPS Code

• commenting code to make it clearer
• David waiting for thresholding from Dan
• then more testing before code gets turned over to Greg to integrate in to the system
• this is almost ready but the recent changes in the NRT weren't included. PGE 155 updates and PGE 159 (changes that David)
  • due for a big PGE release when establish current NRT is working on 3 and 4. then do a new release with the changes. can roll back to this version of there are any issues in the new PGE release.

Database 

• 190 days 751 million global pixels
• need to think about how to manage this.

11 March 2021

Priority for Dan

1. GeoTIFFS
2. C6 vs C61
3. Look at thresholds from OPS algorithm from David
4. Evaluate 3 year run in OPS - will take approx one month
5. Pixel sampling from scans using LDOPE tools - Greg thinks David might be able to help with this.
   Dan to send a list of things to be tested to David and Greg

Diane to initiate discussions with Karen and Robert re: archiving the flood product

GeoTIFFS

• wrote and tested code as part of PGE 159- on products in test products in OPS
  • These are on moddev-c7
    • /SSTG/gederer/PGE159-TEST-LOTS-GEOTIFFS/MCDWD_L3_F1
      /SSTG/gederer/PGE159-TEST-LOTS-GEOTIFFS/MCDWD_L3_F1C
      /SSTG/gederer/PGE159-TEST-LOTS-GEOTIFFS/MCDWD_L3_F2
      /SSTG/gederer/PGE159-TEST-LOTS-GEOTIFFS/MCDWD_L3_F3 .
  • Now have about 10k GTIFF images 
  • Greg doesn't have a good GTIFF viewer
  • @ Dan Slayback to sign off on these and then if happy send them to disasters for their approval.
• formal PGE release process needs to be undertaken. 
  • CM would need info and then deploy it 
  • at this point the main concern is that it does what we want
  • already added new products in to dbase on NRT4
    • this would need to be done dbase on NRT3 and LAADS
  • currently no mdata other than std geolocation 
  • using GDAL command
• Asad is registering these as DOIs
  • Dan to send Greg an example of how to add DOI in to metadata tags 

CMR & EMS - diane to follow up with Asad

Flood imagery

• Imagery is going to GIBS correctly.
• Production imagery only comes from April 1st. 
• waiting for Worldview team to schedule when it will go in to Worldview
• Sadashiva to email Gang and Matt

3 year Run in OPS

• complete
  • Dan has the data and needs to take a look

C6 v C61 comparison

• Greg finished this and data is available for Greg to look at
  • Dan needs to take look 

David and Greg working on OPS algorithm

• developed code that can look at the number of observations and determine threshold
• Question is what is a good threshold %? 50% - extension of what was done before 
  • David to run the products for Dan to take a look at.
  • single file is ready for viewable.
  • David needs more to run more tests. Greg will get these to David.
• When done run in OPS as separate archive so can compare. 
• will keep this separate from GTIFF code. as change to thresholds is more complex
  • Greg was hoping to get the GTIFF code added
  • wanted to check C6 vs C61 is okay 

PGE 155

• had some changes too that need to go in to an OPS test (3 day followed by 3 year)

LDOPE tools 

• Dan to take a look at brightness temperature - as this might have less cloud
• This is a re-sampling process.  looks at max observation coverage and takes that one. 
  • if other observation covers the grid cell it is ignored so you are only seeing partial coverage.
  • do we want max observation coverage or aggregate?
  • sometimes weighted average of observations
  • can you take minimum brightness? yes 
    • for flood. if diff surface areas the one with lower brightness would have less cloud. 
    • ? from Dan. are there tools to enable Dan to test the different options?
  • yesterday Greg was trying to get to the bottom of this. via Sudipta Greg found out that LDOPE has tools that enables separate SDS to be read and look at outcomes. sounds like a research thing - not a quick and easy answer. once figured out could back port in to NRT

3 March 2021

LANCE requirements:

• Data - on redundant servers
  • add website updates to
    • earthdata.nasa.gov
    • lance3 and lance 4
• User guide
• DOI and landing page
• Register in CMR - enable earthdata search
  • Asad is doing this 
• Register metrics with EMS 
• imagery in Worldview

GeoTIFFS

• shortnames
• MCDWD_L3_F1_NRT
• MCDWD_L3_F1C_NRT
• MCDWD_L3_F2C_NRT
• MCDWD_L3_F3_NRT
• doi - CMR - EMS registration
• add information to user guide and landing page

GIBS Imagery

• next sprint starts next week - need to get this included!
• Greg to contact Sudipta to check all good.

GeoTIFFS

• added code. not been able to test it yet. 
• will be able to test later today or yesterday 

Latency

• PGE 159 runs every 10 mins to check everything is coming in. 5 mins to process. send results to imagery producing PGE. 
• PGEs that process MOD09 - immediately 
• maybe additional padding in the scheduler - 15-20 mins

3 year Run in OPS

• complete
• Dan has the data and needs to take a look

C6 v C61 comparison

• Greg finished this and data is available for Greg to look at

Database  - is going well 

177 days

David working on OPS algorithm

• looking good.
• now a question of threshold for flood vs non-flood
• in the ST if we want to replicate what is happening in NRT can we do that?
  • current code in 3 year is exactly the same
  • integrate Sanjeeb and our changes - may see some differences.
  • Dan should do evaluation. so step 1 is done.
    
    
• almost same code as for NRT but running in OPS. Not quite - same as it just accounts for pixel from first scan. if there are additional overlaps it won't include them.
• 10 lines at 1km. as you go further from nadir you get more pixels overlapping. these are counted and retained by PGE 155_LG code. takes all the swaths and grids the pixels
  • in NRT look at one swath at a time. just look at one layer. not at overlapping pixels. first pixel in any overlapping scan. those pixels are in the 155 but just look at the first one the rest are thrown away.
  • scan and grid don't align. so go at an angle to the grid cell. even at the center if there is a scan line overlapping 2 grid cells
  • This is a re-sampling process.  looks at max observation coverage and takes that one. 
    • if other observation covers the grid cell it is ignored so you are only seeing partial coverage.
    • do we want max observation coverage or aggregate?
    • sometimes weighted average of observations
    • can you take minimum brightness? yes 
      • for flood. if diff surface areas the one with lower brightness would have less cloud. 
      • ? from Dan. are there tools to enable Dan to test the different options?
      • yesterday Greg was trying to get to the bottom of this. via Sudipta Greg found out that LDOPE has tools that enables separate SDS to be read and look at outcomes. sounds like a research thing - not a quick and easy answer. once figured out could back port in to NRT

Variable (per pixel) threshold modifications will require changes to NRT. This could be done a the same time. 

Action @ Dan to think about this before integrating it in to the OPS code. 

• Dan asked Greg to consider how could the 50% threshold be implemented?
  • thinks it will be straight forward

24 February 2021

discuss GeoTIFFs and where to store them

Revisit: date for removing legacy product

GIBS Imagery

• diane to contact GIBS
• dan to look at imagery in GIBS

GeoTIFFS

• Need to add code to PGE 159 to create GTIFF for each 10 x 10 tiles (4 of each)
  • daily cumulative GTIFFs per tile - updated throughout the day 
  • 1 x file for each tile throughout the day. the more overpasses the more information you see in that tile.
  • each file name has a different
• If we want them to be downloadable we need product names / ESDTs and added to the database tables as we do with other products. 
  • this means they will be staged to the website and put in the download directories on the NRT systems
  • in ops - they would get archived
  • it requires some table edits and scripts
  • NAMES: MCDWD - contain tile and date but not time. 
  • action @ Dan Slayback to come up with names 

Diane to contact Robert about the end of day product - so he is aware and contact Neal and Asad re: GTIFF

User Guide

• Almost finished. Question on the improvements. Should it be done in 2 phases:
  • 1 - getting this one out plus evaluation comparing to legacy and evaluation (keep as beta)
  • 2 - rest of changes (release 1)

3 Day Run in OPS

issue with three day production rules

• days where no data were produced due to sensor calibration etc
• production stops at that point as it can't find previous day 
• need to amend production rules to handle that situation

NRT processing

OPS Processing

• in general working well. compressed water layers decompress properly. 
• weird problem combining them with a ref water mask
  • need to find a way to test this
  • same code that used to work

Database

• 169 days and still running. 700 million pixels 
  • 1 billion records= 1 year - a 40% drop after new terrain shadow masks are in place.

Diff between C6 (OPS processing) and C61 (NRT)

• Greg has been working on a script for this on his dev server

Terrain Shadow zip files

• some terrain shadow zip files that David has received has zero data.
• Dan uploaded them again for David to download but he is still having issues // suggest command line utilities to unpack them

18th February 2021

Dan, Fritz, Greg, Sadashiva, David and Diane

User Guide

• Sadashiva to review today / tomorrow
• Please look at last section wrt who we term the versions.
  • for MODAPS usually beta, provisional, validated
  • in this case Dan is thinking:
    • PHASE 1: close to old system
    • PHASE 2: improved terrain shadow etc

3 Day Run in OPS

• OPS has about 100 recipes.
• Kurt is running the date ranges that Dan sent.
  • Dan can start looking at the data on MOD_DEV_C7
  • Files should remain until we ask them to be removed

NRT processing

•  Still waiting on Jihad and Navid.
  • looks like solution for DNB is not enough
  • not sure how they are looking in to it.
• There is a virualization that happens that makes it look like there is no production time. 
• LVFS has to do work to find them. We need to do work to manipulate the file name.
• Can we do it the old way? No not at this point.
• Agree to provide products at the end of the day to avoid further delays. Imagery will be NRT

OPS Processing

• Greg has been working on getting test files to David
• found bug in L2 G that Sanjeeb has fixed.
  • not impacting NRT
• David has things almost working - seems to be an issue with the reference water mask.
• Test files for T & A seem to be work 

Diff between C6 (OPS processing) and C61 (NRT)

• Greg can check this on his dev server
• some terrain shadow zip files that David has received has zero data.
  • Dan to upload them again for David to download 

Database

• 162 days and still running. 685 million pixels 
  • 1 billion records= 1 year - a 40% drop after new terrain shadow masks are in place.

11th February 2021

User Guide

• Dan to upload to One Drive
  • Sadashiva and Diane to review

3 Day Run in OPS

• 3 year test run. plans are entered for this. currently not running
  • asked for a list of date ranges. these were sent. need to follow up to see if will run 3 years or date ranges.
  • This will need to be run in C6
    • Could to check MOD09 differences between C6 and C61
    • Greg could run flood as c6 and c61

Top 2 row issue

• ignore for now

NRT Processing 

• re: data not getting put on the NRT download site when complete
• Needs changes from Jihad to make them show up in LVFS (end of the month) it is an input and write it back out. original system wasn't set up for this. 
  • Sadashiva to follow with Neal / Jihad

David working on OPS algorithm

• good progress. need better test files
  • Greg to send these to David. Creating these via the dev server. Almost done.

Worldview / GIBS

• GIBS imagery is being produced but they are ignoring it
  • Dan - to give Minnie brief overview
  • once we hear about NRT data we can give GIBS the go ahead

3 day run in OPS

• ran okay
• Dan looked at it briefly and it looks okay.
  • needs to look more rigorously before asking for 2 years. 

NRT Processing

• There is an issue where it not putting the data on to the download site until the end of the day.
• Greg to contact Neal about this
  • Not sure if it is MODAPS issue or PGE issue?

David is working on OPS algorithm

• layers are decompressing properly for the new structure of the data - based on the internal storage structure

Validation plan

• Dan has been collecting data. needs to summarize it still. should have something in the next couple 

User guide

• waiting for validation 

Removing the northern tiles 

• No terrain shadow file for Dec - Jan 22. 
• The algorithm should be using the new TS file so it should be better. 
  • excessive 

Worldview Diane to contact WV and GIBS about possibly going live at end of Feb and adding flood imagery to UAT - GIBS

Database

• 141 days and still running. 630 million pixels (3-day product)
  • images look good
  • if n. terrain shadow is more effective we should be seeing a drop in the number of daily pixels.
    • There has been a 25% reduction in the number of flood pixels.

GMU and UWM have a global flood mapping product. Fritz has been applying his flood alert. They claim to have solved the problem of terrain shadows. Fritz is cutting out Landsat shaped tiles. Looking at the flood pixels (0-20%) high number for Landsat areas. If only look at 100% pixels still think it will be hard to determine which ones are important floods. 

21 January 2021

• David Landis
• Sadashiva 
• Dan Slayback
• Greg Ederer

3 day run in OPS

• Issues: production rules and output in production imagery
  • out put for the test. for all the days it is missing the previous L3  (MCDWD_L3) file. 
  • production rules don't wait to make sure the previous day is available. 
    • not an issue in in NRT but it is in OPS
  • if you look at the 3 day flood, there is often no data but it should have the current days data.
    • most have these narrow pieces of swath so that sounds like a code issue
    • Greg ran the code on his own system - the production algorithm should be resolved next week
    • also need to check the algorithm part re: previous days data. Use current L3 file but use previous day L3 file. Production rules allow you to skip it as they are set up now.
    • Composite is today plus L2 from previous day
  • Suggest address production rule issue first? But there is something going on that needs looking at - consistent error (seeing wedges of data). 
  • Dan will look again at the NRT outputs to make sure they look okay and confirm that he didn't miss this issue. 
    • when previous days L3 is available, don't see the wedge shape.
    • Wedge suggests overlap from swaths and only getting sufficient observations from wedge.
      
      
• David is working on OPS algorithm
  • Greg has been helping 
  • trying to decode the additional compressed layers in the file
    • tweaking algorithm. reading the algorithm slowly as it is building 1 pixel at a time. 

• Database
  • 135 days and still running. 6 million pixels.
    • images look good
    • oval shape flood – eastern Siberia 18th Jan
• Email alerts
  • running daily
  • getting some cloud shadow anomalies
  • 24 x 14 km - 360k of them so bound to have anomalies.
    • starting to work with Landsat grid
• Validation plan
  • release the product without doing the validation just summarizing the statistical
  • in N. getting more cloud shadow issues
    • release before or after this is fixed? N. latitude - will need variable threshold which will need work.
      • could ask GIBS to not put those to rows in to GIBS. NEED TO UPDATE THE TILE SCHEME IN NRT
      • SD: recommendation would be to not produce the high latitude tiles.
        • any attempt to communicate the issues is fraught
        • so what is the best way to implement this?
          • don't produce them? different tiles scheme. requires no adjustments elsewhere
          • can run tests with the tiles in as needed.
    • do we want geoTIFF outputs?
      • we can use WVS

• 3 day run in OPS
  • no updates
  • Kurt is running this but doesn't know he is working with
  • Greg is coordinating it
• Schedule
  • no update
• GIBS imagery
  • no update
• GeoTIFF
  • no update
• Database
  • 128 days and still running.
  • 1.7 billion records - estimate for a year.
    • may change as we move away from the winter solstice and terrain shadow 
• PGE 159 Updates
  • David fixing it to work with PGE 155 inputs. he ran in to a few issues reading the HDF metadata. 
    • Greg working on this
• Dan could come up with a different plan for evaluation - compare legacy and NRT
  • this would enable us to get the flood product
• Email alerts
  • 24 x 14 km in Fritz's
  • we want 10 x 10 degree blocks
  • makes sense to bin things by cloud cover percentage. 
    • need to look at this more carefully as not a linear relationship 100% cloud
    • interesting to incorporate this in to the flood algorithm or flood alert detection
• User Guide
  • Dan needs to work on this - as a beta version
  • and a document comparing LANCE with heritage system 
• Set release date for 14th Feb

7 January 2021

Greg, Sadashiva, Dan, Frtiz, 

• 3 day data in OPS had some issues - missing files
  • hopefully just production rules. Greg will take a look at this.
• GeoTIFF - action for Greg
• Database
  • thresholds from Dan.
  • once these are in they will be ready to test with the pre-composited water products.
  • continuing to run files in to the database: 121 days - 556 million flood pixels (seem to include a bunch of terrain shadows especially in the Himalayas).
• Dan managed to get the legacy system to the new server

17 December

Fritz, Dan, Greg, David, Diane

•  3 day run in OPS
• Dan has collected data
• Action @ Dan to take a look at the data - to be done by 23.
• Schedule
• run 2 years of OPS data.
• This could start this next week but need input from Dan first.
• Action @ Greg to wait to hear from Dan and when he does – to initiate the 2 year run.
• GIBS imagery
• will update code to prevent yellow recurring flood showing for now.
• release late Jan
• GeoTIFF 
  • currently using  GDAL
  • specs: tile by tile just for the flood products
  • 2&3 day
  • naming? in the past never put more than one dataset in GTIFF file* Dan to give this to Greg
  • not all readers can handle multi-layer GTIFF
  • start with a single band.
  • get some user feedback.
  • archiving GTIFFS : need to have names for each file
• needs to be added to PGE 159 - Action @  Greg to do for NRT code
• Database
• Dan sent thresholds to David as a placeholder
• PGE 159 Updates
  • Could get them from Curt but he will need a test plan 
  • Action @ Greg and David. get our own samples by running them in MOD_DevC7   
  • if this doesn’t work ask Curt
  • Sudipta and Sanjeeb running PGE 155 when running in OPS had to remove old files (from their test) as their files wouldn't run wtih PGE 159. 
  • need new test plan that doesn't overwrite. 
• David working on this. He has a sample format. working with the code.
• hope to have multiple samples. 
• User Guide
• Ongoing
• Flood Alerts
• Dummy thresholds sent to David
• Work plan - Ed has submitted this ESDIS
  • ? 1 day = one water detection, 2 day = two water detections etc
  • could change threshold and make it dependent on the number of observations. this would get rid of some of the false positives. 
  • F: will look at compositing 
• Ed, Robert and Sadashiva had a conversation. Fritz will put together a limited proposal for ROSES A33 to do some post processing of the initial flood mapping using fuzzy logic to reduce false positives. also details on the flood alerts.
• Question about NOAA VIIRS Flood product – how does it differ? Should LANCE produce a VIIRS product if NOAA already has one?
  • they claim 95% success in terrain shadow problems
  • use a geometry approach for cloud
  • for terrain they use surface roughness via DEM
  • NOAA doesn't produce a surface water product. they mask and only deliver flood.
  • not sure about archive and long term distribution - only go back to 2019.
• GMU and UWM have a system running and a 2018 paper. In process of transitioning it to NOAA
• they did a comparison with the MODIS product in the 2018 paper.  
• Fritz is still trying to figure their flood alert. 
• VIIRS has SW band - good for water
• What about Sentinel 3 – will we get NRT data?
• Fritz asked Dan how long for experimental server to move to LANCE
• Action @ Dan to add this to the schedule

• 3 day run in OPS. Plan sent  by Greg.
  • Greg to send a reminder and Sadashiva to check 
• Schedule - Diane sent around and would appreicate feedback.
• Flood alerts: 
  • Dan to send sample threshold data to Greg and David. To add in to dbase.
• David adding data to dbase 93 days. Sept 8 to now.
  • 430 million flood pixels
  • looking at images seem to be static floods in a bunch of places that are likely terrain shadow. as sun gets lower they will likely get bigger 
  • Fritz is having to go to a high no of SD to get a reasonable number of alerts. 
• Dan has been looking at GIBS / WV. 
  • terrain shadow were not as up to date as they should be. gave to Greg to update. 
    • only affects Hawaii
• Due to technical problems on Terra will not get as much data on the N. latitudes. 
  • on legacy system there are missing tiles on 70 N.
  • seems to be a difference between NRT3 & 4 for legacy files (not the LANCE).
  • MODIS went in to assembly anomaly on Oct 5: one of the superset went bad. and can't be used for storing onboard data. 
    • to avoid overwrite of data. reduce the day rate and increase the night portion. 
      • 46: 54 day:night ratio
      • Oct 7 - Nov 8: then field operations changed it as you get overlap at the poles so now alternative orbits. You will see something of a zigzag in the imager.
      • to recover the memory it would need to reboot the solid state recorder. too much of a risk.  
• Modify PGE 159
  • David has information from Sudipta
    • needs some end of the day water files – David to get this from Sanjeeb 
• Dan to contact WV / GIBS

•  Still waiting on OPS code for 3 day processing. Greg to follow up
• Do we need to wait for forward processing of C6.1 - No.
  • In Jan may run C6 and C6.1 on separate servers 
• Imagery should keep going to GIBS for test instance of WV.
  • Currently it is the only C6.1 product running on NRT4.
  • The CR imagery from nrt3 is also being sent to the test instance of GIBS.
  • Dan monitoring this - flag if there are any issues.
    • 721 CR - blue color set for surface water looks like snow and ice.

• Still Need to modify PGE 159 – flood detection to run in OPS (to run so fewer output files) and to use outputs from 155.  

  • This will not impact getting the 2 years of data for Dan. as the OPS version of PGE 155 will run with the existing  PGE 159.
  • David is writing the update to PGE 159 - This will be done after new year. This will need to be tested before running in OPS (3 day test) and then we can decide whether to re-run the 2 years worth of data. 

• LAADS production - looks like nothing is running. 
  • Greg to contact Curt to see what is happening
  • Sadashiva thinks Sanjeeb and Sudipta were perhaps confused about which version to run.
    • Seems like a communication error? Greg has asked for it to run. Questions come back.
    • Greg to ask Gang to run it cc'ing Curt
• Dan: product evaluation - waiting for run.
  • working on the user guide
• Flood product in Worldview
  • looks good. useful for pointing out areas where there are problems
• GeoTIFF
  • generate GeoTIFFs
  • there is a tool that works with HDF4 to convert to GeoTIFF
  • this could be done via processing in the PGE
  • need to look at the extra volume it would take to archive GeoTIFFs as well as HDF
• David Landis:  1.5 - 1.8 million flood pixels/records per year in the database. Lots of false positives. As for size - we can deal this as we go forward and think about designing the alerts and web displays. Sounds like we might need a different approach to the FIRMS approach. What is the requirement to view the data? Could use the database for the last month and if you want longer.

• Status of OPS production (Greg):
  • if OPS runs using NRT rule it generates more files it would if the standard files (land L3G) were used. The difference is a factor of 4 in the number of files created
    • some concern re: disk space
    • agreement will run with the current production rules to get Dan's test data done but this could be modified for other OPS running
  • running wrong version of PGE 155 (still failing and not processing all granules overlapping one tile). maybe need correct version in recipe or 
  • need to re-do PGE 159 to use the combined rather than 1 datasets (last granule that came in). may take a month or two.
  • Dan asked about the c-datasets that PGE 159 needs. Does it allow you to see separate granules? Yes its a 1 dimensional array that allows you to identify these.
  • Sadashiva: even in the current implementation the way the L2G works - it finds max  coverage observation and puts it in first layer then puts it in compact layer
  • Name conflict vs production rule issue. Greg to follow up with Sadashiva
  • Go ahead with test for validation data for now, so we can get back on schedule then look at how to update PGE for future exercise.
  • Could run new version of 155 as long as the output goes to another archive. 
• Product evaluation (Dan): almost complete.
  • working on the user guide
• How will the product be advertised?
  • communication all needs to be done via the LANCE website. 
  • Private vs public archive (via authorization)
  • Need to get permission from project scientist Hal Marring to determine if this can be archived as an applications product. Need to revisit this before Ed leaves.
  • Need to coordinate archive NRT forward processing. Should be routine.
  • Greg to talk to Neal about this and Diane to contact Ed re: permission from Hal Marring. 
• GIBS/Worldview
• No word yet from GIBS on whether the data is being ingested for testing. Diane to follow up

• Status of OPS production (Greg):
• No word yet from Kurt, who is needed to run the test. Has been somewhat MIA; Greg will check in with Gang to track him down and see where this is.
• Product evaluation (Dan):
• After more examination of differences, it is apparent that *most* differences are due to the larger number of looks available in the LANCE implementation, where swaths overlap (towards the poles). Which is letting in more cloud shadow false positives, esp in the 2-day product. A second order (after Nov 30 release) project will be to examine alternate compositing thresholds to see if these false positives can be reduced without also significantly reducing true water detections.
• Currently finalizing summary graphs illustrating differences.
• Next will develop User Guide.
• Will focus transitioning legacy system to new server only after User Guide and Evaluation Report are completed, assuming 617 Sys Admins will allow the existing server to continue operating a few weeks past the CentOS 6 EOL on Nov 30.
• Joe Nigro was able to find his raw database of specific dates for the flood and non-flood events used in the existing evaluation report; this will make it much less time consuming to re-evaluate those sites with the LANCE product.
• Current NRT production issue: current day products not reaching nrt site until end of day (Greg):
• Greg needs to work this with Neal
• Release schedule - Nov 30 still feasible? (Dan)
• possibly, if we can get OPS into production to generate the evaluation data.
• GIBS/Worldview
• No word yet from GIBS on whether the data is being ingested for testing.
• Database ingest (David)
• Continues, and data is looking good
• Alerts (Fritz)
• Initial attempt was generating ~5000 alerts per day, based on default parameters...working on ways to revise and filter.

22 October 

Greg, Dan, Sadashiva, David and Fritz

PGE Code

• all PGEs have the most recent version.
  • these are running in NRT but not yet in OPS
  • Greg wanted to re-run the 3 days in OPS to check all working
  • Actions
    • Sadashiva to check with Sudipta and LDOPE to confirm that they should check out the recent version so the 3 day test can be run in OPS 
    • Greg to make sure Curt and Gang run the 3-days
    • Greg to let Dan know when this is running
    • Dan to check the output from the 3-day test

• Greg still waiting to hear back from Navid on hardware. 
  • Greg to follow up

Evaluation

• Dan has developed some tools to look at the NRT product in more detail. Still looking but overall looks good. When will this assessment be complete? Should be next week.
  • the main issue is the one day product looks different due to the different cloud mask. And the far north looks wrong but aware of this and will remove the 70 N tiles. 
• Bhaskar is developing pages in the LAADS website to include one for flood. He was asking about what products will be public. Dan shared the table to clarify.
  • Greg sent a link of a similar page: https://ladsweb.modaps.eosdis.nasa.gov/missions-and-measurements/applications/nacp/
• Dan getting original study from Joe Nigh. 50 flood events and 50 non-flood events. 

Removing 70N tiles

• Dan has confirmed top row need to be removed - this should be done once we have a weeks worth of imagery in Worldview.

Imagery

• Files are being produced. 
  • Diane checking with GIBS: they will be setting up processing in UAT in the next couple of days. So sometime next week it should be viewable for the science team to start their review of it.

Database and flood alert program

• David has written a tool to load the data more easily
• Fritz has looked at 174,000 AOIs - which led to several thousand alerts
• Discussions with David ongoing

Next week: Diane on leave - meeting to go ahead anyway

PGE Code

• All PGEs were updated to NRT 4 and to OPS. 
• Limited name changes in 155 but 152 and 159 the names and metadata are updated.
• Fill values are fixed and consistent.
• ingested all mask files in to OPS
• asked Curt to re-run test days in OPS. once that works will run the year or two years required.
• Greg sent email to Navid

Evaluation 

• Dan is looking at the differences. No big surprises. Wants to look more - trying to quantify it in a meaningful way.
  • Nothing worrisome.
  • would like to look at the specific dates from OPS

Imagery 

• file are being produced

Removing 70N tiles

• Dan to confirm which ones need to be removed - 
  • top row
• will remove in NRT so imagery doesn't show up in GIBS, but not in OPS

Database

• too many rows to use like we do in FIRMS.
• Greg hasn't had time to think about alternatives
  • rolling update?
  • not sure how it fits in with what Fritz is doing with different thresholding.
  • only keeping flood pixels but sounds like we might need all pixels.
  • David to start looking at the flood alerts and how it might work. Fritz to send David what he has already done.

Fritz - running statistics for floods

• wants to run the stats on a chron job - once every 15 mins or so?
• New server - Dan will need SA

8th October 2020

Dan, Greg, David, Sadashiva, Fritz, Diane

PGE Code and Evaluation

Dan still working on the evaluation. Greg discovered terrain shadow files were not there for all tiles and that was a problem for the production rules.

• Dan had to populate empty files to provide to Greg get the code to run. A complete new list of mask files have been ingested to NRT4 and OPS systems. The NRT4 looks like it is processing them.

6 files over the course of the year were missing or incorrect in the 70 N row. That row is a mess and the files need to be updated. It is generally frozen in the winter so there is an argument that we don't need to run those in the winter.
action @ Dan to consider adding that as a caveat for the End User guide

Suggested solution

• remove the 70N from the tile scheme until March action for Greg
• in OPS run all tiles for the dates required for the evaluation
• need to talk to Maki about running a different tile scheme and not produce those images for GIBS

More on PGE Code

• Gang, Neal and Greg have been working on finalizing the production rules. Some issues still re: files not showing up until the end of the day. 
• action @ Greg to solve problem of why files are not showing up as soon as updated. and check to see the imagery will get processed when 
• File name changes requested by Dan. Greg has these and has made changes to the code. He needs to run through the tests to make sure they are okay. will go to production on Monday. 
• action @ Greg needs to notify Sudipta know about the file name changes. To do that today cc'ing Dan to make sure it is correct.  

Imagery 

• changing the file names. Need to get these finalized, then should be okay. Looks like next week before we get the first image tiles out next week.

Database

• managed to load 29 days worth of data (3-day flood layer). 146 million flood pixels for 30 days. that would be (x 12) 1.7 billion pixels for a year.
• purpose of the database: to render images, manage alerts, could be used for statistics (not currently done for FIRMS).
• Based on the information provided by David - the number of pixels will be too big to keep them all. maybe we need a rolling archive? TBD

Alerts

• Need to understand the logic for the alerts.
• Set up a prototype.
• Look at design (data files vs database)

Note: Ed has said we can proceed with the alerts. We don't have a mandate to do this from the UWG but they didn't object when Ed said this was the plan. However we do have a mandate to  get the flood product out so any work on the alerts is fine as long as it doesn't detract from getting the flood product into LANCE.

• Statistics that Fritz was running is complete. Looks at planet tiles that fit in planet-scope size areas.
  • Machine to machine alerts would be useful for the flood community
  • Fritz is going to be looking for new alerts in the data. 

Old Flood system

• Existing Server becomes obsolete at the end of November
• Push the data archive to NCCS. Most of the product data is on the MODAPS distribution server. 

Need to re-visit:

Plans for improving the terrain shadow and cloud shadow. 

1st October 2020

Dan, Greg, David, Sadashiva, Fritz, Diane

PGE Code

• Ed gave the go ahead to keep the code running on NRT4 (C61)
• Dan wrote some code to come up with statistics based on the products being produced in NRT - the results should be available in a few days.
• OPS code: Greg is in the process of updating the code to require the mask files. Should be done later today. Once done - he will run the 3 day and make sure they are being used correctly. Once done we can start on the larger test towards the evaluation.

Question from Dan: can we rename some of the HDF layer files? Yes - it is quite a bit of work.

• action @ Dan to send Greg the list of names that need to be changed. 
• action @ Greg to keep LDOPE in the loop as GIBS layers will change too

Imagery

• PGE is baselined and looking to get this in to operations on NRT4. Not seen anything coming out yet.
  waiting to hear from Neal - action @ Sadashiva to check the status of this

Database

• loading data in to the database. 20 days loaded of the 3-day
  • likely to get more terrain false positives in the winter
  • action @ David to let Dan know how many positive hits on a specific day over the past week so Dan can compare it with his stats – to give us an idea of what the database size will be

Flood hardware

• the hardware is there. There is a need to install mapserver if we are going to generate flood imagery using the dbase.
• need a website. no docker container yet that holds this. 
  • will be setting this up as part of the FIRMS - to docker website. Estimate 1 month. Asen and Otmar will work on this at the same time as FIRMS. 
• will need to put together a docker container for python. SA will not be responsible. Navid's group will likely be responsible for deployment. There is a formal process where approval comes from management (e.g. Robert, Bhaskar for MODAPS). For flood this could be Dan / Sadashiva

User Guide - looking at this.

• Sadashiva: user guide to cover only the products that are released to the public

Stats for the Alerts

8 scripts finished. 5 still running. When done will bring in new datasets and can start issuing warnings when exceed certain thresholds.

Are we still on track for November release? Likely

• biggest issue will be getting the operational code running for the evaluation.  
• action @Dan to send the years required for historical processing in OPS Greg

PGE Code 

OPS code: Greg has a ticket in to update the code to require the mask files 

Dan: quantifying the new vs old product over a couple of weeks.

• the apparent diff is the MOD09 being used for cloud shadow instead of MOD35.
• Then will need the data from OPS to do the full evaluation

Imagery - Sadashiva

• The code is on NRT 4. Neal was trying to figure the streams for GIBS. (one for test and one for OPS).
• need CR from NRT 4

Flood Database - David

• reset to add in the 3 day product only 1/10th of the data. Screenshots of the 12 days → movie.
• flood showing in N Canada and Siberia. huge improvement over what we had a month ago
• Next steps:
  • David to continue loading the data:
  • Get Navid to set up another database as they are on FIRMS2
  • Focus on figuring out the size
  • Greg checked the hardware monitoring system and they have some assigned. need to figure out how to get stuff on to it. prototype flood on FIRMS would need to move it across to new server. action for Greg to figure next steps

Stats code still running

• 13 scripts running one for each latitude band. 
• Breaking things in to the Planet size boxes Quite a few that don't meet the 80% cloud free requirement. Some 110 meet the requirement but had zero % flood coverage

User Guide - Dan working on this.

PGE code

Greg: No changes to the NRT code for 2 weeks. NRT is running well. There was an issue in OPS where they had forgotten to stage the mask files – Greg has been working with the OPS manager to help get them staged. The code runs without the mask files. Sadashiva suggested we make it a requirement (not optional). Greg agreed

@action for Greg to check with the OPS manager to see if he has staged the mask files.

@action to modify the production rules to make it a requirement to have the mask files in order to run.

Dan: Not looked at the NRT yet. Will do later today or tomorrow.

GIBS image product.

Sadashiva: NRT 3 & 4. currently running on NRT 4 (test), NRT3 is operational and visible. Flood map has been running in C6.1. C6.1 is still running in test. Can run every day on NRT4. If we want it visible to very user need on NRT3 but that is running C6.

If you baseline as C6 it can use C6 SR. So the question is – are we ready to go public.

NRT 3 be moved to C61 in October (polarization fix for Terra). Close to approving C61.

Keep it on NRT4 for now as an Alpha. From Matt - If MODAPS and GIBS added the flood product as an operational product, we could process it and just not add it to the public WV. But we would have a test WV instance that would let us see the imagery.

Flood Database

David is seeing some pixels that aren't flood. A lot are clustered along the edge of cloud. so there are cloud shadows that are not being removed. Is the MOD09 good enough? May need to screen out areas where we are not going to report alerts. This should be okay in the 3-day product.

@action for David

Next steps: reset to add in the 3 day product.
Give examples of size of data base for 1-day

May struggle with the size of the database. It is much bigger than the fire data. Dan suggested we could have a rolling archive. Need to think about the design.

Scripts for Alerts

Fritz: running scripts again.

User Guide - documentation - still to be done by Dan when time permits.

• This should also include FAQs and information for the Earthdata LANCE pages

10th September

PGE 

• Running in NRT with both A&T and set up so it can run with just one or the other.
• Ran 3 days in OPS looks like there is an issue with the tiles around border of ocean /land
  • probably as mask file not staged properly as it works fine in NRT
• Visually the data from NRT looks good. 
  • next step will be a statistical comparison
  • once OPS are running we can run the dates for the evaluation

Imagery

• No updates. we were waiting for it to be running in NRT
  • it has been running since the beginning of the week so action for Sadashiva to follow up and transfer GIBS imagery to NRT system.
  • PGE was ready and tested in house. hopefully can transition to NRT and then on to GIBS.

Flood Database

• Loading test data. Seems to be running fine. Noticed the problem of flood 'ringing' bodies of water.
  • visually it still looks like the whole world is flooded. 
  • Action for Otmar

User Guide - documentation - still to be done by Dan when time permits.

• This should also include FAQs and information for the Earthdata LANCE pages

3 September

PGE

NRT Production

• Yesterday production rules to allow code with just one instrument. It seemed to run okay.
• Algorithm code seems to be okay. No issues found since the pan-sharpened product.
• Now that Aqua is back - 

OPS 

• Fixed operation rules. Issue caused by additional PGE (11P - a night time PGE) that runs in OPS but not NRT.
• Some stuck pieces - something that operations needs to deal with. Hope to be finished today.
• Sanjeeb was making changes to handle issues he was dealing with in PGE 152. Preparing a release for that. 

Imagery

• as no products being made due to Aqua loss
• on OPS side same issue - PGE wasn't 

BRDF and Albedo MAIAC. won't come back on as multi-day products using both Aqua and Terra. Working to make combined products with whatever data is available. After 8 days everything will be stable and usable.

Flood Database

• program that loads the flood data has been re-written and runs well. 
• one day worth of data - August 22. except large number of flood pixels along the shore of any large body of water
• From 19 million to 2 million pixels when you go from 1 to 2 day product. 
• want to do more test

Alerts Prototyping activity

• Tiles and river basins to start with. Can we prototype both?
  • would need statistics for the river basins 
  • aim to do this manually / run the stats and see what the alert would look like 

Fritz

• server - end of life.
• checking for funds. not enough to cover a new server until new fiscal year.
• In Nov will need to update the operating system

Found another bug on the flood alerts. Need to start again. for statistics.

20 August

PGE code

• Greg fixed issue with false positives at the edges of some tiles. This was due to an error in band sharpening code (similar to PGE 100 written by Jeff Schmaltz). This was fixed in Jeff's code in 2016 but other code was not updated. Should be resolved now.

Standard Processing

• Code ran for 3 days. Some issues - probably in the production code.
  • Action@Greg to look at this

Flood imagery in GIBS

• Dan sent final color map scheme to Sudipta
• Sadashiva will check on progress. it was agreed it would be great to have a demonstration ready for the LANCE UWG but while desirable realistically there probably isn't enough time. 

Flood Database

• David got the ingest code working so it is ingesting in to the database.

• Otmar put together a prototype: https://firms2.modaps.eosdis.nasa.gov/floods/

  • comments / feedback welcome - to Diane

LANCE UWG

• should be able to show some imagery from Dan and from the prototype

Flood Statistics

• Fritz is running flood statistics for the archive, based on the Planet tiling grid (25 x 25km overlaid on the MODIS grid). Fritz has access to Planet imagery so is able to compare with the high resolution imagery.

Email alerts

• will we use the Planet tiles or watersheds or both? To be confirmed. Possibly both.
  • action @ Dan to confirm how the watershed statistics will be determined and how often they need to be updated. This should be automated.

Hardware for floods

• Greg to check with SAs

13 August 2020

PGE Code

-- Issue with terrain shadow mask not being applied is fixed. It was an issue with production rules not finding the correct file to load.

-- Dan has identified a possible issue with false positives at the edges of some tiles.

-- Dan has identified a possible issue with totally empty L2G files.

Standard Processing

-- we have loaded all the necessary information for standard production into ops, and will run a preliminary test of three days to make sure everything runs correctly.

-- Dan will identify a list of test dates to use for the evaluation test

Flood Alerts Database

-- David has been testing the database code, and it looks good. He has added several days of data and a reasonable number of data points are getting added.

-- Greg will see if Otmar can mock up something that can view the data from the database.

Flood Imagery to GIBS

-- Dan received feedback from Matt on color tables.

-- Dan fill forward color table information to Sudipta.

-- Image generation code is mostly complete, ready for final testing once final lookup table is integrated

6 August 2020

PGE Code

• Data looking good. Dan still checking it. 
• What would happen if we were missing data? 

• it would be a manual process to correct this. the production team are used to re-processing data.
• for the most part - the data are stable. sometimes there are maneuvers.
  
  
• there are differences in the way NRT vs OPS handles granules during maneuvers.
  • NRT 3 steps to check if quality is impacted. not perfect but suspect manuals will be removed as part of the pre-processing. (Automated process).
  • for the OPS system the maneuver data are processed off line and QA checked by LDOPE and a decision made as to whether it should be removed.  
• If the Flood team notices bad data is going through then they should let LDOPE know and they maybe able to fine tune it.
• The missing data issue Dan saw was related to files that overwrote each other. Neal and Greg have fixed this.
  
  

what should we call this product? 

• Alpha after Dan has looked at it. 

• Beta after the Evaluation report of 100 sites (same as existing product). 
  

• Release at the Beta Stage and get it into Worldview. 

• Beta stage timeline? Depends on how long it takes to set up the standard processing. 

• Beyond Beta - recurring flood over 10 years. updating the terrain shadow etc. Essentially a new / better product. 

Evaluation Process

• Dan's plan is evaluation rather than validation; it is mostly inter-comparison where neither is validated using truth data. 100 sites

Standard Processing

• Sadashiva has emailed Gang but not heard back yet. L2 should be straight forward. L3 more work.
  • They won't need to deal with challenges of NRT - can fall back to how it is implemented in OPS. 
  • Gang needs to assign someone. 
  • Action @ Sadashiva to contact Gang to see if we can get the evaulation done by the LANCE UWG on 9 September

GIBS imagery

• Action @ Dan to ping Matt to see if his suggestion is okay 

Flood Alerts Database

•  David got access to the dbase yesterday so will test program later today.
  • previous version was putting out the proper data in to a TXT file.
  • Once data are there:
    • start looking at displaying the map images like we do with FIRMS and start on alerts. There are still some issued with the alerting system. Fritz has been looking at.
    • Fritz has been working on the statistics by latitude bands. This will probably take a week. Based on the planet image size. Software can be changed to do it by 1 x 1 degree

User Guide - documentation - still to be done by Dan when time permits.

• This should also include FAQs and information for the Earthdata LANCE pages

Dan,Greg, David, Sadashiva

PGE Code

1. Something is going on in the northern tiles - the reflectance is low and detecting all as water. Greg thinks the real issue is in the input from MOD09 files. Greg is taking a look at this and will email Sudipta.
   There is a band at high lat that gets classified as water. it is obviously artificial. Maybe a quality flag needs to be turned off
2. Current production includes more tiles than the original product. Option to turn these off for the initial NRT processing. Once the standard processing has been run for some time (long enough to produce additional reference water) then these tiles could be added back in. If they are left in now without reference water mask they will show up as flood when they shouldn't.

Tile schemes list all tiles to be processed. so removing of tiles can be done here.  Sandeep and Neal can help with this.

Sadashiva discussed standard processing with Ed. 

• action for Sadashiva to send follow up email to Ed  and action for Diane to contact Karen to see if she has heard anything

GIBS imagery

• Dan sent email to Matt to check what he needs to provide to Sudipta. 
• action for Dan to provide Sudipta with the final color scheme

Flood Alerts Database

•  David was wondering if he was getting too many pixels for the tip of S. America
  • Greg checked code and thinks it looked reasonable
  • need to test by running code and populating the database

User Guide - documentation - still to be done by Dan when time permits.

• This should also include FAQs and information for the Earthdata LANCE pages

16 July 2020

PGEs

Greg sent Sudipta the issue about PGE 152 about the LPGE today. They are looking at it now. Sounds like the problem maybe to do w/ the deep ocean flag.

Sadashiva discussed standard processing with Ed. There are a couple of points that need to be considered.

1. MODAPS will do the necessary processing for the validation but before that we need to be sure the PGE is generating quality data for validation
   Dan - had planned to do a qualitative comparison between the new and current data. For Dan to do this we need the PGE issues finalized. Need issue with PGE 152 finalized and data checked by Dan
2. Come up with a plan for what we need to do the validation. How much data? Global vs specific sites.
   1. Dan - would not need global for the 50+ validation sites. That will enable the validation to be updated. If all okay then run 10 years of data to update the flood water product. Again could do this strategically. Do certain areas for 10 years then expand

Action @ Dan to come up with proposed plan for Ed and Sadashiva

GIBS imagery

• in progress. they have tested the PGE locally in LDOPE
• Action @ Dan to provide the LUT 

User Survey

• almost finalized.
• hopefully ready to send next week
• Dan is liaising with DFO

Flood Alerts Database

• program is pulling the data out from the images but seems to be pulling too many points. 
• Greg was going to review the code but didn't get to it this week.
• Fritz is doing some tests on alerts on the old system.

Action@ Greg to look at David's code

User Guide - documentation - still to be done by Dan when time permits.

• This should also include FAQs and information for the Earthdata LANCE pages

9 July 2020

PGE 159: small changes made

PGE 152: more significant changes made. some failures due to PERL code. Running since Tuesday. Data should be good. and PGE 159 should be using that data correctly.

• There was still a question over data values in the cloud mask
  • action for Dan to look at this and see if it needs updating
  • not showing today's data on the website. Neal needs to look at this when back from leave - data will appear on the website at the end of the day but it should be available as granules are processed.

Putting the code in to standard processing

• waiting to get the okay from Ed about what level of processing should be supported.
• will support what is needed for validation but maybe looking for something more.
• action for Sadashiva to send follow up email to Ed 

GIBS imagery

• LDOPE has confirmed with Dan that he recommends all three ( 1,2 and 3 day) products.
• Code being written.
• action for Dan to provide Sudipta with the final color scheme

User Survey

• DFO - to run the survey. Can't run this through NASA easily
• Small updates to be finalized by Diane and confirmed by Dan
• then Dan should run by Bob Brackenridge, Albert and Fritz.

Flood Alerts Database

•  David is working on SQL commands. Not hooked in to test table yet. 
  • want to check the right values will be populated (900K entries)
  • currently generating CSV files for testing
  • wondering if getting multiple listings for the same pixel. 
  • action for Greg to help check PDL code.

User Guide - documentation - still to be done by Dan when time permits.

• This should also include FAQs and information for the Earthdata LANCE pages.

1 July 2020

Discussed feedback from LAND MODIS - VIIRS meeting

• A question was asked:  Can Flood Map process generate annual land water mask product similar to the one that Mark Carroll is generating using the MODIS 16-day composite reflectance product?
• re: historic archive for flood (and for other orphaned products). This is a challenge as
  • resources to do the QA are greater than the resources just to archive the product.
  • ideally need a science team to take care of any changes in the instrument calibration.
  • larger issue of re-processing for different collections
• For short term solution: ESDIS funding. fine
• Long term solution: ROSES call.

Dan sent a document sent to Ed about the next steps. Waiting for feedback.

PGE 159:

Greg installed fixes which broke everything in production! The fixes were to do with the cloud and cloud shadow processing. The way it was originally done - it was mixing up pixels. some areas in the image where pixels got shifted. Greg found the error and fixed in in PGE 152. Right now the previous version of the code is running until the new version ready.

Putting the code in to standard processing

• waiting to get the okay from Ed. 
• once we get the okay there will be some work to be done to test the code that Greg has written
• Sadashiva to follow up on his

GIBS imagery

• no progress this week. Sadashiva will follow up.

Flood Alerts Database

• the database has been mostly done by David.
• Greg had given David the code to do the inserts. He had started to do this. not finished or tested

User Survey

• Diane waiting for okay from Karen/Robert - - action Diane to finalize and check okay to move ahead.
• Once we have this - run it by Bob Brackenridge and Fritz.

User Guide - documentation - still to be done by Dan when time permits.

• This should also include FAQs and information for the Earthdata LANCE pages. 

25 June 2020

• Dan has had a closer look at the data and is happy with the overall product.
  • still looking to confirm everything is okay. Action Dan - by next week should have finished.
• Greg did contact Gang to see about standard processing for historical data for validation (STIG). 
  • Action - Greg to follow up with Gang
    
    
• GIBS imagery: still waiting for the test. Not heard back yet. Sudipta is working with Maki
  
  
• Greg emailed Ed, Robert and Navid re: hardware for alerts
  • Navid suggested this should be developed for Kubernetes like deployment using the archive for data storage instead of relying on any local disk
    • No development environment for Kubernetes
    • Action - Greg to check where Navid is expecting the development to take place. perhaps on local machines but as there is no data on the local machines this would involve data having to be copied across. 
      
      
• DBASE table - David has been working on this and made good progress. Code used to take data from the images → the database
  • it will be a lot of data!
  • not connecting to anything yet 
  • prelim database on FIRMS2 for now
    
    
• User Survey - Dan and Diane are working on this. 
• User Guide - documentation - still to be done by Dan when time permits.
  • This should also include FAQs and information for the Earthdata LANCE pages. 

11 June 2020

• Update from Sadashiva
  • LDOPE now has global browses of L3 water composite, and two different cloud masks that Dan wanted to review for suspected differences.
  • LDOPE will soon start working on the GIBS image for the Flood map.
    
    
• Operational PGE –  Greg contacted Gang to have STIG integrate the PGE to MODAPS following different set of production rules for operational processing at MODAPS. Greg to follow up again
  
  
•  Dan has been writing proposals and unable to get to the flood work.
  • needs to check the output from PGE 159
  • draft strawman for flood alerts thresholds

• Otmar has been looking at the alerts.
  • tabled: waiting to understand how the threshold works
    
    
• User Guide - documentation - still to be done by Dan when time permits.

May 28 2020

PGE 159:

• Dan found a few issues with the logic. Won't be too big a deal to fix them. 
  • need to change where threshold is applied
  • no data not showing up - seems to be a threshold issue. Greg reviewing them. 
  • hopefully the changes will be in production on Monday. so Dan could take another look on Wednesday.
  • datasets look mostly there so we can proceed with imagery

Imagery 

• Greg still needs to talk to Maki
• GIBS team (correspondence) only need updated tiles - not whole days data each time
  • they want to modify one of the no data values. Dan to send info to Dan so it can be coded.

Question from Dan

• once you get the correction done - how is the imagery coded?
  • David will do this and stick it in with PGE 159 → image files
    
    • timeline for this? David has a program running that works. it is a bit slow and needs speeding up.
• Re: Imagery Greg said they still need to understand how they get to GIBS and how that interacts with other PGEs (another weeks worth of work). Maki and Sadashiva work with this so can help on this.  Sadashiva says once we have the product we agree on then they can augment a process to generate the GIBS image. Core code is there. Just a matter of adding correct information in to the PGE. Should be straight forward.
  • Sadashiva will aim to start working with LDOPE on this. 

• User Guide - documentation - still to be done by Dan when time permits.

May 21 2020

PGE 159: Looks good.

• Dan has one day downloaded and needs to look more closely at other days.
  • possibly showing more cloud shadow in Day 3 & 2. than it should be
• PGE was running on NRT4 which is currently down 
  
  
• Sadashiva has been making L3 and L2G images but stopped as code was being re-worked. they can start again with LDOPE
  • what images would you like to highlight to enable a better look at what is going on.
  • Sadashiva to work with Sudipta on this
  • Greg mentioned the dataset names have changed again. Dan to verify they are okay

GIBS imagery

• Matt has a prototype in a UAT instance of Worldview. Looks great although possibly a lot of cloud shadow
• need to determine colors of flood vs regular water. 

David was working on PNG or JPEG images from composite files for easier validation. Really close

Next meeting 28th via TEAMS

David, Greg, Sadashiva, Dan, Otmar, Diane

May 14th 2020

PGE 159: Dan reviewed the data in current production and found some issues in the code. Hope to have another release today or tomorrow which (if all goes well) would mean the 2 day product would be ready for Monday. 

GIBS imagery: Could give GIBS the old (ongoing) product, or maybe it would be better to wait for the new code? Dan to get in touch with some 1,2 & 3 day products. Greg is still not sure what format the GIBS folks want the data in despite corresponding with Maki. Dan thinks Paletted PNG.
Maki commented that it might be a lot of imagery to ship to GIBS. MODAPS send the whole world every time it is updated.
Could we ship just the changed files? Dan to ask Matt

David to work on 10 x 10 images which can then convert to 9x9 as expected by GIBS. 

Moving to standard production: wait until PGE 159 is working properly

Alerts:

1. data to the database
2. thresholding - need to determine the steps
   1. statistics thresholding which needs to be derived from standard production

Next steps:

1. finalize PGE 159 
2. move to standard processing environment to enable: 
   1. validation of the NRT product
   2. with 10 years of data it will be possible to generate a new reference water mask 
   3. once the new reference water mask is in place, 10 years of data should be re-processed to generate the statistics for the alerts. this isn't a huge volume of data but need to keep in mind that at MODAPS there is a lot of re-processing already going on.  
3. work on the alert system
   1. determine water shed region
   2. process for determining statistics
   3. backend process for alerts
   4. UI interface

Sadashiva, Greg, David, Fritz, Greg

May 7th 2020

Funding for Dan - seems to be sorted.

Flood alerts: Preliminary discussion. We don't want to alerts for false positives which are in the product. so if we know how many routinely occur in the area we can set the alert accordingly - with MOD44W rivers etc have moved plus terrain shadow. So need threshold based on running stats over time. How many false positives do you expect. Challenge to dev threshold for each area of interest. Could be 10 x 10 tiles initially. Ideally users to submit AOI but then we would need to develop stats based on this. We have 8 years of data but LANCE product will be different, so ideally we would process 10 years of data via LANCE so we can then generate new reference water.

More complex than FIRMS! If we have thresholds - could apply them to incoming data. open question - where can we generate the statistics. Could be done in IDL, python but not acceptable to MODAPS. The layers could be done monthly or yearly.

The threshold will change throughout the year as terrain shadow false positives will change between seasons. 

Possibly develop regions of interest. Need to give this more consideration.

PGE 159: Greg got code from David this morning. Hoping to get it in to production over the next day or two. Will take a few days to see the impact due to the sliding window. Sometime next week can review data. At that point it will be worth spending time to make sure it does what it is supposed to. Can then talk about what we need to do to run it in the operational system to accumulate statistics over the period of the mission. 

Imagery for GIBS: Greg has been in touch with Maki and Dan. A few more things to clear up but it shouldn't be a big deal to create the issue. It will be a lot of imagery and sending it back and forth to GIBS every 10 mins might be problematic. Currently Maki sends entire world as a set of tiles (600 every so often). It is a lot of data to process. We are at 250 tiles - can we just send updates vs just the latest imagery. This needs further discussion with Maki to see how it can be done without impacting other NRT imagery.

Sadashiva: global products. gibs process put its in to the 9x9 tiles. Matt doesn't do any choosing of the data / processing. It has to be done at NRT to convert approx 250 → 9x9 tiles. 

29th April 

PGE 159: running for 3 weeks. getting ready to add final masking stages where we check ref water masks and set flood flags.

• ingested all masks: part of archive set 1 (ancillary data) done this way so if masks are updated they can be added and kept track of.
• wrapper code for PGE is pretty much ready to go. not yet installed.
• Dan found issues with current product. turns out we weren't using the correct thresholds for 2-3 day counts. 
• found issues w/ some of the logic for the counting in the algorithm, so David, Dan and Greg have been clarifying and fixing these issues.
  • hopefully later this week a new version will be in production. then we should get the complete products with masks and flags

• One remaining issue: still don't have the hand masks converted to ENVI files. Dan was to work on that but ran out of money. Greg could try. 
  • issue with the HAND - there is just one file per tile but Dan would like to look at the threshold to generate the HAND mask and decide what threshold to apply. HAND is an index and in the past 30m has been used but Dan would like to take the opportunity to review this before it goes in to production.  It should only impact mountainous areas so not a huge concern but would like the chance to do that. 
  • Greg: can put logic in there with threshold as zero until have the file. Logic should be if file doesn't exist - it should continue without e.g. for flat areas.
  • Will want to run it without the HAND mask to enable Dan to cross compare with existing product. 

Way to handle different masks

• will go in to different archive sets. probably won't change much. discussed this w/ Neal - seems the best way 

GIBS Imagery

• Dan was unable to follow up on this as he ran out of funding for the month
  • SSAI needs a new sub-task to what was the old 6026. 

Funding

• Ed and Frtiz talked about budgets. Ed will talk to Keith / Ernie to see if Dan can start charging to this now. 
• Future? Possibly a MODIS / VIIRS combined product? 

Flood Alerts

• experiment with the alerts / next steps.
• thresholds need to be run for each user AOI
• would need approx 10 years of archive data to come up with the thresholds. Have the existing flood data base from 2012. 
  • depends what mask is used. 
    • improve terrain shadow mask to reduce false positives would change the statistics. so need an archive for current product. 
    • using Mark Carrolls land water mask
      • Dan mentioned they would prefer to derive one from this product so it is more consistent.
  • initially historic processing would be archived at MODAPS. could be delivered elsewhere (LP DAAC or Hydrology DAAC) 
  • Ed thinks we are at the stage where ESDIS seems okay with it

• Ed will let us know when he has talked to SSAI to let us know the plan for charging.

22nd April

Production of PGE 159. Greg has been working on getting the mask file in to the code. Following some back and forth with the CM and DB folks it looks like they have a workable solution.  The masks will be filed like regular files so they can be downloaded (binary and header envi files) by users. DBase are tables ready to go. Greg thinks the ingest will start today or tomorrow.

Once done the next step is for Greg to take the code David has given him (with ref water mask) and put that in to production. 

Dan: looked at the data and thinks like the wrong threshold was used on the 2 & 3 day products.

• Greg to look at these thresholds and see what is going on.

David: Terrain shadow - had some questions for Dan. Cleared up. It turns out the test file being used had no terrain shadow in! 

How to handle and mask and versions of the mask files: Greg said there is no easy solution but suggests the best way forward is to use a "recipe" - this is a description of inputs necessary to run PGEs. the recipes run the production rules to decide what inputs  will get used. so there could be a recipe with mask "on" and another with the mask "of". Once in the recipe it will be easy.  You can also run the recipes simultaneously should you want to compare outputs.  

• Dan asked how the different products would be identified. Different archives, and the metadata would list the input files. Since this is standard input data (just like any other in the production system) the input files could be referenced from the archive system (metadata field = input pointer). 
• Sadashiva confirmed that these alternative runs would be limited for validation / comparison.

GIBS imagery

• Dan has received a reply from Matt. needs to look at this in more detail.
• Greg will talk to Maki about image generation. She can look at what it will take to generate imagery.  It is likely to be a transparent layer with the data points on it. Need to make sure we have a "no data value" available (that can be turned on / off or made opaque).
• Greg to contact Maki

Consider LOE flood alerts.

• there are system issues to discuss:  would it run on FIRMS? a separate system? should it be run it like FIRMS, or we come up with a more modern approach? there are a lot of system questions that we haven't begun to think about.
  • add Otmar to next week's discussion if he is available to look at options.

Fritz needs to tag up with Dan to finalize costs to finish up costs for phase 1 including water mask, user guide etc That didn't include the alerts. The alerts were under the follow on work. Fritz needs to put more money on the task. 

Question on running validation data

• Once running this provisionally, we will need to run some historical data, so this will need to be tested and run on LAADS. 
  • if this is a short period this shouldn't be a problem but it will still need to be set up
    • it would take some time to set up (a couple of weeks). Greg has not put any effort in to getting it running there - but it should be similar to the process on NRT so he doesn't envisage any problems.
  • Sadashiva asked if it would run as in NRT or at the end of the day (as if in standard processing). This means they will run slightly differently so we will need to look at the differences and identify the cause of them.
    • end of the day 
    • is the PGE set up for that? It will have all end granules rather than as they come in. Greg thinks it will cope; it is designed to take any number of input files. if it sees water it notes it, if on a subsequent pixel there is no water the PGE still keeps the water in there. 
    • L3 process will see one file per tile. it is till requires L2 PGE to get water product and something to grid it. 
      • Sadashiva said gridding will be one file for entire day. Greg thinks this is okay.

Production of PGE 159

Looked at the data. from Greg's perspective it looks good. Accumulating data over the 3 days from both Terra and Aqua. Production seems to be stable (approx 200 files per day as expected). Neil and Greg were looking at the files chosen: 6 are over the ocean so after reviewing with Dan it has been decided to remove those, and a few others as recommended by Dan.

Dan - said the 1,2 &3 day all look quite similar. maybe showing up too much cloud shadow. Not sure why...

• Action Dan/Greg Need to dig to see what is happening.
• Non-observations file. not used. this is generated and used by the gridding algorithm but Greg is not sure what it is used for. Stripped images. It is not generated by the L2G nor used by the L3.

Yesterday David sent an update which includes the reference water mask processing. so the next step will be to add this in and run a release.

David still to add terrain shadow to code 

For follow up

HAND mask  Is there an on/off switch could be applied? how could the products be differentiated in the file name? 

• whether HAND mask was applied and what version should be added to the metadata to file. 
• there are standard metadata fields in the HDF

have we had an answer from Matt / GIBS

Greg will talk to Maki about image generation. She can look at what it will take to generate imagery. 

Consider LOE flood alerts

Production of PGE 159

Greg said 2 days of data have run well. That equates to approximately 250 files for days 98 and 99. So looks like the production rules are okay.

• next thing to check the data looks reasonable (same link as before) - action for Dan and Greg
• figure out how to get in process files to show on website (show in production if you know where to look) 

Dan has completed the Terrain Shadow grid.

• HAND - not yet done. validation needs be done first. before it is applied so there is no rush.

David - is close to getting the land/water mask added in to the code. He is still getting errors so will talk to Greg.

• once the land water mask is in the code the terrain shadow (and HAND) mask should be easy to add.

User guide & validation

• Dan has looked at the user guide and doesn't see any issues. 

re: validation:  the plan is to look at concurrent files and processing historical data for certain areas.

• Dan asked how processing the old files would be done?
• Sadashiva: suggests this should be done once the system has been running well for a short time.
  • have it run on the current system. do a thorough review of the data (can get help from LDOPE) including compare it with the existing products coming out of the existing system. then determine what historical data is required to enable the validation 
• Greg mentioned that they would also need to set up production rules as we don't have historical data in NRT. Rules for science processing are slightly different. The inputs would be science quality SR products. Should be a simpler process - run once a day as don't need to keep track of incoming files.
• need to come up with an approach: will have the whole days data so in terms of the product that comes out - it may be different to the NRT product and we will need to understand why it is different. If it is then we need to account for this as part of the validation
  • could run it up as a science test. there are dedicated test machines. historical mod09 data is already there. 
  • NRT vs Standard is done regularly as part of LANCE

HAND mask  Is there an on/off switch could be applied? how could the products be differentiated in the file name? 

• whether HAND mask was applied and what version should be added to the metadata to file. 
• there are standard metadata fields in the HDF

Next meeting 

have we had an answer from Matt / GIBS

Greg will talk to Maki about image generation. She can look at what it will take to generate imagery. 

Consider LOE flood alerts 

Sadashiva, Diane, Greg, David, Dan Fritz

2 APRIL 2020

Greg: production rule fixes seemed okay on 091 - got 250 tiles (expected), but day 092 only got 6 tiles. Looking for problems. 

Dan sent out his converted binary files for the reference water. others still to be done.

Action: David needs to look at the binary file. Greg to send link to Greg.

L2G gridded: 256 tiles in flood tile scheme. of those 250 are being processed consistently. need to see which 6 are missing. maybe in N. hemisphere. 

Ocean tiles are not being produced in Level3

David: got code for water mask - working for  3 day composite to product the flood. Need to compare to 1 day and 2 day and 3 day. 
he has the binary mask files - won't be able to read them in with the HDF reader. maybe PDL or Perl.

AOB:

Validation and the HAND mask. it occurred to Dan that it would be nice to get validation sites reprocessed and compare to make sure we are getting the same date. It would be useful to have a product without the HAND mask applied. Is there an on/off switch could be applied? how could the products be differentiated in the file name? or combined? or don't apply the HAND mask at the get go and then add it in once the validation has been done. 

• Greg would like to get it working correctly first. Then make changes.

Dan will hopefully have updates to the masks when improvements are made. Can we make sure the metadata for the files keeps track of which masks are applied - at some point. Action Greg / David to add this in.

NRT Flood Tag up: 25 March 2020

PGE 11: All files except for Aqua missing when doing production rules. Running now for a week (SR)

PGE 152 and 155: process granules and turn them in to tiles – seem ok

PGE 159: production rules were failing for flood PGE: update files were not being stored correctly. Output files are being produced but not staged to the NRT server. Greg working with Neal on this.  Then will be able to see if the issues Dan saw are resolved. On Greg’s local system the counts show up correctly.

• We agreed no data value from 0 to -99. Still remains to be done. But this doesn’t affect the flood detection itself.

 Action: for Greg to let Dan know when they are on NRT4

Gridding the masks

Dan decided he would do this. Dan has reference water will be done today. The others will be easier – should be this week.
David will then need to update the code to include these masks. David has been working to get the flood code ready to read the state of water map. Once the additional files are in he will add these.
Question: Can they sit on the system and be referenced? Greg thinks the masks will be an array. Since the grids will line up then just need to read in the mask array and overlay it on the composite data or whatever it is masking out. As for where the files are stored – Greg thinks the PGE wrapper code will take care of that. Will need to come up with a number for the different mask files. Greg will need to look at the notes. Either production rules or PGE wrapper. Read them in like how we deal with other masks.

L2G Data
Dan looked at L2G data: When looked at 2 dates (a week ago), the global grid – a lot more tiles being generated that we need (e.g. in the middle of the ocean) probably because they have a spec of land in them. We don’t need them. It is not a problem to keep them.

Level 3 data: there are no ancillary files for these additional tiles. Greg noticed this and is filtering them out and in L3 they will only producing files for the tiles that Dan currently gets. 2 tiles at the tip of South America weren’t generated for a couple of days. Greg to check the files. It could be because the land tile scheme switches as the poles go from spring to summer. Greg said that for some areas MOD09 has tiles for the north in the summer but not in the winter. Sadashiva also mentioned no tiles are produced if there is no data.

 Action: Greg to figure why they are missing.

User guides
Sadashiva sent examples to Dan for him to look at.

AOB
Dan mentioned that the current product – includes ocean map: 5-10km off shore. We can drop this. Fritz asked if that complicate would counting flood pixels? Dan said it shouldn’t alarms should be based on flood not on surface water. Greg said he thought ocean is being masked out as not data. Need to check this. Could it be due to the MOD09 product. Does this generate over oceans.

Action: Greg/Dan to check this

Frtiz: Once up and running will it be available for internal testing. Greg said the data are on NRT4.

How close are we to finishing up?

Greg – 2-4 weeks. Depending on how long it takes to get the masks done. It shouldn’t affect production too much. Talking about functional readiness only. 

Next meeting – Thursday 2^nd April at 11 AM

David, Dan, Greg, Sadashiva, Fritz, Diane

18 March

There have been some issues setting up the production system but it looks like PGE 11 has been fixed and installed. It will be a few days before we have a reasonable set of data. Up to now only 3 x tiles of Aqua data (so a lot missing). It also seems to be processing more tiles than we need.

Need to figure out how to update files throughout the day - currently as each new granule is added it updates the existing water composite file but this doesn't show until the end of the day - due to the way the production system keeps track of them. Neal Devine and Greg to figure out how to make this available as soon as it comes in rather than at the end of the day. So still some work to do.

It doesn't impact the work David is working on getting masks integrated in to the code.

Gridding the masks (hand, terrain shadow and land/water)

• David is discussing possible ideas with Greg. looks like GDAL command is best way forward.
  • once converted to MODIS grid, the terrain shadow needs to be composited. working on ideas.
  • could use Dan's help to come up with proper GDAL commands
  • Dan said the output should just be a 10x10 file 
  • For terrain shadow files these are computed once a month for Aqua and Terra? Ideally when applying the terrain shadow mask we should use the appropriate mask for that month.
    • when doing water count need to mask out observations using the terrain shadow mask. this is outlined in the updated document that Dan circulated. Dan said he could include IDL code for doing this if it would be helpful - yes please. Action@Dan to provide IDL code to David/in document.
  • Dan said the terrain shadow works okay but the problem is the terrain shadow is different each day as the MODIS tracks are different each day.
    • currently the process is compositing different orbits in to one 10 x 10 file. The outcome is the solar geometery will be different for some tiles.
    • in this new processing - you could do create a better terrain shadow for each. is it worth the improvement? don't know, Plus we are not funded to do this.
      • the right way would be to use the actual solar geometry. could be done....
  • The terrain shadow is approximate - it works reasonably well. 
  • David to read through the updated document and let Dan know if he has questions.
  • This information is useful for Greg to figure out how to get them in to the production system.
    • need to decide how to store them. various options: use perl to call them or use co-efficient files (like LUT) or put them directly in the database. or combine them in to one file for each month.
  • need geometry info to see how to convert them in to the right grid information - e.g. corner points and EPSG code for projection. Dan could help
• Naming convention - typical MODIS naming convention or flood naming convention? Sadashiva would prefer standard MODAPS tile format e.g HxxVxx
  • there is a translation that can convert to Lat Long so users can look for a tile of interest. Most search pages do this. It is not in LANCE yet but should be coming soon. (It wasn't done in the past as NRT hardware was limited. With  cuperneides there will be more servers running. Not done yet but coming. 
• GEOTiff conversion: LAADS has a way you search for a file you want. Tell it what conversion you want and then it sends an email saying where to get it.  Aim to have it as an output in GeoTIFF as users are used having the data this way.
  •  Take existing HDF files - stage them and make new products as GEOTIFF files. At this point you would only need to composite the final water layers for 1,2,3 

Changes to the flood map document:

• starting at next steps. modify modwater - done. updated tables. changes are update table 13
• clearer - that Terrain shadow needs to be applied early on. MST1
• code how it would be done in python

User guides. Sadashiva sent examples to Dan for him to look at.

March 11

David, Greg, Dan, Sadashiva, Fritz

The preliminary water composite product is now available at https://nrt4.modaps.eosdis.nasa.gov/archive/allData/61/MCDWD_L3_NRT/2020/.There is still some production rule debugging going on, but the outputs can be checked to see if they look reasonable. Note that this is not a full three days so the three day composite data will most likely look identical to the 2 day composite data, etc. We still need to add in the flood detection using the various masks. 

• The masks have been supplied but need to be regridded and algorithms updated.
• Are they running continuously? Greg said they are running but update files are not yet showing on the website - so some teething issues (file synchronization issue); production rules are still being debugged (Greg and Neal). the impact of this is that all processing for day 68 should be done but in production system 283 PGEs are waiting to run - they are waiting for ocean tiles which won't be coming so need to update production rules. 
• it looks like the files that are running are running correctly but not sure. need to confirm
• Dan briefly looked at one products: looks like there maybe an issue with false positives on the 2 day product. Need to look in more detail. ACTION DAN

Question re: naming convention. Currently using h-v format consistent with other MODAPS science products: 10 degree grid

• This could possibly be changed to the flood tile naming if that is more desirable for end users. Existing flood tile naming scheme uses lat/lon of upper left corner, eg 100E020N.
• Dan and users are used to current naming scheme, but may not be necessary to maintain. . Will be helpful to have conversion page or map. 
• One option to leave as is. There presumably will be a geotiff conversion at the end, and if desired, the lat/lon naming scheme could be reintroduced there (or that may just add more confusion and disconnect between the HDF and geotiff products).  

Action @Greg to check what the options are for this.

• if renaming the file would it cause problems on the database ingest? Gets this from the metadata in the file so this is probably okay. need to make sure the tile metadata works. 

GEOTIFF conversion

• are there standard ESDIS-provided tools? no
• an ongoing issue in building 32 HDF 5 files 
• various groups do have tools - plus modaps have their own tools (e.g. land have linux tools in local repository - written in C, build for linux systems. uses 20 year old technology). Unclear if that can be easily updated and made easily available and usable for Windows, Mac, and Linux end-users.
• Need to figure out recommended tool for GeoTIFF
• potential PGE can spit out GeoTIFF. This is on hold pending a decision by the UWG about how to move forward.
• GIBS GeoTIFF will not be a problem as MODAPS can use their existing code to convert as needed by GIBS.

David working on converting masks to grid: by calling GDAL from Perl (Dan suggests gdalwarp command). Need to give it the grid corners for each 10 deg tile and projection ref. Write a script to generate them based on the L2G. Projection presumably fixed so should be okay. export bounding coordinates from existing L2G products files. test result to ensure pixel boundaries overlap precisely with L2G/L3 products. Some software uses pixel centers, some uses pixel UL corner as the reference point – can be hard to know (or trust) so best to just try and test.  Existing L2G/L3 grid is extractable from the metadata in the L2G file, or presumably is available from within MODAPS?

Funding. Not enough for Dan to finish the transition !!!

• need to figure out validation cost. 
• include updating reference water or not
• can proceed on different levels of funding
• phase 1 = doesn't include user guide (for use by user to know how to use the product. can refer to some of the MODIS user guides. LDOPE can help with that)
• phase 2 = new mask, alerts, other improvements
• Action @ sadashiva  to send a couple of user guide examples to Dan
• Road map document is pretty close to an ATBD. Maybe needs updating.
• Dan currently has 11 hours per month to keep the existing system up and running, and any other tasks
• David G. has said it is pretty much the end of the funding. 
• Action @ Fritz to check situation with Bob and send an email to Diane. 
• Action @ Diane to flag this with Karen / Kevin 

Flood meeting 4 March

Dan, Greg, David, Sadashiva and Diane

on Friday David sent Greg an email regarding the compositing code which David finished.

• Greg has looked at it. To release PGE need to take David's code and put wrapper around it. And to add production rules (when to run).
  • Step 1 - done. Tested and works with all the different combinations of inputs that we get
  • step 2 - working on production rules today (what to run and when to run it). Should be ready to run tests later this afternoon
    • Greg will check it runs without errors and finds things it is looking for. Once validated. will release to production to NRT4 system - on files already there so everyone can look at it.
    • Still waiting for water mask file from Dan. That is not part of the algorithm code yet.
      • Can still run without to show compositing steps but won't show flood.
      • When get this global water file from Dan - can add it in.
      • Greg to move forward with the production anyway so he can test production rules by the end of the week.
        
        
• Dan - not working on this as he has not had time 
  • Ref water file, Hand Mask and Terrain shadow mask (date dependent).
  • Algorithm code doesn't look for them so production rules will need to be modified.
•  David to re- grid Hand Mask and Terrain shadow mask and ref water layer. Action @ Dan to provide these to David 
  • Action @ David to send them to Dan and Greg
    to check 
  • to Greg to update the code
• re: ref water file. Dan was hoping to re-do this. 
  •  Need to check with Fritz to see if there is funding for this.  Meanwhile carry on with old one.

• once David has done these.
  • Send to Greg. could be co-efficient or maybe a query.

• not yet gone in to production.
• needs to be tested.

PGE 155 Gridded data is available on NRT4:

• Dan asked about the fill value. this will be changed when PGE 159 is released. 

FEB 26 2020

Fritz

Dan

Greg

Flood Meeting Notes – Wed 26 Feb

Progress update from Greg

• Greg has started work on the water compositing PGE. He downloaded a bunch of the gridded data which looks good - although he didn't check this for geolocation.
• The data are on NRT4 if anyone else wants to look in the folder for C6.1. File names: MODWD_LG_NRT and MYDWD_LG_NRT.
• MODAPS is producing approx 3000 files a day. 

Dan asked if  this includes polar imagery that we don't need?

Greg: Yes. These are all 10x10 land tiles.

Dan brought up question of end-product pixel resolution and tile size.

• The current operational system uses a pixel size of 00219683655536 degrees, equaling to about 244.55 m at the equator, and resulting in output 10x10 tiles of about 4552x4552 pixels, but this varies dependent on the particular input data; pixels from different products for a given tile will not line up exactly with one another. (the product is not on a fixed global grid).
• The current/testing L2G products are output at a pixel size of 0.00208333 (~231.92 m at equator), resulting in 10x10 degree tiles of 4800x4800 dimension (on a fixed global grid).
• GIBS operates at a pixel size of 0.002197265625 (~244.60 m at equator), but in 9x9 degree tiles, resulting in tiles of 4096x4096.

Thus, if the current L2G resolution is maintained, the flood product will need to be resampled for display in GIBS/Worldview. This can be done, but resampling is never optimal, especially for a thematic product like this.

One alternative would be to instead generate the L2G product on the same grid as GIBS. The downside to this is that would require either: (1) the output product also be delivered in 9x9 tiles (this would allow those tiles to all be the same exact size of 4096x4096, and have integer tile degree boundaries); or (2) if the output product is kept at 10x10 degree tiles (more convenient for users to reference, and continues current product tradition), these output tiles would vary in size (most would be 4551x4551 pixels, but every 9^th would be 4552x4552), and would not have integer degree tile boundaries.

Greg said we would need to work through this. Can change PGE 155 to the required tile size. Greg couldn't recall how Sudipta and Sanjeeb came up with their information.

Action@Greg to figure out options for tile size and which would be best from MODAPS → GIBS perspective.

Action@Dan to consider what would be best for end users.

Gridded data are available (PGE 155).. Next step to get the algorithm using the data.

Progress update from David:

Need to be sure the algorithm is implemented properly. David and Greg have been trying to make sure the code matches road map document. Dan provided updated instructions.  David is happy with the new information from Dan and working on the code. He thinks this will be ready for  testing at the end of the week.

Action@ David to let Dan know when the test data are ready

Action @Dan to look at the outputs at the beginning of next week.

Dan

Greg

David

Landis

Actions from last week with updates:

1. Dan took a look at what it would take to create our own water mask. He looked at the existing land/water mask from Mark Carroll and Alfred. He conclude it would be preferable to create a land/water masks from our water product - as this would make it more consistent and reduce errors. 
   Update: Dan is out of funding so couldn't do this. 
   Frtiz and Dan have come up with a budget of work still to be done. This includes
   i) what they have already committed to do (but is not funded) and
   ii) what they would like to do if funding were made available. 
   This budget was sent by email to Bob Brackenridge who should contact NASA HQ
   
   
2. Re-gridding Hand layer and Terrain Shadow mask. 
   Update: Need to re-visit this. Could be something David does once compositing code is complete.
   
   
3. Dan to look at the updated PGE 153 (land water) output (L2 product) available on NRT4 and provide feedback to Sudipta - who is waiting for the green light to put this in production.
   Update: Dan found a few fill values that needed updating. These were corrected and PGE 152 went in to production today.
   PGE 155 (gridded product) also went in to production but some minor errors were found. Sanjeeb is fixing these and it should be ready today.
   
   Action: Dan, David and Greg all need to check the outputs from PGE 152 and 155
   
   
4. Compositing Code - David to continue working on this following a discussion on Table 13 from the Roadmap. 
1. It was suggested David use a config file for the values in case they change.
   Update: this is in progress. David and Greg to meet tomorrow (20 Feb) to go through Dan's roadmap.  They will send their plan (of proposed values) to Dan to check it is okay. Dan will be available if there are any questions.
   
   
5. Dan and Fritz to revisit the values listed in Table 13 of the road map with Bob Brackenridge and end users. It was suggested the 0 value that = insufficient data could be changed to 255 to be more consistent with other MODAPS products.
   Update: Dan agreed the values should be changed to 255. Just need to make sure users are aware. Ideally they will have new and old products for comparisson 
6. Diane to add Flood product to LANCE UWG agenda - need to make sure the following items are discussed: 

1. production of GeoTIFFS
2. reprocessing and archive of the historic data
3. implementation of an alert system
   Update: Discussion of Flood Product is on the UWG agenda

We also discussed Dan's funding - or lack of funding and the need to communicate status to NASA HQ. Diane to follow up with Ed re: his email to me and David.

NEXT MEETING  - Wednesday 29 Feb at 10 AM

11 Feb 2020

Dan Slayback

Greg Ederer

David Landis

Fritz Policelli

Land/water mask

• Dan would prefer to use our own land/water mask as it will be more consistent
• Looking at our product and see how easy it would be to spit out water reference mask - threshold could be adjusted
• Fritz agrees but not funded. Dan is hoping it won't take to much effort....  
• Alternative is to use Mark Carroll's product.  This could cause issues. This is what we are currently doing and it does end in some erroneous flood detections.
• Action for Dan to look at this some more

Re-gridding 

• Dan has not done that yet.

PGE 153

• Code that creates the L2 MODWATER is in production on NRT4. Greg notified everyone where to get this from
• Greg thinks it looks good but he has not validated it.
• Sudipta ran the L2G (PGE 155/156)on one of the outputs from Greg. Greg looked at results - seems good. Waiting for feedback from Dan. Waiting on this before it goes in to production
• ACTION - Dan to take a look and provide feedback to Sudipta. 

Composite Code

• going well. putting together counting layers that keeps track of the number of layers - need to talk with Dan about roadmap information. having trouble figuring it out. There are some intermediate working layers that could be excluded.
• Looking at latest version of Road map (20200206) - table 13.
  • David questioned the values. Dan to consider changing 0 (insufficient data) to 255. This would make it more consistent with the other MODAPS products. 
  • In table Dan refers to "name" an Abbrev - this could be changed to be more consistent with MODAPS. Greg says you have up to 64 characters so it is ok. you can have a name and description in an HDF file. Name is what shows up in a program like HDF look – so that is the important one. David updated some of the names to make it clear (e.g by adding 250m) in HDF look. There is also an attribute button in HDF look. 

ACTION @Dan to consider values in Table 13 of table and discuss with Bob Brackenridge and Alfred for end user perspective.

ACTION@ DAVID to continue as is in Table 13 suggestion from Greg to use config file (.DAT) that sets values. so if you need to adjust values it can be done in the config file and you don't have update the code. 

Fritz said WFP uses these values so they would need to change their code. 

Question from Dan. Will it possible to create GeoTIFF derived files from HDF? 
this would be very useful for end users. 

Greg said that would be the next step - dealing with the imagery part. 

• initial thoughts replicate what we have in FIRMS. FIRMS generates various files 

4th February

Dan Slayback

David Landis

Greg Ederer

Sadashiva 

PGE 152 MODWATER Product

Greg: On Friday was all set to go with ESTD names in dbase, code integrated with new names etc. Ran a test file for Sudipta, Sanjee and Dan. The file is complete in that it has all metadata, layers, correct names BUT cloud layer only has 2 bits - missing one bit. So they can still process as it is. (David reminded him that the cloud layer was missing one of the bits so need to be modified before releasing the PGE).

David updating code - a day or two to get 2 bit. once done this can do the next step which is to propogate the changes in cloud/cloud shadow through to the water composite product.

Sudipta says they want to keep the original L2G name which is fine.

Re: Flood / water mask

- currently using MOD 44W (now 10 years old). Dan to look at what Alfred did based on Mark's product- which would be through to 2015. It would need to be re-gridded

• can't use Marks product as is. it is conservative so work needs to be done. plus it is less straight forward
• could produce your own using existing flood / water product for last 10 years. how long ?
• Quickest to look at what Alfred has done and use that.

Terrain Layer and Hand Shadow

Terrain layer and hand shadow – need to be included. These are static layers that need to be re-gridded to L2G - David can do this. David to send to David.

There are intermediate flood products - Greg needs to verify this that everyone is on the same page.

Other

Greg - had to run tiles for Dan. Special request for a user who needed old tile. Done. Started running first 3 columns of tiles in LAADs. Finishing this/next week

Dan Slayback

Greg Ederer

Sadashiva

Dave Landis

Diane Davies

Greg gave an update:

• ESTD updates have been done and available for PGE developers to obtain as needed.
• They are working on PGE 152 - water detection PGE (MODWATER). Still going back and forth with Dan on fields. It already includes new product names and uses David's most recent code with cloud bit included as new layer. So cloud mask and cloud shadow mask will be included. Once sorted - will be posted for ingest in to NRT4
• Greg will organize an update of PGE 156 L2G - can see how they work together for final testing for changes David has made in L3 products.

PGE 152 should be ready early next week. 

→ Dan to take a look at a sample file of PGE 152 now and when ready.

→ Sadashiva to also take a look at a sample file of PGE 152 so he can update the L2G code (PGE 155).

MODWATER PRODUCT Now includes

1. water
2. water ratio
3. cloud shadow 
4. cloud (clear / cloudy and mixed - need to make sure these are all brought through)

*David to confirm these values.

PGE 155: L2G should be done later next week.

PGE 152: includes the additional data sets from David. L2G will need to change a bit to accommodate those. Greg will be in touch with Sudipta and Sanjeeb to make those changes.

Samples provide for L2G are with Dan - he needs to take a look at the format of the operational product to help  figure out water / land mask

Sadashiva to coordinate with LDOPE to reflect changes in L2 and integrate with L2G.

• one sample file from Greg will be enough to update the L2G
• Greg can get a sample file to Sadashiva by tomorrow. but developer is on vacation so it won't be integrated in to the L2G until next week.

Next for David: needs to propagate changes through to water composite products. then change count. 

Dan - had a question on archiving. Can we archive the product?

Sadashiva - NRT is different to standard so if we want to archive we may need to consider how to do this.  Sadashiva to talk to Ed / Robert.

Greg said implementation of 152 could be run in LAADS as well. if they decide to reprocess this historic data it would be possible.

if improvements in geo-location come with standard that would be good to reprocess it in standard quality as this would make it more useful for long-term trends. Log Out

13 January 2020

David Landis

Greg Ederer

Dan Slayback

Fritz Policelli

Sadashiva Devadiga

David reworked the water code to add in the cloud layer extracted from MOD09 so in same format but additional layer with a 0 or 1 cloud mask. 

it is interpolated to 250m (like the cloud shadow). David has compared it to the original MOD09 image and it looks good.

So Greg to test that before it is approved 

Still need to change water layer value:  to get rid of 10,11 and 12 from the water image. need to look at code as Dave didn't write that code.

Data - did detect water

No data - didn't detect water

Currently swath data. Did some tests with gridded data put in 100 for all data in the swath but not in the grid. This is interim code. Need to make sure handle no data in L2G.  

Sadashiva - do you have sample L2G data?

L2 Water code running on NRT - not yet released but being run. Greg not seen it - should be there.

Regarding what Neale has accomplished. Even for L3 progress, can ask for L2G to be run so you have some samples to work with.

Greg to deliver revised PGE then can run this week on L2 Swath

Make sure L2G gets these.

Production rules for L3 are written by Neal. Not been tested yet.

Sadashiva: are we using a static land water mask? which one?

Dan - now using MOD44W from 10 years ago. 

Using C6 data they have made annual MOD44W for up to 2015 and then some concern after that. Working with mark Carroll to update these. Maybe interested in using the latest land water mask. 

Alfred Hubbard looked at this with the older data. Dan would ideally take last 3 years and combine that to make a mask, or use their own product to generate an updated land water mask.

Need land water mask in approx 3-4 weeks.

Greg will get code water code in to PGE and released on NRT 4 and science test for a week  then Dan can take a look. Steps to get it done.

Once that is done. Connect it with the L2G PGE and then we will have the output from both phases. Then we can take L2G data and use it with David's new flood algorithm and compositing code to make sure it looks good. Not yet tested it from data in the production code.

Will need land water mask to finish flood algorithm and compositing code. 

GREG owes Dan an email about L2G gridding. Greg needs to look at the code for this. Sadashiva says PGE is already there. integrating it to the existing L2G version now is possible - this could be done by next week. Sadashiva to do this. Then it can be updated with the new code next week.

Will need some revisions - there are at least two different ways to do this.  

Question from Fritz: some faulty data from MODIS.... it was a science test issue. when they started testing C61 the code that was running the current flood process was using data from C6 and C61 - the program had a bug and was writing 0 in the file. It was in the NRT system. Not in the LAADS system - never run as SQ so as far as he is concerned no data archived with that false data.

mid Nov to new year.

So the issue is that they produce flood product from NRT so they archived that. If it is off 2 x orbits in a row it is very obvious. So Dan needs to find days and tiles re-run.

Greg could re-run for the full 3 months in LAADS. 

8 January 2020

Diane

Dan

Greg

David

David and Dan have exchanged emails on the logic of the code - following questions from Dan, David will amend the MODWATER code.

• Dan to send David an updated document outlining in more detail how the code should work - specifically focusing extracting from MOD09 cloud and water pixels (counts of the latter are needed for thresholding), and removing unnecessary layers.
• As the MODWATER code is already running in production Greg needs to look at how easily this can be updated. Usually if the code is in production there are a number of steps that need to be gone through to update code. As this code is not yet being used (and there are not ATBDs) the process should be straightforward. 
  
  
• The status on the Level 2G composition needs to be clarified. Diane to contact Sadashiva get an update. 
  • He said "We have the L2G process ready for integration. Late last year we agreed on a production rule for L2G and L3, that required some implementation at MODAPS at system level (production DB). I believe Neal was working on it or has already finished it. I wasn’t able to fully grasp his last email on this and so we asked Gang to have STIG move forward integrating the L2G PGE based on what Neal had in place to make that happen."
• Dan to provide water mask – still pending

4 December 2019

Greg Ederer

Dan Slayback

David Landis

Neal has started to work on production rules. 

• Greg needs to connect with Sadashiva re: tiled L2G product. Who is putting together the PGE for this? They said they are done.
• David / Greg have inputs. 
  
  • @Greg to find out who will write the PGE wrapper
• Second process will feed in to David's code. Can't test it as we don't have the tiled L2G code. He can simulate it but we need to figure out if there are difference.
• David - has code pulls in all of the new MODIS water products in a day and pulls in the composite water product from the day before. And it pulls apart product from yesterday to create composites for next day. If composite exists today, if it contains yesterday and day before. Take composite files for the day before so using 1 + 2 day so making 2 + 3 composite files for today. This includes information on whether clouds are present (but not the number of times - if making todays 2 day product need to know the number of valid pixels. how should this be stored? separate layer like a book keeping layer?) Need a count of cloudy / no data/ bad values. This helps understand where do we not have enough to know. For each day - book keeping layer would sum value valid observations. 3 layers 
  • 0 land
  • 1 water 
  • 11 cloud
  • 12 cloud bright pixel
  • 13 cloud shadow
  • 100 for no data i.e. outside of swath  
• individual water files have a 1 value for water. these get overlaid last. so have an array that pulls out each value for each new product. Flood water overwrites land if there is a flood.
• action @David Landis to write up logic and send it to Dan.
• aim to have book keeping layer in final file.

What is left: still not generating flood product

We have the water composite. But there is a separate process that compares water product with standard reference to get final flood product.

• aim was to update what we have been using (MOD44W). single layer - in time at that point. This is being generated by Mark Carroll? Should / could be updated each year.
• Can't just use the previous year as that won't be available reliably.
• Global layer to be provided by Dan
  • will it be on the MODIS grid?
• Where should comparisson be done? pull it in to the code being worked on now.

Also need TERRAIN SHADOW mask and HAND SHADOW mask. These will need to be pulled in and may change the value from flood to not flood.

• label the shadow? or make it no data? if it is shadow it is not useful data.

NOTES from Sadashiva

Attendees

In person: Robert Wolfe, Sadashiva Devadiga, Gregory Ederer, Dan Slayback, Sudipta Sarkar, Zhousen Wang

By Phone: Diane Davies, David Landis

Summary:

-          The NRT DNB radiance (VNP46A1) and the Flood Map process will include two PGEs – first PGE that will generate the L2G product followed by the second that will generate the L3 tiled product. The end product from the second PGE is delivered to public and is used to make GIBS images displayed on worldview.

-          The first PGE (L2G tiled process) is to be scheduled and run for every input L2 granule (5 minutes for MODIS or 6 minutes for VIIRS) generating one file per tile intersecting the granule. The PGE will generate new file(s) for the tile(s) and not update the existing file generated by an earlier run of the PGE. So there could be multiple versions of the L2G file for a tile at any moment.

-          This will require a new file naming convention for the L2G gridded  product. Proposed plan is to use granule time in addition to the data day and tile id.   

-          Second process will run at interval of granule time for the tiles that have new L2G files since the last run of the PGE generating a new L3 file for the tile. Similar to the L2G process, the L3 process will create a new file for the tile at every run of the PGE. So there will be multiple files for the same tile and so will need to follow the same file naming convention used for the L2G files. Rule could be to use the current granule id field from the L2G file used as input in the L3 process.

Action Items:

-          Neal will have to come up with necessary updates to database to support the processing of the PGE and new filenaming convention.

-          L2G process for MOD_WATER has been delivered. L2G for DNB will be same as the OPS: - integration can proceed only after Neal has finished his updates.

-          L3 process – Flood map team is working on their PGE. The operational version of the L3 DNB PGE (VNP46A1) cannot be used for NRT since this is an incremental assimilation of observations. A new NRT variation of the PGE is needed.

Dan Slayback

David Landis

Greg Ederer

Diane Davies

David - Greg and David made good progress and got through various problems in the code.

About to start tweaking to produce the first 3:2:1 Day composites so they have a library of them to generate the final products.

However, Dan said This would be different to what is being generated. Need to read MODWater every day - this makes things simpler.

Code will be updated to use water product from all three days.

Next steps: 

• David and Greg will get prototype products generated. just at the point where starting to put this together.
• within a week should have prototype products coming out that Dan can look at. 
• Greg asked what Dan uses to define standard water: MOD44W - single static layer. Needs to be updated but should remain a static layer e.g. generated from last 5 years?
  • action @Gregto look at MOD44W.
  • action @Dan to check his notes on what he did.
• Greg to work on PGE production rules with Neal and Sadashiva
  • talking with Neal so have narrowed this down.
  • todays info will simplify this
• Dan's asked how the process would be organized it so granules processed as soon as possible. If time stamping these, is there value in holding on to older version once newer version has been generated? Do we overwrite the old with the new? or save the old as it takes up some more space. Agreed we should overwrite the older file

Greg will be working with Neal on production rules: take as many inputs as you give it and create tile as output. 

Robert talking about every 3rd orbit. others talking about sessions. as soon as granule becomes available. So need to work on production rules or use separate task. RR task - in order to update the fire data. Either way - would work. Probably opt to stick with PGE. Modify code so PGE processes granules as often as they comes in.

Dan Slayback

David Landis

Greg Ederer

Fritz Pollicelli

Diane Davies

Flood Notes - edited by Dan and Greg

Ryan Boller

Karen Michael

Dan Slayback

Fritz Pollicelli

Minnie Wong

Greg Ederer

Otmar Olsina

Matt Cechini

• Dan provided an overview of existing products – these can be found at:  https://floodmap.modaps.eosdis.nasa.gov/
  • presenting 1,2 and 3 day products without it being confusing.
  • Consider adding all 3 as toggle-able options when layer is loaded in Worldview. In the existing application there is also a 14 day product which was to enable users to see flood in recent past without having to go to calendar. This is probably not required in Worldview. could drop 1 day of historical reprocessing.
  • native resolution is ~250m (product is in geographic (lat/lon))
    
  • NRT or just historical data? Hopefully both. TBD - there is interest in back processing the archive as this would be useful for users. Robert Wolfe suggested it be considered for a standard product - this would require a proposal.
    
• Points to consider when visualizing the flood product:
• would need to let users know that it is an experimental product

• Output options:
  • 0- no/missing/saturated data or clouds
  • 1 – no water detected
    
  • 2 - water detected, which overlaps with ref water (eg, normal water)
    
  • 3* – flood
• transparent PNGs with a layer where there is water. Matt asked if they would you be able to bake in different RGB values. Currently there are 4 values.
• There is a plan to update 3 to distinguish between recurring (seasonal) flood and non-seasonal flooding.
• If it was a GeoTIFF you would be able to see the values
• Is there anything to visualize as a vector?
  • Dan doesn't think it adds much to the vector, and actually reduces information provided (4-5 raster values are difficult to efficiently vectorize into a user-friendly package). But some users may want SHP files? Still talking about it with Bob Brackenridge.
  • Ryan mentioned a planned Worldview / GIBS feature that will enable users to toggle transparent values but this is currently scheduled for mid (?) 2020
    

• Status of product: still in progress. Code being written for second PGE – in terms of getting imagery to GIBS, it will follow standard procedure for other MODAPS products
  • Timeline: few months out but could get sample imagery prior to that
    
    
• Fritz asked how would users access the flood imagery - from a webpage? straight to WV?
  • possibly from hazards and disasters page?
  • There could still be a flood webpage
  • In the Worldview Flood category it was suggested to put flood product at the top and add other layers as supporting products. It was agreed this could also be useful for other layers (e.g. various options for active fire).This may require updates to the WV code to support.
    
• There was a question on the archive / download: it is an experimental product would it still be stored at MODAPS. Hopefully yes but this and the ongoing QA and support of the product need to be addressed.
• If GIBS / Worldview gets access to the RTC SAR product (from Sentinel 1) via ASF (which may be done on a disaster basis), this could be shown in conjunction with the flood product. 
  • ESA has some restrictions on RTC products so there may be some issues with this.
    
    
• From a GIBS perspective
  • Need to nail down RGB values in imagery (action @Dan Slayback)
  • Get sample geoTIFF and PNG (action @Dan Slayback)
  • Then the color map will need to be sorted.
  • so for now color all values apart from no-water 
• Once this is done and the product is ready, then the formal process can happen
• toggle hidden values - should be ready by summer 2020: enable users to toggle between cloud cover values
• LANCE MODIS will need to finalize output to GIBS. 
• LANCE – MODIS would ship 3 different layers to GIBS (one for each flood product, 1-day, 2-day, 3-day) but GIBS would bundle them in to one
• LANCE-MODIS package all 3 layers in HDF for end users - and then users can separate out imagery 

·       NEXT STEPS

• Dan to send sample of current products as RGBA GEOTiff or palleted PNG (with metadata) to Matt. 
  • Note the current flood products are10 x 10-degree tiles which is not what the final output will be. Matt says that is not a problem for preliminary viewing.
• imager values now are 1- 4 but not colored so needs to send RGB files.
• They need to be RGBA images. Dan needs to add extra value for seasonal flood.

David Landis

Fritz Policelli

Dan Slayback

Greg Ederer

Diane Davies

David Landis

Fritz Policelli

Dan Slayback

Greg Ederer

Diane Davies