NRT Flood Meeting notes

• Old NRT flood system was broken when the SA made an update. Dan has just spent 2 weeks fixing issues. Can we avoid this happening (or at least minimize the impacts) in future?
• Dan is still unsure how to proceed with his code in python. Which version? Will he be allowed to use python libraries? that are not included in RedHat?
• Dan needs put his code in GITHUB by the end of this week so the others can look at it and figure out how to move forward.
  
  • Greg and Gang will look at the code with a view to determining how it can best be set up so that it won't break when there are system updates.
  • Greg to deliver action plan on how to move forward.
  • Can Dan use outside python libraries? The SA won't maintain these. Greg said it is the same with the HDF libraries: the CM librarian does configuration management and makes sure they are up to date and compiled. Perhaps something similar can be set up for the flood mapping
• Frtiz and Dan have been in touch with the Disaster Portal folks. They are waiting on code from Dan

• Update on Disaster Portal: Fritz saw Jessica Seepersand but she was too busy to talk. They have requested Dan’s code for the web map services. Dan needs to provide this.
• Update from Dan
  • First module done. Waiting to hear back from MODAPS on how to deploy it. Impression that they are not familiar with using Python in a production environment. There was a suggestion that they use system python but Dan thinks that would be an issue. Greg said the issue is supporting external libraries and packages. SA in Building 32 do not want to deploy anything that is not part of the standard operating system but only as far as it comes from Red Hat or whoever the distributor is. So anyone that needs custom modules will require installation. Want users that want them to build and maintain them. But currently no there is no repository /deployment system. (There is an old fashioned system in place for Centos 5 but they are not porting it to Centos 7. So stuck in in between place!. It will be resolved. There will be facilities to do this (Navid and Kurt) but still have processes to sort out (Gate keeping / CM / documentation etc. Processes need to be transitioned from old PGE system). Greg thinks a couple of months until everyone is happy. But test environment should be sooner.
  • Greg to talk to Navid to see what the plan is.
  • Dan said it is better to wait for until this is resolved. – no rush to get in to production, as old system is working, so. Greg thinks the flood code might be the test case.
  • Dan said that to be clear - Python Conda system doesn’t need root anything.
  • Greg asked if Dan Dan has access to Git Lab? Navid can add him in. This is the configuration management system that will let Dan get familiar with the environment MODAPS will be deployed from. Put code in. track tickets etc. Should be used as a wedge to push Navid along. Greg to send Dan information.
  • Dan: to do short names.
  • Dan: can still work on compositing assuming python coding will be available. 
• Diane mentioned the GIBS summit coming up on 2-3rd October. Asked Dan for a straw man of how we would like to see the data represented for the summit, to make sure this is in the work plan for the coming 12-18 months.

Progress on Phase 1: Granule Module

•  Dan said that the coding is progressing well the code for the first intermediate MWD product should be ready next week. The next stage will be compositing all the raster files.
• Dan checked with Gang and Ed that:
  • the MODAPS code will monitor when new granule is available and check the validity of the granule files
  • the memory use was within acceptable limits
  • MODAPS will call code when granules are ready
  • Python 3.5 is preferable to the older version currently being used. (Ed to check with SAs and will get back to Dan if there is an issue).
  • Greg's code, that Dan got a year ago is still OK. No significant updates were made that should be tested. Greg said they are looking at the offset again for GEOLOCO but it is OK for Dan to proceed with what he has.
• Dan asked about logging code: are there recommendations on standards / content? Gang to send these to Dan.
  
  

• Product naming

  • Asad has sent Dan naming conventions

  • Intermediate products not to be distributed; don’t need to follow the same naming requirements. Only the official products.

• Dan will provide code to Gang / Greg who will add a PGE wrapper and run the standard science tests, put it in an archive for validation. Then need to determine how to store / add to a database. This may take a couple of months.

  
  

• Providing flood alerts

  • Greg asked about the outputs with a view to setting up a flood alert system.
     
• Diane has been looking at requirements from the flood workshop and feedback from users Fritz contacted:
  • users want a one stop shop for flood products
    • Suggest adding the imagery in Wordview and having a feed that can be ingested in to a devoted flood product viewer.
    • Get updates on what is happening with the Dartmouth Flood Observatory and their bid to have a "flood portal" and the NASA disasters portal being developed with ESRI" that David Green is funding.
  • Users often say products are not easy to understand: having historical data may help? and good explanations of what the data shows and doesn't show.
  • Users want automated polygon generation / product delivery
  • Users want to be able to provide feedback

Notes for Meeting on 17 July 2017

Notes for Meeting on 26 June 2017

• Flood product is currently run using IDL and ENVI and it uses 10 x 10 degree tiles (MOD09, MOD35 and MOD06) as input and generates MODIS Water Detection (MWD) and MODIS Cloud and Shadow (MCS) data files in same format.
• Dan will modify code to use granule data as input so the MWD and MCS can be processed as soon as the data are available
• Dan will assess whether the to use IDL / ENVI or GDAL / Geo Tools
  • no licenses are required for the GDAL option but if it is going to take longer, Ed is willing to buy another ENVI license
  • Dan to provide feedback the week of 10th July
• Dan will then deliver his code to Gang (by 17 July)
• Gang and LAADs team will take the code and wrap it in to PGEs for NRT processing

Phase 2: Mapping and Compositing module

This module creates products on a global 250m grid

• Who:  PI staff can prototype, but TBD who will build and operate this component (LANCE or PI staff).  Schedule: Start after stage 1 complete.

Estimated completion: TBD

• Generate final distribution format files (geotiff, shapefile, KMZ, graphics) from the master product and publish to web, ftp, GIBS (if possible), etc.
• Diane to review information Fritz collected from user survey and to develop additional questions (as appropriate) and contact users to develop end user requirements.