|
|
| Meeting 16 January 2019 |
|---|
David Landis Dan Slayback Greg Ederer Gang Ye Diane Davies | - With the shutdown. David was not allowed on to GSFC and not able to access his computer. No VPN set up
- he was able to go in yesterday and get some files, so should be able to carry on.
- Greg said he can work at SSAI Aerospace building. They have connections that don't require VPN set up
- away next week in Florida but after that David should be able to start testing again.
PROGRESS since last meeting - Testing of PDS commands to replace the IDL commands like morphclose and some others
- there are a bunch of Perl equivalents but the documentation is poor
- with Greg's help (a shell of test code), David has been testing small arrays of data
- some perl code enables you to do more with less code e.g. you can resize images and flag so no zeros on the borders – these means some of the original IDL code is not required. David has been streamlining the code.
- still figuring out different sizes of the various products and inputs: 250m vs 1km. figuring out at what point to they need to be re-gridded e.g. generating the cloud shadow. the initial cloud is 1km. should he reprocess to 250m first? or leave it at 1km will that create a different product?
- if you don't resize image products before hand you are saving processing time but will the product be as good? testing needed.
- looks like all files that go in to cloud shadow - original are are 1km apart from CR data which is being loaded to get the dimension.
- one reason for the longer processing method would be if it enabled a better cloud mask by thresholding reflectance 250m data (down the road)
- run some tests to see if it is obvious which way to go.
- original code was for gridded data. now swath data:
- so cloud mask goes with swath
- Algorithm now takes gridded data for SR, cloud, cloud properties – not swath.
- when writing code David should assume he will get gridded inputs.
- will be working on gridded swath imagery so it will be larger than 10 x 10 box
- Currently set up so the output for given day is produced on the grid for Terra SR file for that day. so for Aqua and files for previous day will have to be re-gridded to match the Terra SR grid OR we can take some other 250m grid. Greg: code allows you to do either, so that option is available.
- Dan asking if Greg could run some test data to map out test granules at native vs fixed grid
- action @ Dan to send a tile / AOI and Greg will order the data
- Question re: funding and the shutdown.
- Greg said the HBS project was reviewed and approved by congress but it was still being reviewed by GSFC and they had not approved spending.
- letter saying we are OK through Feb 1st. Not clear if they will extend it again.
- process that needs to be happen for it to be expended again.
- Next meeting 20 Feb 2019
|
| Meeting for 7 November 2018 |
David Landis Dan Slayback Fritz PollicelliDavid Landis Greg Ederer Gang Ye Diane Davies
| - Almost done converting code - cloud mask. taking flood code using it as a shell and using it to covert idl → perl
- (cloud layer is from MOD35 = mask)
- Working on data structures
- Talking to greg about how data arrays are handled (rather than the perl format) - the standard would
- David is lookin up perl data language (PDL) options on line. Functions that read in the HDF data - read it in a PDL language.
- He is looking at the official PDL website. Not ventured out to look at other sites.
- Hoping for a few months. Depends on how quickly he can get the dilation code working (filling in the holes). At each place you have a cloud get height from cloud top temperature. So need complete arrays and it is patchy. Need to smooth it out to make is spatially complete. Erosion and dilation are Dan's attempt to expand the cloud top temperature. is a simpler way to do this? looking at patching functions in a 2d array - nearest neighbor averaging for non-bad pixels. some handle bad data values and some don't.
- Issue is no-one knows how the functions work and so it is trial and error. Documentation is not good so it is a slow process.
- If python was allowed this would be much quicker however for now the SA will not allow this. Currently the MODAPS machines only use code written in C, Perl or Fortran.
- This maybe an issue for LANCE moving forward if all science code has to be re-written.
- It will be a few months until this is finished.
Meet again in 2 weeks. Need to prepare an update for LANCE UWG. |
|
|
| Notes for meeting on 26 September |
|---|
Dan Slayback Fritz Pollicelli David Landis Greg Ederer Gang Ye Diane Davies | - David converting IDL code to Perl. Good headway. some snags in trying to find the proper function for some of the more sophisticated IDL routines. (congrid, morph, dilate, quadrat)
- Working with Greg yesterday and some questions for Dan.
- may be some work arounds. trying to fill holes in all layers / to project the cloud.
- several routines in perl that do similar task
- Move to testing specific routines to see if they do similar things – perl equivalent code. Still more code conversion.
- Next steps: to get IDL routines mimicked in Perl. or writing simplified version that can do the hole filling. then get it on to the server to test the code – running it through the interpreters.
Next meeting 24 October 2018 |
| Notes for meeting on 29 August 2018 |
Dan Slayback Fritz Pollicelli David Landis Greg Ederer Gang Ye Diane Davies
| - Dan met up with David an hour before the meeting to explain the background
- David has been studying perl tutorials - and looking at code and substituting cloud shadow for water detection
- Dan is open to David / Greg coming up with more efficient ways to run the code
- Steps
- David to make write the code for the cloud shadow detection
- Gregory Ederer can provide the information needed to run the code.
- Greg has a test framework set up already that was used to test the water detection algorithm. Can be modified to test cloud detection algorithm. this will use pieces of production system to run the test.
- Dan Slayback will need to validate the output
- If happy with output – then they will run a science test. The code will be run in as if in production code but only visible internally. then it moves in to production
- Dan requested that David document the the code so it can be modified in the future. The cloud shadow algorithm could do with being refined as the input data not great (it is 1km data and needs to be smoothed).
- David thought in a month he might have the cloud shadow part complete (eg step 1).
- David should ask if he has any questions or suggestions on how the code could be improved.
- Next meeting will be on 26th September 2018
|
| Notes for meeting on 21 August 2018 |
Dan Slayback Fritz Pollicelli Greg Ederer Diane Davies Gang Ye | Brief meeting. David Landis has been hired to work on transitioning the code to Pearl. Dan to drop by and see him in Building 33 G316 Dan and Greg to try and set up weekly meetings with David to touch base on any issues that might arise.
Plan to meet next Wednesday 29th if David is free. |
| Notes for meeting on 19th March 2018 |
|---|
Dan Slayback Fritz Pollicelli Ed Masuoka Diane Davies | - Current Status: Step 1 is complete. The initial swath based water product is running as a PGE and Dan has checked the out put product and is happy with it.
Next steps - Greg is going to amend the PGE to include a cloud shadow mask. Dan and Greg are discussion whether to use the existing cloud shadow mask (likely) or whether to use a different one.
- Dan needs to provide Greg with the code for the existing cloud shadow mask that uses MOD35.
- the current code Dan uses has IDL operators. this will need to be re-written.
- Diane and Dan to meet with Greg this week to discuss next steps this.
- Dan to provide an outline for code to composite and tile the initial product. Dan to work with Greg who will write the code.
- Terrain shadow filter (monthly product computed as part of this process) is to be applied to the tiled product
- HAND (Height Above Nearest Drainage) filter, which identifies areas where flood water should not be present, will also be applied to the tiled product.
|
|
|
| Notes for meeting on 22 February 2018 |
Dan Slayback Greg Ederer Diane Davies | - Greg amended PGE code and has completed step 1. Dan happy with the product
- Discussion on how to implement the cloud mask.
- looking at options
- this should be done as part of the PGE running in step 1
- Diane to contact Eric Vermote to see if there was an improved cloud shadow mask. He said
- "There is a cloud cloud shadow mask in the MOD09 product (at 1km resolution), at this resolution it is very difficult to evaluate cloud shadow mask and I am not sure about the performance.
Cloud shadow masks are currently being evaluated for Landsat 8 and Sentinel 2 data (using an operator derived validation mask) we can probably gain a lot of experience from this and then transition the “best” algorithm to MODIS the best way we can. That will not be right away unfortunately but more like in a year timeframe, As far as terrain shadow it is an interesting problem quite feasible if you have a good Digital Terrain Model but could be computationally expensive because my guess is that you have to do it at a finer spatial resolution than what your actual resolution but that can be studied too once again with Landsat 8/Sentinel 2 data.
|
|
|
