So many great things to work on next, help us understand where you'd like us to start by voting below. You can also edit the page and add a row to the table if we're missing your top need. Voting will only be available for a short period of time, so please vote now!
If there is something that you feel is important but not listed, you may edit the page and add that to the table.
34 Comments
Lindsey Harriman
Re: bulk download - not only to improve user experience, but DAAC experience as well with regards to the removal of the 2k granule limit.
Lindsey Harriman
Can the feedback on the timeline be shared before we vote on that particular feature (please)?
user-c2913
From usability testing at both LP DAAC and NSIDC, the general consensus was that the timeline had useful features, but users did not immediately understand how to use the timeline or what was available to them. Once timeline functionality was demonstrated, every user said that they found it useful and would use it in the future.
So our initial thought is to improve the affordance (intuitiveness) in some way. Another easy option is to add a timeline "help" function. We don't have any hard examples of what we would actually do, but we do have data showing users don't know how to use it, but find it useful once shown.
Lindsey Harriman
I'd be interested (any others?) in breaking out just the ability to hide the timeline. Unless people are still actively asking for the timeline to be improved (I don't have any use cases), maybe any additional work should be tabled for a later date?
Amy Steiker
Lindsey Harriman I'd be interested in adding a feature to hide the timeline, as well as improve intuitiveness, but these are lower priority to us compared to the other features. I could also see hiding the timeline as something related to the display size issue (less congestion on the page).
Lindsey Harriman
Agree on lower priority!
Jennifer Wei
I'd like to combine both bulk download and granules capability. In fact, most users would like to bulk download with subsetted data within their region of interest and time frame.
user-c2913
I think we should keep the features separated in terms of voting. Any sort of "this is more valuable if combined with feature ___" should be noted during the call.
Amy Steiker
Is an improved download manager (for native and/or "transformed" data) included in the Bulk Download feature? If not, I recommend adding this as a new row.
user-c2913
Yes. The idea for an improved bulk download experience is some new feature that isn't just a script. We've had some preliminary discussions about what could/should happen, but obviously no designs or work in earnest yet. But that's the general idea.
Christopher Lynnes
It could still conceivably be a script, but it would be more effective and useful than the current, simple script that is generated.
Amy Steiker
There is a distinction as far as script-user persona and GUI-user persona. Not all of our users know how to use a script and scripts tend to be platform-specific. The GUI user needs to be considered in this feature.
Christopher Lynnes
Poor choice of words on my part: I meant that it could still conceivably include scripts. (Scripts are particularly useful when you want the data to go to a machine that is not the one you are running your browser on.) Scripts might also be plural: one for Windows and one for Mac / Linux. We don't have to decide that in this voting process though.
Amy Steiker
Okay, then I may suggest adding a data processing feature for bulk downloads, including user improvements to processing time expectations and order management.
Christopher Lynnes
That is on the radar, but more in the End-to-End Services Architecture and SDPS Services, which is where most of the work would have to be. (EDSC would just query CMR or SDPS–not sure which right now–for an expected time.)
However, our initial discussions suggest that this is pretty hard, probably at least a good year to get right. Also, a lot of it might become unnecessary for services that can be converted from asynchronous bulk processing to synchronous one-by-one processing. Users would be able to see how long the first ones take, and would be able to start getting serviced data immediately, so a lot of the angst about "where's my order???" is alleviated.
So, this leads to the question of whether we should even try this in the current EOSDIS architecture, or start folding this into Cumulus Services architecture?
Lindsey Harriman
Before we create a new feature, should we see what solutions already exist? And identify all the issues/qualms with the current situation? Will DAACs be involved with that process?
Christopher Lynnes
I would characterize synchronous processing as an existing solution used at other DAACs, just not applied to SDPS services yet. It is essentiallly how OPeNDAP and some of the non-SDPS DAACs RESTful services work there. But yes, we can look at additional solutions as well, like using S3 as an order distribution mechanism. We will also likely do some performance testing and instrumentation of the services so we know what their performance characteristics are.
Also, yes, other DAACs will be involved in that process, likely including some of the non-SDPS DAACs as well going forward. It might actually be an appropriate follow-on activity to the UFG on data transformations, but focused more on the architectural aspect.
Lindsey Harriman
I guess I was referring more to the "download manager" element/Mark's comment on "new feature." DAAC2Disk already exists for bulk download and then whatever is going on with Earthdata Drive.
LP has found the current script to be effective, it just lacked documentation to indicate how it should be used - and then also the pause issue. We documented that this summer and then Jeanne mentioned that on the phone.
One of the Reverb features we had requested to be rolled into Earthdata Search was the text file of download links - so you didn't have to use a browser feature (DownthemAll in Firefox doesn't work with the new version btw?) or a script to get your data - or make your own text file.
Additionally, bulk download of data that has gone through processing (e.g. HEG) has caused issues at some DAACs, so the user-facing messaging could be looked at as well in addition to Amy Steiker's comment on order management.
Daine Wright
I just tested DownloadThemAll! against Earthdata Search and ORNL DAAC's order deliveries. It worked for me. What problems is it giving you? Which 'new version' of which thing?
We've added that as a download mechanism for our orders, so I don't want to get surprised if it breaks.
Lindsey Harriman
The latest version of Firefox - "Quantum": https://blog.mozilla.org/blog/2017/11/14/introducing-firefox-quantum/
No support for Quantum: https://addons.mozilla.org/en-US/firefox/addon/downthemall/
More info: https://www.howtogeek.com/333230/why-firefox-had-to-kill-your-favorite-extension/
Daine Wright
Thanks! I've updated Firefox on one of my machines to experience the issue for myself. Looks like it's time to update some pages.
Lindsey Harriman
Great! We are looking into to doing the same
Amy Steiker
Yep, we're doing the same as well. DownThemAll is still listed on EDSC after clicking on the "Direct Download" option. If you end up pointing to alternatives that are supported in Quantum, I'd be interested to know what those are! For now we'll add a disclaimer.
Amy Steiker
From an operations standpoint, depending on the transformation taking place, client-side HTTPS timeout could be a concern. A queuing mechanism would also need to be in place on the EDSC side to alleviate bottlenecks on the DAAC end. Even if this were implemented, further improvements can (and should) be made to processing transparency and order status communication. Yes, a user watching their granules process in real-time could help, but that won't help set user expectations prior to order submission, or even during order submission if large orders are queued behind others.
Christopher Lynnes
Yes, all valid concerns. Note that doing this in the cloud expands our solution space significantly.
Lindsey Harriman
Can we add an item for natural language/ranking search? I know we provided a lot of comments this summer and would like to see where those are on the game plan.
Christopher Lynnes
Relevancy Ranking is in the CMR task. Some progress has been made, but we think we need lots more automatable tests to move further forward.
However, we could add an EDSC item to allow the user to sort data collections on something other than Relevance, like Amazon does. Could sort on Popularity, Collection End Date, ...
Is that of potential interest? (It's non-trivial because of the way that EDSC pages through its results with CMR, but nothing is impossible.)
Lindsey Harriman
This summer we sent preferred rankings for how results would get returned for particular keywords/search terms/products - did those ever go into work?
Christopher Lynnes
All the rankings of the form "Should see X above Y for query Z" are implemented in an automated test framework for relevancy ranking, which has been used to guide the improvement in relevancy ranking.
We have not yet added "X should be in the top N for query Z" or "X should not be in the top N for query Z". (I forget which category most of yours went into.) The test framework is up on github.
What do people think, should we add the "Top N" set of tests?
Amy Steiker
Yes, NSIDC would be interested in this (maybe not considered high priority, but definitely on the "nice to have" side).
Lindsey Harriman
LP as well - at least be able to get the newer versions ranked above the older versions.
Daine Wright
ORNL DAAC would appreciate this, but our main concern for sorting is the default sort order. That is being address with the CMR task.
Daine Wright
Does CMR currently have a way to rank popularity?
Christopher Lynnes
Yes, it uses the number of users that downloaded a given data collection over the most recent N months. (I think N might be 6.) That information is exported from EMS on a regular basis.