Contents
Introduction
This page contains instructions about how to set up a new Request for Comment (RFC) review. It also contains child pages that are templates that, when copied, provide the wiki pages needed to track the RFC.The ESCO role in processing an RFC is described in the Process for Earth Science Data Systems Standards and References document. Also refer to the material in Instructions to RFC Authors, a description of the process as viewed from the document authors' point of view.
Scope
This page covers only RFC document reviews.
It does not cover RFCs that are evaluated as ASCII File Formats as described in ASCII File Format Guidelines for Earth Science Data.
It does not cover ESCO-PUB documents (they are not usually reviewed).
Previous RFCs
It's always helpful to look at previous RFC review materials, particularly those that are in the same category as this new RFC. Look at the Standards and Practices page and the ESCO Document List to find relevant RFCs. Then look in the ESCO Review and Publication Archives on the wiki to see if there are helpful materials there.
RFC Requirements
Upon initial submission, the document might not meet all these requirements. ESCO staff should work with authors to help them understand the requirements and how to meet them. It is possible that after working with the authors it becomes clear that the document won't meet the requirements. In that case, ESCO should reject the submission (gently!).
Table Enhancer for Confluence
Table Enhancer for Confluence does not have a valid license.
| S | CS | TN | Requirement |
|---|---|---|---|
| Motivation must be clear and relevant to ESDS | |||
| Document must be well-written | |||
Sufficiently well-documented to be technically implementable | |||
At least two representative implementations | |||
Evidence of significant operational experience (In the form of Points of Contact who are using the spec/convention operationally) | |||
Format specifications should be published in open literature and freely available | |||
No patent, copyright or other intellectual property encumbrances on the format or its specification | |||
Besides the commercial entity that developed and/or maintains the format, there should be | |||
at least two (2) independent implementations that read the format | |||
at least one (1) independent implementation that writes the format |
S: Specification or Convention
CS: Commercial Specification or Convention
TN: Tech Note
RFC Development
Generally carried out by non-ESCO individuals or teams with ESCO assistance. Authors should contact ESCO as early in this process as possible to increase chances of success. ESCO will help authors understand the process, how to format the document, other requirements such as completeness, evidence of implementation and operational experience. ESCO should also try to determine whether the proposed material is likely to be approved at the end of the process. Some material may not be suitable for a community review. In that case, ESCO could consider working having the authors submit an ESCO-PUB document instead (see the ESCO Publications section of the ESCO Document List for examples).
Sometimes ESCO will receive fully-formed RFC documents. This is rare if the authors have not had any prior submissions (but it has happened) and less rare for experienced authors. In that case, use this section as a reminder of things that should be checked and discussed with the submitters.
Sometimes ESCO writes the RFC, as was the case with the GeoTIFF File Format specification. In that case, ESCO has to adhere to the same process, and it's probably doubly important to make sure the document is well-written.
Table Enhancer for Confluence
Table Enhancer for Confluence does not have a valid license.
Potential submitters are encouraged to contact ESCO staff at: esco-staff@lists.nasa.gov | |
Authors are recommended to read this material before they start writing:
It's also helpful if they become familiar with the process so they know what to expect:
ESCO should make sure the authors know they should use one of our document templates.
ESDS RFC Template: Generally, the ESDS RFC Template should be used (there are links to various forms of the template on the Instructions to RFC Authors page). COMET Template: The COMET Template is only for material that will be registered with the COMET system. It's not hard to convert an ESDS RFC formatted document to a COMET formatted document but it's better not to have to do that. If at all possible, confer with the NASA ESCO lead about the proposed document to decide whether the document will need to be submitted into the COMET system. The COMET template is available on the Templates and Instructions page. | |
ESCO helps Determine the Document Type: Standard, Convention, Suggested Practice, Lessons Learned, Technical Note, ... | |
ESCO helps Determine the Resource Type (Topics): Software, Specification, Process, Technical Note (If the document doesn't fit one of these, more can be added but there should be a good reason!) | |
ESCO helps set-up communication vehicle (emails, telecons, etc.) that includes the authors and designated ESCO staff members | |
Submitters determine who should be listed as Author(s) and who should be listed as Contributors (if any)
| |
For standards and conventions, authors should focus on providing as many implementations of the standard, specification, or process as possible and to list these in a document known as Evidence of Use or Evidence of Implementation (some RFCs embed this in the RFC, e.g., GeoTIFF). The ESCO will use these references to target reviewers for their views on the operational readiness of the proposed material. |
RFC Submission and Intake
When the authors are ready, the RFC is submitted to ESCO for processing.
Document Naming and Numbering
All versions of RFC documents prior to publication are 0.x
Some authors will use their own document naming and numbering convention during the editing phases.
Versions sent out for public review should be named "ESDS-RFC-nnnv0.x" or "423-ESCO-nnnv0.x"
The first published version of an approved document is version 1.0. The document name of this version does not contain the version number. COMET will use its own naming convention for versions.
Subsequent versions of RFC documents that are created to fix editorial problems are "v1.x". If an RFC has to get bumped to "2.x" because of non-editorial changes, then ESCO has to decide whether there has to be a new public review (the standards process document says it does).
Initial Screening
The initial screening process involves designating a member of ESCO to be the RFC Editor. The RFC Editor is the main point of contact between ESCO and the authors and also serves as the chair of the Technical Working Group (TWG).
Before setting up the review infrastructure, ESCO has to decide whether the document meets all the requirements and whether the material should be processed as an RFC and whether the material should be processed as a different document type than initially proposed by the authors. There are times when authors submit an RFC as a proposed standard or convention, but ESCO feels that it should be re-written as a Technical Note or Suggested Practice. This can happen if the material is not strong enough (not enough use within NASA being a primary reason) to be a standard or convention. (If the authors worked with ESCO to create the document, hopefully this is not the case.)
If the material is strong but the document itself is not well-formulated, then ESCO can work with the authors to correct that.
Table Enhancer for Confluence
Table Enhancer for Confluence does not have a valid license.
Assign an ESCO Staff member to be the RFC Editor to act as the liaison between ESCO, the RFC authors and the TWG. | |
Assure that the RFC meets requirements, see Instructions to RFC Authors: https://earthdata.nasa.gov/standards/instructions-to-rfc-authors | |
Decide whether to accept / reject / or ask for revisions | |
| If RFC requires revisions | |
RFC Editor coordinates with RFC Author till it's ok | |
RFC Editor and ESCO Staff decide when to accept RFC or whether to reject | |
Assign RFC unique document number (i.e. ESDS-RFC-nnn) RFC numbers are assigned consecutively. If the document is going to be posted to COMET, the document number usually becomes 423-ESCO-nnn where nnn is the next available RFC number. In some cases, the COMET number does not follow the 423-ESCO-nnn pattern. |
Create a Digital Object Identifier (DOI)
Updated 04/18/22 - this is VERY early in the process to create a DOI and, based on discussions within ESCO, it was decided that waiting to generate a DOI until the RFC is further through the community review process and near approval would be more efficient.
Follow the process in DOIs for ESCO Documents to create a DOI for the document. Most ESCO documents will be assigned DOIs.
Initial Review Set Up
Once ESCO knows there is a new RFC about to be submitted (this can happen in the RFC Development phase or in the RFC Submission and Intake phase), a set of wiki pages is needed to track the review process.
To set up the wiki tracking pages, copy this page, its attachments and its child pages. During the copy process you can rename the pages to include the RFC number.
Note: RFC numbers are assigned consecutively.
After the copy operation, the new pages will be in the Work in Progress area and should look like this:
| Page title | Page nickname |
|---|---|
| ESDS-RFC-nnn Review Tracking Page | Tracking Page |
ESDS-RFC-nnn Review - <TITLE OF RFC> | Public Review Page |
ESDS-RFC-nnn TWG Work Area | TWG Page |
Table Enhancer for Confluence
Table Enhancer for Confluence does not have a valid license.
Using the Confluence "Copy" function, bring up the page copying dialog box.
Select both "Include attached files and images" and "Include child pages", then click Next. Make the Parent Page be "Work in Progress" | |
Fill in the fields to rename the pages. In this example, the pages are going to be used to track RFC-042
Replace the "RFC" in the title with, e.g., "RFC-042", then click Copy. If it's going to be a COMET document, then you can replace "ESDS-RFC" with the COMET number, e.g. "423-ESCO-042". If you don't know yet which it will be, go with RFC, not COMET. | |
| You can now begin using the new tracking pages in the Work in Progress area. |
Finish Setting up Wiki pages
Table Enhancer for Confluence
Table Enhancer for Confluence does not have a valid license.
If there will be non-ESCO people on the TWG Request creation of a Wiki group named esco-twg-nnn from confluence admins. | |
Move the Public Review Page to be a child of the ESCO Reviews page. | |
Make sure the permissions on the Public Review Page are set so that only members of the ESCO and esco_review groups can view/edit. | |
If there was a special wiki group for this TWG, add the esco-twg-nnn group to be able to view/edit Public Review Page and the TWG Page (Confluence permissions may prevent non-esco members from being able to edit these pages. If it's necessary to allow this, adding them to the esco_review group will work but will also give full access to the entire ESCO Private space.) | |
| Attach the submitted copy of the RFC to the TWG Page. You can just drag and drop the document into the Attachments area. |
At this point, the Tracking Page remains in the Work in Progress area and the Public Review Page and the TWG Page are in the ESCO Reviews area.
Recruit Technical Working Group (TWG)
There is a description of what a TWG does in the ESCO Technical Working Group Guide.
If ESCO members have enough expertise about the topic of the RFC, then the TWG can consist of ESCO members.
The RFC Editor chairs the TWG (this is not spelled out in the process but has generally been the case in practice).
The TWG should use the TWG Page to keep lists of potential reviewers, formulate review questions, etc.
Table Enhancer for Confluence
Table Enhancer for Confluence does not have a valid license.
| Recruit/appoint ate least 2 members into the TWG (in addition to the RFC Editor) | |
| Ensure the TWG members have Confluence access |
Set up the ESCO Reviews Pages on Earthdata and the Wiki
Table Enhancer for Confluence
Table Enhancer for Confluence does not have a valid license.
Add a notice saying that the review is upcoming on the ESCO Reviews page on Earthdata. Don't link to the review since none of the pages are publicly visible yet. This example shows how the page looks with a current review, an upcoming review and two in-progress views. (see Earthdata Page Editing, Creation, etc.)
| |
Add a notice saying that the review is upcoming on the ESCO Reviews page on the wiki. Don't link to the review since none of the pages are publicly visible yet.
|
TWG Activity (pre-review phase)
Table Enhancer for Confluence
Table Enhancer for Confluence does not have a valid license.
| Review the RFC for completeness (this may already have been done as part of the initial screening but if there are problems with the RFC that are discovered in this phase, they should be addressed in cooperation with the RFC authors). | |
Ensure RFC is ready to be reviewed more broadly | |
Ensure there is sufficient evidence of implementation | |
| Identify reviewers | |
Target potential reviewers, including people listed in the Evidence of Implementation | |
| Set up a list of potential reviewers on the TWG Page | |
| Develop a review strategy - does the review need to include a technical review? Should the technical review be combined with the operational readiness review? | |
Develop review questions - try to ask questions that help to uncover strengths, weaknesses, applicability and limitations of the contents of the RFC (needed for the recommendation for adoption that is sent to ESDIS along with the RFC). Reviewers tend to be very good at commenting about the specifics of the document and it helps to get them to think about the overall context. A good question to end with is whether they think this document should be approved. Optional: You can ask the authors for help with the questions, either by asking them what they would like to ask reviewers or by showing them the questions and asking for feedback. |
Pre-Review Logistics
Review Period
The TWG and ESCO decide on when the review should start and how long it should run. Typical periods are on the order of 4 weeks. Reviews in August as well as in late November through early January don't tend to attract as many reviewers. It might be advisable to either avoid those times or set a longer review period. Perhaps also plan for more reminders.
Document Preparation
To set up a review in Jama and on the Wiki, you need to prepare the following documents
- A clean copy of the RFC with no comments left over from the internal reviews. Create a PDF copy and a Word copy. This will get attached to the Public Review Page. This copy should be named "ESDS-RFC-nnnv0.x" where nnn is the RFC number and x is the latest revision number. For COMET, use "423-ESCO-nnnv0.x".
- A copy of the review questions in a separate document. Create a PDF copy and a Word copy. This will get attached to the Public Review Page.
- A copy of the RFC that has a section inserted above all other content, even above the title. This section is titled Review Introduction and Review Questions and contains a brief introduction to the review as well as the questions developed by the TWG. Create only a Word copy. This will get uploaded into Jama.
Add RFC into Jama
Table Enhancer for Confluence
Table Enhancer for Confluence does not have a valid license.
Send request to esco-staff@lists.nasa.gov asking to upload the review to Jama, ( | |
| |
| |
|
Finalize Public Review Pages
Table Enhancer for Confluence
Table Enhancer for Confluence does not have a valid license.
| Let the authors know that the review is about to go live. | |
| Attach the clean copies of the RFC and the questions in both Word and PDF to the Public Review Pages. You have to attach copies to that page so they will be accessible to the public. Non-ESCO or TWG members cannot see files that are links to attachments on the TWG page. | |
| Fill in the red items on the Public Review Page | |
Do a final check of the Public Review Page - If you do these steps in an anonymous browser mode, you'll notice permission problems that might arise.
| |
Edit the ESCO Reviews page on Earthdata to move the review from the Upcoming section to the Current section. Add the comment period and a link to the review's Top Level Review Page. Check the link on this page once it's been published to make sure it points to the right page! This example shows how the page looks with a current review, an upcoming review and two in-progress views. (see Earthdata Page Editing, Creation, etc.)
| |
Edit the ESCO Reviews page on the wiki to move the review from the Upcoming section to the Current section. Add the comment period and a link to the review's Top Level Review Page. Check the link on this page once it's been published to make sure it points to the right page! This example shows how the page looks with a current review, an upcoming review and two in-progress views.
|
Community Review
Draft an Announcement
Create an announcement that includes
- A short introduction to the review
- Why the review is important
- the deadline
- A short note about ESCO's role
- Link to the Public Review Page
- Ask recipients of the email to spread the word to colleagues
Here is a sample review announcement. It's always easier to start from an email sent from a previous review!
Disseminate the Announcement
Email Lists - lists to consider
- DAAC Managers - Confer with the NASA ESCO lead before sending this (and, in fact, have the NASA lead send the email to this list)
- cmr-announce - This list is managed by ESCO.
- eso-sig - This list is managed by ESCO (not functioning as of 5/4/22)
- ESIP mailing lists (such as esip-all) - consider one or more ESIP lists, but confer with ESIP first
Targeted Reviewers
People respond best if they are asked personally. If you can develop a list of potential reviewers, then send a separate email to each one, if possible. If you send an email to a group of potential reviewers, be sure to list them in the BCC field of the email.
Keep track of targeted reviewers on the TWG Page.
Slack Channel
Paste the announcement into the #eosdis-standards channel on Slack.
Verbal mentions
Person-to-person recruiting works well. If you're at a conference or a meeting when the review starts (or is about to start), mention the review and offer to send the review information to people you meet. If you send announcements to anyone, add them to the list of reviewers on the TWG Page.
Add Reviewers to Jama
While the comment period is open, we will get requests from people to be given access to the Jama review.
Table Enhancer for Confluence
Table Enhancer for Confluence does not have a valid license.
Steps
Receive an email asking for Jama access | |
Acknowledge the request so they know what to expect. | |
| Follow the process at How to add reviewers to Jama. | |
Let them know when they can start reviewing | |
| For each person you add, also add their name and the date you added them into the Review Team page near the bottom under the heading Reviewers added based on requests sent to esco-staff@lists.nasa.gov |
Collect Comments
Some people may send comments via email. If they do, copy the email text and save it on the TWG page for the review as a new comment. Thank the sender for the email and add your email to the comment as a reply. Also forward the comment to the authors. Use your discretion whether or not to remove the commenter's name and email address.
Comments that are received in Jama will be retained by Jama.
Reminder Announcements
At about the 1/2 or 3/4 point of the review period consider sending out a reminder announcement. Send reminder emails to targeted reviewers if they have not already submitted any comments.
Review Period Extension
Deciding whether to extend the review can be hard. Many people wait until the last week or last few days to start. However, if there are not enough commenters near the end of the review period, consider extending the review period. This provides a good reason for sending another announcement.
Review Period Conclusion
When the review period is over, the Public Review Page, the ESCO Reviews page on Earthdata and the ESCO Reviews page on the Wiki should be edited to indicate that the review period is over.
Table Enhancer for Confluence
Table Enhancer for Confluence does not have a valid license.
Edit the Public Review page to indicate that the comment period is closed.
| |
Edit the ESCO Reviews page on Earthdata to move the review from the Current section to the Reviews in Progress section. Change the comment period to indicate that it is closed. This example shows how the page looks with a current review, an upcoming review and two in-progress views. (see Earthdata Page Editing, Creation, etc.)
| |
Edit the ESCO Reviews page on the wiki to move the review from the Current section to the Reviews in Progress section. Change the comment period to indicate that it is closed. This example shows how the page looks with a current review, an upcoming review and two in-progress views.
|
Assess Comments
TWG Assessment
When the comment period closes, the TWG should assess the comments to decide
- Did enough people review the document? There is no specific number. Some topics can be very specialized and you may only find a small number of reviewers. Other documents attract as many as 20+ reviewers.
- Do the comments warrant any revision of the document? The answer to this is usually "yes".
- Do the comments support approval by ESDIS for use in EOSDIS? There should be more favorable comments than negative ones to support this.
If the TWG feels that there were not enough comments or that the comments were too unfavorable, ESO needs to decide how to proceed. There is no specific guidance for this and each document has to be handled on a case-by-case basis. In the past, there have been a few different scenarios:
- A documented standard (RFC-005) was found to have little operational use but during the review it became clear that the prior version of that standard was used extensively. The submitters were encouraged to change the category of the current version to be a Technical Note and to submit the prior version as a Specification (RFC-006).
- A documented convention (RFC-013) was submitted by an ESDIS project team. This convention was only in use within the system maintained by that team and data providers who submitted information to that system using the specified format. No other implementations existed or were expected to be developed. The TWG recommendation was to not recommend the spec for adoption. Subsequently, the status was noted as "submitted/in progress" on the ESO Document List. In 2016, ESO made the decision to mark it as Deprecated.
- As part of an ESDIS desire to have the GeoTIFF file format be documented properly (GeoTIFF was a fairly mature format with no current format specification at the time), ESO invited someone to write a GeoTIFF RFC. We expected the RFC to contain an up-to-date format specification, but that turned out to be a far more complex task than originally expected. We had an RFC number dedicated to the document and the author submitted a draft that did not document the current format. The decision was made to stop processing the document even before sending it out for comment and the document number was reused for a different RFC.
- An RFC covering the topic of Datacasting was submitted in 2012 but after over a year it was dropped due to insufficient evidence of implementation. See the comment at the bottom of the page for some background.
If the decision is made to stop processing this RFC or to change course, then the rest of the process will be different.
Authors' Assessment
The RFC Editor notifies the Authors that they can/should start addressing the comments and work towards a revised document. For recent reviews, particularly those with extensive comments, the authors have set up Google spreadsheets to keep track of the process. There is an Excel spreadsheet here that can be used as an example. The authors have shared the spreadsheets with the TWG but the spreadsheets are not meant to be preserved as public documents. The material in the spreadsheets can be valuable resource material for the final TWG report.
The authors will need to be upgraded to Moderator status on the Jama review so that they can work with the comments.
Once the authors have finished addressing the comments, they should provide a new version of the document.
Table Enhancer for Confluence
Table Enhancer for Confluence does not have a valid license.
Contact the authors to work with them on addressing the comments | |
Give the authors copies of comments that were submitted via email | |
Add the authors as Moderators in the Jama review. In the Actions menu, you will find Update Moderators. Add the authors by clicking their names in the left panel, either in Project Team or Outside Project Team.
| |
Provide the authors with the sample spreadsheet and ask them to track the comments. | |
| You may need to periodically check in with the authors to see how they are progressing. |
Community Review - 2nd Round
The 2nd round of reviews is meant to give the 1st round commenters a chance to give feedback on how their comments were handled. Only those people who provided comments in the 1st round are invited to re-review the document.
Pre-review Logistics
Review Period
For the 2nd round, two weeks has been typical in the past. Adjust this as needed.
Document Preparation
For the 2nd round, you don't need new review questions. If all the 1st round commenters used Jama, then you also don't need to post updated documents on the Public Review Page.
You do have to ensure you have a clean copy of the updated document, named and numbered appropriately with a new version number to upload into Jama.
Add the new document into Jama
Table Enhancer for Confluence
Table Enhancer for Confluence does not have a valid license.
Send request to esco-staff@lists.nasa.gov asking to upload the review to Jama that includes: | |
| |
| |
|
Contact the 1st round commenters
Just before you send out the review announcement, add the 2nd round commenters to the Jama review as reviewers. They will probably get an automated email from the Jama system letting them know they've been added to the review.
This is a sample round 2 review announcement.
Keep track of who you've sent the 2nd round review invitation to, their response, etc. on the TWG page.
Post Review
Document Clean-up
The TWG evaluates the comments received. The authors may need to make some additional revisions to the document.
Once the authors have finished the document, create a clean copy with all headings such as version number, title, etc filled in.
Send recommendation to ESDIS
The TWG is responsible for writing a recommendation to ESDIS to approve the document. The recommendation includes a summary of the RFC along with a short summary of the Strengths, Weaknesses, Applicability, and Limitations (SWAL) of the RFC. Most current RFC landing pages (see the ESCO Document List) have examples of the SWAL.
When the recommendation is ready, it is sent to the NASA ESCO lead along with the document. The NASA ESO lead is responsible for either approving the RFC, getting further feedback about whether to recommend from within ESDIS, or rejecting the RFC (the latter has not happened so far).
Publication of the RFC
COMET
Skip this part if the RFC is not going to be published in COMET.
At this point, if the RFC is meant to be published in COMET, then if it's not already in COMET format, it must be turned into a properly formatted COMET document.
Once there is a COMET formatted version, the document has to be submitted to COMET and be approved by ESDIS within the COMET system.
The final, COMET-approved, PDF version of the document is then published the same way a non-COMET RFC would be.
Earthdata
There are several steps to publishing an RFC (or COMET) document. First, the final, cleaned up and properly named PDF file should be uploaded to Earthdata. This is done via Conduit. Then a landing page has to be created for it (or updated if the document will share a landing page with another - see the Dataset Interoperability Recommendations for Earth Science landing page for an example with two documents or the Recommendations from the Data Quality Working Group landing page for an example with three documents).
The document should also be added to the ESCO Document List and probably (but not always) to the Standards and Practices page. There are a few Mission Requirements documents that are on the ESDIS Mission Requirements page and not on the Standards and Practices page.
Once the document is in Conduit on Earthdata and the landing page is public, let the authors know about it. They may have helpful comments about the landing page.
Additional Housekeeping
ESO Reviews pages
Once a document has been published on Earthdata, the review is removed from the ESCO Reviews page on Earthdata and from the ESCO Reviews page on the Wiki.
Wiki Pages
The wiki pages that were public as well as the private ones are gathered into a bundle with the Public Review page as the parent and the rest of the pages as child pages. These are moved to the ESCO Review and Publication Archives on the wiki.
Announcement
When the publication steps have been completed, it's a good idea to send an announcement email and to post the announcement on the #eosdis-standards channel on Slack.
Attachments
Overview
Content Tools