Contents

Introduction

This page contains instructions about how to set up a new Request for Comment (RFC) review. It also contains child pages that are templates that, when copied, provide the wiki pages needed to track the RFC.The ESCO role in processing an RFC is described in the Process for Earth Science Data Systems Standards and References document. Also refer to the material in Instructions to RFC Authors, a description of the process as viewed from the document authors' point of view.

Scope

This page covers only RFC document reviews.

It does not cover RFCs that are evaluated as ASCII File Formats as described in ASCII File Format Guidelines for Earth Science Data.

It does not cover ESCO-PUB documents (they are not usually reviewed). 

Previous RFCs

It's always helpful to look at previous RFC review materials, particularly those that are in the same category as this new RFC. Look at the Standards and Practices page and the ESCO Document List to find relevant RFCs. Then look in the ESCO Review and Publication Archives on the wiki to see if there are helpful materials there.

RFC Requirements

Upon initial submission, the document might not meet all these requirements. ESCO staff should work with authors to help them understand the requirements and how to meet them. It is possible that after working with the authors it becomes clear that the document won't meet the requirements. In that case, ESCO should reject the submission (gently!).

S: Specification or Convention
CS: Commercial Specification or Convention
TN: Tech Note

RFC Development

Generally carried out by non-ESCO individuals or teams with ESCO assistance. Authors should contact ESCO as early in this process as possible to increase chances of success. ESCO will help authors understand the process, how to format the document, other requirements such as completeness, evidence of implementation and operational experience. ESCO should also try to determine whether the proposed material is likely to be approved at the end of the process. Some material may not be suitable for a community review. In that case, ESCO could consider working having the authors submit an ESCO-PUB document instead (see the ESCO Publications section of the ESCO Document List for examples).

Sometimes ESCO will receive fully-formed RFC documents. This is rare if the authors have not had any prior submissions (but it has happened) and less rare for experienced authors. In that case, use this section as a reminder of things that should be checked and discussed with the submitters.

Sometimes ESCO writes the RFC, as was the case with the GeoTIFF File Format specification. In that case, ESCO has to adhere to the same process, and it's probably doubly important to make sure the document is well-written.

RFC Submission and Intake

When the authors are ready, the RFC is submitted to ESCO for processing.

Document Naming and Numbering

All versions of RFC documents prior to publication are 0.x

Some authors will use their own document naming and numbering convention during the editing phases.

Versions sent out for public review should be named "ESDS-RFC-nnnv0.x" or "423-ESCO-nnnv0.x"

The first published version of an approved document is version 1.0. The document name of this version does not contain the version number. COMET will use its own naming convention for versions.

Subsequent versions of RFC documents that are created to fix editorial problems are "v1.x". If an RFC has to get bumped to "2.x" because of non-editorial changes, then ESCO has to decide whether there has to be a new public review (the standards process document says it does).

Initial Screening

The initial screening process involves designating a member of ESCO to be the RFC Editor. The RFC Editor is the main point of contact between ESCO and the authors and also serves as the chair of the Technical Working Group (TWG).

Before setting up the review infrastructure, ESCO has to decide whether the document meets all the requirements and whether the material should be processed as an RFC and whether the material should be processed as a different document type than initially proposed by the authors. There are times when authors submit an RFC as a proposed standard or convention, but ESCO feels that it should be re-written as a Technical Note or Suggested Practice. This can happen if the material is not strong enough (not enough use within NASA being a primary reason) to be a standard or convention. (If the authors worked with ESCO to create the document, hopefully this is not the case.)

If the material is strong but the document itself is not well-formulated, then ESCO can work with the authors to correct that.

Create a Digital Object Identifier (DOI)

Updated 04/18/22 - this is VERY early in the process to create a DOI and, based on discussions within ESCO, it was decided that waiting to generate a DOI until the RFC is further through the community review process and near approval would be more efficient.

Follow the process in DOIs for ESCO Documents to create a DOI for the document. Most ESCO documents will be assigned DOIs.

Initial Review Set Up

Once ESCO knows there is a new RFC about to be submitted (this can happen in the RFC Development phase or in the RFC Submission and Intake phase), a set of wiki pages is needed to track the review process.

To set up the wiki tracking pages, copy this page, its attachments and its child pages. During the copy process you can rename the pages to include the RFC number.

Note: RFC numbers are assigned consecutively.

After the copy operation, the new pages will be in the Work in Progress area and should look like this:

Finish Setting up Wiki pages

At this point, the Tracking Page remains in the Work in Progress area and the Public Review Page and the TWG Page are in the ESCO Reviews area.

Recruit Technical Working Group (TWG)

There is a description of what a TWG does in the ESCO Technical Working Group Guide.

If ESCO members have enough expertise about the topic of the RFC, then the TWG can consist of ESCO members.

The RFC Editor chairs the TWG (this is not spelled out in the process but has generally been the case in practice).

The TWG should use the TWG Page to keep lists of potential reviewers, formulate review questions, etc.

Set up the ESCO Reviews Pages on Earthdata and the Wiki

TWG Activity (pre-review phase)

Pre-Review Logistics

Review Period

The TWG and ESCO decide on when the review should start and how long it should run. Typical periods are on the order of 4 weeks. Reviews in August as well as in late November through early January don't tend to attract as many reviewers. It might be advisable to either avoid those times or set a longer review period. Perhaps also plan for more reminders.

Document Preparation

To set up a review in Jama and on the Wiki, you need to prepare the following documents

• A clean copy of the RFC with no comments left over from the internal reviews. Create a PDF copy and a Word copy. This will get attached to the Public Review Page. This copy should be named "ESDS-RFC-nnnv0.x" where nnn is the RFC number and x is the latest revision number. For COMET, use "423-ESCO-nnnv0.x".
• A copy of the review questions in a separate document. Create a PDF copy and a Word copy. This will get attached to the Public Review Page.
• A copy of the RFC that has a section inserted above all other content, even above the title. This section is titled Review Introduction and Review Questions and contains a brief introduction to the review as well as the questions developed by the TWG. Create only a Word copy. This will get uploaded into Jama.

Add RFC into Jama

Finalize Public Review Pages

Community Review

Draft an Announcement

Create an announcement that includes

• A short introduction to the review
• Why the review is important
• the deadline
• A short note about ESCO's role
• Link to the Public Review Page
• Ask recipients of the email to spread the word to colleagues

Here is a sample review announcement. It's always easier to start from an email sent from a previous review!

Disseminate the Announcement

Email Lists - lists to consider

• DAAC Managers - Confer with the NASA ESCO lead before sending this (and, in fact, have the NASA lead send the email to this list)
• cmr-announce - This list is managed by ESCO.
• eso-sig - This list is managed by ESCO (not functioning as of 5/4/22)
• ESIP mailing lists (such as esip-all) - consider one or more ESIP lists, but confer with ESIP first

Targeted Reviewers

People respond best if they are asked personally. If you can develop a list of potential reviewers, then send a separate email to each one, if possible. If you send an email to a group of potential reviewers, be sure to list them in the BCC field of the email.

Keep track of targeted reviewers on the TWG Page.

Slack Channel

Paste the announcement into the #eosdis-standards channel on Slack.

Verbal mentions

Person-to-person recruiting works well. If you're at a conference or a meeting when the review starts (or is about to start), mention the review and offer to send the review information to people you meet. If you send announcements to anyone, add them to the list of reviewers on the TWG Page.

Add Reviewers to Jama

While the comment period is open, we will get requests from people to be given access to the Jama review.

Steps

Collect Comments

Some people may send comments via email. If they do, copy the email text and save it on the TWG page for the review as a new comment. Thank the sender for the email and add your email to the comment as a reply. Also forward the comment to the authors. Use your discretion whether or not to remove the commenter's name and email address.

Comments that are received in Jama will be retained by Jama.

Reminder Announcements

At about the 1/2 or 3/4 point of the review period consider sending out a reminder announcement. Send reminder emails to targeted reviewers if they have not already submitted any comments.

Review Period Extension

Deciding whether to extend the review can be hard. Many people wait until the last week or last few days to start. However, if there are not enough commenters near the end of the review period, consider extending the review period. This provides a good reason for sending another announcement.

Review Period Conclusion

When the review period is over, the Public Review Page, the ESCO Reviews page on Earthdata and the ESCO Reviews page on the Wiki should be edited to indicate that the review period is over.

Assess Comments

TWG Assessment

When the comment period closes, the TWG should assess the comments to decide

• Did enough people review the document? There is no specific number. Some topics can be very specialized and you may only find a small number of reviewers. Other documents attract as many as 20+ reviewers.
• Do the comments warrant any revision of the document? The answer to this is usually "yes".
• Do the comments support approval by ESDIS for use in EOSDIS? There should be more favorable comments than negative ones to support this.

If the TWG feels that there were not enough comments or that the comments were too unfavorable, ESO needs to decide how to proceed. There is no specific guidance for this and each document has to be handled on a case-by-case basis. In the past, there have been a few different scenarios:

• A documented standard (RFC-005) was found to have little operational use but during the review it became clear that the prior version of that standard was used extensively. The submitters were encouraged to change the category of the current version to be a Technical Note and to submit the prior version as a Specification (RFC-006).
• A documented convention (RFC-013) was submitted by an ESDIS project team. This convention was only in use within the system maintained by that team and data providers who submitted information to that system using the specified format. No other implementations existed or were expected to be developed. The TWG recommendation was to not recommend the spec for adoption. Subsequently, the status was noted as "submitted/in progress" on the ESO Document List. In 2016, ESO made the decision to mark it as Deprecated.
• As part of an ESDIS desire to have the GeoTIFF file format be documented properly (GeoTIFF was a fairly mature format with no current format specification at the time), ESO invited someone to write a GeoTIFF RFC. We expected the RFC to contain an up-to-date format specification, but that turned out to be a far more complex task than originally expected. We had an RFC number dedicated to the document and the author submitted a draft that did not document the current format. The decision was made to stop processing the document even before sending it out for comment and the document number was reused for a different RFC.
• An RFC covering the topic of Datacasting was submitted in 2012 but after over a year it was dropped due to insufficient evidence of implementation. See the comment at the bottom of the page for some background.

If the decision is made to stop processing this RFC or to change course, then the rest of the process will be different.

Authors' Assessment

The RFC Editor notifies the Authors that they can/should start addressing the comments and work towards a revised document. For recent reviews, particularly those with extensive comments, the authors have set up Google spreadsheets to keep track of the process. There is an Excel spreadsheet here that can be used as an example. The authors have shared the spreadsheets with the TWG but the spreadsheets are not meant to be preserved as public documents. The material in the spreadsheets can be valuable resource material for the final TWG report.

The authors will need to be upgraded to Moderator status on the Jama review so that they can work with the comments. 

Once the authors have finished addressing the comments, they should provide a new version of the document.

Community Review - 2nd Round

The 2nd round of reviews is meant to give the 1st round commenters a chance to give feedback on how their comments were handled. Only those people who provided comments in the 1st round are invited to re-review the document.

Pre-review Logistics

Review Period

For the 2nd round, two weeks has been typical in the past. Adjust this as needed.

Document Preparation

For the 2nd round, you don't need new review questions. If all the 1st round commenters used Jama, then you also don't need to post updated documents on the Public Review Page.

You do have to ensure you have a clean copy of the updated document, named and numbered appropriately with a new version number to upload into Jama.

Add the new document into Jama

Contact the 1st round commenters

Just before you send out the review announcement, add the 2nd round commenters to the Jama review as reviewers. They will probably get an automated email from the Jama system letting them know they've been added to the review.

This is a sample round 2 review announcement.

Keep track of who you've sent the 2nd round review invitation to, their response, etc. on the TWG page.

Post Review

Document Clean-up

The TWG evaluates the comments received. The authors may need to make some additional revisions to the document.

Once the authors have finished the document, create a clean copy with all headings such as version number, title, etc filled in.

Send recommendation to ESDIS

The TWG is responsible for writing a recommendation to ESDIS to approve the document. The recommendation includes a summary of the RFC along with a short summary of the Strengths, Weaknesses, Applicability, and Limitations (SWAL) of the RFC. Most current RFC landing pages (see the ESCO Document List) have examples of the SWAL.

When the recommendation is ready, it is sent to the NASA ESCO lead along with the document. The NASA ESO lead is responsible for either approving the RFC, getting further feedback about whether to recommend from within ESDIS, or rejecting the RFC (the latter has not happened so far).

Publication of the RFC

COMET

Skip this part if the RFC is not going to be published in COMET.

At this point, if the RFC is meant to be published in COMET, then if it's not already in COMET format, it must be turned into a properly formatted COMET document.

Once there is a COMET formatted version, the document has to be submitted to COMET and be approved by ESDIS within the COMET system.

The final, COMET-approved, PDF version of the document is then published the same way a non-COMET RFC would be.

Earthdata

There are several steps to publishing an RFC (or COMET) document. First, the final, cleaned up and properly named PDF file should be uploaded to Earthdata. This is done via Conduit. Then a landing page has to be created for it (or updated if the document will share a landing page with another - see the Dataset Interoperability Recommendations for Earth Science landing page for an example with two documents or the Recommendations from the Data Quality Working Group landing page for an example with three documents).

The document should also be added to the ESCO Document List and probably (but not always) to the Standards and Practices page. There are a few Mission Requirements documents that are on the ESDIS Mission Requirements page and not on the Standards and Practices page.

Once the document is in Conduit on Earthdata and the landing page is public, let the authors know about it. They may have helpful comments about the landing page.

Additional Housekeeping

ESO Reviews pages

Once a document has been published on Earthdata, the review is removed from the ESCO Reviews page on Earthdata and from the ESCO Reviews page on the Wiki.

Wiki Pages

The wiki pages that were public as well as the private ones are gathered into a bundle with the Public Review page as the parent and the rest of the pages as child pages. These are moved to the ESCO Review and Publication Archives on the wiki.

Announcement

When the publication steps have been completed, it's a good idea to send an announcement email and to post the announcement on the #eosdis-standards channel on Slack.

Attachments