rfc9561xml2.original.xml   rfc9561.xml 
<?xml version="1.0" encoding="UTF-8"?> <?xml version='1.0' encoding='UTF-8'?>
<?xml-stylesheet type="text/xsl" href="rfc2629.xslt" ?>
<!-- generated by https://github.com/cabo/kramdown-rfc version (Ruby 3.1.2) -
->
<!DOCTYPE rfc [ <!DOCTYPE rfc [
<!ENTITY nbsp "&#160;"> <!ENTITY nbsp "&#160;">
<!ENTITY zwsp "&#8203;"> <!ENTITY zwsp "&#8203;">
<!ENTITY nbhy "&#8209;"> <!ENTITY nbhy "&#8209;">
<!ENTITY wj "&#8288;"> <!ENTITY wj "&#8288;">
]> ]>
<?rfc strict="yes"?> <rfc xmlns:xi="http://www.w3.org/2001/XInclude"
<?rfc comments="yes"?> ipr="trust200902"
<?rfc inline="yes"?> docName="draft-ietf-nfsv4-scsi-layout-nvme-07"
<?rfc editing="no"?> number="9561"
<?rfc tocompact="yes"?> category="std"
<?rfc compact="yes"?> consensus="true"
<?rfc subcompact="no"?> submissionType="IETF"
obsoletes=""
updates=""
tocInclude="true"
sortRefs="true"
symRefs="true"
version="3"
xml:lang="en"
>
<rfc ipr="trust200902" docName="draft-ietf-nfsv4-scsi-layout-nvme-07" category=" std" consensus="true" submissionType="IETF" tocDepth="3" tocInclude="true" sortR efs="true" symRefs="true">
<front> <front>
<title abbrev="pNFS SCSI Layout for NVMe">Using the Parallel NFS (pNFS) SCSI <title abbrev="pNFS SCSI Layout for NVMe">Using the Parallel NFS (pNFS) SCSI
Layout to access NVMe storage devices</title> Layout to Access Non-Volatile Memory Express (NVMe) Storage Devices</title>
<seriesInfo name="RFC" value="9561"/>
<author initials="C." surname="Hellwig" fullname="Christoph Hellwig" role="e ditor"> <author initials="C." surname="Hellwig" fullname="Christoph Hellwig" role="e ditor">
<organization></organization> <organization/>
<address> <address>
<email>hch@lst.de</email> <email>hch@lst.de</email>
</address> </address>
</author> </author>
<author initials="C." surname="Lever" fullname="Charles Lever"> <author initials="C." surname="Lever" fullname="Charles Lever">
<organization abbrev="Oracle">Oracle Corporation</organization> <organization abbrev="Oracle">Oracle Corporation</organization>
<address> <address>
<postal> <postal>
<country>United States of America</country> <country>United States of America</country>
</postal> </postal>
skipping to change at line 68 skipping to change at line 72
<street>176 South Street</street> <street>176 South Street</street>
<city>Hopkinton</city> <city>Hopkinton</city>
<region>MA</region> <region>MA</region>
<code>01748</code> <code>01748</code>
<country>United States of America</country> <country>United States of America</country>
</postal> </postal>
<email>david.black@dell.com</email> <email>david.black@dell.com</email>
</address> </address>
</author> </author>
<date year="2024" month="January" day="10"/> <date year="2024" month="March"/>
<area>Transport</area> <area>Transport</area>
<workgroup>NFSv4</workgroup> <workgroup>NFSv4</workgroup>
<keyword>NFSv4</keyword> <keyword>NFSv4</keyword>
<abstract> <abstract>
<t>This document specifies how to use the Parallel Network File System (pN
<t>This document specifies how to use the Parallel Network File System (pNFS) FS)
Small Computer System Interface (SCSI) Layout Type to access storage devices Small Computer System Interface (SCSI) Layout Type to access storage devices
using the Non-Volatile Memory Express (NVMe) protocol family.</t> using the Non-Volatile Memory Express (NVMe) protocol family.</t>
</abstract> </abstract>
</front> </front>
<middle> <middle>
<section anchor="sec_intro">
<section anchor="sec:intro"><name>Introduction</name> <name>Introduction</name>
<t>NFSv4.1 <xref target="RFC8881"/> includes a pNFS feature that allows
<t>NFSv4.1 (in <xref target="RFC8881"/>) includes a pNFS feature that allows reads and writes to be performed by means other than directing read and
reads and writes to be performed by other means than directing read and
write operations to the server. Through use of this feature, the server, write operations to the server. Through use of this feature, the server,
in the role of metadata server is responsible for managing file and in the role of metadata server, is responsible for managing file and
directory metadata while separate means are provided for execution of directory metadata while separate means are provided to execute
reads and writes.</t> reads and writes.</t>
<t>These other means of performing file reads and writes are defined by
<t>These other means of performing file reads and writes are defined by individual mapping types, which often have their own specifications.</t>
individual mapping types which often have their own specifications.</t> <t>The pNFS Small Computer System Interface (SCSI) layout <xref target="RF
C8154"/> is a layout
<t>The pNFS Small Computer System Interface (SCSI) layout <xref target="RFC8154" type that allows NFS clients to directly perform I&wj;/&wj;O to block storage de
/> is a layout vices
type that allows NFS clients to directly perform I/O to block storage devices
while bypassing the Metadata Server (MDS). It is specified by using while bypassing the Metadata Server (MDS). It is specified by using
concepts from the SCSI protocol family for the data path to the storage devices. </t> concepts from the SCSI protocol family for the data path to the storage devices. </t>
<t>NVM Express (NVMe), or the Non-Volatile Memory Host Controller Interfac
<t>NVM Express (NVMe), or the Non-Volatile Memory Host Controller Interface e
Specification, is a set of specifications to talk to storage devices over Specification, is a set of specifications to talk to storage devices over
a number of protocols such as PCI Express (PCIe), Fibre Channel (FC) and a number of protocols such as PCI Express (PCIe), Fibre Channel (FC),
TCP/IP or Remote Direct Memory Access (RDMA) networking. NVMe is currently TCP/IP, or Remote Direct Memory Access (RDMA) networking. NVMe is currently
the by far dominant protocol used to access PCIe Solid State Disks (SSDs), the predominantly used protocol to access PCIe Solid State Disks (SSDs),
and increasingly adopted for remote storage access where it replaces and it is increasingly being adopted for remote storage access to replace
SCSI-based protocols such as iSCSI.</t> SCSI-based protocols such as iSCSI.</t>
<t>This document defines how NVMe Namespaces using the NVM Command Set <xr
<t>This document defines how NVMe Namespaces using the NVM Command Set <xref tar ef target="NVME-NVM"/>
get="NVME-NVM"/>
exported by NVMe Controllers implementing the exported by NVMe Controllers implementing the
NVMe Base specification <xref target="NVME-BASE"/> are to be used as NVMe Base specification <xref target="NVME-BASE"/> are to be used as
storage devices using the SCSI Layout Type. storage devices using the SCSI Layout Type.
The definition is independent of the underlying transport used by the The definition is independent of the underlying transport used by the
NVMe Controller and thus supports Controllers implementing a wide variety NVMe Controller and thus supports Controllers implementing a wide variety
of transports, including PCI Express, RDMA, TCP and Fibre Channel.</t> of transports, including PCIe, RDMA, TCP, and FC.</t>
<t>This document does not amend the existing SCSI layout document. Rather
<t>This document does not amend the existing SCSI layout document. Rather, ,
it it
defines how NVMe Namespaces can be used within the SCSI Layout by defines how NVMe Namespaces can be used within the SCSI Layout by
establishing a mapping of the SCSI constructs used in the SCSI layout establishing a mapping of the SCSI constructs used in the SCSI layout
document to corresponding NVMe constructs.</t> document to corresponding NVMe constructs.</t>
<section anchor="ssc_intro_reqlang">
<name>Requirements Language</name>
<t>
The key words "<bcp14>MUST</bcp14>", "<bcp14>MUST NOT</bcp14>",
"<bcp14>REQUIRED</bcp14>", "<bcp14>SHALL</bcp14>", "<bcp14>SHALL NOT</bcp14>
",
"<bcp14>SHOULD</bcp14>", "<bcp14>SHOULD NOT</bcp14>",
"<bcp14>RECOMMENDED</bcp14>", "<bcp14>NOT RECOMMENDED</bcp14>",
"<bcp14>MAY</bcp14>", and "<bcp14>OPTIONAL</bcp14>" in this document are to
be
interpreted as described in BCP&nbsp;14 <xref target="RFC2119"/> <xref
target="RFC8174"/> when, and only when, they appear in all capitals, as
shown here.
</t>
<section anchor="ssc:intro:reqlang"><name>Requirements Language</name> </section>
<section anchor="ssc_intro_defs">
<t>The key words "<bcp14>MUST</bcp14>", "<bcp14>MUST NOT</bcp14>", "<bcp14>REQUI <name>General Definitions</name>
RED</bcp14>", "<bcp14>SHALL</bcp14>", "<bcp14>SHALL <t>The following definitions are included to provide context for the rea
NOT</bcp14>", "<bcp14>SHOULD</bcp14>", "<bcp14>SHOULD NOT</bcp14>", "<bcp14>RECO der.</t>
MMENDED</bcp14>", "<bcp14>NOT RECOMMENDED</bcp14>", <dl>
"<bcp14>MAY</bcp14>", and "<bcp14>OPTIONAL</bcp14>" in this document are to be i <dt>Client:</dt>
nterpreted as <dd>
described in BCP 14 <xref target="RFC2119"/> <xref target="RFC8174"/> when, and <t>The "client" is the entity that accesses the NFS server's
only when, they
appear in all capitals, as shown here.</t>
</section>
<section anchor="ssc:intro:defs"><name>General Definitions</name>
<t>The following definitions are provided for the purpose of providing
an appropriate context for the reader.</t>
<dl>
<dt>Client</dt>
<dd>
<t>The "client" is the entity that accesses the NFS server's
resources. The client may be an application that contains the resources. The client may be an application that contains the
logic to access the NFS server directly or be part of the operating logic to access the NFS server directly, or it may be part of the operating
system that provides remote file system services for a set of system that provides remote file system services for a set of
applications.</t> applications.</t>
</dd> </dd>
<dt>Metadata Server (MDS)</dt> <dt>Metadata Server (MDS):</dt>
<dd> <dd>
<t>The Metadata Server (MDS) is the entity responsible for coordinating <t>The Metadata Server (MDS) is the entity responsible for coordinat
ing
client access to a set of file systems and is identified by a server client access to a set of file systems and is identified by a server
owner.</t> owner.</t>
</dd> </dd>
</dl> </dl>
</section>
</section> <section anchor="ssc_intro_conv">
<section anchor="ssc:intro:conv"><name>Numerical Conventions</name> <name>Numerical Conventions</name>
<t>Numerical values defined in the SCSI specifications (e.g., <xref targ
<t>Numerical values defined in the SCSI specifications, e.g., <xref target="SPC5 et="SPC5"/>) and the
"/>, and the NVMe specifications (e.g., <xref target="NVME-BASE"/>) are represented using the
NVMe specifications, e.g., <xref target="NVME-BASE"/>, are represented using the same
same
conventions as those specifications wherein a 'b' suffix denotes a binary conventions as those specifications wherein a 'b' suffix denotes a binary
(base 2) number, e.g., 110b = 6 decimal, and an 'h' suffix denotes a (base 2) number (e.g., 110b = 6 decimal) and an 'h' suffix denotes a
hexadecimal (base 16) number, e.g., 1ch = 28 decimal.</t> hexadecimal (base 16) number (e.g., 1ch = 28 decimal).</t>
</section>
</section> </section>
</section> <section anchor="sec_slm">
<section anchor="sec:slm"><name>SCSI Layout mapping to NVMe</name> <name>SCSI Layout Mapping to NVMe</name>
<t>The SCSI layout definition <xref target="RFC8154"/> references only a
<t>The SCSI layout definition <xref target="RFC8154"/> references only a
few SCSI-specific concepts directly. This document provides a mapping few SCSI-specific concepts directly. This document provides a mapping
from these SCSI concepts to NVM Express concepts that are used from these SCSI concepts to NVM Express concepts that are used
when using the pNFS SCSI layout with NVMe namespaces.</t> when using the pNFS SCSI layout with NVMe namespaces.</t>
<section anchor="ssc_volident">
<section anchor="ssc:volident"><name>Volume Identification</name> <name>Volume Identification</name>
<t>The pNFS SCSI layout uses the Device Identification Vital Product Dat
<t>The pNFS SCSI layout uses the Device Identification Vital Product Data (VPD) a (VPD)
page (page code 83h) from <xref target="SPC5"/> to identify the devices used by page (page code 83h) from <xref target="SPC5"/> to identify the devices used by
a layout. Implementations that use NVMe namespaces as storage devices a layout. Implementations that use NVMe namespaces as storage devices
map NVMe namespace identifiers to a subset of the identifiers map NVMe namespace identifiers to a subset of the identifiers
that the Device Identification VPD page supports for SCSI logical that the Device Identification VPD page supports for SCSI logical
units.</t> units.</t>
<t>To be used as storage devices for the pNFS SCSI layout, NVMe namespac
<t>To be used as storage devices for the pNFS SCSI layout, NVMe namespaces es
<bcp14>MUST</bcp14> support either the IEEE Extended Unique Identifier (EUI64) o r <bcp14>MUST</bcp14> support either the IEEE Extended Unique Identifier (EUI64) o r
Namespace Globally Unique Identifier (NGUID) value reported in a Namespace Namespace Globally Unique Identifier (NGUID) value reported in a Namespace
Identification Descriptor, the I/O Command Set Independent Identify Identification Descriptor, the I&wj;/&wj;O Command Set Independent Identify
Namespace Data Structure, and the Identify Namespace Data Structure, Namespace data structure, and the Identify Namespace data structure,
NVM Command Set <xref target="NVME-BASE"/>. If available, use of the NGUID value is NVM Command Set <xref target="NVME-BASE"/>. If available, use of the NGUID value is
preferred as it is the larger identifier.</t> preferred as it is the larger identifier.</t>
<t>Note: The PS_DESIGNATOR_T10 and PS_DESIGNATOR_NAME have no equivalent <aside><t>Note: The PS_DESIGNATOR_T10 and PS_DESIGNATOR_NAME have no equ
in NVMe and cannot be used to identify NVMe storage devices.</t> ivalent
in NVMe and cannot be used to identify NVMe storage devices.</t></aside>
<t>The pnfs_scsi_base_volume_info4 structure for an NVMe namespace <t>The pnfs_scsi_base_volume_info4 structure for an NVMe namespace
<bcp14>SHALL</bcp14> be constructed as follows:</t> <bcp14>SHALL</bcp14> be constructed as follows:</t>
<ol spacing="normal" type="1"><li>
<t><list style="numbers"> <t>The "sbv_code_set" field <bcp14>SHALL</bcp14> be set to PS_CODE_S
<t>The "sbv_code_set" field <bcp14>SHALL</bcp14> be set to PS_CODE_SET_BINARY. ET_BINARY.</t>
</t> </li>
<t>The "pnfs_scsi_designator_type" field <bcp14>SHALL</bcp14> be set to <li>
<t>The "pnfs_scsi_designator_type" field <bcp14>SHALL</bcp14> be set
to
PS_DESIGNATOR_EUI64.</t> PS_DESIGNATOR_EUI64.</t>
<t>The "sbv_designator" field <bcp14>SHALL</bcp14> contain either the NGUID or </li>
<li>
<t>The "sbv_designator" field <bcp14>SHALL</bcp14> contain either th
e NGUID or
the EUI64 identifier for the namespace. If both NGUID and EUI64 the EUI64 identifier for the namespace. If both NGUID and EUI64
identifiers are available, then the NGUID identifier <bcp14>SHOULD</bcp14> be identifiers are available, then the NGUID identifier <bcp14>SHOULD</bcp14> be
used as it is the larger identifier.</t> used as it is the larger identifier.</t>
</list></t> </li>
</ol>
<t>RFC 8154 <xref target="RFC8154"/> specifies the "sbv_designator" field as an <t>RFC 8154 <xref target="RFC8154"/> specifies the "sbv_designator" fiel
XDR variable d as an XDR variable
length opaque&lt;&gt;. The length of that XDR opaque&lt;&gt; value (part of length opaque&lt;&gt; (refer to Section <xref target="RFC4506" sectionFormat="ba
re" section="4.10"/> of RFC 4506 <xref target="RFC4506"/>). The length of that X
DR opaque&lt;&gt; value (part of
its XDR representation) indicates which NVMe identifier is present. its XDR representation) indicates which NVMe identifier is present.
That length <bcp14>MUST</bcp14> be 16 octets for an NVMe NGUID identifier and That length <bcp14>MUST</bcp14> be 16 octets for an NVMe NGUID identifier and
<bcp14>MUST</bcp14> be 8 octets for an NVMe EUI64 identifier. All other lengths <bcp14>MUST</bcp14> be 8 octets for an NVMe EUI64 identifier. All other lengths
<bcp14>MUST NOT</bcp14> be used with an NVMe namespace.</t> <bcp14>MUST NOT</bcp14> be used with an NVMe namespace.</t>
</section>
</section> <section anchor="ssc_fencing">
<section anchor="ssc:fencing"><name>Client Fencing</name> <name>Client Fencing</name>
<t>The SCSI layout uses Persistent Reservations (PRs) to provide client
<t>The SCSI layout uses Persistent Reservations (PRs) to provide client
fencing. For this to be achieved, both the MDS and the Clients have to fencing. For this to be achieved, both the MDS and the Clients have to
register a key with the storage device, and the MDS has to create a register a key with the storage device, and the MDS has to create a
reservation on the storage device.</t> reservation on the storage device.</t>
<t>The following subsections provide a full mapping of the required
<t>The following sub-sections provide a full mapping of the required
PERSISTENT RESERVE IN and PERSISTENT RESERVE OUT SCSI commands <xref target="SPC 5"/> PERSISTENT RESERVE IN and PERSISTENT RESERVE OUT SCSI commands <xref target="SPC 5"/>
to NVMe commands which <bcp14>MUST</bcp14> be used when using to NVMe commands that <bcp14>MUST</bcp14> be used when using
NVMe namespaces as storage devices for the pNFS SCSI layout.</t> NVMe namespaces as storage devices for the pNFS SCSI layout.</t>
<section anchor="ssc_fencing_keys">
<section anchor="ssc:fencing:keys"><name>PRs - Key Registration</name> <name>PRs - Key Registration</name>
<t>On NVMe namespaces, reservation keys are registered using the
<t>On NVMe namespaces, reservation keys are registered using the
Reservation Register command (refer to Section 7.3 of <xref target="NVME-BASE"/> ) Reservation Register command (refer to Section 7.3 of <xref target="NVME-BASE"/> )
with the Reservation Register Action with the Reservation Register Action
(RREGA) field set to 000b (i.e., Register Reservation Key) and (RREGA) field set to 000b (i.e., Register Reservation Key) and
supplying the reservation key in the New Reservation Key (NRKEY) supplying the reservation key in the New Reservation Key (NRKEY)
field.</t> field.</t>
<t>Reservation keys are unregistered using the Reservation Register
<t>Reservation keys are unregistered using the Reservation Register
command with the Reservation Register Action (RREGA) field set to command with the Reservation Register Action (RREGA) field set to
001b (i.e., Unregister Reservation Key) and supplying the reservation 001b (i.e., Unregister Reservation Key) and supplying the reservation
key in the Current Reservation Key (CRKEY) field.</t> key in the Current Reservation Key (CRKEY) field.</t>
<t>One important difference between SCSI Persistent Reservations
<t>One important difference between SCSI Persistent Reservations
and NVMe Reservations is that NVMe reservation keys always apply and NVMe Reservations is that NVMe reservation keys always apply
to all controllers used by a host (as indicated by the NVMe Host to all controllers used by a host (as indicated by the NVMe Host
Identifier). This behavior is analogous to setting the ALL_TG_PT Identifier). This behavior is analogous to setting the ALL_TG_PT
bit when registering a SCSI Reservation key, and is always supported bit when registering a SCSI Reservation Key, and it is always supported
by NVMe Reservations, unlike the ALL_TG_PT for which SCSI support is by NVMe Reservations, unlike the ALL_TG_PT for which SCSI support is
inconsistent and cannot be relied upon. inconsistent and cannot be relied upon.
Registering a reservation key with a namespace creates an Registering a reservation key with a namespace creates an
association between a host and a namespace. A host that is a association between a host and a namespace. A host that is a
registrant of a namespace may use any controller with which that registrant of a namespace may use any controller with which that
host is associated (i.e., that has the same Host Identifier, host is associated (i.e., that has the same Host Identifier,
refer to Section 5.27.1.25 of <xref target="NVME-BASE"/>) refer to Section 5.27.1.25 of <xref target="NVME-BASE"/>)
to access that namespace as a registrant.</t> to access that namespace as a registrant.</t>
</section>
</section> <section anchor="ssc_fencing_reg">
<section anchor="ssc:fencing:reg"><name>PRs - MDS Registration and Reservation</ <name>PRs - MDS Registration and Reservation</name>
name> <t>Before returning a PNFS_SCSI_VOLUME_BASE volume to the client, the
MDS
<t>Before returning a PNFS_SCSI_VOLUME_BASE volume to the client, the MDS
needs to prepare the volume for fencing using PRs. This is done by needs to prepare the volume for fencing using PRs. This is done by
registering the reservation generated for the MDS with the device registering the reservation generated for the MDS with the device
(see <xref target="ssc:fencing:keys"/>) followed by a Reservation Acquire (see <xref target="ssc_fencing_keys"/>) followed by a Reservation Acquire
command (refer to Section 7.2 of <xref target="NVME-BASE"/>) with command (refer to Section 7.2 of <xref target="NVME-BASE"/>) with
the Reservation Acquire Action (RACQA) field set to 000b (i.e., Acquire) the Reservation Acquire Action (RACQA) field set to 000b (i.e., Acquire)
and the Reservation Type (RTYPE) field set to 4h (i.e., Exclusive Access and the Reservation Type (RTYPE) field set to 4h (i.e., Exclusive Access
- Registrants Only Reservation).</t> - Registrants Only Reservation).</t>
</section>
</section> <section anchor="ssc_fenceaction">
<section anchor="ssc:fenceaction"><name>Fencing Action</name> <name>Fencing Action</name>
<t>In case of a non-responding client, the MDS fences the client by
<t>In case of a non-responding client, the MDS fences the client by
executing a Reservation Acquire command (refer to Section 7.2 of <xref target="N VME-BASE"/>), executing a Reservation Acquire command (refer to Section 7.2 of <xref target="N VME-BASE"/>),
with the Reservation Acquire Action with the Reservation Acquire Action
(RACQA) field set to 001b (i.e., Preempt) or 010b (i.e., Preempt and (RACQA) field set to 001b (i.e., Preempt) or 010b (i.e., Preempt and
Abort), the Current Reservation Key (CRKEY) field set to the Abort), the Current Reservation Key (CRKEY) field set to the
server's reservation key, the Preempt Reservation Key (PRKEY) field server's reservation key, the Preempt Reservation Key (PRKEY) field
set to the reservation key associated with the non-responding client set to the reservation key associated with the non-responding client,
and the Reservation Type (RTYPE) field set to 4h (i.e., Exclusive and the Reservation Type (RTYPE) field set to 4h (i.e., Exclusive
Access - Registrants Only Reservation). Access - Registrants Only Reservation).
The client can distinguish I/O errors due to fencing from other The client can distinguish I&wj;/&wj;O errors due to fencing from other
errors based on the Reservation Conflict NVMe status code.</t> errors based on the Reservation Conflict NVMe status code.</t>
</section>
</section> <section anchor="ssc_recovery">
<section anchor="ssc:recovery"><name>Client Recovery after a Fence Action</name> <name>Client Recovery after a Fence Action</name>
<t>If an NVMe command issued by the client to the storage device retur
<t>If an NVMe command issued by the client to the storage device returns ns
a non-retryable error (refer to the DNR bit defined in Figure 92 in a non-retryable error (refer to the DNR bit defined in Figure 92 in
<xref target="NVME-BASE"/>), the client <bcp14>MUST</bcp14> commit all layouts t hat <xref target="NVME-BASE"/>), the client <bcp14>MUST</bcp14> commit all layouts t hat
use the storage device through the MDS, return all outstanding layouts use the storage device through the MDS, return all outstanding layouts
for the device, forget the device ID, and unregister the reservation for the device, forget the device ID, and unregister the reservation
key.</t> key.</t>
</section>
</section> </section>
</section> <section anchor="ssc_caches">
<section anchor="ssc:caches"><name>Volatile write caches</name> <name>Volatile Write Caches</name>
<t>For NVMe controllers, a volatile write cache is enabled if bit 0 of t
<t>For NVMe controllers a volatile write cache is enabled if bit 0 of the he
Volatile Write Cache (VWC) field in the Identify Controller Data Volatile Write Cache (VWC) field in the Identify Controller data
Structure, I/O Command Set Independent (see Figure 275 in <xref target="NVME-BAS structure, I&wj;/&wj;O Command Set Independent (refer to Figure 275 in <xref tar
E"/>) get="NVME-BASE"/>)
is set and the Volatile Write Cache Enable (WCE) bit (i.e., bit 00) in is set and the Volatile Write Cache Enable (WCE) bit (i.e., bit 00) in
the Volatile Write Cache Feature (Feature Identifier 06h) the Volatile Write Cache Feature (Feature Identifier 06h)
(see Section 5.27.1.4 of <xref target="NVME-BASE"/>) is set. (refer to Section 5.27.1.4 of <xref target="NVME-BASE"/>) is set.
If a volatile write cache is enabled on an NVMe namespace used as a If a volatile write cache is enabled on an NVMe namespace used as a
storage device for the pNFS SCSI layout, the pNFS server (MDS) <bcp14>MUST</bcp1 4> storage device for the pNFS SCSI layout, the pNFS server (MDS) <bcp14>MUST</bcp1 4>
use the NVMe Flush command to flush the volatile write cache to use the NVMe Flush command to flush the volatile write cache to
stable storage before the LAYOUTCOMMIT operation returns by using the stable storage before the LAYOUTCOMMIT operation returns by using the
Flush command (see Section 7.1 of <xref target="NVME-BASE"/>). Flush command (refer to Section 7.1 of <xref target="NVME-BASE"/>).
The NVMe Flush command is the equivalent to the SCSI SYNCHRONIZE The NVMe Flush command is the equivalent to the SCSI SYNCHRONIZE
CACHE commands.</t> CACHE commands.</t>
</section>
</section>
<section anchor="sec_security">
<name>Security Considerations</name>
</section> <t>NFSv4 clients access NFSv4 metadata servers using the NFSv4
</section>
<section anchor="sec:security"><name>Security Considerations</name>
<t>NFSv4 clients access NFSv4 metadata servers using the NFSv4
protocol. The security considerations generally described in <xref target="RFC88 81"/> protocol. The security considerations generally described in <xref target="RFC88 81"/>
apply to a client's interactions with apply to a client's interactions with
the metadata server. However, NFSv4 clients and servers access the metadata server. However, NFSv4 clients and servers access
NVMe storage devices at a lower layer than NFSv4. NFSv4 and NVMe storage devices at a lower layer than NFSv4. NFSv4 and
RPC security are not directly applicable to the I/Os to data servers RPC security are not directly applicable to the I&wj;/&wj;Os to data servers
using NVMe. using NVMe.
Refer to Section <xref target="RFC8154" section="2.4.6" sectionFormat="bare">Ext Refer to Sections <xref target="RFC8154" section="2.4.6" sectionFormat="bare">Ex
ents Are Permissions</xref> and Section <xref target="RFC8154" section="4" secti tents Are Permissions</xref> and <xref target="RFC8154" section="4" sectionForma
onFormat="bare">Security Considerations</xref> of <xref target="RFC8154"/> for t t="bare">Security Considerations</xref> of <xref target="RFC8154"/> for the
he security considerations of direct access to block storage from NFS clients.</t>
Security Considerations of direct block access from NFS clients.</t> <t>pNFS with an NVMe layout can be used with NVMe transports (e.g., NVMe
<t>pNFS with an NVMe layout can be used with NVMe transports (e.g., NVMe
over PCIe <xref target="NVME-PCIE"/>) that provide essentially no additional sec urity over PCIe <xref target="NVME-PCIE"/>) that provide essentially no additional sec urity
functionality. Or, pNFS may be used with storage protocols such as NVMe functionality. Or, pNFS may be used with storage protocols such as NVMe
over TCP <xref target="NVME-TCP"/> that can provide significant transport layer over TCP <xref target="NVME-TCP"/> that can provide significant transport layer
security.</t> security.</t>
<t>It is the responsibility of those administering and deploying pNFS with
<t>It is the responsibility of those administering and deploying pNFS with
an NVMe layout to ensure that appropriate protection is deployed to that an NVMe layout to ensure that appropriate protection is deployed to that
protocol based on the deployment environment as well as the nature and protocol based on the deployment environment as well as the nature and
sensitivity of the data and storage devices involved. When using IP-based sensitivity of the data and storage devices involved. When using IP-based
storage protocols such as NVMe over TCP, data confidentiality and storage protocols such as NVMe over TCP, data confidentiality and
integrity <bcp14>SHOULD</bcp14> be provided for traffic between pNFS clients and NVMe integrity <bcp14>SHOULD</bcp14> be provided for traffic between pNFS clients and NVMe
storage devices by using a secure communication protocol such as Transport storage devices by using a secure communication protocol such as Transport
Layer Security (TLS) <xref target="RFC8446"/>. For NVMe over TCP, TLS <bcp14>SHO ULD</bcp14> be used as Layer Security (TLS) <xref target="RFC8446"/>. For NVMe over TCP, TLS <bcp14>SHO ULD</bcp14> be used as
described in <xref target="NVME-TCP"/> to protect traffic between pNFS clients a nd NVMe described in <xref target="NVME-TCP"/> to protect traffic between pNFS clients a nd NVMe
namespaces used as storage devices.</t> namespaces used as storage devices.</t>
<t>A secure communication protocol might not be needed for pNFS with NVMe
<t>A secure communication protocol might not be needed for pNFS with NVMe
layouts in environments where physical and/or logical security measures layouts in environments where physical and/or logical security measures
(e.g., air gaps, isolated VLANs) provide effective access control (e.g., air gaps, isolated VLANs) provide effective access control
commensurate with the sensitivity and value of the storage devices and data commensurate with the sensitivity and value of the storage devices and data
involved (e.g., public website contents may be significantly less sensitive involved (e.g., public website contents may be significantly less sensitive
than a database containing personal identifying information, passwords, than a database containing personal identifying information, passwords,
and other authentication credentials).</t> and other authentication credentials).</t>
<t>Physical security is a common means for protocols not based on IP. In e
<t>Physical security is a common means for protocols not based on IP. nvironments where the security requirements for the storage
In environments where the security requirements for the storage
protocol cannot be met, pNFS with an NVMe layout <bcp14>SHOULD NOT</bcp14> be protocol cannot be met, pNFS with an NVMe layout <bcp14>SHOULD NOT</bcp14> be
deployed.</t> deployed.</t>
<t>When security is available for the data server storage protocol,
<t>When security is available for the data server storage protocol,
it is generally at a different granularity and with a different it is generally at a different granularity and with a different
notion of identity than NFSv4 (e.g., NFSv4 controls user access notion of identity than NFSv4 (e.g., NFSv4 controls user access
to files, and NVMe controls initiator access to volumes). As to files, and NVMe controls initiator access to volumes). As
with pNFS with the block layout type <xref target="RFC5663"/>, with pNFS with the block layout type <xref target="RFC5663"/>,
the pNFS client is responsible for enforcing appropriate the pNFS client is responsible for enforcing appropriate
correspondences between these security layers. In environments correspondences between these security layers. In environments
where the security requirements are such that client-side where the security requirements are such that client-side
protection from access to storage outside of the layout is not protection from access to storage outside of the layout is not
sufficient, pNFS with a SCSI layout on a NVMe namespace <bcp14>SHOULD sufficient, pNFS with a SCSI layout on a NVMe namespace <bcp14>SHOULD
NOT</bcp14> be deployed.</t> NOT</bcp14> be deployed.</t>
<t>As with other block-oriented pNFS layout types, the metadata server
<t>As with other block-oriented pNFS layout types, the metadata server
is able to fence off a client's access to the data on an NVMe namespace is able to fence off a client's access to the data on an NVMe namespace
used as a storage device. If a metadata server revokes a layout, the used as a storage device. If a metadata server revokes a layout, the
client's access <bcp14>MUST</bcp14> be terminated at the storage devices via fen cing client's access <bcp14>MUST</bcp14> be terminated at the storage devices via fen cing
as specified in <xref target="ssc:fencing"/>. The client has a as specified in <xref target="ssc_fencing"/>. The client has a
subsequent opportunity to acquire a new layout.</t> subsequent opportunity to acquire a new layout.</t>
</section>
</section> <section anchor="sec_iana">
<section anchor="sec:iana"><name>IANA Considerations</name> <name>IANA Considerations</name>
<t>This document has no IANA actions.</t>
<t>The document does not require any actions by IANA.</t> </section>
</section>
</middle> </middle>
<back> <back>
<references>
<name>References</name>
<references>
<name>Normative References</name>
<xi:include href="https://bib.ietf.org/public/rfc/bibxml/reference.RFC.4
506.xml"/>
<xi:include href="https://bib.ietf.org/public/rfc/bibxml/reference.RFC.5
663.xml"/>
<xi:include href="https://bib.ietf.org/public/rfc/bibxml/reference.RFC.8
154.xml"/>
<xi:include href="https://bib.ietf.org/public/rfc/bibxml/reference.RFC.8
881.xml"/>
<references title='Normative References'> <reference anchor="SPC5">
<front>
<reference anchor='RFC5663'> <title>SCSI Primary Commands - 5 (SPC-5)</title>
<front> <author>
<title>Parallel NFS (pNFS) Block/Volume Layout</title> <organization>INCITS Technical Committee T10</organization>
<author fullname='D. Black' initials='D.' surname='Black'/> </author>
<author fullname='S. Fridella' initials='S.' surname='Fridella'/> <date year="2019"/>
<author fullname='J. Glasgow' initials='J.' surname='Glasgow'/> </front>
<date month='January' year='2010'/> <seriesInfo name="INCITS" value="502-2019"/>
<abstract> </reference>
<t>Parallel NFS (pNFS) extends Network File Sharing version 4 (NFSv4) to a
llow clients to directly access file data on the storage used by the NFSv4 serve
r. This ability to bypass the server for data access can increase both performan
ce and parallelism, but requires additional client functionality for data access
, some of which is dependent on the class of storage used. The main pNFS operati
ons document specifies storage-class-independent extensions to NFS; this documen
t specifies the additional extensions (primarily data structures) for use of pNF
S with block- and volume-based storage. [STANDARDS-TRACK]</t>
</abstract>
</front>
<seriesInfo name='RFC' value='5663'/>
<seriesInfo name='DOI' value='10.17487/RFC5663'/>
</reference>
<reference anchor='RFC8154'>
<front>
<title>Parallel NFS (pNFS) Small Computer System Interface (SCSI) Layout</ti
tle>
<author fullname='C. Hellwig' initials='C.' surname='Hellwig'/>
<date month='May' year='2017'/>
<abstract>
<t>The Parallel Network File System (pNFS) allows a separation between the
metadata (onto a metadata server) and data (onto a storage device) for a file.
The Small Computer System Interface (SCSI) layout type is defined in this docume
nt as an extension to pNFS to allow the use of SCSI-based block storage devices.
</t>
</abstract>
</front>
<seriesInfo name='RFC' value='8154'/>
<seriesInfo name='DOI' value='10.17487/RFC8154'/>
</reference>
<reference anchor='RFC8881'>
<front>
<title>Network File System (NFS) Version 4 Minor Version 1 Protocol</title>
<author fullname='D. Noveck' initials='D.' role='editor' surname='Noveck'/>
<author fullname='C. Lever' initials='C.' surname='Lever'/>
<date month='August' year='2020'/>
<abstract>
<t>This document describes the Network File System (NFS) version 4 minor v
ersion 1, including features retained from the base protocol (NFS version 4 mino
r version 0, which is specified in RFC 7530) and protocol extensions made subseq
uently. The later minor version has no dependencies on NFS version 4 minor versi
on 0, and is considered a separate protocol.</t>
<t>This document obsoletes RFC 5661. It substantially revises the treatmen
t of features relating to multi-server namespace, superseding the description of
those features appearing in RFC 5661.</t>
</abstract>
</front>
<seriesInfo name='RFC' value='8881'/>
<seriesInfo name='DOI' value='10.17487/RFC8881'/>
</reference>
<reference anchor="SPC5" >
<front>
<title>SCSI Primary Commands-5</title>
<author >
<organization>INCITS Technical Committee T10</organization>
</author>
<date year="2019"/>
</front>
<seriesInfo name="ANSI INCITS" value="502-2019"/>
</reference>
<reference anchor="NVME-BASE" target="https://nvmexpress.org/wp-content/uploads/
NVM-Express-Base-Specification-2.0c-2022.10.04-Ratified.pdf">
<front>
<title>NVM Express Base Specification, Revision 2.0c</title>
<author >
<organization>NVM Express, Inc.</organization>
</author>
<date year="2022" month="October"/>
</front>
</reference>
<reference anchor="NVME-NVM" target="https://nvmexpress.org/wp-content/uploads/N
VM-Express-NVM-Command-Set-Specification-1.0c-2022.10.03-Ratified.pdf">
<front>
<title>NVM Express NVM Command Set Specification, Revision 1.0c</title>
<author >
<organization>NVM Express, Inc.</organization>
</author>
<date year="2022" month="October"/>
</front>
</reference>
<reference anchor="NVME-TCP" target="https://nvmexpress.org/wp-content/uploads/N
VM-Express-TCP-Transport-Specification-1.0c-2022.10.03-Ratified.pdf">
<front>
<title>NVM Express TCP Transport Specification, Revision 1.0c</title>
<author >
<organization>NVM Express, Inc.</organization>
</author>
<date year="2022" month="October"/>
</front>
</reference>
<reference anchor='RFC8446'>
<front>
<title>The Transport Layer Security (TLS) Protocol Version 1.3</title>
<author fullname='E. Rescorla' initials='E.' surname='Rescorla'/>
<date month='August' year='2018'/>
<abstract>
<t>This document specifies version 1.3 of the Transport Layer Security (TL
S) protocol. TLS allows client/server applications to communicate over the Inter
net in a way that is designed to prevent eavesdropping, tampering, and message f
orgery.</t>
<t>This document updates RFCs 5705 and 6066, and obsoletes RFCs 5077, 5246
, and 6961. This document also specifies new requirements for TLS 1.2 implementa
tions.</t>
</abstract>
</front>
<seriesInfo name='RFC' value='8446'/>
<seriesInfo name='DOI' value='10.17487/RFC8446'/>
</reference>
<reference anchor='RFC2119'> <reference anchor="NVME-BASE" target="https://nvmexpress.org/wp-content/
<front> uploads/NVM-Express-Base-Specification-2.0d-2024.01.11-Ratified.pdf">
<title>Key words for use in RFCs to Indicate Requirement Levels</title> <front>
<author fullname='S. Bradner' initials='S.' surname='Bradner'/> <title>NVM Express Base Specification</title>
<date month='March' year='1997'/> <author>
<abstract> <organization>NVM Express, Inc.</organization>
<t>In many standards track documents several words are used to signify the </author>
requirements in the specification. These words are often capitalized. This docu <date year="2024" month="January"/>
ment defines these words as they should be interpreted in IETF documents. This d </front>
ocument specifies an Internet Best Current Practices for the Internet Community, <refcontent>Revision 2.0d</refcontent>
and requests discussion and suggestions for improvements.</t> </reference>
</abstract>
</front>
<seriesInfo name='BCP' value='14'/>
<seriesInfo name='RFC' value='2119'/>
<seriesInfo name='DOI' value='10.17487/RFC2119'/>
</reference>
<reference anchor='RFC8174'> <reference anchor="NVME-NVM" target="https://nvmexpress.org/wp-content/u
<front> ploads/NVM-Express- NVM-Command-Set-Specification-1.0d-2023.12.28-Ra
<title>Ambiguity of Uppercase vs Lowercase in RFC 2119 Key Words</title> tified.pdf">
<author fullname='B. Leiba' initials='B.' surname='Leiba'/> <front>
<date month='May' year='2017'/> <title>NVM Express NVM Command Set Specification</title>
<abstract> <author>
<t>RFC 2119 specifies common key words that may be used in protocol specif <organization>NVM Express, Inc.</organization>
ications. This document aims to reduce the ambiguity by clarifying that only UPP </author>
ERCASE usage of the key words have the defined special meanings.</t> <date year="2023" month="December"/>
</abstract> </front>
</front> <refcontent>Revision 1.0d</refcontent>
<seriesInfo name='BCP' value='14'/> </reference>
<seriesInfo name='RFC' value='8174'/>
<seriesInfo name='DOI' value='10.17487/RFC8174'/>
</reference>
</references> <reference anchor="NVME-TCP" target="https://nvmexpress.org/wp-content/u
ploads/NVM-Express-TCP-Transport-Specification-1.0d-2023.12.27-Ratified.pdf">
<front>
<title>NVM Express TCP Transport Specification</title>
<author>
<organization>NVM Express, Inc.</organization>
</author>
<date year="2023" month="December"/>
</front>
<refcontent>Revision 1.0d</refcontent>
</reference>
<references title='Informative References'> <xi:include href="https://bib.ietf.org/public/rfc/bibxml/reference.RFC.8
446.xml"/>
<xi:include href="https://bib.ietf.org/public/rfc/bibxml/reference.RFC.2
119.xml"/>
<xi:include href="https://bib.ietf.org/public/rfc/bibxml/reference.RFC.8
174.xml"/>
</references>
<references>
<name>Informative References</name>
<reference anchor="NVME-PCIE" target="https://nvmexpress.org/wp-content/uploads/ <reference anchor="NVME-PCIE" target="https://nvmexpress.org/wp-content/
NVM-Express-PCIe-Transport-Specification-1.0c-2022.10.03-Ratified.pdf"> uploads/NVM-Express-PCIe-Transport-Specification-1.0d-2023.12.27-Ratified.pdf">
<front> <front>
<title>NVMe over PCIe Transport Specification, Revision 1.0c</title> <title>NVMe over PCIe Transport Specification</title>
<author > <author>
<organization>NVM Express, Inc.</organization> <organization>NVM Express, Inc.</organization>
</author> </author>
<date year="2022" month="October"/> <date year="2023" month="December"/>
</front> </front>
</reference> <refcontent>Revision 1.0d</refcontent>
</reference>
</references>
</references> </references>
<section numbered="false" anchor="acknowledgements">
<section numbered="false" anchor="acknowledgements"><name>Acknowledgements</name <name>Acknowledgements</name>
> <t><contact fullname="Carsten Bormann"/> converted an earlier RFCXML v2 so
urce for this document to a markdown source format.</t>
<t>Carsten Bormann ran the scripted conversion from the XML2RFC v2 format <t>David Noveck provided ample feedback to various drafts of this document
used by earlier drafts to the currently used markdown source format.</t> .</t>
</section>
<t>David Noveck provided ample feedback to various drafts of this document.</t>
</section>
</back> </back>
<!-- ##markdown-source:
H4sIAAAAAAAAA71b63IaS5L+X09Ra/8wzAIWsix7FDMTBwOyiZEQA7LPeC+h
KJpCdKjp5nQ3yIRD77LPsk+2X2ZVdVc3yOec2LP7R4KmLplZefkyK7vdbouF
yvWFPD05PWufdNvdE7FIglit8WyRqmXeDnW+bMfLbHfWzoIsbEdqn2zzdrxb
6/bJO7G7kG9EoPILmeULkeWpVusLORreXopNeCEkHqdhgJ9f7XX2Ct+DZL3W
cZ6VT8I4CmNdfteLMA/jezyIE/qeJ0H5I74k643yV8Sjhd7kKzx5Q9+z/TrV
S2+DLEnz6pODNbLtvHxG2+ZhHoGmzxkokflKy4lKVRTpSI4vZ7Kxwd+mnPVn
I3nFAgEVUgWBzjI5/nKtwXaSqnstF3oX4qlQ83mqISyaWJm3TFKeIRREdyFv
UxVnGxAsHiEBDN6diYdH90mobb5K0gvRlmlC9JGskhQMmCPrr9IQO29W8pOO
osfwnsS5VmF0IVfB6qcoyzsLjclutEojnckrvdO0RpJiy5tUBZGW/SQFFSoP
kxi/OOrNjyzAbZynewgoDnO9kLMcapTJZCl7a40DV+XGwWobPHQi2uOnhOd3
IOqCiFmShrG8VOE8zFYFERsdL9JwpzMey1qkNc6m25UzHT3uYzlN1IIICXNQ
MdaPOROa6ntQfCGve0zkgtQKqn3effX7iM46S0PRT0mdlM0qIW391648775r
v+2etE/OTk/BTxhDwwYd+SFSwUNxJAO1CxfyqnzMDA5wPPJWB6s4iZL7EAri
8fjuHFLBQYNAelJw+SnZPITxDxjtvjt7/zsZXRB5nTnR9tMCRDGTIk7SNc5+
p8mCp5f9t+fnb+zH9923Z+7j+/dd+jib9N/Sf5iisRrW70karlW6hyat1ype
ZO23PMRpMH1uG2GMxv3R7cyIA8RFPCXMc63lLRwSjXReqvtn/pqBCw3TXCZm
ISl7Y2xpFrqQb09O23YsTGvY/tCbDS2BKr0nGa/yfJNdvH5NbuzbJoXZdkDK
68dNO0jiHP7p9XYTQcOy11igPTRD2h9UptuzjQ7CJegk22ifdk4C7HV62ume
dE7O2lM8XoZ60dkslr5IsIy0y0haRlaWackp/ESGT5IWfE5Q3iItOYqDTkU2
p6dtlhazjD9/BMf02R5ge6bzGvPdCvNvfhPz9NmuCFPOn5VD94+Rw21/8kfI
Acu0C9f8B0gB65Wu/v9QBmSlZ2fnF4JsxbNpls2kP/pDzALr6D9COlomCBKS
lvt/EE673UZcg9NF0BdC3K7CTAL7bAmdyMxsCqe5Sh4ptm9hslUYoPPHJH2Q
lyGi5Wyf5XptYYGYrTGElHyzzcGP/XEEAaZLFWjZIP/YdADgdr/RHnqoA4dt
gUDGkOaXJIIcsOO1Xidwrk6fGiS+ptykCQGkSC7VOoz2HfBFbK7DxQJhW7wk
ItJksQ1IlvL7y0wHFyE9ehKCEUanKxsIx9+/W/f+9NREXAui7QKiUAa9LLXK
tymJQ+USnCaPmQB2WWAAbPoxDSnWgKG5lhtwDLVDCJrvZQImUrnWOFiaG8tF
mOqAoJ6k6TRb8GyJmGugB69DvMPhQzU6Ut6u0mR7v+LzQDzL6dQsQS1vZAsK
z18JJtHAtc4Vzl/Z3yWmQXAbbBHOMYJQGFySuidqliRgosYQSHIupj+u6MdM
b6AHoNRwA+hGokckBae0lP6mgy3LOFkeyKZDyqaJfk8gINEKq6DgQKa0zUIv
AZZJnuBwEWLLLQLmWm02rCbQpYxoDFZYEQYrV2rHihumMnmMnV4bY7KUWEz6
25TWwH+rIIACT08kSmWfi5yVuVQMRstBFBLip7M0Eo32jlk5en3DuhIlwcOB
7hthz/cblRVWcO1OYmYOsnE9mDWhGKOcCHF2ywrHpiPguQKkB1CTNFnzEgxP
aqbCx0Y/8tobBfTlVK9KFGTmO3JjeC1pZx+z0U9JlkOwZGbwHGkpU1FzbSzI
DDERylA9KKZFRQ/0v0YP+0yhZLxdz7E46ZHlDNLYQg9URh61JJjcKwi+DAHp
KQWIY3izxmW/ySqPwPR6NCF2pqAeGj7gE3O89IyTakwH172mjI0PhJQ70jhw
sBBs0xTHHe0FCQTHsFQpHCv0WsGxFmKHAS88t8dOf5ZEoUWr2Dd7wEaz2SBr
tgRZAfwQTILOFMelFskmt9aWGkqdYOyKjzAuEJTj5w3gLdSJzr09V7TxoYxC
+rVTjwPG3EwUYAbHAPXZhpaTnmeuYZrv3x0Ge3oSCKQIY0YjeYlSFbDrehNp
2skuJXgEA8SKBrglCcnC5MgTGA/LYlSZqGtFSZyfbFKs6bDNM2MhLw1+4Uo0
5TnEMntVLIxvabTnVYpYzLuBj4JST62J+Xy1JYluaHD2PKNwo3CVcqeA4vO9
oB3dFojXJt7QOE9vEfihci0GTrRTRX0Pjy2BCOIEXghfF8yP/obMmBZleVg3
5iZAfYFLVhw3cvGjQw8Qt5zYH0OEn/hAyHDNOsvVPEL6aLh17tmKlgfDLQF7
IBJnZjF/IetMC3Zw1EGSmnjFgmGqyhXA/8uXMNhftjBWrq6Alvh+SwqBIJ/Z
IH+R6l8iPH8ybv9B7yXMFzHmxfXn2e2Llvkvxzf8eTr8x+fRdDigz7NPvaur
4oOwI2afbj5fDcpP5cz+zfX1cDwwk/FUVh6JF9e9r/iFzvHFzeR2dDPuXb0w
EvBPsdTykFwm9CA3yg4wEqTh3EjtQ3/y3//VPYOF/Ati0mm3+2cYiPnyHgkx
vsATxGa3JIbrMF8h7L3AuWh4J6xCwS9QmxBeFroGf5CtKF6SD4F0//TvJJn/
vJB/mQeb7tnf7ANiuPLQyazykGV2+ORgshHikUdHtimkWXlek3SV3t7Xyncn
d++hYDX6qGOgr0gOCg+RVZQI1pFZDVomFORJIRfe4AM0RHq92aabxGA28xuF
ZtgSTiBNNmlILp9zjG95MYcAEECfEH0GEOJC0qYvDJx4QX6LDRs+Jd9b0MGu
X5sfCHoYtPcq44pJlmxTiuCS1zHLwDb3pGGGlMi5W16N6FEho1WqeVGhJvBC
VnWPEtqAeoK+Ki18qcWz8T1XJxlY8QZWSpmLYAz77ABalT05ScPBAirFlVSS
3YujeMiK6uhvNbnVUXCQwCUgVltyrZQcy0mJUDxiDUqlOEIRpMBfDmxT0esx
5pOEfo23pgZFYDPe0fi6hkHuO0pIioE7FW0hCYd9fVdZxUktqTv3nRbMn2pS
T08tG5VstHpusBdaW6y+gAyQC2jDbmUkzRAGCE4WRCuSZFKP1RZ6kFORr+av
EA+Xy/AbqEdA4ixqDvGme9EgKCJPmxa6OXK63ZO5/Ks8x4QgBCg3PEA/X60O
1xIr/U3ZgdIs2D0/WBEI56/y9L1bkdTmZSVkFflDYkKLyQyzaG0NvRIzS+Tg
ZwGpXoLpmBEpeVkllvqRJ7addGQBxZ2tsCn6Lr+wiCJoCgfbszJwmkUMsQWw
LZ+zJ0hNkBbk7b0zLOvvlh0K4obpuIjyRlGB40GUHFmddkCMFXVHSBXPn/wM
ylt165zQgPFYfZEvFGfkxKTickAm2vgyGTTFhkJ2g/9SUVe+f7NqmsTFqTSx
bc1sbxKWAvGZtNDlYh05cpjLJREkGEqca+xyuKulXpB+bVhp3KlzBNu59QVE
h/ez4J1+wP5kIJnHAiqS5zHyIyerIrGFinF+6qPcg9yniC41+bfqLAoO13Y7
qUPOvGnmaDgcQoVyAr8Lqpf/si3JJZc5/Dw6P2vCqYsCBcqPUTIHYNgfGz/+
+Hk0aBqXRX7EYH92BsUCoiaPAeOZDZgzNQxKiv18YuTBczt175HD+jNjKMhl
EOvziqHy2aHimczF+EJo0FKqnQojgFmsW9RcoEHEpWUyzMSGrT81hxTmLsRE
VFZMPc2g5DmhIhwZzWR2NxjORh/Hvdub6d1t94QJrz4d966HpoYRJ5LwLbYk
KAB58gnTDEBygvpOTXz7OHYV54oe8TK7oxvNO/Kadzs29jsqk57JzAnIBN+4
pk3C4Le5B8EN5wYPZRdCdDsGq2Tz3R0Z8h0M5QVCpo4gZzebjAfUguP+zWB4
Nxve3n0YjXvTr51ygZJMeMXwHnE5Se+oyvLMaoi1VQmy/naqFJVLVVexgMe3
D3PQfMVIX3k170ALCyyEQ6WYpZwn5FV5Lh0RT6OrXs+FkIf2lCsnP13u6G1h
QfCcMJhzBD/WMUQlSWGpEqDKkm7+vBwUQRn5z8GUk1OiTEDf7sFMslEw9f/4
y9+MHN3TpfGqNKMYYe2iYSEgUsqMBxSogq2eCqsL8gBFxc5UUErGwaGdQDk7
drGbsi+bU6SXCVTPuk+npgfyo7KOm/L+2Iz6oeIIe0iHTHnS7GkdKCUYfvZ7
aBsdk0QYxC4vAQgo8pqguTTfjmAKDpcTaAVydJo31YQcbdhqTKZZkwzFggOL
SYVdDtReshKGruqsglWod3rRMmrIVcPBrHCLfVuONKXRRNBVakY1T2VS4tDO
qfqN0q3SWivFm1FBCrhdibQkWCbxkemdesaE4NnOdGBYdJwpudxGUb1ekJrE
fiEmw+lsNLsdjinNmw2nX4ZyNDZe8/CXm8+3DjCZC9gCQggH84pfjP45JTHH
W+Am8etw4dlAzEjqpcQJyrb8O6Q7ZWGnPpay53gB4VNieVNXKQB1X7w0zIJ0
c24+Rhee5ti9dOr4lA0OU3RwMyN5+a7zhqRcCXtNUajA0dV6PFU0ptPhx17T
Og7ry09OAN4bYUd3WuUEfxXIwBRaCYvY8hofcYVBl+KMAaFrs4Ewpn8ffm0K
3pec3THZbONj0jnKj3DS+S1cy2Nci5OTbsH152Lno3zLZ/kWHt99U0M+5L3P
vEvH+02sqbIIjEW15UW4tDkItDh/1FBg04lw3LFwUZlVreJuQguU+ZdDxYse
Ff0jJsiOuG7k1TldhVTJFZX9Gyor3LyrnJqV6VZAlMCx2TGZ0FzDLYUJO38V
K8DhZMuuBqJ2RWKJWH13+/FucivmCIRsqU7opuDIbNcUo+WSdMuCBcNwK64w
7YsBaC+Owgdd3Y/t3DgLk31bQA0IGMYEhqyYq7As1RFVBLabJO6IaYXQut6b
oOIlHcbDkiyEyrIkCM1Yd75WzJwh+xikZ57zQRLL1smnyhS4/R2oAETIVsV7
7yQNJYZVWkXwerSUpQIMWY3nTTgg2BqBufEpz7YlDvzO287pu063c/r2iPfx
K0xYuaSUwIksGTGh1jlXikoV50oy8VWg6myxDHztB40DpQMC2I3NiUzgwe/o
cO++3Fx9vh7eEVnSwGN3JWYCcMtFQxFrvchMhKaLUaM0dgppjN3UOiLQa5Wd
M/+YLomEr791h3jPFcncqyYSs4W7MjFINDKtIcqDiPLUtFHXGaYvlF7AoVX8
KEKcHp4Rby7qvtIuVrrKXv8fPwoQdnxTOGThL8ZtAY3p7dfJsLbE2cotMPwW
RJDpTtt7OdEuVIAAzg3VYbw1m05jHCzrBTXN0IqfQDNGMQzY5HswliRue7cP
tePn47Ww2pYL6QrEXIGzTh0T0u+UeOt4UK6KXDwj8jI6TVKt15uc8nl50j2p
P+fI3JvDpTVbvz0QuY0If7iCc92zmeXcNgfLTbzlRLncgX/0vE8hj6On87/X
KWHven9Vp7xiesBdHXzPtg2zFRcydJomCIyLLbsP5wq4psUZhrADzMWsBc8+
1f0kXkZhkLtsXuXbjKtjTp1tsjHVAd2FQ0hLg+YvGQpUlDy1Y0jDl0X24nQx
zLJtGaYtT0fbAKzLzIQzjjzdU7ZouPWUmotg46mkQO3Vry/Deyov/PkUX0RN
0/3NGZIH3A7JUMOgahMZhOtJqpGW2xYZa54tSyvPp8nASqwodi1RdD7YPAff
73XuPZKjgQEPJao8Bt469gKp6H8wzTwB0jHtCvzmC6R/adueK+BJUdA4mEth
QsckXYhuyZI8samRKPb6mcf3eXzjy899p98WUBalMO/Cmmphwiub/ajqxsHF
Htrpu7cyjOtRm5pPdF4kiUcJGzIXsvFzH/ZHfFibY5ZOqCAgnp17abuuGu6D
V3Q8OV81Tfyr4YuzI6HLENph/f9VeTOOqBeCXRFG1boOflCOLZ5m/k0UaXeh
xbzJJXzPqrBH8hf8wOKJQ1KRfPBVe2kEcwNpaMZV7yuSYLoOHd2WPWXOdIsO
IVak6sYVWUKQh2I0bu8Ize52rahVOi/A8ph9Hfc/TW/Go38bin6v/2lY5ODG
emjTbUr3cn1C04uiD87eyNhfXbte0VrlXj3gh7Vet0qnCr9G4LpfTCnLLcrV
TG9LA7uozF25b/d6AwWnQeYywFDyKjN39coWNgqcVKOpA4z8SK8EtGSNEUoR
LdmGKXGsiCvpikcSqEtJx9ghkaJyD6NdkmL5dNIv+SNkSilJcVNr71JJfewh
wQeYPjVPfLYFk8ig9MU69u/fnX6cds4657LBlwjgoIddkHGuEUxIBCbrdWPP
ROOZE26SjpXVSmtK4jl9wGDDhm2dswrAYdXruYNWsdlVinW26FZvZTE/lp04
smGuD/ntlLIz19oBNQ+TO/EvsiVdvsMpsdLE0IrFgi8KVVQcglhu48A8w7eO
vIEGMIX2Hr6kxp34YatWSRB1A1l68JEux/jeHow5iqjAyxctZIhFHxOrjHA0
QUijop5c3IeHRKCJM3TFqxbrMC4zV5wpokOUcCWjELGoiRh6ouOsbJb1mh2I
LasUlAbxWub2gmN70SdXgUVmGN+U6ngXpklsGmVgaPRCic1BYxMguMqE7XEE
u4IV2+DIZlYzqTCGi93pRUfKn8tL09HEtMyJH5+HdOfRMhvAlyxNRZkPmokh
z3DPylyU9GuNIqla0j2xy+43fveoq9YctLoVflwZLTN4bhu727VClI7e8h2r
K3YdhY01bq8QlYyDOzs7pxuwAqeU/GGQx4BrwKv5SF8nE3fYv5HB2O8xPHrx
CY3t/Qqz6/B+lUtbgaEE3cq4dAe8lwOUdOtTapTrntys9hl3YIC215hsb2dL
l7rWitQ7E9ZXqDCV92pDXXwZxWts+uWqN86apYtYLkntd0WfpgWAwrwYiNXI
OsoivKfAJCBztWJV+SAokFUSqnOq7FzYZjuHo4eRzLPQNRoRl9bpeE4Cjivi
Nny7rxYcWBSvy/0V9pKM7R7Rgb2bu3Ckh8WbFtTSS53L3GNnWljNlQq9sEDj
7ZEFqbaGklGGPnEyL4TMfcEkHgw23eJ8kIUd8iE7NzGadCh5P3KYuR/rU79Z
0OE2K8/S+ZQ1PMTvlnw2lJR9anRN55wZmGFHUmHE3fhVG64tLKz7GGrHpFkl
FOG47wq9ubyHKW8jlTr1sPXDYoAA+aYN356R6RSzSKGIcAaCGEVko0sd+CAI
CtCZtQrzLMdxGwzdHnrtUabqlVFHei8zNYtSasSuCdcuPlBCzv6G3rB7emqJ
AijbBPDIewqaFIyTaC+eiLJB1FRjnIsxnTPFEXDkyzqypiLi11SEsBP7TxNh
mbg2ARLhBTJGH6Us3GmSgyHTt1ZreQ9ZcQV3NAWmoOTpV+V6MOHuiWoSYlRO
2BtJT+V6BnVaW2Nxt5M0NM1cvIMn/MykJjV0SsmcQ4Vc3ALpSx/jljwWGnws
UxJFplS/DeRLcnXwVkqqd8mDLt+nYOpEfVt3T5cTyozZydp2m7pD3IXKVVyE
8t+O4Bjl38k+VbsiVya9owafX7bcFc6FfurKMXjf1t0UQstjeddHbxj1xr3j
yUuoYmXvfg8bta2ycRXeJQ8I7LSae4tpTu/PYode8BAnj8hO741uiu8X29h0
vOkFNuirlG4h5AdywzHyPWUvZLnHBsxzC1+aFRpLP/7z+uqU2gZ2p9K4b+Fu
crRKI0qy+XX44syLtxxMiF6r9GHBL9hwj6ldA5Sb13/HQA+w+gLsKOrMwsno
BTHFfgM+jG567C7u1aaiQV2I/wFd0piarT8AAA==
</rfc> </rfc>
 End of changes. 71 change blocks. 
511 lines changed or deleted 269 lines changed or added

This html diff was produced by rfcdiff 1.48.