Enclave Messages: The Enclave's Logical Interface

The SVR2 enclave interface defined in svr2.edl is generic. It provides initialization and message passing functions that are independent of the application logic. The logical interface of the enclave is defined by these messages and how the enclave responds to them. In what follows we will think of the different messages that can be sent to the enclave as RPCs and refer to them as "calls" or "commands".

Three Interfaces: Host, Peer, and Client

SVR2 enclaves interact with three different types of entities: the host that makes ECALLs and receives OCALLs from the enclave, other peer enclaves, and clients that are using the service to store and recover secure values.

The host interface includes a number of administrative commands (create or join a replica group, get enclave status, tick the Raft timer, etc.). It also has commands to forward wrapped peer or client requests.

The peer interface includes Raft protocol messages, attestation updates, and a number of other "Enclave to Enclave [E2E] transactions" used to get information about a replica group, transfer database state, and join a replica group.

The client interface is the raison d'être for SVR2. It allows clients to backup, restore, or delete a secure value. Everything else in this system is here to ensure that this is done securely and reliably.

We will use this abstraction to organize this document, but it does not align perfectly with the organization of the code. The code organization reflects important implementation details as follows:

All messages to the enclave are sent in an UntrustedMessage (shared/proto/msgs.proto). These may be direct commands or forwarded messages from peers or clients.
Host calls that will not trigger response messages are sent as a simple UntrustedMessage. We will call these synchronous host calls.
Host calls that MAY trigger response messages are sent as a HostToEnclaveRequest inside an UntrustedMessage. It is important to note that all client requests are sent this way.
HostToEnclaveRequests are further subdivided into administrative requests and requests on behalf of clients. Client requests may be Noise encrypted (backup, restore, delete) or unencrypted (create new client). Encrypted client messages are defined in (shared/proto/msgs.proto). Unencrypted ones are defined as submessages of HostToEnclaveRequest in (shared/proto/msgs.proto).
Peer calls are all sent as PeerMessage messages inside an UntrustedMessage. These messages contain raw bytes that either hold handshake information or a Noise encrypted EnclaveToEnclaveMessage. These messages are defined in enclave/proto/e2e.proto

There is another important property that we will note on all of the calls we describe: some calls require that a new Raft log entry be accepted and committed by this node's replica group in order to complete, others do not. We will say that the calls that succeed or fail based on whether a log entry was successfully committed "require Raft consensus".

The Host Interface

Synchronous Calls

There are two messages the host can send to the enclave that act as synchronous calls - once the ECALL returns the action is complete. No messages will be sent from the enclave in response to these calls. None of these require Raft consensus. They are:

TimerTick passes a unix timestamp that causes the enclave's to update its internal time (which is used to obtain a consensus group_time with its peers), then perform a RaftStep.
ResetPeer lets this Raft instance know that the given peer ID may have lost some of the messages we sent to it previously.

Asynchronous Calls

All other calls from the host may cause the enclave to send response messages that must be handled asynchronously. These are all sent as a HostToEnclaveRequest inside an UntrustedMessage. These calls include:

Reconfigure (enclaveconfig.EnclaveConfig) Reconfigure the replica with new host-supplied configuration.
GetEnclaveStatus (bool) Retrieves basic information about the status of a replica. Has more detail if the replica is a leader.
DeleteBackup (DeleteBackupRequest - requires consensus) Used by host to delete a backup, e.g., when the account is deleted.
CreateNewRaftGroup (RaftConfig) Request that we create a new raft group from scratch, setting ourselves as the sole member and leader. This should be done to seed a new Raft, after which we should requst JoinRaft instead.
JoinRaft (JoinRaftRequest - requires consensus) This tells the enclave to join a particular Replica group. This call requires that the target raft group be up and running. Raft joining is a multi-step process described in detail in Healing.md. In this process there will be an enclave-to-enclave call that creates a new Raft configuration. This change requires consensus of the existing voting members. If successful the enclave will be a non-voting, up-to-date member of the specified Raft.
PingPeer (EnclavePeer) Tells an enclave to check connectivity with another peer.
RequestVoting (bool - requires consensus) Tells an enclave that is already a member of a replica group to request voting status. This requires a new Raft configuration to be accepted by a majority of the voting members of the new configuration.
RequestMetrics Get all metrics and gauges collected by the enclave.
RefreshAttestation Refresh attestations for peer and client connections.
SetLogLevel Sets the enclave's logging level with an ::svr2::EnclaveLogLevel enum. These enum values match Open Enclave's oe_log_level_t.
RelinquishLeadership (bool - requires consensus) If we are the Raft leader, give it up and attempt to pass leadership to an up-to-date peer without waiting for the election timers.
RequestRemoval (bool- requires consensus) Request that this replica be removed from the Raft group.
Hashes (bool)Compute and return to the host a hash of the current DB.

The Peer Interface

Peer to peer calls fall into three categories:

Raft messages
Connectivity messages
E2E Transactions

Raft Messages

The Raft protocol messages defined in enclave/raft.proto closely follow the Raft protocol defined in Ongaro's thesis.

Connectivity Messages

These messages are defined in enclave/proto/e2e.proto.

Connect (e2e.ConnectRequest) Sends attestation and handshake information to initiate a connection with a peer. The response to this call contains attestation and handshake information for the called enclave.
AttestationUpdate (Attestation) sends a new attestation to a peer so that peers can ensure their long-term connection with another enclave is still secure.

Enclave to Enclave (E2E) Transactions