SHIFT: a System for Handling Impeccable File Transfers

1 Summary

This spec suggests a modern, extensible alternative to Zmodem which supports directory transfers and is transparent to both Unicode processing and 7-bit serial connections. All data is incapsulated inside OSC control sequences, which are transparent to terminals and terminal emulators, and won't corrupt terminal state if a connection is lost or one of the sides doesn't understand the protocol. Unlike Zmodem, it also allows normal terminal I/O to continue while a transfer is in progress.

Once both sides have announced their presence, any side can offer an outbound transfer, or suggest to the other side to initiate an outbound transfer itself. Multiple transfers in any direction can happen within a single session.

A single transfer can contain either a single file, or a single directory with its contents. If one side accepts an inbound transfer, it grants, in case of a file transfer, write access to a file, or in case of a directory, write access to any number of files within the target directory. The other side can then open files and write data to them.

2 Definitions

host: the controlling application, typically a terminal.
data stream: the main communication channel between the host and the application, in most cases a PTY or an SSH or serial connection.
packet: a single unit of communication on the Transport Layer
message: a single unit of communication on the Message Layer
transport prefix: a byte sequence identifying the start of a packet:
```
\x1b]1337;SHIFT;
```
transport suffix: a single byte sequence identifying the end of a packet:
```
\x07
```
transfer: an operation involving sending one or multiple files.
inbound transfer: a transfer that's incoming from the point of view of the process.
outbound transfer: a transfer that's outgoing from the point of view of the process.
sending: an outbound transfer from the point of view of the process initiating it.
receiving: a inbound transfer from the point of view of the process initiating it.

3 Protocol layers

Transport Layer
Message Layer

3.1 Transport Layer

Provides a communication stream that is transparent to the terminal.

Transport layer encodes/decodes packets into/from the data stream. Packets can be written into the data stream in chunks, but cannot be interleaved with other data.

Note

This encoding was chosen so that:

Output is ignored by unsupported terminals as it's an unknown OSC sequence.
Echoed input is ignored by unsupported terminals for the same reason.
It's transparent to Unicode processing and 7-bit-only channels.

3.1.1 Packet structure

[packet prefix] [base64-encoded contents] [packet suffix]

3.2 Message Layer

Encodes structured protocol messages using Protobuf 3. The protocol definition is available at https://github.com/shift-protocol/shift/blob/main/shift/proto/shift.proto

Note

This encoding choice ensures future compatibility, should new fields or messages be added to the protocol.

Messages are encoded and written into the transport layer one by one. A message may not be split into multiple packets.

3.2.1 Messages

Message: this is the top-level wrapper for all other messages and the only one message that is read or written into the transport layer.

message Message {
    oneof content {
        Init init = 1;
        Disconnect disconnect = 2;
        ReceiveRequest receiveRequest = 3;
        SendRequest sendRequest = 4;
        AcceptTransfer acceptTransfer = 5;
        RejectTransfer rejectTransfer = 6;
        OpenFile openFile = 7;
        FileOpened fileOpened = 8;
        Chunk chunk = 9;
        CloseFile closeFile = 10;
        CloseTransfer closeTransfer = 11;
    }
}

Init: one of the sides announces itself.

message Init {
    uint32 version = 1;
    repeated string features = 2;
}

Version

currently 1.

Features

currently unused. //TODO

FileInfo: a sub-message describing file metadata.
```
message FileInfo {
    string name = 1;
    uint64 size = 2;
    uint32 mode = 3;
}
```
```
mode
```
POSIX file mode, the host is free to ignore all bits except for "is directory".
ReceiveRequest: the sender would like the host to initiate an outbound transfer by transmitting back a SendRequest.
```
message ReceiveRequest {
    bool allowDirectories = 1;
}
```
SendRequest: the sender would like to initiate an inbound transfer on the host.
```
message SendRequest {
    FileInfo fileInfo = 1;
}
```
AcceptTransfer: the sender accepts the transfer requested from it in the last received message.
```
message AcceptTransfer { }
```
RejectTransfer: the sender rejects the transfer requested from it in the last received message.
```
message RejectTransfer { }
```
OpenFile: the sender would like to start transmitting a new file within the currently open transfer.
```
message OpenFile {
    FileInfo fileInfo = 1;
}
```
```
fileInfo.name
```
for file transfers, only . is allowed. For directory transfers, it's a path relative to the directory being transferred.
FileOpened: the sender has the file requested in the last received message for writing and is ready to receive chunks.
```
message FileOpened {
    uint64 continueFrom = 1;
}
```
```
continueFrom
```
if not zero, the sender would like to start receiving chunks starting from this byte offset.
Chunk: the sender is providing the next chunk of the current file's contents.
```
message Chunk {
    uint64 offset = 1;
    bytes data = 2;
}
```
.
CloseFile: the sender would has finished transmitting the current file and would like the receiver to close it.
```
message CloseFile { }
```
CloseTransfer: the sender has transmitted all files and would like to close the current transfer.
```
message CloseTransfer { }
```
Disconnect: the sender would like to immediately cease communication.
```
message Disconnect { }
```

4 Behaviour

The protocol's behaviour is defined as a finite state machine with the following well-defined states and transitions between these states.

4.1 States

The protocol's behaviour is defined as a finite state machine with the following states:

Initial: no connection has been established.
Connecting: sent an Init message and waiting for an Init from the other side.
Idle: ready to start transfers.
InboundTransferRequested: sent a request for an inbound transfer, waiting for a SendRequest.
InboundTransferOffered: got a request for an inbound transfer, will accept or reject.
InboundTransfer: an inbound transfer is in progress, no file open.
InboundTransferOpening: an inbound transfer is in progress and a file is being opened.
InboundFileTransfer: an inbound transfer is in progress and a file is open for writing.
OutboundTransferRequested: sent a request for an outbound transfer, waiting for confirmation.
OutboundTransferOffered: received a request for an outbound transfer, will send a SendRrquest.
OutboundTransfer: an outbound transfer is in progress, no file open.
OutboundTransferOpening: an outbound transfer is in progress and a file is being opened.
OutboundFileTransfer: an outbound transfer is in progress and a file is open for writing.
Disconnected: the connection was closed.

4.2 State transitions

Note

Here, an arrow (→) denotes a transition to another state, emit denotes sending a message, and received denotes receiption of a message, which must then trigger its transition unconditionally.

The possible state transitions are:

From any state:
- Abort connection: emit Disconnect; → Disconnected.
- Received Disconnect: → Disconnected.
From Initial:
- Received Init: emit Init; → Idle.
- Initiate connection: emit Init; → Connecting.
From Connecting:
- Received Init: → Idle.
From Idle:
- Received SendRequest: → InboundTransferOffered.
- Received ReceiveRequest: → OutboundTransferOffered.
- Request inbound transfer: emit ReceiveRequest; → InboundTransferRequested.
- Request outbound transfer: emit SendRequest; → OutboundTransferRequested.
From InboundTransferRequested: Same as Idle. // TODO ?
From InboundTransferOffered:
- Accept transfer: emit AcceptTransfer; → InboundTransfer.
- Reject transfer: emit RejectTransfer; → Idle.
From InboundTransfer, InboundTransferOpening, OutboundTransfer or OutboundTransferOpening:
- Received CloseTransfer: → Idle.
- Close transfer: emit CloseTransfer; → Idle.
From InboundTransfer
- Received OpenFile: → InboundTransferOpening.
From InboundTransferOpening
- File opened: emit FileOpened; → InboundFileTransfer.
From InboundFileTransfer
- Received Chunk: write out chunk.
- Received CloseFile: close file; → InboundTransfer.
- Close file: emit CloseFile; → Idle.
From OutboundTransferRequested:
- Received AcceptTransfer: → OutboundTransfer.
- Received RejectTransfer: → Idle.
From OutboundTransferOffered:
- Accept transfer: emit emit SendRequest; → OutboundTransferRequested.
- Reject transfer: → Idle.
From OutboundTransfer:
- Open a file: emit OpenFile; → OutboundTransferOpening.
From OutboundTransferOpening:
- Received FileOpened: → OutboundFileTransfer.
From OutboundFileTransfer:
- Received CloseFile: → OutboundTransfer.
- Send a chunk: emit Chunk.
- Close file: emit CloseFile; → OutboundTransfer.

A Copyright & Software License

Copyright Notice

Software License

All Software contained in this document ("Software") is protected by copyright and is being made available under the "BSD License", included below. This Software may be subject to third party rights (rights from parties other than Ecma International), including patent rights, and no licenses under such third party rights are granted under this license even if the third party concerned is a member of Ecma International. SEE THE ECMA CODE OF CONDUCT IN PATENT MATTERS AVAILABLE AT https://ecma-international.org/memento/codeofconduct.htm FOR INFORMATION REGARDING THE LICENSING OF PATENT CLAIMS THAT ARE REQUIRED TO IMPLEMENT ECMA INTERNATIONAL STANDARDS.

Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:

Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.
Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.
Neither the name of the authors nor Ecma International may be used to endorse or promote products derived from this software without specific prior written permission.

THIS SOFTWARE IS PROVIDED BY THE ECMA INTERNATIONAL "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL ECMA INTERNATIONAL BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.