Commit 0356198c authored by Bruce Flynn's avatar Bruce Flynn
Browse files

Add README, Exit on eof from command stream.

parent 139874fe
# sftper
Mini in-process API to drive SFTP connections.
There are 2 main drivers behind the creation of this project:
1. Inconsistent and poor performance using [paramiko](https://www.paramiko.org/)
for bulk data transfers. It seems the paramiko v2.4 performs much better than
both v2.5 and v2.6, but event v2.4 as limitations regarding file transfers.
2. Using the native Linux `sftp` binary as a replacement is difficult due to the
scriptablility of its interactive terminal.
This project currently provides support for PUT, GET, LISTDIR, and DELETE operations
against an SFTP server via a simple JSON protocol over file-like objects, typically
stdin, stdout, and stderr.
## Protocol
The process reads commands from the input as JSON objects prefixed by the size of
the command payload as a 4 byte big-endian unsigned int:
```
*******************************
* uint32 Length of payload *
* --------------------------- *
* JSON encoded command *
*******************************
```
The command format is:
```
{"command": <name (str)>,
"args": <args (obj)>
}
```
The result will be of the format:
```
{"status": <ok|error|fail (str)>,
"message": <message (str)>,
"data": <returned data (any)>,
}
```
## Protocol in Python
Running the process:
```
from subprocess import Popen, PIPE
log = open("sftper.log", "wb")
p = Popen(["./sftper", "--url", "sftp://localhost"],
stdout=PIPE, stdin=PIPE, stderr=log, bufsize=0)
```
Sending commands:
```
import json, os, struct
dat = json.dumps({"command": "LISTDIR", "args":{"path", "/path"}}).encode()
p.stdin.write(struct.pack('!I', len(dat), dat))
```
Receiving results in Python
```
import json, os, struct
num = struct.unpack('!I', p.stdout.read(4))
dat = p.stdout.read(num)
assert len(dat) == num
zult = json.loads(dat)
```
## Commands:
### PUT
```
{"command": "PUT", "args": {"source": <path>, "dest": <path>}}
```
### GET
```
{"command": "GET", "args": {"source": <path>, "dest": <path>}}
```
### LISTDIR
```
{"command": "LISTDIR", "args": {"path": <path>}}
```
Return Data:
```
[{"name": "<filename>", "size": <bytes>, "mtime": <unixtime>}, ...]
```
### DELETE
```
{"command": "DELETE", "args": {"path": <path>}}
```
# Building
Requires Go >= 1.11
```
./build.sh
```
......@@ -68,18 +68,31 @@ func (s sftpAPI) commands() <-chan command {
go func() {
for {
num, err := readHeader(s.req)
if err == io.EOF {
debug("EOF from command stream")
break
}
// Can't recover if the basic assumptions regarding the protocol are broken
// TODO: or can we?
if err != nil {
continue
info("ERROR bad command, bailing: %s", err)
break
}
buf, err := readPayload(s.req, num)
if err != nil {
info("ERROR bad command, bailing!!!: %s", err)
break
}
cmd, err := decodeCommand(buf)
if err != nil {
debug("error decoding command: %s", err)
debug("dropping invalid command: %s", err)
continue
}
ch <- cmd
}
close(ch)
}()
return ch
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment