sftper

Mini in-process API to drive SFTP connections.

There are 2 main drivers behind the creation of this project:

1. Inconsistent and poor performance using paramiko for bulk data transfers. It seems the paramiko v2.4 performs much better than both v2.5 and v2.6, but event v2.4 as limitations regarding file transfers.

2. Using the native Linux sftp binary as a replacement is difficult due to the scriptablility of its interactive terminal.

This project currently provides support for PUT, GET, LISTDIR, and DELETE operations against an SFTP server via a simple JSON protocol over file-like objects, typically stdin, stdout, and stderr.

Protocol

The command format is:

{"command": <name (str)>,
 "args": <args (obj)>
}

The result will be of the format:

{"status": <ok|error|fail (str)>,
 "message": <message (str)>,
 "data": <returned data (any)>,
 }

Text Protocol

The process reads commands from the input, one per line, as JSON objects, handles the command an writes results, one-per-line to the output.

NOTE: While text is simple, it will have issues for any commands or strings that may mistakenly have a new-line in it. So, if you use it make sure you are stripping whitespace from your strings.

Binary Protocol

The process reads commands from the input as JSON objects prefixed by the size of the command payload as a 4 byte big-endian unsigned int:

*******************************
* uint32 Length of payload    *
* --------------------------- *
* JSON encoded command        *
*******************************

Python

Running the process:

from subprocess import Popen, PIPE
log = open("sftper.log", "wb")
p = Popen(["./sftper", "sftp://localhost"],
          stdout=PIPE, stdin=PIPE, stderr=log, bufsize=0)

Sending commands:

import json, os, struct
dat = json.dumps({"command": "LISTDIR", "args":{"path": "/path"}}).encode()
p.stdin.write(struct.pack('!I', len(dat)) + dat)

Receiving results in Python

import json, os, struct
num = struct.unpack('!I', p.stdout.read(4))[0]
dat = p.stdout.read(num)
assert len(dat) == num
zult = json.loads(dat)

Commands:

PUT

{"command": "PUT", "args": {"source": <path>, "dest": <path>}}

GET

{"command": "GET", "args": {"source": <path>, "dest": <path>}}

LISTDIR

{"command": "LISTDIR", "args": {"path": <path>}}

Return Data:

[{"name": "<filename>", "size": <bytes>, "mtime": <unixtime>, "mode": <int>}, ...]

mode will be an unsinged int like stat.st_mode. See man pages for stat(2) and inode(7).

DELETE

{"command": "DELETE", "args": {"path": <path>}}

Building

Requires Go >= 1.11

./build.sh