mirror of
https://github.com/retspen/webvirtcloud
synced 2026-11-11 12:46:49 +00:00
perf(storages): an ISO uploads to an SSH compute without waiting for each write
Each 32 MiB chunk went out as 1024 SFTP writes, each waiting for its answer: 19 MiB/s on a gigabit LAN, ~1.9 s per chunk with the SSH session. The writes now go out without waiting and their answers are read as they come and at the end: ~0.53 s per chunk, the same file (checked by sha256). paramiko's own pipelined writes (set_pipelined) drop the errors in their answers, so a full disk would leave a short ISO behind a successful upload. Each write is registered to its file instead, the way paramiko does for prefetched reads: every answer, in any order, reaches the file and an error is raised as soon as one is read.
This commit is contained in:
parent
24231813d7
commit
a87fbf739f
4 changed files with 136 additions and 5 deletions
|
|
@ -523,6 +523,25 @@ class StorageUploadSecurityTests(TestCase):
|
|||
def read(self):
|
||||
return bytes(self.buf)
|
||||
|
||||
# the SFTP client calls of a pipelined write: the file is its own client
|
||||
MAX_REQUEST_SIZE = 32768
|
||||
handle = b"handle"
|
||||
sock = MagicMock(recv_ready=lambda: False)
|
||||
|
||||
@property
|
||||
def sftp(self):
|
||||
return self
|
||||
|
||||
def _async_request(self, fileobj, kind, handle, offset, data):
|
||||
self.seek(offset)
|
||||
self.write(data)
|
||||
|
||||
def _finish_responses(self, fileobj):
|
||||
pass
|
||||
|
||||
def _check_exception(self):
|
||||
pass
|
||||
|
||||
def seek(self, pos):
|
||||
self.pos = pos
|
||||
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue