Travis running tests on test servers (#272)

- README.md for adding test servers
 - Bash script for running tests on remote servers
 - Extend test_runner.py to flash using esptool.py
 - Python3.4 support. README.md update

utils/travis_tests/README.md

@@ -0,0 +1,101 @@
+Travis CI Tests
+====================
+
+This directory contains a script `run_tests.sh` that is executed by Travis CI.
+The script builds a test firmware, deploys it on one of the test servers and
+runs it.
+
+The script will not return an error if deployment to one of the test servers has
+failed. It is done this way not to fail a build if a test server is down.
+The script will return an error if deployment was successful but tests failed.
+
+Test servers
+------------
+
+Test server is a linux host that is accessible from the Internet by a static IP.
+It should have at least one ESP8266 module connected to a USB port. The module
+should be capable restarting and switching to boot mode via a serial port.
+All popular **NodeMCU** and **Wemos** modules will work.
+
+To run tests on a server it should provide SSH access. SSH daemon should be
+configured to authenticate using keys.
+
+Test server running on Raspberry PI:
+
+![Raspberry PI Test server][example-test-server]
+
+### Test server requirements
+
+* Linux host
+* Public static IP
+* One or two ESP8266 modules connected to USB ports
+* SSH access from the Internet (with public key from Travis CI)
+* Python3
+* [esptool.py] installed `pip install esptool`
+* pySerial python module `pip3 install pyserial`
+
+### Create SSH keys for Travis
+
+[Here][travis-ssh-deploy] is a good article about Travis deployment using SSH.
+
+The problem with SSH access from Travis to a server is that it should have
+a private key. But this key should not be publicly available.
+
+Hopefully Travis allows to encrypt certain files and only decrypt them at build
+stage. So the sensitive file is stored in the repository encrypted.
+
+Generate a new key pair:
+```bash
+ssh-keygen -t rsa -b 4096 -C '<repo>@travis-ci.org' -f ./<server_name>_rsa
+```
+
+To encrypt a private key you need a command line Travis client.
+
+To install it run:
+```bash
+gem install travis
+```
+Or refer [the official installation instructions][travis-install].
+
+The following command will encrypt a file and modify .travis.yml:
+```bash
+travis encrypt-file <server_name>_rsa --add
+```
+
+Deploy public key to a test server:
+```bash
+ssh-copy-id -i <server_name>_rsa.pub <ssh-user>@<deploy-host>
+```
+
+Add the following lines in the .travis.yml:
+```yml
+addons:
+  ssh_known_hosts: <server_ip>
+```
+```yml
+before_install:
+- openssl aes-256-cbc aes-256-cbc -K $encrypted_<...>_key -iv $encrypted_<...>_iv -in <server_name>_rsa.enc -out /tmp/<server_name>_rsa -d
+- eval "$(ssh-agent -s)"
+- chmod 600 /tmp/<server_name>_rsa
+- ssh-add /tmp/<server_name>_rsa
+```
+
+Remove keys and stage files for commit:
+```bash
+rm -f <server_name>_rsa <server_name>_rsa.pub
+git add <server_name>_rsa.enc .travis.yml
+```
+
+### Add test server
+
+The final step is to add a server to the test runner script.
+Add a new item into an array in `run_tests.sh`:
+```bash
+TEST_SERVERS[2]="IP=<server_ip>;User=<ssh_user_name>;Type=<solo|dual>"
+```
+
+
+[esptool.py]: https://github.com/espressif/esptool
+[travis-ssh-deploy]: https://oncletom.io/2016/travis-ssh-deploy
+[travis-install]: https://github.com/travis-ci/travis.rb#installation
+[example-test-server]: ./test_server_example.png

utils/travis_tests/run_tests.sh

@@ -0,0 +1,106 @@
+#!/bin/bash
+#
+# This script builds tests, deploys them on one of the available test
+# servers and runs them. If deployment fails it will not return an error code.
+# If tests fail the script will return an error code.
+# It is done this way not to fail Travis build if one of the test servers is
+# down.
+
+# Test servers configuration
+TEST_SERVERS[0]="IP=195.138.84.66;User=pi;Type=solo"
+TEST_SERVERS[1]="IP=195.138.84.66;User=pi;Type=dual"
+
+# It will be populated in 'build' function
+FLASH_CMD=
+
+# Function doesn't accept any arguments. It builds the tests,
+# packages the binaries into the archive and populates FLASH_CMD variable.
+function build {
+    echo "Building tests"
+    make -C ./tests clean
+    make -C ./tests -j8
+    FLASH_CMD=$(make -s -C ./tests print_flash_cmd)
+
+    # Now we need to pack all files that are included in the flash cmd
+    # so they can be transferred to the remote server and run there
+    # Also we need to prepare flash command:
+    #  - remove firmware files path
+    #  - remove serial port parameter
+    mkdir -p /tmp/firmware
+    rm -rf /tmp/firmware/*
+    params=($FLASH_CMD)
+    pushd ./tests
+    for param in "${params[@]}"
+    do
+        if [ -f ${param} ]
+        then
+            file_name=${param##*/}
+            cp ${param} /tmp/firmware/
+            FLASH_CMD=${FLASH_CMD/${param}/${file_name}}
+        fi
+
+        # Removing port parameter from the cmd string
+        if [[ "$param" == "-p" || "$param" == "--port" ]]
+        then
+            FLASH_CMD=${FLASH_CMD/${param}/}
+            next_port=true
+        else
+            # Removing port value from the cmd string
+            if [ "$next_port" ]
+            then
+                FLASH_CMD=${FLASH_CMD/${param} /}
+                unset next_port
+            fi
+        fi
+    done
+    cp test_runner.py /tmp/firmware/
+    tar -czf /tmp/tests.tar.gz -C /tmp/firmware .
+    popd
+}
+
+# $1 - Server IP
+# $2 - Login user name
+function deploy {
+    echo "Deploying tests, server IP=${1}"
+    scp /tmp/tests.tar.gz ${2}@${1}:/tmp/tests.tar.gz
+    ssh ${2}@${1} mkdir -p /tmp/eor_test
+    ssh ${2}@${1} rm -rf /tmp/eor_test/*
+    ssh ${2}@${1} tar -xzf /tmp/tests.tar.gz -C /tmp/eor_test
+}
+
+# $1 - Server IP
+# $2 - Login user name
+# $3 - Type "solo" or "dual"
+function run_tests {
+    echo "Running tests, server IP=${1}, type=${3}"
+    echo "Flash cmd: ${FLASH_CMD}"
+    # Run test runner on the remote server
+    ssh ${2}@${1} "source ~/.profile; /tmp/eor_test/test_runner.py --type ${3} -f -c \"${FLASH_CMD}\""
+}
+
+# First step is to build a firmware
+build
+
+failed=0
+
+for server in "${TEST_SERVERS[@]}"
+do
+    params=(${server//;/ })
+    ip=${params[0]#IP=}
+    user=${params[1]#User=}
+    type=${params[2]#Type=}
+
+    deploy ${ip} ${user}
+    if [ "$?" -eq "0" ]
+    then
+        run_tests ${ip} ${user} ${type}
+        if [ "$?" -ne "0" ]
+        then
+            failed=$((failed+1))
+        fi
+    else
+        echo "Server ${ip} is not available"
+    fi
+done
+
+exit $failed