If an outgoing connection cannot be made because no address is known for
it, it should be removed from the outgoing_list, otherwise it will
prevent it from being re-added later when we do know addresses for it.
This problem occurs on "road-warriors" when tincd setups
outgoing connections but you do not have any active uplink then
dns-lookups will fail and any following attempt to make outgoing
connections will keep failing forever.
This fixes a hairy race condition that was introduced in
1e89a63f16, which changed
the underlying transport of handshake packets from REQ_KEY to ANS_KEY.
Unfortunately, what I missed in that commit is, on the receiving side,
there is a slight difference between req_key_h() and ans_key_h():
indeed, the latter resets validkey to false.
The reason why this is not a problem during typical operation is
because the normal SPTPS key regeneration procedure looks like this:
KEX ->
<- KEX
SIG ->
<- SIG
All these messages are sent over ANS_KEY, therefore the receiving side
will unset validkey. However, that's typically not a problem in practice
because upon reception of the last message (SIG), SPTPS will call
sptps_receive_record(), which will set validkey to true again, and
everything works out fine in the end.
However, that was the *typical* scenario. Now let's assume that the
SPTPS channel is in active use at the same time key regeneration
happens. Specifically, let's assume a normal VPN data packet sneaks in
during the key regeneration procedure:
KEX ->
<- KEX
<- (SPTPS packet, over TCP or UDP)
<- KEX (wtf?)
SIG -> (refused with Invalid packet seqno: XXX != 0)
At this point, both nodes are extremely confused and the SPTPS channel
becomes unusable with various errors being thrown on both sides. The
channel will stay down until automatic SPTPS channel restart kicks in
after 10 seconds.
(Note: the above is just an example - the race can occur on either side
whenever a packet is sent during the period of time between KEX and SIG
messages are received by the node sending the packet.)
I've seen this race occur in the wild - it is very likely to occur if
key regeneration occurs on a heavily loaded channel. It can be
reproduced fairly easily by setting KeyExpire to a short value (a few
seconds) and then running something like ping -f foobar -i 0.01.
The reason why this occurs is because tinc's TX code path triggers the
following:
- send_packet()
- try_tx()
- try_tx_sptps()
- validkey is false because we just received an ANS_KEY message
- waitingforkey is false because it's not used for key regeneration
- send_req_key()
- SPTPS channel restart (sptps_stop(), sptps_start()).
Obviously, it all goes downhill from there and the two nodes get very
confused quickly (for example the seqno gets reset, hence the error
messages).
This commit fixes the issue by keeping validkey set when SPTPS data is
received over ANS_KEY messages.
Unfortunately, libminiupnpc has a somewhat... "peculiar" approach to
backwards compatibility for their API, where they reserve the right to
make breaking changes when they feel like it, forcing users to resort
to #ifdefs to ensure they use the correct API. Sigh.
Previously, tinc would only build against API versions <= 13, because I
was doing my initial development using miniupnpc-1.9.20140610 which is
the version that ships with Debian. The changes in this commit are
required for tinc to build against more recent versions, from
1.9.20150730 to the latest one at the time of this commit, 1.9.20151026.
This commit makes tincd capable of discovering UPnP-IGD devices on the
local network, and add mappings (port redirects) for its TCP and/or UDP
port.
The goal is to improve reliability and performance of tinc with nodes
sitting behind home routers that support UPnP, by making it less reliant
on UDP Hole Punching, which is prone to failure when "hostile" NATs are
involved.
The way this is implemented is by leveraging the libminiupnpc library,
which we have just added a new dependency on. We use pthread to run the
UPnP client code in a dedicated thread; we can't use the tinc event loop
because libminiupnpc doesn't have a non-blocking API.
When tinc is used in router mode with a TAP device, Ethernet (MAC)
headers are not present in packets flowing over the VPN; it is the
node's responsibility to fill out this header before handing the
packet over to the TAP interface (which expects such headers).
Currently, tinc fills out the destination MAC address of the packet
(otherwise the host would not recognize the packets, and nothing would
work), but it does not fill out the source MAC address. In practice this
doesn't seem to cause any real issues (the host doesn't care about the
source address), but it does look weird when looking at the packets with
a sniffer, and it also result in the following valgrind warning:
==13651== Syscall param write(buf) points to uninitialised byte(s)
==13651== at 0x5C4B620: __write_nocancel (syscall-template.S:81)
==13651== by 0x1445AA: write_packet (device.c:183)
==13651== by 0x118C7C: send_packet (net_packet.c:1259)
==13651== by 0x12B70A: route_ipv4 (route.c:443)
==13651== by 0x12D5F8: route (route.c:971)
==13651== by 0x1152BC: receive_packet (net_packet.c:250)
==13651== by 0x117E1B: receive_sptps_record (net_packet.c:904)
==13651== by 0x1309A8: sptps_receive_data_datagram (sptps.c:488)
==13651== by 0x130A90: sptps_receive_data (sptps.c:508)
==13651== by 0x115569: receive_udppacket (net_packet.c:286)
==13651== by 0x119856: handle_incoming_vpn_data (net_packet.c:1499)
==13651== by 0x10F3DA: event_loop (event.c:287)
==13651== Address 0xffeffea3a is on thread 1's stack
==13651== in frame #6, created by receive_sptps_record (net_packet.c:821)
==13651==
This commit fixes the issue by filling out the source MAC address. It is
generated by negating the last byte of the device MAC address, which is
consistent with what route_arp() does.
In addition, this commit stops route_arp() from filling out the Ethernet
header of the packet - this is the responsibility of send_packet(), not
route().
This reverts commit c2319e90b1.
As a general principle, I do not believe it is worthwhile to cache
nodes. Sure, it brings lookup time down from O(log n) to O(1), but
considering that the scalability target of tinc is around 1000 nodes
and log2(1000) is 10, that looks like premature optimization; tree
lookups should already be very fast. Therefore, I believe it makes sense
to remove the cache as a code cleanup initiative.
This commit replaces the node UDP address hash table "cache" with a
full-blown splay tree, aligning it with node_tree (name-indexed) and
node_id_tree (ID-indexed).
I'm doing this for two reasons. The first reason is to make sure we
don't suddenly degrade to O(n) performance when two "hot" nodes end up
in the same hash table bucket (collision).
The second, and most important, reason, has to do with the fact that
the hash table that was being used overrides elements that collide.
Indeed, it turns out that there is one scenario in which the contents of
node_udp_cache has *correctness* implications, not just performance
implications. This has to do with the way handle_incoming_vpn_data() is
implemented.
Assume the following topology:
A <-> B <-> C
Now let's consider the perspective of tincd running on B, and let's
assume the following is true:
- All nodes are using the 1.1 protocol with node IDs and relaying
support.
- Nodes A and C have UDP addresses that hash to the same value.
- Node C "wins" in the node_udp_cache (i.e. it overwrites A in the
cache).
- Node A has a "dynamic" UDP address (i.e. an UDP address that has been
detected dynamically and cannot be deduced from edge addresses).
Then, before this commit, A would be unable to relay packets through B.
This is because handle_incoming_vpn_data() will fall back to
try_harder(), which won't be able to match any edge addresses, doesn't
check the dynamic UDP addresses, and won't be able to match any keys
because this is a relayed packet which is encrypted with C's key, not
B's. As a result, tinc will fail to match the source of the packet and
will drop the packet with a "Received UDP packet from unknown source"
message.
I have seen this happen in the wild; it is actually quite likely to
occur when there are more than a handful of nodes because node_udp_cache
only has 256 buckets, making collisions quite likely. This problem is
quite severe because it can completely prevent all packet communication
between nodes - indeed, if node A tries to initiate some communication
with C, it will use relaying at first, until C responds and helps A
establish direct communication with it (e.g. hole punching). If relaying
is broken, C will not help establish direct communication, and as a
result no packets can make it through at all.
The bug can be reproduced fairly easily by reproducing the topology
above while changing the (hardcoded) node_udp_cache size to 1 to force a
collision. One will quickly observe various issues when trying to make A
talk to C. Setting IndirectData on B will make the issue even more
severe and prevent all communication.
Arguably, another way to fix this problem is to make try_harder()
compare the packet's source address to each node's dynamic UDP
addresses. However, I do not like this solution because if two "hot"
nodes are contending on the same hash bucket, try_harder() will be
called very often and packet routing performance will degrade closer to
O(N) (where N is the total number of nodes in the graph). Using a more
appropriate data structure fixes the bug without introducing this
performance problem.
Left shifts of negative values is undefined in C. This happens a lot in
the Ed25519 code. Cast to unsigned first, then cast the result back to
signed where necessary.
Although not a problem for tinc internally, the size of the struct was 12
bytes instead of 4, causing some problems when interpreting the value
received from tincd by the CLI.
In cases when tinc has all available nodes in outgoing connections and
can not establish those connection due to network outage periodic_handler()
would crash since tmp_node_tree->count is 0.
This commit adds also new flag node->status.has_cfg_address to prevent
update_udp_address() from removing this flag.
Fixed node_status_t->unused - 13 + 19 = 32
When AutoConnect is on tinc needs to know if nodes have Address to defined
in thier hosts files. Currently tinc parsed node's host files if StrictSubnet
was enabled. To reduce the parsing overhead I have merged load_all_subnets
with load_all_nodes, such that load_all_subnets has been removed and
load_all_nodes has if-statement extracting Subnet information from node's host
file.
When AutoConnect is enabled tinc tries to connect to other nodes picking them at random.
This may be sane default behavior but it may take ages if only few nodes have
defined Address in thier config.
Proposed solution to this problem:
- Filter out nodes without known address in periodic_handler
I have added new node->status.has_known_address bool
- On update_node_udp() update this flag
In some cases - mostly when e->to == myself the prevedge is set to NULL,
causing invalid memory access. In rare cases this may lead to malformed mst
or segfaults.
In some cases - mostly when e->to == myself the prevedge is set to NULL,
causing invalid memory access. In rare cases this may lead to malformed mst
or segfaults.
If ADD_EDGE came from tinc version 1.0.x local_address.sa.sa_family is set to 0.
If it came from tinc version 1.1.x forwarded for older verion it will be 255 - AF_UNKNOWN.
When tinc gets ADD_EDGE from older versions it will allocate
new edge in protocol_edge.c:189 due to missed case in lines 149-171 where
local_address is not defined.
The information is of grate value when monitoring multiple
nodes in one window. Without it the user is forced to quit top, exit tinc
and go back to shell to refresh his memory about which node is in
what window.
When tincd setups it's network device some operating systems send router
solicitation packets from local scope ip addresses. tincd forwards it
then to his neighbors then those nodes follow the same routine fowarding it
to the next hops. I may happen that an loop will occur consuming large amount
of bandwith. Constrains: Mode = Router, Broadcast = mst.
Reproduction: ping6 -c 1 ff02::2%<tincd interface>
Sending one packet will, depending on your setup, generate about 3k packets.
Proposed solution in this commit: enable StrictSubnets, tincd will reject such
packets due to unknown subnet.
Future work: check scope of the ip address and make decisions about forwarding
based on Mode tincd is configured to work.
For some reason the edges ware removed in one direction resulting in e->reverse
point into invalid memory.
Do not insert edge into edge_weight_tree if not needed.
If ADD_EDGE came from tinc version 1.0.x local_address.sa.sa_family is set to 0.
If it came from tinc version 1.1.x forwarded for older verion it will be 255 - AF_UNKNOWN.
When tinc gets ADD_EDGE from older versions it will allocate
new edge in protocol_edge.c:189 due to missed case in lines 149-171 where
local_address is not defined.
If ADD_EDGE came from tinc version 1.0.x local_address.sa.sa_family is set to 0.
If it came from tinc version 1.1.x forwarded for older verion it will be 255 - AF_UNKNOWN.
When tinc gets ADD_EDGE from older versions it will allocate
new edge in protocol_edge.c:189 due to missed case in lines 149-171 where
local_address is not defined.
The proper place to clean up resources of objects is in their
destructor. This makes sure proper cleanup when edge_del() is called as
well. At exit, free_edge() is called on all edges by free_edge_tree(),
which is called by exit_nodes().
Let configure include sys/if_tun.h when testing for netinet/if_ether.h
to detect the Kernel/libc header conflict on musl.
After this patch, configure will correctly detect netinet/if_ether.h as
unusable and the subsequent compilation will not attempt to use it.
Conflicts:
src/have.h
With AutoConnect = yes, tinc tries to establish connections to known hosts.
However, you could have set no Address for this host, which is perfectly fine
(as long as there is at least one bootstrap node with an address or a local
discovered node already part of the network)
So log this to LOG_DEBUG
In a "decentrally managed vpn" it is very likely that host config
files for some reachable nodes do not exist. Currently, tinc
fills the logs with "Cannot open config file" messages.
This commit changes the log level to LOG_DEBUG so
syslog doesn't get filled by default.
(gdb) bt
#0 mst_kruskal () at graph.c:107
#1 graph () at graph.c:302
#2 0x00007ffff7b509fe in del_edge_h (c=<optimized out>, request=<optimized out>) at protocol_edge.c:292
#3 0x00007ffff7b4de2e in receive_request (c=0x5555557e3ef0, request=0x555555800e13 "13 3fc17404 node1 node2") at protocol.c:136
#4 0x00007ffff7b43513 in receive_meta (c=0x5555557e3ef0) at meta.c:290
#5 0x00007ffff7b442d9 in handle_meta_connection_data (c=0x5555557e3ef0) at net.c:291
#6 0x00007ffff7b41391 in event_loop () at event.c:287
#7 0x00007ffff7b449b2 in main_loop () at net.c:469
#8 0x0000555555556716 in main (argc=<optimized out>, argv=<optimized out>) at tincd.c:480
==27135== Use of uninitialised value of size 8
==27135== at 0x57BE17B: BN_num_bits_word (in /usr/lib/libcrypto.so.1.0.0)
==27135== by 0x57BE205: BN_num_bits (in /usr/lib/libcrypto.so.1.0.0)
==27135== by 0x57BADF7: BN_div (in /usr/lib/libcrypto.so.1.0.0)
==27135== by 0x57C48FC: BN_mod_inverse (in /usr/lib/libcrypto.so.1.0.0)
==27135== by 0x57C3647: BN_BLINDING_create_param (in /usr/lib/libcrypto.so.1.0.0)
==27135== by 0x5812D44: RSA_setup_blinding (in /usr/lib/libcrypto.so.1.0.0)
==27135== by 0x58095CB: rsa_get_blinding (in /usr/lib/libcrypto.so.1.0.0)
==27135== by 0x580A64F: RSA_eay_private_decrypt (in /usr/lib/libcrypto.so.1.0.0)
==27135== by 0x4E5D9BC: rsa_private_decrypt (rsa.c:97)
==27135== by 0x4E51E1B: metakey_h (protocol_auth.c:524)
==27135== by 0x4E505FD: receive_request (protocol.c:136)
==27135== by 0x4E46002: receive_meta (meta.c:290)
==27135== Uninitialised value was created by a heap allocation
==27135== at 0x4C29F90: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==27135== by 0x575DCD7: CRYPTO_malloc (in /usr/lib/libcrypto.so.1.0.0)
==27135== by 0x57C24E1: BN_rand (in /usr/lib/libcrypto.so.1.0.0)
==27135== by 0x57C216F: bn_rand_range (in /usr/lib/libcrypto.so.1.0.0)
==27135== by 0x57C3630: BN_BLINDING_create_param (in /usr/lib/libcrypto.so.1.0.0)
==27135== by 0x5812D44: RSA_setup_blinding (in /usr/lib/libcrypto.so.1.0.0)
==27135== by 0x58095CB: rsa_get_blinding (in /usr/lib/libcrypto.so.1.0.0)
==27135== by 0x580A64F: RSA_eay_private_decrypt (in /usr/lib/libcrypto.so.1.0.0)
==27135== by 0x4E5D9BC: rsa_private_decrypt (rsa.c:97)
==27135== by 0x4E51E1B: metakey_h (protocol_auth.c:524)
==27135== by 0x4E505FD: receive_request (protocol.c:136)
==27135== by 0x4E46002: receive_meta (meta.c:290)
The definition of the splay_each() macro is somewhat complicated for
syntactic reasons. Here's what it does in a more readable way:
for (splay_node_t* node = tree->head; node;) {
type* item = node->data;
splay_node_t* next = node->next;
// RUN USER BLOCK with (item)
node = next;
}
list_each() works in the same way. Since node->next is saved before the
user block runs, this construct supports removing the current item from
within the user block. However, what it does *not* support is removing
*other items* from within the user block, especially the next item.
Indeed, that will invalide the next pointer in the above loop and
therefore result in an invalid pointer dereference.
Unfortunately, there is at least one code path where that unsupported
operation happens. It is located in ack_h(), where the authentication
protocol code detects a double connection (i.e. being connected to
another node twice). Running in the context of a socket read event, this
code will happily terminate the *other* metaconnection, resulting in its
socket being removed from the io tree. If, by misfortune, this other
metaconnection happened to have the next socket FD number (which is
quite possible due to FD reuse - albeit unlikely), and was part of the
io tree (which is quite likely because if that connection is stuck, it
will most likely have pending writes) then this will result in the next
pending io item being destroyed. Invalid pointer dereference ensues.
I did a quick audit of other uses of splay_each() and list_each() and
I believe this is the only scenario in which this "next pointer
invalidation" problem can occur in practice. While this bug has been
there since at least 6bc5d626a8 (November
2012), if not sooner, it happens quite rarely due to the very specific
set of conditions required to trigger it. Nevertheless, it does manage
to crash my central production nodes every other week or so.
Unfortunately, sptps_logger() cannot know if s->handle is pointing to a
connection_t or a node_t. But it needs to print name and hostname in
both cases. So make sure both types have name and hostname fields at the
start with the same offset.
The sptps_receive_data() was changed in commit d237efd to only process
one SPTPS record from a stream input. So now we have to put a loop
around it to ensure we process everything.
In some harmless places, checks for the return value of ECDSA and RSA
key generation and verification was omitted. Add them to keep the
compiler happy and to warn end users in case something is wrong.
The definition of the splay_each() macro is somewhat complicated for
syntactic reasons. Here's what it does in a more readable way:
for (splay_node_t* node = tree->head; node;) {
type* item = node->data;
splay_node_t* next = node->next;
// RUN USER BLOCK with (item)
node = next;
}
list_each() works in the same way. Since node->next is saved before the
user block runs, this construct supports removing the current item from
within the user block. However, what it does *not* support is removing
*other items* from within the user block, especially the next item.
Indeed, that will invalide the next pointer in the above loop and
therefore result in an invalid pointer dereference.
Unfortunately, there is at least one code path where that unsupported
operation happens. It is located in ack_h(), where the authentication
protocol code detects a double connection (i.e. being connected to
another node twice). Running in the context of a socket read event, this
code will happily terminate the *other* metaconnection, resulting in its
socket being removed from the io tree. If, by misfortune, this other
metaconnection happened to have the next socket FD number (which is
quite possible due to FD reuse - albeit unlikely), and was part of the
io tree (which is quite likely because if that connection is stuck, it
will most likely have pending writes) then this will result in the next
pending io item being destroyed. Invalid pointer dereference ensues.
I did a quick audit of other uses of splay_each() and list_each() and
I believe this is the only scenario in which this "next pointer
invalidation" problem can occur in practice. While this bug has been
there since at least 6bc5d626a8 (November
2012), if not sooner, it happens quite rarely due to the very specific
set of conditions required to trigger it. Nevertheless, it does manage
to crash my central production nodes every other week or so.
Unfortunately, sptps_logger() cannot know if s->handle is pointing to a
connection_t or a node_t. But it needs to print name and hostname in
both cases. So make sure both types have name and hostname fields at the
start with the same offset.
The sptps_receive_data() was changed in commit d237efd to only process
one SPTPS record from a stream input. So now we have to put a loop
around it to ensure we process everything.
In some harmless places, checks for the return value of ECDSA and RSA
key generation and verification was omitted. Add them to keep the
compiler happy and to warn end users in case something is wrong.
It is not unusual for tinc to receive SPTPS packets to be relayed to
nodes that just became unreachable, due to state propagation delays in
the metagraph.
Unfortunately, the current code doesn't handle that situation correctly,
and still tries to relay the packet to the unreachable node. This
typically ends up segfaulting.
This commit fixes the issue by checking for reachability before relaying
the packet.
clang-3.7 warnings surfaced an actual bug:
invitation.c:185:5: error: address of array 'filename' will always evaluate to 'true'
[-Werror,-Wpointer-bool-conversion]
if(filename) {
~~ ^~~~~~~~
The regression was introduced in 3ccdf50beb.
This issue was found through a clang-3.7 warning:
protocol_misc.c:167:46: error: format specifies type 'short' but the argument has type 'int'
[-Werror,-Wformat]
if(!send_request(c, "%d %hd", SPTPS_PACKET, len))
~~~ ^~~
%d
It is entirely possible that the configuration file could contain a
ConnectTo statement refering to its own name; that's a reasonable
scenario when one deploys semi-automatically generated tinc.conf files.
Amusingly, tinc does not like that at all, and actually sets up an
outgoing_t structure to myself (which obviously makes no sense). This is
mostly benign, though it does result in non-sensical "Already connected
to myself" messages every retry interval.
However, that also makes things blow up in close_network_connections(),
because there we delete the entire outgoing list and *then* the myself
node, which still has a reference to the freshly deleted outgoing
structure. Boom.
timeout_handler() calls try_tx(c->node) when c->edge exists.
Unfortunately, the existence of c->edge is not enough to conclude that
the node is reachable.
In fact, during connection establishment, there is a short period of
time where we create an edge for the node at the other end of the
metaconnection, but we don't have one from the other side yet.
Unfortunately, if timeout_handler() runs during that short time
window, it will call try_tx() on an unreachable node, which makes
things explode because that function is not prepared to handle that
case.
A typical symptom of this race condition is a hard SEGFAULT while trying
to send packets using metaconnections that don't exist, due to
n->nexthop containing garbage.
This patch fixes the issue by making try_tx() check for reachability,
and then making all code paths use try_tx() instead of the more
specialized methods so that they go through the check.
This regression was introduced in
eb7a0db18e.
We do this by creating an umbilical between the CLI and the daemon. The
daemon pipes log messages to the CLI until it starts the main loop. The
daemon then cuts the umbilical. The CLI copies all the received log
messages to stderr, and the last byte indicates whether the daemon
started succesfully or not, so the CLI can exit with a useful exit code.
This gets rid of xasprintf() in a number of places, and removes the need
to free() the temporary strings. A few potential memory leaks have been
fixed.
This dumps the name of the invitation file, as well as the name of the
node that is being invited. This can make it easier to find the
invitation file belonging to a given node.
It is possible that opening /dev/net/tun works but that interface
creation itself fails, for example if a non-root user tries to create a
new interface, or if the desired interface is already opened by another
process. In this case, the ioctl() fails, but we actually silently
ignored this condition.
The compile time local state directory is usually /var or
/usr/local/var. If this is not accessible for some reason, for example
because someone ./configured tinc without --localstatedir and
/usr/local/var does not exist, or if tinc is started by a non-root user,
then tinc will fall back to the directory where tinc.conf is stored.
A warning is logged when this happens.
This function is not used for normal traffic, only when a packet from an
unknown source is received and we need to check against candidates. No
failures should be logger in this case; if the packet is really not
valid this will be logged by handle_incoming_vpn_data().
try_tx_sptps() gives up on UDP communication if the recipient doesn't
support relaying. This is too restrictive - we only need the other node
to support relaying if we actually want to relay through them. If the
packet is sent directly, it's fine to send it to an old pre-node-IDs
tinc-1.1 node.
Currently, tinc tries to parse node IDs for all SPTPS packets, including
ones sent from older, pre-node-IDs tinc-1.1 nodes, and therefore doesn't
recognize packets from these nodes. This commit fixes that.
It also makes code slightly clearer by reducing the amount of fiddling
around packet offset/length.
A condition in try_harder() is always evaluating to false when talking
to a SPTPS node because n->status.validkey_in is always false in that
case. Fix the condition so that the SPTPS status is correctly checked.
This prevented recent tinc-1.1 nodes from talking to older, pre-node-ID
tinc-1.1 nodes.
The regression was introduced in
6056f1c13b.
Since commit 13f9bc1ff1, tinc passes the
-I. option to the preprocessor so that version_git.h can be found during
out-of-tree ("VPATH") builds.
The problem is, this option also affects the directory search for files
included *from* system headers. For example, on MinGW, unistd.h contains
the following line:
#include <process.h>
Which, due to -I. putting the tinc directory at the head of the search
order, results in tinc's process.h being included instead of the file
from MinGW. Hilarity ensues.
This commit fixes the issue by using -iquote, which doesn't affect
system headers.
KEY_CHANGED messages are only useful to invalidate keys for non-SPTPS nodes;
SPTPS nodes use a different internal mechanism (forced KEX) for that purpose.
Therefore, if we know we can't talk to legacy nodes, there's no point in
sending them these messages.
There are a number of ways a SPTPS tunnel can get into a corrupt state.
For example, during key regeneration, the KEX and SIG messages from
other nodes might arrive out of order, which confuses the hell out of
the SPTPS code. Another possible scenario is not noticing another node
crashed and restarted because there was no point in time where the node
was seen completely disconnected from *all* nodes; this could result in
using the wrong (old) key. There are probably other scenarios which have
not even been considered yet. Distributed systems are hard.
When SPTPS got confused by a packet, it used to crash the entire
process; fortunately that was fixed by commit
2e7f68ad2b. However, the error handling
(or lack thereof) leaves a lot to be desired. Currently, when SPTPS
encounters an error when receiving a packet, it just shrugs it off and
continues as if nothing happened. The problem is, sometimes getting
receive errors mean the tunnel is completely stuck and will not recover
on its own. In that case, the node will become unreachable - possibly
indefinitely.
The goal of this commit is to improve SPTPS error handling by taking
proactive action when an incoming packet triggers a failure, which is
often an indicator that the tunnel is stuck in some way. When that
happens, we simply restart SPTPS entirely, which should make the tunnel
recover quickly.
To prevent "storms" where two buggy nodes flood each other with invalid
packets and therefore spend all their time negotiating new tunnels, we
limit the frequency at which tunnel restarts happen to ten seconds.
It is likely this commit will solve the "Invalid KEX record length
during key regeneration" issue that has been seen in the wild. It is
difficult to be sure though because we do not have a full understanding
of all the possible conditions that can trigger this problem.
