The definition of the splay_each() macro is somewhat complicated for
syntactic reasons. Here's what it does in a more readable way:
for (splay_node_t* node = tree->head; node;) {
type* item = node->data;
splay_node_t* next = node->next;
// RUN USER BLOCK with (item)
node = next;
}
list_each() works in the same way. Since node->next is saved before the
user block runs, this construct supports removing the current item from
within the user block. However, what it does *not* support is removing
*other items* from within the user block, especially the next item.
Indeed, that will invalide the next pointer in the above loop and
therefore result in an invalid pointer dereference.
Unfortunately, there is at least one code path where that unsupported
operation happens. It is located in ack_h(), where the authentication
protocol code detects a double connection (i.e. being connected to
another node twice). Running in the context of a socket read event, this
code will happily terminate the *other* metaconnection, resulting in its
socket being removed from the io tree. If, by misfortune, this other
metaconnection happened to have the next socket FD number (which is
quite possible due to FD reuse - albeit unlikely), and was part of the
io tree (which is quite likely because if that connection is stuck, it
will most likely have pending writes) then this will result in the next
pending io item being destroyed. Invalid pointer dereference ensues.
I did a quick audit of other uses of splay_each() and list_each() and
I believe this is the only scenario in which this "next pointer
invalidation" problem can occur in practice. While this bug has been
there since at least 6bc5d626a8 (November
2012), if not sooner, it happens quite rarely due to the very specific
set of conditions required to trigger it. Nevertheless, it does manage
to crash my central production nodes every other week or so.
On Windows, the event loop io tree uses the Windows Event handle to
differentiate between io_t objects. Unfortunately, there is a bug in
the io_add_event() function (introduced in
2f9a1d4ab5) as it sets the event after
inserting the object into the tree, resulting in objects appearing in
io_tree out of order.
This can lead to crashes on Windows as the event loop is unable to
determine which events fired.
Commit 611217c96e introduced a regression
because it accidentally reordered the timeout handler calls and the
fdset setup code. This means that any io_add(), io_del() or io_set()
calls in timeout handlers would be ignored in the current event loop
iteration, resulting in erratic behavior.
The most visible symptom is when a metaconnection timeout occurs and the
connection is closed; the timeout handler closes the socket but it still
ends up in the select() call, typically resulting in the following
crash:
Error while waiting for input: Bad file descriptor
tinc is using a separate thread to read from the TAP device on Windows.
The rationale was that the notification mechanism for packets arriving
on the virtual network device is based on Win32 events, and the event
loop did not support listening to these events.
Thanks to recent improvements, this event loop limitation has been
lifted. Therefore we can get rid of the separate thread and simply add
the Win32 "incoming packet" event to the event loop, just like a socket.
The result is cleaner code that's easier to reason about.
Currently, when the tinc service handler callback (which runs in a
separate thread) receives a service shutdown request, it calls
event_exit() to request the event loop to exit.
This approach has a few issues:
- The event loop will only notice the exit request when the next event
fires. This slows down tinc service shutdown. In some extreme cases
(DeviceStandby enabled, long PingTimeout and no connections),
shutdown can take ages.
- Strictly speaking, because of the absence of memory barriers, there
is no guarantee that the event loop will even notice an exit request
coming from another thread. I suppose marking the "running" variable
as "volatile" is supposed to alleviate that, but it's unclear whether
that provides any guarantees with modern systems and compilers.
This commit fixes the issue by leveraging the new event loop Windows
interface, using a custom Windows event that is manually set when
shutdown is requested.
This commit changes the event loop to use WSAEventSelect() and
WSAWaitForMultipleEvents() on Windows. This paves the way for making the
event loop more flexible on Windows by introducing the required
infrastructure to make the event loop wait on any Win32 event.
This commit only affects the internal implementation of the event
module. Externally visible behavior remains strictly unchanged (for
now).
When using socket functions, "sockerrno" is supposed to be used to
retrieve the error code as opposed to "errno", so that it is translated
to the correct call on Windows (WSAGetLastError() - Windows does not
update errno on socket errors). Unfortunately, the use of sockerrno is
inconsistent throughout the tinc codebase, as errno is often used
incorrectly on socket-related calls.
This commit fixes these oversights, which improves socket error
handling on Windows.
There are several reasons for this:
- MacOS/X doesn't support polling the tap device using kqueue, requiring a
workaround to fall back to select().
- On Windows only sockets are properly handled, therefore tinc uses a second
thread that does a blocking ReadFile() on the TAP-Win32/64 device. However,
this does not mix well with libevent.
- Libevent, event just the core, is quite large, and although it is easy to get
and install on many platforms, it can be a burden.
- Libev is more lightweight and seems technically superior, but it doesn't
abstract away all the platform differences (for example, async events are not
supported on Windows).
This relieves some confusion and problems during the libevent transition.
In particular, "event_add" was defined by both.
(The 't' stands for 'timeout', 'tinc', 'temporary', or some such.)
- Convert cp to cp(); so that automatic indenters work.
- Convert constructions like if(x == NULL) to if(!x).
- Move all assignments out of conditions.
