Twisted is the original Python asynchronous networking framework, predating asyncio by more than a decade. Its reactor pattern — the central I/O event dispatcher — influenced the design of asyncio, node.js, and many other frameworks. Yet Twisted’s internals are rarely studied. This post reads through SelectReactor in twisted/internet/selectreactor.py and the base reactor in twisted/internet/base.py to understand exactly how I/O and timed callbacks are scheduled.
→ Related: Python asyncio event loop internals — asyncio’s reactor is directly influenced by Twisted (Blog 01)
The Reactor Pattern in Twisted
Twisted’s reactor is a singleton (per process) that owns all I/O. You register IReadDescriptor or IWriteDescriptor implementors with the reactor via reactor.addReader() / reactor.addWriter(). The reactor polls these descriptors for activity and calls doRead() or doWrite() on them when they are ready. Protocols, transports, and all higher-level abstractions sit on top of this primitive.
SelectReactor.doSelect()
SelectReactor (twisted/internet/selectreactor.py) implements _doReadOrWrite using the POSIX select() system call. Its doSelect() method: collects all registered readers and writers into three sets (read, write, exceptional), calls select(read, write, exceptional, timeout) with a computed timeout, then iterates the returned ready sets and calls _doReadOrWrite() for each ready descriptor.
The timeout is computed from the _pendingTimedCalls heap (the same priority queue concept as asyncio’s _scheduled). If there are pending delayed calls, timeout = time_until_next_call, else timeout = None (block indefinitely).
# Simplified from selectreactor.py def doSelect(self, timeout): reads = set(self._readers) writes = set(self._writers) r, w, e = select.select(reads, writes, reads, timeout) for reader in r: self._doReadOrWrite(reader, reader, POLLIN) for writer in w: self._doReadOrWrite(writer, writer, POLLOUT)
The Base Reactor: runUntilCurrent()
The base reactor class (twisted/internet/base.py) provides runUntilCurrent(), called on every iteration after doSelect(). It processes all delayed calls whose .time <= now from the _pendingTimedCalls heap. Each DelayedCall wraps a callable and a timestamp. When due, it calls the callable and, if it is a LoopingCall, reschedules itself.
This is structurally identical to asyncio’s _run_once() logic: poll I/O, then drain the due timer queue. The key difference is that Twisted uses DelayedCall objects with a cancel() method, while asyncio uses Handle objects.
Deferred: Twisted’s Coroutine Primitive
Twisted predates native Python coroutines, so it uses Deferred (twisted/internet/defer.py) as its async primitive. A Deferred is a callback chain. When a result arrives (or an errback for errors), the chain fires sequentially. reactor.callLater() and addCallback() are the building blocks.
@inlineCallbacks (twisted/internet/defer.py) is Twisted’s equivalent of async def. It wraps a generator function, drives it with a trampoline that sends Deferred results back into the generator via generator.send(), and propagates failures via generator.throw(). This is exactly the mechanism asyncio’s Task uses — Twisted invented it.
→ Related: How Python coroutines actually work — the same send()/throw() mechanism (Blog 02)
pollreactor and epollreactor: Scaling Beyond select()
SelectReactor is limited by select()’s FD_SETSIZE (1024 file descriptors on many platforms). For production systems, Twisted provides EPollReactor (twisted/internet/epollreactor.py), which uses epoll on Linux, and KQueueReactor on macOS. These drop in as reactor implementations — the rest of Twisted is unaware of the difference because all reactors implement IReactorCore and IReactorFDSet.
EPollReactor.doSelect() uses select.epoll() to register fds and poll them. The interface is identical to SelectReactor from Twisted’s perspective — only the OS call changes.
threadCallFromThread and thread safety
The reactor runs in one thread. To schedule a callback from another thread, Twisted provides reactor.callFromThread(f, *args), implemented in base.py via a synchronized queue and a wakeup fd (a self-pipe). The calling thread writes a byte to the pipe, which appears as a read-ready event on the reactor’s event loop, causing it to drain the callback queue immediately.
This is the same wakeup-fd pattern used by asyncio’s call_soon_threadsafe(). Both trace the pattern back to the classic self-pipe trick for interrupting a blocking select/poll/epoll call.
Conclusion
Twisted’s SelectReactor is a clean implementation of the reactor pattern: an I/O polling loop driven by select(), a timer heap for delayed calls, and a callback model for composing async logic. Reading twisted/internet/selectreactor.py alongside base.py reveals the full mechanism in roughly 500 lines of Python. If you understand Twisted’s reactor internals, asyncio’s event loop design becomes immediately familiar — they solve the same problem with the same tools.
Leave a Reply