Arcan is a display server thing that seems to have streaming concepts in the core: an Awk for multimedia.
Consider application windows to be video streams. Applications become appsrc elements in a gstreaming pipeline.
The display server is then simply a sink with capabilites of moving and composing streams around like "windows". Gstreamer has some flexible network transparancy. If the network is fast and cpu is slow faw buffers can be streamed, or if network is slow and cpu is fast compressed buffers, as described here. Streams can also be encrypted.
Open question: how would mouse events (and other events) be piped back to the application? There seems to be rich message passing functionality in gstreamer but would it work for this? Possibly gst_message_new_custom. See also this. Actually navigation events are exactly what we are looking for.
If the display server simply "pipes" frames into GL textures, these can then be rendered on geometry planes, onto 3d objects and with transparency. Streaming is less efficient than vector based methods when the application visuals or audio don't change much, but then most applications are in motion (scrolling, video content, moving graphics etc).