Data transport api’s appropriate for the medium
Despite the wishes of unix, fifo’s are not files. Well they are, but they are fundamentally different from actual files on disk. You can’t seek. But even this thinking is out of date. Seeking made sense back in the day when data was stored on slow spinning magnetic disks. The data comes of 1 byte at a time akin to a fifo or other stream. Seeking involves moving the read-head to the start of the data you want to read. Today most data storage is done on SSD’s. Here the concept of seeking makes absolutely no sense. You are always seeking, and at the same time never seeking. Therefore the correct design for the 21st century is a design that takes into account the fact that your disk isn’t a disk, it’s more like slower, persistant ram.
Files
Files are in a sense very much like memory. Conceptually a long array of bytes. In reality, a long array of blocks. Memory is also conceptually a long array of bytes but actually treated by the cpu and system as a long array of 4096 byte pages. Now consider where computers are headed, Microsoft’s DirectStorage allows your pci-e ssd read/write directly from memory. This allows for way higher data speeds then would otherwise be possible and it frees up the cpu to do other work. Therefore AstraOS goes all in on this concept. The interface to the storage hardware is async read/write of 4096 byte blocks on the block device to pages in memory. And the api is about sending I/O requests for reading and writing many 4096 block chunks at a time. There is no seeking, it’s always async, and it’s always dma. The OS only needs to do some light translation to the requests on their way out. Block 3 of a file might actually be block 568 on the drive.
Fifos/Streams
Streams as they are called in AstraOS require basically no change from their posix counterparts. Each process has in-streams and out-streams. As the names suggest you can read bytes from the in-streams and write bytes to the out-streams. These work almost identically to posix for normal operation.
Graphics as core usecase
Unix is really good at manipulating text files. Writing programs that manipulate text files is easy on unix. Connecting these programs together in useful ways is easy on unix. Why is this? Because unix was designed from day one to aid in the manipulating of text files and character devices. It was so successful at this that we have not replaced it. Except that the unix model is definetly outdated. Just rendering to the screen in a modern environment, 1920x1080@240hz, requires a bandwidth of 1.9 Gibibytes per second. There is no way you can push that much data over a fifo. Therefore we need a new model, we need the unix of graphics. You could say that plan9 was that but plan9 is focused on network transparancy first. AstraOS will be built for the state of the art in terms of local device graphics.
Framebuffers
An AstraOS framebuffer is a description of exactly that, a framebuffer. It has a width and a height, and memory allocated for it. The format used is always 32bit float RGBA, alpha premultiplied. This is way overkill but it ensures that you never will need more dynamic range and you will never need to upgrade to a new format. This means that all programs use the same format and it’s easy to manipulate since the processor has instructions for dealing with 32bit floating point. Unix would not have worked well if you were piping between programs that used different text formats, UTF-8 vs UTF-16 for instance. Framebuffers are mapped into and out of each processes virtual address space, this ensures no copies unless needed. When a framebuffer is to be displayed by the display chip the kernel gives it the ability to access the framebuffer in question via dma. So for an app in exclusive fullscreen the frame data get’s written by the application and read by the display chip with nothing in between, even though the framebuffer is conceptually moving, application -> kernel -> wm -> kernel -> display-chip -> monitor.
Surfaces
Framebuffers aren’t useful on their own, they are the medium of exchange, but surfaces are the higher level organisation. A surface is something that a process can use to output framebuffers. If you are rendering to a window, the windows contents are represented by a surface. If you want to create a htop style terminal gui you would do that through a surface. If you are creating a WM and want to output graphics to your monitors, that too is done with a surface. If framebuffers are the text, surfaces are a type of filedescriptor.
Consumers
Consumers are the process level counterpart to a surface. When a WM for instance is asked by an application for a window, the WM creates a consumer/surface pair. The surface is owned by the application, the consumer is owned by the WM. The application can render and sleep on the surface, the WM has 2 options for using the framebuffers it recieves from the consumer. Either it can access the framebuffer directly and use in it’s compositing. Or it can abdicate it’s rendering responsibilites to the application with a passthrough. Essentially, it can tell the kernel to connect a surface and a consumer that the WM owns, bypassing the WM. This is in my opinion a very elegant way of doing exclusive fullscreen.
PCI-E like bus
Casey Muratori highlights in his talk “The Thirty Million Line Problem” that the reason our operating systems are so bad is because of lack of competition. He points out that the lines of source code to have an operating system is a lot more than 30 million. The reason for this is the great variability of hardware. The current model has NVIDIA creating a unique piece of hardware each generation and then writing a piece of software to talk to their hardware. This piece of software is called a driver and it is the reason new operating systems have no chance to compete with windows or linux. Contrast this with the effort required to create a text mode OS that runs on x86. There are lots of small OS’s that do this.
The solution then is a system wide “ISA”, a machine spec that would allow operating systems to be written for the spec and hardware could be created with support out of the box for all os’s written for the spec. I think the optimal version of this spec looks something like this. A RISC-V RV64-GV system with a pci-e like system bus without the drivers. There is no GPU, instead the risc-v vector extension is used to give great SIMD capabilities to the CPU. The system bus is a high speed interface like pci-e consisting of two parts as viewed by the OS. Messages and DMA. Devices talk to the main system through asynchronous messages mainly. The OS can the tell the bus controller to grant dma access to certain devices for certain memory pages. This way the CPU is still in control of the communication but we can transfer data to and from devices at a much higher rates.
Lets take the example of an SSD. When the OS wants to perform a read, it sets up the controller to allow the SSD to write to certain memory pages, then it sends a message containing an array of block/page number pairs. The SSD can then load the requested blocks and then write them into the correct locations in memory. It can not read anywhere in memory, and it can only write to the pages that the OS has allowed. When the loads have completed the SSD sends a message indicating that the operation is done, to which the OS responds by removing the write permissions and waking up any user threads that were awaiting the I/O operating to be complete.
“AstraOS design concepts as of June 2021” was written by Sam H Smith on the 1st of July 2021, 2021-07-01.