Multics 2nd Generation (Multics^2): Lehman and Belady have studied the history of successive releases in a large operating system. They find that the total number of modules increases linearly with release number, but that the number of modules affected increases exponentially with release number. All repairs tend to destroy the structure, to increase the entropy and disorder of the system. Less and less effort is spent on fixing original design flaws; more and more is spent on fixing flaws introduced by earlier fixes. As time passes, the system becomes less and less well-ordered. Sooner or later the fixing ceases to gain any ground. Each forward step is matched by a backward one. Although in principle usable forever, the system has worn out as a base for progress. Furthermore, machines change, configurations change, and user requirements change, so the system is not in fact usable forever. A brand-new, from-the-ground-up redesign is necessary. -- Fred Brooks, "The Mythical Man-Month", about IBM OS/360 (but it may as well be applied to Unix too, and perhaps even the original Multics) We Unix people have been deceived, and are deceiving ourselves in the worst way possible for all too long a time. One of our core philosophies is backwards / inside out / upside down / flat-out wrong: In unix it's not really true that "everything is a file". Rather it is that "everything is an I/O channel, even including files" We may as well still be computing with all our data accessed via magnetic tape! In Multics the filesystem is merely a logical map of all memory segments stored in the system, each of which may be mapped into a process' address space (assuming appropriate access rights). Meanwhile real I/O in Multics is done via a unified, universal, streams-oriented, and hardware-independent, interface that even works transparently with files as the "device". Multics even allows easy interchange of devices while programs are executing. In Unix everything is a file descriptor (but a file descriptor is not necessarily a file, especially not for sockets and pipes), i.e. everything (except memory) in Unix is I/O, even files. Files must be accessed as if they are I/O streams (though they are seek-able, so somewhat randomly accessible, but that the extreme cost of extra system calls and far more complex algorithms). Unix file I/O is no better than ancient magnetic tape storage, at least w.r.t. algorithm design. (I'm ignoring mmap() because it is far too hobbled to be truly useful as a proper replacement for file access via I/O streams.) Of course in Multics everything is also a file(name), effectively, since files are just segments (the I/O subsystem plays tricks with names that could easily be bound into the same namespaces as data files, though at the time that didn't seem so important in the Multics world). However in Multics the data in a file is just more address space: data is directly addressable and thus randomly accessible, and it appears on demand in the process address space without any thought as to where it was stored. We only think in terms of linear streams of data in Multics when we are forced to communicate with the world outside the process execution space. But that's OK because it doesn't limit what the underlying communications hardware can do, and it fits exactly with the paradigms implied by the underlying hardware and how it links to the outside world. Trying to think of a serial port as an array of data can get you into trouble and it leaves out some extremely critical concepts about the very nature of communications. (Hmmmm.... this probably means Multics with DMA hardware could be made more secure too!) Paging vs. Segments The purpose of paging is to make the allocation of physical memory easier. One may think of paging as the intermediate ground between a fully associative memory, having each word addressed by means of some part of its contents, and a normal memory, having each memory location addressed by a specific integer forever fixed to that physical location. In paging, blocks of memory are assigned differing base addresses. Addressing within a block is relative to the beginning of the block. Thus if association and relative addressing are handled with a break occurring within a normal break of the word (viz. in a binary machine block size is a power of 2), then a number of noncontiguous blocks of memory can be made to look contiguous through proper association. The association between a block and a specific base address can be dynamically changed by program during the execution of appropriate parts of the executive routine. Segments are used not for the allocation of physical memory, but for the allocation of address space. A segment defines some object such as a data area, a procedure (program) or the like. In a sense, each segment corresponds to a part of virtual memory whose size is whatever size, up to a maximum limit, that is required for its content and which can expand or contract as that size changes. In theory, as many such segments can be available to a programmer as necessary. Observe that although segments and pages are two distinctly different entities, they work together to facilitate the allocation of physical memory and virtual memory. Although a large number of segments may be defined, each one having a large number of words, only the currently referenced pages of pertinent segments need to be loaded in physical memory at any time. (The above section is paraphrased from "System Design of a Computer for Time Sharing Applications" by E. L. Glaser, J. F. Couleur, and G. A. Oliver. This paper gives a good abstract overview of the rational and design of the "extension" features required to support Multics on the GE-635 -- i.e. the creation of the GE-645.) Multics securty: The 1970's is the 2020's The high-assurance, military security, mainframe oriented model of the Orange Book is oriented toward the problems of the 1970's. One can assume that Multics systems ran on trusted hardware that was maintained and administered by trained and trusted people. Users didn't buy graphics cards on Canal Street and plug them into the mainframe. In many ways, the security design of Multics-based systems pushed a lot of hard-to-solve issues off onto Operations and Field Engineering. Multics hardware originated from a trusted manufacturer and was shipped on trusted trucks, installed by trusted FEs, and kept in a locked room. (above paraphrased from http://www.multicians.org/b2.html, "4.5 Relevance to current and future security needs") Well what do you know! Those same needs and assumptions can be applied to modern Internet SaaS platforms. Multics and its security as a precursor and model for "cloud computing" In terms of past examples, the substantial Multics efforts over a number of years by Honeywell (initially General Electric) is one of the few. Decades in advance of widespread commercial need Multics addressed the challenge of creating a "computing utility" for which security was a central value proposition. Many decades later essentially this vision has been given the new name of "cloud computing" – unfortunately without significant attention to security. https://intelligence.org/2014/06/23/roger-schell/ Original Multics Goals: - Avoid multiple copies of the same information in memory - Avoid copying or processing each word of memory repeatedly - Control sharing at the segment level - Link subroutines incrementally on demand - Support symmetric shared-memory multiprocessing - Support programming in multiple languages Multics^2 should have: * Single Level Store with a hierarchical filesystem. - Einstein inspired Peter G. Neumann to think in terms of hierarchical systems, which in turn begat the Multics filesystem, and thus Unix and all subsequent similar hierarchical filesystems. - block/page level de-duplication with copy-on-write semantics for all file copies - memory segments are files and vice-versa -- one-to-one with only a hackish streams-based method for accessing segments as an I/O stream for external "serial" communications (and perhaps the benefit of silly old POSIX-compatibility). - All information stored in memory must be directly addressable by any process. I.e. everything that is "just" data, should be accessed directly via pointers into memory. Only interfaces where data must be serialized for some reason, e.g. network communications, ttys, etc. should be done via some I/O stream mechanism. - Access control can be performed at each reference to memory, or at least at each mapping of a page of memory. - abstractions such as buffering in STDIO are pure overhead! * Extensions to C that work similar to the old extensions to PL/1 for multics? Or just stick to pointers? We already depend almost entirely on malloc() and mmap() which return bounded regions, and we even expect that if we have to grow them that they may move. * also possibly offer a tagged, search-based filesystem overlay - http://eecs.harvard.edu/margo/papers/hotos09/paper.pdf * natural dynamic linking with a search path handled by the link(call) trap handler, and pure shareable compiled code (i.e. no modification of code segments is allowed or required). * ability for ALL programs to make all non-basic dynamically linked libraries and "frameworks" entirely optional! The most basic functionality MUST NOT require installing all possible bloatware that a program might want to use to implement all of its options. * designed for real-world general purpose use, as well as for research - fully POSIX-compatible runtime emulation with fread() and fwrite() being effectively memcpy(). * in general it should try to fix or avoid all the "stupid" things in various systems APIs In general the purpose is not to be a research OS, but rather to be a re-implementation of ideas that have fallen by the wayside. BTW, Chris offers a good discussion of what "success" means for a research OS: https://utcc.utoronto.ca/~cks/space/blog/tech/SuccessForResearchOSes Ideas: In many ways Multics^2 is very similar to The Amber Operating System: http://www.mit.edu/~cbf/thesis.htm Transform Xen into, or otherwise use a hypervisor to implement, Multics^2 - idea from MirageOS (http://queue.acm.org/detail.cfm?id=2566628) - maybe each process is a whole virtual machine - you run either a small runtime, or a whole user-dedicated OS, in each VM - threads are always virtual-CPUs in the VM? - use unix-like job control to have multiple threads running in one process? - the hypervisor implements an emulation of segments for a slightly enhanced virtual-CPU+MMU even if the underlying CPU doesn't do segments - this is, perhaps unfortunately, much more in keeping with the heavy-weight nature of the original Multics processes, though if multiple vCPUs can be assigned to each domain then threading is available and good inter-thread communications, e.g. like goroutines and channels, could ease the pressure on having multiple processes for a given task. - See also EthOS: http://www.ethos-os.org/ https://www.cs.uic.edu/~spopuri/ethos.html Or maybe just use hypervisor technology to implement rings? On the other hand Jonathan Shapiro said: "The idea of adapting paravirtualization to a microkernel for use within an operating system is, frankly, silly. Several existing microkernels, including L4.sec (though not L3 or L4), KeyKOS, EROS, Coyotos, and CapROS, have isolation and communication mechanisms that are already better suited to this task." http://www.coyotos.org/docs/misc/linus-rebuttal.html - but then Shapiro isn't known for being open-minded about some OS concepts. :-) Rings have a huge advantage when there's hardware ring protection: Multics can do a direct procedure call to a higher privilege level, with the full generality of high-level language argument passing, without taking a trap or invoking mediation code. The return from higher privilege (lower ring) to lower privilege (higher ring) is also direct. - the kernel code is directly shared with all processes, i.e. kernel code segments are mapped into the address space of each process -- just another segment, or several. - A process can call kernel functions directly using normal function call mechanisms -- there are no "system calls" as in the unix sense. For a practical experiment a 64-bit address space should suffice, even if rings are in software (though we would need multi-segment files): 2 bits ring # 30 bits segment # 32 bits segment address (are there any modern large-scale data apps that expect more than 32-bits of address space for individual arrays? yes, but not many) If we want to avoid multi-segment files and we can do rings in hardware: 24 bits segment # (16 million segments per process) 40 bits segment address (1,099,511,627,776, 1 terabyte) or even: 4 bits ring # 20 bits segment # (1 million segments per process) 40 bits segment address (1,099,511,627,776, 1 terabyte) or: 2 bits ring # 22 bits segment # (4 million segments per process) 40 bits segment address (1,099,511,627,776, 1 terabyte) We can also split the address space into regions using "flag" bits in pointers, and thus also avoid needing multi-segment files at the same time!: Highest bit 63: 0 = regular address, 1 = memory mapped files (i.e. is this a "segment" address?) next highest 62: very large files (or just if any one of the VLF# bits is non-zero -- requires masking and testing for non-zero as opposed to testing just one bit) next 8 bits 54-61: VLF file number (256 very large files open) 2^52 byte max (4,503,599,627,370,496) 4 petabyte next 21 bits 33-53 regular file numbers (1,048,576 files) 2^32 byte max (4,294,967,296) 4 gigabyte 1 2 3 4 5 6 0123456789 123456789 123456789 123456789 123456789 123456789 123 R V VF E L LI G F FL # # ?E ? Mask off the flags and file number and you have the index (offset). Take 2 bits from the regular file numbers to do rings in software. Get rid of ordinary file descriptors, have open(2) return a pointer ala mmap(), and have socket(2) be the only call that returns a file descriptor. So, how do we do dynamic linking with this scheme? Notes about 80386: I've been doing an awful lot of reading and research about Multics again recently, including looking at the x86 segmentation and protection model in view of how it could work for Multics. So far as I can tell IA-32 is more or less _exactly_ what is needed for Multics, or a Multics-like system, with some possible limitations. A virtual address consists of a 16-bit (13+1 are address bits) segment selector and a 32-bit segment offset. The selector is used to fetch a segment descriptor from a table (actually, there are two tables and so the "plus one" of the bits of the selector is used to choose which table) (and the other two bits are the Requested Privilege Level). The 64-bit descriptor itself contains the 32-bit address of the segment (called the segment base) 21 bits indicating its length, and miscellaneous bits indicating protections and other options. The segment length is indicated by a 20-bit limit and one bit to indicate whether the limit should be interpreted as bytes or pages. (The segment base and limit "fields" are actually scattered around the descriptor to provide compatibility with earlier version of the hardware.) If the offset from the original virtual address does not exceed the segment length, it is added to the base to get a "physical" address called the linear address. If paging is turned off, the linear address really is the physical address. Otherwise, it is translated by a two-level page table as described previously, with the 32-bit address divided into two 10-bit page numbers and a 12 bit offset (a page is 4KB). Warning: Apparently the Atom processors have very sucky performance, especially when using non-zero-based code segments. The main addressing limitation in 32-bit x86 is the relatively small 13-bit segment index held in the segment registers. With the two possible descriptor tables that gives an absolute maximum of 2^14-1 segments per process, though practically I'd guess the GDT is only useful for code segments and possibly a very few truly globally needed file segments, so that leaves just 8192 segments for stack, data, and general file access by each process. That's still twice what the practical limit was for packed pointers on the old Honeywell 6180, and 4-8 times the average number of open file descriptors permitted in the average modern unix-like system, so perhaps it's not really an important limit. Perhaps there's some convenient way for a process to switch LDT's on demand too. Also, it's between 2 and 8 times the number of open file descriptors generally limited to by modern Unix systems. Obviously the x86 only has 4 rings, but I don't currently see that as a problem. I think all x86-64 implementations still have full compatibility modes for x86 (IA-32, i.e. 32-bit) protected mode addressing with full segmented addressing and paged segments. They pretty much have to in order to run all 32-bit software. (FYI, the FS and GS registers retain their use as segment descriptor table indexes even in 64-bit mode (i.e. they can still have non-zero base addresses), but without any real protection features, I think.) Paul Green's Multics VM paper suggests another, possibly more efficient, way to avoid multi-segment files: Concatenating segments would be simple if incrementing the word offset past the last word carried into the segment number. You wouldn't want this by default, but with a bit in the SDW you could control which segments were concatenated and which were not. So, can we modify x86 segmentation to allow concatenated segments? This would allow direct, trivial, access to files larger than 4GB without having to re-invent multi-segment files again. There should still be a bit available in a segment descriptor to indicate that it is to be concatenated with the following descriptor. Getting pointer arithmetic to work right might be tricky though. - "segment" registers point into a segment descriptor table (SDT): 2 bits ring # (requestor's privilege level) 1 bit segment descriptor table indicator (global vs. local) 13 bits segment descriptor index (8192 segments per SDT) 32 bits segment address (offset) - segment descriptors (SDT entries, or SDWs in Multics parlance): 16 bit segment limit (in either bytes or 4k pages, 4GB max) - can a process reload its own LSDT without a ring#0 call? Ideally though the x86_64 would have segmentation re-enabled, thus doubling the size of index registers and hopefully also doubling the size of segment selector registers (though maybe not increasing the descriptor table size by the same power of two -- other flag bits might be useful in the visible part of segment selector registers), and of course adding at least 4 more bytes (but ideally doubling) the size of segment descriptors. IA-32 TSS: - dynacube OS uses just two TSS segments in the GDT: http://dynacube.net/ Note about Honeywell 6180 limits: - in practice packed pointers were used with a limit of 4096 segments per process. Other CPUs: CHERI Memory Segmentation to Support Secure Applications "A segment mechanism that implements the capability model of safe, programmatic memory protection" http://www.cl.cam.ac.uk/research/security/ctsrd/pdfs/2013essos-cheri.pdf "CHERI can do anything Multics could do: segmentation, paging, dynamic linking, ring-structured software" -- Peter G. Neumann in "An Interview with..." by Rick Farrow in ;login:, Winter 2017 vol. 42, no. 4 More about "Capabilities" Capability-Based Computer Systems Henry M. Levy This book was published by Digital Press in 1984. It is still the most thorough survey and description of early capability-based and object-based hardware and software systems. The book is now out of print and the copyright belongs to the author, who makes the material available here for viewing or downloading, in Adobe Acrobat PDF format: https://homes.cs.washington.edu/~levy/capabook/ - see also Plessey System 250 - capabilities vs. access control lists: J. H. Saltzer and M. D. Schroeder. 1975. The protection of information in computer systems. Proceedings of the IEEE 63, 9 ( Sept. 1975), 1278–1308. They thought revocation of access was one of the major area where capabilities fell short. In segmentation and with the introduction of "generation" numbers in the segment table revocation is simply a matter of incrementing the generation number. See also "Using segmentation to build a capability-based single address space operating system", Wuyang Chung https://www.bsdcan.org/events/bsdcan_2023/sessions/session/129/slides/50/slides.pdf Mill https://millcomputing.com/ I'm also extremely interested in exploring what hypervisor support in modern x86 CPUs might bring to the table in terms of building a new Multics-like system. What if each process is a virtual machine or maybe even an LPAR (but with overlapping memory, of course)? Virtual CPUs as threads? I think the PowerPC might work too (with software rings?), and I'm also interested in seeing how difficult it might be to add segmentation and ring "hardware" to an ARM, SPARC, or MIPS architecture, perhaps as an experiment in an FPGA implementation. There's also a patent I found about emulating segment bounds checking in a simpler RISC-style MMU: https://www.google.com/patents/US5652872 For now IS IT possible to just do segment + offset calculations manually on non-segmented architectures? - how might CPU support features for a hypervisor help? - does an MMU help emulate segments? - segments as pages? How did the S-1 support a variable boundary between the segment number and the offset? (S-1 segments could take up the whole address space of the system, apparently.) Here's a linux extension that provided multiple global address space mappings for files via mmap(): Srinivas Palnati, Gautam Barua, "Persistent Storage for 64-bit Systems", International Conference on Information Technology (CIT-2001), Gopalpur-on-the-sea, India, December 20-22, 2001. Is there any performance advantage to loading a whole library in one segment (when there are no adverse security implications?). Otherwise every independent function gets its own segment (we would still have a billion to play with on a 64-bit address even with 2-bits reserved for rings!). - probably yes -- at least for when the library calls lots of other members of itself, since then internal calls won't need fixups on the initial invocation. MMU Paging systems can be used to implement segments. From the Amber paper[1]: Architectures that have the necessary hardware for demand paging and a large address space (VAX and IBM 370 with Extended Addressing option) can still use the techniques of segmentation by allocating contiguous ranges of pages to represent a segment. They will not however, have the feature of trapping references that overflow a segments boundaries that the Multics and S-1 hardware provide. This feature is primarily a debugging aid, but it can be an important one. however there are problems with only having page tables: On a segmented machine like Multics, the S-1, or the Prime 500 series, access to a segment is determined from bits in a user's segment descriptor word. On a paged system each page table word has access bits. (The access we are referring to here is read, execute, or write to an area of main memory. The S-1 actually has bits in both the segment and the page table words and it and's them together to determine effective access.) This unfortunately means that two users who have the same segment mapped in with different access rights cannot share the same page table. Files vs. I/O -- everything is a file (descriptor) vs. SingleLevelStore File systems have (at least) two undesirable characteristics: both the addressing model and the consistency semantics differ from those of memory, leading to a change in programming model at the storage boundary. Main memory is a single flat space of pages with a simple durability (persistence) model: all or nothing. File content durability is a complex function of implementation, caching, and timing. Memory is globally consistent. File systems offer no global consistency model. Following a crash recovery, individual files may be lost or damaged, or may be collectively inconsistent even though they are individually sound. from the abstract of: "Design Evolution of the EROS Single-Level Store" by: Jonathan Shapiro, Jonathan Adams Issues with using memory as storage, at least with respect to MMAP: - note that with a Single Level Store all views of the data are implicitly guaranteed to be unified. - a system all similar to msync(2) could be used to "secure" (flush) all writes to a segment, thus careful use of it (like a mutex of sorts), combined with some effect of something akin to mlockall(2) to guarantee that no writes happen until the msync() call, could guarantee write order to secondary storage, though of course efficiecy here requires that writing of a whole page is as efficient as writing some small part of the page (normally the case if I/O is done with DMA transfers I think) [[ how could this be made more transparent? -- perhaps by providing a write(2) like interface? ]] - it is hard to escape from point #3, though presumably that's algorithm dependent. - similarly point #2, but that is in part a language problem. From: Howard Chu via Cyrus-devel To: Cyrus Devel Message-ID: <1c46d2af-c16f-aeb3-9e32-291d3ac4e216@highlandsun.com> Date: Fri, 30 Nov 2018 11:09:23 +0000 I've covered the reasons for/against writing thru mmap in my LMDB design papers. I don't know how relevant all of these are for your use case: 1. writing thru mmap loses any control over write ordering - the OS will page dirty pages out in arbitrary order. If you're using a filesystem that supports ordered writes, it will preserve the ordering of data from write() calls. 2. making the mmap writable opens the possibility of undetectable data structure corruption if any other code is doing stray writes through arbitrary pointers. You need to be very sure your code is bug-free. 3. if your DB is larger than RAM, writing thru mmap is slower than using write() syscalls. Whenever you access a page for the first time, the OS will page it in. This is a wasted I/O if all you're doing is overwriting the page with new data. 4. you can't use mmap exclusively, if you need to grow the output file. You can only write thru the mapping to pages that already exist. If you need to grow the file, you must preallocate the space, otherwise you get a SEGV when referencing unallocated pages. And a side note, multiple studies have shown that skiplists are not cache-friendly, and thus have inferior performance to B+tree organizations. A skiplist is a very poor choice for a read/write data structure. Obviously I would recommend you use something carefully designed and heavily tested, like LMDB, instead of whatever you're using. There's one point in favor of writing thru mmap - if you take care of all the other potential gotchas, it will work on every OS that implements mmap. Using mmap for reads, and syscalls for writes, is only valid on OSs with a unified buffer cache. While this isn't a problem on most modern OSs, OpenBSD is a notable example of an OS that lacks this, and so that approach always results in file corruption there. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/ - Bron replied to say: This is not a concern at all - twoskip is deliberately designed such that it does a single write and then flush to "dirty" the file, all changes made while dirty are fully revertable if it crashes, and then it does a fsync (msync now I guess!) before a single write which clears the dirty flag. So long as a single 256 byte write is consistent, it's safe. hardware transactional memory https://dl.acm.org/citation.cfm?id=2618405 https://fsgeek.ca/2018/04/27/the-future-of-synchronization-on-multicores-the-mulitcore-transformation/ Security What are the modern threats to a Multics-like system that are different from what it faced originally? How should it deal with bugs in programs responsible for the interpretation of structured content such as graphics (HTML, etc., etc., etc.) - security sensitive processes could be prevented from mapping in new code segments just before they start running (and of course it is an error to try to jump to a data (or stack) segment, and the stack(s) and heap segment(s) do not share the same address(offset) space). - separate processes for separate functional purposes, e.g. browser in one, mail in another, DBs each in their own, compiler is run in its own, etc. Each network server in its own, etc. Even an editor like Emacs could run in its own process. How to deal with the administration issue -- i.e. how much sysadmin can be automated to the degree that a non-expert can own and run a machine? Jonathan S. Shapiro argues against segments in: http://seclists.org/risks/2014/q2/1 How distributed can it all be? - e.g.: like Apollo Domain/OS and/or Amoeba - full virtual cloud for all computing devices? - any computer can join the cloud, local resources can be shared or private, some other resources (filesystem and CPU, etc.) can be "more trusted" than the whole cloud. - use something like Apple's security processor to make one device the only thing that can decrypt some segments - additionally be able to store keys into it to allow identities to share devices, migrate between devices, make devices "disposable" What to write it all in? - some "safe" C? with modules - see ~notes/C - why not plain C? "Undefined behaviour". It is a thing "defined" for C, for one: ... we have C compiler maintainers who assert that any behavior not defined in the C standard is not something where they have to pick one behavior and stick with it, but can instead declare it an impossible situation, that code which relies upon common assumptions is [therefore] buggy, and proceed to optimize away the security checks. If I program in C, I need to defend against the compiler maintainers. If I program in Go, the language maintainers defend me from my mistakes. Phil Pennock https://bridge.grumpy-troll.org/2017/04/golang-ssh-radix/ A better language would not define anything as "undefined behaviour", ever! So, not Standard C as defined by modern standards. - Hare, https://harelang.org/ - "Hare definitely does not have is the same kind of 'undefined behavior' that C compilers use as a license to do whatever they want to your program" - "all strings are always valid UTF-8. We also store the length alongside the string, allowing NUL to appear in the string contents, and for len(str) to be an O(1) operation." - can be useful everywhere C is useful, BUT... - it lacks multithreading in the standard library - Go, with extensions and a different runtime? - can a Go-like language automatically and reliably detect when, e.g., features are used which might depend on a hosted runtime (as opposed to the bare metal runtime), e.g. garbage collection. - see GO-OSe and bootgo https://github.com/boomshroom/goose https://github.com/jjyr/bootgo - V https://vlang.io/ https://github.com/vlang/v - hmmm.... this V language is very simple, much like Go, but without (currently) a garbage collector -- cleanup is done dynamically when functions return, etc. V is a statically typed compiled programming language designed for building maintainable software. It's similar to Go and is also influenced by Oberon, Rust, Swift. - Has enums! (and even generics!) - super-safe: - No undefined values - No undefined behavior - Bounds checking There's no garbage collection or reference counting. V cleans everything up during compilation. If your V program compiles, it's guaranteed that it's going to be leak free. - there's a C to V translator and C interoperability is easy - claims compiled binaries have no dependencies (but don't seem to be static-linked -- they're too small, for one) - currently implements concurrency with system threads, but claims to be adding a coroutines and a scheduler - "V"ery interesting! Extremely good candidate! - possible drawbacks: difficult to manage pointers? - there's also Jai: imperative static/strongly typed C-style language by Jonathan Loom https://inductive.no/jai/ https://github.com/BSVino/JaiPrimer/blob/master/JaiPrimer.md No implicit type conversions No header files Jai code can be built on top of existing C libraries - need to know more about its runtime - Nim https://nim-lang.org/ - it's a bit/very annoyingly python-like in look - it has no runtime compiler library -- executables are stand-alone! * Nim generates native dependency-free executables, not dependent on a virtual machine, which are small and allow easy redistribution. * Fast deferred reference counting memory management that supports real-time systems. * Nim is self-contained: the compiler and the standard library are implemented in Nim. - Zig -- http://ziglang.org/ (by Andrew Kelley) ready and running, apparently. BUT oh, gawd -- it's currently written in C++ and requires CMake to build the compiler (because it is a thin front-end to LLVM) (though apparently comes with its own built-in build system so that Zig programs don't need anything like CMake, etc.) BUT, this may be changing (divorce from LLVM): https://kristoff.it/blog/zig-new-relationship-llvm/ See also: https://lists.sr.ht/~sircmpwn/public-inbox/%3C874kh7ovmk.fsf%40dismail.de%3E Some people think Zig is too far away from being C-like to be "good". - Eiffel: https://www.eiffel.org/ too wordy, and possibly suffering the same kind of runtime issues as Go. - Rust: http://rust-lang.org/ (overly excessively complex and baroque, so, probably not) - D: http://dlang.org/ (overly complex, probably not) - Ocaml (probably not -- too different) - XC -- Xmos C - C with Communicating Sequential Processes on multi-core hardware - was heavily influenced by the occam programming language https://en.wikipedia.org/wiki/XC_(programming_language) - something with unsigned integers by default, which can be specified in exact bit widths (a bit like PL/1) - What about Turing Plus? Still available and maintained: - http://research.cs.queensu.ca/~cordy/pub/downloads/tplus/ - only for i386(?) - somewhat wordy (pascal-like begin/end, etc.) - but still maybe better than Oberon/Modula-2/whatever? - Scheme? - there are several good Scheme-to-C compilers: https://justinethier.github.io/cyclone/ http://gambitscheme.org/wiki/index.php/Main_Page - what about a multics-like OS with a garbage collector in the "kernel"??? Why not? - Ada? e.g. as was done for: https://wookey-project.github.io/ewok/index.html - investigate Ada's development: "DEPARTMENT OF DEFENSE REQUIREMENTS FOR HIGH ORDER COMPUTER PROGRAMMING LANGUAGES" * STRAWMAN issued in April 1975 * WOODENMAN issued in August 1975 * TINMAN issued in January 1976 * IRONMAN issued in January 1977 (revised in July 1977) * SANDMAN not published but circulated in January 1978 * STEELMAN issued in June 1978 * PEBBLEMAN issued in July 1978 * PEBBLEMAN Revised and issued in January 1979 * STONEMAN issued in February 1980 What to run it on: Standards: - which to follow, which to ignore, which to violate on purpose? - UTF-8? UTF-32LE? (or UTF-32BE), but UTF-32* still does not have a fixed byte count per displayed character due to combining characters. (also up to four times the size of UTF-8 depending on how many of the characters are in the ASCII subset) On Unix systems, UTF-32 strings are sometimes used for storage, due to the type wchar_t being defined as 32 bit. Here's an old paper by the unix group defending the LP64 programming for 64-bit architectures, and to some extent espousing the very large address space of 64-bit pointers: http://www.unix.org/version2/whatsnew/lp64_wp.html HP claims to be designing a new architecture computer with a "flat" memory design much like Multics needs: https://www.hpl.hp.com/research/systems-research/themachine/ People: - Multicians - Karl Aurbach - Wayne Radinsky See Also: Plessey System 250 - https://en.wikipedia.org/wiki/Plessey_System_250 SOMBRERO AMOEBA: Tanenbaum et al - uses purely the concepts of "objects" and "capabilities", with any style of filesystem (implemented by a set of three user processes running on each CPU) mapped on top of these. - AMOEBA was independently improved by Stefan Bosse to create FSD-AMOEBA http://fsd-amoeba.sourceforge.net/ which then begat: VAM (Virtual Amoeba Machine) - A virtual machine for distributed systems based on OCaML and the Amoeba OS (2000-2006) Vertex Amoeba - An extended and updated version of the distributed Operating System Amoeba from the Vrije University for x86 platforms and embedded PCs (1999-2004) MUNGI: A Distributed Single Address-Space Operating System - "private per-process data can therefore only be provided by using relative addressing from a base register." So why not just use segments? :-) - papers provide good consideration of many issues w.r.t. distributed operation - doc archive at http://www.nic.funet.fi/pub/doc/OS/Mungi/ No. 9302 issued in March, 1993 Title: A Distributed Single Address-Space Operating System Supporting Persistence Author(s): Gernot Heiser, Kevin Elphinstone, Stephen Russell, Graham R. Hellestrand Abstract: Persistence has long been difficult to integrate into operating systems. The main problem is that pointers lose their meaning once they are taken out of their address-space. We present a distributed system which has a single address-space encompassing all virtual memory of every node in the system. This design has become possible (and practicable) with the advent of 64-bit microprocessors. In our system, every pointer retains its meaning independent of its location, even across nodes or on secondary storage. No restrictions are imposed on the use of pointers by application programs. Hence persistence is naturally and elegantly integrated into the system. Further features are uniform addressing and unlimited sharing of data, and memory protection based on password capabilities, making the system easy to use. A reliable paging protocol ensures that the impact of node crashes on other parts of the system is minimised. No. 9314 issued in November, 1993 Title: Mungi: A Distributed Single Address-Space Operating System (The text of this report has been acepted for ACSC-17) Author(s): Gernot Heiser, Kevin Elphinstone, Stephen Russell, Jerry Vochteloo Abstract: With the development of 64-bit microprocessors, it is now possible to combine local, secondary and remote storage into a large single address-space. This results in a uniform method for naming and accessing objects regardless of their location, removes the distinction between persistent and transient data, and simplifies the migration of data and processes. This paper describes the Mungi single address-space operating system. Mungi provides a distributed single level address-space, which is protected using password capabilities. The protection system performs efficiently on conventional architectures, and is simple enough that most programs do not need to be aware of its operation. MONADS A capability-based distributed shared memory. F. A. Henskens, J. Rosenberg, and J. L. Keedy. In Proceedings of the 14th Australian Computer Science Conference, pages 29.1–12, 1991. - MONADS depends on specialised hardware LOCUS LOCUS: a network transparent, high reliability distributed system G. Popek, B. Walker, J. Chow, D. Edwards, C. Kline, G. Rudisin, and G. Thiel. In Proceedings of the 8th ACM Symposium on OS Principles, pages 169–77, 1981 OPAL http://homes.cs.washington.edu/~levy/opal/opal.html - supports a huge (64-bit) global (network-wide) "pure" address space divided into virtual segments which are named by capabilities. Shared code is in segments called modules. All segments are potentially persistent and recoverable. Persistent segments create a virtual filesystem as a single level store. - the paper notes disadvantages of hardware-based segmented addressing in the likes of multics: (1) cross-segment pointers are impossible; (2) multiple pointer forms must be treated differently by application programs; (3) software must be used to also co-ordinate the use of segment registers to create the illusion of a single address space. (but that last point is silly because in a global pure address space software must be used to co-ordinate the paging table and VM hardware) - a major disadvantage of virtual segments is that copy-on-write is impossible (for data segments containing internal pointers) without supporting some form of "pointer conversion" (because internal pointers are always global pointers, never offsets from the segment base) - note that in Multics cross-segment pointers are actually possible if there are tags on segment values such that initially the segment value points to a name which can be resolved at runtime to a mapping to an actual live segment -- i.e. in the way dynamic linking works in multics HYDRA http://www.cis.upenn.edu/~KeyKOS/ http://www.eros-os.org/eros.html http://www.coyotos.org/ ATLAS - first single-level store system? GOTHIC: https://hal.inria.fr/inria-00075356/file/RR-1202.pdf SCAP - secure capability architecture, derivative of Cambridge Capability system Karger, Paul Ashley Improving Security and Performance for Capability Systems PhD thesis, March 1988, Cambridge University. http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.64.7003&rep=rep1&type=pdf This dissertation examines two major limitations of capability systems: an inability to support security policies that enforce confinement and a reputation for relatively poor performance when compared with non-capability systems. Apollo AEGIS (and Domain/OS) https://doc.lagout.org/science/0_Computer%20Science/0_Computer%20History/old-hardware/apollo/Apollo_DOMAIN_Architecture_Feb81.pdf https://doc.lagout.org/science/0_Computer%20Science/0_Computer%20History/old-hardware/apollo/AEGIS_Overview_1985.pdf https://doc.lagout.org/science/0_Computer%20Science/0_Computer%20History/old-hardware/apollo/AEGIS_Internals_and_Data_Structures_Jan86.pdf (via https://filepursuit.com/directory/5128568-AEGIS-Overview-1985-pdf/) Stratus OpenVOS https://www.cs.rochester.edu/u/scott/papers/1992_TR418.pdf very interesting project to add SLS and proper dynamic linking to IRIX, intended for 64-bit systems (see also their Hemlock dynamic linking papers) http://www.greenend.org.uk/rjk/tech/fork.html [1] http://www.mit.edu/~cbf/thesis.htm https://lovelaceos.sourceforge.io/ -- unix-like OS in ADA(2012) Measurements of sharing in Multics Warren A. Montgomery http://dl.acm.org/citation.cfm?doid=800214.806549 - this paper notes that a distributed Multics could be feasible (though not practical due to memory limitations at the time of writing) MULTICS and Plan 9, the Big Bangs in Distributed Computing Systems' Universe Seyedeh Leili Mirtaheri http://drmirtaheri.ir/publication/Journal-papers/a1-1.pdf IO-Lite: A Unified I/O Buffering and Caching System Vivek S. Pai, Peter Druschel, Willy Zwaenepoel https://scholarship.rice.edu/bitstream/handle/1911/17117/1384394.PDF?sequence=1&isAllowed=y http://cs.brown.edu/courses/cs161/papers/io-lite.pdf [DEAD]http://www.cs.rice.edu/~vivek/IO-Lite/TR97-294.ps Effects of Data Passing Semantics and Operating System Structure on Network I/O Performance (1997) Jose Carlos Brustoloni http://www.cs.pitt.edu/~jcb/papers/thesis.ps http://reports-archive.adm.cs.cmu.edu/anon/1997/CMU-CS-97-176.pdf Towards High-Performance Application-Level Storage Management Simon Peter https://www.usenix.org/system/files/conference/hotstorage14/hotstorage14-paper-peter.pdf GEMSOS - Gemini Multiprocessing Secure Operating System for IA32 - Written in Pascal and leverages IA32 (initially iAPX 286) security features to build a highly secure [Class A1 Verified] RTOS kernel (architected by Roger Schell) - "Designing The GEMSOS Security Kernel" paper mentions that for multiprocessor systems they support the concept of "processor-local" memory http://www.helenos.org/ -- portable micro-kernel muli-server (like minix) Apparently Schiller developed a verifiable kernel prototype on an 11/45 that provided segmented virtual memory implemented at a kernel layer. M.A.G.I.C.: Mockapetris And Gregory's Interactive Computer https://gunkies.org/wiki/MagicSix More Taste: Less Greed? (or) Sending UNIX to the Fat Farm C H Forsyth (1990???) http://www.collyer.net/who/geoff/taste.pdf MenuetOS: "MenuetOS is a pre-emptive, real-time and multiprocessor Operating System in development for the PC written entirely in 32/64 bit assembly language." http://www.menuetos.net/