Operating Systems

How Operating Systems Should Be

Last modified: 2009-11-12 10:19:04 UTC

I'm in the process of getting acquainted with two new operating systems: Vista and Leopard. And I'm finding that Microsoft and Apple still haven't gotten it right. So I figured that maybe nobody ever wrote down how an operating system should work, and maybe that's the whole problem. :) So here goes...

Operating systems are way too vulnerable, to user error, to malicious software, and to errors on the part of the application developers. It's possible for users, malware, and poorly-coded software to make system and application changes that result in the breakdown of some or all of the functionality of the computer.

These errors are encouraged by the fact that all of these system and application files are exposed to the user, via the operating system's file management interface (i.e., Explorer or Finder). When opening, saving, renaming, and moving files, the user is constantly exposed to files of which he/she has no knowledge. This is as inconvenient as it is hazardous.

Attempts to solve the problem with user rights of varying degrees have only succeeded in inconveniencing the users as well as the developers. After the 11,000^th time that a user has to click the "continue" button in a security alert dialog box, or type "sudo" in a terminal window, it becomes something that the user just doesn't think about.

What is needed is a system that is designed to not require that the user perform some tasks as administrator, and that simply does not have vulnerabilities as a consequence.

The first door that needs to be closed is access to operating system files, including device drivers. The OS should simply not allow the user, or any application other than OS utilities, to be able to change anything with regards to the OS. Device driver vendors should coordinate with the OS vendor to get their software installed, updated, and patched using built-in OS capabilities. This, of course, puts the responsibility on the OS vendor to make sure that device drivers are fully tested before being automatically installed on the user's machine. But considering what's at stake, and how unqualified the user is in evaluating device drivers, this is not unreasonable. Apple has exercised a great degree of control over the release of device drivers for the Macintosh OS, and this has been one of the central reasons for the robustness of the Mac. First, bad drivers are rarely released to the public, because Apple makes sure that they're tested. Second, Apple knows what drivers are out there, because these drivers have to be reviewed by Apple before they are released. So the testing effort can be effective. Third, Apple doesn't have to grant users access to this part of the OS, which means that users can't screw it up, and viruses can't change these files maliciously.

In short, the only people who should ever be able to screw up the OS should be the OS developers themselves. Everyone else, including users and third-party applications, should simply not have access to OS-level files. To gain access to OS files, users should have to log on as "manufacturer," which will grant them full rights to everything on the machine. There should never be a reason to use this account unless the person is doing OS or device driver development.

The second door that needs to be closed is that one application should not be able to change the files associated with another application. When a well-behaved installer runs, instead of asking the user where the application should be installed, the installer should make a request to the OS for space within a protected area on the hard-drive, into which it can install its files. Along with this request, it will send the OS the name of the executable that it intends to expose to the user. The OS will then create and subsequently manage the hook to that executable, and that hook will be the only way that the user will be able to launch the executable. After installation, outside access to the protected application folder (except via the hook that was registered) will be closed. From this point onwards, no user, nor any other application run by the user, will be able to see or modify the files within this application folder.

The protected application itself will, of course, be able to read and write files within its protected area, and could potentially provide an interface for the user to see and modify protected files. This would probably be poor application design, but the job of the OS is not to eliminate crappy apps. Rather, it is merely the job of the OS to protect itself from user and app error, and to protect well-behaved apps from malicious apps.

Note that once an application has been installed into a protected application folder, the only way for the user to execute that application will be via the hook that was registered at installation time. This has several implications.

First, deleting the hook would effectively un-install the software, since the app would never be accessible again. Therefore, deleting an application hook should be the way that software gets uninstalled, and this should be perfectly clear to the user. It's possible that deleting an application hook should not immediately uninstall the software, but rather, should simply move the hook to an "Unused Applications" folder, from which it can be retrieved later if the user changes his/her mind, or from which it can be deleted later if the user needs to free up some space on the hard-drive. Either way, the significance of deleting an application hook should be extremely obvious.

Second, updating an existing application will have to be accomplished using an existing hook, because only a registered application will have the rights necessary to update files within its protected application folder. To reduce the resultant hardship on the lower tier of developers, the OS vendor should provide an updater mechanism that application developers will be able to integrate easily into their applications. The OS vendor should provide an update system that includes a way of creating and posting compressed and encrypted update files to an FTP server, along with the calls necessary to query the server for updates, download just the files that are needed, and move those files into place during the next application launch. The OS vendor should also provide roll-back capability, in case the application developer releases a bad patch.

Third, applications that support third-party add-ons will have to provide a mechanism for those add-ons to be installed.

Applications that are not capable of operating effectively within these constraints will not be eligible for installation as "protected" applications.

Users logged on as "programmer" will have full rights to all of the protected application folders (but will not have "manufacturer" rights within the protected OS folder). There should never be a reason to use this account except when doing application development where it is important to run the application from its eventual location within a protected application folder. Of course, when logged on as "programmer" it won't be protected, but the app will still be making calls to system services that might make assumptions about the location of the app.

As a convenience to system administrators, there should be a "support files" area outside of the protected application folders, where applications can store runtime-generated data. If the user logs in as "administrator," all of these data folders will be visible and modifiable. Each application, when it registers for protected space, will also be granted rights to a corresponding data folder. Only that application, or users logged on as administrator, will have access to the data folder. This way, rogue apps will not be able to corrupt the data files of another application. It will be the responsibility of the application developer to make sure that administrators will be able to effectively work with data files, such as copying them to other machines to transfer configuration settings and user-generated application support files.

Outside of the protected OS, application, and data folders, there should be a fourth area on the hard-drive for user files. This area will be completely unprotected, except in the sense that other user accounts on a multi-user machine will not have rights to the current user's files.

This, of course, means that malicious software will still have access to the user's files, and will be able to delete, corrupt, and/or steal anything there. Malware could also plant viruses in documents used by applications that support scripting (such as MS Office), and those scripts will then have access to everything within the application's folder. It will also still be possible for the user to install rogue apps into the user area, and these rogue apps will have the same network rights as the user, and they will still be able to access the Internet. So the architecture described above will not eliminate all of the problems associated with user error and with malicious software. But it will guarantee that the OS will be untouchable, and that well-behaved apps, including their support data, will be vulnerable only to themselves by their own design. This will so greatly reduce the footprint of user error and rogue app problems that the remaining issues will be much easier to manage.

The nature of this architecture means that support for legacy applications need not be directly addressed. All legacy applications will install into, and run from, the user area. To put it another way, the user area will correspond to the entire hard-drive in existing OS installations. To support such apps that don't know about the new protected areas, there will be an instance of the previous version of the OS, where the legacy apps expect to find it. When they corrupt this OS, it will be broken for all legacy apps. But this will not affect the new protected OS, nor will it cause any problems for the new well-behaved protected apps.

As an alternative to maintaining a copy of the previous OS on the hard-drive, the OS vendor might decide to permit legacy apps to access the current OS. But the OS will then need a well-defined way of handling bad behavior on the part of these apps. For example, if a poorly-designed application thinks that it needs to write files into the System32 folder, the OS could intercept such actions, and redirect the file I/O to a folder elsewhere. Future attempts to access the so-written file could be redirected to the alternate location. In other words, the rogue app must be allowed to think that it has modified the OS, while in fact the OS remains fully insulated from all apps, and has simply handled all of the requests in an indirect way. While it would be non-trivial to catch every instance of such behavior, this might be easier than coming up with a legacy OS emulator. And this would mean that when a rogue app corrupts the OS, it is really only corrupting its "version" of it. All other apps, including legacy (unprotected) apps, would continue to function.

If this alternative strategy is taken, then without a legacy OS within the "user" area of the hard-drive, the user would see a hard-drive that would be empty except for the files that he/she had explicitly put there, and for the legacy applications that he/she had installed.

Well-behaved apps must effectively manage their temp files, and prevent eventual system shutdown because temp files have chewed up all of the disk space. While this was important before, it will become imperative now, since no third-party clean-up application will have access to these files. To encourage temp file clean-up, the OS should provide a call for creating temp file handles that the OS will then associate with the process ID, and then automatically delete after the session.

Failure to properly handle temp files should prohibit OS logo certification of the app, even if the app was developed by the OS vendor.

OS vendors and application developers frequently decide to store user-specific application settings within the user's folder. While this keeps all of the user-specific files in one place on the hard-drive, which has its advantages, it also exposes the user to a file hierarchy of which he/she has no knowledge. This is as inconvenient as it is risky.

The user should be exposed only to that part of the file system over which he/she rightfully has direct, total control. In other words, the file system should be empty so far as the user is concerned, except for user files that he/she has deliberately created using the "save" feature within an application, or by deliberately copying files to the local machine from somewhere else.

User-specific application support data should be stored in the application support folder. Application-wide settings should be stored in a folder named "All Users" and user-specific settings should be stored in folders named after the users.

Applications that run as services for other applications would inherit the user's rights as well as the rights of the calling application, and then add whatever additional rights they need in order to function effectively. All services would run as child processes of the calling application, and where two or more apps have launched the same service, there will be two or more instances of that service running, each with a different set of rights.

To follow the same model as other applications, services should be accessible through application hooks created at installation time. The only difference would be that service hooks would not be exposed to the user.

Non-protected (legacy) apps installed into the user area would be callable directly, by filename, but would not have any special application rights.

Note that this proposal introduces the concept of "application" rights, which is something that is not present within any existing operating system. An implementation of this proposal could be faked with user rights, assuming the OS vendor wanted to do the wrapper work that would be required. Essentially, existing operating systems provide a way of assigning rights to files on a per-user (or per-group) basis. To instantiate app rights within this framework, the installation process would have to create a new user for the application being installed, and give that user complete control over that application's files. When the (human) user launches that app, the app will then run not as the (human) user, but as the app user, to gain read/write rights within the app and app data folders. The (app) user will also have read/write rights within the (human) user's folders, so that user files can be created and modified.

If these "app" users can be hidden from the existing rights management interfaces, this sort of implementation would not present any additional inconvenience to the user or administrator.