Foreword

Welcome!

Welcome to NightOS' documentation! This book will show you how the system is designed, how it works, and all the little things it can offer to the end user as well as the features availables for application developers.

Before reading this documentation, please note that I'm developing this project on my own and I'm by no mean an operating system expert. I do not even as a professional low-level developer though I have some knowledge on how low-level software and hardware work to some extent.

This means there are surely many flaws in the very low-level aspects, especially in the microkernel, like the way hardware devices are handled and mapped in memory. If you find a flaw, feel free to fill an issue.

The main point of this project is the many middle-level/high-level aspects, not the very low-level ones which will take quite a lot of time to design completely and surely require the knowledge of people more experimented in this field than I am.

On the other hand, specification documents are not like specification documents you may have seen from, let's say, WhatWG or other consortiums. As NightOS is only a prototypal O.S., the documents you will find here are meant to be easily understandable. Specification documents are meant to describe completely a specific concept or part of the system, but without being hard to read. Some will argue it's not close enough to a "real" specification, but I think it's enough for now, regarding the very early state of the project.

Also, this project is far from being finished, so many things are still missing from the documentation. I frequently add new design documents and complete existing ones, but feel free to create a new issue if you think something should be added to the project :)

By the way, you can also find answers to the most frequently-asked questions here.

What is NightOS?

NightOS is an operating system aiming to replace ancient systems like Windows, Linux or MacOS. Well, this is the ideal goal but given how much Windows/Linux/MacOS and other systems are deeply installed in today's computers, NightOS is more of a theorical operating system that shows what we could get if we hadn't to maintain legacy compatibility with ancient architectures.

What? Another O.S.? Why?

The problem with current OSes like Windows, Linux or MacOS is they were built on bad roots. At the time they were created, security wasn't a concern like today, and computers weren't nearly as powerful as they are now. Which means a lot of things we have to deal with today in order to maintain a good level of security or performance isn't met at all with these systems, because they can't just be rewritten from scratch or break compatibility with older pieces of software.

NightOS, on its side, starts from a blank page. There is no compatibility to maintain on the software level, and everything is designed from the start to be future-proof, meaning it will be easy to add new security features for instance, without breaking any software compatibility.

What NightOS isn't

Unlike other operating systems such as Linux which are able to run on a very wide variety of devices, the goal of NightOS is not to be as flexible as this would require to make some big compromises and hugely complexify the system. Here is a list of things NightOS isn't designed for running on:

  • Embedded devices
  • Multi-CPU computers (although this decision might change in the future)
  • Non-64-bit CPU architectures
  • Headless devices (such as servers)

What's the project's structure?

The project is composed of three distinct parts: the kernel, the system, and the desktop environment.

The first one is the part that allows the software to access the hardware, like writing files to the disk or making network requests. It's a microkernel, meaning it's easier to maintain and has a reduced attack surface for potential security issues than a monolithic kernel like Linux. In its early stages though, we will use a the Linux kernel as a base, and the microkernel will be last "brick" to the project.

The system is the higher-level component that deals with everything that is not directly related to hardware or graphical interface. For instance, applications, user accounts and permissions are all managed by the system. It's a read-only part the user cannot modify by itself.

The desktop environment, finally, is the part the user sees when launching NightOS. It noteably contains the status bar, the windows manager, the notifications panel, and all graphical-related things.

The goal of NightOS is to provide an operating system that is far more secure, robust and performant than existing systems, and this for all users.

What license is this project distributed under?

The NightOS project uses the Apache 2.0 license.

Special thanks

Thanks to rmRander for providing the image that served as a base for this project's logo. You can find the original picture here.

Frequently-Asked Questions

Welcome to NightOS' FAQ. This document answers to common questions about the project, completing the foreword.

This project seems too good to be true

Well, it is in a way. NightOS can be designed as such a secure and robust O.S. because we chose to not maintain any backward compatibility with existing software - understand that existing applications from Linux, MacOS or Windows will not be compatible with NightOS. This is why we can afford to introduce so many new features and requirements current operating systems cannot.

The problem is that backward compatibility is absolutely mandatory when it comes to creating a general-purpose operating system. All applications would have to be developed again (in fact, they would just have to be adapted for NightOS, but that's still quite a bit of work) in order to run on this system, which is of course unfeasable today.

This is why NightOS will probably never be completed. The current goal of this project is to build a theorical O.S. that demonstrates how good of an O.S. we could make if we hadn't to worry about backward compatibility. When the theorical specifications will be complete, we shall start to develop the O.S. itself but that will require a lot of work beforehand.

Still, we do all we can to maintain as much compatibility as we can. For instance, we are currently investigating how existing Linux applications may be ran on NightOS, with a system call wrapper that would allow them to run correctly on the system, without compromising security, but with lacking features of course.

What's the current state of the project?

The project is currently in its very early stages. You can find more informations about this in the ROADMAP.

Will this project replace Windows/MacOS/Linux/... one day?

No. Absolutely not. Because, as I said in sooner, it's not viable for an operating system today to just get rid of all backward compatibility with existing software, simply because what users are interested in is applications first, and then the system itself - is it secure, stable, performant, and so on.

Imagine someone introducing the perfect operating system, but with almost no application on it. Would you get rid of your Windows/MacOS/Linux/... system for this one? Surely not, at least not as your everyday system.

This is why this project is for now purely theorical. It aims to describe how great an OS could be if we did not have to care about backward compatibility. Changes to see it in a usable state one day are like very low.

How does this project relates to NightOS v1, v2 and v3?

The current NightOS project is very different from its previous versions (v1, v2 and v3). Originally, it only intended to be a robust and secure desktop environment, written in JavaScript. Performances would of course be very bad, but it was more of a challenge than a serious project. Also, the project was meant to be a Linux kernel under the hood to get rid of the "low-level part". I seriously lacked of knowledge about low-level concepts at this point in time.

This project does not use JavaScript anymore, but Rust, which is both performant and very robust to the most common memory-related bugs (unless you use unsafe code). There is no more "challenge" idea involved in this new version, only the aim to make the best possible (theorical) system.

In terms of similarities:

  • The v1 was more of a draft, which had a lot of problems up to its roots. It was a pretty bad version overall, and was created when I did not have a real knowledge about how an operating system actually worked ;
  • The v2 was a bit more realistic, but still far from being mature enough ;
  • The v3 was more serious and ambitious, but it was still a desktop environment and not designed to be a full operating system, plus it was not designed for optimal performances

Who are you?

My name is Clément Nerma (my last name is a pseudonym). I'm a back-end developer that makes stuff since I'm 10-year old and I love to touch to low-level concepts, like operating systems.

Why did you create NightOS?

I created it as an answer to the frustration modern OSes provided me. All modern systems have many stability and security problems that cannot be resolved easily because they need to maintain backward compatibility with older pieces of software. This is where I had the idea of creating NightOS: an operating system that is both robust (understand stable, with mechanisms to minimize data loss in case something goes wrong) and secure, with many security features available to the average user and some more complex ones for advanced users.

How can I help?

Help is very welcome but there's not many things to help with currently, as the system is still being designed. You can still help by suggesting improvements or fixing an error (whatever it is, a design problem or a simple typo) by submitting an issue.

Project

This folder contains the documentation about the project itself - how is will be developed, maintained, its release schedule, etc. Please note that this part is far from being complete, and many informations will be missing and/or incomplete.

Project's ROADMAP

WARNING: This document is only a draft and as such far from being complete. All informations described here are subject to change anytime.

The project will be cut in the four phases described below.

Conception

The first part is to write conception documents about how the system work, both at a high-level (concepts, features, native applications etc.) and low-level (processes management, kernel design...).

This project is currently in this state.

Validation

Once all conception documents are ready, they will be validated one by one, and frozen. Breaking changes after validation are still possible but should be avoided as much as possible.

Implementation

Once all documents have been validated, the different project's pieces can be implemented.

It may happen that, during implementation, some validated documents get modifications in order to improve or fix some specific points. Each modification will need to be validated in order to preserve the stability of these documents.

Below are described the different phases of the implementation.

Phase 1: Unoptimized kernel

The first piece to be implemented will be the kernel. This is required as the kernel is needed in order to implement higher-level pieces like processes. Still, as not everything is needed right from the start, it will developed with no special matter of performances. In this state, it will be referred to as the name of unoptimized kernel.

Phase 2: Low-level implementation

Once the unoptimized kernel is ready, two implementations will start in parallel:

  • Kernel optimization (as the unoptimized kernel will be really, really slow)
  • System development

The system development will consist in building the following pieces successively:

  1. The BIOS/UEFI bootloader
  2. The system bootloader
  3. The process manager
  4. The native process manager

Phase 3: Applications & System optimization

The third phase consists in developing the native applications, which are meant to be used by the end user, as well as optimizing the system in parallel and improving its stability.

Starting from this point, all conception documents are definitively validated, which means they cannot receive breaking changes anymore, only new points/features.

This also coincides with the system's first alpha release, which will allow persons who are external to the project to test it and suggest improvements.

Phase 4: Completion

The last phase's purpose is to ensure the system is in a fully usable state, in other words to stabilize it.

The kernel and system will also need to be properly optimized, and once all these things are done a first beta release will be deployed.

Evolution, Optimization & Maintenance

The Evolution, Optimization & Maintenance (EPM) part is pretty explicit: it's all about improving NightOS' existing features, introducing new ones as needed, improving performances and stability, and fixing new bugs.

This part will last forever, as for all operating systems - or until the project dies.

Project development

WARNING: This document is only a draft and as such far from being complete. All informations described here are subject to change anytime.

Languages

The project will be developped in Rust, with first-class support for this language. API interopability will be assured for TypeScript as well as Python in the future.

Usage of architecture-specific assembly will be reduced as much as possible, being only used in two cases:

  • The desired behaviour cannot be reached without assembly (e.g. direct register or stack manipulation)
  • Extremely performance-critical pieces (e.g. memory management, context switching, etc.)

Hardware requirements

WARNING: This document is only a draft and as such far from being complete. All informations described here are subject to change anytime.

This document presents the minimal hardware requirements required by NightOS in order to work. This is also means NightOS will work on any computer with these attributes.

  • Processor: x86_64 processor
  • Memory: at least 256 MB or RAM
  • Storage: to be determined
  • Graphics card: Required for graphical access (otherwise see BareEnv)

Recommanded configuration

  • A modern process with TPM (ideally 2.0 or higher)

Concepts

This folder contains documentation for each high-level concept of NightOS. They are designed to be easily readable and understandable.

  • Applications - the way to run software on NightOS
  • Libraries - sharing identical behaviours between multiple applications
  • Users - sharing a computer between multiple persons

Applications

Applications are the system's way to handle software.

NOTE : This document is only an introduction to how applications work.

How applications work

An application is a set of executable files and resources. They are the only way to execute code, as direct binary programs are not supported.

Any user can install applications, which will only be available from his account. Administrator users can install global applications, which are available to every user.

Installation methods

Applications are installed through an application package via Skyer, the applications manager. There are several installation methods:

  • From the store
  • Directly from the application's package (sideloading)
  • As a volatile application

From the store

Applications can be downloaded from NightOS' official applications store (available via Stellar).

  • For closed-source applications, the store only provides pre-compiled programs
  • For open-source applications, the store provides both pre-built programs as well as the source code

For the latter, user can choose either to build the program from source, using the appropriated build tools in order to optimize performances, or to simply use the pre-built programs (which is the option by default).

Sideloading

Applications sideloading (installing an application directly from its package) follows strict rules determined by the sideloading mode, which is either "disabled", "store-checking" or "unsecure".

Disabled mode prevents all sideloading ; it's not possible to install applications from their package in this mode. Volatile applications can stil be run.

Store-checking mode allows sideloading but will first make the system check if the application's AID matches an existing application on the Store. If so, it compares the application's signature to the Store application's one. If they don't match, the application is considered malicious and won't be installed.
Note that this mode only works while connected to internet, as the system needs to check the Store to ensure the application is not malicious. If the computer is offline, sideloading will be disabled.

Unsecure mode allows sideloading without any checking, which is highly dangerous as it allows spoofing.

The sideloading mode can be changed in the control center.

Volatile applications

Applications can be also be ran as volatile applications, which means they are not installed on the disk. There are four different methods:

  • Fully volatile: the app's data are removed when the application closes
  • Session-scoped: the app's data are stored on disk until the system shuts down
  • Locally persistent: the app's data are stored within a data file located in the same folder
  • Persistent: the app's data are stored in a dedicated folder, also enabling common data between users

By default, volatile applications are ran in locally persistent mode. In this mode, the system first checks if a file with the same name as the application's package but with the .vad (Volatile Application's Data) exists. If so, it opens this file as the application's storage. Then, when the application wants to store some data, it is stored inside this data file.

Note that VAD files are disguised VSF files.

Volatile applications running as persistent do not appear in the applications list and can only be managed through a specific option in the Control Center. Their executable files are not stored anywhere and stay in the application's package, while only their data are stored on the disk. This allows to run the same application several times without losing any data and without worrying about a data file. This also allows to store common data between users.

Note that the store has an option for installing applications as volatile.

Permissions

Permissions allow to finely control what an application can do (or not). See the permissions feature document for more details.

Commands

Application can expose shell commands. Multiple commands can be exposed without any risk of clashing as the command name must be prefixed by the AID first.

For instance, if an application with AID superdev.utils exposes an get_time command, the final usable command will be :superdev.utils.get_time.

This is quite a long name but allows to prevent any clashing between commands. Shell scripts must use imports at the beginning to specify the commands they will use. This also allows to directly spot any missing application required for that script. For more informations on the shell language, check the related specifications document.

Note that, by default, shell prompts (not scripts) will allow to directly use commands such as get_time in the short form if no other application exposes a command with the same name, for convenience.

Commands work by launching the application with a specific execution context.

System applications

Some native applications are part of the system itself and are called system applications as such. They get a few specific features:

  • Access to system-reserved features
  • Ability to create system services
  • They cannot be uninstalled

System applications cannot be removed in any way, as some of them are critical for the system to function properly.
Native applications which are not system applications can be removed, though.

These applications can be updated independently of the system itself, thus their version may differ from the system itself.

Services

Application can provide services to run at startup. They must be specified in their manifest.

Services run at startup with the usual application's permissions, and services get one process per active user.

Libraries

Libraries allow to share a program between multiple applications.

How libraries work

Libraries allow to share runtime elements to applications. Unlike the latter, they can be installed in multiple versions. This means you can have three versions of the same library installed at the same time on your computer.

When an application needs to use a library, it explicitly indicates the list of versions it is compatible with. The system then gets the related version and provides it to the application.

Libraries cannot perform interactive tasks by themselves, which means no background process, no commands exposure, no permission granting. Applications simply use libraries as helpers to achieve specific tasks.

For more details, check the specifications document.

Users

Concept

Many computers are intended to be shared by multiple persons. In such case, it is useful to separate the data of each user and to prevent other users from accesing another's.

By default, there is two user accounts: the System and the Administrator. They are called virtual users because it's not possible to log in these accounts. Other users (the ones which are created manually) are called custom users.

Each custom user has a dedicated data directory called the homedir, in /home/[username], as well as a list of files and directories it can read and/or write. By default, each user gets access to:

  • Its homedir in /home/[username] ;
  • The mounted periphericals in /media ;
  • Its temporary directory in /tmp/[username]

User permissions level

Each user has an associated permissions level:

  • Level 1 (Phantom): the user's data are erased after the computer is turned off ;
  • Level 2 (Standard): the default user account type ;
  • Level 3 (Administrator): can run programs directly as administrator ;
  • Level 4 (Main administrator): administrator that can manage global storage encryption ;
  • Level 5 (System): reserved to a specific builtin user that can only be accessed by the main administrator using a special procedure

When a user wants to perform a task it does not have the privileges to, it can (by default) ask to run the task as another user. The other user's credentials are then required.

It's not possible to ask to run a task as system or as administrator as these accounts are virtual and do not have passwords as such, except if an administrator user is tries to run a task as administrator.

Note that there is always one and exactly one main administrator. This user account is created during the installation process, and can be transferred later on to another administrator account.

Dangers of an admin. account

The problem with administrators account is that they can do almost anything (except some very specific things like editing the system's files). They can change system settings, read and change other users' data, and even run background processes at startup. They also have all possible privileges as they can edit their own.

This is why it's extremely discouraged to have two administrator users on the same computer, unless the two accounts are used by really trustworthy persons. As such, a large warning is shown if you try to create a new administrator user.

User Privileges Elevation (UPE)

Users can ask to perform a task with the privileges of another user, such as running a program as administrator. This uses the User Privileges Elevation (UPE) system, builtin the sys::perm service.

In such case, the program is still run as the current user, but with the privileges of both the running user and the user specified to the UPE.

Running a program with UPE requires to know either the other account's password, or to have an authorization from this user. For instance, admin. users have an authorization to use the Administrator account, without providing any password, although a human confirmation is required.

Each request, successful or not, is logged in the log file at /etc/logs/upe.

Complexity level

Each user can define a complexity level, which will affect its default settings.

A higher level of complexity will make the system display more informations, give more details about errors that may require some technical knowledge but providing additional features and informations in exchange.

The levels are:

  • Beginner: only show basic informations, make the permission popups as simple as possible
  • Standard: the default complexity level
  • Advanced: shows additional informations, displays some error reports, makes the permission popups more precise and verbose
  • Power: shows very detailed informations, displays all error reports, makes the permission popups display exhaustive informations

Users' data encryption

See encryption.

Child and supervised users

Child users are users that are supervised by the parental control. Supervised users are users that are part of a domain.

Groups

Groups are a set of privileges defined by the administrator. Access to some folders or application's permissions (e.g. microphone access) can be restricted to a specific group, for instance. When a user is created, it can be put in a group. It then automatically inherits all of the group's privileges, in addition to his own ones. The administrator can add and remove users from groups with the control panel.

When a user is in a group, the group's privileges cannot be revoked for this user.

User privileges

User privileges indicate the list of actions a user can perform or not. You can find more in the specifications of the sys::perm system service.

Features

This folder contains explanations about each major feature of NightOS. Each document is designed to be relatively easy to read and understand.

  • The balancer - improve performances by balancing processes' priority
  • Crash saves - prevent data loss at maximum with crash-proof data saving
  • Domains - manage a network of computers
  • Encryption - encrypt the whole storage and individual user accounts
  • Freeze-prevention system - prevent the system from freezing when all RAM and CPU power are used
  • Parental control - manage children access to the computer
  • Permissions system - prevent applications and users from doing whatever they want
  • Sandboxes - isolate applications to prevent them from harming important data
  • Synchronization - synchronize settings between multiple computers

Balancer

The balancer is a kernel feature that allows to get more performances out of most important applications.

How the balancer works

The balancer allows to manage the priority of userland processes - and only them. Here is the list of its features:

  • Increase priority of this application: gives a priority of 8 to the process linked to the active window ;
  • Give maximum priority to this application: gives a priority of 10 to the process linked to the active window ;
  • Set this application with maximum priority: always give a priority of 8 to this application's processes ;
  • Suspend/resume this application: see below ;
  • Enter performance mode: see below

Application processes suspension

Application processes can be suspended, which is an equivalent of pause where they don't run at all. This is achieved by setting their priority to 0.

A suspended application can then be resumed, and because it was just suspended it will instantly run again, without any data loss.

When a process is suspended, all its child processes are, too.

Performance mode

The performance mode performs the following actions:

  • For all userland processes with a non-null priority, set their priority to 1 ;
  • For all the processes of the application related to the active window, set a priority of 10.

This makes all other applications running a lot slower, but the current one will run a lot faster. The priority is re-calculated whenever the active window changes.

When a fullscreen application uses more than 50% of CPU in fullscreen, or when it asks for it, an overlay suggesting to enter performance mode is shown.

Performance mode is automatically exited when the related process exits.

Crash saves

Crash saves prevent most data loss caused by a crash, in all applications supporting it.

How crash saves work

Every minute (this delay can be changed in the registry), a notification is sent to all running applications. Applications can then answer with their state data, which should contain every data required for the application in order to be restored to its exact current state later. They may join a title message, which is expected to be their main window's title - if they have ones. They can also answer with a specific message telling they won't give a crash save data during the current collect. The signal will still be sent on the next collect process (e.g. a minute later). A last answer method is with another message telling they won't give any crash save for the running instance. The signal won't be sent again for this instance of the application. This message is most of the time encountered when the application doesn't implement the crash save process.

If the application didn't answer when the next collect occurs, the signal is aborted, but the next one will be sent.

When a crash save has been collected for a given application, it is stored in /home/[user]/appdata/[appname]/crashsaves/[timestamp]_[pid].csf.

NOTE: Crash saves' intregrity is controlled during the boot process.

All native applications support crash saves.

Domains

Domains are a set of computers running NightOS which are linked to each other. The way a domain's computers behave and interact are defined in the Domain Supervisor, which is only available to the domain administrator user.

It behaves like an extension of the parental control, though they are two completely distinct features and domains offer a lot more features, while being made not to manage a child's access to a computer but to manage thousands of comupters at once.

The concept

Domains enable all features of parental control (without the dedicated application though), as well as the following ones:

  • Mount a common storage between computers
  • Use remote user accounts for log in
  • Restrict access to applications
  • Restrict available permissions to users
  • Restrict installation and update of new applications
  • Manage how applications and the system are updated
  • Monitor CPU, RAM and storage usage on all computers
  • Get access to every user's storage (unless per-user encryption has been explicitly allowed)
  • Get remote terminal access to every running computer
  • Get virtual desktop access to every running computer
  • Put computers to sleep, hibernation, log out current user, power them off or reboot them
  • Start any computer remotely (if the computer does support it, e.g. through PoE)
  • Limit disk usage per user (in disk usage percentage or absolute value)
  • Limit the session duration per user
  • Limit the number of physical and virtual cores per user
  • Limit the amount of memory per user

Domain supervisor

The Domain Supervisor is a system application that shows up on any computer that is part of a domain. Only domain supervisor users can see it, but every user can run it through command line (though it will ask for a domain supervisor's username and password).

Here is the list of options the domain supervisor proposes:

TODO

Encryption

Encryption allows to prevent unauthorized access to a data with a password.

Installation defaults

During installation, a check-up is performed to check if the system supports Secure Boot and if it has a modern TPM. If both are available, it will suggest to enable full-storage

Relayed encryption

Related encryption is the combination of full-storage and per-user encryption. It works by encrypting the global storage using the master key, but users' data are only encrypted using their own user key. This avoids double encryption and still remain perfectly secure.

Full-storage encryption

Full-storage encryption encrypts the entire system storage. It is incompatible with USGE (only one of these two can be enabled at a time).

At any time, it can either be enabled or disabled through the the control center.

NOTE: The ability to encrypt the storage globally, change the master password or decrypt the storage is by default reserved to the main administrator, but this privilege can be given to normal administrators, though this is highly discouraged.

Passwordless encryption

The installer runs a quick check-up and if the computer is considered secure enough (which notably requires the presence of a TPM and support for Secure Boot), it will suggest to use a passwordless method. The random key is then stored in the TPM and the bootloader is checked by the UEFI using Secure Boot to ensure it has not been compromised, avoiding possible data extraction in case of the device being stolen.

If this method is selected, the whole installation will be encrypted using that key. On startup, the UEFI will check the bootloader's integrity through Secure Boot, and let it start if valid. The (unencrypted) bootloader will then retrieve the key from the TPM and decrypt the final bootloader, which will handle the remaining boot steps.

A recovery key is generated, which must be backed up somewhere, in case the bootloader asks for it.

Master password encryption

If the computer isn't deemed secure enough or if the user chooses to, a master password will be asked for. This password allows to encrypt the whole installation by using a global password.

On startup, the system will run the (unencrypted) bootloader which will ask for the master password. It will then retrieve the encrypted master key and try to decrypt it using the provided master password. If it succeeds, the boot process can continue.

This is more secure than the passwordless method as it does not depend on the UEFI's Secure Boot which reduces the attack surface, and also prevents cold boot TPM attacks, ensuring that (if implemented correctly) the storage will never be decrypted without the master key.

Per-user encryption

Any non-guest user can also use a built-in system tool to encrypt its data using its own password. This way, the user's data become unreadable without his password, making even administrators unable to read his data.

The encryption/decryption key is generated automatically when the user account is created and stored in the user's profile data (in /etc/sys/users).

When enabled, the encrypted directories are:

  • The homedir at /home/[username]
  • The tempdir at /tmp/[username]

This feature is enabled by default, but can be disabled by the administrator. Also, domains communicate by default the secret key to the domain's manager.

Per-user shared global encryption (USGE)

USGE is a method used to perform near full-storage encryption, without a master password and without the need of an UEFI / TPM. It is does not prevent modifications in case of physical access to the device, but prevents data from being extracted if the device is stolen.

When configured, it will wait for the user's name and password to be input in the login screen to decrypt a shared master key, which will be used to decrypt the global storage. User's data will then be decrypted using the user key, just like relayed encryption. The main difference between the two being the way the storage itself is encrypted.

This allows to not have to input a master password, not rely on an UEFI's Secure Boot system and prevent cold boot TPM attacks, while still providing near full-storage encryption + per-user encryption for all users.

This feature forces encryption for all users and requires every user to set up a strong password.

Which method to use?

Every method is suited better for a specific type of environment.

If you want the maximum level for security, be the only one to use your computer and enable only full-disk encryption with a strong master password and a strong user password.

If you want to prevent data from being extracted in case of theft, enable relayed encryption or USGE.

If you want to have the most seamless experience, use USGE. You will not be prompted for a master password but still have system and user data encryption.

Freeze prevention

Freeze prevention prevents nearly all computer freezes, by reserving a little amount of resources to the system. It is enabled by default, but it can be disabled during installation process or from the control panel.

How freeze prevention works

The system forbids applications to access a fixed amount of RAM, called the freeze prevention resource (FPR), which is by default the smallest between 1% of the total RAM and 8 megabytes.

When more than 99% of CPU and/or RAM is used, the system puts itself in freeze prevention mode (FPM), which consists in listening to a specific keyboard shortcut (by default, Ctrl+Alt+P), controllable using the keyboard or the mouse (with a special block cursor, not changeable by applications) which asks user if they want to terminate gently the more resources-consuming application instance (by sending the TERMINATE signal), to kill it, to kill all instances of the application, or to see the list of running application instances with how much resources they consume, to make a choice on another one.

If developer mode is enabled, another option is added to access a tiny shell containing a set of commands made to list and manage running applications in a more powerful way.

Parental Control

Parental control allows to manage a child's access to a computer running NightOS.

How it works

Parental control works using (at least) two accounts: the parent's account, and the child's user account, which is related to as a child user. Note that multiple child users can be managed using parental control.

The parent account then gets access to an application called "Parental Control", which allows to access all parental control-related features for each child user.

It is also possible to get important notifications by email, and even to install a smartphone application (iOS/Android) which brings the parental control application to phones, allowing to manage child users even when not at home.

Integration

Parental control's settings are available to third-party applications using specific permissions. This way, some applications can provide an additionally layer of security for child users, like preventing access to adult contents in a game store.

Features

Restrict usage hours

This feature allows to restrict usage of the computer by the child user during specific hours in the day. These hours can be set independently for each day of the week, allowing for instance to set specific hours in the week-end. It's also possible to set specific hours for some specific weeks or periods of times (e.g. for four specific days in a row), allowing for instance to set specific hours for vacations.

Child user will not be able to log in outside of these hours. If they are logged in when the upper limit is reached (e.g. only 8 AM to 9 AM is allowed and the child user is still logged in as 9 AM), a warning message will be shown 15 minutes before reaching the time limit, 5 minutes before, and 1 minute before, allowing the child to save its data.

When this final minute comes, a pop-up shows telling the session is going to be closed in 60 seconds, and a cooldown is shown in a closable pop-up balloon in the taskbar. An option is also shown to extend the session by up to fifteen minutes, sending a notification to the parent's phone as well as an e-mail.

The parent can then do nothing and the session will extend for a maximum of fifteen minutes, or reject it and the computer will shut down after 1 minute. Then another option shows to ask if the parent wants to permanently disable time extension.

Session time extension can be enabled/disabled during the parental control installation process (enabled by default). If disabled, the time extension option will not be shown to the child user during the timeout.

Restrict session duration

This feature allows to restrict how much time a session can be active, allowing for instance to limit a child user's usage to one hour per day. Like the usage hours, it can be set independently for each day of the week, for specific weeks, and for specific periods.

Session time extension applies here as well, and works the same way as for restricted usage hours. The related settings are independent though, allowing for instance to enable session time extension for usage hours but not for session duration, or the opposite.

Restrict access to applications

This feature restricts access to some applications, either using a whitelist (listing all the allowed applications), or using a blacklist (listing all the forbidden applications).

It's possible to automatically while/black-list new applications.

Restrict access to websites

This feature restricts access to some websites, either using a whitelist or a blacklist. It's also possible to apply these restrictions only to compatible web browsers. Whitelist is strongly discouraged outside of web browsers-only usage, as it may prevent some applications to work properly.

Whenever an application tries to access a forbidden website, the request will fail with a specific code indicating it's forbidden by parental control. Applications can still enforce this rule but this requires a specific permission that cannot be allowed by child users.

These restrictions are enabled through the built-in firewall, Vortex.

Restrict mature contents

The child user's birth date can be declared during setup, so its age (not its birth date) will be available to every application which asks for it (with a specific permission, which is by default automatically granted to all applications which ask it).

This allows, for instance, for web browsers to block websites that declare themselves as showing adult contents. Or even more precise protection, like a movies store not showing movies whom age limit is beyond the child user's age.

Restrict installation of applications

This feature prevents child users from installing and running volatile applications. By default, installation is allowed but sends a notification to the parent user asking if the child user is allowed to install the requested application ; while running volatile applications is simply disabled.

Controlling session remotely

This feature allows parents to log out the child user remotely. This will show a pop-up indicating the session will be closed in 60 seconds, behaving just like usage hours restriction. It may also provide session time extension based on how the log out was triggered (log out the child user or log out the child user and disable session time extension) and the setting selected during the setup process.

Sandboxes

Sandboxes allow to test an application without applying modifications to the system.

How sandboxes work

A sandbox is an execution mode where an application's modifications to the disk are not applied directly to it, but instead to a virtual drive stored in its sandboxes folder. When the app. exits, a confirmation overlay proposes to apply the modifications to the real storage - it's also possible to see the changes before applying them.

In developer mode, it is possible to export sandboxes and import them on other computers.

Persistent sandboxes

Sandboxes can also be created as persistent, which prevents them from being removed after the app. exited. Instead, the next time it will run, the same sandbox will be used again.

Puppet sandboxes

Sandboxes can be controlled by another application (this requires an administrator permission, though). This allows to automatically accept or decline every API usage, like permissions or I/O requests. Such sandboxes are called puppet sandboxes, and they can be especially useful for testing.

Synchronization

Online backup and synchronization is performed through Cloudy.

Technical

This folder contains technical documents which essential describe low-level parts of NightOS.

A global overview of the system can be found in the overview document.

Overwiew

This document presents a global overview of NightOS. It is only a simplified representation of the system, but allows to better understand its architecture and organization.

The referenced documents are mostly technical overviews themselves, which is why links to their associated specification document are also provided (marked as (specs)).

Note that this is only an overview, and as such many topics will not be covered by this document. For more informations, refer to the global document.

Program executions

NightOS doesn't allow to run standalone binaries. Every running code is either:

System services are immutable and run for the entire system's lifespan, while applications can be opened and closed at anytime, and can also run as multiple instances in parallel.

Permissions

While the system has every right on the system itself, applications are restricted and can do almost nothing by default.

Capabitilies can be granted through permissions (specs), which are designed to allow precise control on what an application can do or not, while requiring as few user interactions as possible.

Hardware access

Hardware access is performed through two layers:

Hardware drivers

Unlike most operating systems, hardware drivers are simple applications with no specific integration in the kernel.

Any application can register itself as a driver (specs) using the hardware service (specs).

The relevant driver for each hardware device is selected using various criterias (specs).

Hardware access performances

Hardware access is performed through syscalls (specs) and signals (specs), which use CPU interruptions.

The access process is often:

  • A userland process notifies a system service
  • The system services contacts the hardware service (specs)
  • The hardware service contacts the relevant (specs) driver (specs)
  • The driver performs the requested task by communicating with the hardware through the hardware service
  • The action's result is then transmitted to the hardware service, which then transmits it to the system service, which in turns transmits it to the userland process

Although this process can seem a bit long, CPU interruptions and forced threading (specs) makes the theoric latency low enough for intensive use.

Filesystem access

Accessing and managing filesystems and their content is faster than common hardware operations thanks to direct storage access (and direct driver access for worst-case scenarios).

The typical process involves communication with the filesystem service which communicates directly with storage devices.

For filesystems that aren't natively supported, a filesystem interface is involved to communicate with the storage driver service, which then communicates with the hardware service. This involves a higher latency, but is only limited to edge cases and remains in an acceptable range of performances.

Data loss prevention

Data loss prevention works essentially through crash saves, which ensure applications' state is regularly saved on disk in case something goes wrong.

User interface

The user interface is entirely managed by the desktop environment, which can be any application exposing the relevant service (specs).

Users management

Each person using a computer can have its own user account.

Children protection

Children can get their own user account with parental control.

External security

External security is guaranteed through by-default encryption, and additional per-user encryption.

For developers

Developers can access additional features through the developer mode.

Controller

The controller is a system library that manages permissions of processes.

It is concretely represented by the sys::perm service.

Notion of scope

Permissions are split into several scopes:

  • The application scope contains the permissions a given process is borned from ;
  • The user scope contains the permissions the user who launched the application has ;
  • The mode scope contains the permissions the execution mode (either system or userland) has.

The mode scope restricts the user scope, which itself restricts the application scope. This means that, if the application scope specify a permission that is not covered by the user scope, it is not applyable to the process. This allows to prevent applications and users from getting too high permissions.

The mode scope prevents applications from performing harmful tasks such as writing the system. Only system applications, which run in system mode instead of userland mode, gets an unrestricted mode scope.

The perm system library

The perm system library is an interface for the controller which allows processes to check their own permissions, ensure they can make I/O requests before effectively making them, and extend their permissions (see below).

Permissions extension

A process can, at any moment, send a permission extension request (PER) using the perm library. It allows to gain a new permission by showing an overlay the user can accept or decline. If the permission is accepted, the requested permission is added to the process' one - and sometimes to the application's one.

If the requested permission is out of its maximum scope (e.g. asking for write access to /etc while being ran as standard user), the request is rejected.

Developer mode

The developer mode enables several features that are not available to default users. Only administrator users get access to them when they are enabled.

How to enable

Developer mode can be enabled either by typing the following command in the terminal:

adm central --enable dev-mode

Or, while holding the Ctrl Alt and Maj key, type DEV (three letters).

A warning dialog will appear. If you confirm, it will display an UPE dialog for the administrator account. Then a final confirmation dialog will ensure you're sure about what you're doing. Then development mode will be enabled (no reboot is required).

Features

Application proxies

Application proxies are applications that intercept all applications calls to the system on-the-fly. It is useful for applications debugging.

When a proxy is set up for an application, all system calls sent to this application will be transferred to the proxy, which will be able to do whatever it wants with it. It's possible for the proxy to never answer the signal, to change its content before actually sending it to the system, etc.

Basic usage of an application proxy is as a "listener": it only listens to specific signals and logs them, without cancelling them and/or modifying them. This is useful to check all filesystem access requested by an application, in real time.

Devices

Connecting a new device

When a new device is connected to the computer, a device handler file (DHF) is created in the /dev directory (see filesystem structure).

Depending on its type, it will be put in a different directory, as shown in the above document. Note that new category directories may be introduced later on.

The filename itself is a random unique identifier made of a single lowercase letter, a digit, another lowercase letter and finally another digit.

For instance, if a hard drive is connected to the computer, the DHF may be something like /dev/sst/f5t2.

Interacting with a device

Device handler files are not simple files ; they can only be used through the sys::hw service.

Different actions may happen depending on the device's type:

  • Camera devices: when an application asks to capture a photo/video, the device will be suggested to take the images from
  • Microphones: when an application asks to capture sound, the device will be suggested to capture the sound from
  • Sound output devices: the device will be available for playback
  • Network adapters: the device will be available to make network requests on
  • Other supported wireless devices: depends on the type (Bluetooth adapter, ...)
  • Basic/persistent storage devices: when possible, the device will be automatically mounted in /mnt and visible in the files explorer

There are many different types of devices, all can be found in the specifications of the sys::hw service under the "Driven device type" (DDT) section.

For uncategorized devices (in /dev/etc), a popup is shown to the user, to indicate the connected device is not recognized. Some applications may still be able to interact with it (for instance, a storage device using an unsupported protocol).

Device handler files persistence

When a device is connected, its DHF is not removed. Instead, if a process tries to interact with the DHF, the sys::hw service will indicate the device is currently not connected.

When the device is connected again, it is associated to the same DHF again. This allows applications to persist the device's location and use it later on.

Custom device handler filename

It's possible to give a custom handler filename for intuitivity. In such case, a new DHF is created in the related directory, but the old DHF is kept anyway. This allows to not break compatibility with applications that rely on the old DHF.

The two DHF will point to the exact same device, without any difference. The custom DHF can be removed anytime, although this is discouraged as applications may rely on it.

Also, custom DHF names must always be longer than 4 characters, to avoid confusion with automatically-generated DHFs.

File Formats

This document presents the file formats natively handled by NightOS. Some files are only handled by optional first-party applications that may not be installed on the computer, so some formats may not be supported in specific installs where such applications have been removed/were never installed.

Common formats

Common file formats are natively handled:

  • Text files (.txt, .md, ...) with Gravity
  • Audio files (.mp3, .flac, ...) with Sonata
  • Image files (.png, .jpg, ...) with ShootingStar
  • Video files (.mp4, .mkv, ...) with Milkshake
  • Archive files (.zip, .tar, ...) with Blackhole
  • E-book files (.cbz, .cbr, ...) with Reader
  • E-mail files (.eml, .vcf, ...) with Postal
  • Web files (.html, ...) with Rocket

Virtual storages

Virtual storages are files that contain one or several virtual filesystems. Different filesystems can be put in, but all must be supported natively by NightOS in order to properly work without needing to install any additional application.

The filesystems can be encrypted individually. The whole storage can also be encrypted. They can also be compressed.

They have multiple purposes: store encrypted files, virtual filesystems for sandboxed applications, etc.

Virtual storage files have the .vsf extension and are opened using Blackhole.

Application packages

Applications can be installed from standalone files called application packages. Installable applications have the .nap (NightOS Application Package) extension, while volatile applications have the .nva (NightOS Volatile Application) one.

They are opened using Skyer.

System updates

The system is intended to be updated through the update section in the settings, but sometimes it may be required to install updates offline for specific reasons (e.g. no network connection available at the moment). In such case, it is possible to download incremental updates as system update files.

They may contain one or several updates, and are only installable on a very specific version of the system, to avoid missing some other incremental updates.

System update files have the .nsu (NightOS System Update) extension and are opened using the control center.

Application updates

Applications are intended to be updated through the update section in the settings, but sometimes it may be required to install updates offline for specific reasons (e.g. no network connection available at the moment). In such case, it is possible to download incremental updates as application update files.

They may contain one or several updates, and are only installable on a very specific version of the application, to avoid missing some other incremental updates.

Application update files have the .nau (NightOS Application Update) extension and are opened using the control center.

I/O nano-manager

The Input/Output Nano-manager, formerly known as Ion, is a part of the system which treats input/output requests from processes.

It is concretely represented by the sys::hw service.

Hardware access

When a process tries to access the hardware, it must go through Ion, which will allow it or not to interact with the desired device.

System services such as sys::fs or sys::net use Ion to deal with the related hardware devices.

Agnosticity

Ion only permits agnostic access to hardware, meaning it does not have knowledge of the performed action (filesystem access, sensor reading, ...). It can only be accessed by drivers.

Non-agnostic access can be performed through various services, such as sys::fs.

Requests priority

Requests are treated by priority, which is made both of its arrival timestamp (first one, first out) but also of the process' priority: a process with an higher priority will see its I/O requests treated more quickly.

Multi-platform

NightOS programs can be ran either on Linux, Windows or MacOS systems using a wrapper called the NightOS Calls Wrapper (NCW). It can be downloaded from NightOS' official website.

When installed on a system, it allows to use NightOS applications natively, although permissions management will not be as smooth and many options will not be available (e.g. no desktop, no access to native NightOS applications - only native libraries, etc.).

Performances

Cycle and phases

Because processes' instructions cannot always be processed parallely, the system will let each application run a fixed amount of instructions, before going to the next one. This short period of time is called a phase and generally lasts a few milliseconds. The treatment of a single phase for all running processes is called a run cycle. When a process isn't in a phase, it is "paused" until the next one.

Processes' priority

Each process is started with a given priority, which is a number from 0 to 10, which can be modified during its execution. The more priority it has, the more instructions it will run in a single phase.

A priority of 0 prevents the program from running any instruction at all. In this case, only an external process will be able to increase the priority to let the program run again.

By default, a background process gets a priority of 2, a visible process gets a priority of 4, the process linked to the active window gets a priority of 6, and the process linked to a fullscreen window gets a priority of 7.

When a process asks to set its priority to 9 or 10, a confirmation overlay is shown.

Balancing performances

See the performances balancer.

Pre-compiling

On traditional operating systems, programs are provided as binary and so are only available for one specific platform (Windows, Linux, MacOS, ...) with a specific architecture (x86, ARM, ...).

In order to simplify distribution, and to bring more stability and security to native programs, NightOS programs are shipped as NightOS Pre-compiled Programs (NPP) using a specific language called CommonAssembly.

How it works

CommonAssembly is very similar to WebAssembly: compressed very low-level source-files that are built ahead of time on the machine that will run it. This enables several advantages:

  • Programs are multi-platforms, multi-architectures
  • Programs run faster thanks to being optimized for the specific machine they will run on
  • Programs are still very fast to compile
  • Source code is protected for closed-source applications as CommonAssembly is made of basic instructions
  • Better security thanks to programs being checked ahead-of-time

Compatibility

CommonAssembly can be ran on other operating systems than NightOS using a lightweight wrapper available on NightOS' official website.

NOTE: While CommonAssembly is multi-platform, NightOS applications take advantage of NightOS' features like more powerful signals that are not natively available on other systems. In order to run such applications, you must install the full wrapper which is available on the same website.

Usage

Open-source and closed-source NightOS applications usually bring pre-compiled versions of themselves in order to fasten a lot the installation process, and to get rid of the need of having the heavy build toolchain installed on the target computer.

This allows to install applications really fast, and to optimize them for the current machine.

Processes

WARNING: This document is far from being complete and may lack a good description of what processes are, and/or the features they offer.

NightOS processes are implemented a bit like Linux' ones, with additional features. There are several types of processes:

  • System processes, which are created by the system ;
  • Application processes, in which applications run ;
  • Worker processes, in which applications' workers run

The base and system processes are called low-level processes, while application and worker ones are called userland processes.

You can find the implementation details of processes in the kernel document.

User privileges

Each process is ran as a specific user, which determines the maximum allowed scope for controller requests, and with a list of initial privileges (the ones given to the application).

Child processes

A process can create child processes (it's called a fork). The child process will roughly be a 1:1 copy of the parent process, but with its own unique PID.

To avoid copying the whole memory, copy-on-write is used, which means that pages are identical and require no allocation until they are modified, in which case a new page is created and attached to the process in place of the original one.

Child processes automatically inherit their parent's permissions.

When a process exits, all its child processes are immediatly killed. It's up to the process to ensure its children are properly terminated before it.

Threading

A process can create threads, which are still a part of the process. Threads allow to run multiple part of a process concurrently, as the kernel may run several threads in different processor cores.

All threads share the same address space and memory, although they also have a reserved space called the thread-local storage.

To avoid copying the whole memory, copy-on-write is used, which means that pages are identical and require no allocation until they are modified, in which case a new page is created and attached to the process in place of the original one.

Also, the stack is local to each thread.

Threads work as a hierarchy ; when a thread creates another, it is called the new thread's parent, while the new thread is its child. When a thread terminates, all its children are instantly destroyed.

Main thread

When a process starts, its instruction run its main thread. Due to threads being hierarchised, exiting the main thread will result in all other threads being closed immediatly, which is why the main thread should always first terminate its children properly to ensure all data are synchronized for instance.

Thread-local storage

Threads have a reserved portion of memory in their address space called the thread-local storage (TLS).

It is split into slots, which can contain arbitrary amount of data.

Each slot has a unique identifier, which can be used across threads to access TLS data from any thread of the process.

Automatic permissions inheritance

When an application process gets a new permission, all other processes from the same application inherit it, unless this permission is granted only for this instance of the application.

Registry

The registry contains all system configuration.

Format and content

The registry's format and content can be found in the specifications document.

Debugging

In developer mode, the registry can be exported to the HFRR format and imported back.

Services

A service is a process that launches at startup and may receive IPC messages from any process.

System services

System services are native applications. They have system permissions and as such are allowed to perform any task. Their purpose is to allow processes to perform specific actions like permissions management or filesystem access without hunging up the kernel.

Each system service is a system application, exposing a service. Most also expose command-line tools.

You can get more details about how services work in the specifications document.

You can find the list of system services in their specifications directory.

Application services

Application services, also called userland services, are provided by applications themselves.

Shell

The shell, called Hydre, is the part that interprets scripts.

Commands

Hydre works using commands, which can take arguments. When running an application's exposed command, the said application is run in a new process with a specific execution context.

The special thing about running an application from one of its exposed commands is that its CMDIN, CMDUSR, CMDMSG, CMDERR, CMDOUT and CMDRAW pipes are exposed to the caller.

You can find more informations about how a process can interact with a running command in the IPC documentation.

Pipes

When a command is run from Pluton, the command process' pipes are handled as follows:

  • All user inputs are sent to the CMDUSR pipe
  • All messages sent through CMDMSG and CMDERR pipes are printed as they are received by the shell
  • The command's return value (from CMDOUT) is printed after the command completes
  • Data written to CMDRAW is not shown, as they don't have to be UTF-8 encoded, or even to be a string. They can be redirected through pipes, though

Specifications

Hydre's specifications can be found in the related document.

User Experience

This folder contains technical documents about user experience.

They only act as drafts and may change at anytime. They mainly contain global ideas or technical overview for some parts. Some documents may also contain specifications.

Documents

Desktop Environment

The Desktop Environment (DE) is an application that acts as the main graphical interface of NightOS.

It is launched with the system, does never stop until the system itself stops, and handles most of the user interface.

Desktop Environment Applications

A desktop environment application (DEA) can only be launched by the system, and is only launched once during the whole system's lifetime, except for their commands which can be launched by anyone. It is also the very first application to be run.

There are two default DEA: Nova, and BareEnv.

The former acts as the default, reference and recommanded DE of NightOS, while the latter is for servers and other computers without a graphical output. It only supports text-based workflows, which limits it to command-line usage.

An application can register itself as a DEA by exposing a SYS_DENV service in its manifest.

Permissions

A DEA does not require specific permissions in order to work ; it only requires to expose a specific integration service in its manifest.

Display layers

The graphical interface is split into layers, which represent the display priority when two elements overlap. For instance, when two elements A and B overlap visually (= part of them is displayed at the same coordinate than the other), the element located in the highest layer will be displayed on top of the other element.

Layers are indexed, starting from 0 with the lowest priority.

Windows

Windows are the base display element for applications. They can take any form depending on the DE's display method, but are usually made of the application's drawing zone and a titlebar.

Some desktop environments may not accept windows (for instance, BareEnv which is a text-only desktop environment). In such case, most applications will fail to run as they won't be able to build windows, but this is still allowed.

Security and operation

Unlike other operating systems, DEA can only access their own data and cannot manipulate the data inside other applications, even graphically.

Desktop environments rely on the sys::ui system service to control the data they display.

This service provides them several methods and events, mainly to:

  • Display graphical elements on layer 0
  • Display the content of a window at provided coordinates
  • Notify a window owner that its window should be closed
  • Notify a window that a part of its state should change
  • Reject a window's state change
  • Be notified when a window's state changes

At no moment can the desktop environment access the graphical content of a window, nor affect it, as windows are displayed starting from layer 1.

On the other hand, the desktop environment can force a window owner (the application process that created it) to resize, minimize, restore, put to fullscreen, etc. and can also hide some elements from the system as it is not forced to display a window's content.

Window states

A window's state is made of its characteristics:

The custom attributes are handled by the DEA itself, and can contain any piece of information relevant to displaying the window, such as the virtual desktop number if the DE supports it.

Window display state

Windows have a display state which indicate how they are displayed on the screen. These states can be:

  • Restored: the default
  • Maximized: the window takes all the available space, minus the space used by its optional titlebar and the elements displayed by the desktop environment
  • Minimized: the window is hidden but still available
  • Fullscreenized: the windows takes all the screen, hiding all other windows and elements from the desktop environment, including its own titlebar

Active/inactive windows

All windows are considered inactive unless they are focused, which on most desktop environments put them on the top layer. There are no specific difference between active and inactive windows, but this attribute is indicated to the owner application as this may impact the desired behaviour of the application.

Interactive windows

By default, windows are interactive, meaning user inputs can be sent to them. They may be set non-interactive to stop receiving such events and preventing users from interacting with them.

Fixed-size windows

Fixed-size windows are windows whose size is fixed and cannot be resized by the end user.

Common elements

Desktop environment are traditionally made of several elements:

  • The taskbar lists all opened windows
  • The notification center centralizes all pending notifications
  • The titlebar is an element put on each window to see its title, icon and control its state

These elements are common but are NOT required to be present in a desktop environment.

Interface windows

Interface windows are windows that are created by another application to be used in a specific part of the desktop environment. It results in the integration of an application with the current DEA.

Interface windows are created the same way as usual windows, but do not usually have a titlebar and can often not be moved. They are mostly used to create widgets and display context menus on items.

Typically, an interface windows created by an application is hidden by default, and the DEA shows it only after getting its identifier returned by the service it is communicating with.

Popups

Popups are special windows which are not drawable directly by the application. They usually contain a single icon, as well as a message and at least one clickable button. They are used to transmit informations to the end user, such as an error or a warning message.

Notifications

Notifications are the way for both the applications and the system itself to indicate when an event happens which the user should be notified of, or to indicate a persistent state.

Structure

A notification is made of:

  • A title
  • A description

As well as optional fields:

  • A description
  • An icon (by default the icon of the application)
  • A notification sound (by default selected by the system)
  • A shaking effect
  • A background color (by default chosen by the desktop environment)
  • A set of buttons
  • A cover image
  • A persistent state
  • A type-dependent set of data
  • A timeout
  • A handler called when the notification is clicked
  • A context menu when the notification is right-clicked

Display rules

The way notifications are displayed depends on the desktop environment, therefore the requested shaking effect, background color, and so on may not be respected by the DE, although it's highly recommanded for a DE to follow these attributes.

Also, there is no rule indicating how elements are visually arranged inside a notification. Some DE may for instance put the icon on the left, while some others on the right. Additional elements may also be displayed, such as a cross to close the notification or an arrow to hide it to the notifications center.

Cover image

A notification can request to display a large image. Some desktop environments may decide to hide the application's icon in such case.

Buttons

Notifications can have action buttons, which are then linked to the owner application's event system. Buttons are usually limited in size and colors.

Persistent notifications

Persistent notifications don't disappear after a timeout. They can for instance be used to indicate a continuous state like the progress of a task.

Notification types

There are several notification types, which their own required sets of data.

Free notifications

This is the default type of notifications, which has no particular constraint.

Multimedia notifications

Multimedia notifications are used to indicate a multimedia playback. It must be composed of a cover image and six buttons (previous, rewind, play/pause, stop, fast-forward, next) whose appearance is decided by the desktop environment.

They are always persistent, and disappear when the playback stops completely.

Sound

This document describes how audio is implemented in NightOS.

Audio inputs

Devices recognized as audio inputs can act as global audio input devices. Multiple devices can be used at the same time, but only one default input can be selected.

By default, each time a new audio input device is connected, it is selected as the new default device. When disconnected, the previous default device is selected again.

It is also possible to set a device as always being the default one, even if other devices are connected afterwards.

Playback devices

Devices recognized as audio outputs can act as global audio output devices. Multiple devices can be used at the same time, but only one default output can be selected.

By default, each time a new audio input device is connected, it is selected as the new default device. When disconnected, the previous default device is selected again.

It is also possible to set a device as always being the default one, even if other devices are connected afterwards.

Multimedia playback

When a sound is played, the process must specify the type of the audio resource, which is either "standalone sound" or "multimedia sound". In the second case, a multimedia notification is emitted, which can then be controlled by the application but cannot be closed while the media is playing.

Specifications

This folder contains specification documents, whose purpose is to describe fully a specific concept or feature. Think of these as reference documents - they are not meant to be easily understood without a solid knowledge of how NightOS works. For more informations about low-level concepts these documents refer to, you can check the technical documents first.

Integration services

Driver services

System services

Applications and libraries

This document describes the structure, features and behaviours common to both applications and libraries.

Name and slug

Each application has a name as well as a slug. The name can any valid UTF-8 string, while the slug must respect several rules:

  • Only lowercase letters, underscores and digits
  • Must not start by a digit
  • Must not be the name of a native shell command
  • Must not be the name of a native shell function
  • Must not be the name of a shell type

By default, the slug is auto-generated from the name, but it can also be customized.

Application Identifier

From the slug is generated the Application's IDentifier (AID), which is prefixed by the developer's identifier (DID) specified in the application's manifest (it must match the publisher's identifier on the store), and followed by two double points. The DID is submitted to the same rules as the application's slug.

For instance, an application with a slug of utils made by a developer whose DID is superdev will get an AID of utils::superdev.

The AID is unique across the store as well as in a single NightOS installation. As malicious application may provide the DID and the slug of a legit application (which is called AID spoofing), sideloading is verified by default.

As AID are text-based and can be quite long (up to 512 bytes), programs can instead use the Application's Numeric IDentifier (ANID), which is a 32-bit unique number randomly generated by the system to refer to this particular application. On two different systems, the ANID of a given application will likely be very different, and so cannot be guessed. It is provided by the system during the application's launch through its execution context.

System applications are registered under the reserved sys DID.

Features

Applications as well as libraries can also declare features, which are opt-in parts of the program. They are described in the manifest and can be enabled or disabled when installing the program.

Each feature allows to embed additional assets and declare additional supported languages (e.g. language packs), and a specific build option is provided to the build tool to indicate at build time weither a feature is selected or not.

This allows to save space and build time when a specific feature isn't required, but should still be embedded in the program itself instead of making an additional application or library.

When the manifest declares one or more feature(s), it must also specify a set of default features to enable, which are selected by default.

Extensions

Applications and libraries can have extensions, which are additional content that may or may not be downloaded and installed alongside the original application/library.

If we take the example of a file browser, an extension could be an image previewer with support for many image formats, which would be unreasonable to put in the original application given most people won't use it. In a media player, it could be a set of additional codecs.

Extensions can contain executable code, but they must be run from within the original application/library only. They do not appear in the list of installed applications visible to the end user, and are instead located in a sub-menu for the application/library they belong to.

For apps and libs downloaded from the Store, extensions can be downloaded from within the original content by calling a dedicated API.

Translations

Applications' and libraries' name, description, content, etc. can be translated using translation files. The content itself can be handled in any other manner, e.g. using a translation framework, but the metadata must use a translation file so the system does not have to run code in order to perform the translation.

The manifest

A manifest is a file describing the application or library. It is mandatory to build and distribute it. Its content differ for applications and libraries, but there are common fields (REQ: required, OPT: optional):

# [REQ] Informations about the application
infos:
  # [REQ] Application's name, as shown when installing it
  name: Cloud Notepad App
  # [REQ] Application's slug, as stored on the disk in the /apps directory
  slug: cloud-notepad-app
  # [REQ] Application's description, as shown when installing it
  description: A notepad application that allows syncing your files in the cloud
  # [REQ] Application's version, following semantic versioning
  version: 1.0.0
  # [REQ] Application's authors
  authors:
    - name: Clément Nerma
      email: clement.nerma@gmail.com
  # [REQ] Application's icons (in the archive)
  # [REQ] "%{}": either 16, 32, 64, 128 or 256 pixels (icons must be square)
  icon: assets/icons/app/%{}.png
  # [REQ] Application's license (must be in a list of available licenses)
  license: Apache-2.0

# [REQ] Application package's content
content:
  # [REQ] Program
  program:
    # Packages can either contain source code only, pre-compiled programs only, or both
    # <for source packages> [REQ]
    source:
      # [REQ] Build tool (must in the list of the toolchain's supported build tools)
      toolchain: rust
      # [REQ] Required build tool-related options
      build: {}
      # [OPT] Build tool-related options
      options:
        optimize: O3

    # <for precomp packages> [REQ]
    precomp: main.npp

  # [OPT] Assets
  assets:
    # Set of path to assets
    - ./assets/

# [REQ] Application's dependencies
dependencies:
  # [REQ] Required libaries
  libraries:
    sysver: ^1.0.0 # Any stable version

  # [OPT] Required fonts
  fonts:
    fonts:open-sans: true # Any version

# [OPT] Features
features:
  # Describe each feature, by name
  testing:
    # [OPT] Additional assets
    assets: []

    # [OPT] Feature's dependencies
    dependencies: {}

    # [OPT] Required extensions
    extensions: {}

    # [OPT] Additional languages (e.g. language pack)
    languages: []

# [OPT] Extensions
extensions:
  # Describe each extension, by name
  test:
    # [REQ] Display name
    name: Test
    
    # [REQ] Description
    description: A test extension

Applications

This document describes application-specific structure and behaviour. For common reference, see the applications and libraries document.

Application package

Application packages are files that have either the .nap (NightOS Application Package) or .nva (NightOS Volatile Application).

Content

NAP and NVA files are ZStandard archives which only requirement is to contain, at the archive's root, a manifest.toml file describing the archive itself, a hash.md5 ensuring the archive has not been corrupted.

They are built around a manifest which can be found below.

Pre-compiled applications

By default, and if possible, the system will always try to install pre-compiled programs from applications' package. If the pre-compiled programs are not available, it will be built from source code - which takes a lot more time.

Libraries embedding

Although it's a better practice to split applications and libraries into different packages, sometimes it's more easy to embed both in the same package, especially in two cases:

  • When the application is just a thin layer ahead of the library (e.g. CLI tool)
  • When the library's API changes rapidly and the application relies on it

For such scenarios, it's possible for an application package to embed one or more libraries, and publish them all at once.

The application's and libraries' version may differ if required.

If an another application or a library specifies one of the embedded libraries as a dependency, only the said library will be installed, not the application.

Values encoding

The application's startup arguments and output value use the following encoding:

  • The return value's length (8 bytes) ;
  • The value's shell type code (see the table below) ;
  • The encoded value (see the table below)
Type codeTypeDescriptionRepresentation
0x00voidVoidnothing
0x01boolBoolean1 byte, 0x00 = falsy, 0x01 = truthy
0x02int64-bit signed integer numberTwo's complement
0x03float64-bit signed floating-point numberIEEE 754
0x04charUTF-8 grapheme clusterCharacter's length (8 bytes), followed by the UTF-8 grapheme cluster
0x05stringUTF-8 stringString's length (8 bytes), followed by the UTF-8 encoded string
0x06listTyped linear listType code of the list's number of items (1 byte), length in bytes (64 bits), encoded items
0x07pathFilesystem pathRepresented as an UTF-8 string
0x08commandShell commandRepresented as an UTF-8 string
0x09streamPipe RCRC identifier (8 bytes)

The type code is present to avoid misinterpreting the value in case the command returned a value of the wrong type.

Returning and failing

The value must be returned using the CMDOUT pipe. The data sent through this pipe must follow the above encoding.

A command may also fail. To indicate so, the process must send the 0xFF value through the pipe, and the shell will consider the command as failed (but not invalid, so the process won't be abruptly killed).

Volatile applications

Volatile applications cannot expose commands globally as they are technically not installed.
They can though be used in shell scripts through volatile imports.

Application manifest

Here is the specifications of the application manifest, in additional to the common manifest:

# [OPT] Event triggers
events:
  # [OPT] Should the application start just after being installed?
  postinstall: false
  # [OPT] Should the application start just before being uninstalled?
  preuninstall: false
  # [OPT] Should the application start just before being updated?
  preupdate: false
  # [OPT] Should the application start just after being updated?
  postupdate: false

# [OPT] Exposed commands (see the related document for additional informations)
commands: {}

# [OPT] Does the application expose services?
services:
  # [REQ] Does the application expose a main service?
  main: false
  # [REQ] List of scoped services
  scoped: []
  # [REQ] List of integration services
  integration:
    # [OPT] Desktop environment service
    desktop_env: false
    # [OPT] File manager service
    file_manager: false
    # [OPT] Filesystem opener service
    fs_items_opener: false
    # [OPT] Filesystem interface
    fs_interface: false
  # [REQ] List of driver services
  driver:
    # [OPT] Storage driver service
    storage: false

# [REQ] System features
sysfeatures:
  # [REQ] Does the application support crash saves?
  crash_saves_support: true

# [OPT] Additional informations
additional:
  # [OPT] Available languages (in a list of existing languages)
  languages: ["en-US"]

Commands

Applications can expose commands through their manifest.

The format is the same as the shell's command typing, although adapted to YAML:

  • The pos or dash indicator is turned into a (required) syntax option
  • The optional indicator becomes a boolean that must be set to true
  • The void type is forbidden

Example

# [...beginning of the manifest...]
commands:
  say_hello:
    help: "A program that repeats the name of a list of person"
    author: "Me <my@email>" # Optional
    license: "MIT" # Optional
    return: void
    args:
      # Declare a positional argument named 'names' with a help text
      names:
        syntax: pos
        type: list[string]
        help: "List of names to display"

      # Declare a dash argument named 'repeat'
      repeat:
        syntax: dash
        type: int
        short: r
        long: repeat
        optional: true

      # Return the time this command took to complete
      get-duration:
        syntax: flag
        short: d
        long: duration
        $if:
          cond: present()
          return: void
# [...end of the manifest...]

Execution Context

An application's execution context is a piece of data that is provided when the application starts.
It indicates why the application was started and so what it is supposed to do.

Startup Reason

The most important information is the startup reason, which indicates why the application was started.

It is one-byte long, and is made of the following bits (starting from the strongest):

  • Bits 0-3:
    • 0x1: the application was started as part of its post-installation process
    • 0x2: the application was started as part of its pre-update process
    • 0x3: the application was started as part of its post-update process
    • 0x4: the application was started as part of its pre-uninstallation process
    • 0x4: the application was started by the system as an application service
    • 0x5: the application was started by the desktop environment
    • 0x6: the application was started by itself (from another process of the same application)
    • 0x7: the application was started by another application
    • 0x8: the application was started using one its exposed shell commands
    • 0x9: the application was started as a desktop environment
  • Bit 4: set if the application was started automatically after a crash/improver shutdown and should to the sys::crashsave service to get a crashsave
  • Bit 5: set if the application's raw output (CMDRAW) will be read (e.g. through the use of a shell operator)

The startup reason is especially important as it determines what the application should do (e.g. uninstall itself, run as a command...) but also if it should output data through its CMDRAW in case it was called by a command.

Context header

The context header is stored as a single block of data, consisting of:

  • The startup reason (1 byte)
  • Ambiant informations (1 byte)
    • Bit 0: set if the application is starting for the very first time since it was installed
    • Bit 1: set if the application is starting for the very first time for this specific user
    • Bit 2: set if the application is starting for the very first time as this specific service
    • Bit 3: set if the application is starting for the first time after an update
    • Bit 4: set if other instances of this application are running
  • Special assignment information (1 byte)
    • Bit 0: set if the application is starting for the first time after being assigned as the new desktop environment
    • Bit 1: set if the application is starting for the first time after being assigned as the new default file manager
  • Service type (1 byte):
    • 0x00: this process is not run as a service
    • 0x01: this process is run as the application's main service
    • 0x02: this process is run as an application's scoped service
    • 0x10: this process is run as the application's desktop environment service
    • 0x11: this process is run as the application's file manager service
    • 0x12: this process is run as the application's file opener service
  • Scoped service's name (8 bytes) - filled with zeroes if the process is not run as a scoped service
  • The application's ANID (4 bytes)

If the command was started as a command, it also contains the following informations:

  • The ANID of the caller application (4 bytes)
  • The number of arguments the process was started with (1 byte)
  • The cumulated size of all arguments, in bytes - up to 63.5 KB (2 bytes)
  • RC identifier for the CMDIN pipe (8 bytes)
  • RC identifier for the CMDUSR pipe (8 bytes)
  • SC identifier for the CMDMSG pipe (8 bytes)
  • SC identifier for the CMDERR pipe (8 bytes)
  • SC identifier for the CMDRAW pipe (8 bytes)
  • SC identifier for the CMDOUT pipe (8 bytes)
  • Future-proof shift space (196 bytes)

RC and SC identifiers come from the relevant flows.

Arguments structure

The context header is followed by the list of each command-line argument, taking up to 64 KB.

Arguments are a simple concatenation of encoded values.

Libraries

Libraries are NightOS' way to share code between multiple applications.

Manifest

As libaries are only meant to share code, there manifest is a lot more simple than the applications' one as it is only made of the common fields.

The system library

The system provides a single system library with the sys::lib AID, which is built in the system and not removable. Apart from these points, it acts like a standard library. Its version is identical to the system itself, allowing applications and libraries to specify a minimum system version in their dependencies field.

System library modules

The library contains several modules (called SLM for System Library Module), which act like language namespaces to expose specific features:

  • fs : Filesystem management
  • net : Network communications
  • ipm : Inter-process management (create processes, workers, IPC, shared memory, ...)
  • gui : Graphical user interface library (relies on desktop)
  • apps : Applications management (installation, removal, ...)
  • perm : Permissions controller
  • hydre : Shell interface (run commands, ...)
  • input : Input interface (keyboard, mouse, microphone, ...)
  • sound : Sound interface
  • system : System interface (low-level changes, updates, ...)
  • sandbox : Sandboxes management (run applications in sandboxes, ...)
  • desktop : Desktop management (desktop, windows, notifications, ...)
  • hardware : Hardware management (drivers and devices)
  • i18n: Translations

System applications' libraries

Each system application also exposes a library with the sysl::<app name lowercased> AID which applications can rely on, as these applications are not removable. These are abbreviated SAL for System Application Library.

Filesystem

This document presents how files are stored in NightOS.

Partitions

NightOS uses the Btrfs filesystem for the main storage due to its robustness, performance, and features (e.g. snapshots).

Three partitions are used to store the data:

  • One FAT32 partition for BOOT1 ;
  • One FAT32 partition for BOOT2 (slot 1) ;
  • One FAT32 partition for BOOT2 (slot 2) ;
  • One Btrfs partition for users' data (/etc except /etc/sys, /apps and /home)

Identifiers and limitations

Filesystem unique identifier

Each existing filesystem gets a unique identifier called the Filesystem Iddentifier (FSID).

It is unique across all filesystems, and consistent across reboots.

Element unique identifier

Each filesystem item has an 8-byte identifier, called the Filesystem Element Identifier (FEID).

It is only unique in the filesystem the item is stored in.

Note that some filesystems may not support this feature. For those, the FEID will be derived from the item's split path, which means it will still be unique in the filesystem but will not be resilient to file moving or another file replacing the previous one.

Temporary FEID

A temporary FEID can be created to grant access to a resource to an application without giving actual access to the original item itself. A temporary FEID only lives in memory, and refers an existing filesystem item. If the original item is deleted, the temporary FEID is deleted too.

Paths' size limit

Paths' length is encoded on 2 bytes, allowing up to 65 534 characters plus the NULL character.

This limit exists to avoid too costly string copies on path manipulation, while being long enough for the very large majority of use cases.

Filenames

For naming conventions, see the related specification.

Permissions

Permissions on individual items is achieved through storage permissions maps.

Symbolic links, abbreviated symlinks, are files that point to another location.

Concept

A symlink points to a specific item: file, folder, device, anything. It's just not a shortcut, though, as the symlink will still work if its target is moved.

When a symlink is accessed, the system will transparently access its target item instead.

When a symlink is removed, it does not affect the original target. Also, any number of symlinks can target the same item, and symlinks can target other symlinks to. When accessing a symlink, if its target item is a symlink itself, the latter's target will be accessed instead, and so on, until we do not encounter a symlink anymore.

This can be explicitly disabled when interacting with the filesystem, or limited to a specific number of children.

Also, symbolic links may point to a location on another storage.

Given the following situation:

  1. We create a symlink A which points to a random file
  2. We create a symlink B which points to A
  3. We update the target of A to be B

When we will try to access A, the system will access B, then A, then B, and so on. This is called a cyclic symlink chain. In such case, the chain is reduced to the minimum (for instance, if we had C pointing to A, the minimum chain would not be C A B but just A B), and marked as erroneous. The process that tried to access the symlink will receive a specific error code to indicate a cyclic symlink chain was encountered.

Flows

Flows are a simple and efficient way for shell scripts to allow manipulating asynchronous streams of data.

Concept

A flow is a file without extension, located in the /fl directory, that can either send data to reader processes (read-only) or receive data from writer processes (write-only).

To understand the concept better, here is the list of native flow files that are always available:

Flow fileTypeDescription
/fl/zeroRead-onlyOutputs zeroes all the time ; useful to zero a file or device or to benchmark a storage
/fl/urandRead-onlyOutputs cryptographically-secure random numbers. Useful to randomly fill a storage or memory area
/fl/crandRead-onlyOutputs non-cryptographically-secure random numbers, thus faster that /fl/rand
/fl/nullWrite-onlyReceives data but does nothing with them

Processes are based on pipes.

Creating a flow

When a process wants to create a flow, it follows the following procedure:

  1. The process asks the sys::fs service to create a flow
  2. The service creates the related flow file in /fl
  3. When a process reads from the (readable) flow file, all data is continuely retrieved from the creator's SC (until the flow is closed)
  4. When a process writes to the (writable) flow file, all data is continuely written to the creator's RC (the flow is not closed after that though)
  5. When the creator closes its SC/RC, the IPC channels duo is closed and the flow file is removed

Connecting to a flow

When a process wants to read from or write to a file, it first asks the sys::fs service to connect to this file. If accepted, it receives a SC or RC to interact with the flow.

Structure

Below is the file tree indicating how elements are organized in NightOS.

NOTE: <F> indicates the item is a file.

/
├── app                            Interactables available to all users
│   └── <appname>                  An application's folder (NOTE: one sub-folder per version for libraries)
│       ├── content                Application's program (executables, static resources, ...)
│       ├── crashsaves             Application's crash saves
│       ├── data                   Application's data (e.g. database)
│       ├── packages               Application's packages (original package + update packages)
│       └── sandboxes              Application's sandboxes
├── dev                            Connected devices
│   ├── cam                        Cameras
│   ├── bst                        Basic storage devices (SD cards, USB keys, ...)
│   ├── etc                        Uncategorized devices
│   ├── mic                        Microphones
│   ├── net                        Network adapters (Ethernet adapter, WiFi card, ...)
│   ├── snd                        Sound-related output devices (Sound card, DAC, ...)
│   ├── sst                        Sensitive storage devices (Hard drives, SSDs, ...)
│   └── wrl                        Other supported wireless devices (Bluetooth adapter, ...)
├── etc                            Mutable data folder
│   ├── env   <F>                  Environment variables
│   ├── hosts <F>                  Hosts overriding (e.g. 'localhost')
│   ├── logs                       Log files
│   |   └── upe <F>                History of UPE requests (1)
│   ├── public                     Public data, readable and writable by everyone
│   └── sys (4)                    System's mutable data - available to system only
│       ├── registry    <F>        System's registry
│       ├── awake       <F>        System's shutdown indicator to detect if there was an error during last shutdown
│       ├── integrity              Integrity data used during the boot process (2)
|       |   ├── signkey <F>        Key used for signing system files
|       |   └── signs   <F>        System file signatures
│       ├── enckey      <F>        Global storage's encryption key (3)
│       └── users       <F>        User profiles and groups
├── fl                             Flow files
├── home                           Users' data
│   └── <user>                     A specific user's data
│       ├── apps                   User's applications (same structure as for `/apps`)
│       ├── appdata                User's applications persistent data (not removed when the application is uninstalled)
│       ├── desktop                User's files appearing on the desktop
│       ├── documents              User's documents
│       ├── downloads              User's downloads
│       ├── music                  User's music files
│       ├── pictures               User's pictures
│       ├── videos                 User's videos
│       └── trash                  User's trash
├── mnt                            Mounted storages
│   └── root                       Soft link to `/`
├── sys (4)                        System - immutable outside of installation, repair processes and updates
│   ├── apps                       System applications
│   ├── boot                   (5) System's boot program (BOOT2)
│   ├── langs                      Translation files
│   ├── old                        Old versions of the system, used during the repair process (compressed archives)
│   ├── backup                     Copy of the last system version (compressed archive)
│   ├── kernel                     Custom micro-kernel
│   └── valid   <F>                A file that just contains "ValidMasterKey" to test if the provided master key is valid at startup
├── tmp                            Temporary folder (cleaned during shutdown)
│   └── <user>                     Temporary folder for a specific user

Links:

Notes

Globally installed applications (located in /apps) can store user-specific data in /home/[user]/appdata/[appname], which will only be made of the data, crashsaves and sandboxes folder.

Storage permissions

This document presents how permissions are handled on filesystems.

How permissions work

Each item (identified by its FEID) uses a set of permissions determined by multiple rules.

  • Items in the /dev, /etc and /sys are not available to any user.
  • Items in a /app/<xxx> directory are only available to the xxx application. Users do not have direct access to these files.
  • Items in a /home/<xxx> directory are only available to the xxx user.
  • Items in a /mnt/<xxx> directory can be available to various users depending on the mount options.

Note that an exception is made for the system user which gets access to every single file.

These are the default permissions, but it's possible to restrict or extend permissions based on a Storage Permissions Map (or SPM).

Structure

The SPM is a non-contiguous data structure which can be located anywhere in the filesystem.

The SPM's header is a contiguous block located at its beginning.

It is made of the following:

  • Number of pages (8 bytes)
  • Address of the first page (8 bytes) - 0 if none
  • Address of the last page (8 bytes) - 0 if none

Pages' header

Each page is made of a contiguous header (data block) made of the following:

  • Maximum number of entries (capacity) (8 bytes)
  • Number of free entries (8 bytes)
  • Address of the previous page (8 bytes) - 0 if none
  • Address of the next page (8 bytes) - 0 if none
  • For each entry:
    • Address of the entry (8 bytes)

Entries

Each page contains a set of entries, which may be split on the disk. An entry is made of the following:

  • Address of the page referencing this entry (8 bytes)
  • FEID of the item this entry is for (8 bytes)
  • Number of entities described in the entry (8 bytes)
  • For each entity:
    • Entity type (1 byte):
    • Entity ID (8 bytes)
    • Permission levels
      • Bits 00-01: List directory's content
      • Bits 02-03: Read files
      • Bits 04-05: Create new items
      • Bits 06-07: Edit items
      • Bits 08-09: Delete items
      • Bits 10-11: Read metadata
      • Bits 12-13: Write metadata
      • Bits 14-15: Change owner
      • Bits 16-17: Edit SPM permissions

An item's owner will have all of these permissions set by default.

An entry's content must be contiguous ; if it grows too large to fit inside its allocated space when updated, it must be moved somewhere else.

Permission levels

A permission level is a 2-bit value which can either be:

  • 0b00: inherit from the parent entity / from the system's settings
  • 0b01: refuse for this specific item
  • 0b10: allow for this item, recursively
  • 0b11: allow for this item but only for content owned by the referred entity, recursively

File table pointer

Unless a filesystem does not support it, nor natively or through the use of extended attributes, each file and directory's entry must contain the address of its entry in the SPM, with 0 indicating no entry exists for it.

When the item is created, no entry is created for it by default. It's only when permissions are set for the item that an entry is made and referenced in the file table.

When the entry is cleared, it is destroyed and the 0 value is assigned back to the file table entry's SPM pointer.

Boot process

The system's boot process is divided in two parts: the unsecure bootloader (BOOT1) and the system bootloader (BOOT2).

Note that BOOT1 and BOOT2 have their own dedicated partitions, the latter getting one for slot 1 and another for slot 2 (see below for more details).

Stage 1: unsecure bootloader (BOOT1)

This is the component loading at the very beginning, called directly by the UEFI. It is unencrypted (hence the name "unsecure").

It starts by initializing the minimum required computer's components.

If the "Escape" key is detected as being pressed during a short time frame, it shows a troubleshooting menu, notably switching the boot slot.

If other bootloaders are found on the storage, it displays a boot menu asking which system to boot. If another system is selected, it launches it.

Otherwise, it tries to get the master key depending on the configured encryption method:

  • If storage encryption is disabled or if only USGE is enabled, nothing to do ;
  • If full-storage encryption is enabled:
    • For passwordless authentication, ensure Secure Boot has verified BOOT1
      • If so, retrieve the master key from the TPM and use it to decrypt the storage
      • Otherwise, ask for the recovery key to decrypt the storage (only required once)
    • Otherwise ask for master password and use it to decrypt the master key

It checks the boot slot to use (1 or 2), and read BOOT2 from it.

If BOOT2 is encrypted (full-storage encryption only), the master key is used to decrypt it. Its signature is then checked to ensure it has not been modified. If the signature don't match, the computer will by default refuse to boot to avoid corruption and/or booting malicious programs. By inputting a specific phrase displayed on the screen, the user can force the boot process, at the expense of security.

If signatures match, BOOT2 is launched directly.

Stage 2: system loader (BOOT2)

This component is responsible to user account selection and storage decryption. It is stored in a dedicated partition as the /sys/boot file, a header-less raw executable program.

It starts by checking signature of all system files (everything located in /sys and /etc/sys).

If signatures are not valid, an error message is shown and the booting process is halt. By inputting a specific phrase displayed on the screen, the user can force the boot process, at the expense of security.

BOOT2 also provides more advanced troubleshooting thanks to the whole storage being decryptable.

It then initializes all required drivers, initialize a graphical session, and asks to select a user account. At this point, it also provides more troubleshooting options.

If the provided username and password are valid, it then does the following:

  • If USGE is enabled, the master key and user key are decrypted
  • If per-user encryption is enabled, the user key is decrypted

The user session is then opened by calling the relevant system component.

Note that the BOOT2 partition is read-only except to the system user itself.

It is only updated through the system update process and when rollbacked.

Update processes

This document describes how updates are performed.

System updates

The system selects an update source, which is one of:

  • The official servers (default)
  • An alternative server (configured by the user)
  • A local update repository package

The source is authenticated, then the latest updates (not yet present on the system) are pulled from it as update package files (.nsu).

If multiple updates must be installed, they are installed one after the other. This will probably change in the future when an incremental update system is designed.

System update packages

Each package file is a simple TAR + GZ archive with the following structure:

  • /metadata: contains metadata on the updates
  • /update: "up-down" update program
  • /assets: a directory containing various files (can be anything)

Update steps

  1. The update package is extracted in a temporary path
  2. Update metadata are checked
  3. The system checks the currently used slot
  4. The active slot (= all files currently in /sys) is copied to the other slot
  5. An incremental system backup is created, containing all data from active slot (/sys) as well as the /etc/sys directory is created
  6. A non-incremental (full) system backup is created
  7. A Btrfs snapshot of the main partition is created
  8. The "up-down" update program is run
  9. In case of success, the active slot number is inverted
  10. System reboots
  11. After a week or two more reboots (whichever happens last), the main partition snapshot as well as the non-incremental system backup are deleted

This allows the following:

  • If the new BOOT2 does not work, we can revert it instantly until the next update ;
  • If some modifications created problems, we can roll them back using the main partition snapshot ;
  • If some problems appear later on, the boot snapshot allows to entirely rollback both /sys and /etc/sys

Incremental system snapshots are kept indefinitely in order to allow an easy rollback. A timestamp, system version and update number are associated, in order to quickly determine the state it was in.

Up-down update program

An up-down update program is a program that can either install or uninstall an update.

It is provided an history of all system updates as well as a snapshot of the BOOT2 partition at each point in time.

Application updates

An application update can happen in either scenario:

  • The system checked for updates in the Store (official or custom) and found one ;
  • The application asked the system to check for update and it found one ;
  • A custom application update package (.nua) was provided

The update package file is then checked. It is a simple TAR + GZ archive containing the following:

  • metadata: contains update metadata
  • files: a directory containing all files to replace in the application

Note that an update can only apply to a specific version of the application, specified in the metadata. Otherwise, previous updates must be installed first.

If valid, the system asks the user to close the application if it is opened.

After the application is fully closed (including services), its files are replaced with the ones provided in the archive's files directory.

The application is then started with its new state.

Permissions

Permissions are used to control what applications can do or not. They have no effect by themselves, but allow different system services to ensure an application has enough permissions to perform a specific action.

WARNING: This document is far from being complete, and as such may be edited in any way at any moment.

Levels of permissions

Permissions are split across five different levels:

  • Level 1: non-sensitive permission like creating a window ; these are granted automatically by defualt
  • Level 2: accessing and modifying non-critical parts of the state of the system, like controlling the global volume
  • Level 3: performing sensitive actions, like reading files or accessing the network
  • Level 4: performing highly-sensitive actions, like accessing the microphone or the webcam

There are also domain-controlled permissions as well as application proxies which influence how permissions are granted.

When using visual applications, requesting interactive, sensitive and privacy permissions will show a popup asking the user if they want to grant the permission:

  • Only one time
  • For the active application (until it stops)
  • For the current session (until the user logs out or the system is shutdown)
  • Forever

By default, it is set to "forever". But for the privacy level, this choice is set by default to "forever while the application is active", which prevents the application from accessing them from background tasks.

NOTE: If the application is uninstalled and re-installed later, all the granted permissions will have been dropped, so confirmation will be required again.

NOTE: Permissions granted to volatile applications are saved using the application's hash, so while the exact same application is opened volatilely, permissions are kept in memory.

NOTE: Permissions are specific to the current user (the AUC is not taken in account here), so in the case of a global application, each user will have to approve the permission manually (unless the administrator set it otherwise).

User privileges

Some permissions require the user to be an administrator. These permissions are marked in this document with an {A}.

List of permissions

Below is the exhaustive list of all available permissions, with their level between parenthesis (e.g. (5) for level 5, etc.).

Many permissions require to have an associated scope, indicating the resources they can access. Each scope is a path or filesystem with a set of permissions that apply on this specific path/filesystem.

Devices

Scope: list of device patterns the permission applies on

  • (4) devices.enum: enumerate devices
  • (4) devices.subscribe: subscribe to devices
  • (4) devices.register_driver: register a device driver
  • (4) devices.ask_driver: ask a device's driver to perform a normalized method

For praticity purpose, the list of device patterns is converted to a human-readable format, with more or less informations depending on the user's complexity level.

Filesystems

Scope: filesystems the permission applies on

  • (2) fs.filesystems.mounted: check if a given filesystem is mounted (all other fs.filesystems.* permissions imply this one)
  • (3) fs.filesystems.metadata: get metadata on a given filesystem
  • (3) fs.filesystems.list: enumerate mounted filesystems
  • (3) fs.filesystems.mount: mount an existing filesystem
  • (3) fs.filesystems.unmount: unmount filesystems mounted by other applications
  • (3) fs.filesystems.watch: be notified when a filesystem is mounted / unmounted
  • (3) fs.filesystems.format: format an existing filesystem

Filesystem elements

Scope: paths the permission applies on

These permissions are automatically granted if their scope is one of the application's own data directories.

  • (3) fs.feid.path: convert a given FEID to a split path

  • (3) fs.feid.exists: check if a given FEID exists in a filesystem

  • (3) fs.path.canonicalize: canonicalize a split path

  • (3) fs.items.exists: check if an item exists at a given path

  • (3) fs.items.metadata: get metadata on a given item

  • (3) fs.items.create: create new filesystem elements

  • (3) fs.items.move: rename and move existing filesystem elements

  • (3) fs.items.remove.trash: send items to the current user's trash

  • (3) fs.items.remove: delete items permanently

  • (3) fs.items.read: read an existing filesystem element's content

  • (3) fs.items.write: update an existing filesystem element's content

  • (3) fs.dir.read: list items contained inside given directories

  • (3) fs.dir.read.hidden: list hidden items in directories

  • (3) fs.dir.lock: lock a full directory's content to prevent modifications from other processes

  • (3) fs.dir.lock.full: lock a full directory's content to prevent modifications and access from other processes

  • (3) fs.symlinks.create: create symbolic links

  • (3) fs.symlinks.update: update existing symbolic links

  • (3) fs.symlinks.read: read the target of a symbolic link

Flows

  • (3) flow.list: list flows opened by all applications
  • (3) flow.metadata: get metadata on flows
  • (1) flow.create: create flows
  • (3) flow.read: read from flows
  • (3) flow.write: write to flows

Network

  • (3) net.fetch: fetch a resource
    Scope: list of domains (wildcards supported for subdomains), list of protocols
  • (3) net.expose: listen to a specific port or a range of ports Scope: range of ports (may cover a single port)

User accounts

  • (1) users.current.level: get the current user's permission level
  • (1) users.current.name: get the name of the current user account
  • (1) users.current.icon: get the icon associated to the current user account
  • (4) users.current.photo: get the (optional) photo associated to the current user account
  • (3) users.current.lastconn: get the timestamp of the current user's last login

All these permissions have a variant to get informations about other user accounts (replace current with all). These have the same permission level but require the current user to be administrator.

System

  • (1) system.clock.date.read: get the current date

  • (1) system.clock.time.read: get the current time

  • (1) system.clock.timezone.read: get the current timezone

  • (1) system.clock.uptime: get the system's uptime

  • (1) system.hw.cpu.count: get the number of CPUs

  • (1) system.hw.cpu.list: list CPUs (model, frequency, etc.)

  • (1) system.hw.mem.total: get the amount of total memory

  • (1) system.hw.mem.available: get the amount of available memory

  • (1) system.hw.mem.slots: get the number of RAM slots

  • (1) system.hw.mem.list: list RAM slots (model, frequency, etc.)

Hardware interfaces

  • (4) hwmid.keyboard.write: simulate key presses
  • (5) hwmid.keyboard.read: read key presses from the keyboard
  • (4) hwmid.pointer.write: simulate actions from pointer devices
  • (5) hwmid.pointer.read: read actions from pointer devices
  • (5) hwmid.video.read: access video input streams (cameras, ...)
  • (5) hwmid.audio.read: access audio input streams (microphones, ...)
  • (5) hwmid.screen.read: read portions of the screen as image buffers (screenshots, ...)
  • (1) hwmid.screen.read_self: read portions of the current application's windows (screenshots, ...)

State

  • (1) state.laptop.is: check if the device is a laptop
  • (1) state.laptop.is_open: check if the laptop is physically opened
  • (1) state.battery.has: check if the device has a battery
  • (1) state.battery.level: check the battery level
  • (1) state.battery.is_charging: check if the battery is charging
  • (1) state.battery.is_full: check if the battery is considered full
  • (1) state.battery.is_critical: check if the battery is at a critically low level
  • (1) state.battery.estimated_remaining: get the remaining running time estimated by the system in minutes

Programs

  • (5) + {A} programs.running.processes: get the list of all running processes
  • (5) + {A} programs.running.windows: get informations on each running application's windows
  • (5) programs.running.apps: get the list of running applications

System-reserved permissions

These permissions are reserved to processes marked as system.

  • sysapp.process.kill: suspend or kill external processes
  • sysapp.process.signal: send a signal to an external process

Containers

A container is a virtual space containing a set of limited resources. They are used to isolate sets of applications and establish virtual environments.

Concept of container

A container is represented by a single 64-bit identifier, randomly generated by the kernel. It is attached as a process' attributes.

It is created, managed and destroyed by the sys::proc service.

Containers hierarchy

A given container may create child containers, in which case the creator is called the parent container.

When a container is destroyed, all children containers in the hierarchy are destroyed as well.

Process isolation

Processes in containers can only communicate with processes of the same container. No outside access is permitted.

In practice, communication with system services is possible thanks to them being aware of the container's informations when receiving a process request.

Communication with the kernel is fully permitted, with limitations applying to the container itself.

Virtual hardware

No hardware device is available by default to a container. Real hardware devices can be connected using a passthrough method from the parent container, and virtual hardware devices can be plugged as well.

Virtual filesystem

The main filesystem of a container is by default a virtual filesystem stored inside a virtual storage file (VSF).

The parent container cannot directly connect a part of its own filesystem into a child's one ; it needs to setup a virtual storage device, which can then be mounted using the related system services.

Note that the system has builtin support for a set of virtual devices which are meant to be used on containers, with native support, which permits to prevent any driver overhead.

Performances

Performances of a container are exactly the same as a normal running program, given there is no emulation of any sort.

However, some I/O operations may result in a small overhead given that system services, as well as the kernel, need to check the permissions associated to the container in addition to transmitting methods and notifications from virtual hardware devices.

A container can be created instantly.

Limitations

A container can only run NightOS applications. As it is not a virtul machine, the kernel version and behaviour will also remain exactly identical across all containers, and upgrading the kernel itself will upgrade it for all containers as well.

If an application is not directly installed inside a container but instead shared as a mount point from the parent container, upgrading or removing the application will have the same effect in the child container.

Registry

This document describes the format of the system registy as well as its content.

Format

The registry is located in the /etc/sys/registry file. It's basically a nested B-Tree map.

Each map has string keys and a value that describes the entry: its purpose, its type with optional constraints, and then the type-safe value.

All types are described as a name, optionally followed by a maximum number of entries between brackets or a maximum number of bytes between braces. Here is the list, in HFRR notation:

NameSize (s) in bytesDescription
structs = ?A structure - keys are ASCII strings and values' type may be various
strings = ?An UTF-8 encoded string
string[x]x <= s <= x * 4An UTF-8 encoded string with a maximum capacity of x characters
string{x}s <= x * 4An UTF-8 encoded string with a maximum capacity of x bytes
asciistrs = ?An ASCII encoded string
asciistr[x]s <= xAn ASCII encoded string with a maximum capacity of x characters
asciistr{x}s <= xAn ASCII encoded string with a maximum capacity of x bytes
int{x}s = xA signed integer on x bytes
uint{x}s = xAn unsigned integer on x bytes
float{x}s = xA floating-point number on x bytes
pfloat{x}s = xA positive floating-point number on x bytes
intss = {CPU bits}A signed integer on as many bytes as the CPU
uintss = {CPU bits}An unsigned integer on as many bytes as the CPU
bools = 1A boolean
list:types = ?A list of entries with the provided type
list[x]:types = sizeof(type) * xA list of entries with the provided type with a maximum of x entries
structlists = ?A list of structures guaranteeing its first entry (the structure model) will never be removed
map:(key):(val)s = ?A dictionary mapping keys of the key to values of the val type
mapc:(key):(val)s = ?A dictionary mapping all possible keys of the key to values of the val type
structmap:(key)s = ?A dictionary mapping keys of the key to structures
structmapc:(key)s = ?A dictionary mapping all possible keys of the key to structures
time(p)s <= 8A duration with a precision of p (see below)
tmin(p,min)s <= 8Equivalent of time(p) but with a minimum value
tmax(p,min)s <= 8Equivalent of time(p) but with a maximum value
tbtw(p,min,max)s <= 8Equivalent of time(p) but with a minimum and a maximum value
stime(p)s <= 8Equivalent of time(p) but allows negative durations
size(p)s <= 8A data size with a precision of p (see below)
smin(p,min)s <= 8Equivalent of size(p) but with a minimum value
smax(p,min)s <= 8Equivalent of size(p) but with a maximum value
sbtw(p,min,max)s <= 8Equivalent of size(p) but with a minimum and a maximum value
id:apps <= 256Identifier of an installed application (ASCII string)
id:libs <= 256Identifier of an installed library (ASCII string)
id:users = 4Identifier of an installed user (32-bit unsigned number)
in:(type):(a...)s = max(sizeof(type))Any value in the provided tuple
enum:(a,b,c,...)s = 1One of a, b, c...

The registry's root is a struct. Each key-value association in a struct is called a node. Nodes that are single values (neither a list, a map or a structure) are called leafs.

All sizes that are exceptly provided in bytes must be a multiple of two.

Notes about types

  • Durations (time, tmin, tmax and tbtw types) have a precision which indicate the smallest unit of time they accept. 1 is for years, 2 for months, 3 for weeks, 4 for days, 5 for hours, 6 for minutes, 7 for seconds, 8 for milliseconds, 9 for microseconds and 10 for nanoseconds.

  • Sizes (size, smin, smax and sbtw types) accept values that are a multiple of their precision. 1 is for terabytes, 2 for gigabytes, 3 for megabytes, 4 for kilobytes, 5 for bytes and 6 for bits.

  • For the struct type, there is no need to specify the list of possible keys and the type of associated values, because they are already present in the registry when it's installed by the system - so it's all implicit. The keys in a struct should never change through time, nor the type of the value of each key. As such, it is not possible to create a list of struct, for instance.

  • For map structures (structmap and structmapc), the first key when sorting using the default algorithm for each type is guaranteed to never be removed. Its mapped value acts as the model structure for all over values in the map.

HFFR Format

The registry can be converted for debugging to an HFRR text file (Human-Friendly Registry Format).

In this format, structures' keys are described using the key(type): format. Each line following it describing its value is indented by a tabulation (\t). The tabulation is decreased when the value has been fully describes.

Strings are describes using double quotes, with a r prefix symbol for ASCII strings. Floating-point numbers use a . symbol as their decimal separators, and an explicit + or - symbol indicates all numbers' sign.

Each list entry is prefixed by a - symbol, and each key is described as a set of key: value list (on multiple lines for lists and maps).

Empty lists are represented as [] and empty maps as {}.

Durations are represented as a combination of one or more integer numbers each followed by a time suffix (ns for nanoseconds, us for microseconds, ms for milliseconds, s for seconds, m for minutes, h for hours, d for days, w for weeks, mo for months or y for years). Also, combinations use the space separator and numbers are represented with leading zeros if necessary, except for the first number.

So for instance, a duration of 2 days and 10 seconds will give the string 2d 10s.

Sizes are represented just like times, but with different suffixes (TB for terabytes, GB for gigabytes, MB for megabytes, KB for kilobytes, B for bytes and b for bits), and without leading zeroes.

So for instance a size of two terabytes and three kilobyte will be represented as 1TB 3KB.

The easiest way to understand the format is to look at the default registry structure presented below.

Structure

NOTE: This section is under heavy development and is by no mean complete!

Here is the content of the default registry file (when installing NightOS with all default settings), converted to the HFRR format:

# Kernel configuration (only editable in developer mode)
kernel(struct):
  # Signals configuration
  signals(struct):
    # Delay before forced suspension after WILL_SUSPEND signal
    suspend_delay(time(ms)): 500ms
    # Delay before forced termination (kill) after WILL_TERMINATE signal
    terminate_delay(time(ms)): 500ms
    # Delay for a service to answer a connection request
    service_answer_delay(ntime(ms,100ms)): 500ms

# System configuration
system(struct):
  # Debugging options
  debugging(struct):
    # Is development mode enabled?
    dev_mode(bool): true

  # Encryption
  encryption(struct):
    # Is the global storage encrypted?
    global_encrypted(bool): false

  # Date and time
  datetime(struct):
    # Is it based on the internet? (if not, it's a custom one)
    internet_based(bool): true
    # In the case the date is not internet-based, provide a timestamp which indicates the difference between the computer's
    #  local datetime and the one that should be displayed
    custom_based_diff(stime(ns)): 0ns

  # Crash saves
  crash_saves(struct):
    # Restore crash saves as soon as the user logs in
    restore_on_login(bool): true
    # Do not save crash saves for guest users
    disable_for_guest_user(bool): true
    # Delay between each crash save
    collect_every(tmin(s,1s)): 60s
    # Ask an application to create a new crash save even if the previous new has not been completed yet
    collect_if_previous_unanswered(bool): false

  # Freeze-prevention
  freeze_prevention(struct):
    # Reserved amount of memory (0 = disabled)
    reserved_memory(MB): 16MB
    # Reserved amount of CPU per virtual core (0 = disabled)
    reserved_cpu_vcore(uint{1}): 1

  # Processes' lifecycle
  processes(struct):
    # Delay before suspension
    suspend_delay(time(ms)): 500ms
    # Delay before termination
    terminate_delay(time(ms)): 2000ms
    # Delay for services to answer connections
    service_answer_delay(time(ms)): 2000ms

# Users (keys = user's unique identifier, value = user's type with 0 = main admin., 1 = admin., 2 = standard, 3 = guest)
# Detailed informations about each user (nickname, privileges, encryption password, ...) and groups are stored
#   in the user profiles file at '/etc/sys/users'
users(map:(uint{4}):(enum:(MainAdmin, Admin, Standard, Guest))):
  0: Admin # System
  1: Admin # Administrator

Vocabulary

This document introduces all vocabulary related to NightOS;

Surface

A surface is a two-dimension area displayed on the screen. It is often used to refer to an application's window.

Interaction

An interaction with a given surface consists in the user clicking on it on pressing a key while this surface is focused.

Windows

Window

A window is a 2D figure provided to an application for manipulation. It does not refer to the desktop environment itself, or to the bootloader's screens for example. There are several types of windows: resizable windows and borderless windows.

Borderless windows

Borderless windows are windows without any titlebar nor resizing lines.

Resizable windows

Resizable windows are the most common type of windows. They feature a title bar as well as resizing lines.

Windows' title bar

A window's title bar is a bar located above the window's content. It contains:

  • On its left: a set of four buttons (customizable) which allow to hide the window in the taskbar, restore/maximize it, close it, or suspend/resume the related application ;
  • On its center: the window's title, which is controlled by the application and may change during the window's lifetime

Windows' resizing lines

A window's resizing lines are four lines on all sides of a window, 1-pixel wide or high, which allow to extend or reduce the window's size relatively to the line's position.

For example, dragging the top line to a pixel higher on the screen will result in the window being extended to this direction - but other borders of the window won't move. Dragging the top line to the opposite direction will result in the window being compressed to this direction.

The top line is located above the window's titlebar.

Hydre

The shell is the part responsible for running scripts, which are small text-based programs that are natively available on every NightOS installation. It is called Hydre and is part of the system under the form of the sys::hydre service.

You can get a quick overview of Hydre in its technical document.

NOTE: The behaviours described here only apply when using the Pluton terminal application. Although all terminals are expected to follow these specifications as they act as a standard for terminals, they are technically not forced to and so misbehaviours may appear when using an alternative terminal application.

Shell sessions

Before evaluating commands, a shell session needs to be created using the sys::hydre service.

A session has the following properties:

  • A (changeable) width and height, in number of characters
  • Commands output are concatenated

Commands can then be run inside the created session, and once they are all finished the same session can be closed using the same service. This simplify execution chaining, but also enables commands to be notified through the service's pipes when the session is resized.

Only one shell instruction can be run at a time, but as it may be made of multiple commands (e.g. cmd1 | cmd2), multiple processes may be binded to the same shell session. Also, background commands may be running in parallel of the running command.

For instance, when using the Pluton terminal, it creates on opening a shell session to run the commands in. When a new tab is opened, another shell session is opened for that tab. When the tab is destroyed, the session is destroyed as well, which means all commands running in it (including pipe and background commands) are killed.

Sessions security

When a command is run, the command's PID is registered in the session's actors list. When the command ends, it is removed from the list.
When a process contacts the sys::hydre service to access the session it runs in, Hydre gets the session associated to this process' PID, and returns it. If the PID is not associated to any session, the access is refused and the command won't be able to get any information on any session.

This prevents processes from accessing sessions they aren't part of.

Commands evaluation

Commands are evaluated one by one, as scripts cannot be run in a concurrent way. They are handled as follows:

  • Builtin commands are treated internally by the shell
  • Application commands will result in launching the requested application in a separate process

Command pipes

When an application is started from a command, its execution context indicates it and the process gets access to several pipes called the command pipes:

Pipe identifierStandard pipe namePipe typeFormatDescription
CMDINTyped inputRawtypedData coming either from a command pipe (`
CMDUSRInteractive inputMessageUTF-8Data coming from a terminal session (e.g. user inputs)
CMDMSGMessages outputMessageUTF-8Messages to display in the console, which won't be redirected by default
CMDERRErrors outputMessageUTF-8Messages to display as errors in the console, which won't be redirected by default
CMDRAWRaw bytesRawRawOutput data, which will be redirected if an output pipe (>) is used
CMDOUTTyped outputRawtypedTyped output data, which will be used by shell scripts)

The SC/RC identifiers of these pipes are available in the application's execution context.

The process that launched the command gets the ability to:

  • Send data to the callee's CMDIN/CMDUSR pipe ;
  • Read data from the callee's CMDMSG/CMDERR/CMDRAW/CMDOUT pipes

As this only applies to processes that can be started from commands, this only applies to application processes ; system processes not being linked to commands and worker processes being started from application processes (and so not directly from commands).

Technically speaking, commands are started by the sys::hydre service and so by a system process. This same service creates the pipes and handles them. For more informations on this, please check the service's documentation.

If the process terminates before the return value has been fully transmitted through CMDOUT or if it closes the CMDOUT pipe before fully transmitting the value, the process is considered as faulty and killed immediatly (if still alive). The calling script (if any) exits with an error message, unless the error is caught with catch, the error message being generated by the system.

Even if the process closes its CMDMSG or CMDRAW pipe properly (by calling the CLOSE_PIPE), the command is not considered as finished until the process itself did not terminate.

Note that when a return value has been fully transmitted through CMDOUT, all pipes are closed and the command is considered as finished.

Interactivity

When an application command is run, the pipes are handled as follows.

Input data

All data coming from a command pipe (|) or from an input pipe (<) (if the command's input type is stream) are transmitted through CMDIN. Once the input data have been fully transmitted, the CMDIN pipe is closed.

User inputs

All user inputs (including raw keystrokes) are transmitted to CMDUSR, except a few ones:

  • Ctrl-., which asks the process to suspend (triggers the SUSPEND signal)
  • Ctrl-Shift-., which forces the process to suspend (triggers the WILL_SUSPEND signal)
  • Ctrl-C, which asks the process to terminate (triggers the TERMINATE signal)
  • Ctrl-Shift-C, which forces the process to terminate (triggers the WILL_TERMINATE signal)
  • Custom GUI keystrokes like Alt-F4
  • System-handled keystrokes like Ctrl+Alt+Del

User input messages use the following format (always starting from the strongest byte/bit):

  • Byte 0: modifier keys
    • Bit 0: set if the Command or Windows key is pressed
    • Bit 1: set if the Ctrl key is pressed
    • Bit 2: set if the Alt key is pressed
    • Bit 3: set if the Shift key is pressed
    • Bit 4: set if the Fn key is pressed
    • Bit 5: set if Numeric Lock is enabled
    • Bit 6: set if Caps Lock is enabled
    • Bit 7: set if Scroll Lock is enabled
  • Byte 1: keycode
  • Bytes 2-5: UTF-8 printable character on 4 bytes, or 0x00 if the character is not printable

Text output

Commands may output either simple text messages (via CMDMSG) or error text messages (via CMDERR).
Conventionnally (but it's up to the terminal application), text messages are displayed by default in white while error messages are displayed in red.

Messages must use the following format (always starting from the strongest byte/bit):

  • Byte 0-2: RGB foreground color (0 will use the current one)
  • Byte 3-5: RGB background color (0 will use the current one)
  • Byte 6: style
    • Bit 0: if set, the message will be displayed in bold
    • Bit 1: if set, the message will be displayed in italic
    • Bit 2: if set, the message will be displayed with a line below it
    • Bit 3: if set, the message will be displayed with a line above it
    • Bit 4: if set, the message will be displayed reversed
  • Byte 7: number of control characters
  • Byte 7-(N): control characters on 2 bytes each
  • Bytes (N+1)-(END): The message itself, encoded in UTF-8 (can be empty)

Control characters use the following format:

  • Strongest byte: control character code
    • 0x00: no control character
    • 0x01: move cursor up X times
    • 0x02: move cursor left X times
    • 0x03: move cursor right X times
    • 0x04: move cursor down X times
    • 0x05: move the cursor to the beginning of the line
  • Weakest byte: data byte

For instance, an 0x0305 control character is decomposable in the 0x03 code and the 0x05 data byte, which means moving the cursor to the right 5 times.

Invalid messages

A message is considered invalid if at least one of the following conditions is verified:

  • The message's length is lower than 8 bytes + 2 * (number of control characters)
  • The provided control character is invalid

Messages that do not follow this format will result in displaying Unicode's replacement character (0xFFFD: ďż˝) instead.

Messages providing an invalid foreground and/or background color will conventionnally (but it's up to the terminal application) in a specific color to indicate no right color could be determined.

Events handling

Commands can get informations on the current session using the sys::hydre service.

This allows the command to be notified of events like windows resizing. For more informations, see the service's specifications document.

Scripting language

You can find more about the script language in the language's specifications document.

Pre-evaluation checking

Before executing a script, the shell looks for errors in it, such as unknown command, invalid argument, type mismatch and so on. If an error is foud, the script doesn't even start to run ; an error is directly reported and the command is considered as failed.

This prevents errors from happening in the middle of a script, leaving it in an inconsistent state ; this also makes script errors easier to debug as they are reported at compile time.

Shell scripting

The scripting language of Hydre offers a lot of powerful easy-to-use features. This allows to create complex script that are still very readable and maintanable.

Running a command

Hydre uses an intuitive syntax. Commands are written like they would be in *-sh shells: the command name is followed by arguments, each separated by a space.

Arugments can either be positional (they are written directly), shorts (prefixed by a - symbol and one-character long), or longs (prefixed by two - symbols). Here is an example:

cmdname pos1 pos2 -a --arg1

This line runs the command called cmdname, provides two positional arguments pos1 and pos2, a short one a and a long one arg1. Short and long arguments are called dash arguments.

Combining short arguments

It's possible to combine multiple short arguments in once by writing them one after the other:

cmdname -abc

This line is strictly equivalent to:

cmdname -a -b -c

Argument values

Short and long arguments can also require a value. This value must be provided using an = symbol or by simply using a space:

cmdname -s=1 --long=2
# or:
cmdname -s 1 --long 2

Comments

Comments can be written on a single line with the # symbol:

cmdname # single-line comment

Everything after the # symbol is ignored. For multiple-line comments, it's required to use three # symbols:

###
this
is a
multi-line
comment
###

Variables

Variables are declared with the var keyword:

let age = 19

Here, we declare a variable age with value 19. As variables are typed, this variable will only be allowed to contain numbers from now on - no strings, no booleans, nothing else.

And to assign a new value to it:

age = 20

Value types

All arguments must match their expected type: if the command is expecting a number, we can't give it a list for instance.

Here is the list of all types and how they are written:

# Booleans (bool) (`true` or `false`)
_ = true

# Integers (int)
_ = 3

# Floating-point numbers (float)
_ = 3.14

# Characters (char)
_ = 'a'

# Strings (string)
_ = "abc"

# Lists of a given type (list[type])
_ = [ 3, 3.14 ]

# Maps (key-values) (map[key_type, value_type])
_ = { "a": 1, "b": 2 }

# Paths (path) - must contain at least one `/` to indicate clearly that it's a path and not a string or something else
_ = dir/file.ext
_ = ./file.ext
_ = /tmp

# Commands (used to run custom commands later in functions)
_ = @{ command "pos1" -s --long }

A string is composed of multiple chars, which are made of single codepoints. This means a grapheme cluster made of multiple codepoints will need to be encoded in a string.
There is also the any type which accepts values of all types, and stream which we'll talk about later as it's a special type.
Finally, there is the void type which cannot be written 'as is' but is used in special contexts like commands return values. It's a type that contains no data at all.

There are also flag arguments, which are dash arguments that take no value. The command will simply check if the argument was provided or not.

NOTE: In order to avoid writing errors, positional arguments cannot be provided after a flag argument.

For instance, considering pos1 and pos2 are positional arguments, --flag a flag argument and --val a non-flag long argument:

# VALID
command pos1 pos2 --flag --val 2

# VALID
command pos1 --val 2 pos2 --flag

# VALID
command pos1 --flag --val 2 pos2

# VALID
command --flags --val 2 pos1 pos2

# INVALID (we could think by reading this that "pos2" is the
#          value of the non-flag argument "--flag")
command pos1 --flag pos2 --val 2

As you can see, suites of characters that do not start with a dash (-) can be written without quotes when they're part of a command. Any special character (like spaces or dashes) can be escaped using the \ prefix:

command this\ is\ the\ same\ \-\ argument

# Strictly equivalent to:

command "this is the same - argument"

Variables shadowing

Any variable can, inside a program, be replaced by a new variable with a same name but with a different type. This is called shadowing.

It can be useful when converting data from a type to another, such as:

let names: list[string] = [ "Jack", "John" ]

let names: string = names.join(",") # this is called a _method_, we'll talk about them later

Here, we shadowed the names variable to create a new one with a different type from the data we had before, which means we cannot access the original names variable anymore.

Expressions

To use a variable, we can directly use it like this::

tellage $age

So this code:

let age = 20
tellage $age

Is equivalent to this one:

tellage 20

It's also possible to perform computations using expressions:

let age = 20
let add = 12
tellage (age + add)

String, characters and numbers can also be inserted inside strings:

let name = "Jack"
echo "Hello, $name!" # Hello, Jack!

Other expressions require to use ${...}. For instance, values in lists through their index (starting at 0):

let names = [ "Jack" ]
echo "Hello, ${names[0]}!" # Hello, Jack!

Values in maps through their key:

let ages = { "Jack": 28 }

echo "Jack is ${ages["Jack"]} years old!"

Note that getting an out-of-bound index will make the program panic, which means it exits immediatly with an error message.

let names = [ "Jack" ]
echo "Hello, ${names[1]}!" # Panics

In commands, expressions can be provided wrapped in (...):

echo (2 + 1) # Will display: "3"

Computing values

It's also possible to compute values using operators. Each operator takes one or multiple operands, which can be either a variable or a literal value.

Mathematical operators

Mathematical operators all take two operands. If any of the operands is not a number, it will fail. If both operands are integers, the result will be an integer, but if at least one is a floating-point number, so will be the result.

The operators are:

  • +: addition
  • -: substraction
  • *: multiplication
  • **: pow
  • /: division
  • //: floating-point division (gives a floating-point number even if the two operands are integers)
  • %: remainder (works only with two integers)

Bit-wise operators

Bit-wise operators only take integer operands and produce an integer result:

  • & bit-by-bit and
  • | bit-by-bit or
  • ^ bit-by-bit exclusive or
  • << binary left shift operator
  • >> binary right shift operator
  • ~ one's complement - takes a single number

Logical operators

Logical operators take two operands and return a boolean:

SymbolNameReturns true if...
&&anda and b are true booleans
||ora, b or both are true booleans
==equal toa is equal to b
!=different thana is different than b
>greater thana is greater than b
<lower thana is lower than b
>=greater than or equal toa is greater than or equal to b
<=lower than or equal toa is lower than or equal to b

Finally, there is the ! operator, which takes a single operand on its right, and simply reverts a boolean.

Assignment operators

The neutral assignment operator = can be prefixed by any mathematical, bit-wise or logical operator's symbol(s). The operator's left operand will be the current variable's content, while the right one will be the value on the left of the assignment operator. The result will then be stored in the variable.

Note that for a variable to be modified, it must be declared as so:

# Immutable
let a = 3

# Mutable
let mut a = 3

Otherwise, re-assigning a value to the variable would fail.

# We declare the variable as mutable...
let mut a = 3

# ...and we can then re-assign to it
a += 1
a *= 8
a /= 2

echo $a # 16

Note that the result's type must also be compatible with the variable:

let a = 0

a &&= 1 # ERROR: Cannot assign a 'bool' to an 'int'

There are also the ++ and -- operators, which respectively increase and decrease the desired variable:

let a = 0

a ++
a ++
a --

echo $a # 1

Lists and maps

Lists and maps behave in a similar way: while lists have indexes, which are handled behind-the-scenes, maps have keys, which are provided explicitly.

Here is how we declare a list or a map:

# Type: list[string]
let names = [ "Jack", "John" ]

# Type: map[string, int]
let ages = { "Jack": 28, "John": 29 }

To add a new value (requires the variable to be mutable):

# Lists
names[] = "Paolo"

# Maps
ages["Paolo"] = 26

To get a value:

# Type: int
names[0]

# Type: int
ages["Paolo"]

To remove a value:

# Lists
delete names[0]

# Maps
delete ages["Paolo"]

To get the number of entries:

# Type: int
names.len()

# Maps: int
ages.count()

Blocks

Blocks allow to run a piece of code multiple times or if a specific condition is met. They are useful combined to comparison operators.

Conditionals

Conditionals uses the following syntax:

if # condition
  command1
end

When this block is ran, if condition (which is an expression that must result in a boolean) is equal to true, command1 is ran. Here is an example:

if 2 + 2 == 4
  command1
end

It's possible to specify multiple commands at once:

if 2 + 2 == 4
  command1
  command2
  command3
end

Note that all commands must be indented by one tabulation.

It's also possible to run a set of commands in case the condition isn't met too using else:

if 2 + 2 == 4
  command1
else
  command2
end

Finally, conditions can be chained using elif:

if 1 + 1 == 4
  echo "Bizarre."
elif 1 + 1 == 3
  echo "Bizarre!"
else
  echo "Normal."
end

Switches

A switch allows to perform actions depending on a value. It's roughly equivalent to a combination of multiple if and elif statements.

switch rand_int(0, 10)
  when 0
    echo "It's zero!"

  when 1
    echo "It's one!"

  else
    echo "It's something else"
end

Note that, for blocks that only contain a single instruction, we can shorten this using the following syntax:

switch rand_int(0, 10)
  when 0 => echo "It's zero!"
  when 1 => echo "It's one!"
  else   => echo "It's something else!"
end

Loops

Loops allow to run a piece of code for a while. The most common loop is the range loop:

for i in 0..10
  command $i
end

This will run command 0 to command 9. To include the upper bound, we must add an = symbol:

for i in 0..=10
  command $i
end

This will run command 0 to command 10.

We can also iterate on a list:

let str = "Jack"

let list = [ "Jack", "John" ]

for name in list
  echo $name
end

This will display Jack and John.

To get the indexes as well, we can do:

for i, name in list
  echo "$i: $name"
end

This will display 0: Jack and 1: John.

For maps:

let ages = { "Jack": 28, "John": 29 }

for name, age in ages
  echo "$name is $age years old!"
end

There is another type of loop, which runs a piece of code while a condition is met:

while # condition
  command
end

If the condition is false when the loop is reached, the command will not be ran once. Else, it will be ran as long as the condition is true.

Note that loops can be broke anytime using the break keyword:

for i in 0..10
  command $i

  if i == 2
    break
  end
end

This will run command 0, command 1 and command 2 only.

Filesystem iteration

It's possible to iterate on a list of files and directories:

for file in (./*.txt)
  echo "Found a text file: $file"
end

The pattern between parenthesis must be a glob pattern. Recursivity is supported to:

for file in (**/*.txt)
  echo "Found a text file: $file"
end

Variables scoping

When a variable is declared, it is scoped to the current block, meaning it doesn't exist outside of the current block:

# This variable is declared in the "global" block
# so it's available everywhere in the current script
let firstName = "Jack"

if firstName == "Jack"
  # This variable is declared in an "if" block
  # so it's not available outside of it
  let lastName = "Sparrow"
end

echo $firstName # Prints: "Jack"
echo $lastName # ERROR ("lastName" is not in scope)

Also, variables are not shared between scripts.

Functions

Functions allow to split the code in several parts to make it more readable, as well as to re-use similar pieces of code across the script.

fn hello()
  echo "Hello!"
end

Now we can call hello this way:

hello() # Will print "Hello !"

Arguments

Function can also take arguments, which must have a type.

fn hello (name: string)
  echo "Hello, $name!"
end

hello("Jack") # Will print "Hello, Jack!"

Arguments can be made optional by providing default values. This also allows to get rid of their explicit type as it's now implicit:

fn hello (name = "Unknown")
  echo "Hello, $name!"
end

hello()       # Will print "Hello, Unknown!"
hello("Jack") # Will print "Hello, Jack!"

Note that a function's arguments do not require to wrap expressions between ${...} as it's implicit. Which means we can write:

fn sayNumber (num: int)
  echo "Number is: $num"
end

sayNumber(2 + 1) # Will print "Number is: 3"

We can also combine functions and blocks, for instance:

fn greet (names: list[string]) -> int
  # .len() is called an _extension_, we will see more about that later
  if names.len() == 0
    echo "No one to greet :|"
  else
    for name in names
      echo "Hello, $name!"
    end
  end
end

hello([])               # No one to greet :|
hello(["John", "Jack"]) # Hello, John! Hello, Jack!

Return types

Functions can also return values. In such case, they must specify the type of values they return, and ensure all code paths will return a value of this type:

fn add (a: float, b: float) -> float
  return a + b
end

Methods

All value types expose specific functions that can be used with a dot after a variable of the given type, called methods:

let letters = "abcdef"
let letters = letter.split(",")

Here, we use the split method of the string type, which returns a list[string].

Failing

Functions can also fail to indicate something went wrong:

fn divide (a: float, b: float) -> fallible float
  if b == 0
    fail "Cannot divide by 0!"
  else
    return a / b
  end
end

The fallible keyword must be present before the return type to indicate the function may fail (even if the function doesn't return anything).

When a function fails, the program stops and print the provided error message. But it's also possible to handle the error:

fn handle_bad_div (a: float, b: float) -> float
  catch divide(a, b)
    ok result
      echo "Divided successfully: $a / $b = $result"

    err errmsg
      echo "Division failed :("
      echo "Here is the error message: $errmsg"
  end
end

Note that you may handle only the success or error case depending on your needs ; you do not have to handle both cases.

This keyword also allows to catch errors from commands:

# Run a command and get error messages from CMDERR instead if the command fails
catch $(somecommand)
  ok data => echo "Success: $data"
  err msg => echo "Errors: ${msg.join("; ")}"
end

Retries

It's possible to retry a function until it succeeds using the retry keyword:

fn may_fail ()
  if rand() > 0.5
    fail "I don't like high numbers"
  end
end

retry may_fail()

This will run may_fail, and run it again if it fails, until it succeeds.

It's also possible to specify a maximum of retries:

retry(5) may_fail()

For information, here is the declaration of the native retry_cmd command, which allows to try to run a command until it succeeds:

fn retry_cmd(cmd: command, retries: string) -> fallible
  retry(retries) cmd()
  if status() != 0
    fail "Command did not suceed after $retries retries."
  end
end

It can be used like this:

retry_cmd(@{ read "file.txt" }, 10)

Global failing

A whole script can fail using this keyword, which will result in displaying the error message in CMDERR and exiting immediatly.
The failure may be handled using fallible in the caller script.

Nullable types

Sometimes, it's useful to be able to represent a value that may be either something or nothing. In many programming languages, "nothing" is represented as the null, nil or () value.

Nullable types are suffixed by a ? symbol, and may either contain a value of the provided type or null. Here is an example:

fn custom_rand() -> int?
  let rnd = rand_int(-5, 5)

  if rnd > 0
    return rnd
  else
    return null
  end
end

To declare a variable with an nullable type, we wrap its initialization value in the nullable operator ?(...):

let a = 1 # int
let b = ?(1) # int?

Or we explicitly give the variable a nullable type:

let b: int? = 1 # int?

To initialize the variable with the null value instead, we must use the explicit version:

let c: int? = null # int?

Note that imbricated types are not supported, which means we cannot create int?? values for instance.

Handle the null value

If we try to access an nullable value "as is", we will get a type error:

let a = ?(1)

let b = 0
b = a # ERROR: Cannot use an `int?` value where `int` is expected

We then have multiple options. We can use one of the nullable types' function:

let a = ?(1)

let b = 0

# Make the program exit with an error message if 'a' is null
b = a.unwrap()

# Make the program exit with a custom error message if 'a' is null
b = a.expect("'a' should not be null :(")

We can also detect if a value is null by using the .isNull() method:

let a = ?(1)
let b: int? = null

echo ${a.isNull()} # false
echo ${b.isNull()} # true

There is also the .default(T) method that allows to use a fallback value in case of null:

let a = ?(1)
let b: int? = null

echo ${a.default(3)} # 1
echo ${b.default(3)} # 3

We can also use special syntaxes in blocks:

let a = ?(1)

if some a
  # While we are in this block, 'a' is considered as an 'int'
else
  # While we are in this block, 'a' is considered as 'null'
end

while some a
  # Same here
end

Also, if the program exits in all cases when the argument is considered as null or non-null, the opposite type will be applied to the rest of the program:

let a = ?(1)

if some a
  exit
end

# 'a' is considered as 'null' here

let b = ?(1)

if none b
  exit
end

# 'b' is considered as 'int' here

The case of optional arguments

When a command takes an optional argument, it's possible to provide a nullable value of the same type instead:

let no_newline = ?(false)

echo "Hello!" -n $no_newline

If the value is null, the argument will not be provided. Else, it will be provided with the non-null value.

Nullable any

The any type covers any type of values, meaning it accepts absolutely every single value, except the null value. Indeed, any is not nullable by default, so to make it accept null values we must use any? instead.

Advanced types

Structures

Structures map one or multiple fields to as many values. Here is an example:

fn sayHello(person: struct { firstName: string, lastName: string })
  echo "Hello, ${person.firstName} ${person.lastName}!"
end

sayHello({ firstName: "Bat", lastName: "Man" }) # Prints: "Hello, Bat Man!"

In such a simple example, it's easier to directly use a firstName and lastName parameters instead of a struct, but they are useful when returning heterogenous sets of data, for example in a list:

fn listRecursively(dir: path) -> list[struct { name: path, size: int }]
  let list: list[struct { name: path, size: int }] = []

  for item in $(ls $dir --details)
    if item.isDirectory
      listRecursively(dir)
    else
      list[] = { name: item.path, size: stats.sizeOf }
    end
  end

  return list
end

But, as this is not very readable, it's better to use a type alias:

type fsItem = struct { name: path, size: int }

fn listRecursively(dir: path) -> list[fsItem]
  let list: list[fsItem] = []
  # ...
end

Closures

Closures are anonymous functions which are generally used to repeat the same group of operations.

let test: fn (string, int) = { a, b -> echo (a.repeat(b)) }
test("Hello world! ", 3) # Prints: "Hello world! Hello world! Hello world! "

Note that closures can also return values implicitly:

let test: fn (string, int) -> string = { a, b -> a.repeat(b) }
echo (test("Hello world!", 3)) # Prints: "Hello world! Hello world! Hello world! "

Also, closures are not forced to take their declared parameters:

type testType = fn (string, int)

let test: testType = { a, b, c -> ### ... ### } # NOT VALID
let test: testType = { a, b    -> ### ... ### } # Valid
let test: testType = { a       -> ### ... ### } # Valid
let test: testType = {         -> ### ... ### } # Valid

Here is a concrete usage example:

fn forEachFile(dir: path, callback: fn (path))
  for item in $(ls $dir --details)
    if item.isFile
      callback(item.fullPath)
    end
  end
end

forEachFile(./, { file -> echo "File: $file" })

Streams

An usual type for manipulating large data is stream, which is notably used to treat a chunk of data that is either too large for the memory or is more easier to treat as things progress.

Data validation

Some commands may return a value whose type cannot be predicted before runtime. For instance, a command fetching a JSON object from a remote server will, in case of success, return a value but whose type cannot be known beforehand.

Data validation is a feature allowing to scripts to check the type of an unknown value at runtime, using type assertions.

Here is an example:

let test: any = [ 2, 3, 4 ]

# 1. Here, "test" is considered to be of type "any"

if test is list[int]
  # 2. Here, "test" is considered to be of type "list[int]"
else
  # 3. Here, "test" is considered to be of type "any"
end

The isnt keyword can also be used:

let test: any = [ 2, 3, 4 ]

# 1. Here, "test" is considered to be of type "any"

if test isnt list[int]
  # 2. Here, "test" is considered to be of type "any"
else
  # 3. Here, "test" is considered to be of type "list[int]"
end

If, in an "isnt" conditional, the script exits/fails in all cases (or returns if we're in a function), the value is considered with the asserted type for the rest of the script:

let test: any = [ 2, 3, 4 ]

# 1. Here, "test" is considered to be of type "any"

if test isnt list[int]
  # 2. Here, "test" is considered to be of type "any"
  exit
end

# 3. Here, "test" is considered to be of type "list[int]"
# > Because it's impossible for "test" to not be "list[int]" as in that case the program would have exited

This can be used to check complex structures as well:

# Considering we have a `fn fetchJson(url: string) -> any' function that fetches a JSON value from the web

type User = struct { id: int, firstName: string, lastName: string, email: string }

let json = fetchJson("https://mysuperapi.../users/all")

if json isnt User
  fail "JSON doesn't have the correct structure"
end

# "json" is considered as a "User" here

Event listeners

Scripts can listen to events using the on keyword:

on keypress as keycode
  echo "A key was pressed: $keycode"
end

This program will display a message each time a key is pressed.

If the event listener is registered in a function, the is automatically unregistered when that function returns. If it's registered outside a function, it is unregistered when the script ends.

fn test()
  on keypress as keycode # (1)
    # Do something
  end
  # Event listener (1) is unregistered here
end

on keypress as keycode # (2)
  # Do something
end

echo "Hello world!"
# Event listener (2) is unregistered here

Waiting

Scripts may wait for a specific event before continuing. This can be achieved without a while loop that consumes a lot of CPU, using the wait keyword:

wait condition

The script will block while the provided condition is not true. The checking interval is defined by the system, and the condition should as fast to check as possible to consume as little CPU as possible.

echo "Please press the <F> key to validate your choice"

let validated = false

on keypress as keycode
  if keycode == KEY_F
    validated = true
  end
end

wait validated

echo "Thanks for validating your choice :D"

It's also possible to wait for a variable to not be null:

echo "Please press a key"

let key: int? = null

on keypress as keycode
  key = keycode
end

wait some key

echo "You pressed key: $key"

Imports

As you may already know, command names can be quite long and complicated. In order to prevent from having to repeat very long names that are not really readable, it's recommanded to use imports which are declared at the beginning of script with the form <dev>::<app>::<command>:

import system::fs::read_file

read_file # ...

It's then possible to use the read_file without prefixing.

To perform multiple imports at once:

import system::fs::{read_file, write_file}

It's also possible to only bind the application itself:

import system::fs

fs::read_file # ...

Aliases

Imported commands can also be aliased:

import system::fs as sysfs

sysfs::read_file # ...

Import expansions

It's possible to import all commands from an application, with:

import system::fs::*

read_file # ...

But also to import all applications from a developer, with:

import system::*

fs::read_file # ...

Note that, if a name clash occurs - if two applications or commands with the same name are imported -, the script won't be able to run.

Non-clashing namespace

The non-clashing namespace is a namespace that can be imported, where live all commands whose name is unique across all applications.

For instance, let's imagine we have two applications:

  • AppA by DevA, which exposes a cmd_a and a cmd_z command ;
  • AppB by DevB, which exposes a cmd_b and a cmd_z command

What happens here? While the cmd_z command has a name clash between AppA and AppB, the cmd_a and cmd_b commands don't. This results in these last two commands being also put in the nonclashing namespace, which can then be imported like a traditional application:

import shell::nonclashing

nonclashing::cmd_a # OK
nonclashing::cmd_b # OK
nonclashing::cmd_z # ERROR (not in namespace)

This means we can also import all commands that don't clash with other ones:

import shell::nonclashing::*

cmd_a # OK
cmd_b # ...
cmd_z # ERROR (not in namespace)

Volatile imports

As volatile applications' commands are not exposed globally, there is a special import syntax for such applications, allowing to import their commands directly from their application package:

import ./app.nva::super_command
super_command # ...

# OR
import ./app.nva as app
app::super_command # ...

Commands input & output

Reading a command's output

Commands output data through pipes. There are several output pipes that can be used:

  • CMDOUT is the default output pipe, which returns typed values (the command declares its output type beforehand)
  • CMDRAW allows to send a stream, which is useful when dealing with a lot of data or with external data
  • CMDMSG allows to send string messages that are displayed in the terminal's windows
  • CMDERR allows to send string messages that are also displayed, but as error messages

There is a specific syntax to get the output from each pipe. To get the (typed) output from CMDOUT:

_ = $(echo "Hello!") # Contains: "Hello!"

This is called the typed reception operator. It can be used like this for instance:

echo ${$(echo "Hello!")} # Prints: "Hello!"

But, as this syntax is not very readable, evaluating a single command can be made without the ${...} expression wrapper:

echo $(echo "Hello!") # Prints: "Hello!"

Note that this only work if the command supports piping through CMDOUT.

To get the result from CMDRAW instead (as a stream), if the command supports it:

# '-b' makes 'echo' read from a stream
echo -b $@(streamify "Hello!") # Prints: "Hello!"

To get the output of CMDMSG instead (as a list[string]):

echo $?(echo "Hello!") # Prints: "["Hello!"]"

To get the output of CMDERR (as a list[string]):

echo $!(echo "Hello!") # Prints "[]"

To get the combined output of CMDMSG and CMDERR (as a list[string]):

echo $*(echo "Hello!") # Prints "["Hello!"]"

Note that using the $(...) operator will make the program panic if the command exits with a non-zero status code.

Redirecting the output to a file

It's also possible to redirect the output of a command to a file, using the > operator. The values are converted to strings before being written, except stream values which are written as they are.

echo "Hello!" > ./test.txt

This works because echo outputs by default to CMDOUT, not to CMDMSG. If it did, we could still perform the redirection this way:

echo "Hello!" ?> ./test.txt

The prefixes are the same as for the $(...) operator:

  • > for CMDOUT
  • @> for CMDRAW
  • ?> for CMDMSG
  • !> for CMDERR
  • *> for CMDMSG and CMDERR combined

Output data

If a script is declared as a command, it gets its own CMDIN, CMDOUT and CMDRAW pipes (the CMDUSR, CMDMSG and CMDERR pipes remain as usual).

They can be accessed using three built-in commands: cmdin, cmdout and cmdraw.

Reading from CMDIN

The cmdin command simply writes in its CMDOUT the value provided in the shell's CMDIN. For instance:

# Considering this shell script accepts `string` values as input.

# If this shell script is called with 'Hello world!' as an input:
echo $(cmdin | length) # Prints: 12

The original type is preserved, which means we can perform typed operations on the input value.

Returning with CMDOUT

The cmdout command takes a typed value and writes it to the shell script's CMDOUT pipe. This also makes the program exit.

Writing to CMDRAW

The cmdraw command takes a stream value and writes it to the shell script's CMDRAW pipe. Only one stream can be piped at a time, so if cmdraw is called while another is pending, the command will simply fail (this can be caught with catch).

Input of a command

Some commands accept inputs through the command pipe | operator. They can be used this way:

echo "Hello world!" > ./somefile.txt

read ./somefile.txt # Prints: "Hello world!"

read ./somefile.txt | length # Prints: 12

How does this work exactly? First, read reads the file and outputs it to CMDOUT as a string, which is then passed to length which happens to accept strings as an input. It then computes the length of the provided input and writes it to CMDOUT, as an int. Which means we can do that:

echo ${ $(read ./somefile.txt | length) * 2 } # Prints: 24

The input may be typed and only accept specific types of values. For instance length only accepts strings, so if we try to give it something else:

echo $(pass 2 | length) # ERROR

What happens here is that we use the builtin pass command which writes to CMDOUT the exact same value we gave it as an input. Then we give it to length, which fails because it doesn't accept ints.

There's also a shorthand syntax for providing a file's content as CMDIN to a command:

echo $(length < ./somefile.txt) # Prints: 24

For commands that only accept string inputs, the file is automatically decoded and converted to a string. Else, it's kept as a stream.

Running in background

It's possible to run multiple commands in parallel by using background commands. A background command runs, as the name suggests, in the background, and so its output isn't visible. If it fails, it won't generate any error nor affect the status() code.

To run a command in backgroud, we use the bg keyword:

bg hello = sleep 5 -x { i -> echo "Counter: $i" }

This will declare an hello variable and put an int value inside it, which is the background command's identifier (BGID). The command will be started and run in parallel of the current program. For instance, the following program:

bg hello = sleep 5 -x { i -> echo "Counter: $i" } --end { -> echo "Counter completed!" }

for i in 1..=5
  sleep 1
  echo "Loop: $i"
end

echo "Loop completed!"

Will print:

Counter: 1
Loop: 1
Counter: 2
Loop: 2
Counter: 3
Loop: 3
Counter: 4
Loop: 4
Counter: 5
Loop: 5
Counter completed!
Loop completed!

It's possible to wait for a background command by making it run back in the foreground:

bg hello = sleep 5 # The command runs in background
fg hello # The commands comes back to the foreground
         # The program pauses until it completes like for a normal command

It's also possible to let the command run even when the program finishes with detach:

bg hello = sleep 5
detach hello

Or to stop the background command with kill:

bg hello = sleep 5 -x { i -> echo "Counter: $i" }

sleep 3
kill hello
echo "Killed."

### Output:
Counter: 1
Counter: 2
Counter: 3
Killed.
<Program stops>###

Environment variables

While traditional variables are always scoped, environment variables are variables that are provided to the script when it starts, and are forwarded to all scripts the main scripts calls, recursively.

They are mostly used to share configuration between programs.

They are usually set:

  • Globally, using Central
  • For the terminal, using the terminal application's settings
  • For the current script, by providing them when running the script

The third case is the most common, here is what it looks like:

# Without environment variables
./myscript.ns

# With environment variables
with MESSAGE="Hello world!" run ./myscript.ns

The environment variable will then be provided to ./myscript.ns, and will also be available in all scripts this script calls.

Reading an environment variable

Environment variables cannot be accessed like traditional variables, they must be retrieved through the env builtin function:

# In "myscript.ns"

let message = env("MESSAGE") # any?

As the variable may not be defined, the function returns a nullable value, so we must check if the variable is indeed defined:

let message = env("MESSAGE")

if none message
  fail "MESSAGE environment variable was not provided"
end

Now we are sure that message is defined, we get an any value, because we don't know the type of the environment variable. So we must use type assertions for that:

let message = env("MESSAGE")

if none message
  fail "MESSAGE environment variable was not provided"
end

if message isnt string
  fail "MESSAGE environment variable is not a string"
end

# Here, "message" is a string

Perfect! Note that, if you want to check if the environment variable exists and is of a specific type at the same time, you can skip the first checking, which is only here to perform specific actions in case the environment variable isn't even defined:

let message = env("MESSAGE")

if message isnt string
  fail "MESSAGE environment variable was not provided or is not a string"
end

# Here, "message" is a string

Commands typing

For script files to be called as commands, they must define a main function and declare a command description.

Here is an example of command description:

cmd
  author "Me <my@email>" # Optional
  license "MIT" # Optional
  help "A program that repeats the name of a list of person"
  return void
  args
    # Declare a positional argument named 'names' with a help text
    pos "names"
      type list[string]
      help "List of names to display"
      optional

      # If this argument is omitted, the command will expect the list of names to be provided through CMDIN
      if absent
        cmdin list[string]
      end

    # Declare a dash argument named 'repeat'
    dash "repeat"
      type int
      short "r"
      long "repeat"
      optional

    # Get the time this command took to complete
    flag "duration"
      short "d"
      long "duration"

      # Conditional return type
      # 'present()' also accepts an optional argument name to check if another argument is present
      if present
        return int
      end
end

Arguments type

Arguments type can be any existing type, or:

  • anystr: accepts any type of argument except stream, which will be converted to a string when the command is called (which means the argument will be a string from the command's point of view)

Enumerations

The enum type for arguments indicate the argument only accepts a subset of values (whose type is inferred), which must be specified as a constant. This means the caller cannot use a variable as this argument's value, because the return type may depend on it.

Syntax is: enum[value1 | value2 | value3]

It may also be used as a list of custom values by wrapping the enumeration into a list[...].

Return type

The command's return type can be any existing type.

The options for each argument are:

  • type: Required, the type of the argument (nullable types are forbidden)
  • help: A help message indicating what the argument does
  • short: Short name for a dash argument
  • long: Long name for a dash argument
  • optional: Indicate the optional can be omitted (the type will be converted to a nullable one)
  • default: Make the value optional, but with a default value (so the type will not be nullable)
  • requires: Indicate one or several other arguments are required to use this dash one
  • conflicts: Indicate this dash argument cannot be used when one or several other specific arguments are already in use
  • enum: Allow only a subset of values

For dash arguments, at least short or long must be provided. Also, optional and default cannot be provided at the same time. For flag arguments, at least short or long must be provided. Also, type, optional, default and enum are not accepted.

Conditionals

Conditions can also use the elif and else keywords, and use the present() and absent() operators as well as usual relational operators like == or < for constant values like enums.

Example

Here is an example that uses all these options:

cmd
  args
    # ...
    dash "repeat"
      type int
      enum [1 | 2 | 3 | 4]
      help "How many times to repeat the names"
      short "r"
      long "repeat"
      default 1
      requires "arg1"
      conflicts "arg2" "arg3"
    end
  end
  # ...
end

The main function takes arguments with the same name as described in the cmd block, and in the same order:

fn main(names: list[string], repeat: int?)
  for i in 0..=repeat.default(1)
    echo (names.join(", "))
  end
end

The script can then be called like any command, with the default $(...) operator returning the script's return value:

./myscript.ns ["Jack", "John"] -r 1
# or
let result = $(./myscript.ns ["Jack", "John"] -r 1)

Also, know that scripts can fail too. This allows errors to be handled when the script is run as a function:

# main(names: list[string], repeat: int?) -> fallible

catch $(myscript ["Jack", "John"])
  ok  _ => echo "Everything went fine :)"
  err _ => echo "Something went wrong :("
end

Native library

The native library is a list of functions that are provided by the shell.

All types have extensions, which are functions that can be called using the . symbol, like my_list.extension().

Utilities

env(varname: string) -> any?

Get an environment variable.
Returns null if the environment variable cannot be found.

prompt(message: string) -> string

Ask the user a string.

prompt_int(message: string) -> fallible int

Ask the user an integer number. Fails if the provided input is not a number. Fails if the shell is not interactive.

prompt_float(message: string) -> fallible float

Ask the user a floating-point number. Fails if the provided input is not a floating-point number. Fails if the shell is not interactive.

confirm(message: string) -> fallible bool

Ask the user to confirm a message using an [Y/n] prompt. Fails if the shell is not interactive.

choose(options: list[string]) -> fallible int

Ask the user to pick a value from a list and get the index of the chosen value. Fails if the shell is not interactive.

retry_cmd(cmd: command, retries: int) -> fallible

Run a command and retry it a given number of times if it fails. Fails if the command still fails after all allowed tries.

exit()

Make the program exit.

last_failed() -> bool

Check if the previous command failed.
Returns 0 if no command was ran since the beginning of the script.

rand() -> float

Generate a random floating-point number between 0 and 1.

rand_int(low: int, up: int) -> fallible int

Generate a random integer between low and up. Fails if low is not strictly less than up.

rand_float(low: float, up: float) -> fallible float

Generate a floating-point number between low and up. Fails if low is not strictly less than up.

All types

any.str() -> string

Turns the provided value into a string, depending on the value's type:

_ = (true).str()    # true
_ = (3).str(3)      # 3
_ = (3.14).str()    # 3.14
_ = ('B').str()     # B
_ = ("Yoh").str()   # Yoh
_ = @{ command --arg1 -c 2 -d=4 }.str() # command --arg1 -c 2 -d 4

_ = ["a","b"].str() # [ "a", "b" ]
_ = @{ streamify "Hello world!" }.str() # "<stream>"

Nullable types

T?.isNull() -> T

Check if the value is null.

let a: int? = 1
let b: int? = null

echo ${a.isNull()} # false
echo ${b.isNull()} # true

T?.default(fallback: T) -> T

Use a fallback value in case of null:

let a: int? = 1
let b: int? = null

echo ${a.default(3)} # 1
echo ${b.default(3)}

T?.unwrap() -> T

Make the program exit with an error message if the value is null.

let a = ?(0) # int?
let b = a.unwrap() # int

T?.expect(message: string) -> T

Make the program exit with a custom error message if 'a' is null

let a = ?(0) # INt?
let b = a.expect("'a' should not be null :(") # int

Characters

char.single() -> bool

Indicate if a character is made of a single codepoint.

char.codepoints() -> list[int]

Get the codepoints composing a character.

char.len() -> int

Get the number of codepoints composing a character.

char.bytes() -> int

Get the size of a character, in bytes.

Strings

string.chars() -> list[char]

Get the characters composing a string.

string.codepoints() -> list[int]

Get the codepoints composing a string.

string.len() -> int

Get the number of codepoints composing a string.

string.bytes() -> int

Get the size of a string, in bytes.

string.parse_int(base = 10) -> fallible int

Tries to parse the provided string as a number in the provided base. Leading zeroes are accepted. 0 symbol followed by b, o, d or x is accepted for bases 2, 8, 10 and 16 respectively. Fails if the string does not represent a number in this base.

_ = ("2").parse_int()   # 2
_ = ("A").parse_int()   # <FAIL>
_ = ("A").parse_int(16) # 11

string.parse_float(base = 10) -> fallible float

Identical to string.parse_int(base) but with floats.

string.upper_case() -> string

Convert a string to uppercase.

_ = ("aBc").upper_case() # "ABC"

string.lower_case() -> string

Convert a string to lowercase.

_ = ("aBc").lower_case() # "abc"

string.reverse() -> string

Reverse a string.

_ = ("abc").reverse() # "cba"

string.concat(right: string) -> string

Concatenate two strings (equivalent to "$left$right").

_ = ("a").concat("b") # "ab"

string.split(str: string, sep: string) -> string

Split a string into a list.

split("ab", "")  # [ "a", "b" ]
split("a b", " ") # [ "a", "b" ]

Lists

list[char].stringify() -> str

Turns a list of characters to a string.

_ = ([ 'a', 'b', 'c' ].str() == "abc") # true

list[string].join(sep = ",") -> string

Join a list of strings with a separator.

join([ "a", "b" ])       # "a,b"
join([ "a", "b" ], "; ") # "a; b"

list[T].get(index: int) -> T?

Try to get an item from the list, without panicking if the index is out-of-bounds.

let names = [ "Jack", "John" ]

names.get(0) # "Jack"
names.get(1) # "John"
names.get(2) # null

list[T].expect(index: number, message: string) -> T

Get an item from the list, and panic with a custom error message if the index is out-of-bounds.

let names = [ "Jack", "John" ]

names.expect(2, "Third item was not found")

list[T].unshift(value: T)

Insert a new value at the beginning of the list.

list[T].push(value: T)

Push a new value at the end of the list.

list[T].unshift() -> T?

Remove the first value from the list and return it.

list[T].pop() -> T?

Remove the last value from the list and return it.

list[T].sort(asc = true) -> list[T]

Sorts a list.

_ = [ 2, 8, 4 ].sort()      # [ 2, 4, 8 ]
_ = [ 2, 8, 4 ].sort(false) # [ 8, 4, 2 ]

list[T].reverse() -> list[T]

Reverse a list.

_ = [ 2, 8, 4 ].reverse() # [ 4, 8, 2 ]

list[T].len() -> int

Count the number of entries in a list.

_ = [ 2, 8, 4 ].len() # 3

list[T].concat(another: list[T]) -> list[T]

Concatenate two lists.

_ = [ 1, 2 ].concat([ 3, 4 ]) # [ 1, 2, 3, 4 ]

list[T].concat(lists: list[list[T]]) -> list[T]

Concatenate multiple lists.

_ = [ 1, 2 ].concat([ [ 3, 4 ], [ 5, 6 ] ]) # [ 1, 2, 3, 4, 5, 6 ]

Maps

map[K, V].has(key: K) -> bool

Check if a key exists in a map.

map[K, V].get(key: K) -> V?

Get a value without panicking if the key doesn't exist.

map[K, V].keys() -> list[K]

Get all keys of a map.

map[K, V].values() -> list[V]

Get all values of a map.

map[K, V].values() -> list[struct { key: K, value: V }]

Turn a map into a list.

map[K, V].expect(key: K, message: string) -> V

Get an item from the map, and panic with a custom error message if the key doen't exist.

map[K, V].delete(key: K) -> bool

Delete a key, returns false if the key didn't exist in the map, or true otherwise.

map[K, V].count() -> int

Count the number of entries in a map.

Commands

command.run() -> int

Run the command and gets its status code after exit.

command.fallible()

Run the command and fail if the status code after exit is not 0.
Equivalent to calling the command with simple parenthesis like cmd().

command.ret_str() -> string

Run the command and get its stringified return value.

command.cmdraw() -> stream

Run the command and get its CMDRAW output.

command.cmdmsg() -> list[string]

Run the command and get its CMDMSG output.

command.cmderr() -> list[string]

Run the command and get its CMDERR output.

command.output() -> list[string]

Run the command and get its CMDOUT and CMDERR outputs combined.

Streams

stream.pending() -> bool

Check if the stream is still pending. If the pipe is complete (which means if its pipe is closed), false will be returned.

stream.size_hint() -> int?

Get the stream's size hint. If no size hint was provided for this stream, null will be returned.

Examples

Guess The Number

while true
  let won = false
  let secret = rand_int(0, 100)

  echo "Secret number between 0 and 100 has been chosen."

  while !won
    let user_input = retry prompt_int("Please input your guess: ")

    if user_input < secret
      echo "It's higher!"
    elif user_input > secret
      echo "It's lower!"
    else
      echo "You guessed the number \\o/!"
      won = true
    end
  end

  tries = retry(5) confirm("Do you want to play again?")

  if !(retry(5) confirm("Do you want to play again?"))
    echo "Goodbye!"
    break
  end
end

Native commands

The native commands are commands that are available in every program without import.

echo: display a value

Display a value to CMDOUT.

# echo [-n] { <anystr> | -b <stream>}
#
# "-b": print a stream as UTF-8
# "-n": don't put a newline break at the end of the content

echo "Hello world"
echo -n "Hello world!"
echo -b $(streamify "Hello world!")

wt: write a file

Write a content to a file.

# wt [-a] [-n] [-c] <path> <anystr>
#
# "-a": append to the end of the file
# "-n": don't append a newline symbol at the end of the content
# "-c": fail if the file doesn't exist instead of creating it

Note that sometimes it can be clearer to use a redirection pipe:

# Overwrite file
echo "Hello world!" > ./test.txt

# Append to file
echo "Hello world!" >> ./test.txt

rd: read a file

Read a file as a string value.

# rd [-s] [--stream-size <int>] <path>
#
# "-s": read a stream instead of reading the full content
# "--stream-size": specify the size of the stream when "-s" is provided (rounded to higher pipe buffer multiplier)

mkdir: create a directory

Create a new directory.

# mkd [-s] <path>
#
# "-s": fail if the directory already exists

ren: rename a filesystem item

Rename a filesystem item.

# re [-m] <oldname: path> <newname: path>
#
# "-m": move if the 'newname' is located inside another directory

mv: move a filesystem item

# mv [-n] <file: path> <dest_dir: path>
#
# "-n": create the target directory if it does not exist

rm: remove a filesystem item

Remove a filesystem item.

# rm [-r | --recursive] [-n | --non-empty] [-t | --trash] {<path> | -l <list[path]>}
#
# "-r": allow removing empty directory
# "-n": allow removing even non-empty directories
# "-l": remove a list of paths
# "-t": move the item to the user's trash

ls: list filesystem items

List filesystem items.

# ls [-t | --tree] [-r | --recursive] [-h | --hidden] [--file-only | --dir-only] [<path>]
#
# "-t": display as a tree (implies "-r")
# "-r": list recursively
# "-h": list hidden files
# "--file-only": only display files
# "--dir-only": only display directories

fd: find filesystem items

Find filesystem items matching provided criterias.

# fd [-t | --types enum["dir" | "file" | "symlink" | "device"]]
#    [-a | --absolute]
#    [-L | --follow]
#    [-e | --extension <string>]
#    [-n | --name <string>]
#    [-r | --regex <string>]
#    [-i | --ignore-case]
#    [-x | --exec <command>] <path>
#
# "-t": only list items of a given type (by default, only files and symlinks are shown)
# "-a": list absolute file paths instead of relative ones
# "-L": follow symbolic links
# "-e": only list files with the provided extension (directory will be excluded)
# "-n": only list items whose name contain the provided string (`^` and `$` can be used for indicating items must start or end by it)
# "-r": only list items matching a provided POSIX regex
# "-i": ignore case (requires "-n" or "-r")
# "-x": execute a command for each found item

Manage symbolic links.

# sl {-r | --read} <path>: check a symlink's target
#
# sl [-u | --update] <srcpath> <targetpath>: create a symlink
#
# "-u": update the symlink to the new target path if it already exists instead of failing

Crash Saves

Crash saves is a major feature of NightOS. It consists in collecting the state of running applications and saving it for later restore in case of application or system crash.

Crash saves is handled through the sys::crashsave service.

Applications support

To support crash saves, applications must opt-in through an option in their manifest.

Crash saves collection

Every once in a while (once per minute by default), a CRASH_SAVE_COLLECTION notification is sent to the application, which is expected to write a state it will be able to restore later on in a temporary file provided by the sys::crashsave service.

Crash saves restoration

If a crash save must be restored (occurs after an application is opened for the first time following a detected crash), a RESTORE_CRASH_SAVE signal is sent to the application with a temporary file containing the state it previously wrote in, as well as a few other informations.

The application can then decide if the crash save file must be removed or not.

Crashes detection

Invalid shutdown indicator

When the system starts up, it creates an empty file in /etc/sys/awake. When the system shutdowns gracefully, this file is removed.

If, during startup, this file already exists, the system hasn't shutdowned gracefully. When this happens, a dialog message is shown, suggesting to re-open crashed applications with their last crash save.

For each application that do not have an available crash save (e.g. when the system crashed before a crash save could have been collected for this application), an indicator in the dialog box will show this application cannot be restored.

Application's crash indicator

When an application starts, the system creates an empty file in /home/[user]/appdata/[appname]/crashsaves/awake for user applications (even for global applications).

When the application exits gracefully, the file is removed automatically. When the system detects the application exited, if this file still exists, it shows a dialog box asking if the user wants to relaunch the application with the last crash save.

NOTE: If there is no available crash save (e.g. when the application crashed before a crash save could have been made), the dialog box will simply show the application crashed.

NOTE: Because a crash save could have been collected for several instances of an application, they can all be restored afterwise.

Restoration process

When a crash save is attempted to be restored, the sys::crashsave service. When the application is ready, a SYS_CRASHSAVE_RESTORE confirmable signal is sent, with the application's crash save. The application is expected to confirm the signal when it has finished restoring its state using the crash save. The crash save is not deleted directly, though. It is renamed using the new instance's PID and kept until the next collect process receives a new crash save for this instance.

If the application crashes before another crash save can be made and didn't confirm the restoration signal, when this process happens again, system will indicate the application appear to have crashed during crash save restoration.

Services

This document describes the architecture of services, as well as the list of all system services and their main features.

Types of services

An application can expose four types of service:

  • A main service, which is used as the default one
  • Scoped services, which are used as interfaces for conventional tasks between applications which are not specific to the current one
  • Integration services, which are used to interact with the system in specific ways (see the complete list)
  • Driver services, which are used to interact with hardware devices through the sys::hw service.

A scope's name is made of up to 8 extended ASCII characters.

Applications exposing integration services get special recognition for the tasks associated to these services. These services are not available directly to the end applications ; they can only be used through system services.

Architecture of a service

Applications have exactly one running process per service, and each service has exactly one running instance per active user, to prevent a user to connect to the service of a user with more privileges than itself. System services, on their side, only have one global instance.

An application process can tell if it was started as a service or not by looking at its execution context.

Once a service process indicates it's ready using the READY syscall, other processes will be able to connect to this service.

Connections

A connection essentially consists of a service socket between a service and another process called its client.

When a process wants to connect to a service, it uses the CONNECT_SERVICE to send a connection request to this service.

The service process then receives the request through the SERVICE_CONN_REQUEST.

It is expected to answer under a short delay specified in the registry's system.processes.service_answer_delay key (2000 ms by default).

If the service refuses the connection, it provides its answer through the REJECT_SERVICE_CONN syscall, and the procedure stops here.

Else, it accepts it through the ACCEPT_SERVICE_CONN syscall. A new thread is then created in the service's process, and the signal returns with a code indicating if the current thread is the one that was just created.

Communication

The service and its client(s) communicate through the service socket created during the connection.

The different message formats a client can send to a service are a called the service's methods, while the different message formats the service may send to a client are called the service's notifications.

Thread types

A service process' main thread is called its dispatcher threads, while threads created using the ACCEPT_SERVICE_CONN syscall are called client threads.

Closing a connection

The service cannot terminate a connection by itself. If a client thread terminates brutally, the SERVICE_SERVER_QUITTED signal will be sent to its client.

Only clients can properly close a connection to a service, using the END_SERVICE_CONN syscall. The service socket is then immediatly closed (on both the client and the thread's side).

The service then receives the SERVICE_CLIENT_QUITTED signal.

If the client terminates brutally (before the connection was properly ended), the client thread receives the SERVICE_CLIENT_CLOSED signal.

When a client thread receives on these two signals, it is expected to end as soon as possible (though there is no time limit).

System services

System services are run by the system and get access to special system calls and signals that aren't available otherwise.

They also have a specific process identifier (PID) to identify them easily.

You can find the list of all system services in the related directory.

Third-party communication

An application can enable communication to and from other applications using a service.

For instance, let's consider two applications, A and B. If A wants to be able to be contacted by other applications (such as B), it sets up a service and subscribes to it. Then, when B sends a message to the service, it is forwarded to A through a notification by the service itself.

It can also work the other way: B subscribes to the service, and when A wants to contact B, it simply sends a message to B.

The downside of this structure is that latency is approximately doubled compared to a direct A-B communication. But as they use messages and notifications, which are based on hardware interrupts, the latency is extremely low, mainly composed of the time the kernel spends to ensure the communication between A and the service and then the service and B (or the other way around) is allowed.

Although this method works fine for small pieces of data, it isn't suited for transmitting large chunks in a performant way. For that purpose, A can share a memory segment with the service, who shares it in turn to B (with restricted permissions if required).

Note that this only applies to different applications ; processes from a same application can communicate directly through IPC.

Translations

Translations are a complex topic and can be handled in several ways.

Applications' and libraries' content can either be translated using a custom way (e.g. a translation framework), or by using the system's builtin translation library, i18n.

Translation sets

Translation sets are a way to specify how to translate a set of content.

They use an extremely compact and efficient format to minimize disk usage and maximize performances. Some trade-offs are made to accelerate performance at the expansve of file size.

They are not made to be modified directly, and should instead be derived from other text formats if the translation sets need to be manually edited by a person.

Header

The header is made of the following informations:

  • Number of translatable languages (2 bytes)
  • For each language:
    • Language's code in ASCII (3 bytes, using the ISO 639-2 format)
    • Location of the language's translation table in the file (8 bytes)
  • Code of the reference language(3 bytes)
  • Number of translation models (8 bytes)
  • For each translation model:

The reference language is the one the translations are made from. For instance, a translation set may be procuced by translating english models to french. But when the set is bundled, both english and french will actually be written in their own translation tables, meaning we can't easily reverse the process. The reference language is here to ensure that we can automatically determine which language was used to create the translations. It can be also be used in debug messages in case something goes wrong.

Translation models

A translation model is a translatable string. They can use variables which are provided at runtime.

It only consists in a variable set declaration.

Translation models contain no text as that is the role of the translation strings, which means that decoding a translation set will not allow to directly see what the source text is, as there is in fact no text in the models.

Most translation tools provide a way to start from a source language (like english) and then translate to other languages, but when the file is actually bundled, the source language is just written as any other inside its own translation table.

Variable set declaration

A variable set declaration of made of the following:

  • Number of variables in the set (2 bytes)
  • For each variable:
    • Unique identifier of the variable (4 bytes)
    • Variable type (1 byte):
      • 0x00: delimited string (string)
      • 0x01: boolean (bool)
      • 0x02: 64-bit signed integer (int)
      • 0x03: 64-bit unsigned integer (uint)
      • 0x04: 64-bit floating-pointer number (float)

Translation tables

A translation table represents the translation of each defined model for a provided language.

It consists in the following:

Translation strings must be in the same order as defined in the header.

Translation strings

A translation string is the dynamic translation, for a given language, of a translation model.

They use a complex format as they permit to achieve both compacity in term of file size, high performance when decoding and actually performing the translations, as well as allowing complex translations. It is for instance possible to link together multiple conditionals to only show a text if a set of provided numbers meet a requirement.

Translation strings use the following structure:

  • Length of the translation string, in bytes (4 bytes)
  • Assignable variables set
  • Number of direct translation segments (4 bytes)
  • Relative address of the dynamic library (4 bytes)
  • For each translation segment:
  • Number of optional translation segments in the dynamic library (4 bytes)
  • For each optional translation segments:

Translation segments

  • Length of the segment in bytes (4 bytes)
  • Segment type (1 byte):
    • 0x00: fixed text
    • 0x01: toggle
      • Followed by the identifier of a boolean variable (4 bytes)
      • Followed by the ID of the optional translation segment to use if the variable is truthy
      • Followed by the ID of the optional translation segment to use if the variable is falsy
    • 0x02: comparison
      • Followed by the ID of the variable to compare (must be of a number type)
      • Comparator (1 byte):
        • 0x01: equal
        • 0x02: not equal
        • 0x03: greater than
        • 0x04: less than
        • 0x05: greater than or equal to
        • 0x06: less than or equal to
      • Comparison object (1 byte):
        • 0x01: direct value
          • Followed by the raw number (must be of the same number type)
        • 0x02: variable
          • Followed by the ID of the variable to compare it to (must be of the same number type)
      • Followed by the ID of the optional translation segment to use if the comparison is truthy
      • Followed by the ID of the optional translation segment to use if the comparison is falsy
    • 0x03: property checking
      • Followed by the ID of the variable to check
      • Negation (1 byte)
        • 0x00: perform the check normally
        • 0x01: revert the check's result
      • Check (1 byte):
        • 0x01: (string-only): string is empty
        • 0x02: (string-only): string only contains ASCII characters
        • 0x20: (int-only): number is positive
        • 0x21: (int-only): number could be converted to a uint without loss
        • 0x30: (uint-only): number could be converted to an int without loss
        • 0x31: (uint-only): if the number was signed as an int, it would be negative
        • 0x40: (float-only): number has a non-zero decimal part
        • 0x41: (float-only): number could be converted to an int without loss
        • 0x42: (float-only): number could be converted to a uint without loss
      • Followed by the ID of the optional translation segment to use if the variable is truthy
      • Followed by the ID of the optional translation segment to use if the variable is falsy
    • 0x04: assignments
      • Identifier of the assignable variable (4 bytes)
      • Action type (1 byte):
        • 0x00: Raw assign
          • Followed by the value to assign (must be of the exact same type)
          • Followed by the identifier of a variable (4 bytes)
        • 0x01: (bool-only) Result of a comparison
          • Followed by a comparison's data (type, comparator, ...)
        • 0x02: (bool-only) Result of a property checking
          • Followed by the check's data (negation, check, ...)
        • 0x10: (uint-only): length of a string, in bytes
          • Followed by the identifier of a string variable
    • 0xFF: empty (used for conditionals)

Kernel

NightOS' kernel is named Cosmos. It is a micro-kernel which tries to be as simple and straightforward as possible, delegating all non-trivial tasks such as filesystem access or permissions management to services.

NOTE: This document is in its very early stages, and so is far from being complete. Major changes may and will be made to related documents.

Documents

Hardware management

This document describes how the kernel interacts with hardware.

Hardware detection

Devices are detected during the boot process and then periodically after startup. This allows to hotplug some additional devices afterwards.

As all devices do not use the same connection protocols, the detection process depends on the connection:

  • PCI-Express devices are detected through their Configuration Space
  • IDE/SATA devices are detected through the IDE/SATA controller
  • USB devices are enumerated through the USB protocol stack
  • Motherboard-connected devices are enumerated through the BIOS/UEFI (e.g. CPU and case fans)

Some devices may not be detected through these though, such as some legacy ISA devices, which will be detected through a set of methods like ACPI enumeration or simply checking UART serial ports.

Connection interface identifier

The connection interface identifier (CII) is a 4-byte number describing what a device is connected to:

  • Connection type (1 byte):
    • 0x01: PCI-Express
    • 0x02: IDE
    • 0x03: SATA
    • 0x04: M.2
    • 0x05: USB
    • 0x06: RGB
    • 0x07: Fans
  • Bus number (1 byte)
  • Port number (2 bytes)

For instance, the seventh USB port on the second bus will have the 0x05010006 CII.

Connection-specific device descriptor

All hardware devices expose a normalized identifier whose format depends on the connection type (PCI-Express, SATA, ...). This identifier is called the connection-specific device descriptor (CSDD).

Its size can vary up to 256 bytes.

Kernel device identifier

The kernel device identifier is an 8-byte identifier computed from the CII and the CSDD. It is unique across all devices, consistent across reboots, and identical from one computer to another for the same device. It is only meant for internal use by the sys::hw service, which generates an UDI for external use.

Raw device descriptor

The raw device descriptor (RDD) is a data structure (up to 260 bytes) made of the followings:

  • KDI (8 bytes)
  • CII (4 bytes)
  • Size of the CSDD, in bytes (1 byte)
  • CSDD (up to 256 bytes)

This descriptor is then used by the sys::hw service to expose the device to the rest of the operating system.

Input/output ports

The kernel first detects the I/O ports used by each device, and maps it to the device's KDI.

They are uni-directionals and as such are split in two categories: input ports and output ports.

Input ports are used by devices to transmit informations to the kernel. When this happens, the data is put on a stack, and drivers can then retrieve it using the READ_IO_PORT system call.

Output ports are used by drivers to transmit informations to devices, using the WRITE_IO_PORT system call.

I/O ports identifiers

I/O ports are accessed using relative identifiers, which corresponds to the nth port of the physical device, meaning it starts to 0 and ends to <number of ports in the device> - 1. The kernel transparently translates the relative identifier to the real port number.

The association with the real port number is transparently performed by the kernel.

Drivers

A driver is a process which can directly communicate with hardware devices using the dedicated system calls.

They communicate with external processes through the sys::hw system service, or sys::fs when direct driver access is possible.

You can find more about how drivers work in this section.

Direct communication with device hardwares is made through system calls.

The sys::hw service is considered as a driver for all devices.

Kernel communication

The kernel does not identify devices, as this task is relegated to the sys::hw service. In order to communicate with hardware devices, only the CII is provided.

Memory

This document describes how the kernel organizes and manages the memory.

WARNING: This document is only a short draft and is missing a LOT of informations on how the memory will be structured and managed.

Structure

Every running code that is not part of the kernel itself doesn't have access to the physical memory. Instead, every process has a virtual 64-bit address space split into pages.

Pages

A page can either be small (usually 4 KB) or large (usually 2 or 4 MB), the exact sizes depending on the CPU architecture.

Pages can be allocated through the dedicated syscall syscall, or mapped using abstract memory segments.

Abstract memory segments

An abstract memory segment (AMS) is an identifier which refers to a segment of memory which doesn't actually exist. To be used, they must be mapped in a process' memory to be accessed like regular memory. The kernel then intercepts all memory accesses to these mappings and handle them, depending on their nature which cover three cases:

An AMS can then be mapped at multiple places in a process' memory, or shared with other processes. The kernel handles mappings to get optimal performances and reduce the number of memory accesses as much as possible.

Addresses randomization

Allocations happen at random addresses using address space layout randomization (ASLR).

Kernel memory's randomization using processes like KASLR or KARL are currently being considered.

Write-or-exec

Memory implements a W^X model where memory can be writable or executable, but not both. This way, an attacker cannot write arbitrary instructions in memory and then execute them.

Processes

Apart from the kernel itself, all programs run in processes.

Why processes

Separating programs in threads presents several advantages:

  1. This allows to take advantage of multi-core architectures by running multiple programs in parallel
  2. Each program has its own data space and does not share its data with other programs
  3. Each process has its own permissions and thus cannot bypass what the user chosen

Types of processes

There are two types of processes:

This implies it is not possible to run an executable as standalone in its own process.

Process identifier

Each process is identified by a 64-bit number, which is guaranteed to be unique until the system restarts. When a process terminates, the PID is not freed. The PID is not randomly generated, but retrieved from a system-wide counter incremented when a new process starts. PID cannot be equal to 0.

System services can be identified by a PID strictly smaller than 100, while userland processes have a PID greater or equal to 100.

Process attributes

Each process has a set of attributes which contains critical informations on it (lists are usually PLL):

  • PID (8 bytes)
  • Priority (1 byte)
  • System marker (1 byte) to indicate if the process comes from a system program
  • Container's ID (8 bytes)
  • Running user's ID (8 bytes)
  • Parent application ANID (8 bytes) - 0 for system services
  • Pointer to the execution context (8 bytes) - 0 for system services
  • PLL(e=32) for memory mappings
  • PLL(e=32) for raw permissions
  • PLL(e=32) for drivable devices

Drivable devices

The drivable devices attribute contains the list of all devices' KDI the current process can drive.

The goal of this attribute is to determine if the process is allowed to map a device's memory by creating an AMS from it using the DEVICE_AMS syscall, as well as using DMA-related instructions in the CPU.

This attribute is managed by the sys::hw service and can only be updated by it.

Raw permissions

Raw permissions are used by system services to determine the permissions of a process without sending a message to the sys::perm service and waiting for its answer, which would be costly in terms of performance.

These permissions use a specific structure, specified in the related service's specifications document.

Permissions can be managed using the sys::perm service.

Scheduling

This document presents how tasks are scheduled internally.

Performance balancing

Each process has a priority number, between 1 and 20, which indicates how much its performances must be prioritized compare to other processes.

Scheduling is achieved in a preemptive way, to prevent processes from taking too much CPU time.

The basics can be found here.

More specifically, the higher the priority of a process is, the faster it will run. Here are the priority-dependant aspects of a process:

  • Number of instructions run per cycle
  • Priority when accessing I/O through services

Comparatively, when a process has a high priority, other processes will run a tad slower.

Cycles and context switching

To run processes, the kernel simply iterates over the list of existing processes, and allow them to run a given number of instructions. Then, the control is taken back by the kernel which runs the next process, and so goes on.

This happens as follows:

  1. A process is selected
  2. If the process is currently suspended, it is ignored
  3. Its registers are restored by the kernel (if any)
  4. The process runs a given number of instructions
  5. The kernel takes back the control of the CPU
  6. The process' registers are saved
  7. Go to step 1

These steps are known as a cycle.

Automatic priority attribution

Processes' priority is automatically adjusted by the kernel, unless it is manually assigned through the SYS_PROCESS_ATTRIBUTES syscall.

The priority is determined based on multiple factors:

  • Does the process have a fullscreen window?
  • Does the process owns the active window?
  • Does the process owns a visible window?
  • Is the process a driver or service? If so, how much is it used?

Data structures

This documents describes the structures used by the kernel to represent the data it uses in memory.

All contiguous data structures are made such as it is easy to delimit them by computing their size in the memory.

Booleans

A boolean is a single byte value that's either 0x01 for true or 0x02 for false.

Timestamps

Timestamps are stored as milliseconds, starting from January 1st, 1970. This is to guarantee interopability with existing algorithms using Unix's EPOCH constant.

They are represented as an 8-byte unsigned integer number.

Delimited lists

Delimited lists are made of:

  • Their length in bytes (8 bytes)
  • Their content

Each element must be delimited.

Delimited strings

Delimited strings are made of:

  • Their length in bytes (8 bytes)
  • Their content (UTF-8 encoded)

Buffer pointers

A buffer pointer refers to a buffer that is either readable, meaning its creator process has read permission on its entire memory location, and/or writable, meaning its creator process has write permission on its entire memory location.

It is made of:

  • The memory address of the buffer (8 bytes)
  • The buffer's length (8 bytes)

Options

An option is a data structure that may contain a specific data type or nothing.

It is made of a variance byte, set to 0x01 followed by the data if any, or a single 0x00 byte to indicate no data is present.

Fallible results

A fallible result is a data structure describing a fallible operation's result. It starts by a variance byte to indicate its result type:

  • Either 0x00 to indicate the operation was successfully, followed by the success data
  • Or 0x01 to indicate something went wrong, followed by the error data

Bitmap images

Bitmap images are represented as a header and a pixel list.

The header is composed as a suite of 8 bytes:

  • Image width (IW), in pixels (2 bytes)
  • Image height (IH), in pixels (2 bytes)
  • Number of colors (power of 256) for the red channel (NR), 0 if unused (1 byte)
  • Number of colors (power of 256) for the green channel (NG), 0 if unused (1 byte)
  • Number of colors (power of 256) for the blue channel (NB), 0 if unused (1 byte)
  • Number of colors (power of 256) for the alpha channel (NA), 0 if unused (1 byte)

The pixel list is made of the data for each pixel, contiguously.

Each pixel is encoded as follows:

  • Value for the red channel (NR bytes)
  • Value for the green channel (NG bytes)
  • Value for the blue channel (NB bytes)
  • Value for the alpha channel (NA bytes)

As shown above, if the number of colors is set to 0 for a specific channel in the header, it must not be contained in the pixel's content. The number of bytes used for each channel of a pixel is equal to the number provided for this specific channel in the header, allowing for 256 power <number in the header> different colors.

Pixels are listed from the top left corner of the image to the bottom right corner. They are always square.

The size of pixel list can be calculated as IW * IH * (NR + NG + NB + NA) bytes. Add another 8 bytes for the header.

Bitmap videos

Bitmap videos are represented as a header and a frame list.

The header is composed as a suite of 16 bytes:

  • Frames width (IW), in pixels (2 bytes)
  • Frames height (IH), in pixels (2 bytes)
  • Number of frames per second (1 byte)
  • Number of frames (4 bytes)
  • Number of colors (power of 256) for the red channel (NR), 0 if unused (1 byte)
  • Number of colors (power of 256) for the green channel (NG), 0 if unused (1 byte)
  • Number of colors (power of 256) for the blue channel (NB), 0 if unused (1 byte)
  • Number of colors (power of 256) for the alpha channel (NA), 0 if unused (1 byte)
  • Future-proof (3 bytes)

Each frame is a bitmap image without the header.

All frames will inherit the informations stored in the video's header (width, height, color channels).

Packed linked lists

Packed linked lists (PLL) are linked lists used for items whose size is both small (usually <= 32 bytes) and fixed for all items.

It uses a system of same-size entries, each containing a micro bump allocator.

The goal of a PLL is to provide a blazing fast read and iteration speed, while compromising on insertion and deletion speeds.

Caracteristics

A PLL is caracterized by its item size (in bytes), number of items per entry (NIE, up to 255) and its length (the number of items in the list), which is the number of items which can be stored per entry. It is noted PLL(e=<number of items per entry>[, s=<item size in bytes>][, l=<length>]).

Structure in memory

Each entry is a contiguous suite of bytes which can store up to NIE items contiguously. It starts by either a pointer to the next entry (on 8 bytes), or the number of items actually initialized in the current entry (pre-filled with zeros to be stored on 8 bytes).

For instance, let's take a PLL(e=3, s=2, l=4). Its content is the four following items:

  • 0xDEADBEEF
  • 0x01234567
  • 0x89ABCDEF
  • 0xBEEFDEAD

If the first entry is located at address 0x00001000 and the second at address 0x00002000, here is the PLL's representation in memory with big-endian representation (with _ representing garbade data):

0x0000000000001000: 00 00 20 00 DE AD BE EF 01 23 45 67 89 AB CD EF
0x0000000000002000: 00 00 00 01 BE EF DE AD __ __ __ __ __ __ __ __

As you can see, the first entry contains the address to the next entry, followed by the content of the first three items (contiguously).
The second entry is the last one and so simply contains the number of initialized items, followed by the last item's content, and then garbage as this memory zone is not initialized yet.

Checking an entry's type

To check if the first byte of an entry is the next entry's address or the number of initialized items, we simply have to perform a simple comparison: if the byte is greater than 0xFF (255 in decimal, which is the maximum allowed number of items per entry), then it's an address, else it's a number of initialized elements.

The ratio

A PLL also has a ratio, which is the number of bytes reserved for items in each entry, divided by the total number of bytes. So, in our example, 3 items per entry * 4 bytes = 12 bytes, while the total number of bytes per entry also takes into account the address itself, so 12 bytes + 8 bytes = 20 : our PLL's ratio is 0.6.

This is quite a low ratio, meaning we waste a lot of space. The ratio must be kept as near as 1.0 as possible, while maintaining a reasonable memory footprint for each entry.

Performances bottlenecks

A thing to keep in mind is that PLL have a considerable performances bottleneck: when an entry is filled, the next must be allocated with a size that's a lot larger than a single item's size. That's why, when an entry is full, it should be allocated on the moment time is the less critical.

Performances are especially bad when inserting new items in the list or when removing ones, as many data needs to be moved around.

Regarding updating elements, this requires to write both the item itself. Inserting elements in a not-yet full entry requires to write the element itself as well as incrementing the entry's counter.

Performance advantages

As you can see in the above bottlenecks, PLL are not meant to be performant on writings. They are meant to be fast for reads, especially sequential reads. The higher the NIE of the list, the fastest it will be to find an element in it, or to read every element of the list one by one (iteration).

Also, as the counter is packed with the other data of each entry, it presents a reduced risk of cache miss.

Computing the length is reasonably fast.

Length-first variant

There is a variant of the PLL that stores the total length of the list somewhere in the memory. In that case, the first byte of the last entry does not need to store the number of items in the entry.

This variant is interesting being we can instantly get the number of items in the list, but the downside is that reading sequentially the list will also incur an additional reading to know where the last entry ends, and updating the total length incur a high risk of cache fault as the chances of the memory area where the length is as well as the entry itself be in the cache at the same time are pretty low.

The length-first variant should only be used when accessing the length instantly is critical.

Tweaking the NIE

Increasing the NIE will:

  • Reduce the needs of allocating
  • Speed up iteration times
  • Speed up insertion times when the last entry isn't occupied

But also:

  • Increase the memory cost of the last entry when it isn't occupied
  • Slow down insert times when the last entry is occupied (need to allocate)

Unions

Unions allow to construct multiple types of data in a single one.

It is made of a type ID (1 byte) followed by the data.

For instance, if we want to store either a string or a list, we can associate type ID 0 to the former and type ID 1 to the latter, then append the actual string or list to it.

Enumerations

Enumerations allow to switch between behaviours using a unified data type.

It is made of a variant ID (1 byte) followed by the data (which can be of any fixed type).

Kernel-Process Communication

The Kernel-Process Communication (KPC) describes how a process can interact with the kernel, and vice-versa.

There are two types of KPC:

  • System calls, which are used by a process to ask the kernel to perform an action ;
  • Signals, which are used by the kernel to send informations about an event to a process

Note that, unlike many operating systems like Linux, it's not possible for a process to send a signal to another. Only the kernel is allowed to emit signals.

For more advanced features, like permissions management or filesystem, check IPC.

Inter-Process Communication

This document describes the way Inter-Process Communication (IPC) works.

Pipes

Inter-process Uni-directional Channels (IUC), also called pipes, allow two distinct processes to communicate.

Pipes are made of two uni-directional parts:

  • A sender channel (SC) which can only be written to (write-only)
  • A receiver channel (RC), which can only be read from (read-only)

The two processes sharing a pipes are:

  • The sender process, which uses the SC to send data to the other process
  • The receiver process, which uses the RC to retrieve data sent by the other process

The process that creates the pipe gets both the SC and the RC, and is expected to provide one of them to another process.

Each SC and RC has a unique identifier, which is binded to the process that created it.
The process receiving an SC or RC receives another identifier for it, unique to that process, which prevents IDs collisions.

Opening pipes

A process can open a pipe with another process using the OPEN_PIPE syscall.

The other process will then receive the RECV_PIPE signal. If no handler is set when the signal is sent, the opening syscall fails.

Pipes' pending data

When a pipe is written to, the data is written to a memory zone. This zone is called the pipe's buffer and it's content is called the pending data.
When a pipe is read from, the pending data is progressively retrieved, erased as the read progresses.

The default size of the pipe's buffer is 64 KB, but this can be extended to up to 16 MB during its creation. When it is reached, no data can be written to the pipe anymore, meaning the other process must read data from it in order to free space to write it.

Pipes locking

When a pipe is being written to or read from, it is locked, which means no other writing or reading can happen during this time. This prevents data races which are a common source of bugs which are complex to debug, while not compromising performances.

Closing pipes

Any of the two processes (be it the receiver or the sender) can close a pipe using the CLOSE_PIPE syscall, providing its SC or RC identifier. The pipe is immediatly closed on both sides, and the other process receives the PIPE_CLOSED signal.

Message pipes

Pipes are designed to transmit streams of data, but sometimes we need to use them to transmit messages. This is why there are specific syscalls and signals to deal with this problem.

A message pipe is a pipe that only sends and receives dynamic-sized messages instead of a stream of bytes.
They have a maximum length of 64 KB, which is the pipes' buffer's minimal capacity. Messages must always be sent at once and cannot be sent partially.
Their length is determined when the message is sent which, coupled to pipes locking, allows to retrieve complete messages directly.

It's not possible to send "non-message" data through a message pipe, as the action of writing to a pipe will automatically check if it's a message pipe and ensure the size and "send at once" requirement are met.

Interactive usage

You can find more informations on interactive usage in the shell specifications.

Service sockets

The downside of message pipes is that they are not designed to handle responses from the receiver process, and they also don't have built-in errors handling.

To solve this, it's possible to use service sockets, which as their name indicate are mainly used to communicate with services. A socket is caracterized by the process opening it, called the service, and the process receiving it, the client.

Opening

Sockets are opened with the OPEN_SERV_SOCK syscall. The other process receives the RECV_SERV_SOCK signal on its side.

Exchanges and messages

Service sockets are based on exchanges, which are sets of messages. An exchange is described by a structure made of the following:

  • Identifier (8 bytes)
  • Method or notification code (2 bytes)
  • Status (1 byte): 0x01 if the exchange must be closed, 0x00 otherwise
  • Error code (on 2 bytes) - only valid if the exchange is closed by this message
  • Message counter for this exchange (2 bytes)
  • Message's length (2 bytes)

This structure is handled by the kernel itself and can be managed using a set of system calls and signals.

Messages can then be sent using the SEND_SOCK_MSG syscall and received through the RECV_SOCK_MSG signal. To be read, the receiver process must use the READ_SOCK_MSG syscall.

They must be sent at once, like in message pipes. The maximum size is 4 KB by default, but can be extended through the opening syscall up to 256 MB.

A message can either initiate an exchange or answer to an existing one: in the first case, an identifier will be generated by the kernel and returned by the sending syscall, and in the latter the message will receive an incremented identifier that allows to track where in the exchange the message is. This also prevents a process to send two messages before the other one answers.

Concluding exchanges

Exchanges can be either be concluded either by sending an error message or by sending a message with a close indicator to indicate this message will be the last one in the socket and no answer will be accepted.

Sending a message in a concluded exchange will result in a syscall error.

Methods and notifications

An exchange can be opened by a client to request something to the service, it is then called a method. In that case, the message's body is called the arguments.

When the server opens itself an exchange with the client, it's called a notification. A notification message's body is called its data.

Closing service sockets

Service sockets can be closed with the CLOSE_SERV_SOCK syscall, which triggers the SERV_SOCK_CLOSED signal on the other process' side.

Although exchanges are based on signals which are asynchronous by design, the answer mechanism and messages tracking which ensures the receiver process answers before sending another message allows for simplier communications and synchronization between the two processes.

Shared Memory

Shared memory allows a process to share a part of its memory with another process. It has multiple advantages over pipes:

  • Data is not copied twice, as the sender process directly shares a part of its own memory
  • There is no pipe management, which results in saving operations and less memory accesses
  • There is no pipe buffer to manage which means all the data can be sent at once

Its main disadvantage being that all data is shared at once, so there is no synchronization or "asynchronous sending" mechanism, which is the purpose of pipes.

It works by asking the kernel to share the memory through the SHARE_AMS syscall, the target process receiving the RECV_SHARED_AMS signal.

There are two types of sharing:

  • Mutual sharing allows both the sharer and the receiver processes to access the shared memory ;
  • Exclusive sharing unmaps the shared memory from the sharer and only allows the receiver process to access it

Exclusive mode has several advantages: the sender process to not have to care about managing this memory and avoid overwriting it by accident, but it also ensures the receiving process that the sender will not perform malicious modifications on the shared buffer while the data is processed on its side.

Mutual memory sharing can be stopped using the UNSHARE_AMS syscall, while exclusive sharing are left to the receiver process.

Signals

Signals are a type of KPC. They are used by the kernel to send informations to processes about a specific event.

Technical overview

When a process is created, the kernel associates it:

  • A signals handler table (SHT) ;
  • A signals queue ;
  • A readiness indicator

Each signal has a 8-bit code that identifies it, as well as a 32 bytes datafield which is used to attach additional informations about the signal.

When the kernel sends a signal to a process, it first checks if an handler is already running. If so, it simply pushes the signal to the queue.

Else, it checks the readiness indicator. If it is false (so if the process did not sent the READY syscall yet), the signal is pushed to the queue.

Else, it checks in the SHT if the signal has a handler. If there is no handler, depending on the specific signal, it may either be ignored or use a default behaviour (this is documented for each signal).

If a handler is found, the kernel checks if the pointer points to a memory area that is executable by the current process. If it isn't, the signal is converted to an HANDLER_FAULT one. If the signal that was being sent was already an HANDLER_FAULT, the process is killed.

The kernel then switches the process to its main thread and makes it jump to the handler's address, then resumes it.

When the handler returns (or the default behaviour completes), if the signal was expecting an answer, the kernel reads it from specific registries and does whatever it needs to do. Then, itchecks if the signals queue is empty. If it is, the kernel simply makes the process jump back to the address it was to before the signal was emitted, and switch to the original thread.

Else, it interrupts the process again and proceeds to treat the first signal on the queue after removing it.

On the performances side, signals use interrupts, meaning the process' current tasks are instantly interrupted to let it handle the signal without delay. Also, the datafield and answer are provided through CPU registers, avoiding memory accesses.

List of signals

You can find below the exhaustive list of signals.

0x01 HANDLER_FAULT

Sent when a signal is sent to a process but the registered handler points to a memory zone that is not executable by the current process. If the sending of this signal to the process results to another fault, it's called a double handler fault and the process is immediatly killed.

If no handler is registered for this signal, it will kill the process when received.

Datafield:

  • Faulty signal ID (8 bytes)

0x02 MEM_FAULT

Sent when the process tried to perform an unauthorized access on a memory address.

Datafield:

  • Faulty address (8 bytes)
  • Access error (1 byte):
    • 0x01: tried to read memory
    • 0x02: tried to write memory
    • 0x03: tried to execute memory

0x20 RECV_PIPE

Sent to a process when another process of the same application and running under the same user opened an pipe with this process, giving it the other part.
The command code can be used to determine what the other process is expecting this one to do. This code does not follow any specific format.

Datafield:

  • Pipe creator's PID (8 bytes)
  • Pipe creator's application's ANID (4 bytes)
  • Pipe SC or RC identifier (8 bytes)
  • Command code (2 bytes)
  • Pipe identifier type (1 byte): 0x00 if the pipe identifier is an RC, 0x01 if it's an SC
  • Mode (1 byte): 0x00 if it's a raw pipe, 0x01 if it's a message pipe
  • Size hint in bytes (8 bytes), with 0 being the 'no size hint' value

0x21 PIPE_CLOSED

Sent to a process when a pipe shared with another process is closed.

NOTE: This does not apply to service pipes.

Datafield:

  • Closing type (1 byte):
    • 0x00 if the pipe was closed properly using the CLOSE_PIPE syscall
    • 0x01 if the other process brutally terminated |
  • Pipe identity (1 byte): 0x00 if this process contained the RC part, 0x01 if it contained the SC part (1 byte)
  • RC or SC identifier (8 bytes)

0x26 RECV_SERV_SOCK

Sent to a process when another process opened a service socket with this one.

Datafield:

  • Service socket creator's PID (8 bytes)
  • Service socket creator's application's ANID (4 bytes)
  • Service socket identifier (8 bytes)
  • Size of the buffer, multiplied by 4KB (2 bytes)

0x27 RECV_SOCK_MSG

Sent to a process when a message has been sent through a service socket.
To read the message, the process must use the READ_SOCK_MSG syscall.

Datafield:

  • Servive socket identifier (8 bytes)
  • Exchange identifier (8 bytes)
  • Exchange method (1 byte)
  • Size of the message (4 bytes)
  • Status (1 byte):
    • Bit 0: set if this message did create a new exchange
    • Bit 1: set if this message is an error message
    • Bit 2: set if this message closed the socket

0x29 SERV_SOCK_CLOSED

(was there a message sent before that closed the socket)

Sent to a process when a pipe shared with another process is closed.

NOTE: This does not apply to service pipes.

Datafield:

  • Closing type (1 byte):
    • 0x00 if the pipe was closed properly using the CLOSE_PIPE syscall
    • 0x01 if the other process brutally terminated |
  • Pipe identity (1 byte): 0x00 if this process contained the RC part, 0x01 if it contained the SC part (1 byte)
  • RC or SC identifier (8 bytes)

0x2A SERVICE_CONN_REQUEST

Sent to a service process' dispatcher thread when another process tries to etablish a connection through the CONNECT_SERVICE syscall.

The process is expected to answer using the ACCEPT_SERVICE_CONNECTION under the provided delay, else it's considered as a rejection.

If no handler is registered for this signal, it will kill the process when received.

NOTE: This signal cannot be received if the application does not expose a service.

Datafield:

  • Callee process' ID (8 bytes)
  • Connection's unique request ID (8 bytes)
  • Command code (2 bytes)
  • Registry's system.processes.service_answer_delay key (default: 2000ms) (2 bytes)

0x2B SERVICE_CLIENT_CLOSED

Sent to a client thread to indicate its client closed before the connection was properly terminated. The thread is expected to terminate as soon as possible (there is no time limit though).

0x2C SERVICE_CLIENT_QUITTED

Sent to a client thread to indicate its client asked to close the connection. The associated RC and SC are immediatly closed.

0x2D SERVICE_SERVER_QUITTED

Sent to a process that previously established a connection with a service, to indicate the associated service thread closed before the connection was properly terminated.

Datafield:

  • Connection's unique request ID (8 bytes)

0x33 READ_BACKED_AMS

Sent to a process when a signal-backed abstract memory segment (AMS) is accessed in read mode.

Datafield:

  • AMS ID (8 bytes)
  • Relative address accessed in the segment (8 bytes)
  • Access mode (1 byte): 0x00 for read, 0x01 for execution

Expected answer:

  • Associated data for this file (4 bytes)
  • Page fault (1 byte):
    • 0x00: no page fault
    • 0x01: address is out-of-range
    • 0x02: hardware fault

0x34 WRITE_BACKED_AMS

Sent to a process when a signal-backed abstract memory segment (AMS) is accessed in write mode.

Datafield:

  • AMS ID (8 bytes)
  • Relative address accessed in the segment (8 bytes)
  • Data to write (4 bytes)

Expected answer:

  • Page fault (1 byte):
    • 0x00: no page fault
    • 0x01: address is out-of-range
    • 0x02: hardware fault

0x35 RECV_SHARED_AMS

Sent to a process when an abstract memory segment (AMS) is shared by another process.

Datafield:

  • Sender PID (8 bytes)
  • Command code (2 bytes)
  • AMS ID (8 bytes)
  • Sharing mode (1 byte): 0x00 for mutual sharing, 0x01 for exclusive sharing
  • Access permissions (1 byte):
    • For mutual sharings: strongest bit for read, next for write, next for exec
    • For exclusive sharings: 0b11100000

0x37 UNSHARED_AMS

Sent to a process when an abstract memory segment (AMS) is unshared by the sharer process.

Datafield:

  • Unsharing type (1 byte):
    • 0x00 if the shared memory was unshared properly using the UNSHARE_AMS syscall
    • 0x01 if the other process brutally terminated

0x44 SUSPEND

Sent when the process is asked to suspend. It's up to the process to either ignore this signal or suspend itself using the SUSPEND syscall.

0x45 WILL_SUSPEND

Sent when the process is asked to suspend. If it is not suspended after the provided delay, the process is suspended.

Datafield:

  • Registry's system.processes.suspend_delay key (default: 500ms) (2 bytes)

0x46 UNSUSPENDED

Sent when the process was just unsuspended.

Datafield:

  • Suspension duration, in microseconds (8 bytes)

0x4E TERMINATE

Sent when the process is asked to terminate. It's up to the process to either ignore this signal or terminate itself (preferably by using the EXIT syscall).

0x4F WILL_TERMINATE

Sent when the process is asked to terminate. If it does not terminate by itself before the provided delay, the process is killed.

If no handler is registered for this signal, it will kill the process when received.

Datafield:

  • Registry's system.processes.terminate_delay key (default: 2s) (2 bytes)

0xD4 DEVICE_CHANGED

Sent to the sys::hw service only, when a hardware device is plugged, unplugged, or when its raw device descriptor (RDD) changes.

The KDI is guaranteed to remain the same for each individual devices.

Datafield:

  • Event code (1 byte):
    • 0x01: a device was connected on this port
    • 0x02: the connected device was disconnected from this port
    • 0x03: the device connected to this port changed its RDD
  • RDD

0xD4 DEVICE_INTERRUPT

Sent to the sys::hw service only, when a hardware device raises a CPU interruption.

Datafield:

  • KDI (8 bytes)
  • Interruption code (1 byte)

System calls

System calls, abbreviated syscalls, are a type of KPC. They allow a process to ask the kernel to perform an action.

Technical overview

Syscalls are performed using CPU interruptions to notify the kernel.

A syscall is made of a 8-bit code, as well as up to 8 arguments with up to 64-bit value each.
When performing a syscall, the process will put in a specific CPU register an address poiting to a memory address containing in a row the syscall's code and its arguments. For most syscalls, code and arguments will be not be longer than 128 bits, but some may use larger arguments.

Some syscalls require the process to send a buffer of data. In such case, the process simply provides a pointer to the said buffer - so the argument's size will vary depending on the length of memory addresses.

After preparing the syscall's code and arguments, the process raises a specific exception that is caught by the kernel. When the syscall is complete, the kernel puts the result values in specific registers and resumes the process. This means that all syscalls are synchronous.

System calls always return two numbers: a 8-bit one (errcode) and a 8 bytes one (return value). If the errcode is not null, then an error occured during the syscall. The specific value indicate the encountered type of error:

  • 0x00: cannot read syscall's code or arguments (error while reading memory)
  • 0x01: the requested syscall does not exist
  • 0x02: at least one argument is invalid (e.g. providing a pointer to the 0 address)
  • 0x03: unmapped memory pointer (e.g. provided a pointer to a memory location that is not mapped yet)
  • 0x04: memory permission error (e.g. provided a writable buffer to an allocated but non-writable memory address)
  • 0x10 to 0x1F: Invalid argument(s) provided (constant checking)
  • 0x20 to 0x2F: Provided arguments are not valid in the current context (in relation with other arguments)
  • 0x30 to 0x3F: Provided arguments are not valid (after resources checking)
  • 0x40 to 0x4F: Resource access or modification error
  • 0x50 to 0x5F: Handled hardware errors
  • 0x60 to 0x6F: Other types of errors

System calls' code are categorized as follows:

  • 0x00 to 0x0F: signal handling
  • 0x10 to 0x1F: process management
  • 0x20 to 0x29: pipes
  • 0x2A to 0x2F: services communication
  • 0x30 to 0x3F: memory management
  • 0x40 to 0x4F: AMS management
  • 0x50 to 0x5F: processes management
  • 0x60 to 0x6F: threads management
  • 0x70 to 0x7F: hardware interaction
  • 0xA0 to 0xAF: applications-related syscalls
  • 0xD0 to 0xDF: reserved to system services

Note that advanced actions like permissions management or filesystem access are achieved through the use of IPC.

List of syscalls

You can find below the exhaustive list of system calls.

0x01 HANDLE_SIGNAL

Register a signal handler.
If the address pointed by this syscall's is not executable by the current process when this signal is sent to the process, the signal will be converted to an HANDLER_FAULT signal instead.

Arguments:

  • Code of the signal to handle (1 byte)
  • Pointer to the handler function (8 bytes)

Return value:

Empty

Errors:

  • 0x10: The requested signal does not exist

0x02 UNHANDLE_SIGNAL

Unregister a signal handler, falling back to the default signal reception behaviour if this signal is sent to the process.

Arguments:

  • Code of the signal to stop handling (1 byte)

Return value:

Empty

Errors:

  • 0x10: The requested signal does not exist
  • 0x30: The requested signal does not have an handler

0x03 IS_SIGNAL_HANDLED

Check if a signal has a registered handler.

Arguments:

  • Code of the signal (1 byte)

Return value:

  • 0 if the signal is not handled, 1 if it is (1 byte)

Errors:

  • 0x10: The requested signal does not exist

0x04 READY

Indicate the system this process has set up all its event listeners, so it can start dequeuing signals.

NOTE: When this signal is sent, all queued signals will be treated at once, so the instructions following the sending of this signal may not be ran until quite a bit of time in the worst scenario.

WARNING: Signals will not be treated until this syscall is sent by the process!

Arguments:

None

Return value:

Empty

Errors:

  • 0x30: The process already told it was ready

0x20 OPEN_PIPE

Open a pipe with a process of the same application and running under the same user and get its SC.
The buffer size multiplier indicates the size of the pipe's buffer, multiplied by 64 KB. The default (0) falls back to a size of 64 KB.
The command code can be used to indicate to the target process which action is expected from it. It does not follow any specific format.
The target process will receive the SC/RC's counterpart through the RECV_PIPE signal, unless notification mode states otherwise.

Arguments:

  • Target process' PID (8 bytes)
  • Command code (2 bytes)
  • Pipe type (1 byte): 0x00 to create a write pipe, 0x01 to create a read pipe
  • Buffer size multiplier (1 byte)
  • Transmission mode (1 byte): 0x00 to create a raw pipe, 0x01 to create a message pipe
  • Notification mode (1 byte): 0x00 to notify the process with the RECV_PIPE signal, 0x01 to skip it
  • Size hint in bytes (8 bytes), with 0 being the 'no size hint' value

Return value:

  • Pipe SC identifier (8 bytes)

Errors:

  • 0x10: Invalid transmission mode provided
  • 0x11: Invalid notification mode provided
  • 0x30: The provided PID does not exist
  • 0x31: The target process is not part of this application
  • 0x32: The target process runs under another user
  • 0x33: Notification mode is enabled but the target process does not have a handler registered for the RECV_PIPE signal

0x21 SEND_PIPE

Share an RC or SC identifier with another process.
This will trigger in the target process the RECV_PIPE signal, unless the notification mode tells otherwise.

When the target process writes through the received SC or read from the received RC, the performance will be equal to writing or reading through the original RC/SC identifier.

Arguments:

  • Pipe RC or SC identifier (8 bytes)
  • Target PID (8 bytes)
  • Notification mode (1 byte): 0x00 to notify the process with a pipe reception signal, 0x01 to skip the signal

Return value:

None

Errors:

  • 0x30: Notification mode is enabled but the target process does not have a handler registered for the RECV_PIPE signal

0x22 PIPE_WRITE

Write data through a pipe.
Messages will always be sent at once when writing to message pipes.
If the data is 0-byte long, this pipe will return successfully without waiting, even if the target pipe's buffer is full or locked.

Arguments:

  • Pipe SC identifier (8 bytes)
  • Number of bytes to write (4 bytes)
  • Pointer to a readable buffer (8 bytes)
  • Mode (1 byte): 0x00 = block until there is enough space to write, 0x01 = fail if there is not enough space to write or if the pipe is locked, 0x02 = write as much as possible

Return value:

Encoded on 4 bytes:

  • If mode is 0x00: remaining capacity of the pipe
  • If mode is 0x01: 0x00 if the cause of failure was because the pipe was locked, 0x01 if it was because of of a lack of space in the target buffer
  • If mode is 0x02: number of bytes written

Errors:

  • 0x10: Invalid mode provided
  • 0x30: The provided SC identifier does not exist
  • 0x31: The provided SC was already closed
  • 0x32: The provided SC refers to a message pipe but the provided size is larger than 64 KB
  • 0x33: The provided SC refers to a message pipe but the 0x02 mode was provided
  • 0x40: There is not enough space in the pipe to write all the provided data and the mode argument was set to 0x01

0x23 PIPE_READ

Read pending data or message from a pipe.
If the pipe was closed while the buffer was not empty, this syscall will still be able to read the remaining buffer's data - but the pipe will not be able to receive any additional data. Then, once the buffer is empty, the pipe will be made unavailable.

Arguments:

  • Pipe RC identifier (8 byte)
  • Mode (1 byte): 0x00 = block until there are enough data to read, 0x01 = fail if there is not enough data to read or if the pipe is locked, 0x02 = read as much as possible
  • Readable buffer pointer (16 bytes) (specify 0 bytes length to read as much data as possible)

Return value:

Encoded on 4 bytes:

  • If mode is 0x00: remaining bytes in the buffer
  • If mode is 0x01: 0x00 if the cause of failure was because the pipe was locked, 0x01 if it was because of of a lack of space in the target buffer
  • If mode is 0x02: number of read bytes

Errors:

  • 0x10: Invalid mode provided
  • 0x30: The provided RC identifier does not exist
  • 0x31: The provided RC was already closed
  • 0x32: There is no pending data in the pipe and the mode argument was set to 0x01
  • 0x33: The provided RC refers to a message pipe but the 0x02 mode was provided

0x24 PIPE_INFO

Get informations on a pipe from its RC or SC identifier.

Arguments:

  • Pipe RC or SC identifier (8 bytes)

Return value:

  • Status (1 byte):
    • Bit 0 (strongest): indicates if the pipe is opened
    • Bit 1: indicates if the pipe is a message pipe
    • Bit 2: indicates if the pipe's buffer is full
    • Bit 3: indicates if the pipe is locked
    • Bit 4: indicates if a writing request is pending (waiting for the pipe to be unlocked)
    • Bit 5: indicates if a reading request is pending (waiting for the pipe to be unlocked)
    • Bit 6: indicates if the provided identifier is an SC
  • Pipe's buffer's capacity (8 bytes)
  • Remaining data before the pipe's buffer is full (8 bytes)
  • Pipe's creator's PID (8 bytes)

Errors:

  • 0x30: The provided RC or SC identifier does not exist

0x25 CLOSE_PIPE

Close a pipe properly. The RC and SC parts will be immediatly closed.
The other process this pipe was shared with will receive the PIPE_CLOSED signal unless this pipe was created during a service connection.
If this syscall is not performed on a pipe before the process exits, the other process will receive the same signal with a specific argument to indicate the communication was brutally interrupted.

Arguments:

  • Pipe RC or SC identifier (8 bytes)

Return value:

None

Errors:

  • 0x30: The provided RC/SC identifier does not exist
  • 0x31: The target process already terminated
  • 0x32: The provided RC/SC identifier is part of a service pipe

0x26 OPEN_SERV_SOCK

Open a service socket.
Triggers the RECV_SERV_SOCK signal on the receiver process' side.

Arguments:

  • Client process PID (8 bytes)
  • Buffer size multiplier by 4 KB (2 bytes) - 0 fall backs to 4KB

Return value:

  • Socket identifier (8 bytes)

Errors:

  • 0x30: Unknown PID provided
  • 0x31: Current process is not allowed to communicate with the provided process

0x27 SEND_SOCK_MSG

Send a message through a service socket exchange.
This syscall can also be used to create a new exchange.

Sending a non-zero error code will close the exchange.

Arguments:

  • Socket identifier (8 bytes)
  • Exchange identifier (8 bytes) - 0 creates a new exchange
  • Method or notification code (1 byte) - non-zero value if not creating an exchange to close it with a non-error message
  • Error code (2 bytes)
  • Buffer pointer to the message's content (16 bytes)

Return value:

  • Exchange identifier (8 bytes)
  • Message counter for this exchange (4 bytes)

Errors:

  • 0x30: Unknown socket identifier
  • 0x31: Socket is already closed
  • 0x32: Unknown exchange identifier
  • 0x40: Exchange has already been concluded

0x28 READ_SOCK_MSG

Read the pending message of a service socket.

Arguments:

  • Socket identifier (8 bytes)
  • Address of a writable buffer (8 bytes)

Return value:

  • Number of written bytes (4 bytes)
  • 0x01 if a message was retrieved, 0x00 if none was pending (1 byte)
  • Status (1 byte):
    • Bit 0: set if this message did create a new exchange
    • Bit 1: set if this message is an error message
    • Bit 2: set if this message closed the socket

Errors:

  • 0x30: Unknown socket identifier
  • 0x31: Socket is already closed
  • 0x32: Unknown exchange identifier
  • 0x40: Exchange has already been concluded

0x29 CLOSE_SERV_SOCK

Close a service socket.
Triggers the SERV_SOCK_CLOSED signal on the receiver process' side.

Arguments:

  • Socket identifier (8 bytes)

Return value:

None

Errors:

  • 0x30: Unknown socket identifier
  • 0x40: Socket is already closed

0x2A CONNECT_SERVICE

Ask a service to etablish connection. The current process is called the service's client.

If the current process already has an active connection (a connection that hasn't been closed) to the target service, it will fail unless the flexible mode argument is set.

NOTE: When this signal is sent, the service's answer will be waited, so the instructions following the sending of this signal may not be ran until several seconds in the worst scenario.

Arguments:

  • Target application's ANID (4 bytes)
  • Target type (1 byte):
  • Scope name - filled with zeroes to access the default service (8 bytes)
  • Command code (2 bytes)

Return value:

  • Unique connection ID (8 bytes)
  • Service socket identifier (8 bytes)
  • Flexible mode (1 byte): 0x00 by default, 0x01 returns the existing connection ID if an active connection is already in place with the service instead of failing

Errors:

0x2B END_SERVICE_CONN

Tell a service to properly close the connection. The associated service socke will immediatly be closed.

Arguments:

  • Unique connection ID (8 bytes)

Return value:

None

Errors:

  • 0x30: The provided connection ID does not exist
  • 0x40: This connection was already closed
  • 0x41: The associated service thread already terminated

0x2C ACCEPT_SERVICE_CONN

Confirm the current service accepts the connection with a client.
A dedicated service socket identifier will be provided to communicate with the client.

This will create a new client thread in the current process, which is meant to be dedicated to this specific client.
The client thread will not receive any SERVICE_CONN_REQUEST signal, only dispatcher thread will.

When the associated client terminates, the SERVICE_CLIENT_CLOSED signal is sent to this thread.

Arguments:

  • Connection's unique request ID (8 bytes)

Return value:

  • 0x00 if the current process is now the associated client's thread, 0x01 else
  • Service socket identifier (8 bytes)

Errors:

  • 0x30: This request ID does not exist
  • 0x40: The process which requested the connection already terminated
  • 0x41: Answer was given after the delay set in the registry's system.processes.service_answer_delay key (default: 2000ms)

0x2D REJECT_SERVICE_CONN

Reject a connection request to the current service.

Arguments:

  • Connection's unique request ID (8 bytes)

Return value:

None

Errors:

  • 0x30: This request ID does not exist
  • 0x40: The process which requested the connection already terminated
  • 0x41: Answer was given after the delay set in the registry's system.processes.service_answer_delay key (default: 2000ms)

0x2E HAS_SCOPED_SERVICE

Arguments:

  • Target application's ANID (4 bytes)
  • Scope name (8 bytes) - fill with zeroes to check the default service

Return value:

  • 0x01 if the application has a service for the provided scope, 0x00 otherwise

Errors:

0x30 MEM_ALLOC

Allocate a contiguous block of memory. In case of large allocations, a large memory page may be allocated instead of multiple small ones.

WARNING: Allocated memory will not be zeroes unless it previously belonged to another process. Therefore the caller process shall ensure memory is used correctly.

Arguments:

  • Number of bytes to allocate (8 bytes)
  • Permissions (3 bits, from the weakest):
    • 0b001: Read-only
    • 0b010: Write-only
    • 0b011: Read-write
    • 0b100: Exec-only
  • 0x01 to ensure the zone is zeroed before allocating, 0x00 else (1 byte)

Return value:

  • Pointer to the newly-allocated block of memory (8 bytes)

Errors:

  • 0x10: Invalid permission
  • 0x40: The kernel could not find a contiguous block of memory of the requested size

0x31 MEM_FREE

Unallocate a contiguous block of memory.

Shared memory pages must first be unshared through the UNSHARE_AMS syscall.
Mapped memory pages must be unmapped through the UNMAP_AMS syscall.

WARNING: Memory will not be zeroed, therefore the caller process shall ensure critical informations are zeroed or randomized before freeing the memory.

Arguments:

  • Pointer to the start address to unallocate the memory from (8 bytes)
  • The number of pages to unallocate (8 bytes)

Return value:

None

Errors:

  • 0x10: The provided start address it not aligned with a page
  • 0x30: The provided start address is out of the process' range
  • 0x31: The provided size, added to the start address, would exceed the process' range
  • 0x32: One or more of the provided pages was not allocated (e.g. unmapped page or memory-mapped page)
  • 0x33: One or more of the provided pages are shared with another process

0x40 VIRT_MEM_AMS

Create an abstract memory segment (AMS) from a part of the current process' address space.

Arguments:

  • Address of the first page to register in the AMS (8 bytes)
  • Number of bytes to register (8 bytes)

Return value:

  • AMS ID (8 bytes)

Errors:

  • 0x10: Start address is unaligned
  • 0x11: Number of bytes is unaligned
  • 0x30: Address is out of range

0x41 BACKED_AMS

Create an abstract memory segment (AMS) backed by the READ_BACKED_AMS and WRITE_BACKED_AMS signals.

Copy-on-write support can be enabled to allow the receiver process to write data in its own memory space. Written pages will be allocated by the kernel and won't be backed anymore by the READ_BACKED_AMS signal. The backer process won't be able to see these changes, and the WRITE_BACKEND_AMS signal won't be trigerred on its side.

Arguments:

  • Length of the AMS (8 bytes)
  • Copy-on-write mode (1 byte): 0x00 to disable, 0x01 to enable

Errors:

  • 0x10: Invalid COW mode provided
  • 0x11: Provided length is unaligned

0x42 SHARE_AMS

Share an abstract memory segment (AMS) with another process.

This will trigger in the target process the RECV_SHARED_MEM with the provided command code, unless the notification mode states otherwise.

The mutual mode allows both processes to access the memory, with the sharer setting the permissions for the receiver to limit its access. Copy-on-write can also be enabled to allow the receiver process to write data without affecting the sharer process' memory.

The exclusive mode allows, only when sharing AMS made from existing memory pages from its original process, to unmap the original pages from the said process to let the exclusive access to the target process. This is useful when transferring temporarily large chunks of data to another process. Also, access permissions are ignored when using exclusive mode.

The returned AMS ID is common for both the sender and the receiver, allowing to use it in exchanges.

Arguments:

  • Target process' PID (8 bytes)
  • AMS ID (8 bytes)
  • Command code (2 bytes)
  • Notification mode (1 byte): 0x00 to notify the process with the RECV_SHARED_AMS signal, 0x01 to skip it
  • Mode (1 byte):
    • Mutual: 0 b 0 0 0 0 <1 to enable copy-on-write> <1 to enable read> <1 to enable write> <1 to enable exec>
    • Exclusive: 0 b 0 0 0 0 1 0 0 <1 to unmap original pages>

Return value:

None

Errors:

  • 0x10: Invalid notification mode provided
  • 0x11: Invalid mode provided
  • 0x12: Invalid exclusive mode provided
  • 0x20: Access permissions were not set but the sharing mode is set to mutual
  • 0x21: Access permissions were provided but the sharing mode is set to exclusive
  • 0x40: There is not enough contiguous space in the receiver process' memory space to map the shared memory

0x43 AMS_SHARING_INFO

Get informations about a shared abstract memory segment (AMS).

Arguments:

  • AMS ID (8 bytes)

Return value:

  • Sharer process' PID (8 bytes)
  • Receiver process' PID (8 bytes)
  • Sharing mode (1 byte): 0x00 for mutual mode, 0x01 for exclusive mode
  • Pointer to the shared buffer (16 bytes)
  • Command code (2 bytes)
  • Access permissions (1 byte): for mutual sharings, strongest bit for read, next for write, next for exec ; for exclusive sharings, 0x00

Errors:

  • 0x30: Unknwon AMS ID provided

0x44 UNSHARE_AMS

Stop sharing an abstract memory segment (AMS) started by SHARE_AMS. Note that exlusive sharings cannot be unmapped.

This will trigger in the target process the UNSHARED_AMS signal.

Arguments:

  • AMS ID (8 bytes)
  • PID to stop sharing with (8 bytes) - 0 to stop sharing with all processes

Return value:

None

Errors:

  • 0x30: Unknown AMS ID provided
  • 0x21: Provided AMS ID is exclusive
  • 0x32: Provided AMS was not shared with the provided process

0x45 MAP_AMS

Map an abstract memory segment (AMS) in the current process' address space.

Arguments:

  • AMS ID (8 bytes)
  • Address to map the AMS from (8 bytes)
  • Mapped buffer pointer (16 bytes)

Return value:

None

Errors:

  • 0x30: Unknown AMS ID provided
  • 0x31: Provided mapping address or address+length is out-of-range in the AMS
  • 0x32: Provided address to map or address+length is out-of-orange in this process' address space

0x46 UNMAP_AMS

Unmap an abstract memory segment (AMS) from the current process' address space.
If the AMS is mapped at multiple addresses of this process, only one of the mappings will be unmapped by default.

Arguments:

  • AMS ID (8 bytes)
  • Mapping address (8 bytes) - 0 to unmap from all addresses

Return value:

Empty

Errors:

  • 0x10: Unknown AMS ID provided
  • 0x30: Provided AMS it not mapped at this address

0x50 CREATE_PROCESS

Create a child process from the current one. The new process gets a separate memory space.

Communication can be done through standard IPC.

The initialization data is joined as part of the application's execution context.

If the parent process is part of a container, the child process will be part of the same one.

Arguments:

  • Initialization data (8 bytes)

Return value:

  • Child process identifier (8 bytes)
  • Identity (1 byte): 0x00 if the current thread is the parent, 0x01 for the child
  • Initialization data (8 bytes) - 0 for the parent

Errors:

  • 0x30: The current process is not an application process
  • 0x40: Failed to create a new process due to hardware problem (cannot allocate memory, ...)

0x51 WAIT_CHILD_PROCESS

Wait for a child process to terminate.

Arguments:

  • Process identifier (8 bytes)
  • Timeout in milliseconds (2 bytes)

Return value:

Empty

Errors:

  • 0x30: The provided PID does not exist or does not belong to the current application

0x52 KILL_CHILD_PROCESS

Kill a child process, which will first receive the WILL_TERMINATE signal.

Arguments:

  • Process identifier (8 bytes)
  • Timeout in milliseconds (2 bytes) - if 0, will use the system's global default value (registry's system.processes.terminate_delay)

Return value:

  • Process' exit data (provided through EXIT_PROCESS, 0 otherwise)

Errors:

  • 0x40: Failed to create a new process due to hardware problem (cannot allocate memory, ...)

0x53 GET_PID

Get the current process' PID.

Arguments:

None

Return value:

  • Current process' PID (8 bytes)

Errors:

None

0x54 SUSPEND

Suspend the current process or a child process.

Arguments:

  • PID to suspend (8 bytes) - 0 for the current process

Return value:

  • Amount of time the process was suspended, in milliseconds (8 bytes)

Errors:

  • 0x30: the current process is not an application process
  • 0x31: the current PID was not found or is not a child of the current process

0x55 UNSUSPEND

Unsuspend a child process.

Will trigger the UNSUSPENDED signal on the child process' side.

Arguments:

  • PID to unsuspend (8 bytes) - 0 for the current process

Return value:

  • Amount of time the process was suspended, in milliseconds (8 bytes)

Errors:

  • 0x30: the current process is not an application process
  • 0x31: the current PID was not found or is not a child of the current process

0x56 HAND_OVER

End this process' cycle.

Used to indicate to the kernel the current process has no additional work to do for now (e.g. waiting for asynchronous I/O data).

Arguments:

None

Return value:

None

Errors:

None

0x5F EXIT

Kill the current process.

A SERVICE_CLIENT_CLOSED signal is sent to all services connection the process has.
If the current process is a service, a SERVICE_SERVER_QUITTED signal is sent to all active clients.

Arguments:

  • Exit data (8 bytes)

Return value:

None (never returns)

Errors:

None

0x60 CREATE_THREAD

Create a thread from the current one. The new thread will share the current one's memory space.

Arguments:

  • Initialization data (8 bytes)

Return value:

  • Child thread identifier (8 bytes)
  • Identity (1 byte): 0x00 if the current thread is the parent, 0x01 for the child
  • Initialization data (8 bytes) - 0 for the parent

Errors:

  • 0x40: Failed to create a new thread due to hardware problem (cannot allocate memory, ...)

0x61 CREATE_TLS_SLOT

Create a TLS slot.

Arguments:

None

Return value:

  • TLS slot identifier (8 bytes)

Errors:

  • 0x40: Maximum number of TLS slots was reached
  • 0x41: Could not allocate memory for a new TLS slot

0x62 READ_TLS_SLOT

Read from a TLS slot.

Arguments:

  • TLS slot identifier (8 bytes)
  • Writable buffer (16 bytes) (specify 0 bytes length to read the entire data at once)

Return value:

  • Number of bytes written (8 bytes)

Errors:

  • 0x30: Unknown TLS identifier

0x63 WRITE_TLS_SLOT

Write to a TLS slot.

Arguments:

Return value:

None

Errors:

  • 0x30: Unknown TLS identifier
  • 0x40: Failed to allocate enough memory for the written data

0x64 DELETE_TLS_SLOT

Delete a TLS slot.

Arguments:

  • TLS slot identifier (8 bytes)

Return value:

None

Errors:

  • 0x30: Unknown TLS identifier

0x6F EXIT_THREAD

Kill the current thread and all of its children threads.

Arguments:

  • Exit data (4 bytes)

Return value:

None (never returns)

Errors:

None

0x70 READ_IO_PORT

Read data from the physical I/O port a device, if authorized as a driver.

Complete access is granted to the sys::hw service.

Arguments:

  • KDI of the device to read data from (8 bytes)
  • Relative I/O port number (2 bytes)
  • Writable buffer pointer (16 bytes)

Return value:

  • Number of bytes read from the I/O port (8 bytes)

Error codes:

  • 0x30: This device is not registered in this process' drivable devices attribute
  • 0x31: The provided device KDI was not found
  • 0x32: The provided I/O port does not exist for the provided device
  • 0x33: The provided I/O port is an output port

0x71 WRITE_IO_PORT

Write data to the physical I/O port a device, if authorized as a driver.

Complete access is granted to the sys::hw service.

Arguments:

  • KDI of the device to read data from (8 bytes)
  • Relative I/O port number (2 bytes)
  • Readable buffer pointer (16 bytes)

Return value:

  • Number of bytes written (8 bytes)

Error codes:

  • 0x30: This device is not registered in this process' drivable devices attribute
  • 0x31: The provided device KDI was not found
  • 0x32: The provided I/O port does not exist for the provided device
  • 0x33: The provided I/O port is an input port

0x72 DEVICE_INTERRUPT

Trigger a hardware interruption on a device, if authorized as a driver.

Complete access is granted to the sys::hw service.

Arguments:

  • KDI of the device to trigger an interrupt on (8 bytes)
  • Relative I/O port identifier (2 bytes)
  • Interrupt code (1 byte)

Return value:

None

Error codes:

  • 0x30: This device is not registered in this process' drivable devices attribute
  • 0x31: The provided device KDI was not found
  • 0x32: The provided I/O port does not exist for the provided device
  • 0x33: The provided I/O port is an input port

0x73 DEVICE_AMS

Create an abstract memory segment (AMS) from a device's memory through Mapped Memory Input/Output (MMIO).

Read data from the physical input/output port a device, if authorized as a driver.

Complete access is granted to the sys::hw service.

Arguments:

  • KDI of the device to map in memory (8 bytes)
  • Start address in the device's memory (8 bytes)
  • Number of bytes to map (8 bytes)
  • Start address to map in this process' memory (8 bytes)

Return value:

  • AMS ID (8 bytes)

Errors:

  • 0x10: The mapping's start address is not aligned with a page
  • 0x11: The mapping's length is not a multiple of a page's size
  • 0x12: The mapping's size is null (0 bytes)
  • 0x30: This device is not registered in this process' drivable devices attribute
  • 0x31: The provided device KDI was not found
  • 0x32: The provided device is not compatible with MMIO

0x74 SET_DMA_MEM_ACCESS

Allow or disallow a device to access a range of addresses through Direct Memory Access (DMA) in the current process' address space.

Read data from the physical input/output port a device, if authorized as a driver.

Complete access is granted to the sys::hw service.

Arguments:

  • KDI of the device to map in memory (8 bytes)
  • Start address in the current process' address space (8 bytes)
  • Length (8 bytes)
  • Authorization (1 byte): 0x00 to allow the device to use this range, 0x01 to cancel an authorization

Return value:

None

Errors:

  • 0x10: The range's start address is not aligned with a page
  • 0x11: The range's length is not a multiple of a page's size
  • 0x12: The range's size is null (0 bytes)
  • 0x30: This device is not registered in this process' drivable devices attribute
  • 0x31: The provided device KDI was not found
  • 0x32: The provided device is not compatible with DMA

0xA0 EXECUTION_CONTEXT

Get informations from the application's execution context.

Arguments:

  • Information to get (1 byte):

    • 0x00: all the context
    • 0x01: startup reason
    • 0x02: context header
    • 0x03: command-line arguments

  • Pointer to a writable buffer (8 bytes)

Return value:

  • Number of written bytes (8 bytes)

Errors:

  • 0x10: Invalid information number provided
  • 0x30: Caller process is a system service

0xD0 SYS_CREATE_PROCESS

Syscall resserved to the sys::process service.

Create a userland process.

Arguments:

Return value:

  • PID (8 bytes)

Errors:

  • 0x30: Caller process is not the sys::process service
  • 0x31: Code location token was not accepted by the sys::fs service
  • 0x32: Application context is not valid

0xD1 SYS_MANAGE_PROCESS

Syscall resserved to the sys::process service.

Manage a userland process.

Arguments:

Return value:

None

Errors:

  • 0x10: Invalid action code provided
  • 0x30: Caller process is not the sys::process service
  • 0x31: Unknown PID
  • 0x32: Process is already in the requested state

0xD2 SYS_PROCESS_ATTRIBUTES

Syscall resserved to the sys::process service.

Manage a process' attributes.

Arguments:

For value-based attributes:

  • Attribute code (1 byte):

    • 0x00: PID
    • 0x01: Process' priority
    • 0x02: Running user's ID
    • 0x03: Parent application ID
    • 0x04: Execution context (startup reason)
    • 0x05: Execution context (header)
    • 0x06: Execution context (arguments)
    • 0x07: Container ID

  • Action code:

For list-based attributes:

  • Attribute code (1 byte):

    • 0x00: Memory mappings
    • 0x01: Permissions
    • 0x02: Drivable devices

  • Action code (1 byte) followed by its optional arguments:

    • 0x00: Get the number of elements
    • 0x01: Get the value at a given index => index (4 bytes)
    • 0x02: Update the value at a given index => index (4 bytes) + value (? bytes)
    • 0x03: Insert a value at a given index => index (4 bytes) + value (? bytes)
    • 0x04: Push a value at the end of the list => value (? bytes)
    • 0x05: Remove a value from the list => index (4 bytes)
    • 0x06: Remove the last value from the list
    • 0x10: Check if the list contains a given item => value to look for (? bytes)

Return value:

  • Number of written bytes (if applies) (8 bytes)
  • 0x00 is all data was written, 0x01 otherwise (1 byte)

Errors:

  • 0x10: Invalid action code provided
  • 0x11: Invalid attribute number provided
  • 0x12: Asked to write a read-only attribute
  • 0x30: Caller process is not the sys::process service
  • 0x31: This system service is not allowed to access or edit this attribute
  • 0x32: Provided index is out-of-bounds

0xD4 SYS_ENUM_DEVICES

Syscall resserved to the sys::hw service.

List devices with a provided connection interface identifier (CII).

For each device, its raw device descriptor (RDD) (up to 260 bytes) is written to the provided address.

By prodiving a pattern with all bits set and a full CII, it is possible to retrieve informations from a single connection port.

Arguments:

  • Pattern type (1 byte):
    • Bit 0: match all connection types
    • Bit 1: match all buses
    • Bit 2: match all port numbers
  • CII pattern of the devices to list (4 bytes)
  • Writable buffer (16 bytes)

Return value:

  • Number of devices found with the provided criterias (4 bytes)
  • 0x00 is all devices were written, 0x01 otherwise (1 byte)

Errors:

  • 0x10: Invalid connection type in CII
  • 0x30: Caller process is not the sys::hw service

0xD5 CREATE_CONTAINER

Create a container.

Arguments:

None

Return value:

  • Container ID (8 bytes)

Errors:

  • 0x30: Caller process is not the sys::proc service.

Make a hardware device available to a previously-created container.

The device may be a real hardware device or a virtual one.

Arguments:

  • KDI of the device to link (8 bytes)

Return value:

None

Errors:

  • 0x30: Caller process is not the sys::proc service.
  • 0x31: Unknown KDI provided

0xD7 DESTROY_CONTAINER

Destroy a previously-created container as well as all its children.

Arguments:

  • Container ID (8 bytes)

Return value:

None

Errors:

  • 0x30: Caller process is not the sys::proc service.
  • 0x31: Unknown conatiner ID provided

Integration services

This document lists all integration services, which are services whose methods and notifications follow a convention established by the system itself to permit an integration at different levels.

An application exposing such a service in its manifest will automatically be assignable to specific tasks, such as being set as the default desktop environment or file manager.

These services are not available directly to the end applications ; they can only be used through system services.

Error codes

Error codes follow a specific convention, but all may not be returned by the services in case of errors.

Some error codes are only reserved to services supporting the additional check tied to the said error code.

List of integration services

Desktop Environments

The Desktop Environment (DE) is an application that acts as the main graphical interface of NightOS.

It is launched with the system, does never stop until the system itself stops, and handles most of the user interface.

The exposed service for desktop environments is SYS_DENV.

User interface concepts

User interface is technically free of any rule ; but it should usually follow the conventions described in the user experience document.

Window's state

In this document, a window's state is a data structure describing a window.

It is used both when getting informations about the window, or when changing them.

Custom elements are non-conventional elements handled by this specific desktop environment.

Note that each window has an identifier, used to access it or identify from notifications. These identifiers must either be different or scoped for each client process, as different clients should not be able to access one another's windows.

Header and content

The window's state is made of a header, containing 1 bit for each conventional element to get/update (listed below). Each bit must be set if the element is provided, unset otherwise.

It is followed by the number of custom elements to get/update (2 bytes), and the list of custom elements' identifiers (4 bytes per identifier).

The header is followed by the value of each element to update, only if the element must be updated.

Header bitDescriptionType / Size in bytesContent
0TitleString
1ButtonsButtons
2IconIcon
3Width2 bytesIn pixels
4Height2 bytesIn pixels
5X coordinate2 bytesIn pixels
6Y coordinate2 bytesIn pixels
7Active attribute1 byte0x01 to set the attribute
8Interactive attribute1 byte0x01 to set the attribute
9Fixed-size attribute1 byte0x01 to set the attribute
10Display layer1 byteLayer number
11Display stateDisplay state
12-31Future-proof

Delimited title strings

Delimited title strings are encoded as delimited strings.

Icons

Icons are encoded as bitmap images.

Buttons

  • Buttons count (1 byte)

Then, for each button:

  • Label's delimited title string
  • Label's icon (1 byte):
    • Bit 0: does the button have an icon?
    • Bit 1: is the button placed on the left? (else will be placed on the right)
    • Bit 2: should the icon be displayed in monochrome?
  • Label's icon
  • State (1 byte):
    • Bit 0: Is disabled?
    • Bit 1: Does it use a custom color?
  • Color (4 bytes for RGB and transparency) - filled with zeroes if custom color not set in state
  • Callback code (1 byte) - for internal use in the client

Display state

The display state is one value among:

  • 0x00: Restored (default)
  • 0x01: Maximized
  • 0x02: Minimized
  • 0x03: Fullscreenized
  • 0x04: Kiosk mode

Methods

0x0001 SUBSCRIBE_CAPABILITIES

Subscribe to change of capabilities listed by the CAPABILITIES method.

Arguments:

None

Answer:

None

Errors:

None

0x0002 CAPABILITIES

List elements which can be handled by the desktop environment.

For more informations about such elements, see the user experience document

Arguments:

None

Answer:

Bits set to 1 if the element is handled (supported) by the DE, 0 else. In each byte below, bits are listed from the strongest to the weakest.

Note that the bit should only be set if the desktop environment is currently able to handle the said element, which means that its display settings change, this answer's content will most probably change.

  • Global mechanisms (4 bytes)

    • Bit 0: Windows
    • Bit 1: Moving windows
    • Bit 2: Display layers
    • Bit 3: Pointing devices
    • Bit 4: Notifications
    • Bit 5: Notification center
    • Bit 6: Quick settings
    • Bit 7: Lockscreen
    • Bit 8: Taskbar
    • Bit 9: Dock
    • Bit 10: Desktop icons
    • Bit 11: Wallpaper
    • Bit 12: Customizable wallpaper
    • Bit 13: Popups
    • Bit 14: Is the interface customizable?
    • Remaining bits: Future-proof

  • Windows' outer interface (4 bytes)

    • Bit 0: Titlebar for windows
    • Bit 1: Custom title in titlebars
    • Bit 2: Custom icon in titlebars
    • Bit 3: Close button in titlebars
    • Bit 4: Customizable buttons in titlebars
    • Bit 5: Disabled buttons in titlebars
    • Bit 6: Custom colors for buttons in titlebars
    • Bit 7: Can windows be made non-interactive
    • Bit 8: Can windows be made fixed-size
    • Remaining bits: Future-proof

  • Windows' dynamic changes (4 bytes)

    • Bit 0: Windows' focus
    • Bit 1: Ability to move windows
    • Bit 2: Ability to resize windows
    • Bit 3: Ability to reorder windows in layers (superposition)
    • Bit 4: Ability to hide or minimize windows
    • Bit 5: Ability to restore and maximize windows
    • Bit 6: Ability to set the window to fullscreen
    • Bit 7: Ability to set the window to kiosk mode
    • Remaining bits: Future-proof

  • Windows' dynamic changes by the user (4 bytes)

    • Same as Windows' dynamic changes, but set only if the user can achieve the specified operation themselves

  • Popups' dynamic changes (4 bytes)

    • Bit 0: Popups' focus
    • Bit 1: Ability to move popups
    • Bit 2: Ability to resize popups
    • Bit 3: Ability to reorder popups in layers (superposition)
    • Bit 4: Ability to hide or minimize popups
    • Bit 5: Ability to restore and maximize popups
    • Bit 6: Ability to set the window to fullscreen
    • Bit 7: Ability to set the window to kiosk mode
    • Remaining bits: Future-proof

  • Popups' dynamic changes by the user (4 bytes)

    • Same as Popups' dynamic changes, but set only if the user can achieve the specified operation themselves

  • Notifications (4 bytes) - filled with zeroes if notifications are not supported

    • Basic notifications (4 bytes):
      • Bit 0: Support for custom icons
      • Remaining bits: Future-proof
    • Music player notifications (4 bytes):
      • Bit 0: Support for music player notifications
      • Bit 1: Support for track's title
      • Bit 2: Support for album's title
      • Bit 3: Support for artist's name
      • Bit 4: Support for album art
      • Bit 5: Support for artist art
      • Bit 6: Support for track's year
      • Bit 7: Support for track's day/month/year
      • Bit 8: Support for track's genre
      • Bit 9: Support for playlist's name
      • Bit 10: Support for progress bar
      • Bit 11: Support for codec
      • Bit 12: Support for frequency
      • Bit 13: Support for resolution
      • Bit 14: Support for bitrate
      • Bit 15: Support for visualizer
      • Bit 16: Support for displaying next title in queue
      • Bit 17: Support for current track/queue size display
      • Bit 18: Support for pause/resume button
      • Bit 19: Support for stop button
      • Bit 20: Support for rewind/fast-forward button
      • Bit 21: Support for previous/next buttons
      • Bit 22: Support for like/dislike button
      • Bit 23: Support for shuffle button
      • Remaining bits: Future-proof
    • Video player notifications (4 bytes):
      • Bit 0: Support for video player notifications
      • Bit 1: Support for video's title
      • Bit 2: Support for video's author
      • Bit 3: Support for video's source
      • Bit 4: Support for episode's number
      • Bit 5: Support for episode's serie name
      • Bit 6: Support for codec
      • Bit 7: Support for bitrate
      • Bit 8: Support for video's current track/queue size display
      • Bit 9: Support for pause/resume button
      • Bit 10: Support for stop button
      • Bit 11: Support for rewind/fast-forward button
      • Bit 12: Support for previous/next buttons
      • Bit 13: Support for like/dislike button
      • Bit 14: Support for shuffle button
      • Remaining bits: Future-proof

  • Integrations (4 bytes)

0x1000 CREATE_WINDOW

Create a new window.

Unset elements in the provided window's state will be replaced by default values or behaviours.

Desktop environment-dependant errors can only be used if the error does not fit any other error code.

If the window is successfully create, the client is automatically subscribed to WINDOW_CHANGED notifications for this specific window.

Arguments:

Answer:

Errors:

  • 0x1000: Invalid list of elements to create/update provided
  • 0x1010: Title must be provided
  • 0x1011: Title cannot be empty
  • 0x1012: Title is not a valid UTF-8 string
  • 0x1013: Unspecified error with the title
  • 0x1020: Custom buttons provided but it is not customizable
  • 0x1021: Too many custom buttons were provided
  • 0x1022: A custom button's label is not a valid UTF-8 string
  • 0x1023: A custom button's label cannot be empty
  • 0x1024: Custom button's icon provided but it is not customizable
  • 0x1025: Invalid custom button's icon buffer
  • 0x1026: A custom button's icon is too small
  • 0x1027: A custom button's icon is too large
  • 0x1028: Unspecified error with a custom button's icon
  • 0x1029: Unspecified error with a custom button's label
  • 0x102A: Custom button's color provided but it is not customizable
  • 0x102B: Custom button's color uses transparency but it is not customizable
  • 0x102C: Duplicate custom button's callback code
  • 0x1030: Custom icon provided but it is not customizable
  • 0x1031: Invalid custom icon buffer
  • 0x1032: Custom icon is too small
  • 0x1033: Custom icon is too large
  • 0x1034: Unspecified error with the provided custom icon
  • 0x1040: Window's width provided but it is not customizable
  • 0x1041: Window's width is too small
  • 0x1042: Window's width is too large
  • 0x1043: Window's height provided but it is not customizable
  • 0x1044: Window's height is too small
  • 0x1045: Window's height is too large
  • 0x1046: Window's X coordinate provided but it is not customizable
  • 0x1047: Window's X coordinate is too small
  • 0x1048: Window's X coordinate is too large
  • 0x1049: Forbidden window's X coordinate
  • 0x104A: Window's Y coordinate provided but it is not customizable
  • 0x104B: Window's Y coordinate is too small
  • 0x104C: Window's Y coordinate is too large
  • 0x104D: Forbidden window's Y coordinate
  • 0x104E: Invalid window's width/height ratio
  • 0x104F: Invalid window's X/Y coordinates couple
  • 0x1050: Active attribute provided but it is not customizable
  • 0x1051: Invalid active attribute provided
  • 0x1052: Cannot apply the custom active attribute for unspecified reasons
  • 0x1053: Interactive attribute provided but it is not customizable
  • 0x1054: Invalid interactive attribute provided
  • 0x1055: Cannot apply the custom interactive attribute for unspecified reasons
  • 0x1056: Display layer provided but it is not customizable
  • 0x1057: Display layer is too low
  • 0x1058: Display layer is too high
  • 0x1059: Forbidden display layer
  • 0x105A: Display state provided but it is not customizable
  • 0x105B: Invalid display state provided
  • 0x105C: Cannot set this specific display state
  • 0x1060: Provided custom elements but this DE does not have any custom one
  • 0x1100 to 0x11FF: Errors for custom elements
  • 0x3000: Client is not allowed to create/update a window
  • 0x3100 to 0x31FF: Errors for custom elements
  • 0x3200 to 0x32FF: Desktop environment-dependant errors
  • 0x4000: Failed to create/update the window
  • 0x4100 to 0x41FF: Errors for custom elements
  • 0x4200 to 0x42FF: Desktop environment-dependant errors

0x1001 UPDATE_WINDOW

Update an existing window.

Arguments:

Return value:

None

Errors:

Same errors as the CREATE_WINDOW method, plus 0x3FFF for unknown window identifier.

0x1002 GET_WINDOW_STATE

Get informations about an existing window.

Arguments:

Return value:

Errors:

  • 0x2000: Invalid list of elements to create/update provided
  • 0x3000: Unknown window identifier

0x10FF DESTROY_WINDOW

Destroy an existing window.

Arguments:

  • Window identifier (8 bytes)
  • Skip the WINDOW_CLOSED (1 byte): 0x01 to skip the notification, 0x00 else

Return value:

None

Errors:

  • 0x1000: Invalid skip mode provided
  • 0x3000: Unknown window identifier
  • 0x3001: Failed to destroy the window
  • 0x3100 to 0x31FF: Desktop environment-dependant errors
  • 0x4100 to 0x41FF: Desktop environment-dependant errors

0x2000 CREATE_POPUP

Create a popup.

Arguments:

Same structure than for a window's state, but with the following common elements:

Header bitDescriptionType / length in bytes
0TitleString
1IconIcon
2ButtonsButtons

Return value:

  • Popup identifier (8 bytes)

Errors:

  • 0x1000: Invalid list of elements provided
  • 0x1010: Title must be provided
  • 0x1012: Title cannot be empty
  • 0x1013: Title is not a valid UTF-8 string
  • 0x1014: Unspecified error with the title
  • 0x1020: Custom buttons provided but it is not customizable
  • 0x1021: Too many custom buttons were provided
  • 0x1022: A custom button's label is not a valid UTF-8 string
  • 0x1023: A custom button's label cannot be empty
  • 0x1024: Custom button's icon provided but it is not customizable
  • 0x1025: Invalid custom button's icon buffer
  • 0x1026: A custom button's icon is too small
  • 0x1027: A custom button's icon is too large
  • 0x1028: Unspecified error with a custom button's icon
  • 0x1029: Unspecified error with a custom button's label
  • 0x102A: Custom button's color provided but it is not customizable
  • 0x102B: Custom button's color uses transparency but it is not customizable
  • 0x102C: Duplicate custom button's callback code
  • 0x1030: Custom icon provided but it is not customizable
  • 0x1031: Invalid custom icon buffer
  • 0x1032: Custom icon is too small
  • 0x1033: Custom icon is too large
  • 0x1034: Unspecified error with the provided custom icon
  • 0x3000: Client is not allowed to create a popup
  • 0x4000: Failed to create/update the popup

0x20FF DESTROY_POPUP

Destroy an existing popup.

Arguments:

  • Popup identifier (8 bytes)
  • Skip the POPUP_CLOSED (1 byte): 0x01 to skip the notification, 0x00 else

Return value:

None

Errors:

  • 0x1000: Invalid skip mode provided
  • 0x3000: Unknown popup identifier
  • 0x4000: Failed to destroy the popup

0x3000 SEND_BASIC_NOTIFICATION

Send a basic notification, "basic" meaning a non-special notification.

Arguments:

Subset of a window's state: title, icon, buttons.

Return value:

  • Notification identifier (8 bytes)

Errors:

Relevant subset of the CREATE_WINDOW method's errors.

0x3001 SEND_MUSIC_PLAYER_NOTIFICATION

Send a music player notificaiton.

Arguments:

Same structure than for a window's state, but with the following common elements:

Header bitDescriptionType / length in bytesContent
0Track's titleString
1Album's titleString
2Artist's nameString
3Album artIcon
4Artist artIcon
3Track's year2 bytes
4Track's date4 bytesDay (1 byte) ; Month (1 byte); Year (2 bytes)
5Track's genreString
6Playlist's nameString
7Progress time8 bytesElapsed time in seconds (4 bytes) ; Total time in seconds (4 bytes)
8Pause state1 byte0x01 for active playback, 0x02 for paused, 0x03 for stopped
9Like state1 byte0x01 if liked, 0x02 if disliked, 0x00 if not liked nor disliked
10CodecString
11Frequency4 bytesValue in hertz (Hz)
12Resolution2 bytesValue in bits
13Bitrate4 bytesValue in bits/second
14Visualizer informations24 bytesOne byte per frequency range (see below)
18Next track's nameString
19Track position8 bytesTrack index (4 bytes) ; Tracks in the queue (4 bytes)
20Buttons availibility1 bytePause/resume ; Rewind/Fast-forward ; Previous track ; Next track ; Shuffle

The visualizer shows a value from 0 to 255 to indicate the bar eight. Each bar value corresponds to a specific frequency range. The lower frequency bound of each bar can be found with: exponential(i / 1.88) - 1, where i is the bar number, from 0 to 23.

Return value:

  • Notification identifier (8 bytes)

Errors:

  • 0x1000: Invalid list of elements to create/update provided
  • 0x1001: Track's title is not a valid UTF-8 string
  • 0x1002: Album's title is not a valid UTF-8 string
  • 0x1003: Artist's name is not a valid UTF-8 string
  • 0x1004: Invalid pause state
  • 0x1005: Invalid like state
  • 0x1006: Next track's title cannot be empty
  • 0x1007: Next track's title is not a valid UTF-8 string
  • 0x1008: Button avaibility bit provided for unknown button
  • 0x1100 to 0x11FF: Errors for custom elements
  • 0x3000: Audio player notifications are not supported
  • 0x3001: Client is not allowed to create/update an audio player notification
  • 0x3100 to 0x31FF: Errors for custom elements
  • 0x3200 to 0x32FF: Desktop environment-dependant errors
  • 0x4000: Failed to create/update the notification
  • 0x4100 to 0x41FF: Errors for custom elements
  • 0x4200 to 0x42FF: Desktop environment-dependant errors

0x3002 SEND_VIDEO_PLAYER_NOTIFICATION

Send a video player notificaiton.

Arguments:

Same structure than for a window's state, but with the following common elements:

Header bitDescriptionType / length in bytesContent
0Video's titleString
1Video's authorString
2Video's sourceString
3Episode's number4 bytesCurrent episode (2 bytes) ; Total episodes in the season (2 bytes)
4Episode's serie nameString
5CodecString
6Bitrate4 bytesValue in bits/second
7Video position8 bytesVideo index (4 bytes) ; Videos in the queue (4 bytes)
8Buttons availibility1 bytePause/resume ; Rewind/Fast-forward ; Previous track ; Next track ; Shuffle

Return value:

  • Notification identifier (8 bytes)

Errors:

  • 0x1000: Invalid list of elements to create/update provided
  • 0x1001: Video's title is not a valid UTF-8 string
  • 0x1002: Video's author is not a valid UTF-8 string
  • 0x1003: Video's source is not a valid UTF-8 string
  • 0x1004: Codec is not a valid UTF-8 string
  • 0x1100 to 0x11FF: Errors for custom elements
  • 0x3000: Audio player notifications are not supported
  • 0x3001: Client is not allowed to create/update an audio player notification
  • 0x3100 to 0x31FF: Errors for custom elements
  • 0x3200 to 0x32FF: Desktop environment-dependant errors
  • 0x4000: Failed to create/update the notification
  • 0x4100 to 0x41FF: Errors for custom elements
  • 0x4200 to 0x42FF: Desktop environment-dependant errors

Notifications

0x0001 CAPABILITIES_CHANGED

Sent to clients that subscribed through the SUBSCRIBE_CAPABILITIES method when the elements the desktop environment can handle changed (for instance when the taskbar is hidden, when the windows' layers become fixed, ...).

Datafield:

Equivalent content to the CAPABILITIES' answer.

0x1000 WINDOW_CHANGED

Notification sent when a window's state changed.

Only the elements that changed since the last notification will be set in the sent window's state.

Datafield:

0x1001 WINDOW_INTERACTION

Sent when the user interacts with a window. These events should not occur if the window is not active

Datafield:

  • Window identifier (8 bytes)

This is followed by the interaction type on 1 byte, then by the interaction's data, which can be either:

  • 0x00: Pointing device event

    • Type (1 byte):
      • 0x00: Pointer moved
      • 0x01: Left click pressed
      • 0x02: Left click released
      • 0x03: Middle click pressed
      • 0x04: Middle click released
      • 0x05: Right click pressed
      • 0x06: Right click released
    • X coordinate in pixels (2 bytes) (relative to the window's drawing zone left side)
    • Y coordinate in pixels (2 bytes) (relative to the window's drawing zone top side)

  • 0x01: Scrolling

    • Direction (1 byte):
      • 0x00: Up
      • 0x01: Down
      • 0x02: Left
      • 0x03: Right
      • 0x04: Shake

  • 0x02: Keyboard event

    • Hold type (1 byte):
      • 0x00: Key was pressed
      • 0x11: Key was released
    • Key code (1 byte), normalized by the sys::hw service

0x10FF WINDOW_CLOSED

Datafield:

  • Window identifier (8 bytes)

0x2000 POPUP_INTERACTION

Sent when the user selects a button in the list provided by the popup.

Datafield:

  • Popup identifier (8 bytes)
  • Callback code (1 byte)

0x20FF POPUP_CLOSED

Sent when a popup was closed by the end user.

Datafield:

  • Popup identifier (8 bytes)

0x3000 BASIC_NOTIFICATION_INTERACTION

Sent when the user interacts with a basic notification.

Datafield:

  • Callback code (1 byte)

0x3001 AUDIO_PLAYER_NOTIFICATION_INTERACTION

Sent when the user interacts with an audio player notification.

Datafield:

  • Button code (1 byte):
    • 0x00: Pause/Resume
    • 0x01: Stop
    • 0x02: Rewind
    • 0x03: Fast-forward
    • 0x04: Previous
    • 0x05: Next
    • 0x06: Like
    • 0x07: Dislike
    • 0x08: Like/dislike toggle
    • 0x09: Shuffle

0x3002 VIDEO_PLAYER_NOTIFICATION_INTERACTION

Sent when the user interacts with a video player notification.

Datafield:

  • Button code (1 byte):
    • 0x00: Pause/Resume
    • 0x01: Stop
    • 0x02: Rewind
    • 0x03: Fast-forward
    • 0x04: Previous
    • 0x05: Next
    • 0x06: Like
    • 0x07: Dislike
    • 0x08: Like/dislike toggle
    • 0x09: Shuffle

0x30FF NOTIFICATION_CLOSED

Sent a notification is closed.

Datafield:

  • Notification identifier (8 bytes)

File Manager

A file manager is an application which can manage the filesystem's content. It does not have any restriction on its user interface, but needs to expose an integration service with the standardized methods and notifications described in this document.

Applications can indicate themselves as file manager by specifying an SYS_FMAN service in their manifest. They rely on file openers to open files.

The end user chooses a single file manager (called the default file manager) between all available ones, whose service will be used by other applications.

Methods

0x1000 OPEN_ITEM

Open a filesystem item.

Arguments:

Answer:

None

Errors:

  • 0x3000: Invalid FMP
  • 0x3001: Could not find the provided item
  • 0x4000: User cancelled the opening
  • 0x4001: Could not find an application to open the provided item
  • 0x4002: Failed to open the provided item due to an I/O error
  • 0x4FFF: Unspecified error

0x1001 IS_ITEM_OPENABLE

Check if a filesystem item could be opened without user interaction.

Arguments:

Answer:

  • 0x02 if the file could be opened without user interaction, 0x01 if it couldn't, 0x00 if the file couldn't be opened at all

Errors:

  • 0x3000: Invalid FMP
  • 0x3001: Could not find the provided item
  • 0x4000: Unspecified error

0x2000 GET_THUMBNAIL

Get the thumbnail for a specific item.

The thumbnail should be generated using the sys::fsh system service, which will provide the cached thumbnail (if any) and else ask for a thumbnail buffer, which will be put in the cache if relevant.

The thumbnail can be requested to have a specific width and height. The returned thumbnail may not have the correct size, but this will require the system to resize the picture.

Arguments:

  • Refresh mode (1 byte): 0x00 to get the current thumbnail or a cached one, 0x01 to force generating a new thumbnail for the item
  • Filesystem path
  • Requested thumbnail width (2 bytes) and height (2 bytes) in pixels, or 0 if no request

Answer:

Errors:

  • 0x3000: Invalid FMP
  • 0x3001: Could not find the provided item
  • 0x4000: The thumbnail generator failed
  • 0x4FFF: Unspecified error

0x2100 GET_VIDEO_PREVIEW

Get the video preview for a specific item.

The preview should be generated using the sys::fsh system service, which will provide the cached preview (if any) and else ask for a preview buffer, which will be put in the cache if relevant.

The preview can be requested to have a specific width and height. The returned preview may not have the correct size, but this will require the system to resize the video.

Arguments:

  • Refresh mode (1 byte): 0x00 to get the current preview or a cached one, 0x01 to force generating a new preview for the item
  • Filesystem path
  • Requested preview width (2 bytes) and height (2 bytes) in pixels, or 0 if no request

Answer:

Errors:

  • 0x3000: Invalid FMP
  • 0x3001: Could not find the provided item
  • 0x4000: The preview generator failed
  • 0x4FFF: Unspecified error

0xA000 CONTEXT_MENU

Generate a context menu for a specific filesystem item. Used by the DEA.

Arguments:

Answer:

Errors:

  • 0x3000: Invalid FMP
  • 0x3001: Could not find the provided item
  • 0x4000: User cancelled the opening
  • 0x3001: Could not find an application to open the provided item
  • 0x4002: Failed to open the provided item due to an I/O error
  • 0x4FFF: Unspecified error

File openers

A file opener is an application which can associate open specific items. It exposes an integration service with the standardized methods and notifications described in this document.

The list of openers can be viewed and modified through the sys::fsh.

They are essentially used by the default file manager.

Methods

0x1000 OPEN_ITEM

Open a filesystem item.

Arguments:

Answer:

None

Errors:

  • 0x3000: Invalid FMP
  • 0x3001: Could not find the provided item
  • 0x3002: User cancelled the opening
  • 0x3003: Could not find an application to open the provided item
  • 0x3004: Failed to open the provided item due to an I/O error
  • 0x3FFF: Unspecified error

0x2000 GENERATE_THUMBNAIL

Generate a thumbnail for a given filesystem item.

The thumbnail should be generated using the sys::fsh system service, which will provide the cached thumbnail (if any) and else ask for a thumbnail buffer, which will be put in the cache if relevant.

Arguments:

Answer:

Errors:

  • 0x3000: Invalid FMP
  • 0x3001: Could not find the provided item
  • 0x3002: This application does not support thumbnail generation for this item type
  • 0x3003: Thumbnail could not be generated from the item as its content is invalid
  • 0x3FFF: Unspecified error

0x2001 CAN_GENERATE_THUMBNAIL

Check if the application can generate a thumbnail for a specific filesystem item.
If it cannot, the default file manager will be in charge of providing a placeholder or using an alternative thumbnail generator.

Arguments:

Answer:

  • 0x01 if the application can generate a thumbnail for the provided filesystem item, 0x00 else

Errors:

  • 0x3000: Invalid FMP
  • 0x3001: Could not find the provided item
  • 0x3002: Thumbnail could not be generated from the item as its content is invalid
  • 0x3FFF: Unspecified error

Filesystem interfaces

Filesystem interfaces are services that act as a layer of indirection between a storage driver and the sys::fs service. Their role is to translate all common filesystem operations, such as items management and modification, for filesystems that aren't natively supported. They heavily communicate with the sys::hw service to communicate with the underlying storage device.

A filesystem is tied to only one partition, and each partition has a filesystem. A storage space is made, unless empty, of a partitions table which lists all the available partitions.

Most methods and notifications of filesystems interfaces are also available in the sys::fs service, with first perform permissions checking and determines the device and location to perform the operation on. It also leverages many more powerful features such as items watching, and features that are our of the scope of a filesystem interface such as partitions management.

As an application can only expose one single filesystem interface service, the said service will handle all of the hardware devices driven by the said process. This is why the the SOR of the target partition and filesystem is provided for each request.

Nomenclature

Storage operating range

The storage operating range (SOR) is a 40-byte information transmitted to all operation methods of the filesystem interface to indicate where the requested operation should be performed and on which devices.

It is composed of:

  • The device's UDI (8 bytes)
  • The partition's first byte address on the device (8 bytes)
  • The partition's last byte address on the device (8 bytes)
  • Authorization token (8 bytes)
  • The filesystem's FSID (8 bytes)

The FSID is only useful to identify partitions more easily, and to inform clients in notifications.

All SOR sent to the interface service are guaranteed to follow this specification, but may be invalid in case of a bad timing (e.g. authorization is revoked just before a request is performed).

Filesystem paths

Filesystem paths are used to refer to specific filesystem elements and are encoded as either a 0x01 byte followed by a FEID, or a 0x02 byte followed by a split path.

All filesystem paths sent to the interface service are guaranteed to follow this specification.

Split paths

All paths manipulated and returned by a filesystem interface services are encoded as split paths: delimited lists of delimited strings.

Each entry of the list must be a path's component. The entries, joined with a slash (/), should form a valid path.

Empty and . components are ignored, while components equal to .. will make this component ignored and remove the previous component in the list.

The slash (/) character is forbidden in components.

An empty path refers to the root directory (when applicable).

Filenames

Filenames are a delimited strings with a few restrictions:

  • Slash / characters are forbidden
  • NULL characters (U+0000) are forbidden

Moreover, filenames cannot be:

  • Empty
  • Be made only of spaces or insecable spaces
  • Strictly equal to .
  • Strictly equal to ..
  • Longer than 65535 characters

Additional limitations may apply for some filesystems. Refer to each filesystem's specification to see the eventual limitations on filenames.

All filenames sent to the interface service are guaranteed to be follow this specification.

Capabilities list

The capabitilies list is a 2-byte long value which indicate the capabitilies of a given filesystem:

  • Bit 0: set if the filesystem is writable
  • Bit 1: set if the filesystem handles symbolic links
  • Bit 2: set if the filesystem allows symbolic links to non-existing items
  • Bit 3: set if the filesystem allows symbolic links to cross-filesystem items (need to store the target's FSID)
  • Bit 3: set if the filesystem supports hidden flag
  • Bit 4: set if the filesystem supports readonly flag
  • Bit 5: set if the filesystem can store creation dates
  • Bit 6: set if the filesystem can store modification dates
  • Bit 7: set if the filesystem can store last access dates
  • Bit 8: set if the filesystem can store an owner UID on 8 bytes
  • Bit 9: set if the filesystem can natively store a storage permissions map
  • Bit 10: set if the interface can store a storage permissions map in the filesystem
  • Bit 11: set if the filesystem supports ahead space reservation

In case the filesystem can't natively handle a storage permissions map (SPM), the interface is allowed to store, query and update the map in the filesystem by itself, using non-native ways such as extended attributes. In such case, the relevant bit must be set to indicate a SPM can be stored on this filesystem.

This list acts as a contract between the service and the sys::fs service ; if a capability bit is not set, the related methods are guaranteed to never be called. If it is set, the "incompatibility" error code present in such methods cannot be returned or it will be considered a bug.

Filesystem metadata

A filesystem metadata is a structure describing the a single filesystem:

  • Capabilities list (2 bytes)
  • Cluster size in bytes (4 bytes)
  • Partition's total size in bytes (8 bytes)
  • Partition's used space in bytes (8 bytes)
  • Partition's free space in bytes (8 bytes) - used in case a part of the partition could not be used for any reason
  • Optional partition's name as a delimited string
  • Optional partition's icon

Item type byte

An item type byte is a single-byte value describing the type of a single item in a filesystem:

  • 0x01: directory
  • 0x02: file
  • 0x03: symbolic link
  • 0xFF: unknown item type

Item flags

Each item has associated flags:

  • Bit 0: hidden flag
  • Bit 1: read-only flag

Item metadata

An item metadata is a structure describing the content of a single item in a filesystem:

Methods

0x0001 HANDLED_FS_LIST

Get informations about each filesystem handled by the current interface.

Arguments:

None

Answer:

Errors:

none

0x0002 IS_VALID_PARTITION

Check if a given partition is of the type of filesystem handled by the current interface.

Arguments:

  • First bytes of the partition (512 bytes)
  • Last bytes of the partition (512 bytes)

Answer:

  • Identifier of the filesystem if it can be handled by the current interface, 0 otherwise (1 byte)

Errors:

None

0x0003 FS_METADATA

Get metadata on a given filesystem.

Arguments:

  • SOR (40 bytes)

Return value:

Error codes:

  • 0x3000: Invalid SOR provided

0x1000 ITEM_EXISTS

Check if a given item exists.

Arguments:

Return value:

Errors:

  • 0x3000: Invalid SOR provided

0x1001 FEID_TO_SPLIT

Convert a FEID to the corresponding split path.

Arguments:

Return value:

Errors:

  • 0x3000: Invalid SOR provided
  • 0x3001: The provided FEID was not found in the filesystem

0x1002 ITEM_METADATA

Get the metadata of a given item.

Arguments:

Return value:

Errors:

  • 0x3000: Invalid SOR provided
  • 0x3001: The provided path was not found

0x1003 RENAME_ITEM

Rename an existing item.

Arguments:

Errors:

  • 0x3000: Invalid filename provided
  • 0x3001: Invalid SOR provided
  • 0x3002: The provided path was not foud

0x1004 MOVE_ITEM

Move an existing item.

Arguments:

Errors:

  • 0x1000: Invalid filename provided
  • 0x3000: Invalid SOR provided
  • 0x3001: The provided path was not found
  • 0x3002: Target directory was not found
  • 0x4000: Target directory's maximum capacity has been reached
  • 0x4001: Maximum nested items number has been reached
  • 0x4002: Maximum path length has been reached
  • 0x4003: Item cannot be moved for unspecified reasons
  • 0x4FFF: Unspecified filesystem error

0x1005 DELETE_ITEM

Delete an item.

Arguments:

Return value:

None

Errors:

  • 0x3000: Invalid SOR provided
  • 0x3001: Item was not found
  • 0x3002: Cannot remove a non-empty directory
  • 0x4FFF: Unspecified filesystem error

0x2000 CREATE_DIRECTORY

Create a directory.

Arguments:

Return value:

None

Errors:

  • 0x1000: Invalid filename provided
  • 0x3000: Invalid SOR provided
  • 0x3001: Parent directory was not found
  • 0x4000: Directory's maximum capacity has been reached
  • 0x4001: Maximum nested items number has been reached
  • 0x4002: Maximum path length has been reached
  • 0x4FFF: Unspecified filesystem error

0x2001 READ_DIRECTORY

List all entries in a directory.

If the provided offset is larger than the number of entries in the directory, all remaining items must be returned.
If the number of items to get is larger than the number of entries in the directory less the start offset, all remaining items must be returned.

Arguments:

  • SOR (40 bytes)
  • Filesystem path
  • Start offset (8 bytes)
  • Number of items to get (8 bytes) - 0 to list all items at once
  • Optional total number of entries, if available (1 + 8 bytes)
  • Hidden flag match (1 byte): set to 0x00 to match only non-hidden items, 0x01 to match all items

Return value:

Errors:

  • 0x3000: Invalid SOR provided
  • 0x3001: Directory was not found
  • 0x4FFF: Unspecified filesystem error

0x3000 CREATE_FILE

Create a file.

The reserved size is only provided if the interface reported the filesystem as supporting this feature.

Arguments:

Return value:

None

Errors:

  • 0x1000: Invalid filename provided
  • 0x3000: Invalid SOR provided
  • 0x3001: Parent directory was not found
  • 0x4000: Directory's maximum capacity has been reached
  • 0x4001: Maximum nested items number has been reached
  • 0x4002: Maximum path length has been reached
  • 0x4003: Storage's capacity exceeded
  • 0x4004: Maximum individual file size exceeded
  • 0x4005: Filesystem's free space exceeded
  • 0x4FFF: Unspecified filesystem error

0x3001 READ_FILE_SYNC

Read a file synchronously.

If no read length is provided, the whole file must be read.

Arguments:

  • SOR (40 bytes)
  • File's path
  • Start offset address (8 bytes)
  • Optional length to read (1 + 8 bytes)

Return value:

  • Number of read bytes (8 bytes)
  • File's content

Errors:

  • 0x3000: Invalid SOR provided
  • 0x3001: Start offset is out-of-range
  • 0x4FFF: Unspecified filesystem error

0x3002 READ_FILE_ASYNC

Asynchronously read a file to a writable abstract memory segment (AMS).

THe number of bytes to read is always provided to ensure it does not accidentally exceed the AMS's size.

When the read is complete, a FILE_READ notification must be sent to the client.

Arguments:

  • SOR (40 bytes)
  • File's path
  • Start offset address (8 bytes)
  • Number of bytes to read (8 bytes)
  • AMS identifier (8 bytes)

Return value:

  • Generated task identifier (8 bytes)

Errors:

  • 0x3000: Invalid SOR provided
  • 0x3001: Invalid AMS ID provided
  • 0x4FFF: Unspecified filesystem error

0x3003 WRITE_FILE_SYNC

Synchronously write a buffer to a file.

If no offset address is provided, the file's content must be completely overriden with the provided buffer.

Arguments:

  • SOR (40 bytes)
  • File's path
  • Optional write offset address (1 + 8 bytes)
  • Number of bytes to write (8 bytes)
  • Buffer to write (8 bytes)

Return value:

None

Errors:

  • 0x3000: Invalid SOR provided
  • 0x3001: Offset is out-of-range
  • 0x4000: Storage's capacity exceeded
  • 0x4001: Maximum individual file size exceeded
  • 0x4002: Filesystem's free space exceeded
  • 0x4FFF: Unspecified filesystem error

0x3004 WRITE_FILE_ASYNC

Asynchronously write a readable abstract memory segment (AMS) to a file.

If no offset address is provided, the file's content must be completely overriden with the provided buffer.

When the writing is complete, a FILE_WRITTEN notification must be sent to the client. The notification must not be sent before the current method returned successfully.

Arguments:

  • SOR (40 bytes)
  • File's path
  • Optional write offset address (1 + 8 bytes)
  • Number of bytes to write (8 bytes)
  • AMS identifier (8 bytes)

Return value:

  • Generated task identifier (8 bytes)

Errors:

  • 0x3000: Invalid SOR provided
  • 0x3001: Invalid AMS ID provided
  • 0x4FFF: Unspecified filesystem error

Create a symbolic link.

Arguments:

Return value:

None

Errors:

  • 0x1000: Invalid filename provided
  • 0x3000: Invalid SOR provided
  • 0x3001: Parent directory was not found
  • 0x3002: Cannot create symbolic links to cross-filesystem items
  • 0x3003: Cannot create symbolic links to non-existing items
  • 0x4000: Directory's maximum capacity has been reached
  • 0x4001: Maximum nested items number has been reached
  • 0x4002: Maximum path length has been reached
  • 0x4003: Storage's capacity exceeded

Create a symbolic link.

Arguments:

Return value:

None

Errors:

  • 0x3000: Invalid SOR provided
  • 0x3001: Provided path was not found
  • 0x3002: Cannot crate symbolic links to cross-filesystem items
  • 0x3003: Cannot crate symbolic links to non-existing items

Read a symbolic link's target.

Arguments:

Return value:

Errors:

  • 0x3000: Invalid SOR provided
  • 0x3001: Provided path was not found
  • 0x4000: Symbolic link is cyclic

0xF000 FORMAT_ASYNC

Asynchronously format the partition to get an empty filesystem. Once the formatting is complete,

Arguments:

  • SOR (40 bytes)
  • Optional new partition's sector size, in bytes (8 bytes)

Return value:

  • Generated task identifier (8 bytes)

Error codes:

  • 0x1000: Invalid sector size provided
  • 0x3000: Invalid SOR provided

Notifications

0x3002 FILE_READ

Sent to a client after an asynchronous file reading requested using the READ_FILE_ASYNC method completed.

Datafield:

  • Task identifier (8 bytes)
  • Fallible result with:
    • Success data: number of bytes read (8 bytes)
    • Error code (1 byte)
      • 0x20: Start offset is out-of-range
      • 0x40: Unspecified filesystem error

0x3004 FILE_WRITTEN

Sent to a client after an asynchronous file writing requested using the WRITE_FILE_ASYNC method completed.

Datafield:

  • Task identifier (8 bytes)
  • Fallible result with:
    • Success data: None
    • Error code (1 byte):
      • 0x20: Start offset is out-of-range
      • 0x31: Maximum individual file size exceeded
      • 0x32: Filesystem's free space exceeded
      • 0x40: Unspecified filesystem error

0xF000 FORMATTED

Sent to a client after an formatting requested using the FORMAT_ASYC method completed.

Datafield:

  • Task identifier (8 bytes)
  • Fallible result with:
    • Success data: None
    • Error code (1 byte):
      • 0x40: Unspecified filesystem error

Driver services

This document lists all driver services, which are services whose methods and notifications follow a convention established by the sys::hw service to permit interaction with hardware devices.

An application exposing such a service in its manifest will gain the permission to register itself as a driver through the sys::hw service.

These services are not available directly to the end applications ; they can only be used through system services.

Error codes

Error codes follow a specific convention, but all may not be returned by the services in case of errors.

Some error codes are only reserved to services supporting the additional check tied to the said error code.

Also, all drivers can use:

  • The 0x30 error code to indicate the provided UDI is invalid or not driven by the current driver service
  • The 0x5F error code to indicate an unspecified hardware error occurred.

List of driver services

The list below indicates the service as well as the DDT it is tied to.

Storage drivers

Storage driver services

Storage driver services handle storage devices. They are used to perform basic I/O operations.

As an application can only expose one single storage driver service, the said service will handle all of the hardware devices it drives. This is why the the UDI of the target device is provided for each request.

Methods

0x1001 STORAGE_INFOS

Get the storage's informations.

Arguments:

  • Device's UDI (8 bytes)

Return value:

  • Storage's size in bytes (8 bytes)
  • Sector's size in bytes (8 bytes)
  • Writability (1 byte): 0x00 if the storage is writable, 0x01 else

Errors:

None

0x1100 READ_SYNC

Synchronously read a suite of sectors.

Arguments:

  • Start sector number (8 bytes)
  • Number of sectors to read (8 bytes)

Return value:

  • Number of bytes read (8 bytes)
  • Content of the read sectors

Errors:

  • 0x3000: Start sector is out-of-bounds
  • 0x3001: End sector is out-of-bounds

0x1101 READ_ASYNC

Asynchronously read a suite of sectors to a writable abstract memory segment (AMS).

When the task completes, the SECTORS_READ notification must be sent to the client.

Arguments:

  • Start sector number (8 bytes)
  • Number of sectors to read (8 bytes)
  • AMS identifier (8 bytes)

Return value:

  • Generated task identifier (8 bytes)

Errors:

  • 0x3000: Start sector is out-of-bounds
  • 0x3001: End sector is out-of-bounds

0x1200 WRITE_SYNC

Synchronously write a suite of sectors.

The content to write is guaranteed to be aligned with the sectors' size.

Arguments:

  • Start sector number (8 bytes)
  • Number of bytes to write (8 bytes)
  • Content to write

Return value:

None

Errors:

  • 0x3000: Start sector is out-of-bounds
  • 0x3001: End sector is out-of-bounds

0x1201 WRITE_ASYNC

Asynchronously write a suite of sectors from an abstract memory segment (AMS).

When the task completes, the SECTORS_WRITTEN notification must be sent to the client.

The content to write is guaranteed to be aligned with the sectors' size.

Arguments:

  • Start sector number (8 bytes)
  • Number of bytes to write (8 bytes)
  • AMS identifier (8 bytes)

Return value:

  • Generated task identifier (8 bytes)

Errors:

  • 0x3000: Start sector is out-of-bounds
  • 0x3001: End sector is out-of-bounds

Notifications

0x1101 SECTORS_READ

Sent to a client after an asynchronous sectors reading requested using the READ_ASYNC method completed.

Datafield:

  • Task identifier (8 bytes)
  • Fallible result with:
    • Success data: number of bytes read (8 bytes)
    • Error code (2 bytes): None

0x1201 SECTORS_WRITTEN

Sent to a client after an asynchronous sectors writing requested using the WRITE_ASYNC method completed.

Datafield:

  • Task identifier (8 bytes)
  • Fallible result with:
    • Success data: None
    • Error code (2 bytes): None

System services

As NightOS' kernel is not monolithic but a microkernel, it only handles the most fundamental tasks of the system, like memory and processes management, as well as direct hardware communication.

The vast majority of its features can be found in system services, run by the system itself under the sys DID.

This splitting implies that most low-level features of the system are documented in the individual services' specifications documents, which you will find here.

Nomenclature

System services are referred to as the sys:: services.

All methods and notifications describe the required permissions to use them, their arguments.

They also use common error codes:

  • 0x00: cannot read syscall's code or arguments (error while reading memory)
  • 0x01: the requested syscall does not exist
  • 0x02: at least one argument is invalid (e.g. providing a pointer to the 0 address)
  • 0x03: unmapped memory pointer (e.g. provided a pointer to a memory location that is not mapped yet)
  • 0x04: memory permission error (e.g. provided a writable buffer to an allocated but non-writable memory address)
  • 0x05: insufficient permissions
  • 0x06: driver error
  • 0x10 to 0x1F: invalid arguments provided (e.g. value is too high)
  • 0x20 to 0x2F: arguments are not valid in the current context (e.g. provided ID does not exist)
  • 0x30 to 0x3F: resource errors (e.g. file not found)
  • 0x40 to 0x4F: driver errors
  • 0x50 to 0x5F: hardware errors
  • 0x60 to 0xFF: other types of errors

All methods return an answer, though it may be empty (indicated by a None). System services' answers always conclude the exchange.

List of system services

sys::fs service

The sys::fs service is in charge of operations related to the filesystems.

Behaviour

Operations and latency

A single filesystem operation request from a client process up to the hardware device traverses:

The response then goes up through all layers. Note that in all cases, the sys::hw don't need to be contacted, thanks to direct driver access.

In the best scenario, which is for natively supported filesystems on storage devices that don't require a dedicated driver, direct storage access is possible, reducing the traversal to:

  • Client
  • sys::fs service
  • Hardware storage device

List of natively supported filesystems

The following filesystems are natively supported, meaning they don't require a filesystem interface to work properly:

  • Btrfs
  • Ext2 / Ext3 / Ext4
  • NTFS
  • FAT12 / FAT16 / FAT32
  • exFAT

Extending supported filesystems

It is possible to use other filesystems that the natively supported ones, using filesystem interfaces.

This, however, creates a higher latency as direct access and operations are not permitted anymore. The typical sequence of operations becomes:

  • The client calls the sys::fs service to perform a given operation
  • The operation is transmitted to the related filesystem interface...
  • ...which in turns contact the sys::hw service...
  • ...which itself transmits the operation to the underlying storage driver

The information then goes up:

  • From the driver to sys::hw
  • Then to the filesystem interface which translates it
  • Then back to the sys::fs service
  • And finally to the client

Filesystems detection

The sys::fs serviec is responsible for detecting filesystems. It performs this by contacting the sys::hw service to enumerate and access the different storage devices, as well as being notified when a storage device is connected, disconnected or changes.

Filesystems are detected using a variety of methods. If all fail (which is, if the filesystem is not one that is natively supported), filesystem interfaces are used one by one to find one that can handle the said filesystem, using their IS_VALID_PARTITION method.

Each partition then gets an identifier, the filesystem unique identifier (FSID), which is consistent across reboots but different between computers to avoid collection of informations from the FSID alone.

Methods

0x0001 IS_FS_MOUNTED

Check if a given filesystem is mounted.

Required permission: fs.filesystems.mounted

Arguments:

Return value:

  • 0x01 if the filesystem is currently mounted, 0x00 else

Errors:

None

0x0002 ENUM_FS

Enumerate all available filesystems.

Required permission: fs.filesystems.list

Arguments:

None

Return value:

Errors:

None

0x0003 FS_METADATA

Get informations on a filesystem.

Required permission: fs.filesystems.metadata

Arguments:

Return value:

Errors:

  • 0x3000: The requested filesystem is currently not mounted

0x0004 FS_MOUNT

Mount an existing filesystem. If no mount path is provided, the filesystem will be mounted under the /mnt directory.

New filesystems are automatically detected when storage devices are connected.

Required permission: fs.filesystems.mount

Arguments:

Return value:

None

Errors:

  • 0x3000: Unknown FSID (8 bytes)
  • 0x3001: This filesystem is already mounted

0x0005 FS_UNMOUNT

Unmount a mounted filesystem.

Required permission fs.filesystems.unmount

Arguments:

Return value:

None

Errors:

  • 0x3000: Unknown FSID (8 bytes)
  • 0x3001: This filesystem is current not mounted

0x0006 FS_WATCH

Subscribe to FS_CHANGED notifications when a filesystem is mounted or unmounted.

Required permission: fs.filesystems.watch

Arguments:

None

Return value:

None

Errors:

None

0x0007 FS_UNWATCH

Unsubscribe from FS_WATCH.

Required permission: None

Arguments:

None

Return value:

None

Errors:

None

0x1000 ITEM_EXISTS

Check if a given item exists.

Required permissions:

  • fs.path.exists to check any item type
  • fs.feid.exists to check only FEID

Arguments:

Return value:

Errors:

  • 0x3000: Invalid FSID provided
  • 0x3001: Requested filesystem is currently not mounted

0x1001 FEID_TO_SPLIT

Convert a FEID to the corresponding split path.

Required permissions:

  • fs.path.exists

Arguments:

Return value:

Errors:

  • 0x3000: Invalid FSID provided
  • 0x3001: Requested filesystem is currently not mounted
  • 0x3002: The provided FEID was not found in the filesystem

0x1002 ITEM_METADATA

Get the metadata of a given item.

Required permissions:

  • fs.items.metadata to get symbolic links' metadata

Arguments:

Return value:

Errors:

  • 0x3000: Invalid FSID provided
  • 0x3001: Requested filesystem is currently not mounted
  • 0x3002: The provided path was not found

0x2000 RENAME_ITEM

Rename an existing item.

Required permissions:

  • fs.items.move

Arguments:

Errors:

  • 0x3000: Invalid filename provided
  • 0x3001: Invalid FSID provided
  • 0x3002: Requested filesystem is currently not mounted
  • 0x3003: The provided path was not foud

0x2001 MOVE_ITEM

Move an existing item.

  • fs.items.move

Arguments:

Errors:

  • 0x1000: Invalid filename provided
  • 0x3000: Invalid FSID provided
  • 0x3001: Requested filesystem is currently not mounted
  • 0x3002: The provided path was not found
  • 0x3003: Target directory was not found
  • 0x4000: Target directory's maximum capacity has been reached
  • 0x4001: Maximum nested items number has been reached
  • 0x4002: Maximum path length has been reached
  • 0x4003: Item cannot be moved for unspecified reasons
  • 0x4FFF: Unspecified filesystem error

0x2002 DELETE_ITEM

Delete an item.

Required permissions:

  • fs.items.remove.trash to send items to the trash
  • fs.items.remove to delete items permanently

Arguments:

  • FSID (40 bytes)
  • Filesystem path
  • Deletion mode (1 byte): 0x01 to send the item to the user's trash, 0x02 to delete it permanently

Return value:

None

Errors:

  • 0x3000: Invalid FSID provided
  • 0x3001: Requested filesystem is currently not mounted
  • 0x3002: Item was not found
  • 0x3003: Cannot remove a non-empty directory
  • 0x4FFF: Unspecified filesystem error

0x3000 CREATE_DIRECTORY

Create a directory.

Required permissions:

  • fs.items.create

Arguments:

Return value:

None

Errors:

  • 0x1000: Invalid filename provided
  • 0x3000: Invalid FSID provided
  • 0x3001: Requested filesystem is currently not mounted
  • 0x3002: Parent directory was not found
  • 0x4000: Directory's maximum capacity has been reached
  • 0x4001: Maximum nested items number has been reached
  • 0x4002: Maximum path length has been reached
  • 0x4FFF: Unspecified filesystem error

0x3001 READ_DIRECTORY

List all entries in a directory.

Required permissions:

  • fs.dir.read
  • fs.dir.read.hidden to also list hidden items

Arguments:

Return value:

Errors:

  • 0x3000: Invalid FSID provided
  • 0x3001: Requested filesystem is currently not mounted
  • 0x3002: Directory was not found
  • 0x4FFF: Unspecified filesystem error

0x4000 OPEN_FILE

Open a file with the provided options.

Opening a file with read access will prevent writes and removal while the file is open. Opening a file with write access will prevent read, writes and removal while the file is open.

Note that read and write access can be enabled together.

Required permissions:

  • fs.items.create if the file must be created
  • fs.items.read for read access
  • fs.items.write for write access

Arguments:

Return value:

  • Opened file ID (8 bytes)

Errors:

  • 0x3000: Invalid FSID provided
  • 0x3001: Invalid parent directory provided
  • 0x3002: Requested filesystem is not currently mounted
  • 0x3003: Provided file path was not found
  • 0x3004: Provided directory path was not found
  • 0x4000: File is already opened
  • 0x4001: Directory's maximum capacity has been reached
  • 0x4002: Maximum nested items number has been reached
  • 0x4003: Maximum path length has been reached
  • 0x4FFF: Unspecified filesystem error

0x4001 CLOSE_FILE

Close a file.

Removes all locks on the opened file.

Arguments:

  • Opened file ID (8 bytes)

Errors:

  • 0x3000: Invalid opened file ID provided

0x4002 READ_FILE

Read an open file.

Required permissions:

None (implicitly given by the initial OPEN_FILE call)

Arguments:

  • Opened file ID (8 bytes)
  • Offset to read from (8 bytes)
  • Length to read (8 bytes)
  • Writable buffer pointer

Errors:

  • 0x3000: Invalid opened file ID provided
  • 0x3001: File is not opened in read mode
  • 0x3002: Provided offset is out-of-range
  • 0x3003: Provided offset + length is out-of-range
  • 0x4FFF: Unspecified filesystem error

0x4002 WRITE_FILE

Write an open file.

File will grow if needed.

Required permissions:

None (implicitly given by the initial OPEN_FILE call)

Arguments:

  • Opened file ID (8 bytes)
  • Offset to write from (8 bytes)
  • Length to write (8 bytes)
  • Readable buffer pointer
  • Truncate boolean (1 byte): truncate the file's length to the end of the written data

Errors:

  • 0x3000: Invalid opened file ID provided
  • 0x3001: File is not opened in write mode
  • 0x3002: Provided offset is out-of-range
  • 0x4000: Filesystem capacity exceeded
  • 0x4001: Maximum file size exceeded
  • 0x4FFF: Unspecified filesystem error

Create a symbolic link.

Required permissions:

  • fs.symlinks.create

Arguments:

Return value:

None

Errors:

  • 0x3000: Invalid filename provided
  • 0x3001: Invalid FSID provided
  • 0x3002: Requested filesystem is currently not mounted
  • 0x3003: Parent directory was not found
  • 0x3004: Cannot create symbolic links to cross-filesystem items
  • 0x3005: Cannot create symbolic links to non-existing items
  • 0x4000: Directory's maximum capacity has been reached
  • 0x4001: Maximum nested items number has been reached
  • 0x4002: Maximum path length has been reached
  • 0x4003: Storage's capacity exceeded
  • 0x4FFF: Unspecified filesystem error

Create a symbolic link.

Required permissions:

  • fs.symlinks.update

Arguments:

Return value:

None

Errors:

  • 0x3000: Invalid FSID provided
  • 0x3001: Requested filesystem is currently not mounted
  • 0x3002: Provided path was not found
  • 0x3003: Cannot crate symbolic links to cross-filesystem items
  • 0x3004: Cannot crate symbolic links to non-existing items
  • 0x4FFF: Unspecified filesystem error

Read a symbolic link's target.

Required permissions:

  • fs.symlinks.read

Arguments:

Return value:

Errors:

  • 0x3000: Invalid FSID provided
  • 0x3001: Requested filesystem is currently not mounted
  • 0x3002: Provided path was not found
  • 0x4000: Symbolic link is cyclic

0xA000 WATCH_ITEM

Watch an item for changes on its metadata or content. Any change will trigger a ITEM_CHANGED notification.

Required permission:

  • fs.items.metadata

Arguments:

  • FSID (8 bytes)
  • Path to watch
  • Generated watch identifier (8 bytes)

Errors:

  • 0x3000: Invalid FSID provided
  • 0x3001: Provided path was not found

0xA001 UNWATCH

Stop watching a content watched with WATCH_ITEM.

Arguments:

  • Generated watch identifier (8 bytes)

Errors:

  • 0x3000: Provided watch identifier was not found

0xF000 FORMAT

Asynchronously format the partition to get an empty filesystem.

Required permissions:

  • fs.filesystems.format

Arguments:

  • FSID (40 bytes)
  • Optional new partition's sector size, in bytes (8 bytes)

Return value:

  • Generated task identifier (8 bytes)

Error codes:

  • 0x1000: Invalid sector size provided
  • 0x3000: Invalid FSID provided
  • 0x3001: Requested filesystem is currently not mounted
  • 0x4000: Unspecified filesystem error

Notifications

0x0006 FS_CHANGED

Sent to a client which subscribed through FS_WATCH each time a filesystem is mounted or unmounted.

Datafield:

  • Mount status boolean (1 byte): indicate if the filesystem was mounted
  • FSID (8 bytes)

0xA000 ITEM_CHANGED

Notification sent to clients watching an item through the WATCH_ITEM method.

Datafield:

  • FSID (8 bytes)
  • FEID (8 bytes)
  • Event code (1 byte):
    • 0x01: item's metadata changed (timestamps excluded)
    • 0x02: item was moved
    • 0x03: item was deleted
    • 0x04: item was locked (only for the parent directory if case of a recursive lock)
    • 0x05: item was unlocked (only for the parent directory if case of a recursive lock)

sys::fsh service

The sys::fsh service is in charge of handling high-level filesystem operations, such as file associations and thumbnails generation.

Methods

0x0001 REGISTER_OPENER

Register an application as an opener for a list of file types.

Arguments:

Return value:

None

Errors:

  • 0x1000: One of the provided extensions is empty
  • 0x1001: At least one of the provided extensions is reserved to the system
  • 0x3000: Client does not expose a file opening service
  • 0x3001: Client already handles at least one of the provided extensions

0x0002 UNREGISTER_OPENER

Unregister an application as an opener for a list of file types.

Argument:

Return value:

None

Errors:

  • 0x1001: One of the provided extensions is empty
  • 0x3000: Client does not expose a file opening service
  • 0x3001: Client does not currently handle at least one of the provided extensions

0x0003 LIST_OPENERS

List the file openers associated to a specific type of items.

The list is not ordered, it's up to the file manager to determine the display order if multiple openers are found.

Arguments:

Return value:

Errors:

0x1000 CHECK_ITEM_THUMBNAIL_CACHE

Check if a cached thumbnail exists for a given filesystem item to determine if another should be generated or not.

The cache policy is determined using internal criterias.

Arguments:

Return value:

  • Presence (1 byte): 0x01 if a thumbnail is present, 0x00 else

If a thumbnail is present, this is followed by:

Errors:

0x1001 CACHE_ITEM_THUMBNAIL

Write a filesystem item's thumbnail in the cache.

If a thumbnail already exists in cache, it will be replaced by the new one.

The cache policy is determined using internal criterias.

Arguments:

Return value:

  • Thumbnail TFEID (8 bytes)

Errors:

  • 0x1000: Invalid bitmap data
  • 0x3000: Client is not a file manager service
  • 0x3001: Unknown FSID
  • 0x3002: Unknown FEID

0x1100 CHECK_ITEM_PREVIEW_CACHE

Check if a cached preview exists for a given filesystem item to determine if another should be generated or not.

The cache policy is determined using internal criterias.

Arguments:

Return value:

  • Presence (1 byte): 0x01 if a preview is present, 0x00 else

If a preview is present, this is followed by:

Errors:

0x1101 CACHE_ITEM_PREVIEW

Write a filesystem item's preview in the cache.

If a preview already exists in cache, it will be replaced by the new one.

The cache policy is determined using internal criterias.

Arguments:

Return value:

  • Preview TFEID (8 bytes)

Errors:

  • 0x1000: Invalid bitmap data
  • 0x3000: Client is not a file manager service
  • 0x3001: Unknown FSID
  • 0x3002: Unknown FEID

sys::hw service

The sys::hw service is in charge of hardware devices. It coordinates and manages communications with the hardware.

It is known as the I/O manager, or Ion.

Hardware detection

Hardware detection is handled by the kernel itself, which exposes a raw device descriptor (RDD).

Unique device identifier

Each device gets a unique device identifier (UDI) encoded on 8 bytes, which identifies a unique hardware device. It is unique across devices and consistent across reboots. Unlike the kernel's device identifier (KDI), the UDI is generated randomly to prevent from getting informations on the hardware just from the UDI.

Device formats

This section describes the multiple formats used by this service to deal with devices.

Device type descriptor

From the RDD is derived the device type descriptor (DTD), which describes the device's type. Its composition and size depends on the connection type, but it varies from empty (0 byte) if the connection type guarantees no information, up to 256 bytes.

The format remains to be determined but should be along the lines of a number-based equivalent of ModAlias, like :

  • PCI-Express:
    • Vendor (8 bytes)
    • Sub-vendor (8 bytes)
    • Type (8 bytes)
    • Sub-type (8 bytes)
    • ...
  • ...

Drivable device raw descriptor

A drivable device raw descriptor (DDRD) is a 512-byte long data structure meant to be used by drivers. It uses the following format:

  • UDI (8 bytes)
  • CII (4 bytes)
  • Size of the DTD (1 byte)
  • DTD (up to 256 bytes)
  • Future-proof

Driven device type

A driven device type (DDT), usually referred to as the device type, is generated by the driver for each device it drives from the DDRD. This is a normalized value, used by the system to determine which actions can be performed through this device.

It's a 4-byte value, the strongest two bytes describing the category and the weakest two the sub-category.

The following list contains all possible values for DDTs, but is far from being complete yet. It will also grow over time as new device types appear on the market.

  • 0x0001: Storage
    • 0x0001: Hard drive
    • 0x0002: SSD
    • 0x0003: USB flash drive
    • 0x0004: SD flash memory card
  • ...

Normalization

When a device is driven, other processes can ask this service to use normalized methods. These are methods that allow to perform a specific action or to receive normalized notifications about specific events of a specific device.

Also, interrupts are normalized to ensure constancy across devices of the same type. They are sent to the driver processes using the DEVICE_EVENT notification.

The normalization of methods, notifications and interrupts is performed by the driver in charge of the device.

There are several methods, depending on the device's type (DDT). Notifications differ as well.

You can find the far from being complete list in the relevant specifications directory. It will grow over time as new device types appear on the market and as existing devices evolve to provide new features.

Patterns

Several methods of this service use patterns, which allow to match devices depending on several criterias.

A pattern is a data structure whose size varies from 5 to 277 bytes made of the following:

  • Pattern (1 byte)
    • Bit 0: match all connection types
    • Bit 2: match all buses
    • Bit 3: match all ports
  • Connection type (1 byte)
  • Bus number (1 byte)
  • Port number (1 byte)
  • DTD length (1 byte) - 0 to omit DTD
  • DTD pattern indicator (32 bytes, only if DTD) - indicates which bytes of the DTD must be used as patterns
  • DTD (up to 256 bytes)

It's possible to match only devices that use a given connection type, and more specifically on a given bus and/or port.

It's also possible to list only devices that match a specific DTD pattern. For that, the bit corresponding to the byte number in the DTD pattern indicator must be set.

For instance, providing the DTD 0x0100B2 with the DTD pattern indicator set to 0b01000000, the second byte will match all devices.

Drivers

From a higher level point of view, drivers are services that declare their parent applications as being able to handle certain type of devices through the REGISTER_DRIVER method, using patterns.

Registering as a driver for a pattern requires the application to expose the driver service(s) relevant to the DDT provided in the pattern.

When a device is connected, a driver is selected from the list of drivers able to handle this specific device. If the device is connected for the very first time, the selected driver process first receives an IDENTIFY_DEVICE notification to translate the DDRD into a DDT.

Then, the driver process receives a DEVICE_EVENT notification, which will also be sent if the status of the device changes.

From this point, the driver process can communicate with the device using its I/O ports with the READ_IO_PORT and WRITE_IO_PORT syscalls.

It can also map the device's memory into its own address space using an AMS with the DEVICE_AMS syscall.

Other processes can then ask the driver to perform specific actions depending on the type of device, using normalized methods which can be sent to the driver using the ASK_DRIVER method. The driver receives these informations through the DRIVER_METHOD_REQUEST notification.

All methods and notifications are transmitted through this service, which performs permission checkings and validates some arguments.

The driver is also in charge of translating the interrupts of a device as well as eventual events polled from its (mapped) memory to normalized notifications which can then be sent to processes that subscribed to them using the related normalized methods.

You can see the complete list of methods and notifications for each type of driver services in the related section of the documentation.

Driver selection

A driver is selected for a specific hardware device if it matches any of the following criterias, in decreasing importance order:

  • The user selected this driver for this specific hardware device ;
  • The user selected this driver for this specific type of hardware devices (pattern) ;
  • This driver is the one with the most specific pattern covering this hardware device ;
  • This driver is the only one able to drive this specific hardware device (DTD)

If no criteria is matched, the driver isn't selected to drive the given hardware device.

A note on performances

Although hardware devices' interrupts are notified to the driver through service socket notifications, the latency is still minimal as soon as the driver listens to the RECV_SOCK_MSG signal, which like all signals uses interrupts and so guarantees a very low latency.

Latency reduction for storage devices

Direct driver access for sys::fs

All operations related to storage devices are handled by the sys::fs service. To avoid the cost of using sys::hw as a relay for hardware operations, the sys::fs service is allowed to directly communicate with all storage driver services.

Direct storage access for sys::fs

In the event a specific storage device doesn't require a dedicated driver, the sys::fs service can directly access the said hardware device.

Methods

0x0001 ENUM_DEVICES

Enumerate connected devices, reserved to system services.

It's also possible to only count the number of devices matching the provided criterias by providing a start index and end index of 0.

Required permission: devices.enum

Arguments:

  • Start index (4 bytes)
  • End index (4 bytes)
  • Pattern (277 bytes)

Answer:

  • Number of found devices globally (4 bytes)
  • Number of devices listed in this answer (4 bytes)
  • DDRD of each device (512 bytes * number of devices)
  • 0x01 if some devices were masked due to insufficient permissions, 0x00 else (1 byte)

Errors:

  • 0x1000: Start index is lower than the end index
  • 0x1001: Invalid connection type
  • 0x1002: Bus number was provided without a connection type
  • 0x1003: Port number was provided without a connection type
  • 0x1004: Invalid DTD
  • 0x1005: Range is greater than the available answer size
  • 0x3000: Client is not a system service
  • 0x3001: Provided bus was not found
  • 0x3002: Provided port was not found

0x0002 SUBSCRIBE_DEVICES

Subscribe to events related to devices matching a patterns, reserved to system services.

All current and future devices matching this pattern will cause a DEVICE_EVENT notification.

Required permission: devices.subscribe

Arguments:

  • 0x00 to subscribe, any other value to unsubscribe
  • Pattern (277 bytes)

Answer:

None

Errors:

  • 0x3000: Client is not a system service
  • 0x3001: Asked to unsubscribe but no subscription is active for this pattern

0x1000 REGISTER_DRIVER

Set up a service as a driver for all devices matching a pattern.
If multiple drivers have colliding patterns, the final user will be prompted to choose a driver.

When a new device is connected, the driver process will receive an IDENTIFY_DEVICE notification to translate the DDRD into a DDT.

The driver process will receive DEVICE_EVENT notifications for drivable devices. This notification will only be sent for devices for which the system chose this driver as the main one.
Notifications are also retroactive, which means they will be sent for already-connected devices.

The driver will also have the device registered in its drivable devices attribute, allowing it to use the DEVICE_AMS syscall to map the device's memory in its own.

Required permission: devices.register_driver

Arguments:

  • Pattern of the devices to drive (up to 277 bytes)

Answer:

None

Errors:

  • 0x3000: Current process is not a service
  • 0x3001: Process' parent application does not expose the relevant integration services
  • 0x3002: Current process is already registered as a driver for this pattern

0x1001 UNREGISTER_DRIVER

Unregister a service previously registered as a driver.

Required permission: None

Arguments:

  • Pattern to unsubscribe from (up to 277 bytes)

Errors:

  • 0x3000: Current process is not registered as a driver for this pattern

0x2000 NOTIFY_PROCESS

Send a notification to a process that registered itself for normalized methods through a normalized method.

Required permission: None

Arguments:

Answer:

Expected answer by the notified process for this method if any

Errors:

  • 0x3000: Unknown notification ID

0xA000 ASK_DRIVER

Ask a driver to use a normalized method on a device it drives.

Reserved to system services.

Required permission: devices.ask_driver

Arguments:

  • Device's UDI (8 bytes)
  • Method's code (4 bytes)
  • Method's arguments (size depends on the method)

Answer:

Expected answer format for this method

Errors:

  • 0x3000: Client is not a system service
  • 03x001: Unknown device UDI provided
  • 0x3002: Provided method code is invalid for this device
  • 0x3003: Invalid arguments provided for this method

0xD000 AUTHORIZE_FS_INTERFACE

Authorize a filesystem interface to access a specific part of a storage device.

The interface service will be allowed to perform requests on the provided storage device, only on the provided data segment.

Arguments:

  • Device's UDI (8 bytes)
  • Start byte (8 bytes)
  • End byte (8 bytes)

Answer:

Errors:

  • 0x3000: Client is not the sys::fs service
  • 0x3001: Unknown device UDI provided
  • 0x3002: Start byte is not aligned on the device's sectors
  • 0x3003: End byte is not aligned on the device's sectors
  • 0x3004: End byte is greater than or equal to the start byte

0xD001 UNAUTHORIZE_FS_INTERFACE

Unauthorize a filesystem interface authorization created using the AUTHORIZE_FS_INTERFACE method.

Arguments:

  • Authorization token (8 bytes)

Errors:

  • 0x3000: Client is not the sys::fs service
  • 0x3001: Unknown authorization token provided

0xD002 AUTH_PERFORM_STORAGE

Used by filesystem interfaces which received an authorization beforehand.

Perform an action just like with the ASK_DRIVER method, but restricted to the authorization's scope.

Arguments:

  • Authorization token (8 bytes)
  • Method's code (4 bytes)
  • Method's arguments (size depends on the method)

Answer:

Expected answer format for this method

Errors:

  • 0x3000: The provided authorization token is unknown or not tied to this client

Notifications

0x0002 IDENTIFY_DEVICE

Sent for a specific device the client that was selected as its driver.

Datafield:

Expected answer:

  • DTD (512 bytes)

Errors:

  • 0x3000: The provided DTD is invalid
  • 0x3001: The client does not expose the driver service relevant to this type of device

0x0003 DEVICE_EVENT

Sent for a specific device to clients that either:

Datafield:

  • DDT (512 bytes)
  • Event code (1 byte):
    • 0x10: device was just connected
    • 0x11: a driver was just selected for the device
    • 0x12: the device is ready to use
    • 0x20: device was disconnected (software)
    • 0x21: the device is being disconnected by its driver
    • 0x22: the device has been disconnected by the driver
    • 0x23: the device was brutally disconnected (hardware)
    • 0x30: device was just put to sleep
    • 0x31: device was just awoken from sleep
  • Indicator (1 byte):
    • Bit 0: set if this device is connected for the first time
    • Bit 1: set if this device was disconnected brutally (not by the system itself)
    • Bit 2: set if this device is connected for the first time on this specific port

0x1000 DEVICE_INTERRUPT

Sent to a driver after a device it's currently driving raised an interrupt.

Datafield:

0xA000 DRIVER_METHOD_REQUEST

Sent to a driver after receiving a valid normalized method request.

The driver is expected to answer using the relevant answer format for the provided normalized method and arguments.

The notification ID is generated by this service to allow the driver to send normalized notifications to a process that registers for it through this method without showing the caller process' PID to the driver process.

Datafield:

  • DDT (4 bytes)
  • Notification ID (8 bytes)
  • Method's code (4 bytes)
  • Method's arguments (size depends on the method)

Expected answer:

Expected answer format for this method if any

0xA001 DEVICE_NORM_NOTIF

Sent to a process that subscribed to normalized notifications of a device.
This notification is transferred by the sys::hw service after the driver sent it its content through the NOTIFY_PROCESS method.

Datafield:

sys::perm service

The sys::perm service is used to manage the privileges of users and the permissions of processes.

The purpose of user privileges

In NightOS, executable instructions can run in three different contexts:

The kernel doesn't have any limitation on what tasks it is allowed to perform, of course, as it is the one to decide.
System services communicate directly with the kernel and are trusted processes so they can do anything in their domain, which means for instance the sys::net cannot ask to manipulate the filesystem, as it's the role of sys::fs.

But applications, who run userland processes TODO

List of permissions

The list of permissions can be found in the dedicated specifications document.

Methods

TODO

Notifications

TODO

sys::net service

Methods

TODO

Notifications

TODO

sys::crypto service

WARNING: I am not a cryptography expert, so this document certainly contains mistakes or bad practices. In which case, feel free to let me know!

This service provides ways to perform secure cryptography operations from any process. Most methods do not require any permission.

Methods

0x1000 RANDOM_BUFFER

Fill a number with random values using a cryptographically-secure method.

Arguments:

Return value:

None

Errors:

None

Notifications

TODO

sys::crashsave service

Methods

0x0100 GET_LAST_CRASH_SAVE

Get the last crash save for this application, if any. The crash saves removal can be controlled by the answer provided to the RESTORE_CRASH_SAVE notification.

Arguments:

None

Answer:

  • Optional data provided in the datafield of the RESTORE_CRASH_SAVE notification (none if there is no crash save for this application)

0x0101 REMOVE_CRASH_SAVE

Remove the existing crash save for this application.

Arguments:

None

Answer:

  • Status (1 byte):
    • 0x00: the crash save was successfully deleted
    • 0x01: no crash save was found for this application
    • 0x02: couldn't delete the crash save as the file is currently in use

Notifications

0x0100 CRASH_SAVE_COLLECTION

Sent to a process during crash save collection.

It is only sent to applications indicating they support crash saves in their manifest.

Datafield:

  • Number in seconds under which this notification must be answered (1 byte)
  • Timestamp of the last crash save for this application (1 byte)
  • 0x01 if this notification was not answered in time the last time it was sent, 0x00 otherwise (1 byte)
  • Temporary FEID to write the crash save in

Expected answer:

  • Status (1 byte):
    • 0x00 if the crash save was written succesfully
    • 0x01 if the application didn't have enough time to fully write the crash save
    • 0x02 if the application couldn't create a crash save for whatever reason
    • 0x03 if the application couuldn't create
  • Optional log message which will be written to the system's logs

0x0200 RESTORE_CRASH_SAVE

Sent to a process when a crash save is to be restored.

This notification is sent only if the correct bit is set in the application's startup reason.

Datafield:

  • Temporary FEID to the crash save (8 bytes)
  • Timestamp when the CRASH_SAVE_COLLECTION signal that resulted in this crash save was sent (8 bytes)
  • Timestamp when this crash save was actually written to the disk (8 bytes)
  • 0x00 if this crash save is sent for the first time, 0x01 if the application was interrupted before it answered this notification the last time (e.g. application or system crash) (1 byte)
  • 0x00 if this crash save is sent for the first time or if the application itself didn't crash while trying to restore it the last time, 0x01 if the application itself crashed last time (1 byte)

Expected answer:

  • Status (1 byte):
    • 0x01: crash save was restored successfully
    • 0x02: crash save is corrupted and/or invalid
  • 0x00 if the crash save can be deleted, 0x01 if it should be kept (1 byte)

sys::hydre service

Methods

TODO

Notifications

TODO

sys::app service

Methods

TODO

Notifications

TODO

sys::process service

Methods

TODO

Notifications

TODO

Native applications

This folder contains documentation for each native application.

WARNING: These documents act as a general list of features and technical overview of the applications. These may change at anytime, and are by no mean a final indication of what the application will consist of and what features it will have.

Below is the list of applications that are installed (by default) during the installation process.

Userland interactive applications

Basics

System utilities

  • Central : Settings
  • Monitor : Monitor opened applications, processes, CPU/RAM usage, etc.
  • Registry : View and edit the registry (for advanced users)
  • Skyer : Applications manager
  • Cloudy : Backup & Sync manager

Utilities

  • Gravity : Text editor
  • Thinker : Notes and task lists manager
  • ShootingStar : Pictures viewer and simple editor
  • Sonata : Music player (with Hi-Res support and options)
  • Milkshake : Video player
  • Blackhole : Archives manager
  • Reader : A complete e-book reader with many supported formats and options
  • Postal : An e-mail client

Suites

  • Particle : A complete IDE to make applications
  • Astral : The complete toolchain allowing to build applications

Security

  • Vortex : Firewall
  • Locky : System's encryption tool, with the ability to manage encrypted archives and volumes

System services

System services are listed in their own specifications directory.

Astral

Astral is the toolchain required to build NightOS applications. It's a command-line only application.

Description

Astral is made of several tools:

  • A builder, which builds application packages from their manifest
  • A publisher, which allows to sign the application and upload it to the Store
  • A debugger, which provides tools to run and debug an application (monitor permissions, create puppet sandboxes, ...)

It relies on several external compilers (Rust, TypeScript, ...) to build applications from source. Infrastructure for the main languages is installed by default, but additional toolchains can be installed manually.

BareEnv

BareEnv, for bare environment, is text-based desktop environment meant for servers as well as fallback when no graphical output is available or when the primary desktop environment does not work for any reason.

Graphical features are disabled, and any direct application launch will fail ; they can only be launched through commands, to reduce the risk of crash due to the creation of graphical backends failing.

Tracking devices such as mouses are still supported and can be used in compatible command-line applications.

The application itself allows to manage the environment's settings, as well as to optionally change the behaviour of specific devices and system features to be more adapted for text-based usage.

Note that this environment is not meant for general use ; it's mainly aimed for troubleshooting when graphics are failing.

Blackhole

Blackhole is the default archive manager of NightOS.

It relies on Locky for encrypted archives, filesystems and files.

Features

Blackhole can open, extract and create different formats. It supports compression as well as encryption.

Supported containers

  • 7-Zip archives (.7z)
  • ISO images (.iso)
  • TAR archives (.tar)
  • ZIP archives (.zip)

Are also supported all the archive formats specific to NightOS:

Supported compression formats

  • BZip(2) (.bz, .bz2)
  • LZMA(2) (.lz, .xz)
  • Unix (.z)
  • GZip (.gz)
  • ZStandard (.zstd)

Central

The Central application delivers the Control Center which allows to configure how the system behaves. It is split into several categories.

> Current user settings
    > User Account
        > Set profile picture
        > Set nickname

    > Change security level
        | Basic
        | Standard
        | Restricted
        | Extreme
        | Total
            ! Only available in developer mode

> Applications management
    > Sideloading
        > Change sideloading mode
            | Disabled
            | Secure
            | Unsecure

> Global settings
    > Date & Time
        > Set date
        > Set time
        > Change timezone
        > Synchronize time

    > Encryption
        > Manage storage encryption
            | Enable
            | Disable

        > Change encryption password
            ! Disabled if storage is not encrypted yet

        > Manage per-user encryption
            | Enable
            | Disable

> Users
    > Create new user
    > Create new administrator user
    > Create new supervised user

This part show settings that are only available in developer mode.

> Applications
    > Set application proxies
        | Enable
        | Disable

Cloudy

While the system and main data can be backed up using TimeTravel, it's not always handy to manage backups on an external hard drive and keep it safe.

This is where Cloudy comes: it allows to synchronize data on the cloud, using a master password which is stored nowhere but in the user's mind.

How it works

In order to work, Cloudy needs a storage account where it can store its data. Multiple providers are supported: Dropbox, Google Drive, Amazon AWS, SFTP storages, etc.

It will then store all data on this account, in a dedicated folder (the path can be customized). All data will be encrypted by a master password prompted when the application is opened for the first time, so no one will be able to access these date except the user who set it up itself.

Integration

Applications can ask to store and synchronize their own data through Cloudy. They will not be able to access any other application's data, though.

Collected data

The data that can be backup-ed and synchronized by Cloudy are:

  • User's settings, computer's settings if the user is administrator
  • Installed applications and their data
  • Applications can ask to store additional custom data (e.g. a password manager)

Synchronization

The synchronization process has two parts:

  • The up-sync process, which sends new data to the cloud ;
  • The down-sync process, which reflects these changes on the local computer

Up-sync can be performed manually at anytime, or scheduled to be performed frequently. It's also possible to ask, for instance, to update the list of applications when a new one is installed or removed, while only backuping applications' data once a day.

Down-sync can be performed anytime, and a preview will show which items will be restored and what will be overridden. It's possible to only restore a few items. By default, down-sync will be performed in real-time: as soon as an up-sync process is performed, down-sync will happen on every other computer of the synchronization chain.

Synchronization chain

Data are synchronized through all computers of a single synchronization chain, which is simply a set of computers. Synchronization happens per user, which is why Cloudy doesn't backup every user of the computer, but only the one synchronization is enabled on. Of course, every user can enable synchronization for its account.

When a user enabled Cloudy for its account, it must connect with a NightOS account, and it is then added to the synchronization chain consisting of every computer having Cloudy enabled with this NightOS account.

For down-sync to happen, the username must be on the receiving computers than on the upload computer, else a specific setting will need to be changed in Cloudy.

Comet

Comet is the default file manager of NightOS.

Compatible filesystems

Comet can open and manage all filesystems natively supported by NightOS, meaning it also supports custom filesystems handled by the sys::fs service as well as remote drives such as SSH and SFTP storages.

Third-party application can also provide support for additional storage types (e.g. cloud).

Usage

Comet is meant to be used with both a keyboard and a tracking device, but it's also possible to use only the former or the latter.

Features

  • Listing
  • Creation and deletion of items
  • Copy and paste for all types of items
  • Handles parallel operations (with queue management, pause/resume, priorization, forced throttling, ...)
  • Encryption and deletion for files, folders and drives
  • Items caching for remote drives
  • Drag & drop from and to other applications
  • Customizable interface
  • Tabs
  • Real-time refresh

Gravity

Gravity is a text editor featuring syntax highlighting for code.

It supports many different languages, and comes with customizable keyboard shortcuts. Mouse is supported.

Features

  • Basic and semantic syntax highlighting
  • Support for third-party language definitions
  • Support for extensions
  • Customizable themes
  • Fully customizable keyboard shortcuts
  • Mouse support
  • Encryption and compression support
  • Built-in terminal
  • Can open folders
  • Workspaces (single- or multi-folders)
  • Memorize unsaved files, per workspace

Locky

Locky is NightOS' encryption tool. It heavily relies on the sys::crypto service.

User interface features

  • Encryption and decryption of notes
  • Encryption and decryption of files, folders
  • Encryption and decryption of users
  • Encryption and decryption of storages (for supported filesystems)

Library features

  • Encryption and decryption of data streams
  • Support for third-party algorithms
  • Can act as an intermediary file driver

Milkshake

Milkshake is the default video player of NightOS.

Supported containers and codecs

  • WebM (.webm)
  • Matroska (.mkv)
  • Video Object (.vob)
  • Ogg Video (.ovg)
  • Multiple-Image Network Graphics (.mng)
  • Audio Video Interleave (.avi)
  • MPEG Transport Stream (.mts, .m2ts, .ts)
  • QuickTime File Format (.mov, .qt)
  • Windows Media Video (.wmv)
  • RealMedia (.rm)
  • MPEG-1/-2 (.mpg, .mp2, .mpeg, .mpe, .mpv, .m2v)
  • M4V (.m4v)
  • 3rd Generation Partnership Project (.3gp, .3g2)
  • AV1 codec

Features

  • Hardware-accelerated decoding
  • Fully customizable keyboard shortcuts
  • Support for third-party formats through external applications
  • Memorize position in video
  • Selection for audio track and subtitles
  • DVD and BD (Blu-ray) support with menu
  • Media library management

Monitor

Monitor allows to see and manage running applications and processes, as well as to watch resources usage like CPU percentage or used RAM.

Features

Monitor can display as an instant, real-time label or as a graph multiple informations:

  • Memory usage
  • CPU usage (global and per core) for each CPU
  • GPU usage (by type) for each GPU
  • Storage usage (HDD, SSD, USB, ...) for each storage device
  • Network usage for each available connection

Usage can be expressed in percents, and when applicable in bandwidth (MB/s par default).

Additional informations can also be displayed:

  • Frequency of each CPU core
  • Frequency of each GPU
  • Number of concurrent reading and writing to each storage device
  • Temperature for each device supporting it
  • Memory usage (by type)

Hardware informations can also be displayed:

  • General : number of processes, number of threads, uptime, number of opened file handles
  • CPU : model, frequency (total and per core), base frequency, boost frequency, physical cores, logical cores, number of used sockets, caches
  • GPU : model
  • Memory : capacity, in use, panigated, used slots, available slots, frequency
  • Storage : model, capacity, used, locks, cached data, S.M.A.R.T. if applicable
  • Network : type of connection, hardware speed, IPv4 and IPv6 address, used DNS

It also displays informations on processes, services, and sum for applications with at least one running process, separated in 3 tabs:

  • Name
  • Icon
  • Application (+ icon)
  • ANID
  • CPU usage
  • GPU usage
  • Memory usage (percent and absolute amount)
  • Storage usage (percent and absolute speed)
  • Network usage (percent and absolute speed)
  • Approximate battery consumption
  • Process identifier
  • Command-line arguments (when started as a command)
  • Execution context (if applicable)
  • Process priority
  • Number of threads
  • Number of file handles
  • List of file handles
  • List of permissions
  • Is the process frozen? (process only)
  • Number of clients (service and application only)
  • Is a service active? (application only)
  • Number of processes (application only)
  • Number of threads (application only)

Nova

Nova is the default desktop environment of NightOS.

It has a highly-customizable user interface.

WARNING: This document is only a draft and is extremely far from being complete! Anything written in this document is subject to change at anytime.

User interface

The desktop

The desktop itself is visible in the background - it is fully visible when on window on foreground.

It shows all items from /home/[username]/desktop.

Right-clicking on an item shows the default file manager's context menu.

A background image called wallpaper is also displayed under the desktop. The wallpaper can be replaced by a custom one.

Notification center

The notification center is a panel that slides in from one of the screen's side. The side can be customized.

It has a vertical layout and contains the list of each notification.

By default, it also shows a few quick settings as well as the date and hour of the day.

Dock

An optional dock can be shown, made of a list of icons on one side of the screen. It also shows the list of opened windows, even if they are not pinned to the dock. An indicator also shows the number of opened windows (a dot for each window).

Moving the mouse on top of one of the icon will by default show a preview of each opened window, side by side. If a high number of windows are opened (by default more than 4) the previews will be stacked on a grid. By default, previews are shown by recently used order, with a maximum of 16 windows (grid of 4 rows, 4 columns).

Clicking on an icon will, if the application already has a window, show the grid, else open the application.

For custom shortcuts, this will only trigger the selected shortcut.

User experience

Removing native applications

When removing a native applications, a popup will suggest to hide the app instead of removing it, which allows other applications and scripts to still interact with them.

Particle

Particle is a complete IDE for making NightOS applications. It uses the Astral toolchain.

Features

Its features are the same as Gravity, plus many additional ones (too many to be listed here).

It also features a strong integration with the Astral toolchain in order to build, debug and deploy NightOS applications.

Pluton

Pluton is the default terminal of NightOS. It features a strong integration of Hydre.

Postal

Postal is the default e-mail client of NightOS.

Features

  • Handles multiple e-mail accounts
  • Supports SMTP, POP, IMAP
  • Custom arrangment rules (labels, colors, ...)
  • Filters and sorts (by e-mail sender, title, ...)
  • E-mail sending planification
  • Offline support to consult e-mails without an internet connection
  • Support for PGP encryption (end-to-end)

Reader

Reader is an e-book reader with supports for comics. It can also manage libraries of books.

Supported e-books formats

Here is the list of the supported e-book formats, which act as containers. See also the supported image formats.

  • E-Pub (.epub, .epub3)
  • PDF (.pdf)
  • MOBI (.mobi)
  • Rich Text Format (.rtf)
  • Compressed ZIPs (.cbz)
  • Compressed RARs (.cbr)

Supported image formats

Same as ShootingStar (uses the sysl::shootingstar SAL to render pictures).

Registry

Registry allows to view and edit the registry. It is reserved to advanced users.

Features

  • Display / edit everything in the registry
  • Save current registry as a snapshot
  • Restore snapshots

User Interface

The U.I. is made of a set of panes (one by default) containing a tree showing all structures and lists stored in the registry and allowing to edit them.

Rocket

Rocket is the default web browser of NightOS.

ShootingStar

ShootingStar is the default pictures viewer and editor of NightOS. Edition is limited to simple manipulations such as cropping and resizing.

Library usage

ShootingStar exposes the sys::shootingstar library to manipulate, perform basic manipulations and most of all render different image formats.

It can also be used by file managers to generate thumbnails and video previews.

Supported formats

  • Joint Photographic Experts Group (.jpg, .jpeg)
  • Joint Photographic Experts Group 2000 (.jp2, .jpx)
  • Tagged Image File Format (.tiff, .tif)
  • Graphics Interchange Format (.gif)
  • Graphics Interchange Format Video (.gifv)
  • AV1 Image File Format (.avif)
  • Windows Bitmap (.bmp)
  • NightOS Bitmap Image (.nbi)
  • NightOS Bitmap Video (.nbv)
  • Portable Networks Graphics (.png)
  • Portable Pixmap Format (.ppm)
  • Portable Graymap Format (.pgm)
  • Portable Bitmap Format (.pbm)
  • Windows Icon (.ico)

Viewer features

  • View pictures
  • Zoom in/out
  • Fullscreen
  • Fit image to width / height / both
  • Put images side by side
  • View a set of pictures as a gallery
  • View a set of pictures as a slideshow (multiple possible transitions)
  • Display EXIF metadata

Edition features

  • Resize (keep or ignore ratio)
  • Crop
  • Edit/remove EXIF metadata
  • Convert to black and white

Skyer

Skyer is the applications manager of NightOS. It allows to install and remove applications, as well as to manage their permissions and enable application proxies in developer mode.

Sonata

Sonata is the default music player of NightOS. It features support for Hi-Res audio files and can play many different formats (e.g. 32 bits / 384 kHz FLAC, DSD, ...). It also supports bit-to-bit playing to USB DACs.

Features

  • Audio library management
  • Sorting and filtering by tag
  • Tag edition for individual tracks, sets, albums and whole artists
  • Bit-perfect playblack
  • Optional fetching of informations about tracks, albums and artists (biographies, pictures, ...)
  • Multiple playback queues
  • Fully customizable keyboard shortcuts
  • Support for themes
  • Support for extensions
  • Customizable tabs
  • Discovery feature to play never-played tracks
  • Smart playlists (auto-add and removal based on custom criterias)
  • Playlists imports/exports
  • Play counts for tracks
  • Full database import/export
  • Reduced player
  • Play in background

Supported formats

  • Waveform Audio File Format (.wave, .wav)
  • Free Lossless Audio Codec (.flac)
  • Apple Lossless Audio Codec (.alac)
  • Ogg (.ogg, .oga)
  • Audio Interchange File Format (.aiff)
  • Core Audio Format (.caf)
  • MPEG-1/2 Audio Layer 3 / mp3PRO (.mp3)
  • Windows Media Audio (.wma)
  • Advanced Audio Coding (.aac)

Stellar

Stellar is the default application store of NightOS.

Thinker

Thinker is a simple notes application.

TimeTravel

TimeTravel is the backup and versioning program of NightOS.

Vortex

Vortex is NightOS' built-in firewall. It can also be used in combination with parental control.