A modular configuration provider reading config from files, arguments and environment

When creating a Node.js application, one usual task is to read configuration somehow in order to let the user define some settings for it. There are lots of awesome configuration libraries at charge of making easy this task, but each one is specialized in reading config from one single source, such as files, arguments or environment variables. I usually use these libraries to read configuration from arguments or configuration files:

• cosmiconfig - Reads configuration from a file. It searches for many file types and file names, and even supports defining config in the package.json file. Very customizable, it’s an awesome library very flexible both for the app developer and for the app user.
• commander - A great library that allows to read configuration from process arguments, among other things.

But I personally like to be even more flexible with the users, and let them choose the source to define the configuration, because each one may have different requirements that can make easier to define the configuration using one than the others. So, I used to repeat the task of defining, reading and merging configuration of each different source in a lot of my projects. And that’s why I have created the configuration library that I’m going to talk about in this post:

• @mocks-server/config - It allows to define configuration options, and it reads environment variables and uses cosmiconfig and commander under the hood to provide values to them.

Note: The library was developed as a part of the @mocks-server project ecosystem, but it is not coupled to it and it can be used anywhere else, because it is fully configurable.

As a summary, it reads, merge and validates configuration from:

• Default option values
• Configuration received programmatically
• Configuration files (using cosmiconfig internally)
• Environment variables
• Command line arguments (using commander internally)

It also provides:

• Parsing objects from command line arguments or environment vars
• Isolated configuration namespaces
• Objects to get/set options values internally at any moment
• Events when any option value changes

Note: Nconf is another popular library reading configuration from different sources, you can read about the differences with @mocks-server/config in the alternatives section of this post.

Quick start

In this example we are going to create a simple option of type string, and we are going to see how to read its value:

Add the library @mocks-server/config to your package dependencies:

npm i --save @mocks-server/config

Import the library and create a configuration instance. You must provide a moduleName option. It will determine the name of the configuration files that will be searched for, and the prefix of the environment variables:

import Config from "@mocks-server/config";

const config = new Config({ moduleName: "myApp" });

Now that we have created the config instance, we can start adding options to it. In this case, we are going to create an option named myOption, of type string, with a fooValue default value:

const myOption = config.addOption({
  name: "myOption",
  type: "string",
  default: "fooValue",
});

Now, we only have to load the configuration. Note that it is an async process, so we have to wait for it to finish before reading the options values:

config.load().then(() => {
  console.log(myOption.value);
});

At this point, supposing that our file was named app.js, we can define the value for our option simply defining an environment variable named MY_APP_MY_OPTION (Environment variables must be prefixed with the value of the moduleName option, and they must be defined using “screaming snake case”):

MY_APP_MY_OPTION=anotherValue node app.js

Or we can define it using a command line argument:

node app.js --myOption=anotherValue

We can also create a .myApprc.json file at the same folder, and simply run node app.js:

{
  "myOption": "anotherValue"
}

Or a myApp.config.js file:

module.exports = {
  myOption: "anotherValue"
};

Or even a .myApprc.yml file. You can check the whole list of supported file formats at the @mocks-server/config docs.

myOption: anotherValue

Sources priority

When reading sources, the library will try to search for the value of each option in every source (unless it is explicitly configured for skipping some sources). So, the values for different options, or even for the same option, can be defined in different sources at a time. In that case, it applies a priority to the sources, which is, from lower to higher:

• Option default value
• Configuration file
• Environment variable
• Process argument

This is very useful, because you can have a configuration file in your app with some values, but override some of them defining environment variables when you start the application, or even using command line arguments, which will override even the values of environment variables.

Option types

It does not only read values from different sources, but it also parses the values to each correspondent option type.

Options can be of one of next types: boolean, number, string, object or array. The array type also allows to define the type of items contained in it.

For example, if an option is of type boolean and it is defined in a environment variable, its value will be converted from false, true, 1 or 0 strings to a boolean type:

MY_APP_MY_BOOLEAN_OPTION=1 node app.js
# value -> true

If the option is of type number, it will be parsed to a numeric value:

node app.js --myNumberOption=2
# value -> 2 as a number

And it parses even options of type object from command line arguments and environment variables:

MY_APP_MY_OBJECT_OPTION='{"foo":"var"}'
# value -> {foo: "var"}

Changing settings in runtime. Events

Apart from reading the configuration, the library can also be used to modify options in runtime. Suppose that your application provides an API for changing settings while it is running (which is the case of Mocks Server, for example). If that is the case, you can modify the values of the options from the API component, and the other components can listen to changes in the options and act in consequence whenever it is needed, because the library also emits events whenever an option changes its value.

Use the onChange method to add event listeners to value changes:

const myOption = config.addOption({
  name: "myOption",
  type: "string",
});

myOption.onChange((newValue) => {
  console.log(`myOption value has changed to ${newValue}!`);
});

Use the value setter to change the value of an option:

myOption.value = "anotherValue";
// console -> myOption value has changed to anotherValue!

Modularity: namespaces

For sure that you have noticed the word “modular” in the title of this post. But, for the moment, what makes this library to be “modular”? It seems to be a simple configuration provider reading values from some different sources. Well, here is where the “namespaces” concept enter.

But, first of all, why should a configuration provider be modular?

Well, we are going to suppose that we have an app that is very well designed. It has a very clean architecture in which each internal component is responsible to do one single thing (that’s what we all want in our applications, don’t we?). And some of that components need some user configuration to do their job. Suppose also that our configuration contains some logic about the options, such as validation, parsing, etc. In my experience, the configuration is usually something that is provided by a specific component in the application, and it is usually placed very next to the application higher levels. One of the first things that we usually do is reading the configuration in some place in our app, and then we pass that configuration to the other internal components (of course that this wouldn’t be always the case, but I have seen it lots of times, and I usually did it also in the same way).

If that is the case, then it may become a problem, because every time we need to modify or to add an option to any of the internal components, we must modify also our “global” configuration provider. And, in an ideal world, we should modify only the involved component, am I right?

Using the @mocks-server/config library, you could avoid this problem simply passing the config instance to each component, and let them add their options. It is a good solution, but, depending on the scale of the project and the amount of options, it may result in conflicts between the names of the options from different components.

Namespaces to the rescue

In order to avoid that problem, the @mocks-server/config library provides the “namespaces” concept, so each component can be the owner of its own configuration namespace, and it can modify its options whenever it is needed without the risk of conflicts with the other components.

Following with the previous example, we can use the addNamespace config method for creating a namespace. We must pass the name for the namespace as first argument:

const myNamespace = config.addNamespace("myNamespace");

And now that we have our namespace created, we can add options to it as we did in the config instance in the previous example:

myNamespace.addOption({
  name: "myOption",
  type: "string",
  default: "fooSecondValue",
});

Then, when defining environment variables, we must add the namespace name as a prefix to the option name:

MY_APP_MY_NAMESPACE_MY_OPTION=anotherValue node app.js

When using command line arguments, we must add the prefix separated by a dot:

node app.js --myNamespace.myOption=anotherValue

And when using configuration files, each namespace corresponds to an object key:

{
  "myNamespace": {
    "myOption": "anotherValue"
  }
}

Keeping components isolated

So, you can keep your components configuration isolated creating and passing a different namespace for each one of them. The next example shows a theoretical app creating different namespaces for some components:

const dbConnector = new DBConnector({
  config: config.addNamespace("db"), 
});

const api = new Api({
  config: config.addNamespace("api"), 
});

await config.load();
await dbConnector.start();
await api.start();

As complex or simple as you may need

Even when namespaces is a great feature, it may not be useful to you if your app only needs few configuration options, or if there is no risk of conflicts between the component options, or even if you want to keep the configuration as much simple as possible for the user. In that case, you could simply pass the config instance to each component and let them to add their own options to the root namespace.

Or maybe you need even a more complex structure for your configuration, because some of your components depends on many other internal components. In that case, nested namespaces are supported also:

const myNestedNamespace = config.addNamespace("first")
  .addNamespace("second")
  .addNamespace("third")
  .addOption({
    name: "foo",
    type: "number",
  });

Which, for example, would result in a yaml configuration file like this:

first:
  second:
    third:
      foo: 3

Or in an argument like this:

node app.js --first.second.third.foo=3

Breaking the modularity rules

Even when the library was designed to provide modularity, it is flexible enough to allow breaking the rules whenever it is needed. For example, in a previous example I talked about an API changing the configuration. Supposing it is able to change the configuration of any component, then that API is breaking the modularity rules.

If this is needed, you can use some library methods to access to any namespace configuration options, or even provide a whole configuration object that will set every namespaces at a time. For example:

config.set({
  first: {
    second: {
      third: {
        foo: "foo"
      }
    }
  }
});

This would set options for all provided namespaces in the object. It is not desirable to do things like this when we are talking about modular solutions, but it can be used if there is no other better alternative. You can read the library API docs to know more about config available methods.

Alternatives

Another library able to read configuration from files, arguments and environment is Nconf. It is a great and very popular library. The main difference with @mocks-server/config is that it is more focused on the options to be loaded and its types in order to execute validations and parse the data, while Nconf leaves the door more opened to get any value unless you explicitly configure restrictions separately for each different source.

More in detail, the main differences between both libraries are:

• mocks-server/config uses Cosmiconfig under the hood, so it supports more file formats out of the box, such as yaml, cjs, etc.
• Nconf allows to get any key from the sources, unless you use each source options separately to set restrictions. On the contrary, mocks-server/config requires to specifically define the details of each option to be loaded, as its type, etc. Then, it parses the data and executes validations for all sources automatically.
• Nconf requires to initiate separately each different source, while mocks-server/config initiates all sources using only the load method, unless the user specifies another behavior using the config options. On the other hand, mocks-server/config uses exactly the same hierarchy described in the Nconf docs as a good practice.
• mocks-server/config always executes validations and parses data based on the option types using Ajv under the hood. This is something that seems to be defined separately for each different source in Nconf.
• mocks-server/config supports nested namespaces, so keys like foo.var.foo2.var2=x are supported.

Further information

This post tried to be only an introduction to the main features of the library, so there are many other interesting things that were not mentioned here, like:

• Configuring the library itself. Deactivating sources, using custom file names, etc.
• How to define values for each different option type on each different source
• Library lifecycle. Handling complex use cases

For further information, you can read the whole technical docs of the library here.