Environment variables and configuration anti patterns in Node.js applications

Crafting robust and maintainable applications is no small feat. One of the fundamental pillars of building reliable Node.js applications is effective configuration management.

Configuration encompasses a wide array of parameters, from database connection details to API keys, and even application-specific settings. Regardless of the size or complexity of your Node.js project, you’ll inevitably encounter the challenge of handling configuration data.

In this blog post, we’re going to delve deep into the world of configuration patterns for Node.js applications. We’ll explore the various strategies and tools at your disposal to efficiently manage configuration data. Whether you’re a seasoned Node.js developer looking to refine your practices or a newcomer eager to learn, this post will equip you with the knowledge you need to make informed decisions regarding your application’s configuration.

Why managing configuration for Node.js application is crucial?

Before we dive into the intricacies of different configuration patterns, let’s take a moment to understand why configuration management is so crucial in Node.js applications.

Imagine building a Node.js application without a clear and organized way to handle configuration. You’d likely end up hardcoding sensitive information like API keys directly into your codebase (oh no! 😳), scattering configuration values throughout your application files (maintainability is 😭), and making it nearly impossible to adapt to different environments seamlessly.

Pillars of effective and robust configuration management

What makes a maintainable, scalable, and secure configuration management in Node.js applications? Here are some of the key pillars to consider that most often I see developers overlook:

Doesn’t impede security: Storing sensitive information like database credentials and API keys securely is paramount. Without proper configuration practices, your application could be susceptible to data breaches.
Promotes deployment portability: Your application should run consistently across various environments, such as development, testing, and production. Good patterns of configuration management do not couple your environments to your application or to your configuration. Instead, it allows your application to transparently and seamlessly deploy to different environments, unaware of the differences between them.
Simplifies maintenance: As your application evolves, so do its configuration requirements. Having a robust configuration management system in place makes it easier to make changes and updates without rewriting large sections of code or accessing ad-hoc configuration files or environment variables.

How to load configuration in Node.js applications?

Now that we’ve highlighted the importance of configuration management, let’s review the various configuration patterns available for Node.js applications.

You might have heard about some common approaches like using .env files or popular packages like dotenv, convict, and env-schema. We’ll explore these options in detail.

The following depicts some of the possible ways to load configuration in Node.js applications, ordered from the least effective to the most robust configuration management traits:

A config.json file
Direct use of environment variables
Typed configuration
Validated configuration

Concerns with the use of configuration files, such as `config.json`

Generally speaking, using a dedicated configuration file to load configuration, a la config.json, doesn’t necessarily mean that you’re missing out on some of the traits we’ve listed above. For example, you can still bake type safety and validation into your configuration file if you’re using it as a JavaScript module (config.js) instead of a JSON file (config.json). Or, maybe you have another step in your configuration loading process that adds type safety and validation.

However, most often one of the anti-patterns I’ve noticed is that developers tend to hardcode configuration data directly into their source code, which is a big no-no. Another disaster is that deployment environments proliferate into your configuration file. Does this look familiar?

├── config/
│    └── config.dev.json
│    └── config.staging.json
│    └── config.qa.json
│    └── config.prod.json

This is a maintenance nightmare. You’ll end up with a lot of duplicated configuration data, and it’s hard to know what configuration was used for a given deployment. You also tightly couple your deployed environment types to your application, which is not ideal.

Information security is another concern with configuration files. If they are committed to the source code repository, it only takes one mistake in misconfigured web servers or path traversal vulnerabilities, to expose sensitive information like database credentials and API keys. Here’s a timely reminder from Noor AlHomaid on X:

Always include (.yml) files in your recon #HappyHacking #cybersecurite #infosec #ethicalhacking pic.x.com/SpSfyoLXEm
— نُور الحميد (@AlHomaidNoor) October 2, 2023

Concerns with `.env` files:

On first look, .env files may seem to suffer from similar issues as config.json files - a specific file used per environment and so on. However, most notable to recognize about .env files is:

They aren’t committed to the Git repository, hence they are not versioned and aren’t tightly coupled to your application’s code. The hint is in the name, .env use a leading dot to indicate that they a special case of files.
They are most often used as a generic .env file that is populated with environment variables and is mostly relevant for production use, and a .env.local file to refer to configuration needed for local development environment variables.

That said, they are still prone to the same issues as config.json files:

It’s easy to hardcode configuration data directly into your source code
It’s easy to copy your deployment environments into several .env files
It’s easy to make the mistake of committing the .env file to your Git repository, which is a security risk
They are not inherently type safe
They are not inherently validated

ASIDE Is it an advantage or disadvantage that .env files aren’t versioned in source control?

One could argue to the issue of perpetuating configuration drift. For example, .env files are most often not versioned via Git, so that secrets are not leaked, which is a good thing. But this means that the configuration is not versioned either, and it’s hard to know what configuration was used for a given deployment.

Developers will also be concerned that because .env files aren’t committed to the Git repository, then new configuration options aren’t easily discoverable. This can lead to configuration drift, where different environments have different configuration options.

Concerns with `dotenv`

Needless to say, the dotenv npm package is a popular choice for loading configuration from .env files. It’s a simple and lightweight package that’s easy to use.

Some of my reservations as noted above are relevant to dotenv too, but specifically I want to call out a security concern with dotenv that I’ve seen developers overlook: basing configuration on process.env as a first-class citizen in a Node.js pattern.

Let’s explain what this means with a simple dotenv use case:

const dotenv = require('dotenv');
const result = dotenv.config()

console.log('env vars:');
console.log(process.env);

When you run this code, the dotenv package will load configuration data from the local .env file into the process.env object. This means that you can access the configuration data via process.env anywhere in your application.

Developers following this pattern will often use process.env as a first-class citizen in their application, from which they draw configuration items:

const db = require('./db')
const dotenv = require('dotenv');
const result = dotenv.config()

db.connect(process.env.DB_URL);

The security risk that is inherent here is:

You may inadvertently expose sensitive information like database credentials and API keys as part of error messages, stack traces, and other forms of data returned to consuming clients.
You may inadvertently expose sensitive information like database credentials and API keys in your application’s logs.

4 Node.js configuration anti-patterns

What are some common anti-patterns or practices that you should avoid when handling configuration in Node.js applications?

These anti-patterns may seem tempting at times but can lead to maintenance nightmares, security vulnerabilities, and overall codebase chaos.

1. Hard-coding configuration data

One of the most prevalent anti-patterns is hardcoding configuration data directly into your source code.

It sounds so naive that you might be wondering why anyone would ever do this, but more often than not, it’s a common “quick fix” that developers resort to when they’re in a hurry.

While hardcoding configuration data may appear convenient for quick development, it has several downsides ranging from security risks of exposing sensitive information in the codebase to inflexibility and poor maintainability.

Node.js configuration anti-pattern to avoid, demonstrating hardcoded configuration data in the codebase:

const express = require('express');
const mongoose = require('mongoose');

const app = express();

// Anti-Pattern: Hardcoded configuration data
const dbUsername = 'your_db_username';
const dbPassword = 'your_db_password';
const dbHost = 'localhost';
const dbPort = '27017';
const dbName = 'your_database_name';

// Establish a database connection
mongoose.connect(`mongodb://${dbUsername}:${dbPassword}@${dbHost}:${dbPort}/${dbName}`, {
  useNewUrlParser: true,
  useUnifiedTopology: true,
});

const db = mongoose.connection;

app.get('/', (req, res) => {
  res.send('Hello World');
});

// Anti-Pattern: Hardcoded configuration data
app.listen(3000, () => {
  console.log(`Server is running on port ${3000}`);
});

In this code, the database credentials (i.e., dbPassword) are hardcoded directly into the codebase. The HTTP web server port of 3000 is also hardcoded.

2. Scattered configuration data

Another common pitfall is scattering configuration data across multiple files or modules within your application.

This haphazard approach can result in maintenance nightmares as you hunt down which part of your application is responsible for a particular configuration. It also makes it difficult to update configuration values when they’re spread across different parts of your codebase.

To add insult to injury, inconsistencies may arise when different parts of your application use different values for the same configuration parameter. Also, good luck debugging!

Spreading env variable usage across all your source files (or dependencies) is a recipe for an unmaintainable codebase.
— Matteo Collina (@matteocollina) August 16, 2023

Node.js configuration anti-pattern to avoid, demonstrating scattered configuration data across multiple files:

// 
// FILE: server.js
// 
const express = require('express');
const app = express();

// Anti-Pattern: Scattered configuration data
const port = 3000;

app.listen(port, () => {
  console.log(`Server is running on port ${port}`);
});

//
// FILE: database.js
// 
const mongoose = require('mongoose');

// Anti-Pattern: Scattered configuration data
const dbUsername = 'your_db_username';
const dbPassword = 'your_db_password';
const dbHost = 'localhost';
const dbPort = '27017';
const dbName = 'your_database_name';

mongoose.connect(`mongodb://${dbUsername}:${dbPassword}@${dbHost}:${dbPort}/${dbName}`, {
  useNewUrlParser: true,
  useUnifiedTopology: true,
});

//
// FILE: routes.js
//
const express = require('express');
const router = express.Router();

// Anti-Pattern: Scattered configuration data
const apiBaseUrl = process.env.API_BASE_URL;
const apiToken = process.env.API_TOKEN;

router.get('/data', (req, res) => {
  // Make a request to an external API using the scattered configuration data
  // ...
});

module.exports = router;

You’ll notice that a variation of this anti-pattern is when scattered configuration data is accessed via environment variables such as process.env.API_BASE_URL through-out the codebase.

While environment variables are a viable option for configuration management, they can quickly become unwieldy when used in this manner.

3. Environment-dependent configurations

Manually switching between different configuration settings based on the environment (e.g., development, staging, production) is another anti-pattern. This involves writing conditional code blocks to handle different settings, which can lead to:

Human error: Mistakes can happen, and you might forget to switch the environment, potentially leading to data corruption or other issues.
Code complexity: Your codebase can quickly become cluttered with environment-specific logic, making it harder to understand and maintain.

Node.js configuration anti-pattern to avoid, demonstrating environment-dependent configurations

const express = require('express');
const app = express();

let dbUrl;

// Anti-Pattern: Environment-dependent configurations
if (process.env.NODE_ENV === 'development') {
  dbUrl = 'mongodb://localhost:27017/dev_database';
} else if (process.env.NODE_ENV === 'production') {
  dbUrl = 'mongodb://dbserver.prod:27017/prod_database';
} else {
  dbUrl = 'mongodb://localhost:27017/test_database';
}

app.get('/', (req, res) => {
  // Use the dbUrl based on the environment
  // ...
});

const PORT = process.env.PORT || 3000;

app.listen(PORT, () => {
  console.log(`Server is running on port ${PORT}`);
});

Another variation of this pattern is when different configuration files are used for different environments, such as config.staging.json, config.dev.json, and so on. This practice can lead to code duplication and maintenance challenges.

Consider the following directory structure:

- config
  - config.dev.json
  - config.staging.json
  - config.prod.json
- server.js

In this directory structure, there are separate configuration files for different environments.

The config/config.dev.json file:

{
  "dbUrl": "mongodb://localhost:27017/dev_database",
  "apiBaseUrl": "http://localhost:3000/dev"
}

And here’s what the config/config.staging.json file might look like:

{
  "dbUrl": "mongodb://dbserver.staging:27017/staging_database",
  "apiBaseUrl": "http://localhost:3000/staging"
}

Then, a Node.js application might load it’s environment-specific configuration data as follows, demonstrating this anti-pattern:

const express = require('express');
const app = express();

const environment = process.env.NODE_ENV || 'development';
const config = require(`./config/config.${environment}.json`);

app.get('/', (req, res) => {
  // Use configuration values from the loaded config file
  const dbUrl = config.dbUrl;
  const apiBaseUrl = config.apiBaseUrl;

  // ...
});

const PORT = process.env.PORT || 3000;

app.listen(PORT, () => {
  console.log(`Server is running on port ${PORT}`);
});

In fact, even the official dotenv package mentions this anti pattern as part of the README:

Should I have multiple .env files? No. We strongly recommend against having a “main” .env file and an “environment” .env file like .env.test. Your config should vary between deploys, and you should not be sharing values between environments.

4. Non-Asynchronous configuration loading

Another common anti-pattern I see often repeating in how configuration libraries on npm work is that they are solely based on a synchronous API.

In practice, this means that the configuration is treated as “local” and loading configuration data synchronously becomes a problem when you need to fetch configuration from a remote source, such as a database or an API. For example, you might want to load your secrets from a KMS (Key Management Service) or a secrets manager like HashiCorp Vault.

Sure, you can load your synchronous config first, and then have further asynchronous function calls and then wrap it all in a configuration manager factory, however most developers don’t do this to begin with because they are hard-wired to follow synchronous configuration loading due to npm packages built that way.

Here are a few examples of how common this non-asynchronous configuration loading anti-pattern is, demonstrating node-config, dotenv, and convict as some examples:

// Example: node-config (https://www.npmjs.com/package/config)
const config = require('config');
//...
const dbConfig = config.get('Customer.dbConfig');
db.connect(dbConfig, ...);

WARNING

The node-config project is actually making use of the config npm package name. The node-config npm package is something else entirely.

// Example: dotenv (https://www.npmjs.com/package/dotenv)

require('dotenv').config()
console.log(process.env)

How to load configuration from environment variables in Node.js ?

One last and closing section on configuration patterns in Node.js is loading configuration from environment variables.

Through-out this article we’ve mentioned numerous open-source npm packages dedicated to managing a Node.js configuration in various ways:

Yet, if you haven’t been following the latest Node.js 20 feature enhancements, then you might not be aware that Node.js now has built-in support for loading configuration from environment variables:

node --env-file=.env server.js

This new environment variables loading feature was introduced in Node.js v20.6.0 and allows you to specify an INI-compatible file format that is commonly used through-out other libraries and tools:

PORT=3000
DB_URL=mongodb://localhost:27017/dev_database

Once the Node.js application starts, the PORT and DB_URL environment variables defined in the .env file will be available to the application via: process.env.PORT and process.env.DB_URL.

In fact, if you need to pass NODE_OPTIONS configuration then you can now include that in your .env file too and Node.js will automatically pick it up:

NODE_OPTIONS=--max-old-space-size=4096
PORT=3000

This feature was added by Yagiz Nizipli

Appreciate open-source developers ♥️

How do we manage configuration in Node.js applications?

My goal in this write-up was specifically about raising awareness of anti-patterns, security and maintainability issues that are common when handling configuration in Node.js applications.

Friends have also shared other perspectives relating to configuration management in Node.js applications, such as Colin Ihrig’s blog post on NODE_ENV Considered Harmful and Shai Yallin’s dotenv considered harmful walk-through, which I recommend your read too for further perspectives.

So, how do we do better?

Read-up in my follow-up article Best Practices for Bootstrapping a Node.js Application Configuration.