What is and how to configure the MongoDB WiredTiger Cache?

Is your MongoDB performance slower than a Sunday afternoon? Does your development team constantly complain about latency, and you, as an IT manager or DBA, feel like the data infrastructure is about to collapse? If you relate to this scenario, you can be sure: you are not alone. Many medium and large companies face the challenge of scaling their NoSQL database operations without losing agility.

The good news is that the solution might be in a key component: the WiredTiger Cache. But what exactly is it and how do you configure it to prevent your data from becoming a colossal bottleneck?

In this article, we’ll take a deep dive into the MongoDB architecture, uncover the crucial role of WiredTiger, and, most importantly, show how a correct configuration can be the difference between a robust system and one that is begging for help. And if, at the end, you realize that this task is too complex to do on your own, we’ll show you why HTI Tecnologia is the ideal partner to ensure the health and high availability of your data.

Why the WiredTiger Cache is the Heart of Your MongoDB?

In essence, MongoDB uses WiredTiger as its default storage engine. Think of it as the engine of your car: it’s responsible for how data is read, written, and, most importantly, stored on disk. However, the magic of performance doesn’t happen directly on the hard drive. It resides in RAM, specifically, in the WiredTiger Cache.

This cache is an area in memory that stores the most frequently accessed and modified data. The logic is simple but powerful: it’s much faster to read and write data in RAM than on a disk. Therefore, the more “hot” data you can keep in the cache, the faster your database will respond.

MongoDB’s default configuration allocates up to 50% of the server’s total RAM for the WiredTiger Cache. Sounds good, right? Not always. In complex environments, where the same server hosts other applications or the operating system consumes a significant portion of the memory, this default allocation can be insufficient or, worse, cause excessive swapping, leading to a drastic drop in performance.

This is where a DBA’s expertise is crucial. Understanding the workload, the nature of the queries, and the hardware behavior is the first step to optimizing performance.

The 7 Signs of a Problematic Cache Configuration

Your MongoDB may be giving you clear signs that the WiredTiger Cache is not optimized. Ignoring these warnings can lead to serious performance issues and, in extreme cases, service unavailability.

Query Latency: Queries that were once instantaneous start to take time. Your latency monitoring shows frequent spikes, and users start to complain about slowness.

High Disk Usage (I/O): Disk usage (I/O) is always at its limit, indicating that MongoDB has to constantly fetch data from the disk instead of finding it in memory.

Drop in Cache Hit Rate: This is the most direct metric. A low cache hit rate means that most read requests are missing the cache and going directly to disk. Ideally, this rate should be as close to 100% as possible.

Increase in Page Faults: MongoDB tracks page faults, which occur when a required data page is not in memory and needs to be read from disk. An increase in this metric is a clear sign that the cache is too small.

// Connect to your MongoDB shell
// mongo

// Run the command to get server status and look for page faults
db.serverStatus().extra_info.page_faults

Operating System Swapping: The operating system starts using the disk as virtual memory, an extremely slow process. This can be seen through system monitoring tools like vmstat or top.

vmstat -S M 5 5

free -h

top

Inconsistent Response Time: Query latency becomes unpredictable. Sometimes it’s fast, sometimes it’s slow, creating an erratic user experience.

Challenges to Scale: When you try to scale the application or add more users, the database performance does not keep up with the growth.

Have you identified any of these signs? Don’t worry, the situation can be reversed with a precise optimization of the WiredTiger Cache. And for this, manual configuration is the way to go.

How to Configure the MongoDB WiredTiger Cache

WiredTiger’s memory allocation is controlled by the wiredTiger.engineConfig.cacheSizeGB parameter. Adjusting this value is one of the most important optimizations a DBA can make in a MongoDB environment.

Calculating the Ideal Allocation

The golden rule is simple but requires analysis:

Check the server’s total RAM: Use commands like free -h or top.
Calculate the memory to be left for the OS and other processes: Generally, a margin of 1 to 2 GB is sufficient, but this can vary depending on what else is running on the server (monitoring agents, backups, etc.).
Subtract the reserved memory from the total: The result is the memory available for WiredTiger.

Practical Example:

If you have a server with 64 GB of RAM and you want to leave 4 GB for the operating system and other processes, the ideal memory for the WiredTiger Cache would be 60 GB.

storage:
  wiredTiger:
    engineConfig:
      cacheSizeGB: 60

It is important to note that, as of MongoDB 3.4, the default allocation of 50% of RAM was adjusted to a maximum of 1 GB on servers with 1 GB or less of RAM. For servers with more memory, the default allocation is 50% or 1 GB, whichever is greater.

Advanced Adjustments:

In addition to cacheSizeGB, other parameters can be adjusted to optimize WiredTiger:

wiredTiger.engineConfig.journal: The WiredTiger journal guarantees data consistency in case of failure. Disabling it can increase write performance, but with a high risk of data loss. It is not recommended for production environments.

storage:
  journal:
    enabled: true 
  wiredTiger:
    engineConfig:
      cacheSizeGB: 60

wiredTiger.collectionConfig.block_compressor: Data compression is a key feature of WiredTiger. The default compression (snappy) offers a good balance between performance and space savings. Other options like zlib or zstd can be explored for greater compression, but with a possible CPU cost.

storage:
  wiredTiger:
    engineConfig:
      cacheSizeGB: 60
    collectionConfig:
      blockCompressor: zstd

Configuring WiredTiger is not an exact science. It requires continuous monitoring and adjustments based on the analysis of metrics such as cache hit rate and disk I/O. It is a job that requires the experience of a senior DBA, capable of interpreting the data and making strategic business decisions.

// In the mongo shell, to create a new collection with a specific compressor
db.createCollection(
  "myCollectionWithZlib",
  { storageEngine: { wiredTiger: { configString: "block_compressor=zlib" } } }
)

// To change the compressor of an existing collection (requires rebuild)
// This is more complex and can be done with db.collection.reIndex()
// or dump/restore, or by using collMod to change the storage settings
// (but this will not recompress existing data, only new data)
// collMod example (only changes for new documents and future operations):
db.runCommand({
  collMod: "myExistingCollection",
  storageEngine: { wiredTiger: { configString: "block_compressor=zstd" } }
})

Why Hiring a MongoDB Specialist is the Masterstroke?

In a world where agility and performance are competitive differentiators, the temptation to solve everything “in-house” is great. However, the growing complexity of data infrastructures makes outsourcing the DBA not only a smart but an essential strategy.

HTI Tecnologia is the ideal partner for this mission. With a team of dedicated specialists, HTI offers 24/7 consultation, support, and maintenance for MongoDB, ensuring that your database is always operating at its maximum performance, with high availability and cutting-edge security.

Here are the reasons why you should consider outsourcing with HTI:

Specialized Technical Focus: Your developers and DevOps team can focus on what they do best: creating business value. Meanwhile, HTI takes care of all the technical complexity, such as performance tuning of the WiredTiger Cache, ensuring that the database is an ally, not an obstacle.
Risk Reduction: An outsourced DBA does not have a “day off.” HTI’s 24/7 maintenance means that any incident, whether a hardware failure or a performance bottleneck, is handled in real-time, minimizing the risk of downtime.
Operational Continuity: Knowledge is not tied to a single professional. HTI operates with a cohesive team and robust methodologies, ensuring that knowledge about your infrastructure is documented and accessible at any time, avoiding dependence on a single resource.

In one of our success stories, a large Brazilian e-commerce company was facing latency peaks in its catalog operations, which were based on MongoDB. After a thorough consultation by HTI, we identified that the root cause was an inadequate WiredTiger Cache configuration, which was causing frequent swapping. By adjusting the cacheSizeGB and optimizing compression, we managed to reduce latency by 40% and increase processing capacity by 30% in less than 2 weeks, allowing the company to scale its sales without interruptions.

Don’t Leave the Future of Your Data to Chance

Optimizing MongoDB’s WiredTiger Cache is one of the most critical tasks to ensure the performance and stability of your NoSQL database. An incorrect adjustment can lead to slowness, unavailability, and, ultimately, financial losses.

If you realize that internal database management is consuming precious resources from your team, or if the warning signs are flashing, it’s time to act.

Don’t wait until your data infrastructure implodes. Let the experts at HTI Tecnologia take care of it for you.

Your company deserves a robust and reliable data infrastructure. Talk to an HTI Tecnologia specialist now and find out how we can optimize your MongoDB for success.

Schedule a meeting here