Docker Log Rotation - Definition, Configuration Guide, and Best Practices

Docker containers generate logs to monitor their operations, but without a mechanism in place to manage these logs, they can grow indefinitely, leading to excessive disk space consumption and performance degradation. Implementing docker log rotation is crucial to control log file size and quantity, ensuring efficient log management and optimal system performance.

In this guide, we will cover how to configure docker log rotation, starting with an overview of what is log rotation and moving into a step-by-step guide for configuring limits on log size. This configuration will help you maintain a well-organized, high-performing docker environment.

Let's get started!

In this Blog Post

1. What is docker log rotation?
2. How log rotation helps manage log files?
3. Configuring log rotation in docker
4. Configuring logging driver for a container
5. What is the best practice for log rotation?
6. Conclusion

What is docker log rotation?

Docker log rotation is a method used to control the size and the number of log files, preventing them from growing without limit. It helps avoid excessive disk space usage by automatically compressing or removing old log files when certain conditions are met. Once set up, log rotation operates automatically, so you don't need to manually handle the log files as docker will manage the log rotation according to the configured rules.

Before getting to know more about docker log rotation, let’s have a quick look at default docker logging driver.

Default docker logging driver:

Just like any other application, docker containers generate logs to track their activities. Docker uses logging drivers to manage these logs, and by default, it uses the json-file driver to collect and store these logs.

So, where are these docker logs stored?

Docker stores logs in the /var/lib/docker/containers/[container-id]/[container-id]-JSON.log directory on the host machine. Each container has its own folder, named after its container ID, where its logs are kept.

Are docker logs rotated?

Docker logs are not automatically rotated by default. One important thing to note is that docker doesn't set limits on the size or number of log files. As a result, logs can keep growing indefinitely, which may eventually lead to disk space exhaustion and performance problems.

This is where the concept of log rotation comes in. Let’s see how log rotation can help manage log files more effectively and prevent these issues.

How log rotation helps manage log files?

Here is how log rotation can help manage docker log files:

• Log rotation sets size limits for log files, preventing them from exceeding specified limits and consuming excessive disk space.
• It archives and compresses older log files, optimizing disk usage while preserving historical data for future reference.
• By regularly creating new log files, log rotation ensures logs are systematically organized and individual files remain within manageable sizes.
• The process of log rotation automates log file handling, reducing manual intervention and ensuring consistent log management practices.

By now, you have a good understanding of what docker log rotation is and why it's needed. So, let’s dive right into how you can configure it.

Configuring log rotation in docker

Follow the steps give below  to configure log rotation:

Step 1: To configure log rotation, first you need to set up the docker daemon with specific logging settings. Locate the docker daemon configuration file.

On Linux:  /etc/docker/daemon.json

On Windows: C:\ProgramData\docker\config\daemon.json

Step 2: Check if the logging driver is already set. If yes, view the current logging driver for the docker daemon, by running the following command.

docker info --format '{{.LoggingDriver}}'

Docker offers multiple logging drivers, but only the json-file and local logging driver provide direct support for built-in log rotation.

Take a look at logging drivers supported by docker. In addition, you can also install and use logging driver plugins.

Step 3: If the logging driver is not set yet, you can configure it by using the following command.

For local driver:

{
  "log-driver": "local"
}

For json-file driver:

{
  "log-driver": "json-file"
}

Note: If you don’t specify a logging driver, the default is json-file.

Step 4: Configure the log rotation settings.

For json-file logging driver:

{
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "10m",
    "max-file": "5"
  }
}

The max-size specifies the maximum size of log files, and max-file specifies the number of log files to keep. The above configuration allows for a maximum of 5 log files, each not exceeding 10 megabytes in size. You can adjust the max-size and max-file settings based on your needs.

The json-file logging driver offers the following logging options,

For local logging driver:

{
  "log-driver": "local",
  "log-opts": {
    "max-size": "10m",
    "max-file": "5"
  }
}

Similar to json-file logging driver, you can adjust the max-size and max-file settings based on your needs.

The local logging driver offers the following logging options,

Step 5: Save the file and restart the docker service.

Restart docker for the changes to apply to newly created containers. Existing containers will not automatically adopt the new logging configuration.

systemctl restart docker

Configuring logging driver for a specific container

If you prefer not to apply log rotation settings globally and want to configure them specific for a container, you can do so by specifying the logging driver and its options directly when starting a container. Here’s how you can set it up.

You can configure the logging driver for a specific container by using the --log-driver flag with the docker container create or docker run command. The --log-driver flag allows you to specify which logging driver docker should use for a particular container.

Using json file

docker run \
      --log-driver json-file --log-opt max-size=10m \
      mysql echo hello world

Using local file

docker run \
      --log-driver local --log-opt max-size=10m \
      mysql echo hello world

To check the current logging driver for a running container, use this command:

docker inspect -f '{{.HostConfig.LogConfig.Type}}' <CONTAINER>

Set up log rotation for the specified container by using the following command:

docker run \
    --log-driver local \
    --log-opt max-size=15m \
    --log-opt max-file=5 \
    mysql echo hello world

This command starts a new docker container with specific logging settings.

What is the best practice for log rotation?

Here are some of the best practices to consider for effective Docker log management:

• Consider archiving older log files instead of deleting them to retain historical data for analysis or compliance purposes.
• Regularly check disk space usage to find any potential issues early, especially in environments with high log output.
• Before deploying changes to production, test your log rotation configuration in a staging environment to ensure it behaves as expected.
• To prevent overloading local storage and improve security, periodically transfer log files to centralized servers.
• Log retention varies based on data type and storage capacity, ranging from a week to a year or more. Monthly backups ensure important logs are preserved while duplicates are removed.

Conclusion

In this blog post, we have discussed the crucial role of docker log rotation in managing log files effectively, preventing them from consuming excessive disk space and degrading performance. As docker containers generate logs that can grow indefinitely, implementing log rotation is essential for maintaining system efficiency. We covered how to set up log rotation in docker to keep log files in check.

With various log management solutions available, choosing the right one can be challenging. Atatus, a Full Stack Observability Platform, offers a robust solution for storing, managing, and analysing logs at scale.

With Atatus's affordable and scalable logs monitoring platform, you can see everything across your entire system. All your log data is processed in one place, making it easy to visualize. Atatus brings together different types of data say logs, metrics, and traces into one monitoring tool. In addition to logs, it helps you track metrics and traces, giving you a complete picture of your system's performance.

Try Atatus free for 14 days—sign up now!