Python Logging: The Complete Guide

In computing, logging involves maintaining a record of events within a computer system, encompassing issues, errors, or pertinent information about ongoing operations. These events can manifest within the operating system or other software, with each occurrence documented through the creation of a message or log entry.

Logging is crucial for comprehending the application's behavior, aiding in the identification of unexpected issues, debugging, or merely monitoring events. Many software companies use third-party log monitoring tools such as Atatus for logging, while others employ built-in modules for regular performance monitoring in an application.

Many programming languages and frameworks, such as Java, Python, Ruby, .NET, etc., come with built-in logging modules or provide logging facilities as part of their standard libraries. In this article, we will explore Python logging and its built-in module, logger.

Table Of Contents:

1. Python Logging: Overview
2. Key Concepts of Python Logging
3. Use Cases of Python Logging Module
4. Getting Started with Logging in Python
5. Best Methods for Logging in Python

Python Logging: Overview

Python Logging is an integrated module that offers an adaptable and configurable logging framework for capturing messages from a Python application. The main goal of logging is to gather details about how the program is being executed, which helps debug and understand the program's behavior.

Key Concepts of Python Logging

i.) Log Levels

Logging is compatible with several severity levels, including DEBUG, INFO, WARNING, ERROR, and CRITICAL. Depending on the significance of the logged message, developers can select a suitable level.

The default level is WARNING, which implies that only events with this level or above will be monitored unless the logging package is set to do something different.

ii.) Loggers

Loggers are hierarchical entities named after the module or component they represent. They organize and categorize log messages. The root logger is the default logger returned by logging.getLogger().

iii.) Handlers

Handlers decide where log messages are sent. Python includes several built-in handlers, such as StreamHandler for console output and FileHandler for writing logs to files.

iv.) Formatters

Formatters define the structure of log messages by setting their content and layout. Developers can change the formatting of log records, including the timestamp, log level, and message.

v.) Configuration

Logging can be configured via the basicConfig function or by defining a custom configuration. This includes configuring the logging level, adding handlers, and specifying formatting.

vi.) Log Records

Log records are instances of the LogRecord class that carry information about a particular event. They include the log level, logger name, message, and other important data.

vii.) Propagation

Loggers can be set up to send messages to their parent loggers in the hierarchy. This enables a more modular approach to logging, with different portions of the program having their own loggers.

Use Cases of Python Logging Module

1. The Python built-in logging module provides various advantages that improve the logging experience for developers. One of its primary strengths is its adaptability, which allows it to capture log messages at multiple levels (DEBUG, INFO, WARNING, ERROR, and CRITICAL) to meet a variety of needs.
2. Another noteworthy feature is customization, which allows developers to modify how logs, handlers, and formatters behave. This adaptability extends to loggers' hierarchical structure, enabling structured and modular logging. An application's many components can each have its logger, and log messages can go up or down the logger hierarchy as necessary.
3. The module has built-in handlers for typical use cases, making it easier to set up simple logging configurations. Its ease of use is demonstrated by a simple API that allows for quick integration of logging into Python programs. Furthermore, the module is thread-safe, making it suitable for usage in multi-threaded programs without the risk of data corruption or race conditions.
4. A noteworthy benefit is extensibility, which enables programmers to design unique handlers, filters, and formatters to customize the logging system to meet specific application needs. The module's ability to be extended is consistent with its interoperability with numerous third-party libraries and frameworks in the Python environment, which helps to maintain consistency across various parts.
5. The gentle decline of the module is an additional useful feature. The default setup makes sure that the application won't crash and that logs will still be collected even if logging isn't explicitly configured.
6. The organized structure of log messages facilitates easy interaction with third-party logging and monitoring systems. This makes it easier to analyze logs centrally, which helps engineers find patterns and solve problems quickly.
7. The logging module's cross-platform functionality allows it to be compatible with many operating systems, which contributes to its widespread usage in the Python community.

Getting Started with Logging in Python

In Python, basic logging entails setting up the logging module to record and produce log messages. Here is a basic illustration with clarifications:

import logging

# Configure the logging module
logging.basicConfig(
    level=logging.DEBUG,  # Set the minimum logging level
    format='%(asctime)s - %(levelname)s - %(message)s'  # Define the log message format
)

# Example usage of different log levels
logging.debug("This is a debug message")    # Not typically shown in production
logging.info("This is an info message")      # General information
logging.warning("This is a warning message")  # Indication of potential issues
logging.error("This is an error message")     # Indication of a serious problem
logging.critical("This is a critical message")  # A very serious error

# Example of logging variable values
variable_value = 42
logging.info("The value of the variable is %s", variable_value)

# Example of exception logging
try:
    result = 10 / 0
except Exception as e:
    logging.exception("An error occurred: %s", e)

Explanation:

i.) Import the ‘logging’ Module

Let's import the logging module to log messages that you want to display with the Logger tool.

import logging

Bring in the logging module from the Python standard library.

ii.) Configure Logging

To enable simple logging to a file, use the basicConfig() constructor, as illustrated below.

logging.basicConfig(
level=logging.DEBUG,
format='%(asctime)s - %(levelname)s - %(message)s'
)

To set up the logging module:

1. Use basicConfig.
2. Set the logging level to DEBUG to capture messages at all levels.
3. Create a format for log messages that includes the timestamp, log level, and actual message.

iii.) Logging at Different Levels

The log message has five built-in levels.

logging.debug("This is a debug message")
logging.info("This is an info message")
logging.warning("This is a warning message")
logging.error("This is an error message")
logging.critical("This is a critical message")

Display logging at various levels. The real messages will be shown or kept according to the logging level that has been selected.

iv.) Logging Variable Values

To log variable data, we can describe the event with a string and then add the variable data as arguments. Let us see how we can achieve this with the following example:

variable_value = 42
logging.info("The value of the variable is %s", variable_value)

Demonstrate how to log variable values inside log messages.

v.) Logging Exceptions

In Python, the logging.exception() method can be used to log an error. This function logs a message with the ERROR level to the logger.

try:
result = 10 / 0
except Exception as e:
logging.exception("An error occurred: %s", e)

This section shows how to log exceptions together with their tracebacks. The exception technique automatically adds the traceback to the log message.

When you run this code, log messages will be printed to the console in the format you chose. Adjust the logging level and format to meet your specific needs.

Filters and Handlers

Additional parts that provide you more control over which log messages are handled and sent and where handlers and filters are. Here's an illustration using handlers and filters:

import logging

# Define a custom filter to log only messages containing the word 'important'
class ImportantFilter(logging.Filter):
    def filter(self, record):
        return 'important' in record.getMessage().lower()

# Create a logger
logger = logging.getLogger("example_logger")

# Configure the logging module
logging.basicConfig(
    level=logging.DEBUG,
    format='%(asctime)s - %(levelname)s - %(message)s'
)

# Add a custom filter to the logger
logger.addFilter(ImportantFilter())

# Create a file handler and set its level to ERROR
file_handler = logging.FileHandler('error_logs.txt')
file_handler.setLevel(logging.ERROR)

# Create a console handler
console_handler = logging.StreamHandler()

# Create a formatter and set it for both handlers
formatter = logging.Formatter('%(asctime)s - %(levelname)s - %(message)s')
file_handler.setFormatter(formatter)
console_handler.setFormatter(formatter)

# Add the handlers to the logger
logger.addHandler(file_handler)
logger.addHandler(console_handler)

# Example usage of different log levels
logger.debug("This is a debug message")  # Not shown due to the custom filter
logger.info("This is an info message")    # Not shown due to the custom filter
logger.warning("This is a warning message")
logger.error("This error is important")
logger.critical("This is a critical message")

# Example of logging variable values
variable_value = 42
logger.info("The value of the variable is %s", variable_value)

# Example of exception logging
try:
    result = 10 / 0
except Exception as e:
    logger.exception("An error occurred: %s", e)

Explanation:

i.) Custom Filter

The Python logging library contains a Filter class that can be used to filter log records. This is a simple approach to ensure that a logger or handler only outputs the log messages that you want.

class ImportantFilter(logging.Filter):
def filter(self, record):
return 'important' in record.getMessage().lower()

Create a custom filter called ImportantFilter that only permits log messages that contain the word important to be processed.

ii.) Logger Configuration

To begin logging using the Python logging module, call the factory function logging.getLogger(name) is normally called. The getLogger() function receives a single argument: the logger's name. It returns a reference to a logger instance with the specified name, if one is available, or root otherwise.

logger = logging.getLogger("example_logger")
Create a custom logger called "example_logger."

iii.) Adding a Filter

Add the custom filter to the logger so only messages that fulfill the filter parameters are processed.

logger.addFilter(ImportantFilter())

iv. )Handlers

The Python standard logging package includes a Handler class, which essentially defines how a certain log message is handled. For example, we can configure a simple handler that outputs log messages to the terminal as follows:

file_handler = logging.FileHandler('error_logs.txt')
file_handler.setLevel(logging.ERROR)
console_handler = logging.StreamHandler()

Create a FileHandler to save error messages ( error_logs.txt ) and a StreamHandler to print them to the console.

v.) Formatter

To specify the log message format, create a formatter and set it for each handler.

formatter = logging.Formatter('%(asctime)s - %(levelname)s - %(message)s')
file_handler.setFormatter(formatter)
console_handler.setFormatter(formatter)

vi.) Adding Handlers to the Logger

logger.addHandler(file_handler)
logger.addHandler(console_handler)

Add the handlers that have been established to the logger and indicate the destination for log messages at various levels.

Example Usage

logger.debug("This is a debug message")  # Not shown due to the custom filter
logger.info("This is an info message")    # Not shown due to the custom filter
logger.warning("This is a warning message")
logger.error("This is an error message")
logger.critical("This is a critical message")

Showcase various logging levels while excluding debug and info messages with the custom filter.

vii.) Logging Variable Values and Exceptions

logger.info( The value of the variable is %s", variable_value )

try:
result = 10 / 0
except Exception as e:
logger.exception("An error occurred: %s", e)

Describe how to log variable values and exceptions, with the exception log automatically containing the traceback.

This example demonstrates a more comprehensive logging setup that includes a custom filter and several handlers, giving you more flexibility over log message processing and output destinations. Adjust the filter criteria, handlers, and log levels to suit your individual use case.

Best Methods for Logging in Python

A crucial component of software development is logging, which helps with problem-solving, issue identification, and behavior analysis of applications. The following are some recommendations for efficient Python logging:

a.) Use Different Log Levels Appropriately

For every log message, select the relevant log level (DEBUG, INFO, WARNING, ERROR, CRITICAL). Use DEBUG for comprehensive debugging information; use CRITICAL for serious issues that could cause the application to terminate.

b.) Configure Logging Early in the Program

Configure logging in to your program as soon as possible. Ideally, this should happen at the script or application launch. This guarantees that every log message is recorded immediately.

c.) Customize Log Formats

Use formatters to provide crucial elements to log messages, such as timestamps, log levels, and particulars about the log occurrence. This makes log analysis easier.

d.) Create Named Loggers

Use named loggers for various parts or modules of your program. This allows you to regulate how your code logs certain sections individually.

e.) Hierarchical Logger Names

Hierarchically assemble logs according to your application's structure. This helps regulate the flow of log messages via the logger hierarchy and modularize logging.

f.) Avoid Hardcoding Log Levels

Avoid hardcoding log levels into your code. Instead, utilize configuration files or command-line options to specify the appropriate log level. This adds flexibility without requiring code changes.

g.) Use Logging Configuration Files

Consider using configuration files (e.g., JSON or YAML) for more complicated logging configurations. This enables you to change logging options without altering the code.

h.) Implement Rotating Log Files

Rotate logs for long-running applications to keep log files from growing unnecessarily big. This makes log management and analysis simpler.

i.) Avoid Excessive Logging

Take note of how much time you spend logging into your application. Overlogging can affect system efficiency and make it difficult to sift through a deluge of data and locate crucial signals.

j.) Handle Exceptions with Logging

Logging records exceptions, including stack traces. This helps with debugging and offers useful data for determining the source of issues.

k.) Include Contextual Information

Include contextual information in your log messages, such as variable values or extra facts about the application's current state. This information can be quite helpful while troubleshooting.

l.) Test Logging Configurations

Test your logging options, particularly in scenarios with varying log volumes and fault conditions. This guarantees that logs offer the correct information during development and production.

m.) Monitor and Analyze Logs

Set up log monitoring and analysis tools to detect problems and patterns before they arise. Centralised log management solutions can help aggregate logs for easier examination.

n.) Keep Security in Mind

Be cautious about logging sensitive information. To maintain the security of your application, ensure that log messages don't include sensitive data, such as passwords or personal information.

o.) Document Logging Conventions

Document the logging rules and procedures in your project. This will allow new developers to grasp the logging technique and promote uniformity throughout the application.

Conclusion

Python logging is an essential tool for developers, providing a robust and flexible framework for capturing, analyzing, and understanding application runtime behavior.

By following best practices such as selecting appropriate log levels, hierarchically organizing loggers, and customizing log formats, developers can streamline the debugging process and improve the general maintainability of their codebase.

The hierarchical nature of loggers, combined with the ability to configure logging settings early in the program, facilitates modularisation and efficient control over the flow of log messages; additionally, the incorporation of filters and handlers allows for a fine-grained approach to logging message processing, allowing developers to tailor their logging setups to specific needs.

Atatus: Python Performance & Logs Monitoring

As developers, we grasp the hurdles of upholding peak performance and monitoring in dynamic environments. That's precisely why Atatus offers you the capability to monitor your entire application seamlessly.

Access real-time insights into performance metrics, monitor application health with customizable dashboards, and receive proactive alerts. Delve deep into code-level insights, fine-tune performance, and ensure scalability with Atatus Python Performance Monitoring.

Streamline troubleshooting by consolidating logs from across your infrastructure, effortlessly searching and filtering log data, and extracting invaluable insights to resolve issues promptly. Enhance your logging capabilities and optimize troubleshooting with Logs Monitoring.

Try your 14-day free trial of Atatus.