13. How to Use Logging with the Python logging Module¶

Overview¶

The Python logging module is a standard library for recording application events, diagnostics, and runtime information in a structured and configurable manner. It provides a flexible framework to emit log messages with different severity levels, route them to multiple destinations, and control formatting and filtering. Common use cases include application debugging, operational monitoring, audit trails, and incident analysis in both development and production environments.

Logging Architecture¶

Core Concepts¶

The logging system is built around a small set of composable components, each with a clear responsibility.

Severity Levels¶

Logging levels define the importance of a message and are used for filtering.

Basic Configuration¶

Minimal Setup¶

The logging module can be configured with a basic setup suitable for simple scripts.

import logging

logging.basicConfig(
    level=logging.INFO,
    format="%(asctime)s %(levelname)s %(name)s %(message)s"
)

logging.info("Application started")

This configuration routes log messages to standard output using a default handler.

Configuration¶

There are 3 basic type of method of configuration ¹

flowchart LR
lc[logging.config method]
lc --> d[dictConfig] --> dm[Using JSON to declare schema] --> dr[Can use json package read]
lc --> f[fileConfig] --> fm[Using logging.ini to declare schema] --> fr[Use configparser to read]
lc --> l[listen] --> s[Socket Mode]
lc --> k[stopListening] --> s
s --> o[Handle in raw test on the script]

The method with dict or file is familar together and can switch. But from the note of element, there are various differnt which the lower API in the backend.

Note The fileConfig() API is older than the dictConfig() API and does not provide functionality to cover certain aspects of logging. For example, you cannot configure Filter objects, which provide for filtering of messages beyond simple integer levels, using fileConfig(). If you need to have instances of Filter in your logging configuration, you will need to use dictConfig(). Note that future enhancements to configuration functionality will be added to dictConfig(), so it"s worth considering transitioning to this newer API when it"s convenient to do so.

Logger Usage Pattern¶

Creating and Using Loggers¶

Each module should create its own logger using a hierarchical naming convention.

import logging

logger = logging.getLogger(__name__)

logger.debug("Debug message")
logger.info("Informational message")
logger.error("Error message")

Logger names form a hierarchy based on dot notation, enabling centralized control.

Execution Flow¶

Logging Event Lifecycle¶

The logging process follows a deterministic sequence.

(1) Application code calls a logger method with a severity level and message (2) The logger creates a LogRecord containing contextual metadata (3) The logger checks the effective logging level to decide if processing continues (4) Applicable filters are evaluated (5) The LogRecord is passed to one or more handlers (6) Each handler formats the record using its formatter (7) The formatted message is emitted to the configured destination

Handlers and Outputs¶

Common Handler Types¶

Handler Configuration Example¶

import logging
from logging.handlers import RotatingFileHandler

handler = RotatingFileHandler(
    "app.log",
    maxBytes=10_000_000,
    backupCount=5
)

formatter = logging.Formatter(
    "%(asctime)s %(levelname)s %(name)s %(message)s"
)

handler.setFormatter(formatter)

logger = logging.getLogger("app")
logger.setLevel(logging.INFO)
logger.addHandler(handler)

Formatters and Structured Output¶

Log Message Structure¶

Formatters control which attributes from the LogRecord are included.

Common attributes include: (1) asctime (2) levelname (3) name (4) message (5) process (6) thread

Structured logging can be implemented by emitting JSON-formatted messages, but this requires custom formatters or third-party libraries.

Formatter instances are used to convert a LogRecord to text.

Formatters need to know how a LogRecord is constructed. They are responsible for converting a LogRecord to (usually) a string which can be interpreted by either a human or an external system. The base Formatter allows a formatting string to be specified. If none is supplied, the style-dependent default value, "%(message)s", "{message}", or "${message}", is used.

The Formatter can be initialized with a format string which makes use of
knowledge of the LogRecord attributes - e.g. the default value mentioned
above makes use of the fact that the user's message and arguments are pre-
formatted into a LogRecord's message attribute. Currently, the useful
attributes in a LogRecord are described by:

%(name)s            Name of the logger (logging channel)
%(levelno)s         Numeric logging level for the message (DEBUG, INFO,
                    WARNING, ERROR, CRITICAL)
%(levelname)s       Text logging level for the message ("DEBUG", "INFO",
                    "WARNING", "ERROR", "CRITICAL")
%(pathname)s        Full pathname of the source file where the logging
                    call was issued (if available)
%(filename)s        Filename portion of pathname
%(module)s          Module (name portion of filename)
%(lineno)d          Source line number where the logging call was issued
                    (if available)
%(funcName)s        Function name
%(created)f         Time when the LogRecord was created (time.time()
                    return value)
%(asctime)s         Textual time when the LogRecord was created
%(msecs)d           Millisecond portion of the creation time
%(relativeCreated)d Time in milliseconds when the LogRecord was created,
                    relative to the time the logging module was loaded
                    (typically at application startup time)
%(thread)d          Thread ID (if available)
%(threadName)s      Thread name (if available)
%(process)d         Process ID (if available)
%(message)s         The result of record.getMessage(), computed just as
                    the record is emitted

Best Practices and Limitations¶

Operational Guidelines¶

(1) Do not use the root logger directly in application modules (2) Avoid excessive DEBUG logging in production environments (3) Treat log messages as immutable audit data (4) Never log sensitive information such as credentials or personal data

Limitations¶

(1) The standard logging module does not provide native JSON logging (2) Distributed tracing correlation requires manual context propagation (3) Misconfigured logger hierarchies can result in duplicate log entries

Reference¶

• https://docs.python.org/3/library/logging.html
• https://docs.python.org/3/howto/logging.html
• https://docs.python.org/3/howto/logging-cookbook.html