Configure Python Logging Using YAML Files

In this tutorial, you’ll learn how to configure Python logging using YAML files.

We’ll cover everything from basic structure to advanced configurations.

Structure of a YAML Logging Configuration File

Let’s explore how to structure a YAML file for logging configuration.

Loggers

Loggers capture logging information. Each logger is a named entity associated with a part of your application.

Handlers

Handlers determine what to do with a log message. They can write messages to the console, files, or other outputs.

Formatters

Formatters specify the layout of log messages, including timestamps, log levels, and other contextual information.

Configure Loggers

In this section, you’ll learn how to configure loggers, including the root logger, named loggers, and setting logging levels.

Root Logger Configuration

To configure the root logger, define its level and handlers in your YAML configuration file.

version: 1
root:
  level: INFO
  handlers: [console]
handlers:
  console:
    class: logging.StreamHandler
    formatter: simpleFormatter
formatters:
  simpleFormatter:
    format: '%(levelname)s:%(name)s:%(message)s'

You can apply this configuration in your Python script like this:

import logging.config
import yaml
with open('logging_config.yaml', 'r') as f:
    config = yaml.safe_load(f)
    logging.config.dictConfig(config)
logging.info('Application started')
logging.debug('Debugging application')

Output:

INFO:root:Application started

Only the INFO message is displayed because the root logger level is set to INFO. The DEBUG message is not shown since DEBUG is a lower level than INFO.

Named Loggers

You can create loggers with specific names to control logging for different modules.

version: 1
loggers:
  my_module:
    level: DEBUG
    handlers: [console]
handlers:
  console:
    class: logging.StreamHandler
    formatter: detailedFormatter
formatters:
  detailedFormatter:
    format: '%(asctime)s - %(name)s - %(levelname)s - %(message)s'

In your Python script, get the named logger:

import logging
import logging.config
import yaml
with open('logging_config.yaml', 'r') as f:
    config = yaml.safe_load(f)
    logging.config.dictConfig(config)
logger = logging.getLogger('my_module')
logger.debug('Debug message from my_module')
logger.info('Info message from my_module')

Output:

2023-10-10 10:00:00,000 - my_module - DEBUG - Debug message from my_module
2023-10-10 10:00:00,001 - my_module - INFO - Info message from my_module

Both DEBUG and INFO messages are displayed because the my_module logger level is set to DEBUG.

Set Logging Levels

Logging levels control which messages are emitted. You can set levels like DEBUG, INFO, WARNING, ERROR, and CRITICAL.

version: 1
root:
  level: WARNING
  handlers: [console]
handlers:
  console:
    class: logging.StreamHandler
    formatter: simpleFormatter
formatters:
  simpleFormatter:
    format: '%(levelname)s:%(message)s'

In your script:

import logging
import logging.config
import yaml
with open('logging_config.yaml', 'r') as f:
    config = yaml.safe_load(f)
    logging.config.dictConfig(config)
logging.warning('This is a warning')
logging.error('This is an error')
logging.info('This is an info message')

Output:

WARNING:This is a warning
ERROR:This is an error

Only WARNING and ERROR messages appear because the logging level is set to WARNING.

Configure Handlers

Handlers determine where your log messages go. You can configure console handlers, file handlers, and rotating file handlers.

Console Handlers

To log messages to the console, use a StreamHandler.

handlers:
  console:
    class: logging.StreamHandler
    formatter: simpleFormatter
    stream: ext://sys.stdout

This configuration directs log messages to standard output.

File Handlers

To log messages to a file, use a FileHandler.

handlers:
  fileHandler:
    class: logging.FileHandler
    formatter: simpleFormatter
    filename: app.log

In your logging configuration:

root:
  level: INFO
  handlers: [fileHandler]

Now, all log messages at INFO level and above will be written to app.log.

Rotating File Handlers

For log rotation, use RotatingFileHandler.

version: 1
formatters:
  detailedFormatter:
    format: '%(levelname)s %(asctime)s %(module)s %(message)s'
handlers:
  rotatingFileHandler:
    class: logging.handlers.RotatingFileHandler
    formatter: detailedFormatter
    filename: app.log
    maxBytes: 1048576  # 1 MB
    backupCount: 5
loggers:
  root:
    level: WARNING
    handlers: [rotatingFileHandler]

This handler rotates the log file when it reaches 1 MB, keeping up to 5 backups.

In your script:

import logging.config
import yaml
with open('logging_config.yaml', 'r') as f:
  config = yaml.safe_load(f)
  logging.config.dictConfig(config)
logging.warning('This is a time-stamped warning')

Output(app.log):

WARNING 2024-10-13 17:37:51,473 mokhtar This is a time-stamped warning

The RotatingFileHandler will write log messages to app.log.

When the file size exceeds 1 MB, it will rotate the log file, rename it to app.log.1, and start a new app.log.

It will keep up to 5 backup files (app.log.1 to app.log.5).

Configure Formatters

Formatters control the layout of log messages including date and time formatting.

Format Strings

You can define format strings to include various log record attributes.

version: 1
formatters:
  detailedFormatter:
    format: '%(levelname)s %(asctime)s %(module)s %(process)d %(thread)d %(message)s'
handlers:
  console:
    class: logging.StreamHandler
    formatter: detailedFormatter
    stream: ext://sys.stdout
loggers:
  root:
    level: WARNING
    handlers: [console]

In your script:

import logging.config
import yaml
with open('logging_config.yaml', 'r') as f:
  config = yaml.safe_load(f)
  logging.config.dictConfig(config)
logging.warning('This is a time-stamped warning')

Output:

WARNING 2024-10-13 17:23:25 mokhtar 15992 5256 This is a time-stamped warning

This formatter includes the log level, timestamp, module name, process ID, thread ID, and the message.

Date and Time Formatting

You can specify the date format in your YAML file using datefmt key.

version: 1
formatters:
  timeFormatter:
    format: '%(asctime)s - %(message)s'
    datefmt: '%Y-%m-%d %H:%M:%S'
handlers:
  console:
    class: logging.StreamHandler
    formatter: timeFormatter
    stream: ext://sys.stdout
loggers:
  root:
    level: WARNING
    handlers: [console]

In your script:

import logging.config
import yaml
with open('logging_config.yaml', 'r') as f:
    config = yaml.safe_load(f)
    logging.config.dictConfig(config)
logging.warning('This is a time-stamped warning')

Output:

2024-10-13 17:21:35 - This is a time-stamped warning

The message includes the formatted timestamp as specified in datefmt.

Mokhtar Ebrahim

Mokhtar is the founder of LikeGeeks.com. He is a seasoned technologist and accomplished author, with expertise in Linux system administration and Python development. Since 2010, Mokhtar has built an impressive career, transitioning from system administration to Python development in 2015. His work spans large corporations to freelance clients around the globe. Alongside his technical work, Mokhtar has authored some insightful books in his field. Known for his innovative solutions, meticulous attention to detail, and high-quality work, Mokhtar continually seeks new challenges within the dynamic field of technology.