The Missing How-to for Celery Logging

You have just decided to use Celery in your Django project and configured it already. All tasks are now created and are running correctly. So you deploy it on a server. The only issue is that logs will refuse to appear in your application server logs and you are losing any log capture for your celery workers.

Worry not! here is a production-tested config for you so that your celery logs appear in your server logs in all their glory.

If not already follow my previous guide on how to do Django logging properly. Once your Django logging config is sorted out, now you can work on the Celery config.

Generally for configuring Celery you will have created a file in your main Django app in which your settings.py file is present. Add a config_loggers function in that file so that it is now able to use your configured settings for logging instead of its own.

import os

from celery import Celery
from celery.signals import setup_logging  # noqa

# set the default Django settings module for the 'celery' program.
os.environ.setdefault("DJANGO_SETTINGS_MODULE", "src.settings")

app = Celery("src")

# Using a string here means the worker doesn't have to serialize
# the configuration object to child processes.
# - namespace='CELERY' means all celery-related configuration keys
#   should have a `CELERY_` prefix.
app.config_from_object("django.conf:settings", namespace="CELERY")


@setup_logging.connect
def config_loggers(*args, **kwargs):
    from logging.config import dictConfig  # noqa
    from django.conf import settings  # noqa

    dictConfig(settings.LOGGING)


# Load task modules from all registered Django app configs.
app.autodiscover_tasks()

If you have already setup logging as in my previous guide it will there are no further steps required and now when you run your celery workers, you will notice they are spitting out logs with the same configuration and formatting as your Django logs.