Maintenance

Current Technical Maintainer: Udit Agrawal

Maintainer: Bob T.

The maintainers support the ongoing engineering development, quality, compatibility, and sustainability of json-logging.

See MAINTAINERS.md for maintainer details, MAINTENANCE.md for project procedures, and AGENTS.md for coding-agent guidance.

json-logging

Python logging library to emit JSON log that can be easily indexed and searchable by logging infrastructure such as ELK , EFK , AWS Cloudwatch, GCP Stackdriver, etc.

If you're using Cloud Foundry, it might worth to check out the library SAP/cf-python-logging-support which I'm also original author.

Content

1. Features
2. Usage

1. Configuration
2. Python References
3. Framework support plugin development
4. FAQ & Troubleshooting
5. References

1. Features

1. Emit JSON logs (format detail)
2. Lightweight, no dependencies, minimal configuration needed (1 LoC to get it working)
3. Seamlessly integrate with Python native logging module. Support both Python 2.7.x and 3.x
4. Auto extract correlation-id for distributed tracing [\[1\]](#1-what-is-correlation-idrequest-id)
5. Support HTTP request instrumentation. Built in support for FastAPI

1. Highly customizable: support inject arbitrary extra properties to JSON log message, override logging formatter, etc.
2. Production ready, has been used in production since 2017 with more than 8 millions downloads

2. Usage

Install by running this command:

pip install json-logging

By default log will be emitted in normal format to ease the local development. To enable it on production set enable_json in json_logging.init_< framework_name >() method call. Once configured library will try to configure all loggers (existing and newly created) to emit log in JSON format. See following use cases for more detail.

2.1 Non-web application log

This mode don't support correlation-id.

import json_logging, logging, sys
log is initialized without a web framework name
json_logging.init_non_web(enable_json=True)
logger = logging.getLogger("test-logger")
logger.setLevel(logging.DEBUG)
logger.addHandler(logging.StreamHandler(sys.stdout))
logger.info("test logging statement")

2.2 Web application log

FastAPI

import datetime, logging, sys, json_logging, fastapi, uvicorn
app = fastapi.FastAPI()
json_logging.init_fastapi(enable_json=True)
json_logging.init_request_instrument(app)
init the logger as usual
logger = logging.getLogger("test-logger")
logger.setLevel(logging.DEBUG)
logger.addHandler(logging.StreamHandler(sys.stdout))
@app.get('/')
def home():
    logger.info("test log statement")
    logger.info("test log statement with extra props", extra={'props': {"extra_property": 'extra_value'}})
    correlation_id = json_logging.get_correlation_id()
    return "Hello world : " + str(datetime.datetime.now())
if __name__ == "__main__":
    uvicorn.run(app, host='0.0.0.0', port=5000)

Flask

import datetime, logging, sys, json_logging, flask
app = flask.Flask(__name__)
json_logging.init_flask(enable_json=True)
json_logging.init_request_instrument(app)
init the logger as usual
logger = logging.getLogger("test-logger")
logger.setLevel(logging.DEBUG)
logger.addHandler(logging.StreamHandler(sys.stdout))
@app.route('/')
def home():
    logger.info("test log statement")
    logger.info("test log statement with extra props", extra={'props': {"extra_property": 'extra_value'}})
    correlation_id = json_logging.get_correlation_id()
    return "Hello world : " + str(datetime.datetime.now())
if __name__ == "__main__":
    app.run(host='0.0.0.0', port=int(5000), use_reloader=False)

Sanic

import logging, sys, json_logging, sanic
app = sanic.Sanic(name="sanic-web-app")
json_logging.init_sanic(enable_json=True)
json_logging.init_request_instrument(app)
init the logger as usual
logger = logging.getLogger("sanic-integration-test-app")
logger.setLevel(logging.DEBUG)
logger.addHandler(logging.StreamHandler(sys.stdout))
@app.route("/")
async def home(request):
    logger.info("test log statement")
    logger.info("test log statement with extra props", extra={'props': {"extra_property": 'extra_value'}})
    # this will be faster
    correlation_id = json_logging.get_correlation_id(request=request)
    # this will be slower, but will work in context you cant get a reference of request object
    correlation_id_without_request_obj = json_logging.get_correlation_id()
return sanic.response.text("hello world")
if __name__ == "__main__":
    app.run(host="0.0.0.0", port=5000)

Quart

import asyncio, logging, sys, json_logging, quart
app = quart.Quart(__name__)
json_logging.init_quart(enable_json=True)
json_logging.init_request_instrument(app)
init the logger as usual
logger = logging.getLogger("test logger")
logger.setLevel(logging.DEBUG)
logger.addHandler(logging.StreamHandler(sys.stdout))
@app.route('/')
async def home():
    logger.info("test log statement")
    logger.info("test log statement with extra props", extra={'props': {"extra_property": 'extra_value'}})
    correlation_id = json_logging.get_correlation_id()
    return "Hello world"
if __name__ == "__main__":
    loop = asyncio.get_event_loop()
    app.run(host='0.0.0.0', port=int(5000), use_reloader=False, loop=loop)

Connexion

import logging, sys, json_logging, connexion
app = connexion.FlaskApp(__name__)
json_logging.init_connexion(enable_json=True)
json_logging.init_request_instrument(app)
app.add_api('api.yaml')
init the logger as usual
logger = logging.getLogger("test-logger")
logger.setLevel(logging.DEBUG)
logger.addHandler(logging.StreamHandler(sys.stdout))
if __name__ == "__main__":
    app.run()

Custom handler for request instrumentation

you need to explicitly set JSONRequestLogFormatter as default formatter for any extra handler that is added to request_logger

import json_logging.formatters
request_logger = json_logging.get_request_logger()
handler = logging.handlers.RotatingFileHandler(filename='log_req.log', maxBytes=5000000, backupCount=10)
handler.setFormatter(json_logging.formatters.JSONRequestLogFormatter())
request_logger.addHandler(handler)

2.3 Get current correlation-id

Current request correlation-id can be retrieved and pass to downstream services call as follow:

# this will be faster
correlation_id = json_logging.get_correlation_id(request=request)
this will be slower, but will work in context where you couldn't get a reference of request object
correlation_id_without_request_obj = json_logging.get_correlation_id()
use correlation id for upstream service calls here

In request context, if one is not present, a new one might be generated depends on CREATE_CORRELATION_ID_IF_NOT_EXISTS setting value.

Before initialization or after init_non_web(), json_logging.get_correlation_id() returns json_logging.EMPTY_VALUE ("-") instead of raising an error or generating an HTTP correlation ID.

2.4 Log extra properties

Extra property can be added to logging statement as follows:

logger.info("test log statement", extra={'props': {'extra_property': 'extra_value'}, 'tags': ['app:name']})

_For backwards compatibility, properties added in props are extracted to the root and take precedence over those set directly in the root of extra. This is relevant only for logs formatted by JSONLogFormatter, other formatters never printed the props details._

Forced overriding correlation-id

logger.info("test log statement", extra={'props': {'correlation_id': 'custom_correlation_id'}})

2.5 Root logger

If you want to use root logger as main logger to emit log. Make sure you call config_root_logger() after initialize root logger (by logging.basicConfig() or logging.getLogger()) [\[2\]](#2-python-logging-propagate)

logging.basicConfig()
json_logging.init_ < framework_name > ()
json_logging.config_root_logger()

2.6 Custom log formatter

Customer JSON log formatter can be passed to init method. see examples for more detail: non web, web

2.7 Exclude certain URl from request instrumentation

Certain URL can be excluded from request instrumentation by specifying a list of regex into init_request_instrument method like below:

json_logging.init_request_instrument(app, exclude_url_patterns=[r'/exclude_from_request_instrumentation'])

2.8 Exclude fields from the JSON output

Set json_logging.EXCLUDED_FIELDS to omit fields from every JSON record, including request logs:

import json_logging
json_logging.EXCLUDED_FIELDS = ["written_ts"]          # or a tuple/set/frozenset of field names
json_logging.init_flask(enable_json=True)

• The default is (), so output is unchanged.
• Set it before calling init_(): formatters read it when they are constructed, the same way as COMPONENT_.
• A bare string, or a collection containing non-strings, raises TypeError from init_*(),

• Fields are removed after they are collected, so extra and props entries are removed too.
• Exclusions apply to formatters that use BaseJSONFormatter.format(): the built-in JSON formatters and subclasses

• Excluding a field changes the JSON schema. Downstream log consumers may depend on fields such as written_ts,

• This is an output-formatting option, not a security redaction or data-loss-prevention control. A sensitive

3. Configuration

logging library can be configured by setting the value in json_logging, all configuration must be placed before json_logging.init method call

Name | Description | Default value --- | --- | --- ENABLE_JSON_LOGGING | ** DEPRECATED Whether to enable JSON logging mode.Can be set as an environment variable, enable when set to to either one in following list (case-insensitive) ['true', '1', 'y', 'yes']** , this have no effect on request logger | false CORRELATION_ID_FIELD | Name of field in generated log messages containing the correlation-id value | 'correlation_id' CORRELATION_ID_HEADERS | List of HTTP headers that will be used to look for correlation-id value. HTTP headers will be searched one by one according to list order| ['X-Correlation-ID','X-Request-ID'] EMPTY_VALUE | Default value when a logging record property is None | '-' CORRELATION_ID_GENERATOR | function to generate unique correlation-id | uuid.uuid1 JSON_SERIALIZER | function to encode object to JSON | json.dumps COMPONENT_ID | Uniquely identifies the software component that has processed the current request | EMPTY_VALUE COMPONENT_NAME | A human-friendly name representing the software component | EMPTY_VALUE COMPONENT_INSTANCE_INDEX | Instance's index of horizontally scaled service | 0 CREATE_CORRELATION_ID_IF_NOT_EXISTS | Whether to generate a new correlation-id in case one is not present| True

4. Python References

TODO: update Python API docs on Github page

5. Framework support plugin development

To add support for a new web framework, you need to extend following classes in ** framework_base and register support using json_logging.register_framework_support** method:

Class | Description | Mandatory --- | --- | --- BaseRequestInfoExtractor | Helper class help to extract logging-relevant information from HTTP request object | yes BaseResponseInfoExtractor | Helper class help to extract logging-relevant information from HTTP response object | yes BaseFrameworkConfigurator | Class to perform logging configuration for given framework as needed | no BaseAppRequestInstrumentationConfigurator | Class to perform request instrumentation logging configuration | no

Take a look at json_logging/base_framework.py, ** json_logging.flask and json_logging.sanic** packages for reference implementations.

6. FAQ & Troubleshooting

1. I configured everything, but no logs are printed out?

1. Same log statement is printed out multiple times.

1. RuntimeError: Can not call init more than once with FastAPI.

7. References

[0] Full logging format references

2 types of logging statement will be emitted by this library:

• Application log: normal logging statement e.g.:

{
	"type": "log",
	"written_at": "2017-12-23T16:55:37.280Z",
	"written_ts": 1514048137280721000,
	"component_id": "1d930c0xd-19-s3213",
	"component_name": "ny-component_name",
	"component_instance_idx": 0,
	"logger": "test logger",
	"thread": "MainThread",
	"level": "INFO",
	"line_no": 22,
	"filename": "/path/to/foo.py"
	"exc_info": "Traceback (most recent call last): \n  File "<stdin>", line 1, in <module>\n ValueError: There is something wrong with your input",
	"correlation_id": "1975a02e-e802-11e7-8971-28b2bd90b19a",
	"extra_property": "extra_value",
	"msg": "This is a message"
}

• Request log: request instrumentation logging statement which recorded request information such as response time,

{
	"type": "request",
	"written_at": "2017-12-23T16:55:37.280Z",
	"written_ts": 1514048137280721000,
	"component_id": "-",
	"component_name": "-",
	"component_instance_idx": 0,
	"correlation_id": "1975a02e-e802-11e7-8971-28b2bd90b19a",
	"remote_user": "user_a",
	"request": "/index.html",
	"referer": "-",
	"x_forwarded_for": "-",
	"protocol": "HTTP/1.1",
	"method": "GET",
	"remote_ip": "127.0.0.1",
	"request_size_b": 1234,
	"remote_host": "127.0.0.1",
	"remote_port": 50160,
	"request_received_at": "2017-12-23T16:55:37.280Z",
	"response_time_ms": 0,
	"response_status": 200,
	"response_size_b": "122",
	"response_content_type": "text/html; charset=utf-8",
	"response_sent_at": "2017-12-23T16:55:37.280Z"
}

See following tables for detail format explanation:

• Common field

• application logs