nginx 自 1.11.8 起支持保存为 json 格式。配置方法是:
log_format name [escape=default|json|none] string …
其中,escape=none 要求 1.13.10 起。
以下是一个示例配置:
log_format json escape=json '{"timestamp":"$time_iso8601",' '"server_addr":"$server_addr",' '"remote_addr":"$remote_addr",' '"host":"$host",' '"request_body":"$request_body",' '"request_uri":"$request_uri",' '"request_method":"$request_method",' '"request_length":$request_length,' '"request_time":$request_time,' '"request_id":"$request_id",' '"body_bytes_sent":$body_bytes_sent,' '"bytes_sent":$body_bytes_sent,' '"upstream_response_time":$upstream_response_time,' '"upstream_addr":"$upstream_addr",' '"upstream_status":$upstream_status,' '"status":$status,' '"http_referer":"$http_referer",' '"http_x_forwarded_for":"$http_x_forwarded_for",' '"http_user_agent":"$http_user_agent",' '"system_info":"$http_x_system_info"' '}';
这是最简单的方法。但有时并不能满足实际情况。比如 upstream_response_time,这里没有加引号是把它当数值型处理,当是静态请求不经过后端时,往往返回的是空或者-,这时就导致整个 json 非法了。再比如 system_info , 这是前端 app 获取系统信息的一个 json 串,无论加不加引号,经过 escape=json 转义后都不是有效的 json 对象了(想要获得最终结果为 json 嵌套)。如果 escape=none 不经过任何转义,http_x_system_info 不加引号是能获得合法的 json 嵌套对象,但当 http_x_system_info 不存在时,又坏了。这里可以简单的使用 map 指令对一些变量进行进行处理,但终究是不够灵活。而且 escape=none 也不太好。
这里我们可以使用 lua 来解决。它带有 logger 可以写到远程 syslog,不支持写到本地文件。我想简单点写本地文件得了,可以定义一个 log_format,通过 lua 新增变量的方式写 access_log。
安装 lua 可以直接使用 openresty,按官网文档安装好即可。在 nginx 主配置添加一个 jsonlog 的 log_format:
log_format log_format_jsonlog escape=none '$jsonlog';
记得添加 escape=none,我们在 lua 中进行 json_encode,这里就不需再转义了。编写 jsonlog.lua
local cjson = require('cjson') local log = {} log.datetime = ngx.var.time_iso8601 log.request_id = ngx.var.request_id log.request_method = ngx.var.request_method log.host = ngx.var.host log.request_uri = ngx.var.request_uri log.request_length = tonumber(ngx.var.request_length) log.request_body = ngx.var.request_body or "" log.upstream_addr = ngx.var.upstream_addr log.upstream_response_time = tonumber(ngx.var.upstream_response_time) log.upstream_status = tonumber(ngx.var.upstream_status) log.bytes_sent = tonumber(ngx.var.bytes_sent) log.status = tonumber(ngx.var.status) log.request_time = tonumber(ngx.var.request_time) log.remote_addr = ngx.var.remote_addr log.server_addr = ngx.var.server_addr log.http_referer = ngx.var.http_referer or "" log.http_x_forwarded_for = ngx.var.http_x_forwarded_for or "" log.http_user_agent = ngx.var.http_user_agent log.model = "" log.system = "" log.version = "" log.brand = "" log.platform = "" log.screenWidth = "" log.screenHeight = "" log.pixelRatio = "" log.SDKVersion = "" if ngx.var.http_x_system_info then local system_info = cjson.decode(ngx.var.http_x_system_info) if system_info then log.model = system_info.model log.system = system_info.system log.version = system_info.version log.brand = system_info.brand log.platform = system_info.platform log.screenWidth = system_info.screenWidth log.screenHeight = system_info.screenHeight log.pixelRatio = system_info.pixelRatio log.SDKVersion = system_info.SDKVersion end end local jsonlog = cjson.encode(log) ngx.var.jsonlog = jsonlog
这里都是直接取 nginx 全局变量,在 http_x_system_info 存在的情况下,再补充它里边的字段,最终的 log 是一个一维的 json 串。最后在网站配置中用 nginx 来写 access_log
set $jsonlog ''; log_by_lua_file /usr/local/openresty/nginx/conf/jsonlog.lua; access_log /var/log/nginx/huanbao/access.json log_format_jsonlog;
注意这里要先设置一下 jsonlog 变量,否则 lua 文件中为 jsonlog 变量赋值会报找不到变量。最终结果如下:
{ "host":"xxxxx.com", "request_method":"POST", "request_uri":"/api/comment", "http_referer":"https://servicewechat.com/wx1b57b830dd04c6ca/devtools/page-frame.html", "status":200, "screenHeight":812, "bytes_sent":2140, "brand":"devtools", "datetime":"2019-07-20T18:43:05+08:00", "SDKVersion":"2.7.0", "upstream_response_time":0.091, "pixelRatio":3, "request_time":0.091, "upstream_addr":"127.0.0.1:8999", "system":"iOS 10.0.1", "version":"6.6.3", "platform":"devtools", "request_id":"97ac9da809ab48ee832d5b2e8117e87d", "server_addr":"106.14.xxx.xxx", "remote_addr":"27.38.xxx.xxx", "request_body":"{"content":"哈哈","topic_key":"78f87155-277a-4deb-a339-ed60f2cccf88"}", "http_user_agent":"Mozilla/5.0 (iPhone; CPU iPhone OS 10_2 like Mac OS X) AppleWebKit/602.3.12 (KHTML, like Gecko) Mobile/14C92 Safari/601.1 wechatdevtools/1.02.1904090 MicroMessenger/6.7.3 Language/zh_CN webview/", "http_x_forwarded_for":"", "screenWidth":375, "upstream_status":200, "request_length":2048, "model":"iPhone X" }
得到这样的日志后,导入 ES 或者 第三方日志处理平台如阿里云的,处理起来比较方便了。