Logstash Filter 配置

笔者这里仅仅列出配置文件，在研究之后最红并没有采用在logstash的接下日志为json的做法。而是将json的输出放在了各个服务/应用中处理， spring boot的app可以参考：logstash-logback-encoder

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
input {
  beats {
    port => 5044
  }
}
filter {
  #If log line contains tab character followed by 'at' then we will tag that entry as stacktrace
  if [message] =~ "\tat" {
    grok {
      match => ["message", "^(\tat)"]
      add_tag => ["stacktrace"]
    }
  }

  #Grokking Spring Boot's default log format
  grok {
    match => [ 
				#	Record transaction
				"message","(?<timestamp>%{YEAR}-%{MONTHNUM}-%{MONTHDAY} %{TIME})  %{LOGLEVEL:level} %{NUMBER:pid} --- \[\s*(?<thread>[^\]]+)\] (?<class>[A-Za-z0-9.#_]+)\s*: \[\s*(?<transactionInfo>[^\]]+)\]",
				"message", "(?<timestamp>%{YEAR}-%{MONTHNUM}-%{MONTHDAY} %{TIME})  %{LOGLEVEL:level} %{NUMBER:pid} --- \[\s*(?<thread>[^\]]+)\] (?<class>[A-Za-z0-9.#_]+)\s*:\s+(?<logmessage>.*)",
				"message", "(?<timestamp>%{YEAR}-%{MONTHNUM}-%{MONTHDAY} %{TIME})  %{LOGLEVEL:level} %{NUMBER:pid} --- .+? :\s+(?<logmessage>.*)"
             ]
  }

  #Parsing out timestamps which are in timestamp field thanks to previous grok section
  date {
    match => [ "timestamp" , "yyyy-MM-dd HH:mm:ss.SSS" ]
  }
}
output {
 elasticsearch{} 
 stdout{
   codec => rubydebug
  }
}

这里grok配置了三册过滤， 第一层用作统计，message的格式如下：

1
2016-07-15 20:30:30.884  INFO 14624 --- [nio-8081-exec-3] c.l.a.w.controller.OfbizProxyController  : [{"transactionCode":"ofbizProxy","transactionDuration":246}]

使用Grok Debugger 解析后如下

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
{
  "timestamp": [
    [
      "2016-07-15 20:30:30.884"
    ]
  ],
  "YEAR": [
    [
      "2016"
    ]
  ],
  "MONTHNUM": [
    [
      "07"
    ]
  ],
  "MONTHDAY": [
    [
      "15"
    ]
  ],
  "TIME": [
    [
      "20:30:30.884"
    ]
  ],
  "HOUR": [
    [
      "20"
    ]
  ],
  "MINUTE": [
    [
      "30"
    ]
  ],
  "SECOND": [
    [
      "30.884"
    ]
  ],
  "level": [
    [
      "INFO"
    ]
  ],
  "pid": [
    [
      "14624"
    ]
  ],
  "BASE10NUM": [
    [
      "14624"
    ]
  ],
  "thread": [
    [
      "nio-8081-exec-3"
    ]
  ],
  "class": [
    [
      "c.l.a.w.controller.OfbizProxyController"
    ]
  ],
  "transactionInfo": [
    [
      "{"transactionCode":"ofbizProxy","transactionDuration":246}"
    ]
  ]
}

第二层针对普通的log

1
2016-07-15 20:30:07.768  INFO 14624 --- [nio-8081-exec-1] c.l.a.web.controller.LoginController     : Login username:vincent.chen@okchem.com IP is:0:0:0:0:0:0:0:1

解析后的json如下：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
{
  "timestamp": [
    [
      "2016-07-15 20:30:07.768"
    ]
  ],
  "YEAR": [
    [
      "2016"
    ]
  ],
  "MONTHNUM": [
    [
      "07"
    ]
  ],
  "MONTHDAY": [
    [
      "15"
    ]
  ],
  "TIME": [
    [
      "20:30:07.768"
    ]
  ],
  "HOUR": [
    [
      "20"
    ]
  ],
  "MINUTE": [
    [
      "30"
    ]
  ],
  "SECOND": [
    [
      "07.768"
    ]
  ],
  "level": [
    [
      "INFO"
    ]
  ],
  "pid": [
    [
      "14624"
    ]
  ],
  "BASE10NUM": [
    [
      "14624"
    ]
  ],
  "thread": [
    [
      "nio-8081-exec-1"
    ]
  ],
  "class": [
    [
      "c.l.a.web.controller.LoginController"
    ]
  ],
  "logmessage": [
    [
      "Login username:vincent.chen@okchem.com IP is:0:0:0:0:0:0:0:1"
    ]
  ]
}

第三层针对遗漏的无法匹配到的log再次解析， 这里暂时没有示例

 上一篇