2020-8-19 23:17:0 Author: taosecurity.blogspot.com(查看原文) 阅读量:3 收藏
Are you a network security monitoring dinosaur like me? Do you prefer to inspect your Zeek logs using the command line instead of a Web-based SIEM?
I store my Zeek logs in JSON format. Sometimes I like to view the output using jq.
If I need to search directories of logs for a string, like a UID, I might* use something like zgrep with the following syntax:
$ zgrep "CLkXf2CMo11hD8FQ5" 2020-08-16/*
2020-08-16/conn_20200816_06:00:00-07:00:00+0000.log.gz:{"_path":"conn","_system_name":"ds61","_write_ts":"2020-08-16T06:26:10.266225Z","_node":"worker-01","ts":"2020-08-16T06:26:01.485394Z","uid":"CLkXf2CMo11hD8FQ5","id.orig_h":"192.168.2.76","id.orig_p":53380,"id.resp_h":"196.216.2.24","id.resp_p":21,"proto":"tcp","service":"ftp","duration":3.780829906463623,"orig_bytes":184,"resp_bytes":451,"conn_state":"SF","local_orig":true,"local_resp":false,"missed_bytes":0,"history":"ShAdDafF","orig_pkts":20,"orig_ip_bytes":1232,"resp_pkts":17,"resp_ip_bytes":1343,"community_id":"1:lEESxqaSVYqFZvWNb4OccTa9sTs="}
2020-08-16/ftp_20200816_06:26:04-07:00:00+0000.log.gz:{"_path":"ftp","_system_name":"ds61","_write_ts":"2020-08-16T06:26:04.077276Z","_node":"worker-01","ts":"2020-08-16T06:26:03.553287Z","uid":"CLkXf2CMo11hD8FQ5","id.orig_h":"192.168.2.76","id.orig_p":53380,"id.resp_h":"196.216.2.24","id.resp_p":21,"user":"anonymous","password":"[email protected]","command":"EPSV","reply_code":229,"reply_msg":"Entering Extended Passive Mode (|||31746|).","data_channel.passive":true,"data_channel.orig_h":"192.168.2.76","data_channel.resp_h":"196.216.2.24","data_channel.resp_p":31746}
2020-08-16/ftp_20200816_06:26:04-07:00:00+0000.log.gz:{"_path":"ftp","_system_name":"ds61","_write_ts":"2020-08-16T06:26:05.117287Z","_node":"worker-01","ts":"2020-08-16T06:26:04.597290Z","uid":"CLkXf2CMo11hD8FQ5","id.orig_h":"192.168.2.76","id.orig_p":53380,"id.resp_h":"196.216.2.24","id.resp_p":21,"user":"anonymous","password":"[email protected]","command":"RETR","arg":"ftp://196.216.2.24/pub/stats/afrinic/delegated-afrinic-extended-latest.md5","file_size":74,"reply_code":226,"reply_msg":"Transfer complete.","fuid":"FueF95uKPrUuDnMc4"}
That is tough on the eyes. I cannot simply pipe that output to Jq however:
$ zgrep "CLkXf2CMo11hD8FQ5" 2020-08-16/* | jq .
parse error: Invalid numeric literal at line 1, column 28
What I need to do is strip out the filename and colon before the JSON. I learned how to use sed to do this thanks to this post.
$ zgrep "CLkXf2CMo11hD8FQ5" 2020-08-16/* | sed 's/.*gz://' | jq .
{
"_path": "conn",
"_system_name": "ds61",
"_write_ts": "2020-08-16T06:26:10.266225Z",
"_node": "worker-01",
"ts": "2020-08-16T06:26:01.485394Z",
"uid": "CLkXf2CMo11hD8FQ5",
"id.orig_h": "192.168.2.76",
"id.orig_p": 53380,
"id.resp_h": "196.216.2.24",
"id.resp_p": 21,
"proto": "tcp",
"service": "ftp",
"duration": 3.780829906463623,
"orig_bytes": 184,
"resp_bytes": 451,
"conn_state": "SF",
"local_orig": true,
"local_resp": false,
"missed_bytes": 0,
"history": "ShAdDafF",
"orig_pkts": 20,
"orig_ip_bytes": 1232,
"resp_pkts": 17,
"resp_ip_bytes": 1343,
"community_id": "1:lEESxqaSVYqFZvWNb4OccTa9sTs="
}
{
"_path": "ftp",
"_system_name": "ds61",
"_write_ts": "2020-08-16T06:26:04.077276Z",
"_node": "worker-01",
"ts": "2020-08-16T06:26:03.553287Z",
"uid": "CLkXf2CMo11hD8FQ5",
"id.orig_h": "192.168.2.76",
"id.orig_p": 53380,
"id.resp_h": "196.216.2.24",
"id.resp_p": 21,
"user": "anonymous",
"command": "EPSV",
"reply_code": 229,
"reply_msg": "Entering Extended Passive Mode (|||31746|).",
"data_channel.passive": true,
"data_channel.orig_h": "192.168.2.76",
"data_channel.resp_h": "196.216.2.24",
"data_channel.resp_p": 31746
}
{
"_path": "ftp",
"_system_name": "ds61",
"_write_ts": "2020-08-16T06:26:05.117287Z",
"_node": "worker-01",
"ts": "2020-08-16T06:26:04.597290Z",
"uid": "CLkXf2CMo11hD8FQ5",
"id.orig_h": "192.168.2.76",
"id.orig_p": 53380,
"id.resp_h": "196.216.2.24",
"id.resp_p": 21,
"user": "anonymous",
"command": "RETR",
"arg": "ftp://196.216.2.24/pub/stats/afrinic/delegated-afrinic-extended-latest.md5",
"file_size": 74,
"reply_code": 226,
"reply_msg": "Transfer complete.",
"fuid": "FueF95uKPrUuDnMc4"
}
Maybe this will help you too.
*I use the find command in other circumstances.
Update: Twitter user @captainGeech42 noted that I could use grep -h and omit the sed pipe, e.g.:
如有侵权请联系:admin#unsafe.sh