mirror of
https://github.com/wagga40/Zircolite.git
synced 2025-12-05 18:56:41 -06:00
Add field transforms
Enhance db insertion producing a 10% speedup Refactor some code Update docs
This commit is contained in:
wagga40
@@ -4,7 +4,7 @@
|
||||
#
|
||||
ARG PYTHON_VERSION="3.11-slim"
|
||||
|
||||
FROM "python:${PYTHON_VERSION}" as stage
|
||||
FROM "python:${PYTHON_VERSION}" AS stage
|
||||
|
||||
ARG ZIRCOLITE_INSTALL_PREFIX="/opt"
|
||||
ARG ZIRCOLITE_REPOSITORY_URI="https://github.com/wagga40/Zircolite.git"
|
||||
@@ -30,7 +30,7 @@ RUN chmod 0755 \
|
||||
FROM "python:${PYTHON_VERSION}"
|
||||
|
||||
LABEL author="wagga40"
|
||||
LABEL description="A standalone SIGMA-based detection tool for EVTX."
|
||||
LABEL description="A standalone SIGMA-based detection tool for EVTX, Auditd and Sysmon for Linux logs."
|
||||
LABEL maintainer="wagga40"
|
||||
|
||||
ARG ZIRCOLITE_INSTALL_PREFIX="/opt"
|
||||
@@ -46,6 +46,8 @@ WORKDIR "${ZIRCOLITE_INSTALL_PREFIX}/zircolite"
|
||||
RUN python3 -m pip install \
|
||||
--requirement requirements.full.txt
|
||||
|
||||
RUN python3 zircolite.py -U
|
||||
|
||||
ENTRYPOINT [ "python3", "zircolite.py" ]
|
||||
|
||||
CMD [ "--help" ]
|
||||
|
||||
99
README.md
99
README.md
@@ -1,24 +1,33 @@
|
||||
# <p align="center"></p>
|
||||
|
||||
## Standalone SIGMA-based detection tool for EVTX, Auditd, Sysmon for linux, XML or JSONL/NDJSON Logs
|
||||
## Standalone SIGMA-based detection tool for EVTX, Auditd, Sysmon for linux, XML or JSONL/NDJSON Logs
|
||||
|
||||
|
||||
[](https://www.python.org/)
|
||||
[](https://www.python.org/)
|
||||
|
||||
|
||||
> **Zircolite is a standalone tool written in Python 3. It allows to use SIGMA rules on : MS Windows EVTX (EVTX, XML and JSONL format), Auditd logs, Sysmon for Linux, EVTXtract, CSV and XML logs**
|
||||
**Zircolite** is a standalone tool written in Python 3 that allows you to use SIGMA rules on:
|
||||
|
||||
- **Zircolite** is relatively fast and can parse large datasets in just seconds
|
||||
- **Zircolite** is based on a Sigma backend (SQLite) and do not use internal sigma to "something" conversion
|
||||
- **Zircolite** can export results to multiple format with using Jinja [templates](templates) : JSON, CSV, JSONL, Splunk, Elastic, Zinc, Timesketch...
|
||||
- MS Windows EVTX (EVTX, XML, and JSONL formats)
|
||||
- Auditd logs
|
||||
- Sysmon for Linux
|
||||
- EVTXtract
|
||||
- CSV and XML logs
|
||||
|
||||
**Zircolite can be used directly in Python or you can use the binaries provided in [releases](https://github.com/wagga40/Zircolite/releases).**
|
||||
### Key Features
|
||||
|
||||
**Documentation is [here](https://wagga40.github.io/Zircolite/) (dedicated site) or [here](docs) (repo directory).**
|
||||
- **Fast Processing**: Zircolite is relatively fast and can parse large datasets in just seconds.
|
||||
- **SIGMA Backend**: It is based on a SIGMA backend (SQLite) and does not use internal SIGMA-to-something conversion.
|
||||
- **Advanced Log Manipulation**: It can manipulate input logs by splitting fields and applying transformations, allowing for more flexible and powerful log analysis.
|
||||
- **Flexible Export**: Zircolite can export results to multiple formats using Jinja [templates](templates), including JSON, CSV, JSONL, Splunk, Elastic, Zinc, Timesketch, and more.
|
||||
|
||||
**You can use Zircolite directly in Python or use the binaries provided in the [releases](https://github.com/wagga40/Zircolite/releases).**
|
||||
|
||||
**Documentation is available [here](https://wagga40.github.io/Zircolite/) (dedicated site) or [here](docs) (repo directory).**
|
||||
|
||||
## Requirements / Installation
|
||||
|
||||
Python 3.8 minimum is required. If you only want to use base functionalities of Zircolite, you can install dependencies with : `pip3 install -r requirements.txt`. But `pip3 install -r requirements.full.txt` is strongly recommended.
|
||||
The project has only beek tested with Python 3.10. If you only want to use base functionnalities of Zircolite, you can install dependencies with : `pip3 install -r requirements.txt`. But `pip3 install -r requirements.full.txt` is strongly recommended.
|
||||
|
||||
The use of [evtx_dump](https://github.com/omerbenamram/evtx) is **optional but required by default (because it is -for now- much faster)**, If you do not want to use it you have to use the `--noexternal` option. The tool is provided if you clone the Zircolite repository (the official repository is [here](https://github.com/omerbenamram/evtx)).
|
||||
|
||||
@@ -28,28 +37,41 @@ The use of [evtx_dump](https://github.com/omerbenamram/evtx) is **optional but r
|
||||
|
||||
Check tutorials made by other (EN, SP and FR) [here](#tutorials).
|
||||
|
||||
### EVTX files
|
||||
### EVTX files :
|
||||
|
||||
Help is available with `zircolite.py -h`. If your EVTX files have the extension ".evtx" :
|
||||
Help is available with:
|
||||
|
||||
```shell
|
||||
python3 zircolite.py -h
|
||||
```
|
||||
|
||||
If your EVTX files have the extension ".evtx" :
|
||||
|
||||
```shell
|
||||
# python3 zircolite.py --evtx <EVTX FOLDER or EVTX FILE> --ruleset <SIGMA RULESET> [--ruleset <OTHER RULESET>]
|
||||
python3 zircolite.py --evtx sysmon.evtx --ruleset rules/rules_windows_sysmon_pysigma.json
|
||||
```
|
||||
|
||||
The SYSMON ruleset employed is a default one, intended for analyzing logs from endpoints with SYSMON installed.
|
||||
- The `--evtx` argument can be a file or a folder. If it is a folder, all EVTX files in the current folder and subfolders will be selected.
|
||||
- The SYSMON ruleset used is a default one, intended for analyzing logs from endpoints with SYSMON installed.
|
||||
|
||||
### Auditd / Sysmon for Linux / JSONL or NDJSON logs
|
||||
### Auditd / Sysmon for Linux / JSONL or NDJSON logs :
|
||||
|
||||
```shell
|
||||
# For Auditd logs
|
||||
python3 zircolite.py --events auditd.log --ruleset rules/rules_linux.json --auditd
|
||||
# For Sysmon for Linux logs
|
||||
python3 zircolite.py --events sysmon.log --ruleset rules/rules_linux.json --sysmon4linux
|
||||
python3 zircolite.py --events <JSON_FOLDER or JSON_FILE> --ruleset rules/rules_windows_sysmon_pysigma.json --jsononly
|
||||
# For JSONL or NDJSON logs
|
||||
python3 zircolite.py --events <JSON_FOLDER_OR_FILE> --ruleset rules/rules_windows_sysmon_pysigma.json --jsononly
|
||||
```
|
||||
|
||||
:information_source: If you want to try the tool you can test with [EVTX-ATTACK-SAMPLES](https://github.com/sbousseaden/EVTX-ATTACK-SAMPLES) (EVTX Files).
|
||||
- The `--events` argument can be a file or a folder. If it is a folder, all event files in the current folder and subfolders will be selected.
|
||||
|
||||
### Run with Docker
|
||||
> [!TIP]
|
||||
> If you want to try the tool you can test with [EVTX-ATTACK-SAMPLES](https://github.com/sbousseaden/EVTX-ATTACK-SAMPLES) (EVTX Files).
|
||||
|
||||
### Running with Docker
|
||||
|
||||
```bash
|
||||
# Pull docker image
|
||||
@@ -64,7 +86,7 @@ docker run --rm --tty \
|
||||
-r /case/input/a_sigma_rule.yml
|
||||
```
|
||||
|
||||
You can replace `$PWD` with the directory (absolute path only) where your logs and rules/rulesets are stored.
|
||||
- Replace `$PWD` with the directory (absolute path only) where your logs and rules/rulesets are stored.
|
||||
|
||||
### Updating default rulesets
|
||||
|
||||
@@ -72,17 +94,18 @@ You can replace `$PWD` with the directory (absolute path only) where your logs a
|
||||
python3 zircolite.py -U
|
||||
```
|
||||
|
||||
:information_source: Please note these rulesets are provided to use Zircolite out-of-the-box but [you should generate your own rulesets](#why-you-should-build-your-own-rulesets) but they can be very noisy or slow. These auto-updated rulesets are available on the dedicated repository : [Zircolite-Rules](https://github.com/wagga40/Zircolite-Rules).
|
||||
> [!IMPORTANT]
|
||||
> Please note these rulesets are provided to use Zircolite out-of-the-box, but [you should generate your own rulesets](#why-you-should-build-your-own-rulesets) as they can be very noisy or slow. These auto-updated rulesets are available in the dedicated repository: [Zircolite-Rules](https://github.com/wagga40/Zircolite-Rules).
|
||||
|
||||
## Docs
|
||||
|
||||
Everything is [here](docs).
|
||||
Complete documentation is available [here](docs).
|
||||
|
||||
## Mini-Gui
|
||||
|
||||
The Mini-GUI can be used totally offline, it allows the user to display and search results. You can automatically generate a Mini-Gui "package" with the `--package` option. To know how to use the Mini-GUI, check docs [here](docs/Advanced.md#mini-gui).
|
||||
The Mini-GUI can be used totally offline. It allows you to display and search results. You can automatically generate a Mini-GUI "package" with the `--package` option. To learn how to use the Mini-GUI, check the docs [here](docs/Advanced.md#mini-gui).
|
||||
|
||||
### Detected events by Mitre Att&ck (c) techniques and criticality levels
|
||||
### Detected events by Mitre Att&ck (c) techniques and criticity levels
|
||||
|
||||
|
||||
|
||||
@@ -90,7 +113,7 @@ The Mini-GUI can be used totally offline, it allows the user to display and sear
|
||||
|
||||
|
||||
|
||||
### Detected events by Mitre Att&ck (c) techniques displayed on the Matrix
|
||||
### Detected events by Mitre Att&ck (c) techniques displayed on the Matrix
|
||||
|
||||
|
||||
|
||||
@@ -98,20 +121,32 @@ The Mini-GUI can be used totally offline, it allows the user to display and sear
|
||||
|
||||
### Tutorials
|
||||
|
||||
- (EN) [Russ McRee](https://holisticinfosec.io) has published a pretty good [tutorial](https://holisticinfosec.io/post/2021-09-28-zircolite/) on SIGMA and **Zircolite** in his [blog](https://holisticinfosec.io/post/2021-09-28-zircolite/)
|
||||
- **English**: [Russ McRee](https://holisticinfosec.io) has published a detailed [tutorial](https://holisticinfosec.io/post/2021-09-28-zircolite/) on SIGMA and Zircolite on his blog.
|
||||
|
||||
- (SP) **César Marín** has published a tutorial in **spanish** [here](https://derechodelared.com/zircolite-ejecucion-de-reglas-sigma-en-ficheros-evtx/)
|
||||
- **Spanish**: **César Marín** has published a tutorial in Spanish [here](https://derechodelared.com/zircolite-ejecucion-de-reglas-sigma-en-ficheros-evtx/).
|
||||
|
||||
- (FR) [IT-connect.fr](https://www.it-connect.fr/) has published [a very extensive tutorial](https://www.it-connect.fr/) in **French** on Zircolite
|
||||
- **French**: [IT-connect.fr](https://www.it-connect.fr/) has published [an extensive tutorial](https://www.it-connect.fr/) on Zircolite in French.
|
||||
|
||||
### References
|
||||
- **French**: [IT-connect.fr](https://www.it-connect.fr/) has also published a [Hack the Box challenge Write-Up](https://www.it-connect.fr/hack-the-box-sherlocks-tracer-solution/) using Zircolite.
|
||||
|
||||
- [Florian Roth](https://github.com/Neo23x0/) cited **Zircolite** in his [**SIGMA Hall of fame**](https://github.com/Neo23x0/Talks/blob/master/Sigma_Hall_of_Fame_20211022.pdf) in its talk during the October 2021 EU ATT&CK Workshop in October 2021
|
||||
- Zircolite has been cited and used in the research work of the CIDRE team : [PWNJUSTSU - Website](https://pwnjutsu.irisa.fr) and [PWNJUSTSU - Academic paper](https://hal.inria.fr/hal-03694719/document)
|
||||
- Zircolite has been cited and presented during [JSAC 2023](https://jsac.jpcert.or.jp/archive/2023/pdf/JSAC2023_workshop_sigma_jp.pdf)
|
||||
### References
|
||||
|
||||
- [Florian Roth](https://github.com/Neo23x0/) cited Zircolite in his [**SIGMA Hall of Fame**](https://github.com/Neo23x0/Talks/blob/master/Sigma_Hall_of_Fame_20211022.pdf) during his talk at the October 2021 EU ATT&CK Workshop.
|
||||
- Zircolite has been cited and presented during [JSAC 2023](https://jsac.jpcert.or.jp/archive/2023/pdf/JSAC2023_workshop_sigma_jp.pdf).
|
||||
- Zircolite has been cited and used in multiple research papers:
|
||||
- **CIDRE Team**:
|
||||
- [PWNJUTSU - Website](https://pwnjutsu.irisa.fr)
|
||||
- [PWNJUTSU - Academic Paper](https://hal.inria.fr/hal-03694719/document)
|
||||
- [CERBERE: Cybersecurity Exercise for Red and Blue Team Entertainment, Reproducibility](https://centralesupelec.hal.science/hal-04285565/file/CERBERE_final.pdf)
|
||||
- **Universidad de la República**:
|
||||
- [A Process Mining-Based Method for Attacker Profiling Using the MITRE ATT&CK Taxonomy](https://journals-sol.sbc.org.br/index.php/jisa/article/view/3902/2840)
|
||||
|
||||
---
|
||||
|
||||
## License
|
||||
|
||||
- All the **code** of the project is licensed under the [GNU Lesser General Public License](https://www.gnu.org/licenses/lgpl-3.0.en.html)
|
||||
- `evtx_dump` is under the MIT license
|
||||
- The rules are released under the [Detection Rule License (DRL)](https://github.com/SigmaHQ/Detection-Rule-License)
|
||||
- All the **code** of the project is licensed under the [GNU Lesser General Public License](https://www.gnu.org/licenses/lgpl-3.0.en.html).
|
||||
- `evtx_dump` is under the MIT license.
|
||||
- The rules are released under the [Detection Rule License (DRL) 1.0](https://github.com/Neo23x0/sigma/blob/master/LICENSE.Detection.Rules.md).
|
||||
|
||||
---
|
||||
@@ -268,5 +268,68 @@
|
||||
"Hash": {"separator":",", "equal":"="},
|
||||
"Hashes": {"separator":",", "equal":"="},
|
||||
"ConfigurationFileHash": {"separator":",", "equal":"="}
|
||||
},
|
||||
"transforms_enabled": true,
|
||||
"transforms":{
|
||||
"proctitle": [{
|
||||
"info": "Proctitle HEX to ASCII",
|
||||
"type": "python",
|
||||
"code": "def transform(param):\n\treturn bytes.fromhex(param).decode('ascii').replace('\\x00',' ')",
|
||||
"alias": false,
|
||||
"alias_name": "",
|
||||
"source_condition": ["auditd_input"],
|
||||
"enabled": true
|
||||
}],
|
||||
"cmd": [{
|
||||
"info": "Cmd HEX to ASCII",
|
||||
"type": "python",
|
||||
"code": "def transform(param):\n\treturn bytes.fromhex(param).decode('ascii').replace('\\x00',' ')",
|
||||
"alias": false,
|
||||
"alias_name": "",
|
||||
"source_condition": ["auditd_input"],
|
||||
"enabled": true
|
||||
}],
|
||||
"CommandLine": [
|
||||
{
|
||||
"info": "Base64 decoded CommandLine",
|
||||
"type": "python",
|
||||
"code": "\ndef transform(param):\n decoded_values = []\n concatenated_result = ''\n data = param\n\n base64_pattern = r'(?:[A-Za-z0-9+/]{4}){2,}(?:[A-Za-z0-9+/]{2}==|[A-Za-z0-9+/]{3}=)?'\n matches = re.findall(base64_pattern, data)\n for match in matches:\n decoded = base64.b64decode(match)\n encoding = chardet.detect(decoded)['encoding']\n if encoding and encoding in ['utf-8', 'ascii', 'utf-16le', 'ISO-8859-1']:\n decoded = decoded.decode(encoding)\n decoded = decoded.strip()\n if decoded.isprintable() and len(decoded) > 10 :\n decoded_values.append(decoded)\n \n concatenated_result = '|'.join(decoded_values)\n return concatenated_result\n",
|
||||
"alias": true,
|
||||
"alias_name": "CommandLine_b64decoded",
|
||||
"source_condition": ["evtx_input", "json_array_input", "json_input", "evtxtract_input", "db_input"],
|
||||
"enabled": false
|
||||
},
|
||||
{
|
||||
"info": "CommandLine credentials extraction - Regex by Practical Security Analytics - https://practicalsecurityanalytics.com/extracting-credentials-from-windows-logs/",
|
||||
"type": "python",
|
||||
"code": "\ndef transform(param):\n import re\n regex_patterns = [\n r'net.+user\\s+(?P<username>(?:\"((?:\\\\.|[^\"\\\\])*)\")|(?:[^\\s\"]+))\\s+(?P<password>(?:\"((?:\\\\.|[^\"\\\\])*)\")|(?:[^\\s\"]+))',\n r'net.+use\\s+(?P<share>\\\\\\\\\\S+)\\s+/USER:(?P<username>(?:\"((?:\\\\.|[^\"\\\\])*)\")|(?:[^\\s\"]+))\\s+(?P<password>(?:\"((?:\\\\.|[^\"\\\\])*)\")|(?:[^\\s\"]+))',\n r'schtasks.+/U\\s+(?P<username>(?:\"((?:\\\\.|[^\"\\\\])*)\")|(?:[^\\s\"]+)).+/P\\s+(?P<password>(?:\"((?:\\\\.|[^\"\\\\])*)\")|(?:[^\\s\"]+))',\n r'wmic.+/user:\\s*(?P<username>(?:\"((?:\\\\.|[^\"\\\\])*)\")|(?:[^\\s\"]+)).+/password:\\s*(?P<password>(?:\"((?:\\\\.|[^\"\\\\])*)\")|(?:[^\\s\"]+))',\n r'psexec.+-u\\s+(?P<username>(?:\"((?:\\\\.|[^\"\\\\])*)\")|(?:[^\\s\"]+)).+-p\\s+(?P<password>(?:\"((?:\\\\.|[^\"\\\\])*)\")|(?:[^\\s\"]+))'\n ]\n\n matches = []\n \n for pattern in regex_patterns:\n found = re.findall(pattern, param)\n if len(found) > 0:\n for match in list(found[0]):\n if len(match) > 0: \n matches.append(match) \n\n concatenated_result = '|'.join(matches)\n if concatenated_result == None:\n return ''\n return concatenated_result\n",
|
||||
"alias": true,
|
||||
"alias_name": "CommandLine_Extracted_Creds",
|
||||
"source_condition": ["evtx_input", "json_array_input", "json_input", "evtxtract_input", "db_input"],
|
||||
"enabled": false
|
||||
}
|
||||
],
|
||||
"Payload": [
|
||||
{
|
||||
"info": "Base64 decoded Payload",
|
||||
"type": "python",
|
||||
"code": "\ndef transform(param):\n decoded_values = []\n concatenated_result = ''\n data = param\n\n base64_pattern = r'(?:[A-Za-z0-9+/]{4}){2,}(?:[A-Za-z0-9+/]{2}==|[A-Za-z0-9+/]{3}=)?'\n matches = re.findall(base64_pattern, data)\n for match in matches:\n decoded = base64.b64decode(match)\n encoding = chardet.detect(decoded)['encoding']\n if encoding and encoding in ['utf-8', 'ascii', 'utf-16le', 'ISO-8859-1']:\n decoded = decoded.decode(encoding)\n decoded = decoded.strip()\n if decoded.isprintable() and len(decoded) > 10 :\n decoded_values.append(decoded)\n \n concatenated_result = '|'.join(decoded_values)\n return concatenated_result\n",
|
||||
"alias": true,
|
||||
"alias_name": "Payload_b64decoded",
|
||||
"source_condition": ["evtx_input", "json_array_input", "json_input", "evtxtract_input", "db_input"],
|
||||
"enabled": false
|
||||
}
|
||||
],
|
||||
"ServiceFileName":[
|
||||
{
|
||||
"info": "Base64 decoded ServiceFileName",
|
||||
"type": "python",
|
||||
"code": "\ndef transform(param):\n decoded_values = []\n concatenated_result = ''\n data = param\n\n base64_pattern = r'(?:[A-Za-z0-9+/]{4}){2,}(?:[A-Za-z0-9+/]{2}==|[A-Za-z0-9+/]{3}=)?'\n matches = re.findall(base64_pattern, data)\n for match in matches:\n decoded = base64.b64decode(match)\n encoding = chardet.detect(decoded)['encoding']\n if encoding and encoding in ['utf-8', 'ascii', 'utf-16le', 'ISO-8859-1']:\n decoded = decoded.decode(encoding)\n decoded = decoded.strip()\n if decoded.isprintable() and len(decoded) > 10 :\n decoded_values.append(decoded)\n \n concatenated_result = '|'.join(decoded_values)\n return concatenated_result\n",
|
||||
"alias": true,
|
||||
"alias_name": "ServiceFileName_b64decoded",
|
||||
"source_condition": ["evtx_input", "json_array_input", "json_input", "evtxtract_input", "db_input"],
|
||||
"enabled": false
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
@@ -12,13 +12,15 @@ There are a lot of ways to speed up Zircolite :
|
||||
- Using as much CPU core as possible : see below "[Using GNU Parallel](using-gnu-parallel)"
|
||||
- Using [Filtering](#filtering)
|
||||
|
||||
:information_source: There is an option to heavily limit the memory usage of Zircolite by using the `--ondiskdb <DB_NAME>` argument. This is only usefull to avoid errors when dealing with very large datasets and have a lot of time. **This should be used with caution and the below alternatives are far better choices**.
|
||||
> [!NOTE]
|
||||
> There is an option to heavily limit the memory usage of Zircolite by using the `--ondiskdb <DB_NAME>` argument. This is only usefull to avoid errors when dealing with very large datasets and if you have have a lot of time... **This should be used with caution and the below alternatives are far better choices**.
|
||||
|
||||
### Using GNU Parallel
|
||||
|
||||
Except when `evtx_dump` is used, Zircolite only use one core. So if you have a lot of EVTX files and their total size is big, it is recommended that you use a script to launch multiple Zircolite instances. On Linux or MacOS The easiest way is to use **GNU Parallel**.
|
||||
|
||||
:information_source: on MacOS, please use GNU find (`brew install find` will install `gfind`)
|
||||
> [!NOTE]
|
||||
> On MacOS, please use GNU find (`brew install find` will install `gfind`)
|
||||
|
||||
- **"DFIR Case mode" : One directory per computer/endpoint**
|
||||
|
||||
@@ -65,7 +67,8 @@ To speed up the detection process, you may want to use Zircolite on files matchi
|
||||
- `-s` or `--select` : select files partly matching the provided a string (case insensitive)
|
||||
- `-a` or `--avoid` : exclude files partly matching the provided a string (case insensitive)
|
||||
|
||||
:information_source: When using the two arguments, the "select" argument is always applied first and then the "avoid" argument is applied. So, it is possible to exclude files from included files but not the opposite.
|
||||
> [!NOTE]
|
||||
> When using the two arguments, the "select" argument is always applied first and then the "avoid" argument is applied. So, it is possible to exclude files from included files but not the opposite.
|
||||
|
||||
- Only use EVTX files that contains "sysmon" in their names
|
||||
|
||||
@@ -142,6 +145,9 @@ Sometimes, SIGMA rules can be very noisy (and generate a lot of false positives)
|
||||
|
||||
## Forwarding detected events
|
||||
|
||||
> [!WARNING]
|
||||
> Forwarding is DEPRECATED and will likely be disabled in a future release
|
||||
|
||||
Zircolite provide multiple ways to forward events to a collector :
|
||||
|
||||
- the HTTP forwarder : this is a very simple forwarder and pretty much a **"toy"** example and should be used when you have nothing else. An **example** server called is available in the [tools](../tools/zircolite_server/) directory
|
||||
@@ -180,7 +186,8 @@ python3 zircolite.py --evtx /sample.evtx --ruleset rules/rules_windows_sysmon_p
|
||||
|
||||
Since Splunk HEC default to the first associated index, `--index` is optional but can be used to specify the choosen index among the available ones.
|
||||
|
||||
:warning: On Windows do not forget to put quotes
|
||||
> [!WARNING]
|
||||
> On Windows do not forget to put quotes
|
||||
|
||||
### Forward to ELK
|
||||
|
||||
@@ -192,9 +199,11 @@ python3 zircolite.py --evtx /sample.evtx --ruleset rules/rules_windows_sysmon_p
|
||||
--eslogin "yourlogin" --espass "yourpass"
|
||||
```
|
||||
|
||||
:information_source: the `--eslogin` and `--espass` arguments are optional.
|
||||
> [!NOTE]
|
||||
> the `--eslogin` and `--espass` arguments are optional.
|
||||
|
||||
:warning: **Elastic is not handling logs the way Splunk does. Since Zircolite is flattening the field names in the JSON output some fields, especially when working with EVTX files, can have different types between Channels, logsources etc. So when Elastic uses automatic field mapping, mapping errors may prevent events insertion into Elastic.**
|
||||
> [!WARNING]
|
||||
> **Elastic is not handling logs the way Splunk does. Since Zircolite is flattening the field names in the JSON output some fields, especially when working with EVTX files, can have different types between Channels, logsources etc. So when Elastic uses automatic field mapping, mapping errors may prevent events insertion into Elastic.**
|
||||
|
||||
#### No local logs
|
||||
|
||||
@@ -204,7 +213,8 @@ When you forward detected events to an server, sometimes you don't want any log
|
||||
|
||||
Zircolite is able to forward all events and not just the detected events to Splunk, ELK or a custom HTTP Server. you just to use the `--forwardall` argument. Please note that this ability forward events as JSON and not specific `Windows` sourcetype.
|
||||
|
||||
:warning: **Elastic is not handling logs the way Splunk does. Since Zircolite is flattening the field names in the JSON output some fields, especially when working with EVTX files, can have different types between Channels, logsources etc. So when Elastic uses automatic field mapping, mapping errors may prevent events insertion into Elastic.**
|
||||
> [!WARNING]
|
||||
> **Elastic is not handling logs the way Splunk does. Since Zircolite is flattening the field names in the JSON output some fields, especially when working with EVTX files, can have different types between Channels, logsources etc. So when Elastic uses automatic field mapping, mapping errors may prevent events insertion into Elastic.**
|
||||
|
||||
## Templating and Formatting
|
||||
|
||||
@@ -245,7 +255,8 @@ mv data.js zircogui/
|
||||
|
||||
Then you just have to open `index.html` in your favorite browser and click on a Mitre Att&ck category or an alert level.
|
||||
|
||||
:warning: **The mini-GUI was not built to handle big datasets**.
|
||||
> [!WARNING]
|
||||
> **The mini-GUI was not built to handle big datasets**.
|
||||
|
||||
## Packaging Zircolite
|
||||
|
||||
@@ -264,7 +275,8 @@ Then you just have to open `index.html` in your favorite browser and click on a
|
||||
* After Python 3.8 install, you will need Nuitka : `pip3 install nuitka`
|
||||
* In the root folder of Zircolite type : `python3 -m nuitka --onefile zircolite.py`
|
||||
|
||||
:warning: When packaging with PyInstaller or Nuitka some AV may not like your package.
|
||||
> [!WARNING]
|
||||
> When packaging with PyInstaller or Nuitka some AV may not like your package.
|
||||
|
||||
## Using With DFIR Orc
|
||||
|
||||
@@ -311,7 +323,8 @@ Basically, if you want to integrate Zircolite with **DFIR Orc** :
|
||||
</wolf>
|
||||
```
|
||||
|
||||
:information_source: Please note that if you add this configuration to an existing one, you only need to keep the part between `<!-- BEGIN ... -->` and `<!-- /END ... -->` blocks.
|
||||
> [!NOTE]
|
||||
> Please note that if you add this configuration to an existing one, you only need to keep the part between `<!-- BEGIN ... -->` and `<!-- /END ... -->` blocks.
|
||||
|
||||
- Put your custom or default mapping file `zircolite_win10_nuitka.exe ` (the default one is in the Zircolite repository `config` directory) `rules_windows_generic.json` (the default one is in the Zircolite repository `rules` directory) in the the `config` directory.
|
||||
|
||||
@@ -350,7 +363,8 @@ Basically, if you want to integrate Zircolite with **DFIR Orc** :
|
||||
</archive>
|
||||
</toolembed>
|
||||
```
|
||||
:information_source: Please note that if you add this configuration to an existing one, you only need to keep the part between `<!-- BEGIN ... -->` and `<!-- /END ... -->` blocks.
|
||||
> [!NOTE]
|
||||
> Please note that if you add this configuration to an existing one, you only need to keep the part between `<!-- BEGIN ... -->` and `<!-- /END ... -->` blocks.
|
||||
|
||||
- Now you need to generate the **DFIR Orc** binary by executing `.\configure.ps1` at the root of the repository
|
||||
- The final output will be in the `output` directory
|
||||
|
||||
196
docs/Usage.md
196
docs/Usage.md
@@ -1,6 +1,7 @@
|
||||
# Usage
|
||||
|
||||
:information_source: if you use the packaged version of Zircolite don't forget to replace `python3 zircolite.py` in the examples by the packaged binary name.
|
||||
> [!NOTE]
|
||||
> If you use the packaged version of Zircolite don't forget to replace `python3 zircolite.py` in the examples by the packaged binary name.
|
||||
|
||||
## Requirements and Installation
|
||||
|
||||
@@ -159,7 +160,8 @@ python3 zircolite.py --events <EVTXTRACT_EXTRACTED_LOGS> --ruleset <RULESET> --
|
||||
python3 zircolite.py --events auditd.log --ruleset rules/rules_linux.json --auditd
|
||||
```
|
||||
|
||||
:information_source: `--events` and `--evtx` are strictly equivalent but `--events` make more sense with non EVTX logs.
|
||||
> [!NOTE]
|
||||
> `--events` and `--evtx` are strictly equivalent but `--events` make more sense with non-EVTX logs.
|
||||
|
||||
### Sysmon for Linux logs
|
||||
|
||||
@@ -169,7 +171,8 @@ Sysmon for linux has been released in October 2021. It outputs XML in text forma
|
||||
python3 zircolite.py --events sysmon.log --ruleset rules/rules_linux.json --sysmon-linux
|
||||
```
|
||||
|
||||
:information_source: Since the logs come from Linux, the default file extension when using `-S` case is `.log`
|
||||
> [!NOTE]
|
||||
> Since the logs come from Linux, the default file extension when using `-S` case is `.log`
|
||||
|
||||
### JSONL/NDJSON logs
|
||||
|
||||
@@ -287,7 +290,8 @@ python3 zircolite.py -e sample.evtx -r schtasks.yml -p sysmon -p windows-logsour
|
||||
|
||||
The converted rules/rulesets can be saved by using the `-sr` or the `--save-ruleset` arguments.
|
||||
|
||||
:information_source: When using multiple native Sigma rule/rulesets, you cannot differenciate pipelines. All the pipelines will be used in the conversion process.
|
||||
> [!NOTE]
|
||||
> When using multiple native Sigma rule/rulesets, you cannot differenciate pipelines. All the pipelines will be used in the conversion process.
|
||||
|
||||
## Field mappings, field exclusions, value exclusions, field aliases and field splitting
|
||||
|
||||
@@ -422,15 +426,187 @@ The final event log used to apply Sigma rules will look like this :
|
||||
|
||||
```json
|
||||
{
|
||||
"SHA1": "F43D9BB316E30AE1A3494AC5B0624F6BEA1BF054",
|
||||
"MD5": "04029E121A0CFA5991749937DD22A1D9",
|
||||
"SHA256": "9F914D42706FE215501044ACD85A32D58AAEF1419D404FDDFA5D3B48F66CCD9F",
|
||||
"IMPHASH": "7C955A0ABC747F57CCC4324480737EF7",
|
||||
"Hashes": "SHA1=F43D9BB316E30AE1A3494AC5B0624F6BEA1BF054,MD5=04029E121A0CFA5991749937DD22A1D9,SHA256=9F914D42706FE215501044ACD85A32D58AAEF1419D404FDDFA5D3B48F66CCD9F,IMPHASH=7C955A0ABC747F57CCC4324480737EF7",
|
||||
"SHA1": "x",
|
||||
"MD5": "x",
|
||||
"SHA256": "x",
|
||||
"IMPHASH": "x",
|
||||
"Hashes": "SHA1=x,MD5=x,SHA256=x,IMPHASH=x",
|
||||
"EventID": 1
|
||||
}
|
||||
```
|
||||
|
||||
## Field Transforms
|
||||
|
||||
### What Are Transforms?
|
||||
|
||||
Transforms in Zircolite are custom functions that manipulate the value of a specific field during the event flattening process. They allow you to:
|
||||
|
||||
- Format or normalize data
|
||||
- Enrich events with additional computed fields
|
||||
- Decode encoded data (e.g., Base64, hexadecimal)
|
||||
- Extract information using regular expressions
|
||||
|
||||
By using transforms, you can preprocess event data to make it more suitable for detection rules and analysis.
|
||||
|
||||
### Enabling Transforms
|
||||
|
||||
Transforms are configured in the config file (the default one is in `config/fieldMappings.json`) under the `"transforms"` section. To enable transforms, set the `"transforms_enabled"` flag to `true` in your configuration file:
|
||||
|
||||
```json
|
||||
{
|
||||
"transforms_enabled": true,
|
||||
"transforms": {
|
||||
// Transform definitions
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Configuring Transforms
|
||||
|
||||
Transforms are defined in the `"transforms"` section of the configuration file. Each transform is associated with a specific field and consists of several properties.
|
||||
|
||||
### Transform Structure
|
||||
|
||||
A transform definition has the following structure:
|
||||
|
||||
- **Field Name**: The name of the field to which the transform applies.
|
||||
- **Transform List**: A list of transform objects for the field.
|
||||
|
||||
Each transform object contains:
|
||||
|
||||
- **info**: A description of what the transform does.
|
||||
- **type**: The type of the transform (currently only `"python"` is supported).
|
||||
- **code**: The Python code that performs the transformation.
|
||||
- **alias**: A boolean indicating whether the result should be stored in a new field.
|
||||
- **alias_name**: The name of the new field if `alias` is `true`.
|
||||
- **source_condition**: A list specifying when the transform should be applied based on the input type (e.g., `["evtx_input", "json_input"]`).
|
||||
- **enabled**: A boolean indicating whether the transform is active.
|
||||
|
||||
#### Source conditions possible values
|
||||
|
||||
| Sets `source_condition` Value |
|
||||
|-------------------------------|
|
||||
| `"json_input"` |
|
||||
| `"json_array_input"` |
|
||||
| `"db_input"` |
|
||||
| `"sysmon_linux_input"` |
|
||||
| `"auditd_input"` |
|
||||
| `"xml_input"` |
|
||||
| `"evtxtract_input"` |
|
||||
| `"csv_input"` |
|
||||
| `"evtx_input"` |
|
||||
|
||||
#### Example Transform Object
|
||||
|
||||
```json
|
||||
{
|
||||
"info": "Base64 decoded CommandLine",
|
||||
"type": "python",
|
||||
"code": "def transform(param):\n # Transformation logic\n return transformed_value",
|
||||
"alias": true,
|
||||
"alias_name": "CommandLine_b64decoded",
|
||||
"source_condition": ["evtx_input", "json_input"],
|
||||
"enabled": true
|
||||
}
|
||||
```
|
||||
|
||||
### Available Fields
|
||||
|
||||
You can define transforms for any field present in your event data. In the configuration, transforms are keyed by the field name:
|
||||
|
||||
```json
|
||||
"transforms": {
|
||||
"CommandLine": [
|
||||
{
|
||||
// Transform object
|
||||
}
|
||||
],
|
||||
"Payload": [
|
||||
{
|
||||
// Transform object
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Writing Transform Functions
|
||||
|
||||
Zircolite uses `RestrictedPython` to safely execute transform functions. This means that certain built-in functions and modules are available, while others are restricted.
|
||||
The function must be named `transform` and accept a single parameter `param`, which is the original value of the field.
|
||||
|
||||
**Available Modules and Functions:**
|
||||
|
||||
- **Built-in Functions**: A limited set of Python built-in functions, such as `len`, `int`, `str`, etc.
|
||||
- **Modules**: You can import `re` for regular expressions, `base64` for encoding/decoding, and `chardet` for character encoding detection.
|
||||
|
||||
**Unavailable Features:**
|
||||
|
||||
- Access to file I/O, network, or system calls is prohibited.
|
||||
- Use of certain built-in functions that can affect the system is restricted.
|
||||
|
||||
#### Example Transform Functions
|
||||
|
||||
##### Base64 Decoding
|
||||
|
||||
```python
|
||||
def transform(param):
|
||||
import base64
|
||||
decoded = base64.b64decode(param)
|
||||
return decoded.decode('utf-8')
|
||||
```
|
||||
|
||||
##### Hexadecimal to ASCII Conversion
|
||||
|
||||
```python
|
||||
def transform(param):
|
||||
decoded = bytes.fromhex(param).decode('ascii')
|
||||
return decoded.replace('\x00', ' ')
|
||||
```
|
||||
|
||||
### Applying Transforms
|
||||
|
||||
Transforms are automatically applied during the event flattening process if:
|
||||
|
||||
- They are **enabled** (`"enabled": true`).
|
||||
- The current input type matches the **source condition** (`"source_condition": [...]`).
|
||||
|
||||
For each event, Zircolite checks if any transforms are defined for the fields present in the event. If so, it executes the transform function and replaces the field's value with the transformed value or stores it in a new field if `alias` is `true`.
|
||||
|
||||
### Example
|
||||
|
||||
**Use Case**: Convert hexadecimal-encoded command lines in Auditd logs to readable ASCII strings.
|
||||
|
||||
**Configuration:**
|
||||
|
||||
```json
|
||||
"proctitle": [
|
||||
{
|
||||
"info": "Proctitle HEX to ASCII",
|
||||
"type": "python",
|
||||
"code": "def transform(param):\n return bytes.fromhex(param).decode('ascii').replace('\\x00', ' ')",
|
||||
"alias": false,
|
||||
"alias_name": "",
|
||||
"source_condition": ["auditd_input"],
|
||||
"enabled": true
|
||||
}
|
||||
]
|
||||
```
|
||||
|
||||
**Explanation:**
|
||||
|
||||
- **Field**: `proctitle`
|
||||
- **Function**: Converts hexadecimal strings to ASCII and replaces null bytes with spaces.
|
||||
- **Alias**: `false` (the original `proctitle` field is replaced).
|
||||
|
||||
### Best Practices
|
||||
|
||||
- **Test Your Transforms**: Before enabling a transform, ensure that the code works correctly with sample data.
|
||||
- **Use Aliases Wisely**: If you don't want to overwrite the original field, set `"alias": true` and provide an `"alias_name"`.
|
||||
- **Manage Performance**: Complex transforms can impact performance. Optimize your code and only enable necessary transforms.
|
||||
- **Keep Transforms Specific**: Tailor transforms to specific fields and input types using `"source_condition"` to avoid unexpected behavior.
|
||||
|
||||
## Generate your own rulesets
|
||||
|
||||
Default rulesets are already provided in the `rules` directory. These rulesets only are the conversion of the rules located in [rules/windows](https://github.com/SigmaHQ/sigma/tree/master/rules/windows) directory of the Sigma repository. These rulesets are provided to use Zircolite out-of-the-box but [you should generate your own rulesets](#why-you-should-build-your-own-rulesets).
|
||||
@@ -541,7 +717,7 @@ For example :
|
||||
|
||||
## Docker
|
||||
|
||||
Zircolite is also packaged as a Docker image (cf. [wagga40/zircolite](https://hub.docker.com/r/wagga40/zircolite) on Docker Hub), which embeds all dependencies (e.g. `evtx_dump`) and provides a platform-independant way of using the tool.
|
||||
Zircolite is also packaged as a Docker image (cf. [wagga40/zircolite](https://hub.docker.com/r/wagga40/zircolite) on Docker Hub), which embeds all dependencies (e.g. `evtx_dump`) and provides a platform-independant way of using the tool. Please note this image is not updated with the last rulesets !
|
||||
|
||||
You can pull the last image with : `docker pull wagga40/zircolite:latest`
|
||||
|
||||
|
||||
Binary file not shown.
@@ -2,6 +2,7 @@
|
||||
* [Requirements and Installation](Usage.md#requirements-and-installation)
|
||||
* [Basic usage](Usage.md#basic-usage)
|
||||
* [Field mappings, field exclusions, value exclusions, field aliases and field splitting](Usage.md#field-mappings-field-exclusions-value-exclusions-field-aliases-and-field-splitting)
|
||||
* [Field Transforms](Usage.md#field-transforms)
|
||||
* [Generate your own rulesets](Usage.md#generate-your-own-rulesets)
|
||||
* [Generate embedded versions](Usage.md#generate-embedded-versions)
|
||||
* [Docker](Usage.md#docker)
|
||||
|
||||
@@ -13,4 +13,6 @@ pysigma>=0.10.10
|
||||
pysigma-pipeline-sysmon>=1.0.3
|
||||
pysigma-pipeline-windows>=1.1.1
|
||||
pysigma-backend-sqlite>=0.1.1
|
||||
pyyaml
|
||||
pyyaml
|
||||
chardet
|
||||
RestrictedPython
|
||||
|
||||
@@ -1,4 +1,6 @@
|
||||
orjson>=3.9.15
|
||||
xxhash
|
||||
colorama>=0.4.4
|
||||
tqdm>=4.58.0
|
||||
tqdm>=4.58.0
|
||||
chardet
|
||||
RestrictedPython
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
@@ -1,4 +1,3 @@
|
||||
flask>=2.2.5
|
||||
jinja2>=3.1.4
|
||||
flask>=1.1.2
|
||||
jinja2>=2.11.3
|
||||
werkzeug>=3.0.3 # not directly required, pinned by Snyk to avoid a vulnerability
|
||||
zipp>=3.19.1 # not directly required, pinned by Snyk to avoid a vulnerability
|
||||
|
||||
552
zircolite.py
552
zircolite.py
@@ -3,6 +3,8 @@
|
||||
# Standard libs
|
||||
import argparse
|
||||
import asyncio
|
||||
import base64
|
||||
import chardet
|
||||
import csv
|
||||
import functools
|
||||
import hashlib
|
||||
@@ -29,6 +31,12 @@ import xxhash
|
||||
from colorama import Fore
|
||||
from tqdm import tqdm
|
||||
from tqdm.asyncio import tqdm as tqdmAsync
|
||||
from RestrictedPython import compile_restricted
|
||||
from RestrictedPython import safe_builtins
|
||||
from RestrictedPython import limited_builtins
|
||||
from RestrictedPython import utility_builtins
|
||||
from RestrictedPython.Eval import default_guarded_getiter
|
||||
from RestrictedPython.Guards import guarded_iter_unpack_sequence
|
||||
|
||||
# External libs (Optional)
|
||||
forwardingDisabled = False
|
||||
@@ -577,7 +585,7 @@ class JSONFlattener:
|
||||
timeBefore="9999-12-12T23:59:59",
|
||||
timeField=None,
|
||||
hashes=False,
|
||||
JSONArray=False,
|
||||
args_config=None,
|
||||
):
|
||||
self.logger = logger or logging.getLogger(__name__)
|
||||
self.keyDict = {}
|
||||
@@ -587,7 +595,19 @@ class JSONFlattener:
|
||||
self.timeBefore = timeBefore
|
||||
self.timeField = timeField
|
||||
self.hashes = hashes
|
||||
self.JSONArray = JSONArray
|
||||
self.args_config = args_config
|
||||
self.JSONArray = args_config.json_array_input
|
||||
# Initialize the cache for compiled code
|
||||
self.compiled_code_cache = {}
|
||||
|
||||
# Convert the argparse.Namespace to a dictionary
|
||||
args_dict = vars(args_config)
|
||||
# Find the chosen input format
|
||||
self.chosen_input = next(
|
||||
(key for key, value in args_dict.items() if "_input" in key and value), None
|
||||
)
|
||||
if self.chosen_input is None:
|
||||
self.chosen_input = "evtx_input" # Since evtx is the default input, we force it no chosen input has been found
|
||||
|
||||
with open(configFile, "r", encoding="UTF-8") as fieldMappingsFile:
|
||||
self.fieldMappingsDict = json.loads(fieldMappingsFile.read())
|
||||
@@ -596,6 +616,28 @@ class JSONFlattener:
|
||||
self.uselessValues = self.fieldMappingsDict["useless"]
|
||||
self.aliases = self.fieldMappingsDict["alias"]
|
||||
self.fieldSplitList = self.fieldMappingsDict["split"]
|
||||
self.transforms = self.fieldMappingsDict["transforms"]
|
||||
self.transforms_enabled = self.fieldMappingsDict["transforms_enabled"]
|
||||
|
||||
# Define the authorized BUILTINS for Resticted Python
|
||||
def default_guarded_getitem(ob, index):
|
||||
return ob[index]
|
||||
|
||||
default_guarded_getattr = getattr
|
||||
|
||||
self.RestrictedPython_BUILTINS = {
|
||||
"__name__": "script",
|
||||
"_getiter_": default_guarded_getiter,
|
||||
"_getattr_": default_guarded_getattr,
|
||||
"_getitem_": default_guarded_getitem,
|
||||
"base64": base64,
|
||||
"re": re,
|
||||
"chardet": chardet,
|
||||
"_iter_unpack_sequence_": guarded_iter_unpack_sequence,
|
||||
}
|
||||
self.RestrictedPython_BUILTINS.update(safe_builtins)
|
||||
self.RestrictedPython_BUILTINS.update(limited_builtins)
|
||||
self.RestrictedPython_BUILTINS.update(utility_builtins)
|
||||
|
||||
def run(self, file):
|
||||
"""
|
||||
@@ -607,6 +649,25 @@ class JSONFlattener:
|
||||
JSONOutput = []
|
||||
fieldStmt = ""
|
||||
|
||||
def transformValue(code, param):
|
||||
try:
|
||||
# Check if the code has already been compiled
|
||||
if code in self.compiled_code_cache:
|
||||
byte_code = self.compiled_code_cache[code]
|
||||
else:
|
||||
# Compile the code and store it in the cache
|
||||
byte_code = compile_restricted(
|
||||
code, filename="<inline code>", mode="exec"
|
||||
)
|
||||
self.compiled_code_cache[code] = byte_code
|
||||
# Prepare the execution environment
|
||||
TransformFunction = {}
|
||||
exec(byte_code, self.RestrictedPython_BUILTINS, TransformFunction)
|
||||
return TransformFunction["transform"](param)
|
||||
except Exception as e:
|
||||
self.logger.debug(f"ERROR: Couldn't apply transform: {e}")
|
||||
return param # Return the original parameter if transform fails
|
||||
|
||||
def flatten(x, name=""):
|
||||
nonlocal fieldStmt
|
||||
# If it is a Dict go deeper
|
||||
@@ -625,6 +686,7 @@ class JSONFlattener:
|
||||
value = x
|
||||
# Excluding useless values (e.g. "null"). The value must be an exact match.
|
||||
if value not in self.uselessValues:
|
||||
|
||||
# Applying field mappings
|
||||
rawFieldName = name[:-1]
|
||||
if rawFieldName in self.fieldMappings:
|
||||
@@ -635,12 +697,38 @@ class JSONFlattener:
|
||||
e for e in rawFieldName.split(".")[-1] if e.isalnum()
|
||||
)
|
||||
|
||||
# Preparing aliases
|
||||
# Preparing aliases (work on original field name and Mapped field name)
|
||||
keys = [key]
|
||||
if key in self.aliases:
|
||||
keys.append(self.aliases[key])
|
||||
if rawFieldName in self.aliases:
|
||||
keys.append(self.aliases[rawFieldName])
|
||||
for fieldName in [key, rawFieldName]:
|
||||
if fieldName in self.aliases:
|
||||
keys.append(self.aliases[key])
|
||||
|
||||
# Applying field transforms (work on original field name and Mapped field name)
|
||||
keysThatNeedTransformedValues = []
|
||||
transformedValuesByKeys = {}
|
||||
if self.transforms_enabled:
|
||||
for fieldName in [key, rawFieldName]:
|
||||
if fieldName in self.transforms:
|
||||
for transform in self.transforms[fieldName]:
|
||||
if (
|
||||
transform["enabled"]
|
||||
and self.chosen_input
|
||||
in transform["source_condition"]
|
||||
):
|
||||
transformCode = transform["code"]
|
||||
# If the transform rule ask for a dedicated alias
|
||||
if transform["alias"]:
|
||||
keys.append(transform["alias_name"])
|
||||
keysThatNeedTransformedValues.append(
|
||||
transform["alias_name"]
|
||||
)
|
||||
transformedValuesByKeys[
|
||||
transform["alias_name"]
|
||||
] = transformValue(transformCode, value)
|
||||
else:
|
||||
value = transformValue(
|
||||
transformCode, value
|
||||
)
|
||||
|
||||
# Applying field splitting
|
||||
fieldsToSplit = []
|
||||
@@ -671,7 +759,10 @@ class JSONFlattener:
|
||||
|
||||
# Applying aliases
|
||||
for key in keys:
|
||||
JSONLine[key] = value
|
||||
if key in keysThatNeedTransformedValues:
|
||||
JSONLine[key] = transformedValuesByKeys[key]
|
||||
else:
|
||||
JSONLine[key] = value
|
||||
# Creating the CREATE TABLE SQL statement
|
||||
keyLower = key.lower()
|
||||
if keyLower not in self.keyDict:
|
||||
@@ -780,7 +871,13 @@ class zirCore:
|
||||
conn = None
|
||||
self.logger.debug(f"CONNECTING TO : {db}")
|
||||
try:
|
||||
conn = sqlite3.connect(db)
|
||||
if db == ":memory:":
|
||||
conn = sqlite3.connect(db, isolation_level=None)
|
||||
conn.execute("PRAGMA journal_mode = MEMORY;")
|
||||
conn.execute("PRAGMA synchronous = OFF;")
|
||||
conn.execute("PRAGMA temp_store = MEMORY;")
|
||||
else:
|
||||
conn = sqlite3.connect(db)
|
||||
conn.row_factory = sqlite3.Row # Allows to get a dict
|
||||
|
||||
def udf_regex(x, y):
|
||||
@@ -800,9 +897,7 @@ class zirCore:
|
||||
|
||||
def createDb(self, fieldStmt):
|
||||
createTableStmt = f"CREATE TABLE logs ( row_id INTEGER, {fieldStmt} PRIMARY KEY(row_id AUTOINCREMENT) );"
|
||||
self.logger.debug(
|
||||
" CREATE : " + createTableStmt.replace("\n", " ").replace("\r", "")
|
||||
)
|
||||
self.logger.debug(f" CREATE : {createTableStmt}")
|
||||
if not self.executeQuery(createTableStmt):
|
||||
self.logger.error(f"{Fore.RED} [-] Unable to create table{Fore.RESET}")
|
||||
sys.exit(1)
|
||||
@@ -827,19 +922,23 @@ class zirCore:
|
||||
return False
|
||||
|
||||
def executeSelectQuery(self, query):
|
||||
"""Perform a SQL Query -SELECT only- with the provided connection"""
|
||||
if self.dbConnection is not None:
|
||||
dbHandle = self.dbConnection.cursor()
|
||||
self.logger.debug(f"EXECUTING : {query}")
|
||||
try:
|
||||
data = dbHandle.execute(query)
|
||||
return data
|
||||
except Error as e:
|
||||
self.logger.debug(f" [-] {e}")
|
||||
return {}
|
||||
else:
|
||||
"""
|
||||
Execute a SELECT SQL query and return the results as a list of dictionaries.
|
||||
"""
|
||||
if self.dbConnection is None:
|
||||
self.logger.error(f"{Fore.RED} [-] No connection to Db{Fore.RESET}")
|
||||
return {}
|
||||
return []
|
||||
try:
|
||||
cursor = self.dbConnection.cursor()
|
||||
self.logger.debug(f"Executing SELECT query: {query}")
|
||||
cursor.execute(query)
|
||||
rows = cursor.fetchall()
|
||||
# Convert rows to list of dictionaries
|
||||
result = [dict(row) for row in rows]
|
||||
return result
|
||||
except sqlite3.Error as e:
|
||||
self.logger.debug(f" [-] SQL query error: {e}")
|
||||
return []
|
||||
|
||||
def loadDbInMemory(self, db):
|
||||
"""In db only mode it is possible to restore an on disk Db to avoid EVTX extraction and flattening"""
|
||||
@@ -847,20 +946,30 @@ class zirCore:
|
||||
dbfileConnection.backup(self.dbConnection)
|
||||
dbfileConnection.close()
|
||||
|
||||
def escape_identifier(self, identifier):
|
||||
"""Escape SQL identifiers like table or column names."""
|
||||
return identifier.replace('"', '""')
|
||||
|
||||
def insertData2Db(self, JSONLine):
|
||||
"""Build INSERT INTO Query and insert data into Db"""
|
||||
columnsStr = ""
|
||||
valuesStr = ""
|
||||
|
||||
for key in sorted(JSONLine.keys()):
|
||||
columnsStr += "'" + key + "',"
|
||||
if isinstance(JSONLine[key], int):
|
||||
valuesStr += str(JSONLine[key]) + ", "
|
||||
else:
|
||||
valuesStr += "'" + str(JSONLine[key]).replace("'", "''") + "', "
|
||||
|
||||
insertStrmt = f"INSERT INTO logs ({columnsStr[:-1]}) VALUES ({valuesStr[:-2]});"
|
||||
return self.executeQuery(insertStrmt)
|
||||
"""Build a parameterized INSERT INTO query and insert data into the database."""
|
||||
columns = JSONLine.keys()
|
||||
columnsEscaped = ", ".join([self.escape_identifier(col) for col in columns])
|
||||
placeholders = ", ".join(["?"] * len(columns))
|
||||
values = []
|
||||
for col in columns:
|
||||
value = JSONLine[col]
|
||||
if isinstance(value, int):
|
||||
# Check if value exceeds SQLite INTEGER limits
|
||||
if abs(value) > 9223372036854775807:
|
||||
value = str(value) # Convert to string
|
||||
values.append(value)
|
||||
insertStmt = f"INSERT INTO logs ({columnsEscaped}) VALUES ({placeholders})"
|
||||
try:
|
||||
self.dbConnection.execute(insertStmt, values)
|
||||
return True
|
||||
except Exception as e:
|
||||
self.logger.debug(f" [-] {e}")
|
||||
return False
|
||||
|
||||
def insertFlattenedJSON2Db(self, flattenedJSON, forwarder=None):
|
||||
if forwarder:
|
||||
@@ -872,7 +981,7 @@ class zirCore:
|
||||
def saveFlattenedJSON2File(self, flattenedJSON, outputFile):
|
||||
with open(outputFile, "w", encoding="utf-8") as file:
|
||||
for JSONLine in tqdm(flattenedJSON, colour="yellow"):
|
||||
file.write(json.dumps(JSONLine).decode("utf-8") + "\n")
|
||||
file.write(f'{json.dumps(JSONLine).decode("utf-8")}\n')
|
||||
|
||||
def saveDbToDisk(self, dbFilename):
|
||||
self.logger.info("[+] Saving working data to disk as a SQLite DB")
|
||||
@@ -881,71 +990,70 @@ class zirCore:
|
||||
onDiskDb.close()
|
||||
|
||||
def executeRule(self, rule):
|
||||
results = {}
|
||||
filteredRows = []
|
||||
counter = 0
|
||||
if "rule" in rule:
|
||||
# for each SQL Query in the Sigma rule
|
||||
for SQLQuery in rule["rule"]:
|
||||
data = self.executeSelectQuery(SQLQuery)
|
||||
if data != {}:
|
||||
# Convert to array of dict
|
||||
rows = [dict(row) for row in data.fetchall()]
|
||||
if len(rows) > 0:
|
||||
counter += len(rows)
|
||||
for row in rows:
|
||||
if self.csvMode: # Cleaning "annoying" values for CSV
|
||||
match = {
|
||||
k: str(v)
|
||||
.replace("\n", "")
|
||||
.replace("\r", "")
|
||||
.replace("None", "")
|
||||
for k, v in row.items()
|
||||
}
|
||||
else: # Cleaning null/None fields
|
||||
match = {k: v for k, v in row.items() if v is not None}
|
||||
filteredRows.append(match)
|
||||
if "level" not in rule:
|
||||
rule["level"] = "unknown"
|
||||
if "tags" not in rule:
|
||||
rule["tags"] = []
|
||||
if "filename" not in rule:
|
||||
rule["filename"] = ""
|
||||
if self.csvMode:
|
||||
results = {
|
||||
"title": rule["title"],
|
||||
"id": rule["id"],
|
||||
"description": rule["description"]
|
||||
.replace("\n", "")
|
||||
.replace("\r", ""),
|
||||
"sigmafile": rule["filename"],
|
||||
"sigma": rule["rule"],
|
||||
"rule_level": rule["level"],
|
||||
"tags": rule["tags"],
|
||||
"count": counter,
|
||||
"matches": filteredRows,
|
||||
}
|
||||
else:
|
||||
results = {
|
||||
"title": rule["title"],
|
||||
"id": rule["id"],
|
||||
"description": rule["description"],
|
||||
"sigmafile": rule["filename"],
|
||||
"sigma": rule["rule"],
|
||||
"rule_level": rule["level"],
|
||||
"tags": rule["tags"],
|
||||
"count": counter,
|
||||
"matches": filteredRows,
|
||||
}
|
||||
if counter > 0:
|
||||
self.logger.debug(
|
||||
f'DETECTED : {rule["title"]} - Matches : {counter} events'
|
||||
)
|
||||
else:
|
||||
self.logger.debug("RULE FORMAT ERROR : rule key Missing")
|
||||
if filteredRows == []:
|
||||
"""
|
||||
Execute a single Sigma rule against the database and return the results.
|
||||
"""
|
||||
if "rule" not in rule:
|
||||
self.logger.debug("RULE FORMAT ERROR: 'rule' key missing")
|
||||
return {}
|
||||
|
||||
# Set default values for missing rule keys
|
||||
rule_level = rule.get("level", "unknown")
|
||||
tags = rule.get("tags", [])
|
||||
filename = rule.get("filename", "")
|
||||
description = rule.get("description", "")
|
||||
title = rule.get("title", "Unnamed Rule")
|
||||
rule_id = rule.get("id", "")
|
||||
sigma_queries = rule["rule"]
|
||||
|
||||
filteredRows = []
|
||||
|
||||
# Process each SQL query in the rule
|
||||
for SQLQuery in sigma_queries:
|
||||
data = self.executeSelectQuery(SQLQuery)
|
||||
if data:
|
||||
if self.csvMode:
|
||||
# Clean values for CSV output
|
||||
cleaned_rows = [
|
||||
{
|
||||
k: str(v)
|
||||
.replace("\n", "")
|
||||
.replace("\r", "")
|
||||
.replace("None", "")
|
||||
for k, v in dict(row).items()
|
||||
}
|
||||
for row in data
|
||||
]
|
||||
else:
|
||||
# Remove None values
|
||||
cleaned_rows = [
|
||||
{k: v for k, v in dict(row).items() if v is not None}
|
||||
for row in data
|
||||
]
|
||||
filteredRows.extend(cleaned_rows)
|
||||
|
||||
if filteredRows:
|
||||
results = {
|
||||
"title": title,
|
||||
"id": rule_id,
|
||||
"description": (
|
||||
description.replace("\n", "").replace("\r", "")
|
||||
if self.csvMode
|
||||
else description
|
||||
),
|
||||
"sigmafile": filename,
|
||||
"sigma": sigma_queries,
|
||||
"rule_level": rule_level,
|
||||
"tags": tags,
|
||||
"count": len(filteredRows),
|
||||
"matches": filteredRows,
|
||||
}
|
||||
self.logger.debug(
|
||||
f"DETECTED: {title} - Matches: {len(filteredRows)} events"
|
||||
)
|
||||
return results
|
||||
else:
|
||||
return {}
|
||||
return results
|
||||
|
||||
def loadRulesetFromFile(self, filename, ruleFilters):
|
||||
try:
|
||||
@@ -994,79 +1102,110 @@ class zirCore:
|
||||
stream=False,
|
||||
lastRuleset=False,
|
||||
):
|
||||
"""
|
||||
Execute all rules in the ruleset and handle output.
|
||||
"""
|
||||
csvWriter = None
|
||||
# Results are written upon detection to allow analysis during execution and to avoid losing results in case of error.
|
||||
with open(outFile, writeMode, encoding="utf-8", newline="") as fileHandle:
|
||||
with tqdm(self.ruleset, colour="yellow") as ruleBar:
|
||||
if not self.noOutput and not self.csvMode and writeMode != "a":
|
||||
fileHandle.write("[")
|
||||
for rule in ruleBar: # for each rule in ruleset
|
||||
if showAll and "title" in rule:
|
||||
ruleBar.write(
|
||||
f'{Fore.BLUE} - {rule["title"]} [{self.ruleLevelPrintFormatter(rule["level"], Fore.BLUE)}]{Fore.RESET}'
|
||||
) # Print all rules
|
||||
ruleResults = self.executeRule(rule)
|
||||
if ruleResults != {}:
|
||||
if self.limit == -1 or ruleResults["count"] <= self.limit:
|
||||
ruleBar.write(
|
||||
f'{Fore.CYAN} - {ruleResults["title"]} [{self.ruleLevelPrintFormatter(rule["level"], Fore.CYAN)}] : {ruleResults["count"]} events{Fore.RESET}'
|
||||
first_json_output = True # To manage commas in JSON output
|
||||
is_json_mode = not self.csvMode
|
||||
|
||||
# Prepare output file handle if needed
|
||||
fileHandle = None
|
||||
if not self.noOutput:
|
||||
# Open file in text mode since we will write decoded strings
|
||||
fileHandle = open(outFile, writeMode, encoding="utf-8", newline="")
|
||||
if is_json_mode and writeMode != "a":
|
||||
fileHandle.write("[") # Start JSON array
|
||||
|
||||
# Iterate over rules in the ruleset
|
||||
with tqdm(self.ruleset, colour="yellow") as ruleBar:
|
||||
for rule in ruleBar:
|
||||
# Show all rules if showAll is True
|
||||
if showAll and "title" in rule:
|
||||
rule_title = rule["title"]
|
||||
rule_level = rule.get("level", "unknown")
|
||||
formatted_level = self.ruleLevelPrintFormatter(
|
||||
rule_level, Fore.BLUE
|
||||
)
|
||||
ruleBar.write(
|
||||
f"{Fore.BLUE} - {rule_title} [{formatted_level}]{Fore.RESET}"
|
||||
)
|
||||
|
||||
# Execute the rule
|
||||
ruleResults = self.executeRule(rule)
|
||||
if not ruleResults:
|
||||
continue # No matches, skip to next rule
|
||||
|
||||
# Apply limit if set
|
||||
if self.limit != -1 and ruleResults["count"] > self.limit:
|
||||
continue # Exceeds limit, skip this result
|
||||
|
||||
# Write progress message
|
||||
rule_title = ruleResults["title"]
|
||||
rule_level = ruleResults.get("rule_level", "unknown")
|
||||
formatted_level = self.ruleLevelPrintFormatter(rule_level, Fore.CYAN)
|
||||
rule_count = ruleResults["count"]
|
||||
ruleBar.write(
|
||||
f"{Fore.CYAN} - {rule_title} [{formatted_level}] : {rule_count} events{Fore.RESET}"
|
||||
)
|
||||
|
||||
# Store results if needed
|
||||
if KeepResults or (remote and not stream):
|
||||
self.fullResults.append(ruleResults)
|
||||
|
||||
# Forward results if streaming
|
||||
if stream and forwarder:
|
||||
forwarder.send([ruleResults], False)
|
||||
|
||||
# Handle output to file
|
||||
if not self.noOutput:
|
||||
if self.csvMode:
|
||||
# Initialize CSV writer if not already done
|
||||
if csvWriter is None:
|
||||
fieldnames = [
|
||||
"rule_title",
|
||||
"rule_description",
|
||||
"rule_level",
|
||||
"rule_count",
|
||||
] + list(ruleResults["matches"][0].keys())
|
||||
csvWriter = csv.DictWriter(
|
||||
fileHandle,
|
||||
delimiter=self.delimiter,
|
||||
fieldnames=fieldnames,
|
||||
)
|
||||
# Store results for templating and event forwarding (only if stream mode is disabled)
|
||||
if KeepResults or (remote is not None and not stream):
|
||||
self.fullResults.append(ruleResults)
|
||||
if stream and forwarder is not None:
|
||||
forwarder.send([ruleResults], False)
|
||||
if not self.noOutput:
|
||||
# To avoid printing this twice on stdout but in the logs...
|
||||
logLevel = self.logger.getEffectiveLevel()
|
||||
self.logger.setLevel(logging.DEBUG)
|
||||
self.logger.debug(
|
||||
f' - {ruleResults["title"]} [{ruleResults["rule_level"]}] : {ruleResults["count"]} events'
|
||||
)
|
||||
self.logger.setLevel(logLevel)
|
||||
# Output to json or csv file
|
||||
if self.csvMode:
|
||||
if (
|
||||
not csvWriter
|
||||
): # Creating the CSV header and the fields ("agg" is for queries with aggregation)
|
||||
csvWriter = csv.DictWriter(
|
||||
fileHandle,
|
||||
delimiter=self.delimiter,
|
||||
fieldnames=[
|
||||
"rule_title",
|
||||
"rule_description",
|
||||
"rule_level",
|
||||
"rule_count",
|
||||
"agg",
|
||||
]
|
||||
+ list(ruleResults["matches"][0].keys()),
|
||||
)
|
||||
csvWriter.writeheader()
|
||||
for data in ruleResults["matches"]:
|
||||
dictCSV = {
|
||||
"rule_title": ruleResults["title"],
|
||||
"rule_description": ruleResults[
|
||||
"description"
|
||||
],
|
||||
"rule_level": ruleResults["rule_level"],
|
||||
"rule_count": ruleResults["count"],
|
||||
**data,
|
||||
}
|
||||
csvWriter.writerow(dictCSV)
|
||||
else:
|
||||
try:
|
||||
fileHandle.write(
|
||||
json.dumps(
|
||||
ruleResults, option=json.OPT_INDENT_2
|
||||
).decode("utf-8")
|
||||
)
|
||||
fileHandle.write(",\n")
|
||||
except Exception as e:
|
||||
self.logger.error(
|
||||
f"{Fore.RED} [-] Error saving some results : {e}{Fore.RESET}"
|
||||
)
|
||||
if (not self.noOutput and not self.csvMode) and lastRuleset:
|
||||
fileHandle.write("{}]") # Added to produce a valid JSON Array
|
||||
csvWriter.writeheader()
|
||||
# Write matches to CSV
|
||||
for data in ruleResults["matches"]:
|
||||
dictCSV = {
|
||||
"rule_title": ruleResults["title"],
|
||||
"rule_description": ruleResults["description"],
|
||||
"rule_level": ruleResults["rule_level"],
|
||||
"rule_count": ruleResults["count"],
|
||||
**data,
|
||||
}
|
||||
csvWriter.writerow(dictCSV)
|
||||
else:
|
||||
# Write results as JSON using orjson
|
||||
try:
|
||||
# Handle commas between JSON objects
|
||||
if not first_json_output:
|
||||
fileHandle.write(",\n")
|
||||
else:
|
||||
first_json_output = False
|
||||
# Serialize ruleResults to JSON bytes with indentation
|
||||
json_bytes = json.dumps(
|
||||
ruleResults, option=json.OPT_INDENT_2
|
||||
)
|
||||
# Write the decoded JSON string to the file
|
||||
fileHandle.write(json_bytes.decode("utf-8"))
|
||||
except Exception as e:
|
||||
self.logger.error(f"Error saving some results: {e}")
|
||||
|
||||
# Close output file handle if needed
|
||||
if not self.noOutput:
|
||||
if is_json_mode and lastRuleset:
|
||||
fileHandle.write("]") # Close JSON array
|
||||
fileHandle.close()
|
||||
|
||||
def run(
|
||||
self,
|
||||
@@ -1074,7 +1213,7 @@ class zirCore:
|
||||
Insert2Db=True,
|
||||
saveToFile=False,
|
||||
forwarder=None,
|
||||
JSONArray=False,
|
||||
args_config=None,
|
||||
):
|
||||
self.logger.info("[+] Processing events")
|
||||
flattener = JSONFlattener(
|
||||
@@ -1083,7 +1222,7 @@ class zirCore:
|
||||
timeBefore=self.timeBefore,
|
||||
timeField=self.timeField,
|
||||
hashes=self.hashes,
|
||||
JSONArray=JSONArray,
|
||||
args_config=args_config,
|
||||
)
|
||||
flattener.runAll(EVTXJSONList)
|
||||
if saveToFile:
|
||||
@@ -1188,7 +1327,7 @@ class evtxExtractor:
|
||||
)
|
||||
except Exception as e:
|
||||
self.logger.error(
|
||||
f"{Fore.RED} [-] Cannot use PyEvtxParser{Fore.RESET}"
|
||||
f"{Fore.RED} [-] Cannot use PyEvtxParser : {e}{Fore.RESET}"
|
||||
)
|
||||
else:
|
||||
self.logger.error(
|
||||
@@ -1222,10 +1361,6 @@ class evtxExtractor:
|
||||
.replace('"', "")
|
||||
.split("=")
|
||||
)
|
||||
if "cmd" in attribute[0] or "proctitle" in attribute[0]:
|
||||
attribute[1] = str(
|
||||
bytearray.fromhex(attribute[1]).decode()
|
||||
).replace("\x00", " ")
|
||||
event[attribute[0]] = attribute[1].rstrip()
|
||||
except Exception:
|
||||
pass
|
||||
@@ -1927,60 +2062,62 @@ def main():
|
||||
default="9999-12-12T23:59:59",
|
||||
)
|
||||
# Event and log formats options
|
||||
# /!\ an option name containing '-input' must exists (It is used in JSON flattening mechanism)
|
||||
eventFormatsArgs = parser.add_mutually_exclusive_group()
|
||||
eventFormatsArgs.add_argument(
|
||||
"-j",
|
||||
"--json-input",
|
||||
"--jsononly",
|
||||
"--jsonline",
|
||||
"--jsonl",
|
||||
"--json-input",
|
||||
help="If logs files are already in JSON lines format ('jsonl' in evtx_dump) ",
|
||||
action="store_true",
|
||||
)
|
||||
eventFormatsArgs.add_argument(
|
||||
"--json-array-input",
|
||||
"--jsonarray",
|
||||
"--json-array",
|
||||
"--json-array-input",
|
||||
help="Source logs are in JSON but as an array",
|
||||
action="store_true",
|
||||
)
|
||||
eventFormatsArgs.add_argument(
|
||||
"--db-input",
|
||||
"-D",
|
||||
"--dbonly",
|
||||
"--db-input",
|
||||
help="Directly use a previously saved database file, timerange filters will not work",
|
||||
action="store_true",
|
||||
)
|
||||
eventFormatsArgs.add_argument(
|
||||
"-S",
|
||||
"--sysmon-linux-input",
|
||||
"--sysmon4linux",
|
||||
"--sysmon-linux",
|
||||
"--sysmon-linux-input",
|
||||
help="Use this option if your log file is a Sysmon for linux log file, default file extension is '.log'",
|
||||
action="store_true",
|
||||
)
|
||||
eventFormatsArgs.add_argument(
|
||||
"-AU",
|
||||
"--auditd",
|
||||
"--auditd-input",
|
||||
"--auditd",
|
||||
help="Use this option if your log file is a Auditd log file, default file extension is '.log'",
|
||||
action="store_true",
|
||||
)
|
||||
eventFormatsArgs.add_argument(
|
||||
"-x",
|
||||
"--xml",
|
||||
"--xml-input",
|
||||
"--xml",
|
||||
help="Use this option if your log file is a EVTX converted to XML log file, default file extension is '.xml'",
|
||||
action="store_true",
|
||||
)
|
||||
eventFormatsArgs.add_argument(
|
||||
"--evtxtract-input",
|
||||
"--evtxtract",
|
||||
help="Use this option if your log file was extracted with EVTXtract, default file extension is '.log'",
|
||||
action="store_true",
|
||||
)
|
||||
eventFormatsArgs.add_argument(
|
||||
"--csvonly",
|
||||
"--csv-input",
|
||||
"--csvonly",
|
||||
help="You log file is in CSV format '.csv'",
|
||||
action="store_true",
|
||||
)
|
||||
@@ -2286,7 +2423,7 @@ def main():
|
||||
consoleLogger.error(
|
||||
f"{Fore.RED} [-] No events source path provided. Use '-e <PATH TO LOGS>', '--events <PATH TO LOGS>'{Fore.RESET}"
|
||||
), sys.exit(2)
|
||||
if args.forwardall and args.dbonly:
|
||||
if args.forwardall and args.db_input:
|
||||
consoleLogger.error(
|
||||
f"{Fore.RED} [-] Can't forward all events in db only mode {Fore.RESET}"
|
||||
), sys.exit(2)
|
||||
@@ -2300,6 +2437,9 @@ def main():
|
||||
# Init Forwarding
|
||||
forwarder = None
|
||||
if args.remote is not None:
|
||||
consoleLogger.info(
|
||||
f"{Fore.LIGHTRED_EX}[!] Forwarding is not tested anymore and will be removed in the future{Fore.RESET}"
|
||||
)
|
||||
forwarder = eventForwarder(
|
||||
remote=args.remote,
|
||||
timeField=args.timefield,
|
||||
@@ -2380,16 +2520,16 @@ def main():
|
||||
)
|
||||
|
||||
# If we are not working directly with the db
|
||||
if not args.dbonly:
|
||||
if not args.db_input:
|
||||
# If we are working with json we change the file extension if it is not user-provided
|
||||
if not args.fileext:
|
||||
if args.jsononly or args.jsonarray:
|
||||
if args.json_input or args.json_array_input:
|
||||
args.fileext = "json"
|
||||
elif args.sysmon4linux or args.auditd:
|
||||
elif args.sysmon_linux_input or args.auditd_input:
|
||||
args.fileext = "log"
|
||||
elif args.xml:
|
||||
elif args.xml_input:
|
||||
args.fileext = "xml"
|
||||
elif args.csvonly:
|
||||
elif args.csv_input:
|
||||
args.fileext = "csv"
|
||||
else:
|
||||
args.fileext = "evtx"
|
||||
@@ -2422,7 +2562,7 @@ def main():
|
||||
consoleLogger,
|
||||
)
|
||||
|
||||
if not args.jsononly and not args.jsonarray:
|
||||
if not args.json_input and not args.json_array_input:
|
||||
# Init EVTX extractor object
|
||||
extractor = evtxExtractor(
|
||||
logger=consoleLogger,
|
||||
@@ -2430,12 +2570,12 @@ def main():
|
||||
coreCount=args.cores,
|
||||
useExternalBinaries=(not args.noexternal),
|
||||
binPath=args.evtx_dump,
|
||||
xmlLogs=args.xml,
|
||||
sysmon4linux=args.sysmon4linux,
|
||||
auditdLogs=args.auditd,
|
||||
evtxtract=args.evtxtract,
|
||||
xmlLogs=args.xml_input,
|
||||
sysmon4linux=args.sysmon_linux_input,
|
||||
auditdLogs=args.auditd_input,
|
||||
evtxtract=args.evtxtract_input,
|
||||
encoding=args.logs_encoding,
|
||||
csvInput=args.csvonly,
|
||||
csvInput=args.csv_input,
|
||||
)
|
||||
consoleLogger.info(
|
||||
f"[+] Extracting events Using '{extractor.tmpDir}' directory "
|
||||
@@ -2460,11 +2600,9 @@ def main():
|
||||
|
||||
# Print field list and exit
|
||||
if args.fieldlist:
|
||||
fields = zircoliteCore.run(
|
||||
LogJSONList, Insert2Db=False, JSONArray=args.jsonarray
|
||||
)
|
||||
fields = zircoliteCore.run(LogJSONList, Insert2Db=False, args_config=args)
|
||||
zircoliteCore.close()
|
||||
if not args.jsononly and not args.jsonarray and not args.keeptmp:
|
||||
if not args.json_input and not args.json_array_input and not args.keeptmp:
|
||||
extractor.cleanup()
|
||||
[
|
||||
print(sortedField)
|
||||
@@ -2478,12 +2616,10 @@ def main():
|
||||
LogJSONList,
|
||||
saveToFile=args.keepflat,
|
||||
forwarder=forwarder,
|
||||
JSONArray=args.jsonarray,
|
||||
args_config=args,
|
||||
)
|
||||
else:
|
||||
zircoliteCore.run(
|
||||
LogJSONList, saveToFile=args.keepflat, JSONArray=args.jsonarray
|
||||
)
|
||||
zircoliteCore.run(LogJSONList, saveToFile=args.keepflat, args_config=args)
|
||||
# Unload In memory DB to disk. Done here to allow debug in case of ruleset execution error
|
||||
if args.dbfile is not None:
|
||||
zircoliteCore.saveDbToDisk(args.dbfile)
|
||||
@@ -2552,7 +2688,7 @@ def main():
|
||||
if not args.keeptmp:
|
||||
consoleLogger.info("[+] Cleaning")
|
||||
try:
|
||||
if not args.jsononly and not args.jsonarray and not args.dbonly:
|
||||
if not args.json_input and not args.json_array_input and not args.db_input:
|
||||
extractor.cleanup()
|
||||
except OSError as e:
|
||||
consoleLogger.error(
|
||||
|
||||
442
zircolite_dev.py
442
zircolite_dev.py
@@ -3,6 +3,8 @@
|
||||
# Standard libs
|
||||
import argparse
|
||||
import asyncio
|
||||
import base64
|
||||
import chardet
|
||||
import csv
|
||||
import functools
|
||||
import hashlib
|
||||
@@ -29,6 +31,12 @@ import xxhash
|
||||
from colorama import Fore
|
||||
from tqdm import tqdm
|
||||
from tqdm.asyncio import tqdm as tqdmAsync
|
||||
from RestrictedPython import compile_restricted
|
||||
from RestrictedPython import safe_builtins
|
||||
from RestrictedPython import limited_builtins
|
||||
from RestrictedPython import utility_builtins
|
||||
from RestrictedPython.Eval import default_guarded_getiter
|
||||
from RestrictedPython.Guards import guarded_iter_unpack_sequence
|
||||
|
||||
# External libs (Optional)
|
||||
forwardingDisabled = False
|
||||
@@ -389,11 +397,11 @@ class eventForwarder:
|
||||
# Wait until all worker tasks are cancelled.
|
||||
await asyncio.gather(*tasks, return_exceptions=True)
|
||||
await session.close()
|
||||
|
||||
|
||||
class JSONFlattener:
|
||||
""" Perform JSON Flattening """
|
||||
|
||||
def __init__(self, configFile, logger=None, timeAfter="1970-01-01T00:00:00", timeBefore="9999-12-12T23:59:59", timeField=None, hashes=False, JSONArray=False):
|
||||
def __init__(self, configFile, logger=None, timeAfter="1970-01-01T00:00:00", timeBefore="9999-12-12T23:59:59", timeField=None, hashes=False, args_config=None):
|
||||
self.logger = logger or logging.getLogger(__name__)
|
||||
self.keyDict = {}
|
||||
self.fieldStmt = ""
|
||||
@@ -402,7 +410,17 @@ class JSONFlattener:
|
||||
self.timeBefore = timeBefore
|
||||
self.timeField = timeField
|
||||
self.hashes = hashes
|
||||
self.JSONArray = JSONArray
|
||||
self.args_config = args_config
|
||||
self.JSONArray = args_config.json_array_input
|
||||
# Initialize the cache for compiled code
|
||||
self.compiled_code_cache = {}
|
||||
|
||||
# Convert the argparse.Namespace to a dictionary
|
||||
args_dict = vars(args_config)
|
||||
# Find the chosen input format
|
||||
self.chosen_input = next((key for key, value in args_dict.items() if "_input" in key and value), None)
|
||||
if self.chosen_input is None:
|
||||
self.chosen_input = "evtx_input" # Since evtx is the default input, we force it no chosen input has been found
|
||||
|
||||
with open(configFile, 'r', encoding='UTF-8') as fieldMappingsFile:
|
||||
self.fieldMappingsDict = json.loads(fieldMappingsFile.read())
|
||||
@@ -411,6 +429,28 @@ class JSONFlattener:
|
||||
self.uselessValues = self.fieldMappingsDict["useless"]
|
||||
self.aliases = self.fieldMappingsDict["alias"]
|
||||
self.fieldSplitList = self.fieldMappingsDict["split"]
|
||||
self.transforms = self.fieldMappingsDict["transforms"]
|
||||
self.transforms_enabled = self.fieldMappingsDict["transforms_enabled"]
|
||||
|
||||
# Define the authorized BUILTINS for Resticted Python
|
||||
def default_guarded_getitem(ob, index):
|
||||
return ob[index]
|
||||
|
||||
default_guarded_getattr = getattr
|
||||
|
||||
self.RestrictedPython_BUILTINS = {
|
||||
'__name__': 'script',
|
||||
"_getiter_": default_guarded_getiter,
|
||||
'_getattr_': default_guarded_getattr,
|
||||
'_getitem_': default_guarded_getitem,
|
||||
'base64': base64,
|
||||
're': re,
|
||||
'chardet': chardet,
|
||||
'_iter_unpack_sequence_': guarded_iter_unpack_sequence
|
||||
}
|
||||
self.RestrictedPython_BUILTINS.update(safe_builtins)
|
||||
self.RestrictedPython_BUILTINS.update(limited_builtins)
|
||||
self.RestrictedPython_BUILTINS.update(utility_builtins)
|
||||
|
||||
def run(self, file):
|
||||
"""
|
||||
@@ -422,6 +462,23 @@ class JSONFlattener:
|
||||
JSONOutput = []
|
||||
fieldStmt = ""
|
||||
|
||||
def transformValue(code, param):
|
||||
try:
|
||||
# Check if the code has already been compiled
|
||||
if code in self.compiled_code_cache:
|
||||
byte_code = self.compiled_code_cache[code]
|
||||
else:
|
||||
# Compile the code and store it in the cache
|
||||
byte_code = compile_restricted(code, filename='<inline code>', mode='exec')
|
||||
self.compiled_code_cache[code] = byte_code
|
||||
# Prepare the execution environment
|
||||
TransformFunction = {}
|
||||
exec(byte_code, self.RestrictedPython_BUILTINS, TransformFunction)
|
||||
return TransformFunction["transform"](param)
|
||||
except Exception as e:
|
||||
self.logger.debug(f"ERROR: Couldn't apply transform: {e}")
|
||||
return param # Return the original parameter if transform fails
|
||||
|
||||
def flatten(x, name=''):
|
||||
nonlocal fieldStmt
|
||||
# If it is a Dict go deeper
|
||||
@@ -438,6 +495,7 @@ class JSONFlattener:
|
||||
value = x
|
||||
# Excluding useless values (e.g. "null"). The value must be an exact match.
|
||||
if value not in self.uselessValues:
|
||||
|
||||
# Applying field mappings
|
||||
rawFieldName = name[:-1]
|
||||
if rawFieldName in self.fieldMappings:
|
||||
@@ -446,12 +504,28 @@ class JSONFlattener:
|
||||
# Removing all annoying character from field name
|
||||
key = ''.join(e for e in rawFieldName.split(".")[-1] if e.isalnum())
|
||||
|
||||
# Preparing aliases
|
||||
# Preparing aliases (work on original field name and Mapped field name)
|
||||
keys = [key]
|
||||
if key in self.aliases:
|
||||
keys.append(self.aliases[key])
|
||||
if rawFieldName in self.aliases:
|
||||
keys.append(self.aliases[rawFieldName])
|
||||
for fieldName in [key, rawFieldName]:
|
||||
if fieldName in self.aliases:
|
||||
keys.append(self.aliases[key])
|
||||
|
||||
# Applying field transforms (work on original field name and Mapped field name)
|
||||
keysThatNeedTransformedValues = []
|
||||
transformedValuesByKeys = {}
|
||||
if self.transforms_enabled:
|
||||
for fieldName in [key, rawFieldName]:
|
||||
if fieldName in self.transforms:
|
||||
for transform in self.transforms[fieldName]:
|
||||
if transform["enabled"] and self.chosen_input in transform["source_condition"] :
|
||||
transformCode = transform["code"]
|
||||
# If the transform rule ask for a dedicated alias
|
||||
if transform["alias"]:
|
||||
keys.append(transform["alias_name"])
|
||||
keysThatNeedTransformedValues.append(transform["alias_name"])
|
||||
transformedValuesByKeys[transform["alias_name"]] = transformValue(transformCode, value)
|
||||
else:
|
||||
value = transformValue(transformCode, value)
|
||||
|
||||
# Applying field splitting
|
||||
fieldsToSplit = []
|
||||
@@ -476,16 +550,19 @@ class JSONFlattener:
|
||||
|
||||
# Applying aliases
|
||||
for key in keys:
|
||||
JSONLine[key] = value
|
||||
if key in keysThatNeedTransformedValues:
|
||||
JSONLine[key] = transformedValuesByKeys[key]
|
||||
else:
|
||||
JSONLine[key] = value
|
||||
# Creating the CREATE TABLE SQL statement
|
||||
keyLower =key.lower()
|
||||
keyLower = key.lower()
|
||||
if keyLower not in self.keyDict:
|
||||
self.keyDict[keyLower] = key
|
||||
if isinstance(value, int):
|
||||
fieldStmt += f"'{key}' INTEGER,\n"
|
||||
else:
|
||||
fieldStmt += f"'{key}' TEXT COLLATE NOCASE,\n"
|
||||
|
||||
|
||||
# If filesize is not zero
|
||||
if os.stat(file).st_size != 0:
|
||||
with open(str(file), 'r', encoding='utf-8') as JSONFile:
|
||||
@@ -556,7 +633,13 @@ class zirCore:
|
||||
conn = None
|
||||
self.logger.debug(f"CONNECTING TO : {db}")
|
||||
try:
|
||||
conn = sqlite3.connect(db)
|
||||
if db == ':memory:':
|
||||
conn = sqlite3.connect(db, isolation_level=None)
|
||||
conn.execute('PRAGMA journal_mode = MEMORY;')
|
||||
conn.execute('PRAGMA synchronous = OFF;')
|
||||
conn.execute('PRAGMA temp_store = MEMORY;')
|
||||
else:
|
||||
conn = sqlite3.connect(db)
|
||||
conn.row_factory = sqlite3.Row # Allows to get a dict
|
||||
|
||||
def udf_regex(x, y):
|
||||
@@ -574,7 +657,7 @@ class zirCore:
|
||||
|
||||
def createDb(self, fieldStmt):
|
||||
createTableStmt = f"CREATE TABLE logs ( row_id INTEGER, {fieldStmt} PRIMARY KEY(row_id AUTOINCREMENT) );"
|
||||
self.logger.debug(" CREATE : " + createTableStmt.replace('\n', ' ').replace('\r', ''))
|
||||
self.logger.debug(f" CREATE : {createTableStmt}")
|
||||
if not self.executeQuery(createTableStmt):
|
||||
self.logger.error(f"{Fore.RED} [-] Unable to create table{Fore.RESET}")
|
||||
sys.exit(1)
|
||||
@@ -599,19 +682,23 @@ class zirCore:
|
||||
return False
|
||||
|
||||
def executeSelectQuery(self, query):
|
||||
""" Perform a SQL Query -SELECT only- with the provided connection """
|
||||
if self.dbConnection is not None:
|
||||
dbHandle = self.dbConnection.cursor()
|
||||
self.logger.debug(f"EXECUTING : {query}")
|
||||
try:
|
||||
data = dbHandle.execute(query)
|
||||
return data
|
||||
except Error as e:
|
||||
self.logger.debug(f" [-] {e}")
|
||||
return {}
|
||||
else:
|
||||
"""
|
||||
Execute a SELECT SQL query and return the results as a list of dictionaries.
|
||||
"""
|
||||
if self.dbConnection is None:
|
||||
self.logger.error(f"{Fore.RED} [-] No connection to Db{Fore.RESET}")
|
||||
return {}
|
||||
return []
|
||||
try:
|
||||
cursor = self.dbConnection.cursor()
|
||||
self.logger.debug(f"Executing SELECT query: {query}")
|
||||
cursor.execute(query)
|
||||
rows = cursor.fetchall()
|
||||
# Convert rows to list of dictionaries
|
||||
result = [dict(row) for row in rows]
|
||||
return result
|
||||
except sqlite3.Error as e:
|
||||
self.logger.debug(f" [-] SQL query error: {e}")
|
||||
return []
|
||||
|
||||
def loadDbInMemory(self, db):
|
||||
""" In db only mode it is possible to restore an on disk Db to avoid EVTX extraction and flattening """
|
||||
@@ -619,20 +706,30 @@ class zirCore:
|
||||
dbfileConnection.backup(self.dbConnection)
|
||||
dbfileConnection.close()
|
||||
|
||||
def escape_identifier(self, identifier):
|
||||
"""Escape SQL identifiers like table or column names."""
|
||||
return identifier.replace("\"", "\"\"")
|
||||
|
||||
def insertData2Db(self, JSONLine):
|
||||
""" Build INSERT INTO Query and insert data into Db """
|
||||
columnsStr = ""
|
||||
valuesStr = ""
|
||||
|
||||
for key in sorted(JSONLine.keys()):
|
||||
columnsStr += "'" + key + "',"
|
||||
if isinstance(JSONLine[key], int):
|
||||
valuesStr += str(JSONLine[key]) + ", "
|
||||
else:
|
||||
valuesStr += "'" + str(JSONLine[key]).replace("'", "''") + "', "
|
||||
|
||||
insertStrmt = f"INSERT INTO logs ({columnsStr[:-1]}) VALUES ({valuesStr[:-2]});"
|
||||
return self.executeQuery(insertStrmt)
|
||||
"""Build a parameterized INSERT INTO query and insert data into the database."""
|
||||
columns = JSONLine.keys()
|
||||
columnsEscaped = ', '.join([self.escape_identifier(col) for col in columns])
|
||||
placeholders = ', '.join(['?'] * len(columns))
|
||||
values = []
|
||||
for col in columns:
|
||||
value = JSONLine[col]
|
||||
if isinstance(value, int):
|
||||
# Check if value exceeds SQLite INTEGER limits
|
||||
if abs(value) > 9223372036854775807:
|
||||
value = str(value) # Convert to string
|
||||
values.append(value)
|
||||
insertStmt = f'INSERT INTO logs ({columnsEscaped}) VALUES ({placeholders})'
|
||||
try:
|
||||
self.dbConnection.execute(insertStmt, values)
|
||||
return True
|
||||
except Exception as e:
|
||||
self.logger.debug(f" [-] {e}")
|
||||
return False
|
||||
|
||||
def insertFlattenedJSON2Db(self, flattenedJSON, forwarder=None):
|
||||
if forwarder:
|
||||
@@ -644,7 +741,7 @@ class zirCore:
|
||||
def saveFlattenedJSON2File(self, flattenedJSON, outputFile):
|
||||
with open(outputFile, 'w', encoding='utf-8') as file:
|
||||
for JSONLine in tqdm(flattenedJSON, colour="yellow"):
|
||||
file.write(json.dumps(JSONLine).decode('utf-8') + '\n')
|
||||
file.write(f'{json.dumps(JSONLine).decode("utf-8")}\n')
|
||||
|
||||
def saveDbToDisk(self, dbFilename):
|
||||
self.logger.info("[+] Saving working data to disk as a SQLite DB")
|
||||
@@ -653,41 +750,58 @@ class zirCore:
|
||||
onDiskDb.close()
|
||||
|
||||
def executeRule(self, rule):
|
||||
results = {}
|
||||
filteredRows = []
|
||||
counter = 0
|
||||
if "rule" in rule:
|
||||
# for each SQL Query in the Sigma rule
|
||||
for SQLQuery in rule["rule"]:
|
||||
data = self.executeSelectQuery(SQLQuery)
|
||||
if data != {}:
|
||||
# Convert to array of dict
|
||||
rows = [dict(row) for row in data.fetchall()]
|
||||
if len(rows) > 0:
|
||||
counter += len(rows)
|
||||
for row in rows:
|
||||
if self.csvMode: # Cleaning "annoying" values for CSV
|
||||
match = {k: str(v).replace("\n","").replace("\r","").replace("None","") for k, v in row.items()}
|
||||
else: # Cleaning null/None fields
|
||||
match = {k: v for k, v in row.items() if v is not None}
|
||||
filteredRows.append(match)
|
||||
if "level" not in rule:
|
||||
rule["level"] = "unknown"
|
||||
if "tags" not in rule:
|
||||
rule["tags"] = []
|
||||
if "filename" not in rule:
|
||||
rule["filename"] = ""
|
||||
if self.csvMode:
|
||||
results = ({"title": rule["title"], "id": rule["id"], "description": rule["description"].replace("\n","").replace("\r",""), "sigmafile": rule["filename"], "sigma": rule["rule"], "rule_level": rule["level"], "tags": rule["tags"], "count": counter, "matches": filteredRows})
|
||||
else:
|
||||
results = ({"title": rule["title"], "id": rule["id"], "description": rule["description"], "sigmafile": rule["filename"], "sigma": rule["rule"], "rule_level": rule["level"], "tags": rule["tags"], "count": counter, "matches": filteredRows})
|
||||
if counter > 0:
|
||||
self.logger.debug(f'DETECTED : {rule["title"]} - Matches : {counter} events')
|
||||
else:
|
||||
self.logger.debug("RULE FORMAT ERROR : rule key Missing")
|
||||
if filteredRows == []:
|
||||
"""
|
||||
Execute a single Sigma rule against the database and return the results.
|
||||
"""
|
||||
if "rule" not in rule:
|
||||
self.logger.debug("RULE FORMAT ERROR: 'rule' key missing")
|
||||
return {}
|
||||
|
||||
# Set default values for missing rule keys
|
||||
rule_level = rule.get("level", "unknown")
|
||||
tags = rule.get("tags", [])
|
||||
filename = rule.get("filename", "")
|
||||
description = rule.get("description", "")
|
||||
title = rule.get("title", "Unnamed Rule")
|
||||
rule_id = rule.get("id", "")
|
||||
sigma_queries = rule["rule"]
|
||||
|
||||
filteredRows = []
|
||||
|
||||
# Process each SQL query in the rule
|
||||
for SQLQuery in sigma_queries:
|
||||
data = self.executeSelectQuery(SQLQuery)
|
||||
if data:
|
||||
if self.csvMode:
|
||||
# Clean values for CSV output
|
||||
cleaned_rows = [
|
||||
{k: str(v).replace("\n", "").replace("\r", "").replace("None", "") for k, v in dict(row).items()}
|
||||
for row in data
|
||||
]
|
||||
else:
|
||||
# Remove None values
|
||||
cleaned_rows = [
|
||||
{k: v for k, v in dict(row).items() if v is not None}
|
||||
for row in data
|
||||
]
|
||||
filteredRows.extend(cleaned_rows)
|
||||
|
||||
if filteredRows:
|
||||
results = {
|
||||
"title": title,
|
||||
"id": rule_id,
|
||||
"description": description.replace("\n", "").replace("\r", "") if self.csvMode else description,
|
||||
"sigmafile": filename,
|
||||
"sigma": sigma_queries,
|
||||
"rule_level": rule_level,
|
||||
"tags": tags,
|
||||
"count": len(filteredRows),
|
||||
"matches": filteredRows
|
||||
}
|
||||
self.logger.debug(f'DETECTED: {title} - Matches: {len(filteredRows)} events')
|
||||
return results
|
||||
else:
|
||||
return {}
|
||||
return results
|
||||
|
||||
def loadRulesetFromFile(self, filename, ruleFilters):
|
||||
try:
|
||||
@@ -719,51 +833,99 @@ class zirCore:
|
||||
if level == "critical":
|
||||
return f'{Fore.RED}{level}{orgFormat}'
|
||||
|
||||
def executeRuleset(self, outFile, writeMode='w', forwarder=None, showAll=False, KeepResults=False, remote=None, stream=False, lastRuleset=False):
|
||||
def executeRuleset(self, outFile, writeMode='w', forwarder=None, showAll=False,
|
||||
KeepResults=False, remote=None, stream=False, lastRuleset=False):
|
||||
"""
|
||||
Execute all rules in the ruleset and handle output.
|
||||
"""
|
||||
csvWriter = None
|
||||
# Results are written upon detection to allow analysis during execution and to avoid losing results in case of error.
|
||||
with open(outFile, writeMode, encoding='utf-8', newline='') as fileHandle:
|
||||
with tqdm(self.ruleset, colour="yellow") as ruleBar:
|
||||
if not self.noOutput and not self.csvMode and writeMode != "a":
|
||||
fileHandle.write('[')
|
||||
for rule in ruleBar: # for each rule in ruleset
|
||||
if showAll and "title" in rule:
|
||||
ruleBar.write(f'{Fore.BLUE} - {rule["title"]} [{self.ruleLevelPrintFormatter(rule["level"], Fore.BLUE)}]{Fore.RESET}') # Print all rules
|
||||
ruleResults = self.executeRule(rule)
|
||||
if ruleResults != {}:
|
||||
if self.limit == -1 or ruleResults["count"] <= self.limit:
|
||||
ruleBar.write(f'{Fore.CYAN} - {ruleResults["title"]} [{self.ruleLevelPrintFormatter(rule["level"], Fore.CYAN)}] : {ruleResults["count"]} events{Fore.RESET}')
|
||||
# Store results for templating and event forwarding (only if stream mode is disabled)
|
||||
if KeepResults or (remote is not None and not stream):
|
||||
self.fullResults.append(ruleResults)
|
||||
if stream and forwarder is not None:
|
||||
forwarder.send([ruleResults], False)
|
||||
if not self.noOutput:
|
||||
# To avoid printing this twice on stdout but in the logs...
|
||||
logLevel = self.logger.getEffectiveLevel()
|
||||
self.logger.setLevel(logging.DEBUG)
|
||||
self.logger.debug(f' - {ruleResults["title"]} [{ruleResults["rule_level"]}] : {ruleResults["count"]} events')
|
||||
self.logger.setLevel(logLevel)
|
||||
# Output to json or csv file
|
||||
if self.csvMode:
|
||||
if not csvWriter: # Creating the CSV header and the fields ("agg" is for queries with aggregation)
|
||||
csvWriter = csv.DictWriter(fileHandle, delimiter=self.delimiter, fieldnames=["rule_title", "rule_description", "rule_level", "rule_count", "agg"] + list(ruleResults["matches"][0].keys()))
|
||||
csvWriter.writeheader()
|
||||
for data in ruleResults["matches"]:
|
||||
dictCSV = { "rule_title": ruleResults["title"], "rule_description": ruleResults["description"], "rule_level": ruleResults["rule_level"], "rule_count": ruleResults["count"], **data}
|
||||
csvWriter.writerow(dictCSV)
|
||||
else:
|
||||
try:
|
||||
fileHandle.write(json.dumps(ruleResults, option=json.OPT_INDENT_2).decode('utf-8'))
|
||||
fileHandle.write(',\n')
|
||||
except Exception as e:
|
||||
self.logger.error(f"{Fore.RED} [-] Error saving some results : {e}{Fore.RESET}")
|
||||
if (not self.noOutput and not self.csvMode) and lastRuleset:
|
||||
fileHandle.write('{}]') # Added to produce a valid JSON Array
|
||||
first_json_output = True # To manage commas in JSON output
|
||||
is_json_mode = not self.csvMode
|
||||
|
||||
def run(self, EVTXJSONList, Insert2Db=True, saveToFile=False, forwarder=None, JSONArray=False):
|
||||
# Prepare output file handle if needed
|
||||
fileHandle = None
|
||||
if not self.noOutput:
|
||||
# Open file in text mode since we will write decoded strings
|
||||
fileHandle = open(outFile, writeMode, encoding='utf-8', newline='')
|
||||
if is_json_mode and writeMode != 'a':
|
||||
fileHandle.write('[') # Start JSON array
|
||||
|
||||
# Iterate over rules in the ruleset
|
||||
with tqdm(self.ruleset, colour="yellow") as ruleBar:
|
||||
for rule in ruleBar:
|
||||
# Show all rules if showAll is True
|
||||
if showAll and "title" in rule:
|
||||
rule_title = rule["title"]
|
||||
rule_level = rule.get("level", "unknown")
|
||||
formatted_level = self.ruleLevelPrintFormatter(rule_level, Fore.BLUE)
|
||||
ruleBar.write(f'{Fore.BLUE} - {rule_title} [{formatted_level}]{Fore.RESET}')
|
||||
|
||||
# Execute the rule
|
||||
ruleResults = self.executeRule(rule)
|
||||
if not ruleResults:
|
||||
continue # No matches, skip to next rule
|
||||
|
||||
# Apply limit if set
|
||||
if self.limit != -1 and ruleResults["count"] > self.limit:
|
||||
continue # Exceeds limit, skip this result
|
||||
|
||||
# Write progress message
|
||||
rule_title = ruleResults["title"]
|
||||
rule_level = ruleResults.get("rule_level", "unknown")
|
||||
formatted_level = self.ruleLevelPrintFormatter(rule_level, Fore.CYAN)
|
||||
rule_count = ruleResults["count"]
|
||||
ruleBar.write(f'{Fore.CYAN} - {rule_title} [{formatted_level}] : {rule_count} events{Fore.RESET}')
|
||||
|
||||
# Store results if needed
|
||||
if KeepResults or (remote and not stream):
|
||||
self.fullResults.append(ruleResults)
|
||||
|
||||
# Forward results if streaming
|
||||
if stream and forwarder:
|
||||
forwarder.send([ruleResults], False)
|
||||
|
||||
# Handle output to file
|
||||
if not self.noOutput:
|
||||
if self.csvMode:
|
||||
# Initialize CSV writer if not already done
|
||||
if csvWriter is None:
|
||||
fieldnames = ["rule_title", "rule_description", "rule_level", "rule_count"] + list(ruleResults["matches"][0].keys())
|
||||
csvWriter = csv.DictWriter(fileHandle, delimiter=self.delimiter, fieldnames=fieldnames)
|
||||
csvWriter.writeheader()
|
||||
# Write matches to CSV
|
||||
for data in ruleResults["matches"]:
|
||||
dictCSV = {
|
||||
"rule_title": ruleResults["title"],
|
||||
"rule_description": ruleResults["description"],
|
||||
"rule_level": ruleResults["rule_level"],
|
||||
"rule_count": ruleResults["count"],
|
||||
**data
|
||||
}
|
||||
csvWriter.writerow(dictCSV)
|
||||
else:
|
||||
# Write results as JSON using orjson
|
||||
try:
|
||||
# Handle commas between JSON objects
|
||||
if not first_json_output:
|
||||
fileHandle.write(',\n')
|
||||
else:
|
||||
first_json_output = False
|
||||
# Serialize ruleResults to JSON bytes with indentation
|
||||
json_bytes = json.dumps(ruleResults, option=json.OPT_INDENT_2)
|
||||
# Write the decoded JSON string to the file
|
||||
fileHandle.write(json_bytes.decode('utf-8'))
|
||||
except Exception as e:
|
||||
self.logger.error(f"Error saving some results: {e}")
|
||||
|
||||
# Close output file handle if needed
|
||||
if not self.noOutput:
|
||||
if is_json_mode and lastRuleset:
|
||||
fileHandle.write(']') # Close JSON array
|
||||
fileHandle.close()
|
||||
|
||||
def run(self, EVTXJSONList, Insert2Db=True, saveToFile=False, forwarder=None, args_config=None):
|
||||
self.logger.info("[+] Processing events")
|
||||
flattener = JSONFlattener(configFile=self.config, timeAfter=self.timeAfter, timeBefore=self.timeBefore, timeField=self.timeField, hashes=self.hashes, JSONArray=JSONArray)
|
||||
flattener = JSONFlattener(configFile=self.config, timeAfter=self.timeAfter, timeBefore=self.timeBefore, timeField=self.timeField, hashes=self.hashes, args_config=args_config)
|
||||
flattener.runAll(EVTXJSONList)
|
||||
if saveToFile:
|
||||
filename = f"flattened_events_{''.join(random.SystemRandom().choice(string.ascii_uppercase + string.digits) for _ in range(4))}.json"
|
||||
@@ -837,7 +999,7 @@ class evtxExtractor:
|
||||
for record in parser.records_json():
|
||||
f.write(f'{json.dumps(json.loads(record["data"])).decode("utf-8")}\n')
|
||||
except Exception as e:
|
||||
self.logger.error(f"{Fore.RED} [-] Cannot use PyEvtxParser{Fore.RESET}")
|
||||
self.logger.error(f"{Fore.RED} [-] Cannot use PyEvtxParser : {e}{Fore.RESET}")
|
||||
else:
|
||||
self.logger.error(f"{Fore.RED} [-] Cannot use PyEvtxParser and evtx_dump is disabled or missing{Fore.RESET}")
|
||||
|
||||
@@ -861,8 +1023,6 @@ class evtxExtractor:
|
||||
else:
|
||||
try:
|
||||
attribute = attribute.replace('msg=','').replace('\'','').replace('"','').split('=')
|
||||
if 'cmd' in attribute[0] or 'proctitle' in attribute[0]:
|
||||
attribute[1] = str(bytearray.fromhex(attribute[1]).decode()).replace('\x00',' ')
|
||||
event[attribute[0]] = attribute[1].rstrip()
|
||||
except Exception:
|
||||
pass
|
||||
@@ -1363,15 +1523,16 @@ def main():
|
||||
eventArgs.add_argument("-A", "--after", help="Limit to events that happened after the provided timestamp (UTC). Format : 1970-01-01T00:00:00", type=str, default="1970-01-01T00:00:00")
|
||||
eventArgs.add_argument("-B", "--before", help="Limit to events that happened before the provided timestamp (UTC). Format : 1970-01-01T00:00:00", type=str, default="9999-12-12T23:59:59")
|
||||
# Event and log formats options
|
||||
eventFormatsArgs = parser.add_mutually_exclusive_group()
|
||||
eventFormatsArgs.add_argument("-j", "--jsononly", "--jsonline", "--jsonl", "--json-input", help="If logs files are already in JSON lines format ('jsonl' in evtx_dump) ", action='store_true')
|
||||
eventFormatsArgs.add_argument("--jsonarray", "--json-array", "--json-array-input", help="Source logs are in JSON but as an array", action='store_true')
|
||||
eventFormatsArgs.add_argument("-D", "--dbonly", "--db-input", help="Directly use a previously saved database file, timerange filters will not work", action='store_true')
|
||||
eventFormatsArgs.add_argument("-S", "--sysmon4linux", "--sysmon-linux", "--sysmon-linux-input", help="Use this option if your log file is a Sysmon for linux log file, default file extension is '.log'", action='store_true')
|
||||
eventFormatsArgs.add_argument("-AU", "--auditd", "--auditd-input", help="Use this option if your log file is a Auditd log file, default file extension is '.log'", action='store_true')
|
||||
eventFormatsArgs.add_argument("-x", "--xml", "--xml-input", help="Use this option if your log file is a EVTX converted to XML log file, default file extension is '.xml'", action='store_true')
|
||||
eventFormatsArgs.add_argument("--evtxtract", help="Use this option if your log file was extracted with EVTXtract, default file extension is '.log'", action='store_true')
|
||||
eventFormatsArgs.add_argument("--csvonly" ,"--csv-input", help="You log file is in CSV format '.csv'", action='store_true')
|
||||
# /!\ an option name containing '-input' must exists (It is used in JSON flattening mechanism)
|
||||
eventFormatsArgs = parser.add_mutually_exclusive_group()
|
||||
eventFormatsArgs.add_argument("-j", "--json-input", "--jsononly", "--jsonline", "--jsonl", help="If logs files are already in JSON lines format ('jsonl' in evtx_dump) ", action='store_true')
|
||||
eventFormatsArgs.add_argument("--json-array-input", "--jsonarray", "--json-array", help="Source logs are in JSON but as an array", action='store_true')
|
||||
eventFormatsArgs.add_argument("--db-input", "-D", "--dbonly", help="Directly use a previously saved database file, timerange filters will not work", action='store_true')
|
||||
eventFormatsArgs.add_argument("-S", "--sysmon-linux-input", "--sysmon4linux", "--sysmon-linux", help="Use this option if your log file is a Sysmon for linux log file, default file extension is '.log'", action='store_true')
|
||||
eventFormatsArgs.add_argument("-AU", "--auditd-input", "--auditd", help="Use this option if your log file is a Auditd log file, default file extension is '.log'", action='store_true')
|
||||
eventFormatsArgs.add_argument("-x", "--xml-input", "--xml", help="Use this option if your log file is a EVTX converted to XML log file, default file extension is '.xml'", action='store_true')
|
||||
eventFormatsArgs.add_argument("--evtxtract-input", "--evtxtract", help="Use this option if your log file was extracted with EVTXtract, default file extension is '.log'", action='store_true')
|
||||
eventFormatsArgs.add_argument("--csv-input", "--csvonly", help="You log file is in CSV format '.csv'", action='store_true')
|
||||
# Ruleset options
|
||||
rulesetsFormatsArgs = parser.add_argument_group(f'{Fore.BLUE}RULES AND RULESETS OPTIONS{Fore.RESET}')
|
||||
rulesetsFormatsArgs.add_argument("-r", "--ruleset", help="Sigma ruleset : JSON (Zircolite format) or YAML/Directory containing YAML files (Native Sigma format)", action='append', nargs='+')
|
||||
@@ -1480,7 +1641,7 @@ def main():
|
||||
# Check mandatory CLI options
|
||||
if not args.evtx:
|
||||
consoleLogger.error(f"{Fore.RED} [-] No events source path provided. Use '-e <PATH TO LOGS>', '--events <PATH TO LOGS>'{Fore.RESET}"), sys.exit(2)
|
||||
if args.forwardall and args.dbonly:
|
||||
if args.forwardall and args.db_input:
|
||||
consoleLogger.error(f"{Fore.RED} [-] Can't forward all events in db only mode {Fore.RESET}"), sys.exit(2)
|
||||
if args.csv and len(args.ruleset) > 1:
|
||||
consoleLogger.error(f"{Fore.RED} [-] Since fields in results can change between rulesets, it is not possible to have CSV output when using multiple rulesets{Fore.RESET}"), sys.exit(2)
|
||||
@@ -1490,6 +1651,7 @@ def main():
|
||||
# Init Forwarding
|
||||
forwarder = None
|
||||
if args.remote is not None:
|
||||
consoleLogger.info(f"{Fore.LIGHTRED_EX}[!] Forwarding is not tested anymore and will be removed in the future{Fore.RESET}")
|
||||
forwarder = eventForwarder(remote=args.remote, timeField=args.timefield, token=args.token, logger=consoleLogger, index=args.index, login=args.eslogin, password=args.espass)
|
||||
if not forwarder.networkCheck():
|
||||
quitOnError(f"{Fore.RED} [-] Remote host cannot be reached : {args.remote}{Fore.RESET}", consoleLogger)
|
||||
@@ -1529,16 +1691,16 @@ def main():
|
||||
zircoliteCore = zirCore(args.config, logger=consoleLogger, noOutput=args.nolog, timeAfter=eventsAfter, timeBefore=eventsBefore, limit=args.limit, csvMode=args.csv, timeField=args.timefield, hashes=args.hashes, dbLocation=args.ondiskdb, delimiter=args.csv_delimiter)
|
||||
|
||||
# If we are not working directly with the db
|
||||
if not args.dbonly:
|
||||
if not args.db_input:
|
||||
# If we are working with json we change the file extension if it is not user-provided
|
||||
if not args.fileext:
|
||||
if args.jsononly or args.jsonarray:
|
||||
if args.json_input or args.json_array_input:
|
||||
args.fileext = "json"
|
||||
elif (args.sysmon4linux or args.auditd):
|
||||
elif (args.sysmon_linux_input or args.auditd_input):
|
||||
args.fileext = "log"
|
||||
elif args.xml:
|
||||
elif args.xml_input:
|
||||
args.fileext = "xml"
|
||||
elif args.csvonly:
|
||||
elif args.csv_input:
|
||||
args.fileext = "csv"
|
||||
else:
|
||||
args.fileext = "evtx"
|
||||
@@ -1565,9 +1727,9 @@ def main():
|
||||
if len(FileList) <= 0:
|
||||
quitOnError(f"{Fore.RED} [-] No file found. Please verify filters, directory or the extension with '--fileext' or '--file-pattern'{Fore.RESET}", consoleLogger)
|
||||
|
||||
if not args.jsononly and not args.jsonarray:
|
||||
if not args.json_input and not args.json_array_input:
|
||||
# Init EVTX extractor object
|
||||
extractor = evtxExtractor(logger=consoleLogger, providedTmpDir=args.tmpdir, coreCount=args.cores, useExternalBinaries=(not args.noexternal), binPath=args.evtx_dump, xmlLogs=args.xml, sysmon4linux=args.sysmon4linux, auditdLogs=args.auditd, evtxtract=args.evtxtract, encoding=args.logs_encoding, csvInput=args.csvonly)
|
||||
extractor = evtxExtractor(logger=consoleLogger, providedTmpDir=args.tmpdir, coreCount=args.cores, useExternalBinaries=(not args.noexternal), binPath=args.evtx_dump, xmlLogs=args.xml_input, sysmon4linux=args.sysmon_linux_input, auditdLogs=args.auditd_input, evtxtract=args.evtxtract_input, encoding=args.logs_encoding, csvInput=args.csv_input)
|
||||
consoleLogger.info(f"[+] Extracting events Using '{extractor.tmpDir}' directory ")
|
||||
for evtx in tqdm(FileList, colour="yellow"):
|
||||
extractor.run(evtx)
|
||||
@@ -1582,18 +1744,18 @@ def main():
|
||||
|
||||
# Print field list and exit
|
||||
if args.fieldlist:
|
||||
fields = zircoliteCore.run(LogJSONList, Insert2Db=False, JSONArray=args.jsonarray)
|
||||
fields = zircoliteCore.run(LogJSONList, Insert2Db=False, args_config=args)
|
||||
zircoliteCore.close()
|
||||
if not args.jsononly and not args.jsonarray and not args.keeptmp:
|
||||
if not args.json_input and not args.json_array_input and not args.keeptmp:
|
||||
extractor.cleanup()
|
||||
[print(sortedField) for sortedField in sorted([field for field in fields.values()])]
|
||||
sys.exit(0)
|
||||
|
||||
# Flatten and insert to Db
|
||||
if args.forwardall:
|
||||
zircoliteCore.run(LogJSONList, saveToFile=args.keepflat, forwarder=forwarder, JSONArray=args.jsonarray)
|
||||
zircoliteCore.run(LogJSONList, saveToFile=args.keepflat, forwarder=forwarder, args_config=args)
|
||||
else:
|
||||
zircoliteCore.run(LogJSONList, saveToFile=args.keepflat, JSONArray=args.jsonarray)
|
||||
zircoliteCore.run(LogJSONList, saveToFile=args.keepflat, args_config=args)
|
||||
# Unload In memory DB to disk. Done here to allow debug in case of ruleset execution error
|
||||
if args.dbfile is not None:
|
||||
zircoliteCore.saveDbToDisk(args.dbfile)
|
||||
@@ -1638,7 +1800,7 @@ def main():
|
||||
if not args.keeptmp:
|
||||
consoleLogger.info("[+] Cleaning")
|
||||
try:
|
||||
if not args.jsononly and not args.jsonarray and not args.dbonly:
|
||||
if not args.json_input and not args.json_array_input and not args.db_input:
|
||||
extractor.cleanup()
|
||||
except OSError as e:
|
||||
consoleLogger.error(f"{Fore.RED} [-] Error during cleanup {e}{Fore.RESET}")
|
||||
|
||||
Reference in New Issue
Block a user