macos-unifiedlogs

A simple Rust library that can help parse the macOS Unified Log files.

Unified Logs were introduced in macOS version 10.12 (Sierra, 2016). Part of Apple's goal to create a unified log format for all Apple products. They exist on macOS, iOS, watchOS, tvOS. The Unified Logs replace many of the old log formats Apple used. This simple library can be used to parse files.
Data that is currently extracted includes:

Process ID
Thread ID
Activity ID
Log Message
Timestamp (Intel and ARM supported)
Effective User ID (EUID)
Log Type
Event Type
Library
Subsystem
Category
Process
Raw message - Message extracted from UUID file
Message entries - Message parts from tracev3 file. Combines with Raw message to get the Log Message
Library UUID
Process UUID
Boot UUID
Timezone

Running

Three (3) simple example binaries are available in examples.

unifiedlog_parser - Can parse all logs into a single CSV file. It can also be run on a live system. The resulting CSV file will likely be quite large
unifiedlog_parser_json - Can parse all logs into JSON files. It can also be run on a live system. Each log file (tracev3 file) will correspond to a single JSON file. Depending on the logs, hundreds of JSON files may get created
parse_tracev3 - Can parse a single tracev3 file without any timesync or uuidtext files, to a JSON file. However, without the uuidtext or timesync files the resulting JSON file will be heavily incomplete.

See RUNNING.md for overview of running the example binaries

Using as Library

If you want to import this project into a Rust application add the following to you Cargo.toml file

macos-unifiedlogs = {git = "https://github.com/mandiant/macos-UnifiedLogs"}

If you want to pin to a specific commit

macos-unifiedlogs = {git = "https://github.com/mandiant/macos-UnifiedLogs", rev = "commit hash"}

See Library.md for overview of how to use the library. Simple example projects are also available to review and use

Status

This library has been heavily tested on log data from macOS Sierra (10.12.5) to Monterey (12).
Its been tested against 100+ million log entries. However, due the complexity of the Unified Log format there are some limitations:

No printf style error code lookup support. This library does not do any error code lookups for log messages. The native log command on macOS supports error code lookups when it encounters printf style %m messages.
An example base log messsage: 'Failed to open file, error: %m'
a. This Library outputs:
```
  Failed to open file, error: 1
```
b. The macOS Log command outputs:
```
  Failed to open file, error: no such file or directory
```
Here the error code 1 gets translated to the error string message
No support for custom object decoders. The Unified Log format allows a developer to log abrirtary data to the logs. Apple also includes a handful of custom objects that developer can use to log raw data. An example list can be found in man os_log. However, it is not a complete list.

man os_log
...
     To format a log message, use a printf(3) format string.  You may also use the "%@" format specifier for use with Obj-C/CF/Swift objects, and %.*P which can be used to decode arbitrary binary data.  The logging system also supports custom decoding of values by denoting value types inline in the format %{value_type}d.  The built-in value type decoders are:

     Value type      Custom specifier         Example output
     BOOL            %{BOOL}d                 YES
     bool            %{bool}d                 true
     darwin.errno    %{darwin.errno}d         [32: Broken pipe]
     darwin.mode     %{darwin.mode}d          drwxr-xr-x
     darwin.signal   %{darwin.signal}d        [sigsegv: Segmentation Fault]
     time_t          %{time_t}d               2016-01-12 19:41:37
     timeval         %{timeval}.*P            2016-01-12 19:41:37.774236
     timespec        %{timespec}.*P           2016-01-12 19:41:37.2382382823
     bytes           %{bytes}d                4.72 kB
     iec-bytes       %{iec-bytes}d            4.61 KiB
     bitrate         %{bitrate}d              123 kbps
     iec-bitrate     %{iec-bitrate}d          118 Kibps
     uuid_t          %{uuid_t}.16P            10742E39-0657-41F8-AB99-878C5EC2DCAA
     sockaddr        %{network:sockaddr}.*P   fe80::f:86ff:fee9:5c16
     in_addr         %{network:in_addr}d      127.0.0.1
     in6_addr        %{network:in6_addr}.16P  fe80::f:86ff:fee9:5c16

Currently when the library encounters log messages that contain arbitrary binary data, it will base64 the data as a string. Support for custom decoders will hopefully be added in version 2 of the library.

No support for log messages that have custom object structures or protocol buffer data. Some logs contain binary plist files, custom object structures, or protocol buffer data. This library currently supports parsing binary plist data, but it does not support custom object structures, or protocol buffer data. The custom object structures are similar to the custom decoders mentioned above. But there is no list of decoders
Support for custom object structures will hopefully be added in version 2 of the library

References

https://github.com/ydkhatri/UnifiedLogReader
https://github.com/libyal/dtformats/blob/main/documentation/Apple%20Unified%20Logging%20and%20Activity%20Tracing%20formats.asciidoc
https://eclecticlight.co/2018/03/19/macos-unified-log-1-why-what-and-how/
https://www.crowdstrike.com/blog/how-to-leverage-apple-unified-log-for-incident-response/

mandiant / macos-UnifiedLogs

macos-unifiedlogs

Running

Using as Library

Status

References

About

Languages