PropertiesColumnWriter: ability to exclude specified properties

[SeppPenner]

See b00ted/serilog-sinks-postgresql#21 and b00ted/serilog-sinks-postgresql#22

[SeppPenner]

I still do not see the benefit: b00ted/serilog-sinks-postgresql#21 (comment)

[SeppPenner]

Will be closed until someone responds in b00ted/serilog-sinks-postgresql#21

[Gaz83]

Just stumbled upon this, this would be a great feature.
We are currently having a look at switching over to Linux and using PostgreSQL. I have been using Serilog with the MS SQL sink and this has 2 nice features that I use.

1)If there is a column for a property then don't also store it in the properties column. This is controlled by a setting in the config, which is off by default.

Example: If there is a property, say "RequestContent", and you create a column for this property then the property is removed from the properties column automatically as there is no need to write it twice.

Before
Column Properties: "{ "Properties": [ "RequestContent": "blah blah blah", "ActionId": "SomeID" ]}"
Column RequestContent: "blah blah blah"

After with auto property removed
Column Properties: { "Properties": [ "ActionId": "SomeID" ]}
Column RequestContent: "blah blah blah"

2. Being able to specify which standard columns you want/don't want. I use this feature a lot and in one particular Table I actually don't use the LogEvent or Properties columns as I have created columns for the parts I care about. This make it more efficient for querying.

Hope this all makes sense.

[lonix1]

To answer the question above, of why this is needed:

• "Level" is contained inside LogEvent. We write that property as a separate column, so we can query against it. But then it would exist as a column, as well as inside the LogEvent JSON column. So it should be excluded.
• "SourceContext" is contained inside LogEvent.Properties. We write that property as a separate column, so we can query against it, and also apply a column index. But then it would exist as a column, as well as inside the LogEvent JSON column. So it should be excluded.

So this problem is actually comprised of two sub-problems: excluding standard properties from LogEvent and excluding custom properties from LogEvent.Properties.

Remove custom properties from LogEvent.Properties

I customised LogEventSerializerColumnWriter for exclusions:

using System.Text;
using NpgsqlTypes;
using Serilog.Events;
using Serilog.Formatting.Json;

namespace Serilog.Sinks.PostgreSQL.ColumnWriters;

public sealed class CustomLogEventSerializedColumnWriter : ColumnWriterBase
{

  private readonly string[]? _excludedStandardPropertyNames;
  private readonly string[]? _excludedCustomPropertyNames;

  public CustomLogEventSerializedColumnWriter() : base(NpgsqlDbType.Jsonb, order:0) { }

  public CustomLogEventSerializedColumnWriter(
    NpgsqlDbType dbType = NpgsqlDbType.Jsonb,
    int? order = null,
    string[]? excludedStandardPropertyNames = null,
    string[]? excludedCustomPropertyNames = null
  ) : base(dbType, order:order)
  {
    if (excludedStandardPropertyNames?.Any(x => string.IsNullOrWhiteSpace(x)) ?? false) throw new ArgumentNullException(nameof(excludedStandardPropertyNames));
    if (excludedCustomPropertyNames  ?.Any(x => string.IsNullOrWhiteSpace(x)) ?? false) throw new ArgumentNullException(nameof(excludedCustomPropertyNames));
    _excludedStandardPropertyNames = excludedStandardPropertyNames;
    _excludedCustomPropertyNames   = excludedCustomPropertyNames;
  }

  public override object GetValue(LogEvent logEvent, IFormatProvider? formatProvider = null)
  {
    ArgumentNullException.ThrowIfNull(logEvent, nameof(logEvent));
    return LogEventToJson(logEvent, formatProvider);
  }

  private object LogEventToJson(LogEvent logEvent, IFormatProvider? formatProvider) {

    var jsonFormatter = new CustomJsonFormatter(excludedPropertyNames:_excludedStandardPropertyNames);

    if (_excludedCustomPropertyNames != null) {
      foreach (var excludedInnerPropertyName in _excludedCustomPropertyNames)
        logEvent.RemovePropertyIfPresent(excludedInnerPropertyName);
    }

    var sb = new StringBuilder();
    using var writer = new StringWriter(sb);
    jsonFormatter.Format(logEvent, writer);

    return sb.ToString();
  }

}

Remove standard properties from LogEvent itself

This is more difficult, because it uses Serilog's JsonFormatter, which automatically serialises all the properties.

So I wrote a custom formatter which is based on CompactJsonFormatter:

using System.Globalization;
using Serilog.Events;
using Serilog.Parsing;

namespace Serilog.Formatting.Json;

public class CustomJsonFormatter : ITextFormatter {

  private readonly JsonValueFormatter _valueFormatter;
  private readonly string[] _excludedPropertyNames;


  public CustomJsonFormatter(JsonValueFormatter? valueFormatter = null, string[]? excludedPropertyNames = null)
  {
    _valueFormatter = valueFormatter ?? new JsonValueFormatter(typeTagName: "$type");

    if (excludedPropertyNames != null) {
      foreach (var excludedPropertyName in excludedPropertyNames)
        ArgumentNullException.ThrowIfNullOrWhiteSpace(excludedPropertyName, nameof(excludedPropertyNames));
    }

    _excludedPropertyNames = excludedPropertyNames ?? [];
  }

  public void Format(LogEvent logEvent, TextWriter output)
  {
    FormatEvent(logEvent, output, _valueFormatter);
    output.WriteLine();
  }

  public void FormatEvent(LogEvent logEvent, TextWriter output, JsonValueFormatter valueFormatter)
  {
    ArgumentNullException.ThrowIfNull(logEvent, nameof(logEvent));
    ArgumentNullException.ThrowIfNull(output, nameof(output));
    ArgumentNullException.ThrowIfNull(valueFormatter, nameof(valueFormatter));

    var haveWrittenAtLeastOneProperty = false;

    output.Write("{");

    if (!_excludedPropertyNames.Contains("Timestamp")) {
      output.Write("\"@t\":\"");
      output.Write(logEvent.Timestamp.UtcDateTime.ToString("O") + "\"");
      haveWrittenAtLeastOneProperty = true;
    }

    if (!_excludedPropertyNames.Contains("MessageTemplate")) {
      output.Write((haveWrittenAtLeastOneProperty ? "," : "") + "\"@mt\":");
      JsonValueFormatter.WriteQuotedJsonString(logEvent.MessageTemplate.Text, output);
      haveWrittenAtLeastOneProperty = true;
    }

    var tokensWithFormat = logEvent.MessageTemplate.Tokens
      .OfType<PropertyToken>()
      .Where(pt => pt.Format != null);

    // Better not to allocate an array in the 99.9% of cases where this is false
    if (tokensWithFormat.Any()) {
      output.Write((haveWrittenAtLeastOneProperty ? "," : "") + "\"@r\":[");
      var delim = "";
      foreach (var r in tokensWithFormat) {
        output.Write(delim);
        delim = ",";
        using var space = new StringWriter();
        r.Render(logEvent.Properties, space, CultureInfo.InvariantCulture);
        JsonValueFormatter.WriteQuotedJsonString(space.ToString(), output);
      }
      output.Write(']');
      haveWrittenAtLeastOneProperty = true;
    }

    if (!_excludedPropertyNames.Contains("Level")) {
      output.Write((haveWrittenAtLeastOneProperty ? "," : "") + "\"@l\":\"");
      output.Write(logEvent.Level + "\"");
      haveWrittenAtLeastOneProperty = true;
    }

    if (logEvent.Exception != null) {
      if (!_excludedPropertyNames.Contains("Exception")) {
        output.Write((haveWrittenAtLeastOneProperty ? "," : "") + "\"@x\":");
        JsonValueFormatter.WriteQuotedJsonString(logEvent.Exception.ToString(), output);
        haveWrittenAtLeastOneProperty = true;
      }
    }

    if (logEvent.TraceId != null) {
      if (!_excludedPropertyNames.Contains("TraceId")) {
        output.Write((haveWrittenAtLeastOneProperty ? "," : "") + "\"@tr\":\"");
        output.Write(logEvent.TraceId.Value.ToHexString());
        output.Write("\"");
        haveWrittenAtLeastOneProperty = true;
      }
    }

    if (logEvent.SpanId != null) {
      if (!_excludedPropertyNames.Contains("SpanId")) {
        output.Write((haveWrittenAtLeastOneProperty ? "," : "") + "\"@sp\":\"");
        output.Write(logEvent.SpanId.Value.ToHexString());
        output.Write("\"");
        haveWrittenAtLeastOneProperty = true;
      }
    }

    foreach (var property in logEvent.Properties) {
      var name = property.Key;
      if (name.Length > 0 && name[0] == '@') {
        // Escape first '@' by doubling
        name = '@' + name;
      }
      output.Write(',');
      JsonValueFormatter.WriteQuotedJsonString(name, output);
      output.Write(':');
      valueFormatter.Format(property.Value, output);
    }

    output.Write('}');
  }

}

Alternative approaches here and here.

Usage

new Dictionary<string, ColumnWriterBase> {
  { "LogEvent",      new CustomLogEventSerializedColumnWriter(excludedStandardPropertyNames: new string[] { "Level", "Timestamp" },
                                                              excludedCustomPropertyNames:   new string[] { "SourceContext", "Foo" }) },
  { "Level",         new LevelColumnWriter() },
  { "Timestamp",     new TimestampColumnWriter() },
  { "SourceContext", new SinglePropertyColumnWriter("SourceContext") },
  { "Foo",           new SinglePropertyColumnWriter("Foo") },
}

Both of those classes are generic enough for general use, and should be in this library.