DocumentDb simple representations of BSON types
dlvenable opened this issue · comments
Problem/Background
The upcoming documentdb
source is putting BSON types directly into the Event
model. These types are not supported by downstream processors or sinks. So at the best, they will get the toString()
representations.
Solution
When constructing the Data Prepper event in the documentdb
source, create simple representations of BSON types.
ObjectId
For the BSON ObjectId, make this the string representation.
Input:
{
"name" : "Star Wars",
"directorId" : ObjectId("fdd898945")
}
Output from documentdb
source:
{
"name" : "Star Wars",
"directorId" : "fdd898945"
}
BinData
For the BSON BinData type, make the simple representation a base64 encoded string for that binary data. The subtype will be lost.
Input:
{
"filepath" : "/usr/share/doc1",
"myBinary" : BinData(binary="[0x88]", subtype="MD5") # Not actual format; just a conceptual representation
}
Output:
{
"filepath" : "/usr/share/doc1",
"myBinary" : "X7ah=="
}
Timestamp
For the BSON Timestamp type, create an integer with the epoch seconds. The ordinal will be lost.
Input:
{
"name" : "Star Wars",
"lastUpdatedAt" : Timestamp(time=1713536835 ordinal=12) # Not actual format
}
Output:
{
"_id" : "abcdef12345",
"name" : "Star Wars",
"lastUpdatedAt" : 1713536835 # The time part. The ordinal is lost
}
Regular Expressions
The BSON RegEx type has both a pattern and options part. I'm unsure if the patterns would be sufficient on its own. So perhaps this one must have nested fields.
Input:
{
"use-case" : "email_address",
"myRegex" : RegEx(pattern="[A-Za-z0-9+_.-]+@([\w-]+\.)+[\w-]{2,4}", options="i") # Not actual format
}
Output:
{
"use-case" : "email_address",
"myRegex" : {
"pattern" : "[A-Za-z0-9+_.-]+@([\w-]+\.)+[\w-]{2,4}",
"options" : "i"
}
}