Giters

alephdata / aleph

Search and browse documents and data; find the people and companies you look for.

Home Page:http://docs.aleph.occrp.org

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

BUG: Creating entities with certain date property values fails

tillprochaska opened this issue · comments

commented

Describe the bug
Creating entities that have date properties in the format yyyy-MM-dd'T'HH:mm and yyyy-MM-dd HH:mm fails.

FollowTheMoney supports date properties in this format according to the documentation and Aleph also supports formats such as yyyy-MM-dd HH:mm:ss (datetime with seconds), yyyy-MM (year and month), and any other date and time precision, so it’s very unintuitive that these specific formats are not supported.

To Reproduce
Steps to reproduce the behavior:

This is just one way to trigger this bug, but really any way to create or update entity properties should have the same problem.

  1. Create an investigation.
  2. In the left-hand sidebar, select "Add new entity type" and "Event".
  3. In the table editor, create a new event entity and set the start date to 2024-01-01 00:00.
  4. Aleph will seemingly save the changes, but once you reload the page, the change will be gone.

Expected behavior
Aleph should successfully create and update entities in the formats mentioned above.

Aleph version
3.15.5

Additional context
This bug likely occurs due to a problem in the Elasticsearch field mappings for date properties. Date properties use the following custom format: yyyy-MM-dd'T'HH:mm:ss||yyyy-MM-dd||yyyy-MM||yyyy. This custom format doesn’t match date-time values with minute precision.

Unfortunately, adding the missing date formats to the custom format in the mapping would require recreating the index (as Elasticsearch would consider the new mapping incompatible with the existing mapping).

See:

Elasticsearch error message when trying to update an existing mapping:

Index [aleph-entity-succession-v1] error: Mapper for [properties.startDate] conflicts with existing mapper:
Cannot update parameter [format] from [yyyy-MM-dd'T'HH:mm:ss||yyyy-MM-dd||yyyy-MM||yyyy] to [yyyy-MM-dd'T'HH:mm:ss||yyyy-MM-dd'T'HH:mm||yyyy-MM-dd||yyyy-MM||yyyy]

Elasticsearch error message when trying to index an entity with a (currently) unsupported date format:

Bulk index error: {'index': {'_index': 'aleph-entity-event-v1', '_type': '_doc', '_id': '...', 'status': 400, 'error': {'type': 'mapper_parsing_exception', 'reason': "failed to parse field [properties.startDate] of type [date] in document with id '...'. Preview of field's value: '2023-01-01T00:00'", 'caused_by': {'type': 'illegal_argument_exception', 'reason': "failed to parse date field [2023-01-01T00:00] with format [yyyy-MM-dd'T'HH:mm:ss||yyyy-MM-dd||yyyy-MM||yyyy]", 'caused_by': {'type': 'date_time_parse_exception', 'reason': 'Failed to parse with all enclosed parsers'}}}}}