Support for Complex Features in WFS provider

Question

Support for Complex Features in WFS provider

rouault opened this issue 9 months ago · comments

Even Rouault commented 9 months ago

QGIS Enhancement: Support for Complex Features in WFS provider

Date 2023/11/13

Author Even Rouault (@rouault)

Contact even.rouault at spatialys.com

Maintainer @rouault

Version QGIS 3.36 or 3.38

Sponsored by QGIS-DE user group (QGIS Anwendergruppe Deutschland e.V.)

Summary

The QGIS WFS provider (WFS client) supports WFS 1.0, 1.1 and 2.0 protocols, but it is currently restricted to consuming features returned as GML simple features, that is features that have attributes of simple pre-defined types (string, integer, floating-point number, datetime). While WFS servers serving GML simple features are common, there are various data models around that don't fit into that model and use more complex GML schemas, where feature properties can be repeated or be made of nested XML constructs. This is typically the case for Inspire. Currently, QGIS rejects such WFS layers. While there are workarounds, using the WFS 2.0 client plugin or the QGIS GML Application Schema Toolbox plugin, they require the user to be aware of their existence, and don't provide the same level of user experience than when working with a simple feature WFS layer. Hence this proposal of enhancing the existing WFS provider to be able to deal with complex feature schemas. The implementation will expose properties of complex types as JSON content, converted from XML.

Proposed Solution

Let's state it in a straighforward way: dealing with WFS/GML complex features is a pain and based on past experience developing the OGR GMLAS - Geography Markup Language (GML) driven by application schemas - driver I don't believe there is an ideal solution. The solution proposed here is both a compromise from an implementation and usability point of view. The choice of exposing complex properties as a JSON serialized string from nested XML is what makes it possible to use the existing infrastructure of the WFS provider. The other alternative that could be imagined, and which was taken by the OGR GMLAS driver, would have been to expose a (complex) relational data model of many simple feature tables with lots of relationships between them. It woulnd't fit at all as a QGIS provider, as a QGIS provider is bound to a single layer, and such output is not necessarily easy to manipulate by end users. Hence the QGIS GML Application Schema Toolbox plugin is not totally deprecated by the enhanced WFS provider, in use cases where a relational view of the data model is needed.

Concretely, the QgsWFSProvider::readAttributesFromSchema() method which analyzes the XML schema returned by the WFS DescribeFeatureType request on the layer of interest will be modified to fallback to the OGR GMLAS driver when the schema is not natively understood by QGIS (that is, it is not simple features). More precisely, only the schema analysis capability of the GMLAS driver will be used, by querying the special _ogr_fields_metadata layer which returns the properties of a FeatureType and their nature (if they are simple or complex, their cardinality). To be noted too that the GMLAS driver automatically "flattens" the data structure, even when the content is nested, but when the constraints schema show that there is a 1:1 cardinality (cf below example where ps:inspireID/base:Identifier/base:localId can be flattened as inspireid_identifier_localid). The QgsGmlStreamingParser class will be modified to use the hints (mapping between XPath and QGIS fields) provided by the GMLAS schema analyzer to identify properties. A QgsXML class will be added to offer the translation from XML to equivalent JSON.

For server-side filtering of simple content that is not at the first level of nesting, QgsOgcUtils::expressionToOgcExpression() and QgsOgcUtils::SQLStatementToOgcFilter() will need to be enhanced to map QGIS field names to proper XPath.

The first time the OGR GMLAS driver needs to access a remote schema, it downloads its content and caches the file locally. However with complex schemas, this can involve downloading several tens of files in a cascaded way, which can take several tens of seconds depending on network and server speed. The OGR GMLAS driver will be modified to allow cancellation of the downloading of files, and the QGIS WFS provider, when invoked from QGIS graphical user interface, will cause the OGR GMLAS driver to be invoked in a background task that can be canceled to avoid blocking the user interface. The OGR GMLAS driver will also be modified to use the the pluggable CPL networking capability of CPLHTTPPushFetchCallback(), to be able to use QGIS networking infrastructure to download the XML schemas.

The WFS provider automated tests will be enhanced to test for the above mentioned (non UI-based) changes/enhancements.

Example

Let's look at the output of a Inspire data model: https://inspire.brandenburg.de/services/ps_schutzg_wfs?service=WFS&version=2.0.0&typename=ps:ProtectedSite&request=getfeature&count=1

<?xml version='1.0' encoding='UTF-8'?>
<wfs:FeatureCollection xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.opengis.net/wfs/2.0 http://schemas.opengis.net/wfs/2.0/wfs.xsd http://www.opengis.net/gml/3.2 http://schemas.opengis.net/gml/3.2.1/gml.xsd http://inspire.ec.europa.eu/schemas/ps/4.0 https://inspire.brandenburg.de/services/ps_schutzg_wfs?SERVICE=WFS&amp;VERSION=2.0.0&amp;REQUEST=DescribeFeatureType&amp;OUTPUTFORMAT=application%2Fgml%2Bxml%3B+version%3D3.2&amp;TYPENAME=ps:ProtectedSite&amp;NAMESPACES=xmlns(ps,http%3A%2F%2Finspire.ec.europa.eu%2Fschemas%2Fps%2F4.0)" xmlns:wfs="http://www.opengis.net/wfs/2.0" timeStamp="2023-10-19T09:09:24Z" xmlns:gml="http://www.opengis.net/gml/3.2" numberMatched="unknown" numberReturned="0">
  <wfs:member>
    <ps:ProtectedSite xmlns:ps="http://inspire.ec.europa.eu/schemas/ps/4.0" gml:id="ProtectedSite_FFH_553_DE4546-303">
      <gml:identifier codeSpace="https://inspire.ec.europa.eu/ids">https://registry.gdi-de.org/id/de.bb.inspire.ps.schutzg/ProtectedSite_FFH_553_DE4546-303</gml:identifier>
      <ps:geometry>
        <!--Inlined geometry 'ProtectedSite_FFH_553_DE4546-303_PS_GEOMETRY'-->
        <gml:Polygon gml:id="ProtectedSite_FFH_553_DE4546-303_PS_GEOMETRY" srsName="http://www.opengis.net/def/crs/EPSG/0/25833">
<!--- ... snipped ... -->
        </gml:Polygon>
      </ps:geometry>
      <ps:inspireID>
        <base:Identifier xmlns:base="http://inspire.ec.europa.eu/schemas/base/3.3">
<base:localId>ProtectedSite_FFH_553_DE4546-303</base:localId>
<base:namespace>https://registry.gdi-de.org/id/de.bb.inspire.ps.schutzg</base:namespace>
        </base:Identifier>
      </ps:inspireID>
      <ps:legalFoundationDate xsi:nil="true" nilReason="other:unpopulated"/>
      <ps:legalFoundationDocument xsi:nil="true" nilReason="other:unpopulated"/>
      <ps:siteDesignation>
        <ps:DesignationType>
          <ps:designationScheme xmlns:xlink="http://www.w3.org/1999/xlink" xlink:href="http://inspire.ec.europa.eu/codelist/DesignationSchemeValue/natura2000"/>
          <ps:designation xmlns:xlink="http://www.w3.org/1999/xlink" xlink:href="http://inspire.ec.europa.eu/codelist/Natura2000DesignationValue/specialAreaOfConservation"/>
        </ps:DesignationType>
      </ps:siteDesignation>
      <ps:siteName>
        <gn:GeographicalName xmlns:gn="http://inspire.ec.europa.eu/schemas/gn/4.0">
          <gn:language>deu</gn:language>
          <gn:nativeness xsi:nil="true" nilReason="other:unpopulated"/>
          <gn:nameStatus xsi:nil="true" nilReason="other:unpopulated"/>
          <gn:sourceOfName xsi:nil="true" nilReason="other:unpopulated"/>
          <gn:pronunciation xsi:nil="true" nilReason="other:unpopulated"/>
          <gn:spelling>
            <gn:SpellingOfName>
              <gn:text>Große Röder</gn:text>
              <gn:script>latn</gn:script>
            </gn:SpellingOfName>
          </gn:spelling>
        </gn:GeographicalName>
      </ps:siteName>
<ps:siteProtectionClassification>natureConservation</ps:siteProtectionClassification>
    </ps:ProtectedSite>
  </wfs:member>
</wfs:FeatureCollection>

Querying the structure of the protectedsite layer with the OGR GMLAS driver yields:

ogrinfo GMLAS: -oo XSD="https://inspire.brandenburg.de/services/ps_schutzg_wfs?SERVICE=WFS&VERSION=2.0.0&REQUEST=DescribeFeatureType&TYPENAME=ps:ProtectedSite" _ogr_fields_metadata -where "layer_name = 'protectedsite'" -al -q

Layer name: _ogr_fields_metadata
OGRFeature(_ogr_fields_metadata):0
  layer_name (String) = protectedsite
  field_index (Integer) = 1
  field_name (String) = id
  field_xpath (String) = ps:ProtectedSite/@gml:id
  field_type (String) = ID
  field_is_list (Integer(Boolean)) = 0
  field_min_occurs (Integer) = 0
  field_max_occurs (Integer) = 1
  field_category (String) = REGULAR
  field_documentation (String) = The attribute gml:id supports provision of a handle for the XML element representing a GML Object. Its use is mandatory for all GML objects. It is of XML type ID, so is constrained to be unique in the XML document within which it occurs.

OGRFeature(_ogr_fields_metadata):1
  layer_name (String) = protectedsite
  field_index (Integer) = 2
  field_name (String) = metadataproperty
  field_xpath (String) = ps:ProtectedSite/gml:metaDataProperty
  field_is_list (Integer(Boolean)) = 0
  field_min_occurs (Integer) = 0
  field_max_occurs (Integer) = 2147483647
  field_repetition_on_sequence (Integer(Boolean)) = 0
  field_category (String) = PATH_TO_CHILD_ELEMENT_NO_LINK
  field_related_layer (String) = protectedsite_metadataproperty

OGRFeature(_ogr_fields_metadata):2
  layer_name (String) = protectedsite
  field_index (Integer) = 3
  field_name (String) = description_href
  field_xpath (String) = ps:ProtectedSite/gml:description/@xlink:href
  field_type (String) = anyURI
  field_is_list (Integer(Boolean)) = 0
  field_min_occurs (Integer) = 0
  field_max_occurs (Integer) = 1
  field_category (String) = REGULAR

OGRFeature(_ogr_fields_metadata):3
  layer_name (String) = protectedsite
  field_index (Integer) = 4
  field_name (String) = description_title
  field_xpath (String) = ps:ProtectedSite/gml:description/@xlink:title
  field_type (String) = string
  field_is_list (Integer(Boolean)) = 0
  field_min_occurs (Integer) = 0
  field_max_occurs (Integer) = 1
  field_category (String) = REGULAR

OGRFeature(_ogr_fields_metadata):4
  layer_name (String) = protectedsite
  field_index (Integer) = 5
  field_name (String) = description_nilreason
  field_xpath (String) = ps:ProtectedSite/gml:description/@nilReason
  field_type (String) = anySimpleType
  field_is_list (Integer(Boolean)) = 0
  field_min_occurs (Integer) = 0
  field_max_occurs (Integer) = 1
  field_category (String) = REGULAR

OGRFeature(_ogr_fields_metadata):5
  layer_name (String) = protectedsite
  field_index (Integer) = 6
  field_name (String) = description
  field_xpath (String) = ps:ProtectedSite/gml:description
  field_type (String) = string
  field_is_list (Integer(Boolean)) = 0
  field_min_occurs (Integer) = 0
  field_max_occurs (Integer) = 1
  field_category (String) = REGULAR
  field_documentation (String) = The value of this property is a text description of the object. gml:description uses gml:StringOrRefType as its content model, so it may contain a simple text string content, or carry a reference to an external description. The use of gml:description to reference an external description has been deprecated and replaced by the gml:descriptionReference property.

OGRFeature(_ogr_fields_metadata):6
  layer_name (String) = protectedsite
  field_index (Integer) = 7
  field_name (String) = descriptionreference_href
  field_xpath (String) = ps:ProtectedSite/gml:descriptionReference/@xlink:href
  field_type (String) = anyURI
  field_is_list (Integer(Boolean)) = 0
  field_min_occurs (Integer) = 0
  field_max_occurs (Integer) = 1
  field_category (String) = REGULAR

OGRFeature(_ogr_fields_metadata):7
  layer_name (String) = protectedsite
  field_index (Integer) = 8
  field_name (String) = descriptionreference_title
  field_xpath (String) = ps:ProtectedSite/gml:descriptionReference/@xlink:title
  field_type (String) = string
  field_is_list (Integer(Boolean)) = 0
  field_min_occurs (Integer) = 0
  field_max_occurs (Integer) = 1
  field_category (String) = REGULAR

OGRFeature(_ogr_fields_metadata):8
  layer_name (String) = protectedsite
  field_index (Integer) = 9
  field_name (String) = descriptionreference_nilreason
  field_xpath (String) = ps:ProtectedSite/gml:descriptionReference/@nilReason
  field_type (String) = anySimpleType
  field_is_list (Integer(Boolean)) = 0
  field_min_occurs (Integer) = 0
  field_max_occurs (Integer) = 1
  field_category (String) = REGULAR

OGRFeature(_ogr_fields_metadata):9
  layer_name (String) = protectedsite
  field_index (Integer) = 10
  field_name (String) = identifier_codespace
  field_xpath (String) = ps:ProtectedSite/gml:identifier/@codeSpace
  field_type (String) = anyURI
  field_is_list (Integer(Boolean)) = 0
  field_min_occurs (Integer) = 0
  field_max_occurs (Integer) = 1
  field_category (String) = REGULAR

OGRFeature(_ogr_fields_metadata):10
  layer_name (String) = protectedsite
  field_index (Integer) = 11
  field_name (String) = identifier
  field_xpath (String) = ps:ProtectedSite/gml:identifier
  field_type (String) = string
  field_is_list (Integer(Boolean)) = 0
  field_min_occurs (Integer) = 0
  field_max_occurs (Integer) = 1
  field_category (String) = REGULAR
  field_documentation (String) = Often, a special identifier is assigned to an object by the maintaining authority with the intention that it is used in references to the object For such cases, the codeSpace shall be provided. That identifier is usually unique either globally or within an application domain. gml:identifier is a pre-defined property for such identifiers.
gml:CodeWithAuthorityType requires that the codeSpace attribute is provided in an instance.

OGRFeature(_ogr_fields_metadata):11
  layer_name (String) = protectedsite
  field_index (Integer) = 12
  field_name (String) = name
  field_xpath (String) = ps:ProtectedSite/gml:name
  field_is_list (Integer(Boolean)) = 0
  field_min_occurs (Integer) = 0
  field_max_occurs (Integer) = 2147483647
  field_repetition_on_sequence (Integer(Boolean)) = 0
  field_category (String) = PATH_TO_CHILD_ELEMENT_NO_LINK
  field_related_layer (String) = protectedsite_name

<!--- ... snipped ... -->

OGRFeature(_ogr_fields_metadata):22
  layer_name (String) = protectedsite
  field_index (Integer) = 23
  field_name (String) = legalfoundationdocument_owns
  field_xpath (String) = ps:ProtectedSite/ps:legalFoundationDocument/@owns
  field_type (String) = boolean
  field_is_list (Integer(Boolean)) = 0
  field_min_occurs (Integer) = 0
  field_max_occurs (Integer) = 1
  field_default_value (String) = false
  field_category (String) = REGULAR

OGRFeature(_ogr_fields_metadata):23
  layer_name (String) = protectedsite
  field_index (Integer) = 24
  field_name (String) = legalfoundationdocument_ci_citation_pkid
  field_xpath (String) = ps:ProtectedSite/ps:legalFoundationDocument/gmd:CI_Citation
  field_type (String) = string
  field_is_list (Integer(Boolean)) = 0
  field_min_occurs (Integer) = 0
  field_max_occurs (Integer) = 1
  field_category (String) = PATH_TO_CHILD_ELEMENT_WITH_LINK
  field_related_layer (String) = ci_citation

OGRFeature(_ogr_fields_metadata):24
  layer_name (String) = protectedsite
  field_index (Integer) = 25
  field_name (String) = sitedesignation
  field_xpath (String) = ps:ProtectedSite/ps:siteDesignation
  field_is_list (Integer(Boolean)) = 0
  field_min_occurs (Integer) = 1
  field_max_occurs (Integer) = 2147483647
  field_repetition_on_sequence (Integer(Boolean)) = 0
  field_category (String) = PATH_TO_CHILD_ELEMENT_NO_LINK
  field_related_layer (String) = protectedsite_sitedesignation

OGRFeature(_ogr_fields_metadata):25
  layer_name (String) = protectedsite
  field_index (Integer) = 26
  field_name (String) = sitename
  field_xpath (String) = ps:ProtectedSite/ps:siteName
  field_is_list (Integer(Boolean)) = 0
  field_min_occurs (Integer) = 0
  field_max_occurs (Integer) = 2147483647
  field_repetition_on_sequence (Integer(Boolean)) = 0
  field_category (String) = PATH_TO_CHILD_ELEMENT_NO_LINK
  field_related_layer (String) = protectedsite_sitename

OGRFeature(_ogr_fields_metadata):26
  layer_name (String) = protectedsite
  field_index (Integer) = 27
  field_name (String) = siteprotectionclassification
  field_xpath (String) = ps:ProtectedSite/ps:siteProtectionClassification
  field_is_list (Integer(Boolean)) = 0
  field_min_occurs (Integer) = 1
  field_max_occurs (Integer) = 2147483647
  field_repetition_on_sequence (Integer(Boolean)) = 0
  field_category (String) = PATH_TO_CHILD_ELEMENT_NO_LINK
  field_related_layer (String) = protectedsite_siteprotectionclassification

The layer fields exposed by QGIS will be:

  id (String) = ProtectedSite_FFH_553_DE4546-303
  identifier_codespace (String) = https://inspire.ec.europa.eu/ids
  identifier (String) = https://registry.gdi-de.org/id/de.bb.inspire.ps.schutzg/ProtectedSite_FFH_553_DE4546-303
  inspireid_identifier_localid (String) = ProtectedSite_FFH_553_DE4546-303
  inspireid_identifier_namespace (String) = https://registry.gdi-de.org/id/de.bb.inspire.ps.schutzg
  legalfoundationdate_nilreason (String) = other:unpopulated
  legalfoundationdate (DateTime) = (null)
  legalfoundationdocument_nilreason (String) = other:unpopulated
  legalfoundationdocument_owns (Integer(Boolean)) = 0
  ... other fields omitted for brievity ...
  siteName(Map) = { "gn:GeographicalName" : {
          "@xmlns:gn": "http://inspire.ec.europa.eu/schemas/gn/4.0",
          "gn:language": "deu",
          "gn:nativeness" : { "xsi:nil": true, "nilReason": "other:unpopulated" },
          "gn:nameStatus" : { "xsi:nil": true, "nilReason": "other:unpopulated" },
          "gn:sourceOfName" : { "xsi:nil": true, "nilReason": "other:unpopulated" },
          "gn:pronunciation" : { "xsi:nil": true, "nilReason": "other:unpopulated" },
          "gn:spelling" : {
            "gn:SpellingOfName": {
              "gn:text": "Große Röder",
              "gn:script": "latn"
            }
          }
        } }
  geometry = POLYGON (( .... snipped .... ))

Affected Files

In QGIS:

src/providers/wfs/*
src/core/qgsgml.h|cpp
src/core/qgsogcutils.h|cpp
src/core/expression/qgsexpressionfunction.cpp

In GDAL:

GMLAS driver

Limitations

Only one geometry field per layer will be handled, as currently. Managing several geometry fields per WFS layer would be possible in the future, similarly to other providers, with one QGIS layer per geometry field
WFS-T(ransactional) support (that is creation, update and deletion) for WFS Complex Features layers will not be implemented.
Server-side filtering will be limited to non-XML fields.
This will only work for servers that properly expose the location of all remote XML Schema files that need to be accessed and that such files are actually accessible.

Performance Implications

No measurable performance hit expected on simple features WFS layers.

For complex features WFS layers, an initial delay of several seconds the first
time a new schema is read is expected.

Backwards Compatibility

There should be no backward compatibility issues as this is a new feature.

Issue Tracking ID(s)

#27076 - WFS 2.0 complex features not supported
#52227 - Error loading WFS

Votes

To be done

Nyall Dawson · Answer 1 · Tue Nov 14 2023 06:51:15 GMT+0800 (China Standard Time)

@rouault interesting discussion!

While working on #257 (a SensorThings provider) , I've been wondering if we need some generic / cross provider approach to handle feature relationships in a more flexible way. Currently relationships in QGIS (as you've noted) are tied heavily to the concept of individual layers, but this doesn't translate well to data models like you've described here or the SensorThings model where the relationship structure is more complex and tightly tied into the data model itself.

Another example would be handling "attachments" from an ArcGIS REST server. These aren't available as a standard layer, and indeed can't even be retrieved in bulk. Rather you need to call a specific "get attachments" API per feature that you want to retrieve the attachments for. Again, this setup doesn't fit in well with the current layer based approach to relationships.

So I'm wondering if we shouldn't use this opportunity to instead develop a "get related features" concept within the vector data provider itself, which would allow for interactive feature-based (and nested) related feature retrieval.

I'm keen to hear your thoughts... it's quite a different approach to the "flattening" technique you've proposed here, but might ultimately be a better fit for this data model...

Even Rouault · Answer 2 · Tue Nov 14 2023 17:25:41 GMT+0800 (China Standard Time)

a "get related features" concept

In the case of GML complex features, a property can indeed be sometimes a reference to another object through a URL, but quite often, it is just inline nested XML content without necessary an identifier (as gn:GeographicalName in ps:siteName in my above example), and such content doesn't really qualify for being called a feature and is already returned in the stream of the main feature.
I'm not clear where the data obtained from "get related features" would land: in a dedicated QGIS layer (probably a in-memory one, although I can imagine users would want to see serialization of related features they would have resolved) for each related attribute ? The idea makes sense, but I don't have the bandwidth to implement that on top of the work exposed in this QEP. If such a "get related features" capability was developed, I could see that as a potential enhancement that could potentially be leveraged by the WFS provider on top of what I'm proposing.
Dealing with related features could potentially involve other solutions, like developing a QGIS processing tool, that would resolve related features of already loaded features (although there could be a risk of making the remote server struggle with a big number of requests)
In short, this QEP aims at providing an initial integrated capability of WFS complex features, with an experience similar to simple features, for fields that are simple, and for complex content, the proposed QGIS expression function should provide a initial way of making use of that information.

Martin Dobias · Answer 3 · Tue Nov 14 2023 19:07:15 GMT+0800 (China Standard Time)

A few years ago I was putting together a proposal to better handle INSPIRE data in QGIS and I was leaning towards the GMLAS approach, where the GML file would be exploded into many tables (spatial and non-spatial) that get loaded in QGIS as layers, and then relationships between the layers would be automatically set up.

But given the complexity of the problem, probably we may end up having both approaches - one that is proposed here (with nested XML) and another one with multiple related tables. I guess the former approach is better for simpler data models, the latter is better for complex models with lots of 1:N and N:M relationships. The GMLAS plugin for QGIS also offers both options: https://brgm.github.io/gml_application_schema_toolbox/usage/read_files.html

It would be great if it would be possible to also just drag'n'drop a GML file with complex features to get this functionality - I have seen quite a few broken WFS 2.0 servers (running out of memory, timing out, returning bad results) that it sometimes makes sense to just download GML data when possible and avoid WFS altogether 😄 - but not sure if it will be easy if the logic would be in the WFS provider code?

By the way, the use of XQuery will need xmlpatterns Qt module which I believe we do not require yet, but probably it is already included in the installers, so that should not be a big deal.

+1 from me on this QEP

@nyalldawson as for your "get related features" concept - how would be that different from the existing concept of relations in QGIS, and the discoverRelations() provider call?

Even Rouault · Answer 4 · Tue Nov 14 2023 19:21:54 GMT+0800 (China Standard Time)

It would be great if it would be possible to also just drag'n'drop a GML file with complex features to get this functionality

Everything is doable, but I'm afraid that would complicate the WFS provider in non-trivial & inelegant ways. Plus we would need to hack around the OGR provider to pass such files to the WFS one.

By the way, the use of XQuery will need xmlpatterns Qt module which I believe we do not require yet, but probably it is already included in the installers, so that should not be a big deal.

Actually @pathmapper pointed to me that qtxmlpatterns was unfortunately deprecated in Qt 5.13 (https://doc.qt.io/qt-5/qtxmlpatterns-index.html) and has been removed in Qt 6.0. After looking around, I'm leaning towards using libxml2 which has XPath 1.0 capabilities and is a widely available dependency, used for example by most GDAL builds (and I see I've used libxml2 XPath in the past in GDAL, for unrelated purposes)

Tobias Kraft · Answer 5 · Fri Nov 17 2023 03:10:42 GMT+0800 (China Standard Time)

A most welcome initiative! I'd like to suggest, however, to serialize the content of complex attributes as JSON rather than XML if possible. This would

dispense the need to introduce XPath capabilities
allow to leverage the already available data extraction capabilities in QGIS expressions like subscripting or functions such as array_contains, e.g.:
provide improved readability with the JSON View widget:
be in line with the general tendency to drop XML in favor of JSON, allowing users to focus on and become familiar with just one common format

Even Rouault · Answer 6 · Fri Nov 17 2023 03:26:05 GMT+0800 (China Standard Time)

@kraftto

I'd like to suggest, however, to serialize the content of complex attributes as JSON rather than XML if possible. This would

I indeed considered exposing complex content as JSON in my initial analysis, but my counter arguments are:

if one day one wants to implement the transactional part, it will be much easier to do with XML content rather than with JSON that might be difficult to remap correctly to the expected XML schema
Another issue is that the QGIS editor for JSON fields is seriously limited to apparently only dictionaries of key/value pairs of simple types, and not arbitrarily nested content. I'm not sure what is the exact typing of the "properties" field in your above screenshot, but I suspect it is a string field and not a QVariant::Map.

UPDATE: I now see in Layer Properties / Attributes Form, that one should select "JSON View" as the widget type to get the output of your screenshot, instead of "Key/Value"! Grrr, I stumbled upon that issue for some time.
==> to anyone looking at this, should we change the default widget type for QVariant::Map to "JSON View" to avoid the issue with truncated content of the default "Key/Value" widget ?

So given the above, my point about using XML is much weaker, as transactional support for complex features is likely a marginal use case, and JSON could be a reasonable choice

Tobias Kraft · Answer 7 · Fri Nov 17 2023 03:52:33 GMT+0800 (China Standard Time)

but I suspect it is a string field and not a QVariant::Map

It is a JSONB Postgres field, which is adequately assigned the JSON View widget, but there seems to be an issue in that regard with Map type subtypes.

I get the point regarding WFS-T, but I do agree that it's unlikely to be implemented any time soon, so it might be superseded by a transactional OGC API Features by then.

Even Rouault · Answer 8 · Fri Nov 24 2023 20:00:17 GMT+0800 (China Standard Time)

I've updated the description to reflect that complex XML content will be exposed as JSON

Nyall Dawson · Answer 9 · Tue Nov 28 2023 11:27:22 GMT+0800 (China Standard Time)

@wonder-sk

as for your "get related features" concept - how would be that different from the existing concept of relations in QGIS, and the discoverRelations() provider call?

The current framework for relations is completely dependent on the participants in the relation being representable as vector layer themselves. This constraint breaks the use case of relationships where there is no fixed structure that can be represented as a child vector layer. I'm thinking here of cases where the children have completely different fields depending on which parent feature they have.

Or, in my immediate use case, for the situation where we can't efficiently construct the child as a vector layer. The ArcGIS REST API only permits querying related media feature-by-feature, so in order to construct a vector layer of related media for the children we'd have to do something very bad like iterating over all the parent features and firing off a get children request for each individually.

What I am thinking is something like:

struct RelatedFeature
{
    //! Child feature, containing direct attributes and geometry for the child
   QgsFeature feature;
 
   //! Map of child features belonging to this feature, where map keys are relation identifiers
   QMap< QString, QList< RelatedFeature > > children; 
}


class QgsVectorDataProvider
{
    ...

    /**
      * Retrieves related child features for the parent feature with the specified \a id.
      *
      * The keys in the returned map are the relation identifiers.
      */
    QMap< QString, QList< RelatedFeature > > queryRelatedFeatures( QgsFeatureId id ); 

    ...
}