QGIS Enhancement Proposal: Paged Fetching of Features and Browser Attribute Table Improvements

[Mathieu Pellerin]

QGIS Enhancement: QGIS Enhancement Proposal: Paged Fetching of Features and Browser Attribute Table Improvements

Date 2024/03/14

Author Mathieu Pellerin (@nirvn), Matthias Kuhn (@m-kuhn)

Contact [e-mail]

Version QGIS 3.X

Introduction

The attribute table of QGIS has long been earmarked for significant enhancements to improve performance and usability, especially when dealing with large datasets. A key feature under consideration is the introduction of pagination, which allows users to navigate through data in manageable chunks rather than loading extensive datasets all at once resulting in poor UX. The complete redesign of the main attribute table is a major project. It requires starting from the beginning and will be tackled in the future.
The browser’s attribute table allows previewing the attributes of a dataset before loading a layer.

Current Limitations

The proposal will focus on adding paging support through improving one user-end limitation: currently, the browser's attribute table preview caps the number of rows displayed to the first 100 entries. This limitation can hinder the ability to efficiently browse and preview large datasets, as users cannot easily access records beyond this initial subset without resorting to custom queries or loading full datasets through the main vector layer attribute table, potentially impacting performance.

Proposed Solution

This QEP proposes a phased approach to address the attribute table limitation described in the introduction by adding the necessary pagination capabilities within QGIS’ vector data provider and feature request classes, and exposing those new capabilities through the browser panel details’ attributes widget. The goal is to enable users to navigate through large datasets using paging functionalities exposed through a simple UI. This initial scope will lay the groundwork for more extensive attribute table improvements in the future.

Deliverables

1. Data Provider Pagination Capabilities: Data providers within QGIS will be extended to expose pagination capabilities. Providers can indicate support for no pagination, or offset-based pagination. The architecture will keep in mind the need for a cursor-based pagination in the future. This proposal will primarily focus on introducing offset-based pagination for the postgis provider, but recognizes the need to extent in the future to other interesting formats such as OAPIF (OGC API - Features).
2. Paginated Feature Requests: When building feature requests, the QgsFeatureRequest class will gain new properties - such as offset or startFromCursor- to complement existing parameters like limit and orderBy. This enhancement will allow efficient data retrieval in a paginated fashion. Consideration will be given to developing a high-level QgsPagedFeatureIterator within the QgsVectorLayer class. This interface would simplify pagination by enabling methods such as QList page = pagedIterator.nextPage(), facilitating straightforward navigation through paged data.
3. Paginated Controls for the Browser Panel Attributes Tab: The browser attribute table interface will be updated to include pagination controls, allowing users to navigate between pages of data when the data provider supports the new capability. This UI improvement will enhance data accessibility and improve the overall user experience when exploring large datasets within the browser preview. It also aligns with the ongoing efforts to retire DB Manager by implementing its functionalities into QGIS itself.

Risks

Relatively as the introduced API will be used only in the browser panel to begin with.

Conclusion

By introducing paged loading capabilities to the browser's attribute table preview, this QEP aims to significantly improve the user experience in previewing large datasets within the QGIS browser. This initiative also represents a critical foundational stone towards a more comprehensive overhaul of the main vector layer attribute table functionality, with a focus on performance, usability, and modern data management practices. Further discussions and technical explorations will refine the approach to ensure the successful implementation of this enhancement.

The improvements done to the browser panel’s attribute table also supports ongoing efforts to render the DB Manager plugin obsolete by incorporating its functionalities into QGIS itself.

[Giovanni Manghi]

Anything that can improve the current situation is very welcome!

[Nyall Dawson]

@nirvn

While I'm positive to the described improvements, I have a couple of concerns here:

• Can you reword the title and phrasing here to make it super-clear that this DOESN'T refer to the main attribute table for layers? I'm afraid that this proposal will get misinterpreted as "make the attribute table faster", when it's not directly going to do that. Maybe change it to "Browser Attribute Preview" instead of "Browser Attribute Table" and replace all references to "attribute table" to "attribute preview" just to make the distinction super-clear?

Now, on to the technical stuff:

• Because paging in feature iterators and sort order for returned features are intrinsically linked, I wanted to make sure you're aware that implementing server-side pagination for postgres provider will also require fixing server-side sort handling for the provider. This has been disabled for some time, see https://github.com/qgis/QGIS/blob/master/src/providers/postgres/qgspostgresfeatureiterator.cpp#L173 . This is NOT a trivial fix either, and I'd estimate that it would require at least 2 days for an experienced QGIS developer (like yourself 😉 ) to fix. Please make sure you've accounted for this in your costing, I'd hate to see this project run over budget if this is not considered.
• Can you add some more detail about how you plan to support pagination on a low level? I'm afraid it won't be as simple as just appending an "offset" and "limit" clause to the postgres query. It's actually going to be incredibly complex to handle this correctly, when we consider expression compilation and sort handling. E.g. for expression compilation, we often (usually?) end up requesting more features from the provider then will match the filter expression and filter these on the QGIS client side. For the existing feature iterator limit option we handle this purely on the client side, by just stopping requesting more features when that limit of matching features is hit (on the client side). But when we introduce offset pagination, I'm not sure how this will work with alongside expression compilation. We won't be able to just append "offset 50" to the postgres query and assume that the first 50 features skipped on the postgres side actually correspond to 50 features which match the QGIS client side filtering options, because we may have had to fetch 75 features from the provider in order to actually find 50 client side filtered features which satisified the first 50 page requirements.
• This also gets trickier if sort ordering is specified in the feature request (as it would be when pagination is eventually supported by the main attribute table, since that requires a specific sort order on the feature request). Depending on the provider/sort options, server-side sorting may not always be possible. In this case QGIS currently falls back to just fetching ALL features from the provider then handling the sort client side, before returning any features on the iterator. But how will that work with pagination of features? If we have to fetch ALL features from the provider before we can return the the first page of 50 features, then there's no benefit at all to the pagination. (And actually, it's going to be much worse then before, because for every subsequent page request we'll actually be fetching EVERY SINGLE feature from the provider 😱😱😱 ). So I'm really keen to see some details on how this situation will be handled.

Again, I'm keen to see improvements like this, but I just want you to be aware of the full depth of this work before committing to it!