[Bug] Not able to build Segments on CustomDimensions
jorgeuos opened this issue · comments
What happened?
I have created a Segment, looks like this:
mysql> SELECT *
-> FROM matomo_segment\G
*************************** 1. row ***************************
idsegment: 1
name: NameOfSegment
definition: dimension1==quepapoloco
hash: 2aae533a0383136ee4f1d30576bca987
login: jorgeuos
enable_all_users: 1
enable_only_idsite: 1
auto_archive: 1
ts_created: 2024-05-10 10:00:16
ts_last_edit: 2024-05-15 13:07:00
deleted: 0
*************************** 2. row ***************************
Other rows...
7 rows in set (0.02 sec)
mysql> And I want to invalidate past dates to run the archiver on these segments:
./console core:invalidate-report-data --dates=2024-05-01,2024-05-24 --segment=1
ERROR [19:30:44] 21673 Uncaught exception: /var/www/html/core/Segment.php(253): Segment 'dimension1' is not a supported segment. [Query: , CLI mode: 1]
In Segment.php line 253:
Segment 'dimension1' is not a supported segment.
core:invalidate-report-data [--dates DATES] [--sites SITES] [--periods PERIODS] [--segment SEGMENT] [--cascade] [--dry-run] [--plugin PLUGIN] [--ignore-log-deletion-limit]And if I use the Segment definition:
./console core:invalidate-report-data --dates=2024-05-01,2024-05-24 --segment=dimension1==quepapoloco
ERROR [19:37:24] 21682 Uncaught exception: /var/www/html/core/Segment.php(253): Segment 'dimension1' is not a supported segment. [Query: , CLI mode: 1]
In Segment.php line 253:
Segment 'dimension1' is not a supported segment.
core:invalidate-report-data [--dates DATES] [--sites SITES] [--periods PERIODS] [--segment SEGMENT] [--cascade] [--dry-run] [--plugin PLUGIN] [--ignore-log-deletion-limit]And if I invalidate without a segment definition or id and then rerun the core:archive, I just see rows like:
INFO [20:34:09] 22708 Archived website id 1, period = week, date = 2024-05-06, segment = 'dimension1==quepapoloco', 0 visits found. Time elapsed: 0.766sBut I can see that there is data in that custom_dimension:
mysql> SELECT
-> COUNT(mlv.idvisit)
-> FROM matomo.matomo_log_visit mlv
-> WHERE mlv.idsite = 1
-> AND mlv.custom_dimension_1="quepapoloco"
-> AND mlv.visit_last_action_time BETWEEN "2024-05-01" AND "2024-05-24";
+--------------------+
| COUNT(mlv.idvisit) |
+--------------------+
| 23312 |
+--------------------+
1 row in set (0.32 sec)Why is this happening? I have tried with multiple segments and custom dimensions.
I have tried with numerical values and strings.
I have tried with building the segment in the UI and with API calls:
Operator Behavior Example
== Equals &segment=dimension1==quepapoloco
!= Not equals &segment=dimension1!= (E.g. Not empty)
=@ Contains &segment=dimension1=@quepapo
=^ Starts with &segment=dimension1=^quepapo
I have tried with custom dimensions in both visits scope and action scope. I don't know why it's not working but the error refers to this line:
private function getSegmentByName($name)
{
$segments = $this->getAvailableSegments();
if (array_key_exists($name, $segments)) {
if ($segments[$name] === null) {
throw new NoAccessException("You do not have enough permission to access the segment " . $name);
}
return $segments[$name];
}
throw new Exception("Segment '$name' is not a supported segment.");
}And if I follow that getAvailableSegments() function, I can see that it's calling a PiwikCache::getTransientCache(); function, so I thought it might be an issue with that.
I've tried clearing the cache too:
# Both:
./console cache:clear
# And:
./console core:clear-cachesBut nothing changes
Config relevant to the segments:
./console config:get --section=General --format=text | grep segment
enable_segments_cache = 1
anonymous_user_enable_use_segments_API = 1
enable_create_realtime_segments = 1
enable_segment_suggested_values = 1
adding_segment_requires_access = view
allow_adding_segments_for_all_websites = 1
process_new_segments_from = beginning_of_time
disable_archiving_segment_for_plugins =
pivot_by_filter_enable_fetch_by_segment = 0
data_comparison_segment_limit = 5
rearchive_reports_in_past_exclude_segments = 0Any thoughts on why this is happening or is it not possible to use custom_dimension_x with segments?
What should happen?
I wan't to add a custom dimensions for a visits, so that a session can be tracked and filtered with from the Segment editor.
How can this be reproduced?
Add a Custom dimension with visit scope, populate the custom_dimension_x column for the matching rows, invalidate past dates, run core archiving.
Matomo version
5.0.2-5.0.3
PHP version
8.2.19
Server operating system
Alpine Linux 3.19
What browsers are you seeing the problem on?
Chrome
Computer operating system
MacOS
Relevant log output
ERROR [21:08:02] 28942 Uncaught exception: /var/www/html/core/Segment.php(253): Segment 'dimension1' is not a supported segment. [Query: , CLI mode: 1]
In Segment.php line 253:
Segment 'dimension1' is not a supported segment.
core:invalidate-report-data [--dates DATES] [--sites SITES] [--periods PERIODS] [--segment SEGMENT] [--cascade] [--dry-run] [--plugin PLUGIN] [--ignore-log-deletion-limit]Validations
- Read our Contributing Guidelines.
- Follow our Security Policy.
- Check that there isn't already an issue that reports the same bug to avoid creating duplicates.
- The provided steps to reproduce is a minimal reproducible of the Bug.
@jorgeuos Custom Dimensions are site specific, so maybe it fails as the segment isn't available globally. We need to investigate that.
As a workaround: Did you try running the invalidation with --sites=1, to limit it to the site where it is available?
I've quickly reproduced that locally. It is indeed a problem around the segment not being available globally, but only site specific.
We could rework the command to automatically skip sites where a segment is not available and print an info in that case.
The suggested workaround from above works as expected, so this might not need to be highest priority.
The workaround looks promising!
I'll continue to update some sites and if I hit any roadblocks, I'll update here.
I'm not sure, but I'm guessing that there's a bug with how custom dimensions are used in segments.
I believe that in the segment, it's looking for the custom dimension by the id, but when the custom dimension log table is created, it's looking for the index.
Because I just ran:
./console core:invalidate-report-data --dates=2024-02-01,2024-05-28 --sites=30 --segment=15
./console core:archive --force-idsegments=15 --force-idsites=30And all of a sudden, another custom dimension is being used in the segment.
Could this be the issue?
File: matomo/plugins/CustomDimensions/Dao/LogTable.php
public static function buildCustomDimensionColumnName($indexOrDimension)
{
if (is_array($indexOrDimension) && isset($indexOrDimension['index'])) { // <-- indexOrDimension
$indexOrDimension = $indexOrDimension['index'];
}
$indexOrDimension = (int) $indexOrDimension;
if ($indexOrDimension >= 1) {
return 'custom_dimension_' . (int) $indexOrDimension;
}
}File: matomo/plugins/CustomDimensions/DataTable/Filter/AddSegmentMetadata.php
/**
* @param DataTable $table
*/
public function filter($table)
{
$dimension = CustomDimensionsRequestProcessor::buildCustomDimensionTrackingApiName($this->idDimension);
foreach ($table->getRows() as $row) {
$label = $row->getColumn('label');
if ($label !== false) {
if ($label === Archiver::LABEL_CUSTOM_VALUE_NOT_DEFINED) {
$label = '';
}
$row->setMetadata('segment', $dimension . '==' . urlencode($label));
}
$subTable = $row->getSubtable();
if ($subTable) {
$subTable->filter('Piwik\Plugins\CustomDimensions\DataTable\Filter\AddSubtableSegmentMetadata', array($this->idDimension, $label));
}
}
}