Select indexes slow performance, doesn't use indexes parameter

[Dipesh Bhundia]

Description of the issue

The index optimize procedure when updating statistics can be very slow on large databases with lots of objects. The two attached sections will try and get all of the statistics information for every stat/column/table in the database even if you use the index parameter.

SELECT [IndexesStatistics].[SchemaID] , [IndexesStatistics].[SchemaName] , [IndexesStatistics].[ObjectID] , [IndexesStatistics].[ObjectName] , [IndexesStatistics].[ObjectType] , [IndexesStatistics].[IsMemoryOptimized] , [IndexesStatistics].[IndexID] , [IndexesStatistics].[IndexName] , [IndexesStatistics].[IndexType] , [IndexesStatistics].[AllowPageLocks] , [IndexesStatistics].[IsImageText] , [IndexesStatistics].[IsNewLOB] , [IndexesStatistics].[IsFileStream] , [IndexesStatistics].[IsColumnStore] , [IndexesStatistics].[IsComputed] , [IndexesStatistics].[IsTimestamp] , [IndexesStatistics].[OnReadOnlyFileGroup] , [IndexesStatistics].[ResumableIndexOperation] , [IndexesStatistics].[StatisticsID] , [IndexesStatistics].[StatisticsName] , [IndexesStatistics].[NoRecompute] , [IndexesStatistics].[IsIncremental] , [IndexesStatistics].[PartitionID] , [IndexesStatistics].[PartitionNumber] , [IndexesStatistics].[PartitionCount] , [IndexesStatistics].[Order] , [IndexesStatistics].[Selected] , [IndexesStatistics].[Completed] FROM ( SELECT [SchemaID] = [schemas].[schema_id] , [SchemaName] = [schemas].[name] , [ObjectID] = [objects].[object_id] , [ObjectName] = [objects].[name] , [ObjectType] = RTRIM ([objects].[type]) , [IsMemoryOptimized] = [tables].[is_memory_optimized] , [IndexID] = NULL , [IndexName] = NULL , [IndexType] = NULL , [AllowPageLocks] = NULL , [IsImageText] = NULL , [IsNewLOB] = NULL , [IsFileStream] = NULL , [IsColumnStore] = NULL , [IsComputed] = NULL , [IsTimestamp] = NULL , [OnReadOnlyFileGroup] = NULL , [ResumableIndexOperation] = NULL , [StatisticsID] = [stats].[stats_id] , [StatisticsName] = [stats].[name] , [NoRecompute] = [stats].[no_recompute] , [IsIncremental] = [stats].[is_incremental] , [PartitionID] = NULL , [PartitionNumber] = [dm_db_incremental_stats_properties].[partition_number] , [PartitionCount] = NULL , [Order] = 0 , [Selected] = 0 , [Completed] = 0 FROM [sys].[stats] [stats] INNER JOIN [sys].[objects] [objects] ON [stats].[object_id] = [objects].[object_id] INNER JOIN [sys].[schemas] [schemas] ON [objects].[schema_id] = [schemas].[schema_id] LEFT OUTER JOIN [sys].[tables] [tables] ON [objects].[object_id] = [tables].[object_id] OUTER APPLY [sys].dm_db_incremental_stats_properties ([stats].[object_id], [stats].[stats_id]) [dm_db_incremental_stats_properties] WHERE [objects].[type] IN ('U', 'V') AND [objects].[is_ms_shipped] = 0 AND NOT EXISTS (SELECT * FROM [sys].[indexes] [indexes] WHERE [indexes].[object_id] = [stats].[object_id] AND [indexes].[index_id] = [stats].[stats_id]) ) [IndexesStatistics];

SQL Server version and edition
Microsoft SQL Server 2019 (RTM-CU16) (KB5011644) - 15.0.4223.1 (X64) Apr 11 2022 16:24:07 Copyright (C) 2019 Microsoft Corporation Developer Edition (64-bit) on Windows Server 2016 Datacenter 10.0 (Build 14393: ) (Hypervisor)

Version of the script
2022-01-02 13:58:13

What command are you executing?

EXECUTE [dbo].[IndexOptimize]
@databases = 'DataWarehouse',
@FragmentationLow = NULL,
@FragmentationMedium = NULL,
@FragmentationHigh = NULL,
@UpdateStatistics = 'COLUMNS',
@StatisticsSample = 40,
@Timelimit = 59400,
@LogToTable = 'Y',
@StatisticsModificationLevel = 20,
@indexes = 'Datawarehouse.datawarehouse.FactSales'

What output are you getting?

Takes a very long time to start actually updating statistics as the two screenshotted commands take a long time on a large database

[yobble]

I have the same problem. On one particular database which has a huge number of tables and indexes (most of them empty) - using indexOptimize to update statistics takes 6 hrs (whereas a much older version of indexOptimize runs in 20 mins). And as you say most of the time is spent in that select query, and also as you say it does not matter if you specify just one index (with the @indexes parameter).

[kuijperserik]

We also had this problem on SQL 2016, which then could be fixed by replacing "UNION" statement with "UNION ALL"'.
See information on other topic: #162
After upgrading to SQL 2019 we experience the same long running times. Upgrading the tools to latest one didn't help, also the above mentioned work-around does not help anymore.
Our database contains 360.000 tables and it takes 22 hours to analyze the data, before it actually start updating stats.

[kristinlk]

We are encountering the same issue again on SQL 2019 and using the most recent IndexOptimize procedure. In SQL 2016 the job took 16 hours at it's slowest. However on SQL 2019, using the same database, it takes 2 days to complete, with or without the nonclustered index. The database has over 330K tables and 700K indexes. Has anyone else encountered a similar problem?

[kristinlk]

Using the workaround listed here, I updated the most recent IndexOptimize procedure and the slow query now executes in less than 5 minutes for our database with 330K tables and 700K indexes.
indexoptimize_sql2019_updated.zip