query on turbot_active_grant - running out of memory in Linux

Question

query on turbot_active_grant - running out of memory in Linux

nivethj opened this issue 2 years ago · comments

Describe the bug
A clear and concise description of what the bug is.

I am trying to query turbot_active_grant with limit of 10 records in my turbot v5 environment which has 20+k records . i am getting out of memory error . I have 4GB Mem. I tried using latest Steampipe v0.16.0-rc.8 version . but no luck.

Steampipe version (steampipe -v) v0.16.0-rc.8
Example: v0.3.0

Plugin version (steampipe plugin list) 0.5.0
Example: v0.5.0

To reproduce
Steps to reproduce the behavior (please include relevant code and/or commands).

run below query in amazon linux 2

select grant_id,
resource_id,
identity_profile_id,
identity_display_name,
identity_email,
identity_status,
level_uri,
resource_type_uri,
workspace,
create_timestamp
from nonprod.turbot_active_grant
where resource_trunk_title = 'turbot'
and level_trunk_title = 'superuser'
limit 10;

Expected behavior
A clear and concise description of what you expected to happen.

retrun limited rows in few seconds

Additional context
Add any other context about the problem here.

David Boeke · Answer 1 · Fri Aug 12 2022 02:59:54 GMT+0800 (China Standard Time)

I find when I do this query on an environment with user mode enabled that it results in many duplicate records. I wonder of that is the underlying cause of the issue?

David Boeke · Answer 2 · Fri Aug 12 2022 03:00:31 GMT+0800 (China Standard Time)

select 
  grant_id,
  resource_id,
  identity_profile_id,
  identity_display_name,
  identity_email,
  identity_status,
  level_uri,
  resource_type_uri,
  workspace,
  create_timestamp,
  resource_trunk_title
from 
  turbot_active_grant
where 
  resource_type_uri = 'tmod:@turbot/turbot#/resource/types/turbot'
  and level_uri = 'tmod:@turbot/turbot-iam#/permission/levels/superuser'

David Boeke · Answer 3 · Fri Aug 12 2022 03:01:14 GMT+0800 (China Standard Time)

both of the where constraints should hit indexes

Ved misra · Answer 4 · Fri Aug 19 2022 20:43:22 GMT+0800 (China Standard Time)

Hello @nivethj, I tried to replicate the issue but without much success.

To replicate the issue, could you please provide a few more details -

How many plugins other than turbot are you using in the Amazon Linux 2 environment?
Do other queries work or is it only the turbot_active_grant table queries which are failing?
If you restart steampipe and rerun the query, do you still see the issue?

nivethj · Answer 5 · Thu Sep 08 2022 03:12:37 GMT+0800 (China Standard Time)

Below are the requested details

How many plugins other than turbot are you using in the Amazon Linux 2 environment?
I’m only using only turbot plugin. I have AWS plugin installed , but not using it.

Do other queries work or is it only the turbot_active_grant table queries which are failing?
I haven’t worked on other tables . turbot_active_grant is working in sandbox which has few rows (<20 rows)

If you restart steampipe and rerun the query, do you still see the issue?
Restart is not helping . query runs for 15 mins and end with error

Ved misra · Answer 6 · Tue Sep 13 2022 13:51:31 GMT+0800 (China Standard Time)

Hey @nivethj, sorry for the delayed response.

Sorry to see that you are still blocked with this issue. We released a new version of the turbot plugin (v0.9.0) which includes essential memory management fixes.

Could you please upgrade the plugin and check if the issue persists?

cbruno10 · Answer 7 · Tue Sep 13 2022 20:16:29 GMT+0800 (China Standard Time)

@nivethj To add onto @misraved 's comment, can you please also ensure that you're running Steampipe CLI v0.16.1? This CLI has some updates to allow it to better handle the memory management improvements mentioned. Thanks!

nivethj · Answer 8 · Tue Sep 20 2022 04:11:29 GMT+0800 (China Standard Time)

updated steampipe to (steampipe version 0.16.3). turbot plugin to 0.9.0.

row limit 10 is working with 60% Mem , when i update row limit 100 , max memory reached

github-actions · Answer 9 · Thu Oct 20 2022 07:40:39 GMT+0800 (China Standard Time)

'This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 30 days.'

nivethj · Answer 10 · Tue Oct 25 2022 22:19:54 GMT+0800 (China Standard Time)

im following this request

Ved misra · Answer 11 · Wed Nov 02 2022 14:21:50 GMT+0800 (China Standard Time)

Hello @nivethj, we are extremely sorry for the delay.

While running the query, could you please grab the plugin logs for us to investigate? I tried to replicate this error but without any success.

Hoping to catch something unusual in those logs which might help us in debugging this issue.

nivethj · Answer 12 · Wed Nov 30 2022 21:18:11 GMT+0800 (China Standard Time)

It ran long and timed out.

Console

limit 10;
Error: Post "https://server/api/latest/graphql": net/http: TLS handshake timeout (SQLSTATE HV000)

Log file

2022-11-29 19:59:22.753 UTC [16105] STATEMENT: select grant_id,
resource_id,
identity_profile_id,
identity_display_name,
identity_email,
identity_status,
level_uri,
resource_type_uri,
workspace,
create_timestamp
from nonprod.turbot_active_grant
where resource_trunk_title = 'turbot'
and level_trunk_title = 'superuser'
limit 10;
2022-11-29 20:00:38.460 UTC [16105] LOG: disconnection: session time: 3:58:52.816 user=steampipe database=steampipe host=127.0.0.1 port=37158

Ved misra · Answer 13 · Tue May 16 2023 21:34:48 GMT+0800 (China Standard Time)

Apologies for the radio silence on this issue @nivethj!!

Are you still seeing this error with the new v0.19.5 CLI and the latest turbot plugin versions?

nivethj · Answer 14 · Tue Jun 06 2023 20:27:58 GMT+0800 (China Standard Time)

tried on v0.20.5 . still the same issue

Welcome to Steampipe v0.20.5
For more information, type .help
steampipe-postgres-fdw updated to 1.7.2.

=> Same query used

net/http: TLS handshake timeout (SQLSTATE HV000)

souravTurbot · Answer 15 · Fri Jun 30 2023 20:10:30 GMT+0800 (China Standard Time)

Hello @nivethj, I attempted to replicate the issue, but unfortunately, I couldn't succeed. Could you kindly clarify whether by 20+k records you are referring to 20+k resources or active grants? If it pertains to 20+k resources, please provide me with the number of active grants you currently have.

souravTurbot · Answer 16 · Mon Jul 03 2023 15:31:16 GMT+0800 (China Standard Time)

We have tested in a similar env with 20+K active grants, with limits 10, 100, 1000 - (Works fine)

Welcome to Steampipe v0.20.7
For more information, type .help
> select grant_id,
resource_id,
identity_status,
level_uri,
resource_type_uri,
workspace,
create_timestamp
from turbot_active_grant
limit 10;
+-----------------+-----------------------------------------------------+--------------------------------------------+----------------------+
| identity_status | level_uri                                           | resource_type_uri                          | create_timestamp     |
+-----------------+-----------------------------------------------------+--------------------------------------------+----------------------+
| Active          | tmod:@turbot/turbot-iam#/permission/levels/owner    | tmod:@turbot/turbot#/resource/types/turbot | 2021-06-15T17:50:36Z |
| Active          | tmod:@turbot/turbot-iam#/permission/levels/admin    | tmod:@turbot/turbot#/resource/types/turbot | 2022-02-22T07:39:50Z |
| Active          | tmod:@turbot/turbot-iam#/permission/levels/metadata | tmod:@turbot/turbot#/resource/types/folder | 2022-02-22T11:01:09Z |
| Active          | tmod:@turbot/turbot-iam#/permission/levels/metadata | tmod:@turbot/turbot#/resource/types/folder | 2022-02-22T11:01:09Z |
| Active          | tmod:@turbot/turbot-iam#/permission/levels/owner    | tmod:@turbot/turbot#/resource/types/turbot | 2021-08-20T11:21:59Z |
| Active          | tmod:@turbot/turbot-iam#/permission/levels/metadata | tmod:@turbot/turbot#/resource/types/folder | 2022-02-22T11:01:09Z |
| Active          | tmod:@turbot/turbot-iam#/permission/levels/owner    | tmod:@turbot/turbot#/resource/types/turbot | 2022-02-03T18:54:05Z |
| Active          | tmod:@turbot/turbot-iam#/permission/levels/owner    | tmod:@turbot/turbot#/resource/types/turbot | 2021-10-19T19:52:26Z |
| Active          | tmod:@turbot/turbot-iam#/permission/levels/metadata | tmod:@turbot/turbot#/resource/types/folder | 2022-02-22T11:01:09Z |
| Active          | tmod:@turbot/turbot-iam#/permission/levels/owner    | tmod:@turbot/turbot#/resource/types/turbot | 2022-01-27T02:56:21Z |
+-----------------+-----------------------------------------------------+--------------------------------------------+----------------------+

 select
identity_status,
level_uri,
resource_type_uri,
create_timestamp
from turbot_active_grant
limit 100;
+-----------------+-----------------------------------------------------+--------------------------------------------+----------------------+
| identity_status | level_uri                                           | resource_type_uri                          | create_timestamp     |
+-----------------+-----------------------------------------------------+--------------------------------------------+----------------------+
| Inactive        | tmod:@turbot/turbot-iam#/permission/levels/metadata | tmod:@turbot/turbot#/resource/types/folder | 2022-02-22T11:01:10Z |
| Active          | tmod:@turbot/turbot-iam#/permission/levels/admin    | tmod:@turbot/aws#/resource/types/account   | 2022-02-26T07:56:14Z |
| Active          | tmod:@turbot/turbot-iam#/permission/levels/owner    | tmod:@turbot/turbot#/resource/types/turbot | 2021-06-15T17:50:36Z |
| Active          | tmod:@turbot/turbot-iam#/permission/levels/metadata | tmod:@turbot/turbot#/resource/types/folder | 2022-02-22T11:01:11Z |
| Active          | tmod:@turbot/turbot-iam#/permission/levels/owner    | tmod:@turbot/turbot#/resource/types/turbot | 2021-08-20T11:21:59Z |
| Active          | tmod:@turbot/turbot-iam#/permission/levels/metadata | tmod:@turbot/turbot#/resource/types/folder | 2022-02-22T11:01:11Z |
| Active          | tmod:@turbot/turbot-iam#/permission/levels/owner    | tmod:@turbot/turbot#/resource/types/turbot | 2021-10-19T19:52:26Z |
| Active          | tmod:@turbot/turbot-iam#/permission/levels/metadata | tmod:@turbot/turbot#/resource/types/folder | 2022-02-22T11:01:11Z |
| Active          | tmod:@turbot/turbot-iam#/permission/levels/owner    | tmod:@turbot/turbot#/resource/types/turbot | 2022-01-27T02:56:21Z |
| Active          | tmod:@turbot/turbot-iam#/permission/levels/metadata | tmod:@turbot/turbot#/resource/types/folder | 2022-02-22T11:01:11Z |
| Active          | tmod:@turbot/turbot-iam#/permission/levels/owner    | tmod:@turbot/turbot#/resource/types/turbot | 2022-02-03T18:54:05Z |
| Active          | tmod:@turbot/turbot-iam#/permission/levels/metadata | tmod:@turbot/turbot#/resource/types/folder | 2022-02-22T11:01:11Z |
| Active          | tmod:@turbot/turbot-iam#/permission/levels/admin    | tmod:@turbot/turbot#/resource/types/turbot | 2022-02-22T07:39:50Z |
| Active          | tmod:@turbot/turbot-iam#/permission/levels/metadata | tmod:@turbot/turbot#/resource/types/folder | 2022-02-22T11:01:11Z |
| Active          | tmod:@turbot/turbot-iam#/permission/levels/metadata | tmod:@turbot/turbot#/resource/types/folder | 2022-02-22T11:01:09Z |
| Active          | tmod:@turbot/turbot-iam#/permission/levels/metadata | tmod:@turbot/turbot#/resource/types/folder | 2022-02-22T11:01:11Z |
| Active          | tmod:@turbot/turbot-iam#/permission/levels/metadata | tmod:@turbot/turbot#/resource/types/folder | 2022-02-22T11:01:09Z |
| Inactive        | tmod:@turbot/turbot-iam#/permission/levels/metadata | tmod:@turbot/turbot#/resource/types/folder | 2022-02-22T11:01:11Z |
| Active          | tmod:@turbot/turbot-iam#/permission/levels/metadata | tmod:@turbot/turbot#/resource/types/folder | 2022-02-22T11:01:09Z |
| Active          | tmod:@turbot/turbot-iam#/permission/levels/metadata | tmod:@turbot/turbot#/resource/types/folder | 2022-02-22T11:01:11Z |
| Active          | tmod:@turbot/turbot-iam#/permission/levels/metadata | tmod:@turbot/turbot#/resource/types/folder | 2022-02-22T11:01:09Z |
| Active          | tmod:@turbot/turbot-iam#/permission/levels/metadata | tmod:@turbot/turbot#/resource/types/folder | 2022-02-22T11:01:11Z |
| Active          | tmod:@turbot/turbot-iam#/permission/levels/metadata | tmod:@turbot/turbot#/resource/types/folder | 2022-02-22T11:01:09Z |
| Active          | tmod:@turbot/turbot-iam#/permission/levels/metadata | tmod:@turbot/turbot#/resource/types/folder | 2022-02-22T11:01:11Z |
| Active          | tmod:@turbot/turbot-iam#/permission/levels/metadata | tmod:@turbot/turbot#/resource/types/folder | 2022-02-22T11:01:09Z |
| Active          | tmod:@turbot/turbot-iam#/permission/levels/metadata | tmod:@turbot/turbot#/resource/types/folder | 2022-02-22T11:01:11Z |
| Active          | tmod:@turbot/turbot-iam#/permission/levels/owner    | tmod:@turbot/turbot#/resource/types/folder | 2022-02-22T11:01:09Z |
| Active          | tmod:@turbot/turbot-iam#/permission/levels/metadata | tmod:@turbot/turbot#/resource/types/folder | 2022-02-22T11:01:11Z |

 select
identity_status,
level_uri,
resource_type_uri,
create_timestamp
from turbot_active_grant
limit 1000;
+-----------------+-----------------------------------------------------+-----------------------------------------------+----------------------+
| identity_status | level_uri                                           | resource_type_uri                             | create_timestamp     |
+-----------------+-----------------------------------------------------+-----------------------------------------------+----------------------+
| Active          | tmod:@turbot/turbot-iam#/permission/levels/admin    | tmod:@turbot/aws#/resource/types/account      | 2022-02-26T07:56:14Z |
| Active          | tmod:@turbot/turbot-iam#/permission/levels/user     | tmod:@turbot/aws#/resource/types/account      | 2022-02-26T07:55:05Z |
| Active          | tmod:@turbot/turbot-iam#/permission/levels/owner    | tmod:@turbot/turbot#/resource/types/turbot    | 2021-06-15T17:50:36Z |
| Active          | tmod:@turbot/turbot-iam#/permission/levels/user     | tmod:@turbot/aws#/resource/types/account      | 2022-02-26T07:55:06Z |
| Active          | tmod:@turbot/turbot-iam#/permission/levels/owner    | tmod:@turbot/turbot#/resource/types/turbot    | 2021-08-20T11:21:59Z |
| Active          | tmod:@turbot/turbot-iam#/permission/levels/owner    | tmod:@turbot/turbot#/resource/types/turbot    | 2021-10-19T19:52:26Z |
| Active          | tmod:@turbot/turbot-iam#/permission/levels/owner    | tmod:@turbot/turbot#/resource/types/turbot    | 2022-01-27T02:56:21Z |
| Active          | tmod:@turbot/turbot-iam#/permission/levels/admin    | tmod:@turbot/aws#/resource/types/account      | 2022-02-26T07:55:13Z |
| Active          | tmod:@turbot/turbot-iam#/permission/levels/owner    | tmod:@turbot/turbot#/resource/types/turbot    | 2022-02-03T18:54:05Z |
| Active          | tmod:@turbot/turbot-iam#/permission/levels/admin    | tmod:@turbot/aws#/resource/types/account      | 2022-02-26T07:55:13Z |
| Active          | tmod:@turbot/turbot-iam#/permission/levels/admin    | tmod:@turbot/aws#/resource/types/account      | 2022-02-26T07:55:15Z |
| Active          | tmod:@turbot/turbot-iam#/permission/levels/admin    | tmod:@turbot/turbot#/resource/types/turbot    | 2022-02-22T07:39:50Z |
| Active          | tmod:@turbot/turbot-iam#/permission/levels/metadata | tmod:@turbot/turbot#/resource/types/folder    | 2022-02-22T11:01:09Z |
| Active          | tmod:@turbot/turbot-iam#/permission/levels/admin    | tmod:@turbot/aws#/resource/types/account      | 2022-02-26T07:55:19Z |
| Active          | tmod:@turbot/turbot-iam#/permission/levels/metadata | tmod:@turbot/turbot#/resource/types/folder    | 2022-02-22T11:01:09Z |
| Active          | tmod:@turbot/turbot-iam#/permission/levels/metadata | tmod:@turbot/turbot#/resource/types/folder    | 2022-02-22T11:01:09Z |
| Active          | tmod:@turbot/turbot-iam#/permission/levels/metadata | tmod:@turbot/aws#/resource/types/account      | 2022-02-26T07:55:30Z |
| Active          | tmod:@turbot/turbot-iam#/permission/levels/owner    | tmod:@turbot/turbot#/resource/types/folder    | 2022-02-22T11:01:09Z |
| Active          | tmod:@turbot/turbot-iam#/permission/levels/admin    | tmod:@turbot/aws#/resource/types/account      | 2022-02-26T07:55:45Z |
| Active          | tmod:@turbot/turbot-iam#/permission/levels/owner    | tmod:@turbot/turbot#/resource/types/folder    | 2022-02-22T11:01:09Z |
| Active          | tmod:@turbot/turbot-iam#/permission/levels/metadata | tmod:@turbot/aws#/resource/types/account      | 2022-02-26T07:55:47Z |
| Active          | tmod:@turbot/turbot-iam#/permission/levels/admin    | tmod:@turbot/aws#/resource/types/account      | 2022-02-26T07:55:51Z |
| Active          | tmod:@turbot/turbot-iam#/permission/levels/metadata | tmod:@turbot/turbot#/resource/types/folder    | 2022-02-22T11:01:09Z |
| Active          | tmod:@turbot/turbot-iam#/permission/levels/admin    | tmod:@turbot/aws#/resource/types/account      | 2022-02-26T07:55:58Z |
| Active          | tmod:@turbot/turbot-iam#/permission/levels/metadata | tmod:@turbot/turbot#/resource/types/folder    | 2022-02-22T11:01:10Z |
| Active          | tmod:@turbot/turbot-iam#/permission/levels/admin    | tmod:@turbot/aws#/resource/types/account      | 2022-02-26T07:56:00Z |
| Active          | tmod:@turbot/turbot-iam#/permission/levels/metadata | tmod:@turbot/turbot#/resource/types/folder    | 2022-02-22T11:01:10Z |
| Active          | tmod:@turbot/turbot-iam#/permission/levels/admin    | tmod:@turbot/aws#/resource/types/account      | 2022-02-26T07:56:03Z |
| Active          | tmod:@turbot/turbot-iam#/permission/levels/metadata | tmod:@turbot/turbot#/resource/types/folder    | 2022-02-22T11:01:10Z |
| Active          | tmod:@turbot/turbot-iam#/permission/levels/admin    | tmod:@turbot/aws#/resource/types/account      | 2022-02-26T07:56:04Z |
| Active          | tmod:@turbot/turbot-iam#/permission/levels/metadata | tmod:@turbot/turbot#/resource/types/folder    | 2022-02-22T11:01:10Z |
| Active          | tmod:@turbot/turbot-iam#/permission/levels/metadata | tmod:@turbot/aws#/resource/types/account      | 2022-02-26T07:56:13Z |
| Active          | tmod:@turbot/turbot-iam#/permission/levels/metadata | tmod:@turbot/turbot#/resource/types/folder    | 2022-02-22T11:01:10Z |
| Active          | tmod:@turbot/turbot-iam#/permission/levels/user     | tmod:@turbot/aws#/resource/types/region       | 2022-02-26T07:56:13Z |
| Active          | tmod:@turbot/turbot-iam#/permission/levels/metadata | tmod:@turbot/turbot#/resource/types/folder    | 2022-02-22T11:01:10Z |
| Active          | tmod:@turbot/turbot-iam#/permission/levels/admin    | tmod:@turbot/aws#/resource/types/account      | 2022-02-26T07:56:14Z |
| Active          | tmod:@turbot/turbot-iam#/permission/levels/user     | tmod:@turbot/aws#/resource/types/account      | 2022-02-26T07:55:10Z |
| Active          | tmod:@turbot/turbot-iam#/permission/levels/admin    | tmod:@turbot/turbot#/resource/types/folder    | 2022-02-22T11:01:10Z |
| Active          | tmod:@turbot/turbot-iam#/permission/levels/admin    | tmod:@turbot/turbot#/resource/types/folder    | 2022-02-22T11:01:10Z |
| Active          | tmod:@turbot/turbot-iam#/permission/levels/metadata | tmod:@turbot/turbot#/resource/types/folder    | 2022-02-22T11:01:10Z |

Could you please update the CLI version to the latest and try?

souravTurbot · Answer 17 · Mon Jul 10 2023 11:50:41 GMT+0800 (China Standard Time)

Hey @nivethj, We are closing this issue because we have not heard from you. Please feel free to reopen the issue if you want to share or discuss anything.