Enable Aurora Enhanced Monitoring on existing instances - "IAM role invalid or wrong perms", followed by "MonitoringRoleARN value is required"
hashibot opened this issue · comments
This issue was originally opened by @ssm1th as hashicorp/terraform#9435. It was migrated here as part of the provider split. The original body of the issue is below.
Similar issue as @deanmraz reported in hashicorp/terraform#5455 I think.
Attempting to enable Enhanced Monitoring on existing Aurora instances initially throws the following error:
InvalidParameterValue: IAM role ARN value is invalid or does not include the required permissions for: ENHANCED_MONITORING
When attempting an apply again following that error, I'm getting:
InvalidParameterCombination: A MonitoringRoleARN value is required if you specify a MonitoringInterval value other than 0
I suspect the latter is being thrown because the first apply actually added monitoring_role_arn to tfstate.
Have tried using the existing AWS provided managed policy and attaching it to a Terraform created role (all within Terraform), and I've also tried creating my own policy as per hashicorp/terraform#5455 and attaching that to a self made role too, but the behaviour is still the same.
The addition of monitoring_role_arn to tfstate following the first error sounds like a bug to me, but does anybody have any ideas or suggestions on how to get around this and get it working?
Thanks.
Did this get addressed somehow? I am currently facing the exact same issues reported above.
+1 Also running into this.
I'm going to try the solution here: hashicorp/terraform#5455
but it would be nice to consistently use "aws_iam_policy_document" throughout
@helenes-r7 Did the solution you linked above work for you? It did not for me.
I'm getting the same issue as well. My monitoring role exists, my remote TF state says my monitoring interval is > 0 with the monitoring role correctly set. However, during the apply, the deploy fails with the "MonitoringRoleARN value is required" error. I turned on DEBUG output for the apply and noticed the POST call to the RDS API only has the MonitoringInterval parameter set to my value, but the MonitoringRoleARN is nowhere to be found. This happens even when I explicitly set the aws_db_instance.monitoring_role_arn with the actual value. Anybody find a solution for this?
Hello @terrysdev9 ,
I also have the exact same issue for MySQL.
It's like the variable is not populated...
I got InvalidParameterCombination: A MonitoringRoleARN value is required if you specify a MonitoringInterval value other than 0.
I ended up modifiying the DB instance by hand via AWS Console.
Here's my config:
# IAM Role Definition for RDS Enhanced Monitoring
data "aws_iam_policy_document" "rds-assume-role-policy" {
statement {
actions = ["sts:AssumeRole"]
principals {
type = "Service"
identifiers = ["monitoring.rds.amazonaws.com"]
}
}
}
# Create RDS Enhanced Monitoring role
resource "aws_iam_role" "rds_monitoring_all_iam_role" {
name = "${var.project}-role-all-rds-monitor"
path = "/"
assume_role_policy = "${data.aws_iam_policy_document.rds-assume-role-policy.json}"
}
# Attach RDS Enhanced Monitoring role policy(ies)
resource "aws_iam_role_policy_attachment" "rds_monitoring_all_iam_role_policies_attach" {
role = "${aws_iam_role.rds_monitoring_all_iam_role.id}"
policy_arn = "arn:aws:iam::aws:policy/service-role/AmazonRDSEnhancedMonitoringRole"
}
resource "aws_db_instance" "db_pre_za" {
...
monitoring_interval = 60
monitoring_role_arn = "${aws_iam_role.rds_monitoring_iam_role.arn}"
...
}
Hi @hegyre,
I ended up re-creating the RDS instance with the monitoring role ARN set, but my monitoring interval remained at 0. I was then able to change the interval to some value >0 and back to 0 with no issues. If you changed the DB instance via the console, did your refresh on your TF state and do a plan to see if it picked up your changes?
The refresh picked up the changes I manually did on the DB instance:
"monitoring_interval": "60",
"monitoring_role_arn": "arn:aws:iam::200274800666:role/dcsie-role-all-rds-monitor",
Howerver, since I have commented these in the tf file:
# monitoring_interval = 60
# monitoring_role_arn = "${aws_iam_role.rds_monitoring_all_iam_role.arn}"
Then the plan would like to set it to 0, but only for the monitoring_interval, not for the monitoring_role_arn:
Terraform will perform the following actions:
~ aws_db_instance.db_pre_za
monitoring_interval: "60" => "0"
Now If I uncomment these, the plan is fine
To summup:
This is not working if the RDS instance has been created before enabing this monitoring option.
I have to test this with a new DB AND this monitoring option enabled during RDS instance creation time.
In our experience, we have an RDS instance that we created via terraform a few weeks ago. When we tried adding enhanced monitoring by setting monitoring_interval and monitoring_role_arn at the same time, terraform failed with the error FIXME. Subsequent runs always failed with InvalidParameterCombination: A MonitoringRoleARN value is required if you specify a MonitoringInterval value other than 0., unless we modified the tfstate file to forget about the monitoring_* properties.
The initial failure we suspect is due to either the role not being created properly before its ARN was applied to the RDS instance (is that even possible?), or that the assumerole policy of our role was using the service rds.amazonaws.com instead of monitoring.rds.amazonaws.com.
Once we had role created with monitoring.rds.amazonaws.com as our AssumeRolePolicy Service, and removed the monitoring settings from our tfstate, terraform was able to successfully add enhanced monitoring to the RDS instance.
@geofffranks Thanks!!! We found same error 11.7, and soleved it your solution.
We ran into this today. It was quite aggravating. Rather than modify the tfstate directly we created a second, temporary rule, assigned monitoring_role_arn to that and then reverted to our original role that had formerly had rds.amazonaws.com and since changed to monitoring.rds.amazonaws.com
Terraform v0.11.7
provider.aws v1.41.0
Yeah, got the same issue. There's two problems here:
The first issue is a race condition problem: the RDS declaration has a direct dependency on the monitoring role, so as soon as the role is available, the modification of the RDS is fired. So the modification of the RDS (adding the monitoring role) and the aws_iam_role_policy_attachment are happening at the same time. RDS is checking if the role has indeed the right to use enhanced monitoring, but at that moment, that's not the case, hence the first error.
I tried to fix this issue by forcing a dependency on aws_rds_instance to aws_iam_role_policy_attachment but in my tests, even if the calls are done sequentially, they happen so fast that RDS is still checking too soon if the role has the correct right, leading to the same error.
Here's a breakdown of first run of TF, with the manual dependency on aws_iam_role_policy_attachment:
- request creation of the role @ 2018-11-09T12:40:35.128+0100
- answer HTTP 200 @ 2018-11-09T12:40:35.813+0100
- request attachment policy
AmazonRDSEnhancedMonitoringRole@ 2018-11-09T12:40:36.481+0100 - answer HTTP 200 @ 2018-11-09T12:40:37.188+0100
- request
iam/ListAttachedRolePolicies(I suppose to confirm that the attachment is correctly done) @ 2018-11-09T12:40:37.848+0100 - answer HTTP 200 with the policy
AmazonRDSEnhancedMonitoringRolelisted @ 2018-11-09T12:40:38.495+0100 - request to modify the DB instance and add the enhanced monitoring @ 2018-11-09T12:40:38.509+0100
- answer HTTP 400
Bad Request@ 2018-11-09T12:40:39.498+0100. Reason isIAM role ARN value is invalid or does not include the required permissions for: ENHANCED_MONITORING
As you can see, there's less than 2s between the confirmation that the policy is correctly attached, and the call to modify the instance with the call. I suspect some cache on AWS side, that is not refreshed when we make the call to modify the RDS. Policy attachment is clearly done when we initiate the DB modify API call.
The second problem is a Terraform bug: whatever the outcome of the first call, the monitoring_role_arn attribute is set to the ARN, so next calls are only pushing the monitoring_interval. As a consequence, the monitoring ARN is not sent in the AWS API call, making the request incorrect.
TL;DR: this is the combination of two bugs, one on AWS side and one on TF side.
I've opened an issue on AWS side for this. I'll keep you guys posted, but looking at this forum thread, it's very likely that IAM information is being propagated but haven't reached yet the RDS API.
In that case, it means that TF may need to retry the RDS call a few times to see if the RDS api finally get the correct answer.
Got the confirmation from AWS support that there's indeed a propagation delay between the IAM call and the propagation to all AWS services.
So I think that the AWS provider needs to be updated in two ways:
- correct the bug that fill the
monitoring_role_arnif themodify-db-instancecall is not successful. There's no valid reason to update the state if the call is unsuccessful. - Try multiple times the
modify-db-instancecall when activating the enhanced monitoring on an RDS / Aurora instance. Each call should be made with a 5s sleep between them, in order to allow the IAM modification to correctly propagate and then, allow themodify-db-instanceto successfully validate that the role being passed has indeed the correct set of rights to collect metrics for the enhanced monitoring.
It's worth mentioning that, if the first bug is solved, then the second is less critical, as TF will fail the first time, but will succeed at the next run. Not pretty, but this can work.
Thanks,
I can confirm that if done as @geofffranks described in here it works.
Doing a two stage apply by creating the role first and then updating the RDS instance with the monitoring_role_arn and monitoring_interval works.
Also facing this issue curently. When aws is handling this?
I'm encountering the same exact issue as @hegyre. I have an existing Postgres 10 instance and when I apply the monitoring_role_arn it never gets set which results in the exception.
InvalidParameterCombination: A MonitoringRoleARN value is required if you specify a MonitoringInterval value other than 0.
What's the status on this issue?
In my case I'm observing this error when
1st apply)
- create the DB instance
2nd apply) - add monitoring role
3rd apply) - increase monitoring interval
~ monitoring_interval = 0 -> 30
monitoring_role_arn = "arn:aws:iam::xxxxxxxx:role/rds-enhanced-monitoring-role"
Fix submitted: #9747
The fix for this has been merged and will release in version 2.25.0 of the Terraform AWS Provider, in about a week.
This has been released in version 2.25.0 of the Terraform AWS provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.
For further feature requests or bug reports with this functionality, please create a new GitHub issue following the template for triage. Thanks!
I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thanks!