I have created an RDS Proxy using Terraform. However, it does not seem to be working.
My application code cannot connect to the proxy (timeout) and aws rds describe-db-proxy-targets
gives the following:
{
"Targets": [
{
"Endpoint": "mydb.aaaaaaaaaaaa.eu-west-2.rds.amazonaws.com",
"RdsResourceId": "mydb",
"Port": 5432,
"Type": "RDS_INSTANCE",
"TargetHealth": {
"State": "UNAVAILABLE",
"Description": "DBProxy Target unavailable due to an internal error"
}
}
]
}
How can I go about debugging this?
Here is the Terraform script for the proxy. The RDS instance is described elsewhere, but is working.
data "aws_subnet" "mydb_rds" {
filter {
name = "availability-zone"
values = [ aws_db_instance.mydb.availability_zone ]
}
}
resource "aws_secretsmanager_secret" "mydb_rds_proxy" {
name = "mydb-rds-proxy"
}
resource "aws_secretsmanager_secret_version" "mydb_rds_proxy" {
secret_id = aws_secretsmanager_secret.mydb_rds_proxy.id
secret_string = var.db_password
}
resource "aws_iam_role" "mydb_rds_proxy" {
name = "mydb-rds-proxy"
assume_role_policy = <<EOF
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "",
"Action": "sts:AssumeRole",
"Effect": "Allow",
"Principal": {
"Service": "rds.amazonaws.com"
}
}
]
}
EOF
}
resource "aws_iam_policy" "mydb_rds_proxy_policy" {
name = "mydb-rds-proxy"
policy = <<EOF
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "GetSecretValue",
"Action": [
"secretsmanager:GetSecretValue"
],
"Effect": "Allow",
"Resource": [
"${aws_secretsmanager_secret.mydb_rds_proxy.arn}"
]
},
{
"Sid": "DecryptSecretValue",
"Action": [
"kms:Decrypt"
],
"Effect": "Allow",
"Resource": [
"${aws_secretsmanager_secret.mydb_rds_proxy.arn}"
]
}
]
}
EOF
}
resource "aws_iam_role_policy_attachment" "mydb_rds_proxy_policy_attachment" {
role = aws_iam_role.mydb_rds_proxy.name
policy_arn = aws_iam_policy.mydb_rds_proxy_policy.arn
}
resource "aws_db_proxy" "mydb" {
name = "mydb-rds-proxy"
debug_logging = false
engine_family = "POSTGRESQL"
idle_client_timeout = 1800
require_tls = true
role_arn = aws_iam_role.mydb_rds_proxy.arn
vpc_security_group_ids = [ aws_security_group.mydb_rds.id ]
vpc_subnet_ids = [
data.aws_subnet.mydb_rds.id,
aws_default_subnet.subnet_a.id,
aws_default_subnet.subnet_b.id
]
auth {
auth_scheme = "SECRETS"
iam_auth = "DISABLED"
secret_arn = aws_secretsmanager_secret.mydb_rds_proxy.arn
}
}
resource "aws_db_proxy_default_target_group" "mydb" {
db_proxy_name = aws_db_proxy.mydb.name
connection_pool_config {
connection_borrow_timeout = 120
max_connections_percent = 100
max_idle_connections_percent = 50
}
}
resource "aws_db_proxy_target" "mydb" {
db_instance_identifier = aws_db_instance.mydb.id
db_proxy_name = aws_db_proxy.mydb.name
target_group_name = aws_db_proxy_default_target_group.mydb.name
}
locals {
proxied_pg_connection_string = "postgres://${aws_db_instance.mydb.username}:${var.db_password}@${aws_db_proxy.mydb.endpoint}:5432/postgres?client_encoding=UTF8"
}
There are several things you need to get right for this to work:
A useful debugging query is:
aws rds describe-db-proxy-targets --db-proxy-name <proxy-name>
To understand the error message it gives back, see this page.
The username / password is the hardest thing to discover, since Terraform does not support it yet. What you need to do is construct a JSON string in Terraform that matches what RDS Proxy can understand:
resource "aws_secretsmanager_secret_version" "my_db_proxy" {
secret_id = aws_secretsmanager_secret.my_db_proxy.id
secret_string = jsonencode({
"username" = aws_db_instance.my_db.username
"password" = var.db_password
"engine" = "postgres"
"host" = aws_db_instance.my_db.address
"port" = 5432
"dbInstanceIdentifier" = aws_db_instance.my_db.id
})
}
You then need ensure these security group rules allowing TCP traffic on port 5432
(for Postgres) exist:
ingress
Lambda to RDS Proxyingress
RDS Proxy to RDSegress
RDS Proxy to "0.0.0.0/0"
The RDS Proxy role should have a policy like this:
resource "aws_iam_policy" "my_rds_proxy_policy" {
name = "my-rds-proxy"
policy = <<EOF
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "",
"Action": [
"rds:*"
],
"Effect": "Allow",
"Resource": [
"${aws_db_instance.my_db.arn}"
]
},
{
"Sid": "GetSecretValue",
"Action": [
"secretsmanager:GetSecretValue"
],
"Effect": "Allow",
"Resource": [
"${aws_secretsmanager_secret.my_rds_proxy.arn}"
]
},
{
"Sid": "DecryptSecretValue",
"Action": [
"kms:Decrypt"
],
"Effect": "Allow",
"Resource": [
"*"
]
},
{
"Sid": "DecryptKms",
"Effect": "Allow",
"Action": "kms:Decrypt",
"Resource": "*",
"Condition": {
"StringEquals": {
"kms:ViaService": "secretsmanager.${var.aws_region}.amazonaws.com"
}
}
}
]
}
EOF
}
Good luck!
Using
describe-db-proxy-targets
is very helpful and can give you the answer you need in some cases. I was getting "DBProxy Target unavailable due to an internal error" as described in the question. Very frustrating! In my case it was caused by having an incorrect security group setup that caused the endpoint to not have access to the target. In my testing I thought using the same security group on the proxy as the RDS cluster made sense, only I forgot that group didn't allow access from its own group. Hopefully AWS can catch this error in the future and improve the error messaging.