It appears that something must have changed (probably on the AWS side of things), but when I’m attempting to launch the Cloudformation Template to build the entire OHDSIonAWS Stack, I’m getting an error message that states: Properties validation failed for resource RedshiftCluster with message: #/NumberOfNodes: expected type: Integer, found: String. I’ve been trying to work around this, but to no avail. Has anyone else experienced this recently and does anyone know what I should do to fix it?
I appear to have solved this problem. AWS Redshift requires that the NumRedshiftNodes parameter is passed as a number rather than a string. When this parameter is assigned, AWS automatically determines whether it will be creating a SingleNodeRedshift instance or a MultiNodeRedshift instance. The problem with this, though, is that the current form of the databases.yaml file needs to be changed. The following is the version of the YAML file that I proceeded with that allowed me to build a single node instance successfully:
# Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License"). You may not use this file except in compliance with the License.
# A copy of the License is located at
# http://aws.amazon.com/apache2.0/
# or in the "license" file accompanying this file. This file is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND,
# either express or implied. See the License for the specific language governing permissions and limitations under the License.
AWSTemplateFormatVersion: '2010-09-09'
Description: This CloudFormation Template deploys a complete OHDSI environment. It
depends on the OHDSI-VPC CloudFormation Template.
Parameters:
DatabaseType:
AllowedValues:
- "Highly Available"
- "Single Instance"
- "Serverless"
Default: "Single Instance"
Description: Specifies whether a to deploy the AWS Aurora MySQL Database in Multi-AZ configuration.
Type: String
DatabaseInstanceType:
AllowedValues:
- db.t3.medium
- db.r4.large
- db.r4.xlarge
- db.r4.2xlarge
- db.r4.4xlarge
- db.r4.8xlarge
- db.r4.16xlarge
ConstraintDescription: Must be a valid RDS instance class.
Default: db.r4.large
Description: The Amazon RDS database instance class.
Type: String
DatabaseMasterPassword:
Description: Must be letters (upper or lower), numbers, spaces, and these special characters `~#$%^&*()_+,-
Type: String
NoEcho: true
AllowedPattern: ^([a-zA-Z0-9`~#$%^&*()_+,\\-])*$
ConstraintDescription: The Amazon RDS master password. Letters, numbers, spaces, and these special characters `~#$%^&*()_+,-
RedshiftInstanceType:
Type: String
Description: DC instance types provide faster, but smaller storage. DS instance types provide larger, but slower storage.
AllowedValues:
- dc2.large
- dc2.8xlarge
- ds2.xlarge
- ds2.8xlarge
- dc1.large
Default: dc2.large
NumRedshiftNodes:
Default: 1
Description: Specifies the number of nodes in your Redshift cluster.
Type: Number
SubnetDataA:
Type: AWS::EC2::Subnet::Id
SubnetDataB:
Type: AWS::EC2::Subnet::Id
SGData:
Type: AWS::EC2::SecurityGroup::Id
RSRoleArn:
Type: String
Conditions:
DeployMultiAZDB:
!Equals [ "Highly Available", !Ref DatabaseType ]
SingleNodeRedshift:
!Equals [ 1, !Ref NumRedshiftNodes ]
MultiNodeRedshift:
!Not [ !Equals [ 1, !Ref NumRedshiftNodes ] ]
Serverless:
!Equals [ "Serverless", !Ref DatabaseType ]
NotServerless:
!Not [ !Equals [ "Serverless", !Ref DatabaseType ] ]
Resources:
# Deploys the RDS Aurora Postgres cluster used to store the application data for WebAPI.
RDSCluster:
Type: AWS::RDS::DBCluster
Properties:
MasterUsername: 'master'
MasterUserPassword: !Ref DatabaseMasterPassword
Engine: aurora-postgresql
EngineVersion: 10.7
EngineMode: !If [ Serverless, 'serverless', 'provisioned' ]
StorageEncrypted: 'True'
Port: 5432
DBSubnetGroupName:
Ref: RDSDBSubnets
DBClusterParameterGroupName:
Ref: RDSDBClusterParameterGroup
VpcSecurityGroupIds:
- !Ref SGData
RDSDBInstance1:
Condition: NotServerless
Type: AWS::RDS::DBInstance
Properties:
DBSubnetGroupName:
Ref: RDSDBSubnets
DBParameterGroupName:
Ref: RDSDBParameterGroup
Engine: aurora-postgresql
EngineVersion: 10.7
DBClusterIdentifier:
Ref: RDSCluster
PubliclyAccessible: 'false'
DBInstanceClass: !Ref DatabaseInstanceType
# Only create the second instance if the user specified a Multi-AZ Database
RDSDBInstance2:
Condition: DeployMultiAZDB
Type: AWS::RDS::DBInstance
Properties:
DBSubnetGroupName:
Ref: RDSDBSubnets
DBParameterGroupName:
Ref: RDSDBParameterGroup
Engine: aurora-postgresql
EngineVersion: 10.7
DBClusterIdentifier:
Ref: RDSCluster
PubliclyAccessible: 'false'
DBInstanceClass: !Ref DatabaseInstanceType
RDSDBClusterParameterGroup:
Type: AWS::RDS::DBClusterParameterGroup
Properties:
Description: CloudFormation Sample Aurora Cluster Parameter Group
Family: aurora-postgresql10
Parameters:
rds.force_ssl: 1
RDSDBParameterGroup:
Type: AWS::RDS::DBParameterGroup
Properties:
Description: CloudFormation Sample Aurora Parameter Group
Family: aurora-postgresql10
Parameters:
log_rotation_age: 60
RDSDBSubnets:
Type: AWS::RDS::DBSubnetGroup
Properties:
DBSubnetGroupDescription: Subnet Group for RDS
SubnetIds:
- !Ref SubnetDataA
- !Ref SubnetDataB
RedshiftCluster:
Type: "AWS::Redshift::Cluster"
Properties:
DBName: "mycdm"
MasterUsername: "master"
MasterUserPassword: !Ref DatabaseMasterPassword
NodeType: !Ref RedshiftInstanceType
ClusterType: !If [ MultiNodeRedshift, "multi-node", "single-node" ]
# NumberOfNodes: ""
Encrypted: "True"
PubliclyAccessible: "False"
ClusterParameterGroupName: !Ref RedshiftClusterParameterGroup
IamRoles:
- !Ref RSRoleArn
VpcSecurityGroupIds:
- !Ref SGData
ClusterSubnetGroupName:
Ref: RedshiftSubnetGroup
RedshiftSubnetGroup:
Type: 'AWS::Redshift::ClusterSubnetGroup'
Properties:
Description: "Redshift Security Groups"
SubnetIds:
- !Ref SubnetDataA
RedshiftClusterParameterGroup:
Type: "AWS::Redshift::ClusterParameterGroup"
Properties:
Description: "My parameter group"
ParameterGroupFamily: "redshift-1.0"
Parameters:
-
ParameterName: "require_ssl"
ParameterValue: "true"
ParameterName: "statement_timeout"
ParameterValue: "3600000"
Outputs:
RDSEndpoint:
Value: !GetAtt RDSCluster.Endpoint.Address
RedshiftEndpoint:
Value: !GetAtt RedshiftCluster.Endpoint.AddressThanks Michael this was very helpful. Paul
I worked with the main source at Amazon to get these fixes implemented, and now the CloudFormation templates appear to be building the resource stacks properly.
I used the yaml from https://s3.amazonaws.com/ohdsi-rstudio/00-master-ohdsi.yaml and it keeps failing on RedshiftClusterParameterGroup. I was able to deploy AWS yesterday using the same yaml, but today it keeps failing. Does anybody have anysuggestions what to do?