How We Improved Our AWS Integration with CloudFormation Custom Resource

As a constantly evolving tool, we always try to improve customer experience based on their essential feedback. Resmo AWS Integration is our most popular integration; ~72% of Resmo accounts have at least 1 AWS account monitored by Resmo. Since onboarding our private beta users, we've monitored how they use Resmo. We've noticed that configuring an AWS integration can be troublesome for some users.

72% of Resmo accounts have at least 1 AWS account monitored by Resmo.

Initial AWS integration required the user to use our supplied CloudFormation template with some parameters embedded in the URL and copy the resulting role ARN to the Resmo interface. We've eliminated the need to manually copy Role ARNs by implementing CloudFormation Custom Resource to get the callbacks upon completion, providing a smoother experience so that our customers can see their AWS resources with minimal effort and most convenient way.

How the simple AWS Integration worked before 

Overview of early AWS Integration Flow:

The order of actions expected from a customer to create an AWS Integration with CloudFormation template is as follows:

1. The user navigates to Create AWS Integration on Resmo UI
2. Entering a name, UI opens AWS CloudFormation in a new tab 
3. CloudFormation creates a new IAM Policy
4. CloudFormation creates a new IAM Role with a trust policy for the Resmo AWS account
5. IAM Policy is Attached to new created IAM Role
6. CloudFormation finishes, and the user copies to IAM Role ARN to Resmo UI
7. The user clicks Create integration, and hundreds of async jobs to collect data are scheduled to the engine.

As simple as it might seem from a top-down view, the success ratio was low. We've noticed people were creating the roles incorrectly, not following the instructions in order (creating integration before the role), or entering the wrong role ARNs to the Resmo UI. As a result, Resmo shows the integration as unready.

Example of unready integration due to early integration creation

We needed to fix that experience ASAP. Our primary goal is to show immediate value to the customer, and the integrations must be smooth. Most of our integrations use OAuth2, but unfortunately, AWS does not support such an authentication mechanism.

Without moving forward on how we improved our AWS integration, we first want to talk about the serious security implications of building an AWS integration and why you need to use a controlled External ID when creating assume role policies.

Wall of shame, static credentials, a.k.a IAM Users

Unfortunately, some providers and tools still ask you to create an IAM User and supply its access keys because it's the easiest way. We can go for hours on why it's a terrible practice, given that we can easily build a trust relationship without resenting to static, dangerous credentials. Additionally, AWS Partners must stop asking customers to create access keys for them since June 2022, so we hope to see we will not see providers asking for access keys anymore.

Static credential seems easiest but comes with the periodic access key rotation burden. Plus, it does not comply with IAM-005 (Use IAM roles and its temporary security credentials to provide access to third parties) of AWS Partner Hosted Foundational Technical Review (FTR).

External IDs and Confused Deputy Problem 

A more convenient way is delegating access with AWS IAM Role, which provides temporary security credentials to the third-party tool. However, delegating access via IAM Role without an external ID is less secure and prone to data leaks because it's easier to guess an IAM Role name.

What is External ID? An external ID is a secret identifier that is only (supposed/expected to) known by a customer and a third-party tool. A customer allows a role to be assumed by the third-party tool only if it provides the agreed external ID.

External IDs are used to prevent cross-account confused deputy problems. In short, if a principal trusts an external role by role ARN only, anyone could guess the role name and account ID (more problematic if it's generated predictably).

In the case of no external ID, the customer creates an IAM Role in its account and trusts the third-party tool as a "deputy "that can act on the customer's behalf. However, the ARN of an IAM Role is easy to guess or obtain as none of the account IDs or role names are considered secret. So if another customer with bad intentions of the third-party tool is managed to obtain the account id and role name, it will be able to trick the tool into acting on/accessing resources owned by another customer.

Resmo uses AWS IAM Role with an external ID to obtain temporary security credentials and access resources in a customer account, as complying with CAA-002 (Use external ID with cross-account roles to access customer accounts) of Partner Hosted FTR.

However, allowing customers to decide on an external ID is still prone to a confused deputy problem, as external ID can be chosen weakly and give a chance to users with bad intentions to try and guess the external IDs of other customers, as stated in CAA-004 and CAA-006 of Partner Hosted FTR. Therefore, at Resmo, we ensure external IDs are tied to customer IDs and cannot be used by any other customer, eliminating the confused deputy problem and providing the most secure way to integrate with the AWS accounts of our customers.

One-Click CloudFormation Integration with Callbacks

Okay, it's not exactly one click, it is three clicks, but it does not require the user to follow a sequence or copy ARNs manually. Previously, we've used CloudFormation to help customer to create the necessary policy and role. After the Cloudformation stack has been completed, the customer was expected to return back to Resmo and provide the necessary fields (Account ID, Role Name, External ID), and press the Create Integration button.

With the new AWS Integration create flow, customers only need to run the CloudFormation stack; Resmo will know your IAM resources have been successfully created by listening to relevant callbacks and blocking the UI until completion. No need to go back to Resmo and enter the AWS account ID, role name, and external ID.

The order of actions expected from a customer to create an AWS Integration with a one-click CloudFormation template is as follows:

1. Selecting AWS from add integration screen.
2. Entering a name, description (optional), tags(optional)
3. Creating the required policy and role via running a CloudFormation template and waiting approximately 2-3 minutes for relevant callbacks.

The callbacks are visualized as follows:

Enter name, description, and tags for the integration and launch stack.

Acknowledge that the stack will create a Managed IAM Policy and click Create Stack.

Stack completed; necessary policy and role have been created.

If you go back to Resmo, you will see that the integration creation is completed; you can start investigating your resources within 2 minutes.

How does it work in the background? 

The New CloudFormation template uses CloudFormation Custom Resource to get notified when necessary resources are successfully created.

The structure of the Custom Resource definition in the template is as follows:

{
  "Type": "Custom::ResmoAWSIntegrationNotification",
  "DependsOn": "DataCollectionRole",
  "Properties": {
    "ServiceToken": {
      "Fn::Join": [
        ":",
        [
          "arn:aws:sns",
          {
            "Ref": "AWS::Region"
          },
          {
            "Ref": "ResmoAWSAccountId"
          },
          "resmo-cross-account-cf-listener"
        ]
      ]
    },
    "ExternalID": {
      "Ref": "ExternalID"
    },
    "RoleName": {
      "Ref": "RoleName"
    }
  }
}

Like any other resource declaration in a CloudFormation template, it takes a Logical ID (Custom in our case). Resource Type can be one of Custom::Anything or AWS::CloudFormation::CustomResource. We have continued with Custom::ResmoAWSIntegrationNotification. ServiceToken property is used by CloudFormation to send a request whenever this template is used for creating, updating, or deleting the declared custom resource. Custom Resources, currently, can be backed by Lambda or SNS. Thus, ServiceToken can be either a Lambda ARN or an SNS ARN. Properties other than ServiceToken are sent in requests as ResourceProperties.

A sample creates custom resource request sent by CloudFormation:

{
  "RequestType": "Create",
  "ResponseURL": "http://pre-signed-S3-url-for-response",
  "StackId": "cloudformation stack id",
  "RequestId": "unique id for this create request",
  "ResourceType": "Custom::ResmoAWSIntegrationNotification",
  "LogicalResourceId": "Custom",
  "ResourceProperties": {
    "ExternalID": "Resmo generated unique external id",
    "RoleName": "Role name you have entered, during stack creation"
  }
}

CloudFormation expects a response in ResponseURL location. The body of PUT request to ResponseURL is similar to the following:

{
  "Status": "SUCCESS",
  "PhysicalResourceId": "Custom",
  "StackId": "cloudformation stack id",
  "RequestId": "unique id for this create request",
  "LogicalResourceId": "Custom",
  "Data": {
    "o1": "v1"
  }
}

You can also output Data if you want and access it using Fn::GetAtt function. e.g. value of output o1 can be retrieved with { "Fn::GetAtt" : [ "Custom", "o1" ] } in a CloudFormation template. Additionally, you can add a reason field if the request is "FAILED" somehow. For example, Resmo stack would fail if AWS Account you are trying to integrate has already been integrated with your Resmo account.

The below diagram shows how CloudFormation Stack in a customer AWS account notifies Resmo.

We have chosen SNS-backed Custom Resource with a topic in our account.

We have chosen SNS over Lambda because of the price gap between Lambda and SNS. Additionally, even though we would have set a proper resource-based access policy for Lambda, the idea of letting a CloudFormation stack trigger a function in our account is not comforting at all from a security perspective as well.

We have chosen our AWS account over Customers to minimize our presence in the customer account and have one less thing to worry about. Also, updating it per customer would be problematic in scale.

CloudFormation is unable to publish messages to a topic cross-region. Thus the decision to continue with SNS topics in our account led us to create a resmo-cross-account-cf-listener topic in every AWS region, even regions not available by default, to allow our customers to run their Cloudformation stack at whatever region they want.

To Sum Up

Resmo has observed the behaviors of private beta customers and listened to their feedback on the difficulty of creating AWS Integration which is our, as might be expected, most popular integration. With the new One-Click CloudFormation Integration, we have observed the ratio of successfully creating AWS Integration on the first try has increased noticeably.

While redesigning our AWS integration flow, we noticed possible security problems and fixed them as well in accordance with how AWS Hosted Partner FTR recommends.

In the future, we imagine a world where we put our “app” in the AWS Marketplace and make it even smoother to integrate AWS, similar to a Slack or GitHub app auth flow. Until then, this seems like the easiest and most secure way to integrate with AWS!