Filtering S3 Bucket Location.LocationConstraint for us-east-1

[jcderose]

Howdy! I want to run a simple "custodian run" command against untagged S3 resources in all AWS regions, but filter on the buckets that actually exist in that particular region. (e.g. when I run in us-east-1, I only want to work with untagged S3 resources that live in us-east-1; when I run in us-east-2, I only want to work with untagged S3 resources that live in us-east-2; etc.) I can get this filter to work for all regions except us-east-1 and need help building a filter string that works for all regions, including us-east-1. What would an appropriate policy filter look like that says show me any resources where "tag:Name" is absent and the bucket's Location.LocationConstraint value is the current region "{region}" (e.g. us-east-1, us-east-2, us-west-1, us-west-2, etc.)?

Here's the command I'm running:

custodian run -v -s /home/custodian/output --region=all -u /home/custodian/policy.yml

My policy file that works for all regions except us-east-1 is this:

policies:
  - name: s3-untagged
    resource: aws.s3
    filters:
      - "tag:Name": absent
      - Location.LocationConstraint: "{region}"

With Kapil's help on gitter (thank you, Kapil!) I know that us-east-1 is a bit special - it shows up as null instead of us-east-1 in custodian's output.

I tried this policy file:

policies:
  - name: s3-untagged
    resource: aws.s3
    filters:
      - "tag:Name": absent
      - or:
        - Location.LocationConstraint: "{region}"
        - Location.LocationConstraint: null

But that defeats the purpose of the Location.LocationConstraint filter I was working towards (e.g. it returns every bucket for every region, rather than only the buckets that exist in the particular region I'm iterating over).

I tried this policy file:

    resource: aws.s3
    filters:
      - "tag:Name": absent
      - or:
        - Location.LocationConstraint: "{region}"
        - and:
          - type: value
            key: region
            value: "us-east-1"
            op: equal
          - Location.LocationConstraint: null

But that doesn't return any results for us-east-1. (I assume something in my and filter is incorrect.)

Based on the JMESPath tutorials, I tried this policy file:

policies:
  - name: s3-untagged
    resource: aws.s3
    filters:
      - "tag:bucket-name": absent
      - or:
        - Location.LocationConstraint: "{region}"
        - and:
          - Location.LocationConstraint: null
          - [?region=='us-east-1']

But that returns the error:

2020-07-03 22:00:02,423: c7n_org:WARNING Error running policy in ACCOUNTNAME @ us-east-1 exception: 'list' object has no attribute 'get'

I feel like I'm missing something with the JMESPath string but can't figure it out. Any help would be appreciated!

[kapilt]

I think what your looking for would be an enhancement to allow the value filter to match against execution variables. so the us-east-1 .. alternatively it would be nice if we can inject a normalized region value into the bucket so this is a bit simpler to express, instead of trying to match the default api value here and special casing us-east-1.

[jcderose]

@kapilt I'm happy to look into this deeper but I'm not familiar with the code. Any suggestions where I would start to add this type of functionality?

[jcderose]

@kapilt any suggestions for where I could start to code this type of feature enhancement?

[ajkerrigan]

We got a question about this in Gitter today. I think the recap is that we can make a filter that "works" today but is pretty awkward because of the way the S3 API reflects us-east-1 (null) and eu-west-1 (EU):

filters:
  - type: value
    key: >
      Location | LocationConstraint == 'EU' && 'eu-west-1' || LocationConstraint || 'us-east-1'
    value: "{region}"

We already do some location normalization here - perhaps we could use that to add a normalized region column like c7n:BucketRegion as part of augment?

[JonGilmore]

Thanks @ajkerrigan for posting this workaround, it seems like it might be working, but I'm getting some unexpected results. Here's my full policy:

policies:
  - name: s3-logging
    resource: aws.s3
    description: find and remediate s3 buckets  that do not have logging enabled OR are logging to the wrong location
    filters:
      - type: bucket-logging # check to see if the bucket has logging enabled and is logging to the correct location
        op: not-equal
        target_bucket: "{account_id}-access-{region}"
        target_prefix: "{source_bucket_name}/access-logs/"
      - not: # check to make sure that the bucket isn't an access log bucket
          - "tag:app": "access-logs"
      - type: value # https://github.com/cloud-custodian/cloud-custodian/issues/5925#issuecomment-975635394
        key: >
          Location | LocationConstraint == 'EU' && 'eu-west-1' || LocationConstraint || 'us-east-1'
        value: "{region}"
    actions:
      - type: toggle-logging
        target_bucket: "{account_id}-access-{region}"
        target_prefix: "{source_bucket_name}/access-logs/"

When I run this policy with c7n-org, here is an example "match" from the resources.json:

[
  {
    "Name": "myorg-bucket",
    "CreationDate": "2014-10-24T20:44:16+00:00",
    "Location": {
      "LocationConstraint": null
    },
    "Tags": [
      {
        "Key": "app",
        "Value": "myapp"
      },
      {
        "Key": "stack",
        "Value": "production"
      }
    ],
    "Policy": null,
    "Acl": {
      "Owner": {
        "DisplayName": "OrgPlatformProduction",
        "ID": "11111111"
      },
      "Grants": [
        {
          "Grantee": {
            "DisplayName": "OrgPlatformProduction",
            "ID": "11111111",
            "Type": "CanonicalUser"
          },
          "Permission": "FULL_CONTROL"
        }
      ]
    },
    "Replication": null,
    "Versioning": {
      "Status": "Enabled"
    },
    "Website": null,
    "Logging": {
      "TargetBucket": "aws-logs-111117771111-us-east-1",
      "TargetPrefix": "myorg-bucket"
    },
    "Notification": null,
    "Lifecycle": null,
    "c7n:MatchedFilters": [
      "Location | LocationConstraint == 'EU' && 'eu-west-1' || LocationConstraint || 'us-east-1'\n"
    ]
  }
]

As you can see, the Logging does NOT match my desired standard bucket logging pattern, but that filter does not show up on the c7n:MatchedFilters. Any thoughts on this? Is my policy just misconfigured? Is the region value filter screwing with this result?

[kapilt]

I think the challenge has always been how to make it backwards compatible, or if we need to make some sort of v2 resource, or maybe just a specialized location filter.

[ajkerrigan]

Ah maybe a special filter is better 🤔 . I was thinking If we added a new c7n:Something column during augment, new policies could start to target it but existing policies would keep working. A special filter would be more explicit and obvious from the schema side though 👍 . Probably less jarring than a whole new resource unless there were other drivers to go that way.

[linuxananth1976]

Thanks @ajkerrigan for providing the solution, it works!!!!!