Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We鈥檒l occasionally send you account related emails.

Already on GitHub? Sign in to your account

google_pubsub_subscription - add 'filename_datetime_format' to cloud_storage_config #17911

Comments

@husseyd
Copy link

husseyd commented Apr 19, 2024

Community Note

  • Please vote on this issue by adding a 馃憤 reaction to the original issue to help the community and maintainers prioritize this request.
  • Please do not leave +1 or me too comments, they generate extra noise for issue followers and do not help prioritize the request.
  • If you are interested in working on this issue or have submitted a pull request, please leave a comment.
  • If an issue is assigned to a user, that user is claiming responsibility for the issue.
  • Customers working with a Google Technical Account Manager or Customer Engineer can ask them to reach out internally to expedite investigation and resolution of this issue.

Description

Hi

Pub/Sub Subscriptions with Cloud Storage config now allow definition of the filenameDatetimeFormat. Links below to Gcloud CLI and Pub/Sub API contain further information.

I suggest adding a field filename_datetime_format to the cloud_storage_config block for google_pubsub_subscription to handle this.

References to Pub/Sub API in reference section and sample configuration provided. Please note that the Pub/Sub service/API enforces some constraints on this field - namely than all of YYYYMMDDhhmmss must be used, but can be broken up with characters like /, -, _,:

New or Affected Resource(s)

  • google_pubsub_subscription

Potential Terraform Configuration

resource "google_pubsub_subscription" "example" {
  name  = "example-subscription"
  topic = google_pubsub_topic.example.id

  cloud_storage_config {
    bucket = google_storage_bucket.example.name

    filename_datetime_format = "YYYYMM/DD/hh/mm_ss"
    filename_prefix = "pre-"
    filename_suffix = "-%{random_suffix}"

    max_bytes = 1000
    max_duration = "300s"
  }
  depends_on = [ 
    google_storage_bucket.example,
    google_storage_bucket_iam_member.admin,
  ]
}

References

See:

b/336319061

@github-actions github-actions bot added forward/review In review; remove label to forward service/pubsub labels Apr 19, 2024
@husseyd husseyd changed the title google_pubsub_subscription - add 'file_datetime_format' to cloud_storage_config google_pubsub_subscription - add 'filename_datetime_format' to cloud_storage_config Apr 19, 2024
@melinath melinath added this to the Goals milestone Apr 22, 2024
@melinath melinath removed the forward/review In review; remove label to forward label Apr 22, 2024
@husseyd
Copy link
Author

husseyd commented May 2, 2024

Added note:

The google_pubsub_subscription resource will actually silently overwrite any filename_datetime_format config on the subscription that has been set on the subscription.

I discovered this as follows:

  1. Create pubsub subscription in Terraform.
  2. Update filename datetime format attribute of my pubsub subscription via Pubsub API (since the TF resource does not cater for this).
  3. Make another change to the pubscription in Terraform (changed cloud_storage_config.max_bytes) (No diff was shown in the plan for the datetime format change)
  4. Filename datetime format of the subsciption was reverted to the default after TF apply

This could be a bit of an issue as if any of my subscriptions are updated in TF, they will then revert to the default datetime format and files will appear in a different directory in GCS than they should

@lahuang4
Copy link

Apologies for the trouble. I'm working on adding filename_datetime_format support and hope to have it in your hands soon!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment