Skip to content

Python package to manage Google Cloud Data Catalog tags, loading metadata from external sources -- currently supports the CSV file format

License

Notifications You must be signed in to change notification settings

ricardolsmendes/datacatalog-tag-manager

Repository files navigation

datacatalog-tag-manager

A Python package to manage Google Cloud Data Catalog tags, loading metadata from external sources. Currently supports the CSV file format.

license pypi issues continuous integration continuous delivery

Table of Contents


1. Environment setup

1.1. Python + virtualenv

Using virtualenv is optional, but strongly recommended unless you use Docker.

1.1.1. Install Python 3.6+

1.1.2. Create a folder

This is recommended so all related stuff will reside at the same place, making it easier to follow the next instructions.

mkdir ./datacatalog-tag-manager
cd ./datacatalog-tag-manager

All paths starting with ./ in the next steps are relative to the datacatalog-tag-manager folder.

1.1.3. Create and activate an isolated Python environment

pip install --upgrade virtualenv
python3 -m virtualenv --python python3 env
source ./env/bin/activate

1.1.4. Install the package

pip install --upgrade datacatalog-tag-manager

1.2. Docker

Docker may be used as an option to run datacatalog-tag-manager. In this case, please disregard the above virtualenv setup instructions.

1.2.1. Get the source code

git clone https://github.com/ricardolsmendes/datacatalog-tag-manager
cd ./datacatalog-tag-manager

1.3. Auth credentials

1.3.1. Create a service account and grant it below roles

  • BigQuery Metadata Viewer
  • Data Catalog TagTemplate User
  • A custom role with bigquery.datasets.updateTag and bigquery.tables.updateTag permissions

1.3.2. Download a JSON key and save it as

  • ./credentials/datacatalog-tag-manager.json

1.3.3. Set the environment variables

This step may be skipped if you're using Docker.

export GOOGLE_APPLICATION_CREDENTIALS=./credentials/datacatalog-tag-manager.json

2. Manage Tags

2.1. Create or Update

2.1.1. From a CSV file

  • SCHEMA

The metadata schema to create or update Tags is presented below. Use as many lines as needed to describe all the Tags and Fields you need.

Column Description Mandatory
linked_resource OR entry_name Full name of the BigQuery or PubSub asset the Entry refers to, or an Entry name if you are working with Custom Entries
template_name Resource name of the Tag Template for the Tag
column Attach Tags to a column belonging to the Entry schema
field_id Id of the Tag field
field_value Value of the Tag field
  • SAMPLE INPUT
  1. sample-input/upsert-tags for reference;
  2. Data Catalog Sample Tags (Google Sheets) might help to create/export a CSV file.
  • COMMANDS

Python + virtualenv

datacatalog-tags upsert --csv-file <CSV-FILE-PATH>

Docker

docker build --rm --tag datacatalog-tag-manager .
docker run --rm --tty \
  --volume <CREDENTIALS-FILE-DIR>:/credentials --volume <CSV-FILE-DIR>:/data \
  datacatalog-tag-manager upsert --csv-file /data/<CSV-FILE-PATH>

2.2. Delete

2.2.1. From a CSV file

  • SCHEMA

The metadata schema to delete Tags is presented below. Use as many lines as needed to delete all the Tags you want.

Column Description Mandatory
linked_resource OR entry_name Full name of the BigQuery or PubSub asset the Entry refers to, or an Entry name if you are working with Custom Entries
template_name Resource name of the Tag Template of the Tag
column Delete Tags from a column belonging to the Entry schema
  • SAMPLE INPUT
  1. sample-input/delete-tags for reference;
  2. Data Catalog Sample Tags (Google Sheets) might help to create/export a CSV file.
  • COMMANDS

Python + virtualenv

datacatalog-tags delete --csv-file <CSV-FILE-PATH>

Docker

docker build --rm --tag datacatalog-tag-manager .
docker run --rm --tty \
  --volume <CREDENTIALS-FILE-DIR>:/credentials --volume <CSV-FILE-DIR>:/data \
  datacatalog-tag-manager delete --csv-file /data/<CSV-FILE-PATH>

3. How to contribute

Please make sure to take a moment and read the Code of Conduct.

3.1. Report issues

Please report bugs and suggest features via the GitHub Issues.

Before opening an issue, search the tracker for possible duplicates. If you find a duplicate, please add a comment saying that you encountered the problem as well.

3.2. Contribute code

Please make sure to read the Contributing Guide before making a pull request.

About

Python package to manage Google Cloud Data Catalog tags, loading metadata from external sources -- currently supports the CSV file format

Topics

Resources

License

Code of conduct

Stars

Watchers

Forks

Packages

No packages published