PoC migration generator #544
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* sql/*/*.sql * test/*.sql * updates/pg_partman--4.7.3--5.0.0-beta.sql Trailing whitespaces are redundant and removed by IDE automatically. Gather all these changes in a single commit. Historical files usually left "as is" but could also be slightly reformatted later.
Hard to exploit (need a double quote for extension schema) but it is better to have correct query. Also adjust type of a var: it can never overflow but make linters be quiet. Also change named notation[1] for function arguments to modern style. [1] https://www.postgresql.org/docs/current/sql-syntax-calling-funcs.html#SQL-SYNTAX-CALLING-FUNCS-NAMED
The `wildcard` function already returns sorted files list. https://www.gnu.org/software/make/manual/html_node/Wildcard-Function.html If is is required strictly ordered prerequisites a pipe symbol should be used: https://www.gnu.org/software/make/manual/html_node/Prerequisite-Types.html
It is useful for update/install files generator three commits later. The old install file is copied from a state as of tag v4.7.3: updates/pg_partman--4.7.3.sql It is useful to have two of them: install file and previous one to check upgrade script. Install the newest is usual: ... # CREATE EXTENSION pg_partman; Test of upgrade can be done via commands: ... # CREATE EXTENSION pg_partman VERSION "4.7.3"; ... # ALTER EXTENSION pg_partman UPDATE;
It allows to have (as far as possible) single point of truth. This commit changes nothing in the current migration file: updates/pg_partman--4.7.3--5.0.0-beta.sql
Split table creation and copying data. The best way to watch changes is: git diff --color-moved HEAD^..HEAD
Gather table's DDL together. It makes the last commit in the series (5/5) be simpler (when tables DDL split into several files). The best way to watch changes is: git diff --color-moved HEAD^..HEAD
Join INSERT & UPDATE to a single UPDATE.
Split single file with table creation DDL into different files.
It will help to use it in the next commit.
Also it helps to compare similar tables, e.g.
git diff \
--word-diff=color \
--word-diff-regex='[a-zA-Z0-9]|[^[:space:]]' \
--no-index \
sql/tables/part_config.sql sql/tables/part_config_sub.sql
The best way to watch changes is:
git diff --color-moved HEAD^..HEAD
Use table creation DDL in generated upgrade script.
Only empty lines, comments and unnecessary type cast ("::regclass")
are changed in the reference file:
updates/pg_partman--4.7.3--5.0.0-beta.sql
|
Thanks for this work. I'd been trying to figure out something to make the handling of the update files and individual files like this for a while. This is an interesting idea. Only thing I'm not 100% on is that the Makefile would have to be updated every single release. |
|
I made Makefile to looks like this to only be sure (and prove) changes from "updates/pg_partman--4.7.3--5.0.0-beta.sql" are correctly applied to individual files correctly. Of course if this method will be accepted Makefile for future development process can be [slightly] different. |
|
Sure I understand. Just giving my initial impression and some feedback. I definitely appreciate the work and look forward to having something to automate this process better. My method is a manual one using Meld, but it's worked for a long while now :) |
vitaly-burovoy
force-pushed
the
PoC/migration_generator
branch
from
340bebc
to
baf33dc
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
I tried to play with the code but it is very hard when have some functions in a single "update" file, others - in "base" files (one per function).
Moreover it will be difficult to apply changes from those "update" file back to "base" files.
I somehow managed to write Makefile rules to have DDLs mostly in "base" files which can be merged to get mentioned "update" file.
It improves (at least for me) inspection of the code and keeps the updated migration file in a consistent state.
At this point generated "update" SQL file (to v5.0.0-beta) is almost the same as the original one (except several withespaces, comments and type casting). Other changes can be made right into "base" files and they will be automatically reflected in the "update" one.
Unfortunately I could not solve a problem with autodetection of changes (e.g. if you change "09.cleanup_temp.sql", thenSolved (fixed), thank's @keithf4make alldoes not reassemble files), a workaround is to domake clean all.There are (in the beginning of the branch) two commits with improvements of
pg_partman_bgw.c. If you want to move them to another PR - let me know.Pointing out to typos, comments, tips, etc. are welcome.