Prisma Migrate Drops Custom Migrations

[ApocalypseCalculator]

Bug description

For a multitude of different reasons users need to use custom SQL for unsupported Prisma features. One such feature I am trying to implement is native full text search, which requires me to add a tsvector column to my table and a GIN index. The documentation outlines how to customize a migration, which has worked for me. However, I am now trying to create another table, and when I try to create a migration for it, the generated migration drops my custom column and index.

Since my development database was populated with test data, I was able to catch on as Prisma warned of its attempt to push these changes onto the database itself. However, without data in dev, there is no warning and the migration is generated straight away. If deployed to production, this would wipe out the data I had, which is not really acceptable.

I believe that Prisma should a) warn of destructive actions like dropping columns/indices before generating migrations and b) not have this issue of dropping in the first place. Thoughts?

How to reproduce

1. Create migration with given schema
2. Generate a new migration without applying it with prisma migrate dev --create-only
3. Add the given SQL into the empty migration
4. Apply migration
5. Optionally edit the schema in some way
6. Create migration for changes (if no changes, a migration with only the drop commands are created)

Expected behavior

Prisma shouldn't drop my custom migration

Prisma information

datasource db {
    provider = "postgresql"
    url      = env("DB_URL")
}
generator client {
    provider = "prisma-client-js"
}
model Video {
    id        Int         @id @default(autoincrement())
    title     String
    alttitle  String
    summary   String
}

ALTER TABLE "Video" ADD search tsvector GENERATED ALWAYS AS 
    (setweight(to_tsvector('english', title), 'A') || ' ' || 
    setweight(to_tsvector('english', alttitle), 'B') || ' ' ||
    to_tsvector('english', summary)) STORED;

CREATE INDEX idx_search ON "Video" USING GIN(search);

Environment & setup

• OS: Windows
• Database: PostgreSQL
• Node.js version: 18.17.1

Prisma Version

5.13.0

[janpio]

What exactly is the destructive migration that is being generated here?

Fundamentally the problem is that Prisma does not support what you are trying to use in Prisma Schema (a generated column I think), and hence will of course try to clean up your database to bring it into the state that the Prisma schema represents. But that is also why --create-only exists so you can review and clean up any generated migration.

[ApocalypseCalculator]

The "destructive actions" generated by Prisma that I mentioned are

-- DropIndex
DROP INDEX "idx_search";

-- AlterTable
ALTER TABLE "Video" DROP COLUMN "search";

I realize that --create-only exists, but I think the expectation is that I shouldn't need to go through every generated migration afterwards manually. If I need to add more custom features, or edit my Prisma schema further, it's not hard to see how it can become extremely messy. I had assumed (perhaps incorrectly) that it was a one and done sort of thing.

[janpio]

We would love to be able to offer that, but how should Prisma know what is expected in the database (because of a manually modified migration) and what is not (and hence needs to be fixed to get to the desired state)?

If I was you, I would probably add the search column and the Gin index on it to the Prisma schema, then you just need to see how Prisma deals with the generated column definition (it will possibly try to remove it, which then is something you need to modify with each future migration).

[ApocalypseCalculator]

One possible solution is to tag each migration to the corresponding Prisma schema. Then a newly generated migration would diff the changes in the Prisma schema, as opposed to diff against what is in the current database. This, I believe, makes a bit more sense and is in my opinion a better system overall, but I understand it is likely time consuming to implement.

As for your solution, how can I add the column/index to the Prisma schema? Since tsvector and related functions aren't supported.

[janpio]

One possible solution is to tag each migration to the corresponding Prisma schema. Then a newly generated migration would diff the changes in the Prisma schema, as opposed to diff against what is in the current database. This, I believe, makes a bit more sense and is in my opinion a better system overall, but I understand it is likely time consuming to implement.

Yes, but that would be pretty much a completely different migration tool than what Prisma Migrate currently is.
Still, we are aware of the current challenges and are considering far reaching changes like that - but that will definitely take some time.

As for your solution, how can I add the column/index to the Prisma schema? Since tsvector and related functions aren't supported.

I was thinking about adding it as a supported field, and then manipulate the migration that would create it. But you are correct, in the details this might not actually work as hoped :/

[chrbala]

@ApocalypseCalculator, you might be interested in #24248, in which I suggest we make diffs based on changes to the Prisma schema in different git versions. You might be able to use this for your use case.