[Bug]: Lack of handling for L0 segments in binlog import

[bigsheeper]

Is there an existing issue for this?

• I have searched the existing issues

Environment

- Milvus version: master&2.4
- Deployment mode(standalone or cluster):
- MQ type(rocksmq, pulsar or kafka):    
- SDK version(e.g. pymilvus v2.0.0rc2):
- OS(Ubuntu or CentOS): 
- CPU/Memory: 
- GPU: 
- Others:

Current Behavior

In the current binlog import process, we have not handled L0 segments (#27349), which will result in the loss of user delete data. See also: zilliztech/milvus-backup#316

Expected Behavior

During the backup-restore process, we need to consider L0 segments and import them into the collection. The strategy is to replay the L0 segments rather than merge delete data into L1 and L2 segments because users might be performing the incremental-backups, and the delete data in L0 segments could be from a long time ago.

We will define the order of files passed in during binlog import. The first input file will be the prefix for the insert log, the second input file will be the prefix for the L1 and L2 segments delta log, and the third and subsequent input files will be the prefixes for the L0 segments delta log:

    [
        "files/insert_log/col_0/part_0", -- insert log prefix for L1, L2
        "files/delta_log/col_0/part_0",  -- delta log prefix for L1, L2
        "files/delta_log/col_0/-1",      -- delta log prefix for L0
        "files/delta_log/col_0/part_0",  -- delta log prefix for L0
    ],

To distinguish between segments under the same prefix (for example, "files/delta_log/col_0/part_0" as mentioned above) and determine whether they are L0 or non-L0, it is necessary to pass a list of L0 segment IDs in the options:

    "options": {
        "backup": "true",
        "l0_segments": "l0_seg_0,l0_seg_1,l0_seg_2,...",
    }

Steps To Reproduce

No response

Milvus Log

No response

Anything else?

No response

[xiaofan-luan]

what does -1 means?

[xiaofan-luan]

when we do bulk insertion L0 also need to do a handoff?

[bigsheeper]

what does -1 means?

Currently logs of L0 segments will be instored with the prefix files/delta_log/{collectionID}/-1.

[wayblink]

"replay" means parse delta log and convert to delete messages?

[bigsheeper]

when we do bulk insertion L0 also need to do a handoff?

We need to ensure that after users invoke load(refresh=true), all bulk insertion L0 segments are loaded by the query node. Is this currently achievable? @congqixia

[bigsheeper]

"replay" means parse delta log and convert to delete messages?

Yep, L0 segments will be generated, instead of applying deletes on L1, L2 segments.

[wayblink]

"replay" means parse delta log and convert to delete messages?

Yep, L0 segments will be generated, instead of applying deletes on L1, L2 segments.

backup l0 segments -> new l0 segments? So why we need to import L1/L2 segment and L0 segment in one request? I mean, we can import partition l0 segments after all l1/l2 segments in one partition, and import non-part l0 segments after all partitions restored.

[bigsheeper]

The proposed approach above indeed has a drawback: when users enable partitionKey (e.g., 64), the backup tool will invoke import once for each partition. Consequently, all data under the files/delta_log/{collectionID}/-1 prefix will be redundantly read 64 times, resulting in significant read amplification.

Proposal 2
Therefore, we propose a new approach: refrain from importing L0 segments while importing L1, L2. Instead, after L1, L2 for all 64 partitions imported, a special L0 import will be invoked.
For this L0 import, we define the input files with only one prefix, which is the prefix for L0 segments:

• When L0 import doesn't specify a partition name, L0 segments are applied to the entire table. For example:

    [
        "files/delta_log/col_0/-1"
    ],

• When L0 import specifies a partition name, L0 segments are applied to the specified partition. For example:

    [
        "files/delta_log/col_0/part_0"
    ],

To distinguish between L0 import from L1, L2 imports, a new flag needs to be added in the options:

    "options": {
        "backup": "true",
        "l0_import": "true",
    }

[congqixia]

"-1" means all partition, FYI

[congqixia]

when we do bulk insertion L0 also need to do a handoff?

We need to ensure that after users invoke load(refresh=true), all bulk insertion L0 segments are loaded by the query node. Is this currently achievable? @congqixia

IMHO, this shall be done after #32990 change

[xiaofan-luan]

上面提出的方法确实有一个缺点：当用户启用partitionKey（例如64）时，备份工具将为每个分区调用一次导入。因此，该前缀下的所有数据files/delta_log/{collectionID}/-1将被冗余读取 64 次，导致显着的读取放大。

建议2 因此，我们提出一种新的方法：在导入L1、L2的同时避免导入L0段。相反，在导入所有 64 个分区的 L1、L2 后，将调用特殊的 L0 导入。对于此 L0 导入，我们定义仅具有一个前缀 的输入文件，即 L0 段的前缀：****

• 当 L0 导入未指定分区名称时，L0 段将应用于整个表。例如：

    [
        "files/delta_log/col_0/-1"
    ],

• 当 L0 导入指定分区名称时，L0 段将应用于指定的分区。例如：

    [
        "files/delta_log/col_0/part_0"
    ],

为了区分 L0 导入和 L1、L2 导入，需要在选项中添加一个新标志：

    "options": {
        "backup": "true",
        "l0_import": "true",
    }

why not add a partition tag?