Connector builder not updating state after upgrade to 0.59.1 (0.52.1 previously working)

[orjanbd]

Connector Name

connector-builder

Connector Version

0.65

What step the error happened?

During the sync

Relevant information

After upgrading from 0.52.1 to 0.59.1 our connector no longer updates the streamstate. Since the API can't guarantee that we receive records in sorted order based on lastmodifieddate, we have created a transformation field in the connector that is set to the current time and this is used to set the state and record the time. After upgrading our destination table also includes the time from the original state in the new records instead of current_timestamp.

YAML for the stream:

   SalesOrder:
      name: SalesOrder
      type: DeclarativeStream
      retriever:
        type: SimpleRetriever
        paginator:
          type: DefaultPaginator
          page_size_option:
            type: RequestOption
            field_name: pageSize
            inject_into: request_parameter
          page_token_option:
            type: RequestOption
            field_name: pageNumber
            inject_into: request_parameter
          pagination_strategy:
            type: PageIncrement
            page_size: 500
            start_from_page: 1
            inject_on_first_request: true
        requester:
          $ref: '#/definitions/base_requester'
          path: v2/salesorder
          http_method: GET
          request_parameters:
            lastModifiedDateTimeCondition: '>='
        record_selector:
          type: RecordSelector
          extractor:
            type: DpathExtractor
            field_path: []
      schema_loader:
        type: InlineSchemaLoader
        schema:
          $ref: '#/schemas/SalesOrder'
      transformations:
        - type: AddFields
          fields:
            - path:
                - data_download
              value: '{{ now_utc().strftime(''%Y-%m-%d %H:%M:%S'') }}'
        - type: RemoveFields
          field_pointers:
            - - soBillingContact
        - type: RemoveFields
          field_pointers:
            - - soBillingAddress
        - type: RemoveFields
          field_pointers:
            - - customerVATZone
        - type: RemoveFields
          field_pointers:
            - - soShippingContact
        - type: RemoveFields
          field_pointers:
            - - soShippingAddress
      incremental_sync:
        type: DatetimeBasedCursor
        cursor_field: data_download
        start_datetime:
          type: MinMaxDatetime
          datetime: '{{ config[''start_date''] }}'
          datetime_format: '%Y-%m-%d %H:%M:%S'
        datetime_format: '%Y-%m-%d %H:%M:%S'
        start_time_option:
          type: RequestOption
          field_name: lastModifiedDateTime
          inject_into: request_parameter
        cursor_datetime_formats:
          - '%Y-%m-%d %H:%M:%S'

Relevant log output

Log excerpt (0.52.1)

2024-05-13 22:40:01 platform > airbyte/source-declarative-manifest:0.65.0 was found locally.
2024-05-13 22:40:01 platform > Creating docker container = source-declarative-manifest-read-3178-0-kxpsp with resources io.airbyte.config.ResourceRequirements@50518f52[cpuRequest=0.5,cpuLimit=1,memoryRequest=1Gi,memoryLimit=2Gi,additionalProperties={}] and allowedHosts null
2024-05-13 22:40:02 source > Starting syncing ManifestDeclarativeSource
2024-05-13 22:40:02 source > Marking stream SalesOrder as STARTED
2024-05-13 22:40:02 source > Syncing stream: SalesOrder 
2024-05-13 22:40:02 source > Setting state of ManifestDeclarativeSource stream to {'data_download': '2024-05-12 22:40:07'}
2024-05-13 22:40:05 source > Marking stream SalesOrder as RUNNING
2024-05-13 22:40:05 source > Read 106 records from SalesOrder stream
2024-05-13 22:40:05 source > Marking stream SalesOrder as STOPPED
2024-05-13 22:40:05 source > Finished syncing SalesOrder
2024-05-13 22:40:05 source > ManifestDeclarativeSource runtimes:
Syncing stream SalesOrder 0:00:02.723845

2024-05-13 22:40:45 platform > removing 6841 from the stored stateIds for the stream null:SalesOrder, state received time 2024-05-13T22:40:05.605145463stagedStatsList size after poll: 0, stateIds size before removal 1

Log excerpt (0.59.1:

2024-05-14 10:40:03 platform > Creating docker container = source-declarative-manifest-read-2484-0-bihje with resources io.airbyte.config.ResourceRequirements@38dd952f[cpuRequest=0.5,cpuLimit=1,memoryRequest=1Gi,memoryLimit=2Gi,additionalProperties={}] and allowedHosts null
2024-05-14 10:40:03 platform > airbyte/source-declarative-manifest:0.85.0 was found locally.
2024-05-14 10:40:06 source > Starting syncing ManifestDeclarativeSource
2024-05-14 10:40:06 source > Marking stream SalesOrder as STARTED
2024-05-14 10:40:06 source > Setting state of ManifestDeclarativeSource stream to {'data_download': '2024-05-08 22:53:28'}
2024-05-14 10:40:06 source > Syncing stream: SalesOrder 
2024-05-14 10:40:15 source > Marking stream SalesOrder as RUNNING
2024-05-14 10:42:46 source > Read 9161 records from SalesOrder stream
2024-05-14 10:42:46 source > Marking stream SalesOrder as STOPPED
2024-05-14 10:42:46 source > Finished syncing SalesOrder
2024-05-14 10:42:46 source > ManifestDeclarativeSource runtimes:
Syncing stream SalesOrder 0:02:39.823051
2024-05-14 10:43:07 platform > Attempt 0 to update stream status complete null:SalesOrder

Contribute

• Yes, I want to contribute

[marcosmarxm]

@natikgadzhi, can the team discuss this bug in the next sprint? Thanks!