Can Sensor tick be manually triggered outside the usual schedule?

[mudravrik]

Deployment type: Dagster Cloud Hybrid

We have a sensor serving for picking up new files from GCS into new partitions.
Usually new files appear in the bucket daily, so it is perfectly fine for us to have minimum_interval_seconds set to couple of hours.
However once in a while something unexpected may happened, like a manual addition bunch of files to be processed "right now". So we have to wait for sensor to tick before process these files which is not quite convenient.

Obvious solution for the issue will be manual sensor tick trigger, but unfortunately I'm really struggle with finding how to do it.
I tried:

• start/stop the sensor
• redeploy Code Location with the sensor
• looking through graphql API for some "methods" to trigger a sensor
  without any luck yet.

So can (and how?) can I trigger a Sensor run?

[garethbrickman]

There's no way to manually trigger a sensor evaluation.

The closest approximation is in the GraphQL API you can use the sensorDryRun mutation to get back the DryRunInstitgationTick which has details about how the sensor's RunRequest would be composed (example query in this Gist)

You could then use this information to send a manual RunRequest, but this would not really be equivalent to the sensor itself performing an evaluation.

[mudravrik]

I see, that's pretty much an edge case of ours, thank you!
Probably, it will be easier for us to change minimum_interval_seconds back and forth in this case.

Also, according to your workaround, will it create partitions to track based on RunRequest sent?
If not, I assume, it may be pretty dangerous for non-idempotent jobs because further Sensor tick will re-develop new files and rerun their jobs, right?

[garethbrickman]

If the job is partitioned and you include a partition_key in the RunRequest, the run will be associated with that partition, but it does not create the partition itself. Partitions are typically defined separately as part of your job's configuration.

There may be some way to handle the idempotence risk using the run_key parameter but I expect that would get complex and create other fragilities in terms of state management between manual Run Requests and Sensor-evaluated Run Requests.

[mudravrik]

Ah, I see, in this particular case we use sensor to create new keys in dynamic partitions for job.
So to achieve what sensor does, we have should run sensor dry run, add partitions key (is possible for jobs by the way? I haven't seen add partition button there as in partitioned assets) and then run jobs for partitions.

However, probably not worth the time to setup such a complexity, honestly :)
Thank you!