Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[RFC] Add proper support for dynamic channel creation for things #4048

Open
spacemanspiff2007 opened this issue Jan 17, 2024 · 10 comments
Open
Labels
enhancement An enhancement or new feature of the Core

Comments

@spacemanspiff2007
Copy link
Contributor

More and more bindings dynamically create channels for an added thing.
While this makes it easy to support various variations of a device it does also lead to unexpected behaviour and really hard to track down issues. Some of these issues can be attribitued to bugs/issues in bindings but imho there should be some core changes to make things easier for both the binding developers and users.

For examples I use mostly the shelly binding because there I've observed the issues first hand:

  1. After an openHAB restart the roller shutters dynamically create the channels. Due to some issue the rollershutter channel was not created for every roller shutter but the device still went online. So in the thing overview everything looked good, yet the command to close/open the shutters did not work because the link from the item was pointing nowhere.
  2. Battery powered devices so they are asleep most of the time. After a restart they are stuck in "CONFIG PENDING" and will not report e.g. smoke/flood back to openhab. Typically they wake up every 24h but it's not enough for the binding to complete initialisation. Sometimes the devices are stuck for weeks, months and very often they never finish. In the best case they will not work until the first wake up.
  3. The battery powered smoke alarm is especially tricky: Sometimes it changes to ONLINE, but the thing only has the channel for battery level but not for smoke alarm. In the event log it properly reports the battery level every 24h, but it will not report a smoke alarm even though the item and the corresponding link is correct.
  4. When they are added and not yet discovered battery powered z-wave device are added as online, yet they do not have any channels at all and are not usable in any way.

Improvement to things

Currently the thing lifecycle does not properly represent these steps. Thus it should be extended accordingly:

  1. Currently we have the INITIALIZING status for things and only there should it possible to add channels to the things. Once it goes ONLINE the channels should be "frozen" and not be added/removed any more
  2. A new status detail CHANNEL_DISCOVERY. If should be used when this thing is still being discovered. When using textual thing configuration and channels are provided there the CHANNEL_DISCOVERY should be skipped because the channels are supplied by the user. If there are no channels provided the CHANNEL_DISCOVERY should still run.
  3. A new status detail REGISTERING_OPENHAB. Some things support sending notifications to a host controller. Most of the time this has to set up in some way so this should be the status after CHANNEL_DISCOVERY. Some device (e.g. Shelly) allow to set this persistent on the device so there should be an option to skip this is the user has done this.
  4. Make it possible to share the discovered channels
    Currently it's possible to share the thing through the Code tab. However without the discovered channels the same thing configuration will lead to totally different thing behavior depending on the device. Thus the discovered channels should also be shown in the Code tab.
  5. Save/restore the discovered channels
    I think this already implemented but currently it does not work for textually defined things. When channels are discovered they should be saved and after an openHAB restart restored accordingly assuming the device is still running correctly.
    Some bindings implemented a custom storage (e.g. the node_xxx.xml of the z-wave binding) which should be standardized.
    Some bingings also add possibilities to create thing types through parameters (ecovacs) which should also be standardized.
  6. Add a validation for thing channels / config / etc
    Sometimes devices change (e.g. firmware update) and the user forgets to re-add the thing to openhab. When the thing goes online there should be a one time check which validates everything and logs a warning/error in case of inconsistencies.

I'm aware that things more or less "work" right now but these issues are very hard to pin down even for experienced openHAB users and lead to very frustrating behavior (e.g. it works after one restart but doesn't after another),
Anything that makes Things more reproducable is worth the effort and a step in the right direction.

@spacemanspiff2007 spacemanspiff2007 added the enhancement An enhancement or new feature of the Core label Jan 17, 2024
@rkoshak
Copy link

rkoshak commented Jan 17, 2024

Thus the discovered channels should also be shown in the Code tab.

I don't have any opinions on this RFC except that at least for Zwave and Zigbee, the discovered channels are shown in the Code tab. If they are not shown for other bindings (Shelly), maybe that part at least is a problem with the binding implementation instead of core?

but currently it does not work for textually defined things.

Isn't it the whole point of textually defined things to textually define everything and not do discovery? It seems very weird to me to have a textually defined Thing but not include the Channels in the textual definition. I'm totally on board with standardization of this sort of thing.

@spacemanspiff2007
Copy link
Contributor Author

... at least for Zwave and Zigbee, the discovered channels are shown in the Code tab.

This is what gets shown for a fully discovered Smart Color Button (Z-Wave) with multiple channels.

UID: zwave:device:controller:node33
label: "Philio PSR04 (Node 33): Smart Color Button"
thingTypeUID: zwave:philio_psr04_00_000
configuration:
  config_1_1: 0
  config_2_1: 99
  config_10_1: 12
  wakeup_node: 1
  wakeup_interval: 86400
  group_1:
    - controller
  config_25_1: 0
  config_26_1: 1
  group_2:
    - controller
  node_id: 33
bridgeUID: zwave:serial_zstick:controller

For me it does not show the channels there (OH 4.0.0). Do you see something different?

It seems very weird to me to have a textually defined Thing but not include the Channels in the textual definition.

Since the docs are always lacking the idea is to just define the plain thing and still have auto discovery / automatic channel creation. Once the device channels are properly discovered one can than edit the textual thing config to include the channel definition if desired (which then would be available from the code tab).
Basically discover what you use and then "freeze" that.


Another idea would be to have an additional thing state TARGET_INITIALISATION with the status detail CHANNEL_DISCOVERYand REGISTERING_OPENHAB.
That way the device states would clearly be decoupled from the thing initialization states which IMHO makes sense

@rkoshak
Copy link

rkoshak commented Jan 18, 2024

This is what gets shown for a fully discovered Smart Color Button (Z-Wave) with multiple channels.

That's weird as all of my Zwave and Zigbee (and Shelly for that matter) Things have all their Channels listed in the Code tab.

Here are a few but I've looked at them all.

UID: zwave:device:zw_controller:node7
label: Home Energy Meter
thingTypeUID: zwave:aeon_dsb09_00_000
configuration:
  config_10_1: 5
  config_102_4_00000100: 0
  group_1:
    - controller
  config_102_4_00002000: 0
  config_101_4_00000008: 1
  config_101_4_00000800: 1
  config_101_4_00000004: 1
  config_101_4_00000400: 0
  config_112_4: 720
  config_101_4_00000001: 0
  config_101_4_00000002: 1
  config_101_4_00000200: 1
  config_103_4_00002000: 0
  config_103_4_00000100: 0
  config_255_4_wo: 0
  config_102_4_00001000: 0
  config_1_2: 110
  config_101_4_00001000: 1
  config_7_2: 25
  config_9_1: 5
  config_5_2: 25
  config_102_4_00000008: 0
  config_102_4_00000800: 0
  config_103_4_00001000: 0
  config_11_1: 5
  config_102_4_00000004: 0
  config_102_4_00000400: 0
  config_102_4_00000002: 0
  config_102_4_00000200: 0
  config_102_4_00000001: 0
  config_110_4_wo: 0
  config_113_4: 720
  config_101_4_00000100: 1
  config_111_4: 720
  config_101_4_00002000: 0
  config_103_4_00000001: 0
  config_103_4_00000002: 0
  config_103_4_00000200: 0
  config_103_4_00000004: 0
  config_103_4_00000400: 0
  config_3_1_00000008: 0
  config_103_4_00000008: 0
  config_103_4_00000800: 0
  config_3_1_00000004: 0
  config_3_1_00000002: 0
  config_3_1_00000001: 0
  config_100_4_wo: 0
  config_6_2: 25
  config_8_1: 5
  config_4_2: 25
  node_id: 7
bridgeUID: zwave:serial_zstick:zw_controller
location: Global
channels:
  - id: meter_watts
    channelTypeUID: zwave:meter_watts
    label: Electric meter (watts)
    description: Indicates the instantaneous power consumption
    configuration: {}
  - id: meter_kwh3
    channelTypeUID: zwave:meter_kwh
    label: Electric meter (kWh) 3
    description: Indicates the energy consumption (kWh)
    configuration: {}
  - id: meter_kwh2
    channelTypeUID: zwave:meter_kwh
    label: Electric meter (kWh) 2
    description: Indicates the energy consumption (kWh)
    configuration: {}
  - id: battery-level
    channelTypeUID: system:battery-level
    label: Battery Level
    description: null
    configuration: {}
  - id: meter_kwh
    channelTypeUID: zwave:meter_kwh
    label: Electric meter (kWh)
    description: Indicates the energy consumption (kWh)
    configuration: {}
  - id: meter_reset
    channelTypeUID: zwave:meter_reset
    label: Reset Meter
    description: Reset the meter
    configuration: {}
  - id: meter_kwh1
    channelTypeUID: zwave:meter_kwh
    label: Electric meter (kWh) 1
    description: Indicates the energy consumption (kWh)
    configuration: {}
  - id: meter_watts1
    channelTypeUID: zwave:meter_watts
    label: Electric meter (watts) 1
    description: Indicates the instantaneous power consumption
    configuration: {}
  - id: meter_watts2
    channelTypeUID: zwave:meter_watts
    label: Electric meter (watts) 2
    description: Indicates the instantaneous power consumption
    configuration: {}
  - id: sensor_power2
    channelTypeUID: zwave:sensor_power
    label: Sensor (power) 2
    description: Indicates the energy consumption (kWh)
    configuration: {}
  - id: sensor_power
    channelTypeUID: zwave:sensor_power
    label: Sensor (power)
    description: Indicates the energy consumption (kWh)
    configuration: {}
  - id: sensor_power3
    channelTypeUID: zwave:sensor_power
    label: Sensor (power) 3
    description: Indicates the energy consumption (kWh)
    configuration: {}
  - id: meter_watts3
    channelTypeUID: zwave:meter_watts
    label: Electric meter (watts) 3
    description: Indicates the instantaneous power consumption
    configuration: {}
  - id: sensor_power1
    channelTypeUID: zwave:sensor_power
    label: Sensor (power) 1
    description: Indicates the energy consumption (kWh)
    configuration: {}


UID: zwave:device:zw_controller:node4
label: Family Room Outlet
thingTypeUID: zwave:ge_45605_00_000
configuration:
  switchall_mode: 255
  nodename_location: ""
  config_3_1: 0
  config_4_1: 0
  nodename_name: FamilyRoomLamp
  node_id: 4
bridgeUID: zwave:serial_zstick:zw_controller
location: Family Room
channels:
  - id: config_decimal_param3
    channelTypeUID: zwave:ge_45605_00_000_config_decimal_param3
    label: Night Light
    description: Defines the behavior of the blue LED.
    configuration: {}
  - id: switch_binary
    channelTypeUID: zwave:switch_binary
    label: Switch
    description: Switch the power on and off
    configuration: {}

UID: zigbee:device:zg_coordinator:0022a3000040d061
label: Front Porch Light Switch
thingTypeUID: zigbee:device
configuration:
  zigbee_macaddress: 0022A3000040D061
bridgeUID: zigbee:coordinator_ember:zg_coordinator
location: Front Yard
channels:
  - id: 0022A3000040D061_1_switch
    channelTypeUID: zigbee:switch_onoff
    label: "ON_OFF_LIGHT: Switch"
    description: null
    configuration:
      zigbee_reporting_polling: 900
  - id: 0022A3000040D061_3_switch
    channelTypeUID: zigbee:switch_onoff
    label: "ON_OFF_LIGHT_SWITCH: Switch"
    description: null
    configuration:
      zigbee_reporting_polling: 2147483647

UID: zigbee:device:zg_coordinator:282c02bfffe8f86f
label: Rich's Button
thingTypeUID: zigbee:device
configuration:
  zigbee_macaddress: 282C02BFFFE8F86F
bridgeUID: zigbee:coordinator_ember:zg_coordinator
channels:
  - id: 282C02BFFFE8F86F_1_batterylevel
    channelTypeUID: system:battery-level
    label: Battery Level
    description: null
    configuration:
      zigbee_reporting_polling: 7200
  - id: 282C02BFFFE8F86F_1_dimmer
    channelTypeUID: zigbee:switch_level
    label: "REMOTE_CONTROL: Level Control"
    description: null
    configuration:
      zigbee_reporting_polling: 2147483647
  - id: 282C02BFFFE8F86F_1_batteryalarm
    channelTypeUID: zigbee:battery_alarm
    label: Battery Alarm
    description: null
    configuration:
      zigbee_reporting_polling: 7200
  - id: 282C02BFFFE8F86F_1_batteryvoltage
    channelTypeUID: zigbee:battery_voltage
    label: Battery Voltage
    description: null
    configuration:
      zigbee_reporting_polling: 7200

The last Thing shown above was added last week so it's not just the old Things I have hanging around.

I wonder what the difference is. Maybe you are doing a mixture of .things files and looking in MainUI?

I cannot find a single one of my Zwave or Zigbee Things that does not list all the Channels in the Code tab. In truth I cannot figure out how you can even have a Thing with Channels showing on the Channels tab without them showing on the Code tab. Both the Channels tab and the Code tab are rendered from the same JSON so the UI would have to go out of it's way to not show the Channels in the Code tab and I can't think of a good reason to do that.

I'm on 4.1 but I don't recall ever not seeing the Channels in the code tab for any Thing that has Channels. All of my things are auto discovered where that's possible.

Do you see Channels in the Channels tab? In the RAW JSON from the REST API?

Since the docs are always lacking the idea is to just define the plain thing and still have auto discovery / automatic channel creation. Once the device channels are properly discovered one can than edit the textual thing config to include the channel definition if desired (which then would be available from the code tab).
Basically discover what you use and then "freeze" that.

Wouldn't all that become moot when the YAML config files become a thing? Then the user can discover the Thing and, if desired, export the whole lot to a text file in one step, possibly even through an export function. Is there a case where the Thing has to be manually defined but the Channels are discovered? All my experience is that either the Thing is completely discovered or the Thing is completely manually defined.

@spacemanspiff2007
Copy link
Contributor Author

Do you see Channels in the Channels tab? In the RAW JSON from the REST API?

Yes - the rest response for the thing correctly returns the channels and the gui shows them correctly in the channels tab

grafik

But we digress ...


Wouldn't all that become moot when the YAML config files become a thing?

I don't think so. If I have to add 10 rollershutters it's very quickly to do so through a textual config (e.g. set the IP, proper name, and label). But I am not sure about the channels so I want to use auto discovery in a first step and only in a second step define the channel configuration.
The thing type is typically very clear from the openHAB docs but the channel types typically aren't.
Also if we talk backwards compatibility:
All my shelly things are currently defined in *.things files however the channels have to be auto discovered.
Additionally it will only become moot if we finally have a separation between thing settings and device settings (see other issue) because until then it's not possible to change device settings if the thing is textually defined.

Btw. I just looked at my things again one flood alarm and one smoke alarm is still not working after over two weeks because it's stuck in CONFIG PENDING.

@rkoshak
Copy link

rkoshak commented Jan 19, 2024

I don't think so. If I have to add 10 rollershutters it's very quickly to do so through a textual config (e.g. set the IP, proper name, and label). But I am not sure about the channels so I want to use auto discovery in a first step and only in a second step define the channel configuration.

That seems like even more work than discovering the Thing in the first place, exporting the YAML complete with Channels, and then copy/paste/editing from there. I don't see any benefit to starting with a file defined Thing and only then going to the UI to pull the Channels and then needing to modify your already created Thing definition with the Channels. It just seems like more work.

@spacemanspiff2007
Copy link
Contributor Author

You assume that all devices are fully discoverable by auto discovery in the first place which is not always the case. Sometimes the thing discovery also does not work or only works after providing some form of data (e.g. I want to create 10 astro things which require the coordinates). Also the thing type discovery could be faulty so it's good to have an option to set the type.
Last but not least this would be downwards compatible with the current implementation in .things file so everyone could benefit from the new things

@rkoshak
Copy link

rkoshak commented Jan 22, 2024

You assume that all devices are fully discoverable by auto discovery in the first place which is not always the case.

Are there cases where the Thing is manually created but the Channels are discovered? I know of places where the Things are wholly manually configured and other cases where the Things are wholly discoverable. I know of no cases where one has to manually create the Thing but the Channels are discovered.

Is there a binding that works like this? Or is it s theoretical concern?

I'm not talking about Bridges though, that's a different use case.

Sometimes the thing discovery also does not work or only works after providing some form of data (e.g. I want to create 10 astro things which require the coordinates).

I don't see any difference in this use case. Astro does autodiscover a Thing (two Things actually, one for sun and one for moon) based on the regional settings coordinates.

And this use case too seems to be more work to:

  1. define the Thing in a .things file
  2. navigate to the Thing in MainUI, open the code tab
  3. copy the Channels
  4. incorprate the Channels into the .things file
  5. copy/paste/edit to create new Things

compared to

  1. discover the Thing from the inbox
  2. navigate to the Thing in MainUI, open the code tab
  3. copy the full Thing definition and paste into a .things file
  4. copy/paste/edit to create new Things

Even if for some reason the Thing cannot be discovered (assuming the theoretical case), the fact that you don't have to jump between the editor to MainUI and then back to the editor is an improvement in workflow. At least for cases where the initial configuration of the Thing is simple like with Astro.

Also the thing type discovery could be faulty so it's good to have an option to set the type.

Seems like that should be caught and fixed instead of hidden and/or worked around. Do we have lots of cases of this? I've seen some stuff around Shelly but haven't followed that closely.

@spacemanspiff2007
Copy link
Contributor Author

Are there cases where the Thing is manually created but the Channels are discovered? I know of places where the Things are wholly manually configured and other cases where the Things are wholly discoverable. I know of no cases where one has to manually create the Thing but the Channels are discovered.

Many bindings do not support auto discovery because it's just not possible. And almost all the time when the user currently supplies the thing via text file the channels are automatically created or discovered (since from the user perspective this is the same I'm not going to differentiate between these two).

As you can see the docs provide the thing definition but never the channel definition.


And this use case too seems to be more work to:

1. define the Thing in a .things file
2. navigate to the Thing in MainUI, open the code tab
3. copy the Channels
4. incorprate the Channels into the .things file
5. copy/paste/edit to create new Things

compared to

1. discover the Thing from the inbox
2. navigate to the Thing in MainUI, open the code tab
3. copy the full Thing definition and paste into a .things file
4. copy/paste/edit to create new Things

For me it's like this (since I want an astro thing for a different location).

  1. Copy past from binding docs
  2. Edit location and save
  3. navigate to the Thing in MainUI, open the code tab
  4. ...

compared to

  1. Click + on things file
  2. Select astro binding
  3. Select "astro:sun"
  4. Edit location
  5. Click "Create Thing"
  6. navigate to the Thing in MainUI, open the code tab
  7. ...

However the discussion which one is faster is futile since obviously you are faster with the gui and I am faster with text files.
Since I am going to create the thing file anyway it's easier for me to start that way.
Even if I am/would be slower for me it's easier to work that way.
Channel discovery always comes after thing creation so there is no reason to prevent automatic channel discovery even when the thing was created through text.


Seems like that should be caught and fixed instead of hidden and/or worked around. Do we have lots of cases of this? I've seen some stuff around Shelly but haven't followed that closely.

The problem with these issues is that they are really hard to identify, harder to debug and even harder to reproduce. I consider myself an above average openHAB user and it took me very long to find the cause of my Shelly issues. And even with the help of the binding developer we were not able to pin the issue.
I know for a fact of two other openHAB users who face the same issues, one even stopped using the Shelly binding completely because it was so unstable and switched to MQTT (which the Shellies luckily support out of the box).
Additionally I've read a few forum posts that seem to be related to that issue so I would definitely say it's an issue but I can't quantify how big it is. I guess many users just restart openHAB once or twice which most of the time fixes the issue so they never bother to report it. Or in case of a smoke alarm or a flood alarm they just don't realize it because how often do you have a fire or a flood.

Since I've found more and more bindings which do the automatic channel generation (e.g. not yet released esphome binding, too) it's necessary to address these systematic issues because all bindings which do dynamic channel discovery are affected by this.

@rkoshak
Copy link

rkoshak commented Jan 23, 2024

  1. Click + on things file
  2. Select astro binding
  3. Select "astro:sun"
  4. Edit location
  5. Click "Create Thing"
  6. navigate to the Thing in MainUI, open the code tab
  7. ...

That includes a bunch of unnecessary steps that are only really necessary only if something went wrong.

  1. If you already have an Astro thing skip to step 5
  2. Click the Inbox
  3. If Astro was just installed, you will already have a local-sun and local-moon Thing in the Inbox. Select local-sun and go to step 5
  4. If you've previously ignored the local-sun Thing, click "show ignored". Select the local-sun Thing and go to step
  5. Only if something went wrong do you need to click + and go through all the steps to create the Thing. Note the location can be 0,0 for all it matters. You're going to edit that in the .things file anyway.
  6. Navigate to the Thing in MainUI, open the code tab ...

But that's getting off topic.

To summarize:

  1. Standardization is good. Individual bindings shouldn't need to invent a new way to store information about discovered Things.
  2. I don't understand how/why you do not see the Chanels in the Code tab even for discovered Things. That is IMO a bug, maybe worthy of a separate issue. You should always be able to see the Chanels in the code tab and it definitely shouldn't be the case where it's different between different users.
  3. I think a distinction needs to be made between truly automatically discovered Channels and hard coded Channels. Astro and Network binding Things always have the same Channels. These are not auto discovered, they are hard coded. I don't think it makes sense to treat them the same as the Channels of the Shelly or Zwave binding (for example).
  4. I have no opinion on the rest.

@openhab-bot
Copy link
Collaborator

This issue has been mentioned on openHAB Community. There might be relevant details there:

https://community.openhab.org/t/get-data-from-webserver-provided-by-esp8266/154112/17

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement An enhancement or new feature of the Core
Projects
None yet
Development

No branches or pull requests

3 participants