Saturday, December 30, 2023

MQTT 102: Add Home Assistant Discovery to your DIY Devices

 


This article will show the steps necessary to add Home Assistant discovery to your MQTT projects and devices.  MQTT discovery means your project will be automatically discovered and integrated into Home Assistant without any YAML configuration necessary on the Home Assistant side.
In a couple of previous articles and videos, I covered the basics of MQTT, how to add MQTT to your DIY electronic projects and how to manually integrate these types of devices into Home Assistant via YAML configuration files.

If you are not familiar with these topics, you may wish to read/watch the following before proceeding:



And of course, there is the video version of this article which you can also watch on YouTube.

Prerequisites


This article will assume that, regardless of the language you are using, you know the basics of how to connect to an MQTT broker and publish messages.  I'll be showing Arudino/C++ code here, but since auto-discovery simply involves publication of special messages and topics, the concept transfers to any language that supports MQTT, whether that be Python, Java, JavaScript, .NET, etc.  If you are unfamiliar with MQTT and how to add MQTT capabilities to your own DIY project, I suggest beginning with the MQTT 101 video listed above.

Also note that both Home Assistant and some language libraries (I'll be using the popular pubsub library) are constantly undergoing changes and some of these future changes may involve changes to my examples.  I'll try to update this article if that occurs, but you should always check the latest official documentation, which you can find linked at the end of this article.

Next, you or anyone that wishes to integrate your MQTT device must have an MQTT broker (standalone or the Home Assistant add-on) and the MQTT integration installed and properly configured in their Home Assistant installation.

Finally, the Home Assistant MQTT integration must have discovery enabled.  This is enabled by default, but can be disabled via the advanced MQTT integration settings.  In addition, the default discovery topic prefix is homeassistant.  But again, this can be changed via the advanced integration configuration.  For most Home Assistant users, discovery will be enabled and will use the default prefix.  The remainder of this article will make those assumptions, but it may be worth noting in your own documentation if publishing a project that uses Home Assistant MQTT discovery.

How Home Assistant MQTT Discovery Works


For a device that implements MQTT, but not Home Assistant MQTT discovery, all of the MQTT topics and payloads for the device must be manually configured via a series of YAML statements in the Home Assistant configuration file(s).  Not only is this prone to error on the part of the end user, but with more recent versions of Home Assistant reduce the need for users to write and understand YAML configurations and therefore many are unfamiliar with using YAML.  Many users may opt to skip your project or not implement your device because of the need to create all the manual YAML configuration.

But when you implement Home Assistant MQTT discovery in your project, you device/project shows up as a new integration (much like any native integration) and can be adopted by the end user with little to no manual configuration.

If the MQTT integration has been installed in Home Assistant, discovery is enabled by default... although it can be disabled by the end user.  

When enabled, Home Assistant will listen on a special topic for a message.  It will then use the payload for that topic to create the device and necessary entities directly in Home Assistant... with no manual YAML required.

The Discovery Topic


To have your device or entities automatically discovered and added to Home Assistant, you have to publish and MQTT message to a special topic.  The overall structure of this topic is:

<discovery_prefix>/<component>/[<node_id>/]<object_id>/config

Discovery Prefix:

As mentioned under the prerequisites, the default discovery prefix is homeassistant and this will work for all Home Assistant users unless they changed this prefix in the advanced MQTT integration settings.  This article will assume the default value is used.

Component:

This must be one of the MQTT supported domains/entity integrations, such as sensor, switch, light, etc.  This is a list of currently supported MQTT integrations as of the time of publication:

Node ID (optional)

This is optional and not used by Home Assistant but may be used to structure the MQTT topic if desired.  I won't be using any Node IDs in my example, but see the official documentation link above if this is something you want to do.

Object ID

This is the ID of the device.  It is not the name or entity ID of the device.  It is used to create different topics for each device/entity.  Note that the object ID must consist of only alphanumeric characters, plus underscore and hyphen.  With most brokers, capitalization also matters.  But to be safe and assure proper use for all brokers, each Object ID should be unique without regard to capitalization.

Examples

Here are two examples of discovery topics for simple temperature and humidity sensors:

homeassistant/sensor/bedroomtemp/config

homeassistant/sensor/bedroomhum/config

A discovery topic for a switch might look like this:

homeassistant/switch/kitchen/config

Just remember that each discovery topic must be unique or it will overwrite the configuration of the previous device or entity that used the configuration topic before.

The Discovery Payload


The discovery payload must be a serialized JSON payload.  It will be validated like a YAML entity.  So, for example, if you configured an MQTT in YAML like this:

- name: "MQTT Temperature
  unique_id: "mqtttem01a"
  state_topic: "stat/mydevice/temperature"
  unit_of_measurement: "°F"

Then your JSON payload should be constructed as follow:

{
   "name":"MQTT Temperature",
   "unique_id":"mqtttemp01a",
   "state_topic":"stat/mydevice/temperature",
   "unit_of_measurement":"°F"
}

You can include any key:value pair that is valid for the type of MQTT entity that you are creating and that you would normally include in a YAML definition.  This includes things like availability topics, command topics, value templates, etc.  See the official documentation for list of valid keys for the type of device/entity you are creating.

You can repeat this process of publishing a discovery topic and payload for each entity that you want to add to Home Assistant and each one will be created as a separate entity.




An Arduino/C++ Example


The process of publishing the MQTT discovery topic and payload should be similar regardless of language... just the syntax will change.  Regardless of language, you just need to create a serialized JSON payload and publish this payload to the special discovery topic.

Special note for the Arduino pubsub library:

If you are using the pubsub MQTT library in your Arduino code, note that it has a default maximum buffer size of only 256 bytes for sending and receiving messages.  This took me a while to discover, as some of my discovery messages were working and others weren't.  Those longer than 256 bytes were silently failing.  And it is very possible that some of these longer payloads could result in exceeding that default.  But you easily increase the buffer size in your MQTT client setup:

  client.setServer(MQTTSERVER, MQTTPORT);
  client.setBufferSize(512); 
  client.setCallback(callback);

In my case, setting the buffer to 512 bytes was large enough for all my MQTT messages.  Don't set this too large if your device doesn't have the memory to handle it.  But I was using an ESP8266 (Wemos D1 Mini) and 512 bytes didn't cause an issue.

Below is just a simple example of how you might implement discovery in an Arduino program.... along with a few notes and recommendations.

void setupHaDiscovery () {
  if (auto_discovery) {
    char buffer[2048];
    DynamicJsonDocument doc(2048);
    doc.clear();
    
    // A simple MQTT Temperature Sensor
    doc["name"] = "MQTT Temperature";
    doc["unique_id"] = "mqtttemp01a";
    doc["state_topic"] = "stat/mqttdiscovery/sensor/temperature";
    doc["unit_of_measurement"] = "°F";
    serializeJson(doc, buffer);

    client.publish("homeassistant/sensor/
                    mqtt_temperature_sensor/ 
                    config", buffer, true);
  } else {
    //remove all entities by publishing empty payloads
      client.publish("homeassistant/sensor/
                      mqtt_temperature_sensor/config", "");
  }
}

First, note the use of a boolean (auto_discovery) to determine whether discovery for this project is enabled or not.  Ideally, if providing your code or project for others, you should provide a convenient way for the end user to enable or disable Home Assistant MQTT discovery for your device/code.  When enabled, the Home Assistant entities will simply be created as soon as the code publishes the discovery message.  Some users may prefer to manually configure the MQTT entities, so not having an option to disable discovery may irk some users.

You can provide an end user option to toggle discovery off/on via a web settings form, by posting an HTTP URL to the device or even by having another MQTT topic your code subscribes to that can toggle the boolean value off/on.  Of course there are other ways as well.

For my devices where I do not provide a web settings interface, I use a simply HTTP post to:  ip_address/discovery_on or ip_address/discovery_off to toggle discovery.  By default, the discovery is off (false) upon device boot.  This way the device can be installed, configured for WiFI, MQTT or whatever setup steps are necessary, then the user can enable discovery when ready and the device/entities will be immediately added to Home Assistant.  When set back to 'off', all previously created entities will be removed.

Removing Discovered Entities


A nice feature is that discovered entities can easily be removed from Home Assistant at any time.  This is done by simply publishing the original discovery topic but using an empty payload.  You can see this in the else branch in the above example:

client.publish(<original discovery topic>, ""); 

This will immediately remove any entities created by the original (or last) discovery message.  Do note that it only removes the entities, so if these were used in automations, scripts, dashboards, etc. in Home Assistant, these will have to be fixed for the now missing entities.

By adding a user-accessible toggle for enabling/disabling the discovery, it allows the user to add or remove the MQTT entities from your device in Home Assistant at any time by just toggling the state of this variable.

Creating a Home Assistant Device


Up until this point, we've only created simple entities.  And in Home Assistant, these entities are unrelated and not tied to a device.  But a device can be created that has multiple entities.  You may be familiar with other integrations that create a device that can be viewed via the settings:


A device may have one or more controls (generally with a default primary control), one or more sensors and one or more "diagnostic" entities.  Having a single device with multiple entities has a lot of advantages, especially for end-users with limited knowledge of YAML and entity configurations.  You can also include device-specific  information like a device name, firmware version, author/manufacturer, etc. that isn't possible when creating manually integrations via YAML.  I'll cover some of these below.

The following shows how to create a single device with multiple entities.

The topic for creating a device with multiple entities is the same as it is for creating an individual entity.  You create the topics and payloads just like you are creating individual entities as above.  Except in the payload of each entity, you include a nested JSON object for the device:

{
  "name":"MQTT Switch",
  "unique_id":"switch043b",
    ...
  "device": {
     "name":"My MQTT Device",
     "identifiers":"mymqttdevice01",
     "manufacturer":"Resinchem Tech",
     "model":"ESP8266",
     "hw_version":"1.02",
     "sw_version":"2.45",
     "configuration_url":"http://192.168.1.226"
  }
}

(Note: most configuration keys have abbreviations that can be used in place of the full key name to keep the total size of the payload smaller.  Abbreviations can be found in the documentation linked at the end of this article).

For Arduino code, the nested object would be created similar to the following (abbreviations used):

    doc["name"] = "MQTT Switch";
    doc["uniq_id"] = "switch043b";
    ...
    JsonObject device = doc.createNestedObject("device");
    device["name"] = "My MQTT Device";
    device["ids"] = "mymqttdevice01";
    device["mf"] = "Resinchem Tech";
    device["mdl"] = "ESP8266";
    device["sw"] = "1.24";
    device["hw"] = "0.45";
    device["cu"] = "http://192.168.1.226/config";
    ... 
    serializeJson(doc, buffer1);


The full device configuration only needs to be in the payload of one entity (recommended to be included with the main control of the device if there is one).  All other entity payloads only need to include "name" and "ids".  All entities where the name and ids values match within the device key will be grouped together as a single device when discovered by Home Assistant.

Example of a Device with Multiple Entities - click to enlarge

The above shows where the various device keys show on the Home Assistant device page.  All entries are optional except for name and ids.  If omitted, the missing values simply won't be shown or will have a value of 'unknown'.

A note on the device ids key:  The 'ids' key can be an array of identifiers as opposed to a single value.  This means an entity could theoretically be a member of more than one device.  This is probably rare for Home Assistant cases, but just know that it is possible if you have a need. 

Again, if you have your code designed so that discovery can be turned off with the appropriate topics publishing an empty payload, then turning off discovery will remove the device and all related entities from Home Assistant.  And if you included unique IDs you can remove and re-add the device/entities multiple times and duplicates will not be created in Home Assistant.

The Importance of Unique Identifiers


You may notice that each of the examples I give above include a unique ID key and value.  The unique ID is used by Home Assistant to 'register' the device in its database.  It is important because having a unique ID allows the entity, including the entity name, to be edited in the Home Assistant front end.  And why is this important?

Without a unique ID, the none of the entity properties can be edited in Home Assistant.  Trying to bring up the properties are met with this message:


But if a unique ID is included, it allows the end user to change the entity id, friendly name and other properties if the ones you provided as part of the discovery isn't what they need or want.  Once the entity has been added, it can be edited by editing the gear icon on the entity 'more info':


Being able to edit the entity id may also be needed for another reason.  Each entity id must also be unique.  So if your discovery uses a device/entity ID name that already exists, Home Assistant will add a numeric suffix to the entity ID.


In this example, MQTT discovery added a new entity called sensor.mqtt_temperature, but an entity with that name already existed in Home Assistant.  Therefore, Home Assistant appended a _2 to the incoming entity name.  Adding a unique ID to your discovery payload will allow this to be corrected directly in Home Assistant by changing one of the entity IDs to a more meaningful name.

But like entity IDs, your unique ID must be distinct across all of Home Assistant.  Unlike entity IDs, Home Assistant will not append a value to a duplicate unique ID.  Instead, an error will occur and your entity will not be created.  So, your code needs to assure a unique ID is generated when use discovery for an entity.  

Ideally, this should be a random generated value that is unique for each separate instance of your device or project.  Many developers use something like a prefix and the last 5 or 6 digits of the MAC address from the controller.  For example, rctsensor34AC24.  According to the official Home Assistant documentation, the following are not considered acceptable sources for a unique ID:
  • IP address
  • Device Name
  • Hostname
  • URL
  • Email address
  • Usernames
You should also avoid, where possible, hardcoding a unique ID into your project code.  Why?  It would prevent someone from installing multiple copies of your device or project within a single Home Assistant installation. For example, let's say you created a super duper multi-sensor.  And a user likes it so well, they want to install three of them in their home.

The first install works just fine.  It creates the device using hardcoded topics and unique IDs.  Now when the user attempts to install the second device, it uses the same discovery topic.  This will actually update the configuration of the initial device (republishing a discovery payload on the same topic updates the existing configuration and doesn't create a new one).

So, even if you have a dynamic topic of some sort, if the entity ID is duplicated, an error will occur and the device will not integrate.

Ideally, you would dynamically create a unique ID at run time based on some device-specific parameter.... like the controller's MAC address.  You can use this dynamically created ID for both the unique ID of the entity and for the discovery topic:

   my_unique_id = "myprefix" + <last 6 MAC characters>

Then when generating the actual discovery topic and payload:

Topic:  homeassistant/sensor/<my_unique_id_temp>/config

Payload:
  { 
     "name":"Temperature",
     "unique_id":"my_unique_id_temp",
     ...
  }

Note: If your device has more of one entity remember that each discovery topic and unique ID for that entity must be unique, so you can't just use the unique_id alone for more than entity.  But you can incorporate it to avoid duplication of another device.

Using this technique would allow multiple versions of your device/project to be installed within a single Home Assistant instance and would likely avoid any conflicts with existing discovery topics/unique IDs.  While it is still very possible that one of your entity IDs could already exist in someone else's Home Assistant, including the unique ID would allow the end user to easily change the entity ID if necessary.  You can see one example of how I used the device's MAC address to create a unique ID that can be used for topics and entities in a sample gist file, linked at the end of this article.

More Information


This article only covers the basics of using Home Assistant discovery with your own MQTT devices and projects.  It skips over many other concepts, like including availability topics and other configuration details that you may wish to include with your own devices.  However, I hope with this basic understanding, you can make a little more sense out of the (sometimes hard to decipher) "official" Home Assistant documentation... which can be incomplete, outdated and often fragmented.

Links



If you'd like to support future content on this blog and the related YouTube channel, or just say thanks for something that helped you out, you can say thanks by buying me a one-off cup of coffee at:


on

No comments:

Post a Comment

To help eliminate spam and to keep all conversations civil, submitted comments are moderated. Therefore your post may not show up immediately. Please be patient as most reviews are completed within a few hours.

Subscribe to: Post Comments (Atom)