Smart Home Enthusiasts Guide To MQTT

What is MQTT

MQTT (Message Queuing Telemetry Transport) is a lightweight messaging protocol for specifically suited to Internet of Things devices. It’s useful for handling things like command, control, and status messages between devices, known as MQTT Clients, in an asynchronous way that remains reliable even when devices are not always online and listening.

Invented in 1999, MQTT has been widely used in many applications across industry, commerce, and now the smart home. MQTT is overseen by a technical committee consisting of several major tech companies, and as of v3.1.1 the protocol has been ratified by the ISO as a standard.

MQTT uses a publish/subscribe model where devices can send messages via a broker that can be retrieved by subscribers that are interested in those specific messages. These ‘message types’ are known as Topics and allow clients to target exactly which messages they want to know about. By sending these messages through a broker, clients do not need to maintain connections between each other and can receive messages when convenient.

MQTT is can be used on a local network or through the cloud and supports the use of TLS for secure communication and authentication. The very light nature of MQTT messages and the publish/subscribe model allows the protocol to scale to literally millions of devices, handle bi-directional communication, and do so reliably thanks to configurable levels of Quality of Service (QoS).

MQTT publish/subscribe model via a broker

Where is MQTT used in the smart home?

MQTT is one of those ‘workhorse’ protocols that quietly do their thing in the background without most smart home users ever being aware of its existence. Major consumer smart home platforms like Alexa, HomeKit and Google Home don’t really use MQTT directly, and you’ll never see it mentioned in those contexts.

Devices themselves, however, often do use MQTT for sending commands and status updates between devices and their first party apps and cloud services. Brands like Meross, Nuki, and even Ring use MQTT for command and control in this way, but it’s all behind the scenes. It’s for these reasons more enthusiast focused platforms like SmartThings and Home Assistant surface MQTT as an option. It allows for the integration of supported devices directly into the smart home instead of having to communicate via the maker’s cloud service.

For those wanting local control, and the better performance and reliability that comes with it, MQTT can be the tool that makes this possible. For those who want to get into building your own smart solutions, MQTT provides a great way of managing communication with things based on the popular Tasmota SoCs or Arduinos and your smart home.

MQTT Basics

MQTT Broker

The broker is a centralized service that receives published messages and passes them on to subscribers that are interested in them. Each MQTT message includes a topic that other clients can subscribe to and this, along with QoS settings, determines message routing. Unlike the clients, the broker needs to be an always-on service. There are commercial brokers that can be purchased, public brokers on the internet that can be used, and local open-source broker services that you can run yourself.

MQTT Client

Every device that uses MQTT is a client. Clients can be publishers or subscribers, or both depending on their use case. For example, A smart bulb might subscribe to a topic to receive commands to change its power state, brightness and color, and publish such changes to another topic for other clients that need to display or act on this information, such as a smart home app, dashboard, or automation server.

MQTT QoS

Quality of Service is defined per message by the publisher and determines how aggressive the broker will be in making sure it’s delivered. This where the reliability of MQTT comes from, and how it far outstrips more basic methods like HTTP for messaging. MQTT has three levels available for clients to use:

1. At most once (level 0) - This is similar to UDP packets in that it is sent blindly with no concern for whether it actually was received. You might call it ‘fire and forget’.

2. At least once (level1) - This causes the message to be repeated at regular intervals until the recipient responds. This is a basic check for receipt, but it’s only one way. If the sender doesn’t get the response it will continue to repeat the message anyway.

3. Exactly once (level 2) - Similar to level 1, but with the addition of bi-directional confirmation akin to TCP packets. The recipient acknowledges receipt of the message, the sender then releases the stored message and tells the recipient, and the recipient then confirms the transaction has completed. At this level, the recipient knows if the message is a repeat as it’s flagged as a duplicate. This prevents a command, for instance, being repeated erroneously and allows the recipient to reconfirm that it got it the first time. This is the most useful level in a smart home context as the extra overhead is not really significant in our smaller environments over a local network.

Persistent Sessions

Another aspect that makes MQTT resilient is the ability of the broker to store client information for later use. This ensures a client can pick up right where it left off after being disconnected for any length of time. In the smart home this is relevant for battery powered devices, like sensors, that may sleep for extended periods, or even be offline completely if the battery dies and doesn’t get replaced for a while (we’ve all done that right?).

The broker will store everything it needs to resume communication with a given client including all its topic subscriptions, and QoS level 1 or 2 messages which have not been acknowledged by the client, as well as anything new for the client that was received while it was offline.

Clients have the option not to take up this service by requesting a ‘clean session’ when the initially connect. If such a client disconnects the broker will flush anything queued for that client, including from any previous persistent session it had established.

Retained Messages

Because of the ad-hoc and asynchronous nature of MQTT messages, a new client subscribing to a topic may not receive anything for some time. If the topic is intended to provide the caurrent state of a device this can be a problem, as a new client won’t have any idea what that state is until it is changed and a new message sent. The retained message feature provides a way around this by allowing the last message sent to be held by the broker and sent to any new subscribers on connection.

The onus is on the publishing device to specify a message as ‘retained’, which effectively makes it a snapshot of the last known state of that device. Retained messages are entirely seperate from persistent sessions, as they are primarily used to update new clients, rather than previously existing ones. These retained messages are sticky and will stay on the broker indefinitely until the publisher replaces it or explicitly removes it by sending a new, blank retained message.

Last Will and Testament

This is a pretty cool feature that allows for clients to gracefully inform subscribers they have unexpectedly gone offline. How can a device tell anyone it’s offline? By asking the broker to hold a message in escrow on its behalf - kind of.

This disconnect message is specified in dedicated fields as part of the initial connection to the broker. A keep alive duration is also specified during connection which tells the broker how many seconds to wait before checking up on the device. If no message is received from the client in that time, the broker sends a ping request so it knows if that device is no longer responding. In such an event it will then broadcast the LWT message to all subscribed clients of the specified topic.

Why is this useful? Because of the intermittent nature of messages, it might be quite normal to not receive an update from a device for some time. In a smart home context, you won’t know if that device is working or not which can be a problem, particularly with security related devices. LWT ensures the smart home controller will be informed the device is offline, and you can then take more timely action to resolve the issue.

Using MQTT Topics

The topics used in MQTT are string values with a basic hierarchy. These values can be pretty much anything you like, so if you’re setting up your own MQTT topics you’ll want to thin carefully about how best to structure them. This is dependent on the nature of the device and layout of your home. Wildcards can be used to subscribe to multiple topics at once. This is useful if you want to get updates from all devices of the same type, so making sure the topic structure is consistent is important.

A topic is defined by a case-sensitive simple letter string, with levels separated by a forward slash ‘/’. The number of levels you have is arbitrary. For example:

myhome/groundfloor/bedroom/temperature

kitchen/motionStatus

entry/door/lockstate

Wildcards can be used to select everything in a single level, or multiple levels. To select any value of a single level, replace it with a +, for example:

livingroom/ + /brightness

This will select the brightness topic of any device in the living room.

Conversely, using the # multi-level wildcard allows you to subscribe to all sublevels at once, for eaxmple:

livingroom/lamp/ #

This will capture all the topics for the lamp which might include brightness, power, and color topics.

Conclusion

MQTT is not something you’re likely to come across early in your smart home journey, but if you enjoy tinkering and start getting deeper into more advanced platforms and integrations you are likely to run into it. It’s a relatively simple yet powerful protocol that drives a lot of the underlying communication between devices and apps, so it’s worth having a basic working knowledge of what it does and how it works.

If you start developing your own home-built solutions, you’ll need to get more specifics on implementing the protocol directly. There are open-source client libraries you can leverage in your own code. You can also grab the official specification from mqtt.org and get more in-depth details using the MQTT Essentials eBook from HiveMQ, an enterprise MQTT solutions developer.