One of the first considerations when putting together an M2M / IoT system is what protocol to use between the remote device or asset (the client) and the central server (e.g. the cloud server). This article discusses the factors to consider when choosing a protocol and looks at some of the more popular ones used when connecting remote devices to a server.
Over the years we have seen a huge array of protocols used to communicate between the remote device and the central server. Some have been good, well thought out and worked well, others (most) have not. The ideal situation is where we get to choose the protocol before the project starts, whether that be an existing open protocol or one designed from scratch. The preferred route is to use something open, stable, well understood and well tested instead of something recently invented. However there are situations when an open protocol just doesn’t suit and something more bespoke is required. These days on the rare occasions when we use one of our bespoke protocols (usually for data efficiency reasons) we have one or two that have been used in multiple projects over a long period that have existing robust client and server implementations.
Considerations when choosing a protocol
- Security
Can the desired level of security be achieved using this protocol? This is a difficult one to solve after the event so needs careful consideration. There are many other factors outside the protocol that also impact on security – for more security related info see our blog on M2M / IoT Security. - Data usage efficiency
i.e. anything cellular or satellite based will incur more cost the more data that is used. Other radio technologies simply don’t allow large amounts of data to be sent, especially in a single packet. - Power consumption
Many of the devices we develop are battery powered and therefore the protocol must be one that is suited to low power consumption on the client side. - Readability/simplicity/ease of use
There is no point using something that is more difficult to understand and develop with than need be - you’re going to have to troubleshoot this at some point. This is often overlooked and if you never sort the initial issues, because the protocol you choose is too difficult to debug, then your project never gets off the ground or is plagued with unresolved issues when deployed. - Scalability
Will it scale as the device numbers grow? Can you test the system with many thousands of clients connecting at the same time? - Does it fit the network connection type?
E.g. no point choosing a protocol that requires the device to run a listening socket if the device in on a cellular network that will inevitably use NAT.
The Internet
By definition IoT protocols use the Internet and the same is generally true for older more traditional M2M systems that were born after the turn of the century. There are very good reasons for using the internet and indeed for cellular data there is little option but to use it for data. Just like many other protocols, Internet protocols are layered on top of one another. Nearly all IoT / M2M protocols consist of multiple layers on top of Internet Protocol (IP). Most start with TCP/IP (TCP layered on top of IP). TCP and UDP (both of which sit on top of IP) are the basis of the vast majority of protocols used on the Internet and pretty much all of the ones used for IoT / M2M – there are exceptions but they are beyond the scope of this article.
Popular Protocols in M2M / IoT
Here are some of the popular ones (but there are many others as well as proprietary ones). Unless stated otherwise assume that these layer on top of TCP. For each protocol the advantages and disadvantages are listed.
HTTP
HTTP is the protocol used by the world wide web (the protocol that was used to get this web page to your browser). On the server side you run a web server and handle HTTP requests as you would in any traditional web application.
Advantages
- On the server side the devices connect to a web server so no need to develop your own data collection server. Web servers are freely available, very good at handling a lot of connections, fit into IT infrastructure models easily and are pretty much infinitely scalable.
- The server side is stateless making it efficient and less complex than state based protocols
- The secure version (HTTPS) makes use of SSL/TLS for encryption and certificates for authentication.
- Lots of libraries and example code for many platforms freely available.
Disadvantages
- HTTP headers can be a little bloated if you are not careful, leading to higher data usage.
- You have to know the length of a message before you send it (as the content length needs to go in the HTTP header). This normally is not a problem e.g. a web browser using HTTP typically runs on device with lots of memory so buffering what you are about to send (so you can calculate its length) isn’t a problem but on resource limited electronics, having to know how much you are going to send before you send it can be a big pain.
- The advantage of a stateless server mentioned above can also be a disadvantage if sending multiple messages as the HTTP header and any other state information has to be sent with each message
FTP
File Transfer Protocol is used to send and receive files to/from a server. It has been around a very long time (predates TCP).
Advantages
- The server does not have to be developed from scratch, an off the shelf FTP server can be used. These are easily available and the good ones are robust and well tested. The less good ones have problems when it comes to scaling up to support large numbers of client devices (so choose carefully!).
- If you don’t need a fancy web application then the consumer of the data just has to FTP to the same server to access the uploaded data using an FTP client (e.g. web browsers can be used as FTP clients) or an existing back office system could consume the data, again by just FTPing to the server.
Disadvantages
- Not very data efficient for a few reasons e.g. requires two TCP sockets (one acts as a control channel, the other as a data channel)
- Has some security issues (although FTPS addresses most of these). File and folder permissions need to be carefully assigned.
- The data ends up in files that then need to be parsed and put into a database so you need another layer of processing. Likewise data to go down to the device needs to be extracted from the database into file format ready to be collected from the FTP server. Having data in lots of files (as well as in a database) can be a pain.
- Some relatively simple things are clunky when using FTP e.g. how do you ensure the remote device only downloads data it needs and not data it has already downloaded previously?
- The different active / passive use methods of FTP seem to have a limitless ability to confuse some people (active is the default but usually passive is preferred/required)
MQTT
Unlike HTTP and FTP, which are general Internet protocols, MQTT is specifically aimed at M2M / IoT although it has been used for other things. It is designed with small code/memory footprint and data efficiency in mind.
Advantages
- Publish and subscribe messaging architecture which has the following advantages over command and response
- if the server is a low power device this is much better than for example the server having to run a web server
- Supports different message delivery types e.g. fire and forget messages (think meter reading, if one fails there will be another one soon so no need for guaranteed delivery)
- Philosophy of small code and memory footprint to aid implementation on resource constrained devices
- Built in quality of service metrics
- Data messages can be efficient (e.g. binary if you want it to be) and the header information is also efficient
Disadvantages
- The “publish and subscribe” model is a bit overkill for many applications (including most of the ones we get involved in). For traditional M2M type applications the simple “request and response” type architecture works just fine. But more modern consumer IoT type applications can benefit from the “publish and subscribe” architecture of MQTT.
- Some criticism that it is based on TCP (as are HTTP and FTP) and not UDP. Personally I don’t see this as a major issue. Apparently there is a UDP variant but I’ve never used it.
XMPP
XMPP is an open standard for messaging and presence. XMPP has many uses including IWebRTC, social and IoT.
Advantages
- Mature and extensively used, even before it was applied to IoT
- Open, free, extendable and adaptable with lots of implementations available across multiple platforms
- Security built in
Disadvantages
- XMPP has a large overhead and can be quite inefficient for certain types of data exchange as it’s based on XML
- Binary data has to be base64 encoded before it can be sent
- Guaranteed delivery has to be layered on top of XMPP
Proprietary TCP
You can make your own protocol on top of TCP. Remember TCP layers on top of IP (often referred to as TCP/IP). TCP is a connection based protocol so a connection (often called a socket) is established at the beginning of a connection and is unestablished at the end of a connection.
Advantages
- Compared to UDP, TCP provides reliable, end to end, error free, in order data. This means that when you put something into one end of a TCP connection (a socket) it will pop out the other end exactly as intended (or else the socket will have closed if the link has been lost).
- TCP supports cumulative acknowledgements, selective acknowledgments, window scaling, flow control, congestion control and many more advanced features that are not part of UDP - that’s not to say such things could not be achieved with UDP but the desired functionality would have to be built on top of UDP i.e. built into the proprietary UDP protocol.
- Compared to some of the other protocols mentioned above it is possible to make a proprietary TCP protocol more efficient in terms of data usage and make it easier to implement on resource constrained devices e.g. not having to know the data length before sending as with HTTP (at least at the application layer).
Disadvantages
- Compared with UDP setting up and tearing down the connection (socket) can use a lot of data – although if you only connect/disconnect once per session this is often not too significant.
- Compared to UDP the header is bigger (as it must contain state and other information about the connection)
- Proprietary is not standard
Proprietary UDP
You can make your own protocol on top of UDP. Remember UDP layers on top of IP. UDP is connectionless i.e. there is no session/connection maintained so no setup or tear down is carried out. UDP is datagram based i.e. you fire off datagrams to an IP address and they may or may not arrive at the other end and the order of reception is not guaranteed to be as they were sent.
Advantages
- Compared to TCP, UDP has less header information per packet because it does not maintain a session (socket). There is also no session setup (at the start) or tear down (at the end) so the total data used can be much less than TCP based protocols.
- Suited to situations where end to end error correction is not required or would not be efficient e.g. when streaming real-time video you just want to keep sending the next piece of data and not stop to resend previous data that might have got lost or arrived out of order etc..
Disadvantages
- You lose (or have to implement yourself) a lot of the nice features of TCP e.g. end to end error correction, cumulative acknowledgements, selective acknowledgements, window scaling, congestion control etc. etc. We’ve seen a few customers come to us with UDP protocols saying it all worked fine until we had to shift a lot of data or started to download firmware. This is usually because their simple UDP based protocol requires an ACK for every packet and when you consider the turn-around time for this on cellular can be seconds it’s not hard to see why it becomes painfully slow – with TCP based protocols this is usually not an issue.
- When it comes to Iot/M2M, UDP based protocols are generally more difficult to get right and require more development time than TCP based protocols - there are more pitfalls. E.g. We’ve seen weak UDP protocols with very hard to find message sequence bugs.
- Proprietary is not standard
Troubleshooting
The ability to identify and debug protocol issues cannot be overstated – we have had customers come to us with remote devices using proprietary binary protocols with no debug information available anywhere (no serial port, no JTAG, nothing) and lots of problems. This is where our experience of protocol sniffing and decoding come to use but I’d rather spend the customers time on something more productive to be honest. After all a simple TTL serial port adds zero cost to the bill of materials but could save weeks or months of development/debugging time.
We usually develop both sides of the connection i.e. device firmware and server side software and in these cases things tend to run a lot smoother. We can put in all the nice diagnostics information at key points in the system (on both sides of the connection) making any issues very easy to solve.
Conclusion
The choice of M2M / IoT protocol is a very important one. Getting stuck with the wrong protocol can hamper a project for years after the decision is made. As important is the format of the messages e.g. CSV, JSON, XML, YAML, binary etc….. This could be the topic for a follow on blog in future. In future blogs we will illustrate using some of these protocols to connect devices like Raspberry Pi or Arduino to our web platform MyM2M® Web.
How Can We Help?
We know how complex M2M / IoT systems can be and how difficult protocol related problems can be to resolve. We’re experts at using protocols in IoT/M2M systems. If you think your organisation could benefit from our expertise in this area, or any other, we’d love to help.
Even if you want to do all this yourself a small amount of time talking to us could save many headaches and much development time (and money).