What is a good package data format to send over LoRaWAN?

Lora packets need to be small and it is not a good practice to send ASCII or JSON. It is advised to pack as much data using byte (or even bit) streams.

Example:

You have to send temperature and humidity value from a sensor which measures temperatures between 0 and 99 degrees celsius and humidity between 0 and 99%. The project specifies that it is enough to receive temperature and humidity values rounded to the closes whole number. Value that you measure and want to send is (eg.) 28 degrees celsius and 54% humidity.

JSON:
A longer JSON would look like this:

{"sensor":{"temp": "28","hum":"54"}}

if we format it a bit, it can get a bit shorter:

{"s":{"t":"28","h":"54"}}

but this is still a long format taking 25 bytes (200 bits).

ASCII:

Using plain ASCII might shorten it to just 6 bytes (48 bits):

T28H54

Or even more to 4 bytes (32 bits) if you know that the first two digits are temperature and the last two humidity

2854

Bit stream:

The most efficient way of sending data (out of these three) would be in a bit stream.

Let us take the values and write them in binary:
28(10) = 00011100(2)
54(10) = 00110110(2)

Now, since the values cannot go over 100, which is in binary 01100100(2), we know that the first bit will never change we can say that if the first bit is 0, the next 7 will be the value of a temperature and if it is 1, the next 7 will be the value of humidity.

Therefore, you would only need to send 2 bytes (16 bits). It is fair to say that there are ways to reduce it further, ways, which will be explained in detail in a different post.

00011100 10110110

2 Likes

Digging a little deeper on this matter, let’s say you have two values to send:

  1. temperature from -50ºC to 150ºC in 0.5ºC steps
  2. humidity from 0% to 100% in 1% steps

The following table is formed by the power of two:

bits	values			bits		values
1		2				9		512
2		4				10		1024
3		8				11		2048
4		16				12		4096
5		32				13		8192			
6		64				14		16384
7		128				15		32768
8		256				16		65536

Now we calculate the values and the amount of bits needed:

  1. for temperature: (150 - (-50)) / 0.5 = 400 values total, fits in 9 bits
  2. for humidity: (100 - 0) / 1 = 100 values total, fits in 7 bits

Total 9 bits (temperature) + 7 bits (humidity) = 16 bits. The bitlength of 16 bits fits exactly into two bytes (2x 8bits). A LoRaWAN payload consists of X amount of bytes, so per payload only a multiple of 8 bits can be sent. If you need to send 17 bits for example, 3 bytes are needed: 2 for the first 16 bits and one for the last bit (the remaining 7 bits can be reserved for further use).

Now let’s say the temperature is 25ºC and the humidity is 63%.

  1. temperature value = (actual value - minimum value) / step = (22.5 - (-50)) / .5 = 145
  2. humidity value = (actual value - minimum value) / step = (63 - 0) / 1 = 63

Convert to bits:

  1. temperature: 145 = 010010001 (9 bits)
  2. humidity: 63 = 0111111 (7 bits)

Will yield a bitstream (append temperature and humidity):
0100101100111111 = 0x4B, 0x3F (two bytes).

1 Like

Hereby some C examples on how to put and get data to/from a bitstream / array.

Please note that these functions are not tested yet…

Usage:

uint16_t temperature = 145;
uint8_t humidity = 63;

putData(temperature, 0, 9);			// put 9 bits temperature at offset 0 
putData(humidity, 9, 7);			// put 7 bits humidity at offset 7

Functions:

uint32_t bitstream[32];			// assign an array of 128 bytes (32 * 4) 

void putData(uint32_t data, uint16_t offset, uint8_t bitSize)		// maximum data size = 32 bits
{
	if (bitSize == 0) return;										// filter invalid length
	bitSize = (bitSize - 1) & 31;									// limit to 0...31 shifts (1...32 bits)
	uint32_t bitMask = ~(0xFFFFFFFE << bitSize);					// set bitmask
	data &= bitMask;												// clear redundant bits
	bitSize += offset & 31;											// bitSize is now right bit index
	offset >>= 5;													// offset is now word index
	if (bitSize > 31)												// spread over two 32 bit words?
	{
		bitSize -= 32;												// adjust shifts to 32 bits
		bitstream[offset] &= ~(bitMask >> (bitSize + 1));			// clear addressed bits
		bitstream[offset] |= data >> (bitSize + 1);					// fill with data
		offset++;													// advance to next word
	}
	bitstream[offset] &= ~(bitMask << (31 - bitSize));				// clear addressed bits
	bitstream[offset] |= data << (31 - bitSize);					// fill with data
}

uint32_t getData(uint32_t offset, uint8_t bitSize)					// maximum data size = 32 bits
{
	if (bitSize == 0) return 0;										// filter invalid length
	bitSize = (bitSize - 1) & 31;									// limit to 0...31 shifts (1...32 bits)
	uint32_t retVal = 0;											// initialize return value
	uint32_t bitMask = ~(0xFFFFFFFE << bitSize);					// set bitmask
	bitSize += offset & 31;											// bitSize is now right bit index
	offset >>= 5;													// offset is now word index
	if (bitSize > 31)												// spread over two 32 bit words?
	{
		bitSize -= 32;												// adjust shifts to 32 bits
		retVal = (bitstream[offset] & (bitMask >> (bitSize + 1))) << (bitSize + 1);	// get high part of data
		offset++;													// advance to next word
	}
	retVal |= (bitstream[offset] & (bitMask << (31 - bitSize))) >> (31 - bitSize);	// get data
	return retVal;
}
2 Likes

A fantastic explanation which can be understood by anyone without even prior knowledge on the topic. No where on the internet I could find such sort of detailed explanation. :+1: @Rolf Thanks

3 Likes

Hi

I am new to LoRa but have experience of saving and sending numerical data on MODBUS.
I usually convert the data to integers. Then send the binary data in one byte or two bytes or even 3 or 4 bytes to cover large numbers! The definition of the data type is given in the documentation for MODBUS Data tables. The receiver can then decode and display the data in suitable formats.

This is working fine for MODBUS.

Hope this technique can be used to send data on LoRa by reducing the number of bytes to be sent.

Hope this helps some.