IoT Data Collection Servers: DeviceHive, Node-Red, MQTT, Custom?

[hmjswt 9]

@zeke This is a very interesting project.

Doe you have any idea witch sensors you are going to use an how the communication will be.
Maybe Bluetooth or serial ( RS485 )?

I have very good results using the Analog TMP36 in monitoring different places in our garden.
For logging I use Bluetooth and a PC with local MicroChip 24LC1025 EEPROM backup.
I use home made boards with a MSP430G2553IN20

Henk Siewert.

[zeke 693]

@hmjswt I am presently making use of the ds18b20, ds28ae00, and max31851 sensors in my prototype system.

Most IoT demonstration projects that I see online are using just a couple of sensors. I'm kind of sad about that because I intend on using a boatload of sensors. I am thinking that my dashboard will become very complex.

So, I have to spend some time designing what the user interface could be and then research the readily available options.

My instincts tell me that I will probably have to roll my own dashboard interface because of the massive quantity of data that could be shown to the user.

I don't think that I can just display a single graph with a lot of traces on it. I have to come up with a 3D interface or similar abstraction. Maybe use colour as well as position to show the data? 

 

[Rei Vilo 695]

@zeke Do you need to display the temperature of all the 600+ sensors, or only those that have adnormal temperatures and / or trends?

One flow can acquire the data and populate an SQL database, and another search for anomalies and display them. 

[zeke 693]

@Rei Vilo  Good points!

I had been imagining a transparent 3D image of a grain bin or silo that had vertical lines arranged inside of it. 

The vertical lines would have a number of data points along their lengths.

Each data point would change its colour based on the sensor reading.

When viewed as a whole, the sensors that are hot would trend towards the red colour and sensors that are normal would trend towards a green or other neutral colour.

At a glance, the user would understand the relative temperatures inside the grain bin.

I agree that viewing all of the sensor data at once would be foolish and overwhelming.

I also agree that the data should flow into a database for safe keeping before being processed and passed into the User Interface. 

 

[zeke 693]

Additionally, I performed some tests in node-red. 

Using a python program, I published data to five MQTT temperature streams.

I did this at a rate of five samples every 250ms, 500ms, and 1 second.  It liked a 1 second update rate the most.

I composed a node-red dashboard sketch that would display all five streams simultaneously using both a chart and a gauge for each stream.

The dashboard displayed all the data successfully but there was a noticeable lag between publishing and displaying of the data.

The lag has me thinking that chrome may be overloaded during this process. I had a lot of tabs open.

I should note that I am doing all of these tests locally on my Ubuntu desktop machine and RAM isn't a problem. I might have to upgrade the spinning hard disk to an SSD though.

 

Hmm... I just realized that we are talking about node-red more than DeviceHive in this thread. Does it matter to anyone?

 

[Rei Vilo 695]

Have a tried running Node-RED on a dedicated Raspberry Pi Wireless? At USD10, it is worth experimenting to get a precise idea of the lag.

Does your application need to display all the thermometers, or only those with an issue?

7 hours ago, zeke said:

Hmm... I just realized that we are talking about node-red more than DeviceHive in this thread. Does it matter to anyone?

You may rename the thread on the first post

[zeke 693]

Thank you Sir!  Topic edited successfully. 

With regards to the temperature display, I still have to figure out that.

Spontaneously, I imagine that I would display the average temperature of the entire grain container.

If there were a hot spot then I would have to figure out how to indicate a problem to the user.

The challenge is to figure out a paradigm that hides the complexity of the data initially but allows the User to dig deeper to reveal more detail. Kind of like the zooming function of a satellite map - you start out at 30,000 meters and zoom in. New data is presented as you zoom in.  I think this is called Data Decimation. 

The best example I have found is the iOS application called Netatmo - a weather station app. When you look at the temperature graph over a month span, it presents the data with a few data points. As you pinch to zoom, more data points are included and the trending line is recomputed. I admire them for their engineering.