Jump to content
zeke

IoT Data Collection Servers: DeviceHive, Node-Red, MQTT, Custom?

Recommended Posts

I'm going to have to check this out! I was wondering if something open source would pop up after Phant died.  I killed off my IoT devices once I realized that wasn't going anywhere. 

Share this post


Link to post
Share on other sites

Yeah, I was pretty bummed out too when phant died.

Lately, I have been considering rolling my own tiny IoT system. My concept is to start small - very small, and then work outwards and upwards in complexity and scale.

I have a perfect test case for this - my one wire data logger (photos attached).

I also just signed up for a course on Interactive Python Dashboards with Plotly and Dash. I couldn't resist because it was only $10 because it's a new course. Click the link to get the deal, if you want. 

I figure that I can collect the data from my data logger over the serial port, stuff it into a local database, then display/interact with it via dashboard of some sort.

But, it's got to be simple.

I want to keep the complexity, scale, and scope low so that I can achieve success more quickly.

Also, I don't want to get board or overwhelmed before I finish it. I have started so many things and not finished them. Grrr....

 

IMG_4737.JPG

IMG_4736.JPG

Share this post


Link to post
Share on other sites

@zeke Why not give Node-RED a try?

I’m very happy with Node-RED. It provides a GUI IDE, supports natively a variety of formats / protocols like JSON and MQTT, and features its own dashboard. It is expandable with additional nodes, for example to  interface with an SQL database.

See my reviews and projects at 

To get this...

75C4D568-7AF3-4706-8903-66F4ADC25144.jpeg.fe871b686d38a8b109e9f79ed4f075eb.jpeg

... you just need that...

3E592E2A-48C8-422C-9A04-76C3E5C2E3C4.jpeg.067d6251b8b1661219108f943d07346d.jpeg

Share this post


Link to post
Share on other sites

Thank you Sir for recommending node-red. That is very interesting.

In about an hour, I was able to figure out how to:

  1. Install node-red
  2. Connect it to my mosquitto server
  3. Install the node-red dashboard
  4. Configure node-red to display live data from one mqtt channel

Nice!

Now I have to program my datalogger to gather and report data. \o/ yay

node-red Screenshot from 2018-04-30 12-56-59.png

Share this post


Link to post
Share on other sites

Hallo,

Why not program it yourself.
If you can program a TI LaunchPad you can program almost anything...

Processing.png.937f2cd1c8a5184f5600abc7055ee531.png In Processing. It has a lot op functions. Including voice reporting on all functions and data is saved in .CSV file.

Python.png.337de9465972b9cac1ae33c946e4b130.png In Python, live graphic.

Why use third party software when you can do it yourself?

Greetings from the Netherlands,

Henk Siewert

Share this post


Link to post
Share on other sites
17 hours ago, zeke said:

Thank you Sir for recommending node-red. That is very interesting.

In about an hour, I was able to figure out how to:

  1. Install node-red
  2. Connect it to my mosquitto server
  3. Install the node-red dashboard
  4. Configure node-red to display live data from one mqtt channel

You're welcome, and congratulations for the test! I'm impressed by how many things you've achieved in just one hour :)

Share this post


Link to post
Share on other sites

@hmjswt - you could easily say that about a lot of things (C vs Assembly, roll your own database, etc.)  :) 

I'm a web developer - but trying to figure out everything from hardware to reporting can kill a project pretty quickly.  Like @zeke - I've got a lot of unfinished projects - usually because I get bored trying to get over obstacles in the project.  Since DeviceHive supports MQTT - it should be pretty easy to port a project over to something else at a later point in a project.

I was able to get a simple POC working with a Raspberry Pi on DeviceHive.  Not too bad - it'll be a good place to start working on MQTT for some of the hardware I've got sitting around.  I'm not so sure about deploying DeviceHive though - there are a lot of components to install - although they do have docker instructions.

Share this post


Link to post
Share on other sites

@cubeberg I haven't tried DeviceHive because the dashboard (provided by Grafana) is external.

I'm surprised how well Node-RED works on a very basic and cost-effective Raspberry Pi Zero Wireless.

As an additional benefit, the Raspberry Pi Zero Wireless features Bluetooth, so I can connect my SensorTag as per this project or this one.

I'm using a USD25 Chinese tablet running on Android as a screen with Chrome running in kiosk mode.

screenshot-2017-03-16-17-54-28_orig.thumb.png.8d8d6830ce71158e403c087419e0d30d.png

Just like many other hobbyists, I can't afford the not-invented-here syndrome due to lack of time and resources, as you rightly point out. 

Good luck with your project!

Share this post


Link to post
Share on other sites

Wow! All the good ideas are starting to flow. Keep it up everyone!!!

 

@hmjswt, I had no idea Processing could do that. Cool!

Share this post


Link to post
Share on other sites

Now that I have tinkered with a small proof of concept test, I am wonder how far I can scale node-red.

My temperature datalogger is quite the capable beast. The maximum number of sensors should be around 1600 which should only happen in the lab.

Practical use cases would range between 50 and 650 sensors per installation.

In those cases, I am wondering if DeviceHive (or anything else not custom) would be able to receive, store, and display that data in human friendly way?

This is why I am considering a customized software system.

Whatever I chose, it is going to be the path of least resistance to start with.

Share this post


Link to post
Share on other sites
49 minutes ago, zeke said:

Wow! All the good ideas are starting to flow. Keep it up everyone!!!

 

@hmjswt, I had no idea Processing could do that. Cool!

O ja, how about a wether station:

Processing is perfect for this sort of things.

face002.jpg.ac4eef30a4cb1cbf22d3c189ef4bad37.jpg

Share this post


Link to post
Share on other sites
11 hours ago, zeke said:

Now that I have tinkered with a small proof of concept test, I am wonder how far I can scale node-red.

My temperature datalogger is quite the capable beast. The maximum number of sensors should be around 1600 which should only happen in the lab.

You can easily interface Node-RED with a SQL database to store and retrieve all the data. Among many other options, mySQL, SQLlite...

How often each sensor is reporting temperature? How critical is temperature?

Share this post


Link to post
Share on other sites

The DeviceHive python examples for the Pi are keeping my CPU at ~98%

I couldn't get their dashboard tool connected either.  I may mess around with Node-Red and see if that's not too hard.  I used that before with my old IoT project and was able to get it to work pretty well.

Share this post


Link to post
Share on other sites
13 hours ago, Rei Vilo said:

You can easily interface Node-RED with a SQL database to store and retrieve all the data. Among many other options, mySQL, SQLlite...

How often each sensor is reporting temperature? How critical is temperature?

Good questions!

I would like to monitor the temperature of a silo full of grain. Maybe a million bushels?

Any pile of grain is a gigantic thermal sink therefore the temperature ought not to change very rapidly. I would expect the temperature to be sampled no more than once every hour. But, that time interval should be user configurable.

The neat thing about the logger design is that it has a large on board FRAM chip and an SD card slot that I can use as a data buffers. I haven't worked out the capacity yet because I haven't figured out the expected size of one measurement record. The FRAM is a 256Kbyte unit. The SD card is whatever an MSP430 can work with.

The overarching goal is to watch the temperature and humidity of a grain pile. If it gets too wet then it can spoil. If the spoiling gets out of hand then the grain will spontaneously combust. Here are a number of interesting and, in some cases, deadly examples: [1], [2], [3], [4], [5], [6], and [7]. 

 

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

×