LoRaWAN TTN und openSenseMap


#3

Eine weitere gute Seite um die LoRa Verfügbarkeit über TTN zu checken ist https://ttnmapper.org/


#4

Hallo Heiner und Gerald,

wie schon geschrieben, die Daten kommen definitiv über LoRa bei TTN an, denn ich habe definitiv ein Gateway was erreicht wird, die Angaben und auch das kann man gleich im Eingangspost entnehmen.

Ich habe die Daten per HTTP Integration zum Schauen was bei TTN rausgeht, statt zur openSenseMap einfach mal auf https://requestbin.com umgeleitet.

Dazu war nur nötig bei der HTTP Integration die Adresse von https://ttn.opensensemap.org/v1.1 auf https://requestbin.com zu ändern und schon konnte ich sehen, was bei TTN rausgeht und bei der openSenseMap ankommen müsste.

Ich Vermute daher bei mir einen Fehler in Richtung Decoder, da muss ich irgendwas noch falsch machen. Ich hatte nämlich erwartet, das TTN gleich die Messwerte direkt mit der Sensor ID, die ich ja beim Anlegen der sensBox erhalten habe und natürlich auch im Dekoder auf der TTN Seite angegeben habe, rausschickt.

Aber zu meinem erstaunen stellte ich fest, das TTN neben den Payload, nur einen Link mit schickt, der mir aber im Browser keine Daten anzeigt, wenn ich ihn aufrufe.

Wie nun die Kommunikation TTN zu openSenseBox läuft ist für mich der knackpunkt.

Danke für eure Mühe mir am Ostersonntag mit Rat beiseite zu stehen.

Ich wünsche noch einen schönen Ostermontag.

Viele Grüße

Holger


#5

Vielleicht sollte ich das Vorgehen mit https://requestbin.com noch etwas genauer beschreiben.

Als erstes geht man auf die Webseite https://requestbin.com und klickt dort auf den Button “Create a Request Bin”.

Darauf Hin öffnet sich ein neues Fenster und man erhält einen Link zu dem erzeugten Endpunkt.

"Your endpoint is

https://enc0znmu0pd0g.x.pipedream.net" ( Beispiel Link wie er dann aussieht, ist natürlich Ohne Funktion da immer wieder ein neuer Link generiert wird… )

Diesen Link kann man dann nehmen und auf der Webconsole von TTN bei Intregration eintragen, wo normalerweise bei uns der Link https://ttn.opensensemap.org/v1.1 hin gehört.

TTN schickt darauf hin die Daten, die normal zur openSenseMap gehen würde, auf die Seite von requestbin und man kann sich genauer anschauen, was bei TTN rausgegangen ist, so jedenfalls mein Verständnis bis zu dem Punkt.

Viele Grüße

Holger


#6

Moin Holger,

ich habe das Beispiel mit “request.bin” gerade mal nachvollzogen und komme zu einem ähnlichen Ergebnis. Die gesendeten Daten kommen also beim TTN an und werden auch weitergeschickt.

Jetzt solltest Du noch mal bei opensensemap.org überprüfen, ob die richtigen Parameter für TheThingsNetwork (TTN) eingetragen sind, damit Deine Daten auch zugeordnet werden können. Speziell die TTN Application-ID und die Device-ID.

Das Portal opensensemap.org ist heute sehr langsam. Das kann also etwas dauern.

Gruß Loobster


#7

Hallo Heiner,

vielen Herzlichen Dank für ausprobieren, das hilft mir wirklich sehr weiter.

Ich werde so schnell wie möglich schauen, was bei mir noch nicht ganz rund läuft, aber ich habe jetzt einen Anhaltspunkt mehr.

Viele Grüße

Holger


#8

Hallo Zusammen,

hab den Fehler gefunden, es lag wirklich wie vermutet am Decoder bei TTN.

Ich habe erst alles rauskopiert, dann den Standart Decoder ausprobiert und danach einfach wieder alles eingefügt und auf einmal Funktionierte es.

Es ist mir grade noch ein Rätsel, wo ich was falsch gemacht hatte, aber ich habe durch die Fehlersuche sehr interessante dinge gelernt.

Viele Grüße

Holger


#9

Dear Holger,

Akzeptiere es bitte das ich im English weiter schreibe weil ich die Deutsche sprache nicht gut kontrolliere, aber jedoch verstehe.

A little question, in your last post you stated that you found the solution on the TTN side for your decoding problem.
Can you describe a bit further while we have the same problem.
With our sensebox defined as a lora node on TTN, how can we show the data in OSeM when it arrives on TTN and from their onwards my fine dust sensor data is being forwarded to OSeM with an HTTP integration to https://ttn.opensensemap.org/v1.1 ?
I read somewhere in the docs about the JSON profile to choose in the decoding profile on OSeM. But what I do not understand is what kind of payload format I have to use on the TTN side for my registered device. Can you show me yours (decoder or encoder) on the console of TTN?

I really appreciate your help as I am stuck for this moment.
Have a great evening.


#10

Dear Sensebox members,

i want to bring up this post again as we still struggle with our fine dust sensor.
Uptil now, we succeeded in using a lora connection via TTN to display our temperature and humidity values. But the fine dust values are not in the correct format as seen here (https://opensensemap.org/explore/5dd3c45c4ec04e001ae4aef7):
fine_dut_osem
No way the actual dust level is 4096.1 µg/m³ !!!

It is mentioned in the docs that a JSON profile should be used to format the PM values in the right way. As a result, in TTN one should create a decoder to format the payload. I have used this payload format in the console of TTN, but to no avail:

function Decoder(bytes, port) {
 // bytes is of type Buffer.
 'use strict';
 var TEMPSENSOR_ID="5dce6d08306947001ae15f4e",
   HUMISENSOR_ID="5dce6d08306947001ae15f4d",
   PM10_ID="5dd27c104ec04e001a8596eb",
   PM25_ID="5dd27c104ec04e001a8596ea";
  var lat = 0,lng = 0;
 var bytesToInt = function (bytes) {
   var i = 0;
   for (var x = 0; x < bytes.length; x++) {
     i |= +(bytes[x] << (x * 8));
   }
   return i;
 };

 var latLng = function(bytes) {
  if (bytes.length !== latLng.BYTES) {
    throw new Error('Lat/Long must have exactly 8 bytes');
  }

  lat = (bytesToInt(bytes.slice(0, latLng.BYTES / 2))/1e2)/10000;
  lng = (bytesToInt(bytes.slice(latLng.BYTES / 2, latLng.BYTES))/1e2)/10000;

};
latLng.BYTES = 8;

 var uint8 = function (bytes) {
   if (bytes.length !== uint8.BYTES) {
     throw new Error('int must have exactly 1 byte');
   }
   return bytesToInt(bytes);
 };
 uint8.BYTES = 1;

 var uint16 = function (bytes) {
   if (bytes.length !== uint16.BYTES) {
     throw new Error('int must have exactly 2 bytes');
   }
   return bytesToInt(bytes);
 };
 uint16.BYTES = 2;


 var humidity = function (bytes) {
   if (bytes.length !== humidity.BYTES) {
     throw new Error('Humidity must have exactly 2 bytes');
   }

   var h = bytesToInt(bytes);
   return h / 1e2;
 };
 humidity.BYTES = 2;
var decode = function (bytes, mask, names) {

   var maskLength = mask.reduce(function (prev, cur) {
     return prev + cur.BYTES;
   }, 0);
   if (bytes.length < maskLength) {
     throw new Error('Mask length is ' + maskLength + ' whereas input is ' + bytes.length);
   }

   names = names || [];
   var offset = 0;
   return mask
     .map(function (decodeFn) {
       var current = bytes.slice(offset, offset += decodeFn.BYTES);
       return decodeFn(current);
     })
     .reduce(function (prev, cur, idx) {
       prev[names[idx] || idx] = cur;
       return prev;
     }, {});
 };
 var bytesToSenseBoxJson = function (bytes) {
   var json;

   try {
     json = decode(bytes,
       [
       uint16,
       humidity,
       uint16,
       uint16,
       latLng,
       uint16,
       uint16,
       uint16,
       uint16,
       uint16,
       uint16
       ],
       [
       TEMPSENSOR_ID,
       HUMISENSOR_ID,
       PM10_ID,
       PM25_ID,
       "location",
       "year",
       "month",
       "day",
       "hour",
       "minute",
       "second",
       ]);

    if(json['month'] === 0){
      json['month'] = 1;
    }
    if(json['day'] === 0){
      json['day'] = 1;
    }
     //temp
     json[TEMPSENSOR_ID] = [String(parseFloat(((json[TEMPSENSOR_ID] / 771) - 18).toFixed(1))),String(json["year"])+"-"+String(("0" + json["month"]).slice(-2))+"-"+String(("0" + json["day"]).slice(-2))+"T"+String(("0" + json["hour"]).slice(-2))+":"+String(("0" + json["minute"]).slice(-2))+":"+String(("0" + json["second"]).slice(-2))+"Z",[lng,lat]];

     //hum
     json[HUMISENSOR_ID] = [String(parseFloat(json[HUMISENSOR_ID].toFixed(1))),String(json["year"])+"-"+String(("0" + json["month"]).slice(-2))+"-"+String(("0" + json["day"]).slice(-2))+"T"+String(("0" + json["hour"]).slice(-2))+":"+String(("0" + json["minute"]).slice(-2))+":"+String(("0" + json["second"]).slice(-2))+"Z",[lng,lat]];

     // pm10
     json[PM10_ID] = [String(parseFloat((json[PM10_ID] / 10).toFixed(1))),String(json["year"])+"-"+String(("0" + json["month"]).slice(-2))+"-"+String(("0" + json["day"]).slice(-2))+"T"+String(("0" + json["hour"]).slice(-2))+":"+String(("0" + json["minute"]).slice(-2))+":"+String(("0" + json["second"]).slice(-2))+"Z",[lng,lat]];
     // pm25
     json[PM25_ID] = [String(parseFloat((json[PM25_ID] / 10).toFixed(1))),String(json["year"])+"-"+String(("0" + json["month"]).slice(-2))+"-"+String(("0" + json["day"]).slice(-2))+"T"+String(("0" + json["hour"]).slice(-2))+":"+String(("0" + json["minute"]).slice(-2))+":"+String(("0" + json["second"]).slice(-2))+"Z",[lng,lat]];
     delete json.year;
     delete json.month;
     delete json.day;
     delete json.hour;
     delete json.minute;
     delete json.second;

   } catch (e) {
     json = { payload: bytes };
   }

   return json;
 };

 return bytesToSenseBoxJson(bytes);
}

Any one has a clue on where the problem is?


#11

Moin lieber Mitglieder von die SenseBox community,

Ich wollte gern wieder diesem Umsatz Problem ansprechen. In English leider :sweat_smile:

We are using the SDS fine dust and the HDC temp/hum sensor on the MCU with a lora connectivity via TheThingsNetwork. Their data is well received on the TTN console and an HTTP integration is made to OSeM, no problem with this one.

But the values of the dust sensor are not the correct ones when visualised in OSeM. I have tried a JSON profile with a Payload Function in TTN as described in my previous post.

Today I tried a lora-serialisation profile with these decode options filled in:

[{“decoder”:“latLng”},{“decoder”:“temperature”,“sensor_title”:“Temperatur”},{“decoder”:“humidity”,“sensor_title”:“rel. Luftfeuchte”},{“decoder”:“temperature”,“sensor_title”:“PM2.5”},{“decoder”:“temperature”,“sensor_title”:“PM10”}]

I found this info on https://github.com/sensebox/mobile-sensebox/tree/master/lora-gps
But no result whatsoever; no measurements are coming in on https://opensensemap.org/explore/5ddf8b232b3516001a522ec3 after resetting the decoding profile in the OSeM page.

This is a real difficulty and I wonder if some one could help me out in displaying the right PM values when using lora connectivity.

Danke vielmal.


#12

Hi @wdebbaut,

sorry for this late response and that the values are not correctly displayed on openSenseMap. I have some pretty busy work days. I will try to find some spare time this weekend and have a look at it.

Best, Matthias


#13

Dear OSeM members,

I would like to raise up again this particular problem for which after several weeks of research I still do not find a solution.

I follow exactly the instructions in the project of Paul about the Mobilerfeinstaubloggerlora.

Can Paul present himself and give a clue about how to tackle this issue?
When I choose the JSON profile in OSem, then there are no measurements coming through.


On the other hand, when using the sensebox_home profile the correct temp and humidity values are shown, but the PM values are of no sense
.pm_values_sensebox_home

In TTN itself we constructed a payload format as described though. and illustrated in my previous post.

I would appreciate a lot if some one could point me to the right direction.


#14

Best OSeM members,

as I did not found a solution yet for my TTN-OSeM integration problem, I decided to try another way of resolving my issue that my PM values are not the correct ones as displayed here.

Based on the GitHub information found here, I have created a new sensebox with the MQTT client enabled for TTN.
But as you can see here, the measurements are not arriving from the TTN platform, let alone in the right format.

Has any one else tried this way of retrieving his data with lora on TTN?

Best wishes everyone for Christmas and 2020.


#15

I have had help of an experienced javascript decoder (avanbentem) active on the TTN forum to create a decoder for the PM values. He also made a contribution to https://github.com/sensebox/node-sketch-templater/issues/26 and argues some code errors, especially in the LMIC library.

With the following code snippet, I would hope the decoding in the TTN payload format of the fine dust values with a JSON profile in OSeM, results in the right values of PM10 and PM25.

/**
 * Convert the array of bytes to an unsigned integer, LSB. 
 *
 * BEWARE: This is only safe up to 0x1FFFFFFFFFFFFF, so: 6 bytes.
 */
function uint(bytes) {

  return bytes.reduceRight(function(acc, b) {
    // We expect an unsigned value, so to support more than 3 bytes
    // don't use any bitwise operators, which would always yield a
    // signed 32 bits integer instead.
    return acc * 0x100 + b;
  }, 0);
}

function Decoder(bytes) {

  var i = 0;     
  var decoded = {};
  // The index in the bytes array that needs to be handled next;
  // use along with "i++" which returns the current value, and
  // then increments it for the next usage


  // Temperature with 0.0013 precision, -18.000 through 67.000, LSB
  var temperature = uint(bytes.slice(i, i+=2)) / 771 - 18;
  // Unary plus operator to cast string result of toFixed to number
  decoded.temperature = +temperature.toFixed(3);
  // Relative humidity with 0.01 precision, 0.00 through 100.00, LSB
  decoded.humidity = uint(bytes.slice(i, i+=2)) / 100;
  // Pressure with 0.012 precision, 300.00 through 1100.00, LSB
  var pressure = uint(bytes.slice(i, i+=2)) / 81.9187 + 300;
  decoded.pressure = +pressure.toFixed(2);
  decoded.lux = uint(bytes.slice(i, i+=3));
  decoded.uv = uint(bytes.slice(i, i+=3));
  decoded.pm10 = uint(bytes.slice(i, i+=2)) / 10;
  decoded.pm25 = uint(bytes.slice(i, i+=2)) / 10;
  
  return decoded;
}

But to no avail, and without any further help from this forum, our story unfortunately stops here I guess.:sleepy:


#16

Hi,

I guess most of the people are on vacations. If no one is replying go and write a mail to support@sensebox.de

But if you have a functional decoder (you posted) you can use it on TTN side and on openSenseMap side use the json profile.


#17

Danke vielmal für Ihre Reaktion Matthias,
und wie gefragt, habe ich diesen Morgen mein support mail an Sensebox verschickt.

Copy/paste was ich geschrieben habe:

I would like to come back to my mail from several weeks ago.

We have several senseboxes at the highschool UCLL and uni from Leuven/Belgium which we would like to have them connected with lora on the TheThingsNetwork platform. Just like https://www.thethingsnetwork.org/u/stbkoeln at the https://stadtbibliothekkoeln.blog/2018/11/28/aufs-dach-gestiegen-eine-sensebox/ , we would like to have the same result, i.e. https://opensensemap.org/explore/5be04ae455d0ad001adebb3f

In your documentation on https://sensebox.github.io/books-v2/home/de/komponenten/bees/lora.html#decoder it is written that one should use a JSON profile in the OSeM configuration and creating a payload format on the console of TTN.

I have used the example in the book but we still do not receive the correct PM values, even more, there are no measurements once we switched from the sensbox:home profile to the JSON profile. So I guess the decoder function is not the right, though with postman I can see that a JSON object is sent from the TTN intergration to a test environment (requestbin).

In short, could you provide me with a decoder function on TTN, just as the folks did at the Stadtbibliothek Köln? Matthias Pfeil von uni Münster suggested me to write you this mail as no reaction is coming from the forum.

We wish you a fine end of the year, and hope to receive a reaction from your part. 

Ich wünsche Ihnen noch einen guten EndeJahrs Wechsel.


#18

Moin Allen,

I would like to express a great many thanks to the SenseBox developper Felix from the uni of Münster for having found a solution to my problem after asking for some support.

I have been searching for weeks in order to visualize our PM values when the sensebox is registered on TTN. Indeed, I had to choose a JSON profile in OSeM, but our previous payload function in TTN was wrong.
Fortunately Felix figured it out during the weekend, and now I am able to view my sensebox on : https://opensensemap.org/explore/5e10d6b0b0c088001bf22393 with the correct PM values.
His solution can be found here : https://gist.github.com/felixerdy/f8434ee0c081f1b6e711439cb6a001b7
We even succeeded in transferring those values to the luftdaten platform with a node-red server.

Also he adjusted a bit the Arduino code for the MCU to have more correct values https://gist.github.com/felixerdy/75644b9b047810422b950a972a3b832b

Again, congrats for the developper.


#19

I created a small “tutorial” to explain decoding a little bit. I hope it does not create too much of confusion. Maybe it helps somebody to understand the way of decoding LoRaWAN measurements.

https://sensebox.de/projects/de/2020-01-21-ttn-decoding


#20

Herzlichten Dank Felix,

it is already for some weeks that https://opensensemap.org/explore/5e17250e7e9052001a95502b is running indoor with our lorawan gateway in range that forwards the data to TTN, and from there onwards with an HTTP integration to OSeM.
Schöne Leistung and unabhängig von Wifi! Lorawan also gaining in popularity …

https://blockly.sensebox.de/ is a nice feature, especially for our students to program. Unfortunately, there is no Web block for lorawan. Are there any intentions to develop these, as you know of?
That would finish it completely! An example below from your collegues in Birkenfeld (Octopus) illustrates rapid prototyping should be possible.


Grüsse.


#21

Yes we are currently working on that. There is already a preview https://deploy-preview-140–sensebox-blockly.netlify.com/ardublockly/?board=sensebox-mcu where you can create a simple LoRaWAN sketch. To my surprise, online compiling already works. It’s not tested that much and there may be issues with other components like the display or SD card. Additionally, you will need to convert the measurements by yourself (in order to keep decimals).
You can check it out and feel free to give some feedback.


#22

Great job done on your testplatform Felix,

I have tried your block Web/lora by filling out the OTAA parameters for my specific MCU board (lsb for both DevEUI and AppEUI and msb for AppKey).


A JOIN request/accept happens on TTN (lorawan1.1), meaning that my TTN gateways are in range. But an uplink only happens after a while, which is probably due to the LMIC library on my MCU.
So OTAA seems to work, but not optimally.

Could you therefore also make a block for ABP (Activation By Personalization)?
In your code you need to define your session keys and dev address statically and setting them in the LMIC setup.
Example:
static u1_t NWKSKEY[16] = { 0x36, 0x10, 0x85, 0xB1, 0x9E, 0xC7, 0x9C, 0xAD, 0x88, 0x15, 0xB2, 0xAE, 0x38, 0x58, 0x44, 0xAB };
static u1_t APPSKEY[16] = { 0xA5, 0x69, 0x06, 0x8D, 0xB5, 0xB8, 0x94, 0x04, 0x15, 0x31, 0xF1, 0xAB, 0x47, 0x6E, 0xA8, 0x27 };
static u4_t DEVADDR = 0x260118CD ;

and in the void setup of the Arduino code:
LMIC_setSession (0x1, DEVADDR, NWKSKEY, APPSKEY);

With this ABP activation, things will go smoothier and easier to debug as well. Later on, we can then examine what the reason is for my long OTAA activation of my specific node.

That would complete the whole block for TTN, as they did with Ardublock from octopus as well. See below.