## How to Work with Data
#### How to create a solution for hackathons without writing a single line of code
[Miroslav "mirek" Biňas](https://bletvaska.github.io)
/ [**Act in Space 2022**](https://actinspace.org/)
[](https://bletvaska.github.io)
## Traditional Hackathon Use Case
download/steal > process > show/upload
notes:
* ukradnut data, spracovat ich, poskytnut pouzivatelom
* stiahneme informácie o letoch nad slovenskom pomocou komunikačného protokolu HTTP
* spracujeme ich a zobrazíme ich na vlastnej mape
* zautomatizujeme proces sťahovania a aktualizácie údajov na mape
* získané údaje prepošleme ďalej pomocou komunikačného protokolu MQTT
* údaje začneme lokálne ukladať
* poskytneme ich ďalej pomocou REST API
* urobíme jednoduchú webovú stránku
* budem robit nazivo a samozrejme budem podvadzat
## It's all about **data**!
notes:
* to, co je dolezite je, ze je to vsetko a vzdy o datach
[](https://nodered.org)
notes:
* pocas celej prezentacie budem pracovat vylucne s nastrojom nodered
* s cielom - nenapisem riadok kodu
* (sablony sa nerataju)
## 1. Collecting Data
notes:
* running
```bash
$ docker container run --publish 1880:1880 --name nodered --volume $(pwd)/data:/data --rm nodered/node-red
```
[](https://openskynetwork.github.io/opensky-api/index.html)
https://opensky-network.org/api/states/all?lamin=47.4912&lomin=16.4517&lamax=49.8981&lomax=25.9378
## The Message
```json
{
"_msgid": "89cbb063.7f7a7",
"payload": "Hello world!",
"topic": "Greetings!"
}
```
notes:
* len pre uplnost - medzi jednotlivymi uzlami sa posielaju spravy, ktore sa skladaju z uvedenych troch casti/klucov:
* `_msgid` - id spravy, ktore sluzi na nutornu identifikaciu a pre beznu pracu nie je potrebne
* `topic` - hlavicka spravy
* `payload` - obsah spravy
* rozlicne uzly si mozu pridavat dalsie kluce, ale Node-RED ocakava, ze kazda jedna sprava bude mat minimalne tieto kluce
## 2. Presenting the Data
[](https://flows.nodered.org/node/node-red-contrib-web-worldmap)
[node-red-contrib-web-worldmap](https://flows.nodered.org/node/node-red-contrib-web-worldmap)
notes:
* nainstalujeme plugin worldmap
* nastavit zemepisnu vysku a sirku, zoom level na hodnotu 8
* nainstalujeme plugin pre handlebars
## Template for Data Processing
```json
[
{{#each states}}
{
"name": "{{0}}",
"icao24": "{{0}}",
"callsign": "{{1}}",
"origin_country": "{{2}}",
"lon": {{5}},
"lat": {{6}},
"icon": "plane",
"speed": {{9}},
"last_contact": {{4}},
"heading": {{10}},
"ttl": 60,
"layer": "flights"
}
{{#unless @last}},{{/unless}}
{{/each}}
]
```
## 3. Updating Data
notes:
* nechame udaje stahovat napr. kazdych 10s
* cron job / systemd timers
* udaje budeme aktualizovat kazdych 5s, ale netreba zabudnut na cakanie 6s na zaciatku kvoli inicializacii mapy
## 4. Publishing Data with MQTT
notes:
* MQTT je jeden zo zakladnych komunikacnych protokolov v IoT rieseniach
* pouzijeme `broker.hivemq.com:1883` a topic `hackathon/ais/flights`
## 5. REST API
* HTTP protocol
* endpoint: `/api/planes/:flight`
notes:
## Local Database
* REST API must return data immediately after receiving a request
* db as global variable
* simple dictionary (`key-value` store)
```json
{
{{#each this}}
"{{ icao24 }}": {
"icao24": "{{icao24}}",
"callsign": "{{callsign}}",
"origin_country": "{{origin_country}}",
"lon": {{lon}},
"lat": {{lat}},
"speed": {{speed}},
"last_contact": {{last_contact}},
"heading": {{heading}}
}{{#unless @last}},{{/unless}}
{{/each}}
}
```
notes:
* vysledok vlozime pomocou uzla `CHANGE` do premennej `global.flights = msg.payload`
notes:
* `CHANGE` uzol urobi `msg.payload = flights[msg.req.params.flightid]`
## Testing the REST API
http://localhost:1880/api/flights/:flightid
## 6. Web Page
(similar to REST API)
notes:
```html
<h1>Flight nr {{payload.callsign}}</h1>
<ul>
<li>
<b>ICAO24</b>: {{ payload.icao24 }}
</li>
<li>
<b>Callsign</b>: {{ payload.callsign}}
</li>
<li>
<b>Country of Origin</b>: {{ payload.origin_country }}
</li>
<li>
<b>Longitude</b>: {{ payload.lon }}
</li>
<li>
<b>Latitude</b>: {{ payload.lat }}
</li>
<li>
<b>Speed</b>: {{ payload.speed }}
</li>
<li>
<b>Heading</b>: {{ payload.heading }}
</li>
<li>
<b>Last Contact</b>: {{ payload.last_contact }}
</li>
</ul>
```
notes:
* vysledok vlozime pomocou uzla `CHANGE` do premennej `global.flights = msg.payload`
notes:
* `SWITCH`:
* `property = global.flights`
* `has key = msg.req.params.flight`
* `otherwise`
* `CHANGE`
* `msg.payload = global.flights[msg.req.params.flight]`
### What else can Node-RED do?
* websockets
* connected hw (serial)
* dashboards
* integration with DBS
* integration with external services
* own functions as nodes (JavaScript, Python)
* machine learning
* ...
## Resources
* [Node-RED Homepage](https://nodered.org)
* [FRED: Front End For Node-RED](https://fred.sensetecnic.com)
* [The Result Flow](resources/flows-bikes.json)
* [Node-RED YouTube Channel](https://www.youtube.com/channel/UCQaB8NXBEPod7Ab8PPCLLAA)
(**http://bit.ly/3UTieDB**)