separated data collectors, generalized template, extended README files

Author: markfodor

README.md

@@ -12,27 +12,19 @@ This repo can help you set up an e-ink board and display your Trello and Google
 
 ![InkCheck](https://user-images.githubusercontent.com/3463702/250285813-c93ab4b4-c946-4134-a144-b92ad8b61ca0.jpg)
 
-### Google Keep mapping
-![Keep](https://user-images.githubusercontent.com/3463702/250350631-abae8a92-2ef2-48c3-8082-0c4bc58d943a.jpg)
-It also works with lists as the following Trello example shows. It is rendered the same way.
-
-### Trello mapping
-![Trello](https://user-images.githubusercontent.com/3463702/250350632-9970813a-3a66-47c2-b825-de2e0113df19.jpg)
-
-## General description
-This is a forked repo from [MagInkDash](https://github.com/markfodor/MagInkDash). That project was used as a base for everything here.
-So if you have the opportunity, buy that guy a coffee (link on the MagInkDash README). :v:
-
-InkCheck however differs in many aspects.
-It is able to display your Google Keep and/or Trello data on an [Inkplate E-Ink Display](https://soldered.com/product/soldered-inkplate-10-9-7-e-paper-board-with-enclosure-copy/).
-
 You can use it to display:
 - your daily schedule
 - your long-term goal
 - a TODO list
 - any text/quote you want to keep in mind
 
-## Hardware Required
+## Features
+- Periodical image refresh (1 hour with the default setup)
+- Landscape and portrait mode
+- Renders 2 data columns
+- Data providers: Trello, Google Keep
+
+## Hardware
 - [Inkplate 10 E-Ink Display](https://soldered.com/product/soldered-inkplate-10-9-7-e-paper-board-with-enclosure-copy/) - Used as a client to display the generated image. If you go with this it will be less hardware tinkering.
 - A server, which is powerful enough to run the image generation. It could be a [Raspberry Pi](https://www.raspberrypi.org/)
 
@@ -74,28 +66,28 @@ cd InkCheck
 pip install -r requirements.txt
 ```
 
-6. Fill the variables in the config.json.
+6. Fill the variables in the global.json. Most of the variables are pre-filled, _destinationFolder_ should be the path where your Apache server is running. Sidenote: You can switch between portait and landscape mode by swtiching the _imageWidth_ and _imageHeight_ values.
+
+7. Fill the config.json files for the collectors. More info at the [Data Collectors](#data-collectors)
 
-7. Do a test run and check the logs. If everything is ok you should not see any error logs.
+8. Do a test run and check the logs. If everything is ok you should not see any error logs.
 ```bash
 python3 main.py
 ```
-This might takes a bit longer (depends on your hardware - 2-3 mins on a Raspberry Pi Zero). When it is done, you should be able to find the rendered html file (renderer/inkcheck.html) and the screenshot (output/inkcheck.png). The image should be available on your network if you check in a browser: YOUR_SERVER_IP/inkcheck.png
+This might takes a bit longer (depends on your hardware, 2-3 mins on a Raspberry Pi Zero). When it is done, you should be able to find the rendered html file (renderer/inkcheck.html) and the screenshot (output/inkcheck.png). The image should be available on your network if you check in a browser: YOUR_SERVER_IP/inkcheck.png
 
-7. Copy all the files (other than the "inkplate" folder) over to your RPi using your preferred means. 
-
-8. Run the following command in the RPi Terminal to open crontab.
+9. Run the following command in the RPi Terminal to open crontab.
 ```bash
 crontab -e
 ```
 
-9. Specifically, add the following command to crontab so that the InkCheck Python script runs on the hour, every hour.
+10. Specifically, add the following command to crontab so that the InkCheck Python script runs on the hour, every hour.
 ```bash
 0 * * * * cd /location/to/your/InkCheck && python3 main.py
 ```
 If you want to set an other interval check out the [crontab.guru](https://crontab.guru/) site.
 
-10. As for the Inkplate, I'm not going to devote too much space here since there are [official resources that describe how to set it up](https://inkplate.readthedocs.io/en/latest/get-started.html). It may take some trial and error for those new to microcontroller programming but it's all worth it! Only the Arduino portion of the guide is relevant, and you'll need to be able to run *.ino scripts via Arduino IDE before proceeding. From there, compile and upload the "inkplate.ino" file from the "inkplate" folder in the Arduino IDE when connected to the Inkplate. And do not forget to fill the ssid, password and imgurl fields at the top of the ino file. Oh, and the BOTtoken if you want to get notifications about the battery.
+11. As for the Inkplate, I'm not going to devote too much space here since there are [official resources that describe how to set it up](https://inkplate.readthedocs.io/en/latest/get-started.html). It may take some trial and error for those new to microcontroller programming but it's all worth it! Only the Arduino portion of the guide is relevant, and you'll need to be able to run *.ino scripts via Arduino IDE before proceeding. From there, compile and upload the "inkplate.ino" file from the "inkplate" folder in the Arduino IDE when connected to the Inkplate. And do not forget to fill the ssid, password and imgurl fields at the top of the ino file. Oh, and the BOTtoken if you want to get notifications about the battery.
 Common problems:
 - Inkplate is not connected
 - Inkplate is not ON - a light blue led should shine through the 3D-printed case next to the ON button.
@@ -105,17 +97,25 @@ Common problems:
 
 12. That's all! Your InkCheck should now be refreshed every hour!
 
+## Data Collectors
+Different data sources (e.g: Trello or Google Keep) are handled differently in separated data collectors. You can read more about the setup here:
+- [Google Keep](/collectors/googlekeep/README.md)
+- [Trello](/collectors/trello/README.md)
+
 ## Acknowledgements
+- [MagInkDash](https://github.com/markfodor/MagInkDash) - Source of the fork. If you have the opportunity, buy that guy a coffee (link on the MagInkDash README). :v:
 - [Lexend Font](https://fonts.google.com/specimen/Lexend)
 - [Jinja](https://jinja.palletsprojects.com/)
 - [Bootstrap](https://getbootstrap.com/)
 
 ## Contribution
 Feel free to fork/modify the code to your needs. If you want something to be in this repo, then just open an issue or a pull request.
 
-Things I want to add:
-- Generalize the code -> it would be easier to add a new source
-- Generalize the template to handle single and double coulmns
-- Portrait and landscape mode
+Things I want to add/modify:
+- Validate input values in the global.json (e.g: timezone)
+- Eliminate Apache server and make it pure Python.
+- Improve GoogleKeepCollector to use token if possible
+- Adjust template for checked list items (sort list and display in a different way)
+- Generalize the template to handle single and double columns
 - Docker support -> easier to setup the server
-- Optional: server endpoint to kickstart the image generation so you do not need to wait for the next refresh
+- Optional: server endpoint to kickstart the image generation so you do not need to wait for the next refresh, and easier to test

collectors/abstractBaseCollector.py

@@ -0,0 +1,8 @@
+from abc import ABC, abstractmethod
+from model.columnData import ColumnData
+
+
+class AbstractBaseCollector(ABC):
+    @abstractmethod
+    def get_data(self) -> ColumnData:
+        pass

collectors/googlekeep/README.md

@@ -0,0 +1,26 @@
+# Google Keep Collector 
+
+## Mapping
+![Keep Text note](https://user-images.githubusercontent.com/3463702/250350631-abae8a92-2ef2-48c3-8082-0c4bc58d943a.jpg)
+![Keep List note](https://user-images.githubusercontent.com/3463702/253660376-61843d5a-88c1-4d07-8404-19f8a102fc95.jpg)
+
+
+## Setup
+To generate an app password:
+
+1. Go to your [Google Account](https://myaccount.google.com/)
+2. Select Security.
+3. Under "Signing in to Google," select 2-Step Verification.
+4. At the bottom of the page, select App passwords.
+5. Enter a name that helps you remember where you’ll use the app password.
+6. Select Generate.
+7. Now you can copy the generated password to the config.json
+```
+{
+    "username": "YOUR_MAIL_ADDRESS",
+    "password": "YOUR_PASSWORD",
+    "nodeName": "Daily schedule",
+    "onlyUncheckedItems": true
+}
+```
+The "onlyUncheckedItems" field is only interesting when you work with lists. If the value is _true_, it will only display the unchecked items.

collectors/googlekeep/config.json

@@ -1,5 +1,6 @@
 {
     "username": "YOUR_MAIL_ADDRESS",
     "password": "YOUR_PASSWORD",
-    "nodeName": "YOUR_NODE_NAME"
+    "nodeName": "YOUR_NODE_NAME",
+    "onlyUncheckedItems": true
 }

collectors/googlekeep/googleKeepCollector.py

@@ -3,17 +3,19 @@
 from logger.logger import logger
 from model.columnData import ColumnData
 from util.configHelper import read_config
+from collectors.abstractBaseCollector import AbstractBaseCollector
 
 
-class GoogleKeepHelper:
+class GoogleKeepCollector(AbstractBaseCollector):
 
     def __init__(self):
         config = read_config(str(pathlib.Path(__file__).parent.absolute()))
 
         if config:
             self.username = config['username']
             self.password = config['password']
-            self.nodeName = config['nodeName']
+            self.node_name = config['nodeName']
+            self.only_unchecked_items = config['onlyUncheckedItems']
 
             logger.info('Google Keep config is set.')
         else:
@@ -27,20 +29,20 @@ def __init__(self):
         else:
             logger.error('Google Keep login failed.')
 
-    def get_items_on_node_by_node_id(self, node_id, only_unchecked):
+    def _get_items_on_node_by_node_id(self, node_id):
         node = self.client.get(node_id)
 
         if isinstance(node, gkeepapi.node.List):
-            data = ColumnData(title=self.nodeName, is_list=True)
+            data: ColumnData = ColumnData(title=self.node_name, is_list=True)
             for item in node.items:
-                if not only_unchecked or not item.checked:
+                if not self.only_unchecked_items or not item.checked:
                     data.items.append(item.text)
 
             logger.info('Data collection from Google Keep is done.')
             return data
         # assuming it is a simple text node
         else:
-            data = ColumnData(title=self.nodeName, is_list=False)
+            data: ColumnData = ColumnData(title=self.node_name, is_list=False)
             data.text = node.text
             return data
 
@@ -53,9 +55,10 @@ def _search_node_id(self, name):
                 return node.id
         return None
 
-    def search_node_id_by_name(self, only_unchecked):
-        nodeId = self._search_node_id(self.nodeName)
+    def get_data(self) -> ColumnData:
+        nodeId = self._search_node_id(self.node_name)
         if not nodeId:
-            logger.error(f"node not found with name: {self.nodeName}")
+            logger.error(f"node not found with name: {self.node_name}")
+            return None
         else:
-            return self.get_items_on_node_by_node_id(nodeId, only_unchecked)
+            return self._get_items_on_node_by_node_id(nodeId)

collectors/trello/README.md

@@ -0,0 +1,24 @@
+# Trello Collector 
+
+## Mapping
+![Trello](https://user-images.githubusercontent.com/3463702/250350632-9970813a-3a66-47c2-b825-de2e0113df19.jpg)
+
+```
+{
+    "key": "YOUR_API_KEY",
+    "token": "YOUR_TOKEN",
+    "board": "Test board",
+    "list": "Long-term Goals"
+}
+```
+
+## Setup
+Everything you need to do is written on the [Atlassian guide](https://developer.atlassian.com/cloud/trello/guides/rest-api/api-introduction/).
+
+When you have your newly created app you will be able to see the corresponding **API key** -> click on the app.
+![Trello app](https://user-images.githubusercontent.com/3463702/253630254-3f9e753a-4f27-49ce-baef-cd55cba5d0e2.png)
+
+After that you just need to authorize it and get the **token**.
+![Trello token](https://user-images.githubusercontent.com/3463702/253630914-5f3f9e95-e28f-4d7f-9625-3643b2f75ec5.png)
+
+Once you have a board created on your workspace and have a list on it, you will be able to fill the config.json.

collectors/trello/trelloCollector.py

@@ -3,9 +3,10 @@
 from logger.logger import logger
 from model.columnData import ColumnData
 from util.configHelper import read_config
+from collectors.abstractBaseCollector import AbstractBaseCollector
 
 
-class TrelloCollector:
+class TrelloCollector(AbstractBaseCollector):
 
     def __init__(self):
         config = read_config(str(pathlib.Path(__file__).parent.absolute()))
@@ -23,7 +24,7 @@ def __init__(self):
 
         self.client = TrelloClient(self.key, None, self.token, None)
 
-    def get_data(self):
+    def get_data(self) -> ColumnData:
         boards = self.client.search(self.board, True, ['boards'])
 
         if not boards:
@@ -35,7 +36,7 @@ def get_data(self):
 
         # "There can be only one" in the result set
         selected_board = boards[0]
-        data = ColumnData(title=self.board, is_list=True)
+        data: ColumnData = ColumnData(title=self.board, is_list=True)
 
         # these are a columns on the board
         lists = selected_board.all_lists()

global.json

@@ -3,6 +3,6 @@
     "timestampFormat": "%Y.%m.%d %H:%M:%S",
     "imageWidth": 1200,
     "imageHeight": 825,
-    "rotateAngle": 0,
-    "destinationFolder": ""
+    "collectors": ["GoogleKeepCollector", "TrelloCollector"],
+    "destinationFolder": "YOUR_DESTINATION_FOLDER"
 }

main.py

@@ -4,33 +4,47 @@
 CSS stylesheet.
 """
 
+import sys
 import json
 from datetime import datetime as dt
 from pytz import timezone
 from logger.logger import logger
 from renderer.renderer import Renderer
-from collectors.googlekeep.googleKeepCollector import GoogleKeepHelper
+from model.columnData import ColumnData
+from collectors.abstractBaseCollector import AbstractBaseCollector
+from util.configHelper import validate_collectors
+
+# Import the collector here so it can be in globals
+from collectors.googlekeep.googleKeepCollector import GoogleKeepCollector
 from collectors.trello.trelloCollector import TrelloCollector
 
 
 if __name__ == '__main__':
-    configFile = open('global.json')
-    config = json.load(configFile)
+    config_file = open('global.json')
+    config = json.load(config_file)
+
+    time_zone: str = timezone(config['timezone'])
+    timestamp_format: str = config['timestampFormat']
+    image_width: int = config['imageWidth']
+    image_height: int = config['imageHeight']
+    collectors: list[str] = config['collectors']
+    destination_folder: str = config['destinationFolder']
 
-    timeZone = timezone(config['timezone'])
-    timestampFormat = config['timestampFormat']
-    imageWidth = config['imageWidth']
-    imageHeight = config['imageHeight']
-    rotateAngle = config['rotateAngle']
-    destinationFolder = config['destinationFolder']
+    valid = validate_collectors(collectors)
+    if not valid:
+        logger.error('Invalid config. Exiting...')
+        sys.exit(1)
 
     logger.info('Global config is set.')
 
-    timestamp = dt.now(timeZone).strftime(timestampFormat)
-    google_keep_data = GoogleKeepHelper().search_node_id_by_name(True)
-    trello_data = TrelloCollector().get_data()
+    timestamp = dt.now(time_zone).strftime(timestamp_format)
+    data_list: list[ColumnData] = []
+    for collector_name in collectors:
+        clazz = globals()[collector_name]
+        collector: AbstractBaseCollector = clazz()
+        data_list.append(collector.get_data())
 
-    renderer = Renderer(imageWidth, imageHeight, rotateAngle)
-    renderer.render(timestamp, google_keep_data, trello_data, destinationFolder)
+    renderer = Renderer(image_width, image_height)
+    renderer.render(timestamp, data_list, destination_folder)
 
     logger.info("Inkcheck image is updated.")