Blog
pedro-psb opened this issue · comments
Blog ideas
Here I'll save ideas about the things I've learned and the ideas I had during the development process.
Ideas
[ code quality ] Database Acess Chaos
Write about how database access became a huge mess, and provide some MWE to refator it.
[ architecture ] Multi-Tenancy for each canteen. Partition vs Pooled
Write something about how I tried to implement it.
[ tests ] How tests are useful for refactoring and evolving the code
I needed to extend a function that returned a client's list from the database to return more data (get_client_list_by_canteen_id). In theory, it shouldn't break the other usage of the function, as it would still provide the old data, just return more fields. But then I made the changes and ran all the tests (because I didn't have a unit test for this function). There was an error in a frontend test that checked if the pages were loaded successfully, and while it was not so hard to find where the function was used, it was a bit counterintuitive, and too much overhead, since this test faked a client interaction with request/response, which was really not needed to make this change to the function safely. Write unit tests. End of story.
[ database ] Calculated columns: persistent or dynamic?
Research if updating total with update is safe. An alternative is to calculate the total based on transactions, which seems more robust on the coast of system overload.
Refactor: use sqlite3 computed columns to calculate product_item total. Test if it can be done with multiple column values (by the docs, it seems it only works for same-row values).
[ database ] Trade-offs of SQL vs NoSQL on the transaction data
The transaction schema was very challenging to conceive and to implement, as there was some variations on the types of transactions to be recorded (Regular Purchases, User Account Purchases, etc). The queries became very complex, with lots of inner and left joins. At that point I was considering implementing a document oriented NoSQL solution, as it would just be a process of throwing whatever data I had on each transaction in the same bucket (and later filtering whatever I needed in a report). But I'm not sure if this is a good idea, as I'm not aware of the tradeoffs involved. That post could be a way of exploring such trade-offs.
[ database ] Trade-offs of using ORM over raw queries
I've been migrating to the use of dataclasses (pydantic) to handle data and started putting all validation inside them, including validation that needs db queries. So to keep these classes concise, these validation queries run on every instantation, even when they are not needed. Thinking about it, I may just be able to parametrize which db validations are needed on instantiation, or subclass for specific uses. Eg: when updating a user, I don't need the same validation as when inserting an user. Or when I add an owner, the validation about the canteen is a special case (oh yeah, I should make a class for Canteen...). Well, maybe I shouldn't talk about ORM here. But still valid topic.
[ tests ] Organizing test for webapps: database, API processing, frontend navigation, etc
While developing test for the PyPos App, I've began to realize some types of testing that should be made for different sections of the application, and that ideally I should design the system in such a way that I can test them independently. For example, I made a lot of test on database, checking if and endpoint processed the form and added to the database. This introduces some complexity to the test, as it should account for validation and database interface. I'm not yet sure how to refactor it all to be more well designed, but I know it needs it.
Eg: use and test of User models validators without the overhead of request/response interactions.
#13 [ databases ] Model for multiple canteens system: alternatives study and sqlite limitaitons
Each canteen should have it's own data independent from eachother, in other words, all tables for users, roles, transactions should strictly related to just one canteen. After modeling all app data, now I face the problem wrapping it inside the canteen structure. In other database systems I may create one 'database' for each canteen. In sqlite, it would mean creating a new sqlite file for each canteen. Another solution is to create a relation of every entity with a single canteen, which seems very messy and redundant, but still doable.
#9 [ databases ] Retrieve nested structures with group_concat, instead of use multiple queries
I wanted to get all transaction and all products inside a transaction. I could, for each transaction, query for all it's products. Other solution was to query a group by with concatenated products (comma-separated) and parse it later.
Alternatives: It is worth researching (or benchmark testing) the real optimization value of this. Does a single group by better than multiple queries
#7 [ frontend ] Add stringyfied items to <form> to post nested structures
I wanted to send a nested json structure with transactions and it's products.
Alternatives: I could have send just a json attach. That might be cleaner, but if I've used a JSON post via JS, could I still use flask url_for to get the endpoints?