Trif

Project deployed to ztrif.herokuapp.com.

Overview

Trif is a web application which classifies and offers for sale high-definition prints of abstract fractal images created in Java using a plethora of mathematical functions iterated over the complex plane.

The project has been developed using Django, with the goal of fulfilling the requirements of the final, Full-Stack Milestone Project of the Code Institute Full-Stack Software Developer Course.

All images displayed are stored in an AWS S3 bucket, as are static files like CSS, icons and user-uploded profile pictures.

The project is hosted at Heroku. A Postgresql database, also hosted at Heroku, is used to store image data and user data.

The E-Commerce side is handled by Snipcart.

Automated testing uses django.test. CI/CT is implemented through Travis CI.

At the outset, 348 images will be available, printed with professional fade-resistant pigments on high-quality photo paper.

I generate these images by pouring equal measures of mathematics and computer programming into a pot, mixing well and cooking gently for a long time! ;)

They are the result of several years of developing a Java program which uses polynomial and trigonometric functions iterated over the complex plane to produce often highly detailed, high-resolution images (normally 14032 x 9920 pixels), which can consequently be printed even on large format media without having to be scaled up and thereby losing quality and in extreme cases becoming pixellated.

All prints, and the images on this site, are made from these 14032x9920 originals. The images in the grid view are scaled down to 438 x 310 pixels, and the individual views show a higher-resolution version, 877 x 620 pixels.

tl;dr

Each image is generated by a combination of two complex functions iterated over the complex plane. In most cases the functions are composed, while sometimes they're alternated and now and then combined by taking the arithmetic or geometric mean of the two at each iteration.

Most images are produced using somewhere between 32 and 1024 maximum iterations per pixel. Most functions have a strong trigonometric element, using the real sine and cosine functions applied separately to the real and imaginary parts, in various combinations, as the complex trig functions tend to produce fragmented, jagged images.

Use

Site follows conventions by having a consistent style with every page sporting a navigation bar at the top, with links to the main pages, and a footer with social media icons at the bottom. In between is the varying content.
Navigation bar links vary according to whether user is logged in.
Root of site displays, paginated to 18 per page, all images in the database. In a sidebar two tabs are available. Information about the images is presented in the 'Info' tab, and filtering options are shown in the 'Filters' tab.
A user can filter the images displayed by some of the parameters that were used to produce them, such as by the ids of the mathematical functions used, and the type, or flavour (Mandelbrot or Julia).
If a user hovers over an image it is scaled up in size.
If a user clicks on an image it is displayed on its own page, larger. Details of the parameters used are available in a panel below the image. So for example, if a user likes a couple of images created with Function 353, s/he might want to show only those images produced with that as the main function. S/he can do this simply from the 'Filters' tab.
If a user clicks 'Buy' s/he is taken to a Snipcart modal dialogue where the image has been added to the cart. This follows the usual conventions for checkout functionality, credit card payment, etc. This is currently a demonstration web project so Snipcart remains in test mode.
'About' page displays information about the site and the developer.
An unauthenticated user also sees links to 'Login' and 'Register'.
An authenticated user sees links to her/his profile page where s/he can upload a new profile picture, and to a 'Likes' page displaying all images s/he has liked, as well as to a 'Checkout' page for purchasing prints.
A user must be logged in to 'like' or purchase. 'Like' buttons are not shown to unauthenticated users. 'Buy' buttons are shown, but link to the 'Login' page.
If a user forgets his/her password, s/he can click on a 'Forgot password' link, enter an emaill address, and if this matches a user in the database, s/he will be sent a password reset link.

UX

User Stories

Bob, who happens to like abstract but not figurative or representational art, finds the site by chance. He should be drawn in by the images, be able, by filtering, to view other images similar to ones he likes and be tempted to click "Buy!".
Alice, who's looking for a large colourful print for a white wall in her new flat, should be encouraged to browse all the images until she finds one she likes and is drawn to buy it.

UI

Sketches for list view and detailed (single image) view:

I wanted a simple design because the images themselves are complex. Also sans-serif fonts, as the images have lots of curly stuff! I just sketched out on paper (scans embedded below) some rough ideas for the main views, as a starting point, knowing that that could change during the development process. In the end, knowing that design is not my strongpoint, I am fairly satisfied with how it has turned out.

Some ideas for improvement will no doubt become apparent with time, and will be appraised for possible implementation as appropriate. Pull requests welcome!

My Original Specification

Note: Actual implementation differs in some respects: see "Use" above.

The application must store an indeterminate number of images in non-volatile web-accessible storage.

It must provide methods to classify and filter these images based on various characteristics.

In order to do this, it must store or be able to extract data about the functions used to generate an image. This will be facilitated by these data having being encoded in the filenames of the images.

The administrator must be able to add or delete images.

Images must be stored in low resolution (877x620 pixels and 436x310 pixels) but the originals are 14032x9920 pixels, and it is these originals from which high-definition prints (A0, A1, A2, A3) will be made for purchasers. The filenames of the originals have a one-to-one relationship to the filenames of the low-resolution versions, so orders can be processed without ambiguity.

The application must be able to display the full set of images to the user, suitably scaled in size so as to display a number per viewport, and also let the user select images similar to a given image. An "Images like this" link should be available for all images, which filters the set by the functions used to generate the images. Most images have been generated by two functions, usually composed but in some cases alternated, so the user should be able to filter the set by either or both, and by an optional pre-transform.

Advanced functionality should allow the user to filter by other parameters used in the image generation and encoded in the filename, in particular subc, scp and sri. These are Boolean parameters which when true signify, respectively, that the added constant in the escape-time fractal algorithm has subsequently been subracted, that the same has been done for the pre-transformation function, and that the real and imaginary parts of z have been switched prior to iteration.

In future versions the user should also be able to filter by the predominant colour or quantity of black. This, however, requires some thinking about as it involves building logic to create histograms and make decisions based on their values, so will not be available in the initial version.

The application must provide secure user registration, authentication and authorisation. It must provide a secure password reset facility in case of forgotten passwords. It will not provide a "Forgot my email" facility.

When a user clicks on an image it should be enlarged, and a link to buy should be prominent.

An authenticated user should also be able to 'like' an image, and subsequently view the set of images s/he has liked.

If a user clicks to buy, they should be taken to a page where they can select a size, be informed of the price and complete the transaction with a credit card or Paypal etc.

The application must maintain a "shopping basket" or "cart" so that the user can buy more than one print in a single transaction. These data should be stored in the database so that if the user does not complete a purchase the items will remain in the basket and be there next time they log in.

A checkout and payment facility must be provided using a trusted third-party library compatible with Django.

In order not to distract or detract from the images themselves, the layout and design of the rendered pages should be as clean and simple as possible.

Design decisions

Deployment platform: Heroku
Database: Postgresql as Heroku add-on
Image storage: S3
Versions: Python(3.7.1), Django(2.1.7)
CI / testing: Travis
Payment processing: ~~Stripe~~ Snipcart
Styling: ~~Materialize~~ Bootstrap and own CSS

Models

User

Represents a user, who can register with an email address and thereafter be authenticated;
Fields: for each user we need name, email, id, whether registered and whether authenticated. We will need to link to the orders table with a foreign key.

Image

Represents a fractal image that can be displayed, that can be ordered as a print, that contains information about the functions and parameters used to create it, and that remembers orders;
Fields: name, image_id. We could pre-extract the creation data from the filenames and store as fields, but it will perhaps be simpler, and minimally expensive, to create functions to extract the data as needed. Each Image item must provide a method to get both small and large versions from storage.

Order

Represents an Order made by a User for an Image;
Fields: order_id, user_id (foreign key), order_details (containing despatch address, list of Image items and quantities, whether paid).
It was not found necessary to implement this as Snipcart handles and stores the orders; however, in a future version it could be useful.

Views and Templates

base.html will contain header and footer with links to register / login. Other templates will inherit from this:
index.html will be a Home Page displaying some images with links;
about.html will provide information about the application and the developer;
Other views will enable the user to view the entire set of images (maybe with pagination) or subset based on filtering criteria;
Clicking on a single image will cause it to be rendered alone, at ~80% screen width, with a "Buy" button at bottom right.
Other views will enable a user to register, deregister or amend account details.

Development process log / overview

Set up S3 and Stripe accounts, create S3 bucket for static files
Set up basic Django 2 project structure (project: 'trif')
Create and set environment variables
S3 and Stripe settings into settings.py
Other configuration in settings.py
Set up on Heroku and connected Github for automatic deployment on push
Configure Heroku Postgres database; using SQLite locally
Upload images to S3 bucket: Source Location: "trif-store/static/images/"
Create app: 'fract' and tested with 'Hello World' page
Do migrations, create superuser etc.
Store secrets in environment variables (config vars in Heroku)
Create User model and views following Corey Schafer's Django 2 tutorials, as suggested using code from Bootstrap and CS in base template. So I will try Bootstrap instead of Materialize
Create Image model, migrate
Instantiate a set of Image objects (which simply contain the filename and size, as strings), corresponding to the images actually stored as static assets in my S3 bucket
Iterate over this set, saving each Image object to the database. Used local images folder. See the [console log][1]
Many issues ensued, initially caused, I think, by my mistaken deletion of a migrations directory. At one point I upgraded Django from 2.1 to 2.2, backed up relevant files, recreated the project and apps. Later, after mistakenly pushing a faulty commit, I got into a mess with git and ended up doing a git rebase --hard and then git push -f. Not a very clean solution, rewriting history etc. but it got me sorted.
Minor tweaks to styling etc.
Change home page to a class-based view, subclassing ListView
Implement random ordering and pagination using a custom template tag
Style pagination buttons
Add favicon.ico
Set up gmail address with two-factor authentication
Implement and debug password reset via gmail, using Django's built-in PasswordResetView etc.
Improve styling
Attempt unsuccessfully to override save() method of django.contrib.auth.models.Model in order to resize profile images before saving; also tried by using the post_save() signal method; need to revisit this as otherwise large profile images will take too long to load
Create branch to implement Snipcart shopping facility
When mostly working, merge into master and deploy
Added and styled footer with social media icons
Implement code to let an authenticated user 'like' an image, and display the number of likes for each image
Tweak styling and fonts
Make sidebar into tabbed panel, 2nd tab displaying a form for filtering
Successfully implement views filtered by image parameters, by using a lambda in the override of 'get_queryset()' in class FilteredImageListView in fract/views.py
Write content for 'About' page and style it
Step by step, fix the various issues in the below 'To Do' list
Reasonably comprehensive manual testing: see the Testing" section below.
Fix Snipcart issue.
Solve static issue.
Address miscellaneous styling issues.
Create some tests, of Image model.
Set up CI/CT with Travis CI
Write more tests...

Final Stages: To Do:

- Snipcart - Empty cart on logout - DONE 4842df9
- Fix Snipcart sometimes not finding product when scraping page - DONE 44924f9
- Check all code and update comments and docstrings - DONE
- Finish this README.md, including acknowledgements and deployment guide - DONE (for now)
- Add username to 'Liked Images' header - DONE bb2fe87
- Add 'now hosted at...' to About page 1st milestone link - DONE 3dfc478
- Style pagination buttons and label - DONE e605b7c 186979b
- Fix 'Filters' link at bottom of Info panel not working - DONE (removed link, changed text)
- Redirect user back to previous page upon login (if straightforward) - DONE f14c651
- Scan draft sketches of list & detail pages and include here above - DONE 34dade7
- Fix favicon not being seen by browser - DONE 8dc29b6
- Write content for "About" page - DONE 657a470
- Add 'Details' popup or toast to show image parameters in detail view - DONE e3cc0a2
- Side panel - make tabs, show filtering options - DONE 73542c2 &c.
- Fix footer background - DONE 0a70b0b
- Adjust navbar icon placement and font size - DONE b61cf01 875cbf0 a3acae7
- Implement filtering options - DONE 73542c2
- Make background black beneath image card text - DONE 2a5c562
- Fix disappearing 'Heart' icon on mobile - NOT AN ISSUE, only when not logged in, by intention! 
- Automated Testing - ONGOING e2ac872 &c.
- Code linting / validation - DONE
- Run CSS through Autoprefixer - DONE f031628
- Test responsiveness on mobile devices - ONGOING

Features left to implement in future versions:

- Fix issue of Django admin static files being included in 'collectstatic'
- Find best way to shrink profile pictures, to obviate storing and serving too-large user-uploaded images
- Multiple sizes of prints
- Send a 'Like' to the server with Javascript rather than reloading the page, I guess with an AJAX request
- Store a user's cart in case s/he logs out or clears browser cache
- Improve styling and find a better way to present the images, perhaps with a higher-resolution version (perhaps 2806 x 1984) for users prepared to wait, and a smaller version for list view on mobile and thumbnail for shopping cart
- Further customise the Snipcart workflow 
- Store orders in the user profile, display for the user's convenience, and think how to use for marketing purposes
- Add Java function definitions to database so we can display as image detail for geeks
- Test in Opera, Safari, Edge

Testing

The project was tested manually at all stages, both on my local machine and on the Heroku dynos. In this way many bugs were uncovered and subsequently fixed.
At the point where most structure and functionality was in place, automated testing was introduced using the django.test module.
Continuous integration was then introduced using Travis CI
As of now, various indicators are tested against the home page view and the 'likes' view.
Development of further tests is in progress; currently of filtered views.

Browsers and Devices Manually Tested:

- Firefox Quantum 67.0 on Ubuntu 18 on two laptops
- Chromium 73 on Ubuntu 18 on two laptops
- Chrome 74 on Android (Asus Zenpad tablet)
- Android Browser 7.1.2 org.lineageos.jelly on small old Motorola phone
- A variety of virtual devices via Chrome Dev Tools Responsive Design Mode

Functionality Manually Tested:

- Check logging in and out, views change accordingly
- Check registering as new user and logging in and out and in again
- Check all links in navbar and footer, confirm opening in new tabs
- Check all external links in 'About' page, confirm opening in new tabs
- Click 'Forgot password' and confirm email link
- Add likes with different users, check Likes total updated appropriately
- Check upload and display of user profile pictures
- Check switching between pages
- Check switching between tabs in sidebar
- Check display is reasonable with reasonable resizings of browser window
- Check that filtering works by confirming in the Python shell
- Change things in the admin panel and try to break stuff (e.g. delete a user and then check profile removed by CASCADE)

Functionality Automatically Tested:

- Image parameters returned correctly for two test images
- `name_large` returned correctly for both small and large images
- New image has no likes
- If n users like a new image it has n likes
- If a user's profile is deleted and the user has liked an image then the number of likes returned for that image has one fewer like
- Home page returns 200 OK status
- Home page has correct number of images, taking pagination into account
- After user logs in, s/he is authenticated
- Likes page returns 200 OK status
- Empty set returned for user's 'Likes' page when no images liked
- When user has liked a set of images, the same set is returned by the 'Likes' view

Deployment

The project's git repository is hosted at Github and a push to this remote origin triggers a subsequent push and new build on Heroku, where the project is hosted on the Heroku free tier.
For security, all secret keys (Django, Amazon, Snipcart) are stored in Heroku config variables (environment variables).
The database used is Heroku's own Postgresql database.
The project uses Amazon S3 free tier for non-volatile static and media files, like CSS, icons, uploaded user profile pictures and the actual images displayed on the site.
The project uses Snipcart to handle user purchase orders, shopping cart, payments and backend notifications
The deployed site can be accessed with a web browser at Teraspora Fractals

Deployment Procedure

The project is called `trif`, and contains two non-system apps, `fract` and `users`.

1. Clone the repository into a clean directory on your local machine.

2. Create an account at Heroku and create an app.

3. Create a PostgreSQL database as a Heroku addon for the app.

4. Set up a 'bucket' on Amazon S3 for static and media file storage.

5. Upload appropriately-named images to S3 and adjust settings in `settings.py`.   The image naming schema is demonstrated in the comments in fract/img_params.py.

6. Store Django secret key, AWS secrets, Database keys etc. as Heroku 'Config Vars', or environment variables:
    - AWS_ACCESS_KEY_ID
    - AWS_SECRET_ACCESS_KEY
        These two from Amazon S3 account
    - DATABASE_URL
        From Heroku
    - EMAIL_USER
        Email address to send password reset links from
    - EMAIL_PASS
        Special app password for above address
    - SECRET_KEY
        Django secret key

7. Set up a Gmail account, enable app access for the project, enter the address and special app-enabled password in the two EMAIL_* vars above.

8. Push the code to Heroku, either directly or via Github.

9. Make and run migrations on Heroku.

10. Create a superuser on the PostgreSQL database in order to access Django's Admin interface.

11. `DEBUG` is set to `True` in `settings.py` only if `DEVELOPMENT` is set as an environment variable, so be sure not to set any config var with this name on the Heroku server as doing so risks exposing secrets in error messages.

12. When all seems to be working, run the app on Heroku (`heroku open -a <your Heroku App Name>`).

13. To run the project for real ecommerce disable the testing mode in Snipcart's dashboard.

Ongoing Issues

When I run python3 manage.py collectstatic, Django picks up my static files ok and uploads them to S3, but it also picks up a static directory inside my virtual environment, /lib/python3.7/site-packages/django/contrib/admin/static and uploads nearly 100 files to my S3 bucket. Evidently Django goes looking for them there, too, because if I delete them then the Django Admin UI is unstyled. My workaround has been to change the basename of this directory to static_temp, unless I need to use the Admin interface, in which case I change it back. I set up aliases to do this quickly. I am conscious, though, that it's a workaround, if not a fudge, but no-one has been able to discern the cause and I have not found an answer online, though it 'must' be something to do with the interaction of my settings in settings.py. If, alternatively, I run the collectstatic function in the Heroku CLI, Django finds static folders in .heroku/ instead! So I am keen to learn more about how Django handles static files so that I can resolve it soon.

UPDATE: I have realised that it's probably not an issue! It only need happen once, as Django's static files will not change unless I specify a different version of Django in my requirements.txt. Subsequent runs of collectstatic will not result in further uploads; it only happened twice for me because I deleted the files from my S3 bucket. Django-admin needs to find its static files somewhere!

A user can upload a large profile picture. Storing and serving such a file is unneccessary since a profile picture only needs to be 300x300 pixels at most. So I want to shrink it before saving it. I have tried this by overriding the post_save method of django.db.models.signals and by using Pillow to shrink it. I can get the shrink code working locally but ran into a number of issues implementing it properly, and my mentor assured me it would not lose me marks!
I'm not totally happy about reloading the page when a user clicks 'Like' on an image. I want to re-implement it with an asynchronous AJAX request so the user is not disturbed by a page reload. Again, though, my mentor opined that it would not lose me marks to leave it thus.
Snipcart functionality is not working properly every time. Some orders work, others fail when Snipcart tries to crawl the page.

I cannot get Snipcart to recognise my products consistently when it back-crawls my page. I suspect it's to do with data-item-url or data-item-name. I have tried with both a relative URL - occasionally it worked - and an absolute URL - doesn't seem to work at all.

UPDATE:
- I have fixed this, with the suggestion of Jean-Sébastien from Snipcart, who observed that I have a 'Login' button. Of course, Snipcart's page back-crawler will not be logged in, so the page it looks at will have the button removed by the Django template engine, as it's inside an {% if %} {% else %} block.
- My solution: include it also in an unauthenticated user's view, but with a class applied to give it display: none;.

Acknowledgements

The tutors, mentors and support staff at Code Institute
The tutors, mentors, alumni and fellow students on the Code Institute Slack channels
Corey Schafer for his excellent teaching on Youtube about Django, Python, Server setup and more
The Django docs, which are a paragon of documentation
The Python docs
My mentor Nishant Kumar
All the good answers and guides and tutorials and blogs on the web, on Stack Overflow and elsewhere, that have helped me on the way

Conclusion

This website is the culmination of over a year of study online. It is my fifth and final "Milestone Project" for the Code Institute course. It is built in Python 3.7 on the backend, with Django 2.2. The frontend templates are built using the Django template language, with Bootstrap 4 and my own CSS. It is hosted at Heroku. It uses a PostgreSQL database also hosted at Heroku to store image data and user profile data. The images themselves are stored in an Amazon S3 bucket, along with other static files. The project uses Travis CI for continuous integration and testing.

teraspora / trif