circleci cli docker gitlab-ci pytest python pipenv

coda-cli

This application is a Python CLI that interacts with Coda.io, which is a SaaS. This project consist of (1) python CLI; (2) CI pipeline that automates your static analysis and image deployment; and (3) CD pipeline to deploy a custom Docker image that interacts with Coda.io (Blocked).

coda-cli
Introduction
- Purpose
- Audience
System Overview
User Personas
- RACI Matrix
Requirements
- Local workstation
Usage
- Running coda-cli in your terminal
Shaping
Building
- Building and testing a command and return its output
Troubleshooting and FAQs
- Error The following signatures couldn't be verified because the public key is not available
References
- Things to learn and research

1. Introduction

1.1. Purpose

This document describes the coda-cli command-line wrapper for Coda.io API, an SaaS service that provides interactive documents similar to Notion, and Google Docs.

1.2. Audience

The audience for this document includes:

User who will create and maintain YAML config files to deploy Coda documents using the CLI.
DevSecOps Engineer who will develop the CLI, create unit tests, configure build tools, create and maintain continuous integration and deployment (CI/CD) pipelines, and write documentation.

2. System Overview

2.1. Benefits and Values

Currently, creating a Coda document requires the User to navigate the Coda user interface (UI) and perform click operations, which may be inefficient and error prone.
This CLI allows the User to create and maintain YAML config files that defines a Coda document as code, hence reducing the error rate, while increasing reusability and adding version control for a document as code.

2.2. Workflow

This project uses several methods and products to optimize your workflow.

Use a version control system (GitHub) to track your changes and collaborate with others.
Use a static analyzer (prospector) to help write your clean code.
Use a build tool (Makefile) to automate your development tasks.
Use a package manager (Pipenv) to manage your dependencies.
Use a testing framework (pytest and click.testing) to automate your testing.
Use a containerization platform (Docker) to run your application in any environment.
Use a continuous integration pipeline (CircleCI) to automate your static analysis and image deployment.
Use an artifactory (Docker Hub) to store and pull your image.
Use a continuous delivery pipeline (GitLab) to automate your Docker container deployment and interaction with Coda.io (Blocked: Coda.io doesn't support POST request for many of its resources.)

2.3. Place, Affordance, Connection

Places users can navigate
- Circle CI https://app.circleci.com/settings/project/github/dennislwm/coda-cli
- GitLab CD https://gitlab.com
- Docker Hub e.g. https://hub.docker.com/repository/docker/dennislwm/coda-cli
Affordance users can act
- Docker pull docker pull dennislwm/coda-cli:latest

2.4. Limitation

The CD pipeline is currently blocked, because Coda.io API does not support POST requests for many of its resources.

3. User Personas

3.1 RACI Matrix

Category	Activity	User	DevSecOps
Usage	Running `coda-cli` in your terminal	R,A
Shaping	Creating a single CLI command		R,A
Shaping	Creating test cases for each command		R,A
Shaping	Creating a Dockerfile and testing it locally		R,A
Shaping	Requiring status checks to pass before merging PR		R,A
Shaping	Building, testing, tagging and uploading our app image using CI		R,A
Building	Building and testing a command and return its output		R,A

4. Requirements

4.1. Local workstation

Before running the Python app on your local workstation, you need the following:

Generate a CODA_API_KEY and save it in an .env file, or run the command export CODA_API_KEY=<CODA_API_KEY>
Download and install jq command line application.
Install Python dependencies in a virtual environment within the app folder with pipenv shell && make install_new.

5. Usage

5.1. Running `coda-cli` in your terminal

python coda.py
Usage: coda.py [OPTIONS] COMMAND [ARGS]...

  This script prints coda data

Options:
  --version                       Show the version and exit.
  -o, --out [csv|json|markdown|text]
                                  Output type, default=text
  -h, --help                      Show this message and exit.

Commands:
  list-docs      Returns the list of documents in Coda
  list-sections  Returns the list of sections in a doc
  list-tables    Returns the list of tables in a doc

6. Shaping

As shaping and building have independent cycles, we will define shaping as any work that does not involve implementation of code. This work may include evaluation, feasibility, comparison, research, etc.

We set a time constraint of 9 workdays, for shaping, and an additional 9 workdays for building. Hence, the total time for this project is approximately 20 workdays with a cool-down of 2 workdays.

6.1. Creating a single CLI command

Create a single CLI command with Python

6.2. Creating test cases for each command

Create test cases for each command with Python

6.3. Creating a Dockerfile and testing it locally

Create a Dockerfile and test it locally
- Create a Dockerfile
- Generate a requirements file

6.4. Requiring status checks to pass before merging PR

Require status checks to pass before merging PR
- Status Checks
- Continuous Integration

6.5. Building, testing, tagging and uploading our app image using CI

Build, test, tag and upload our app image using CI

The project started on 17-Aug-2022 and is currently a work-in-progress and ahead of schedule. The shaping cycle was completed on 19-Aug-2022 (2 days), while the building cycle is in progress.

Note: All the list-* commands have been implemented and tested.

7. Building

7.1. Building and testing a command and return its output

This steps are repeatable, i.e. create and test a command.

Build and test a command and return its output

9. Troubleshooting and FAQs

9.1. Error The following signatures couldn't be verified because the public key is not available

Context Analysis:

Project: coda-cli
CI Workflow: static-analysis-workflow
CI Job: static-analysis
CI Step: Build image
CI Last Command: cd app && make ci_test_build
CI Output:

 => ERROR [2/7] RUN apt-get update                                         0.5s
------
 > [2/7] RUN apt-get update:
#5 0.263 Get:1 http://deb.debian.org/debian bookworm InRelease [151 kB]
#5 0.270 Get:2 http://deb.debian.org/debian bookworm-updates InRelease [52.1 kB]
#5 0.271 Get:3 http://deb.debian.org/debian-security bookworm-security InRelease [48.0 kB]
#5 0.301 Err:1 http://deb.debian.org/debian bookworm InRelease
#5 0.301   The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 0E98404D386FA1D9 NO_PUBKEY 6ED0E7B82643E131 NO_PUBKEY F8D2585B8783D481

Problem Analysis:

The docker build command fails on the second layer, i.e. RUN apt-get update, with error The following signatures couldn't be verified because the public key is not available.
StackOverflow top answer recommends upgrading the docker on the host.
CircleCI documentation suggests possible docker versions:

default - set to latest Docker 24.
docker23 - set to previous Docker 23.
20.10.24 - specific Docker version.

The custom CI config .circleci/config.yml has an older docker version, i.e. 19.03.13, used in both static-analysis and deploy-image jobs.
The solution appears to be updating the docker version to default.

8. References

8.1. Things to learn and research

In no particular order, my plan is to use the following resources to learn and research.

Title	Author	Publisher Date [Short Code]
E-Book: Shape Up	Ryan Singer	Basecamp 2021
GitHub Repo: Blasterai/codaio	BlasterAI	2022
E-Doc: Click	Pallets	2022
GitHub Repo: dennislwm/archiveso	Dennis Lee	2022

About

Python CLI application using Shape Up technique and GitOps methodology