The-Academic-Observatory / coki-oa-web

COKI Open Access Dashboard Web Application

COKI Open Access Website

COKI Open Access Website is the Next.js based front end and Cloudflare Workers backend for the Curtin Open Knowledge Initiative's Open Access Dashboard: open.coki.ac

1. Requirements
2. Web App Development
3. REST API Development
4. Configuring GitHub Actions
5. REST API Endpoints
6. Contributors
7. License

1. Requirements

OS Requirements

• Linux, Windows or MacOS.
• NodeJS 20: https://nodejs.org/en/
• yarn 4: https://yarnpkg.com/getting-started/install
• Install system dependencies: sudo apt-get -y install libc++1 unzip (libc++1 may be required for Wrangler 3)
• Vercel CLI: npm install --global vercel

Preparing Data Files

The data for the COKI OA Website is generated by the oa_web_workflow. Apache Airflow workflow. This workflow needs to be run first to produce the data files that will be visualised by the website. Normally the following steps are performed automatically by the workflow when the website is built and deployed, however, during development this needs to be done manually.

From the Google Cloud Storage bucket, download the v10/data.zip file, saving it in the ./data folder of the project.

Run the ./bin/data.sh . script, which will extract the data files and copy them to the right places (this will also run on Windows).

2. Web App Development

See below for instructions specific to running the web application.

Install dependencies

Install dependencies:

yarn install

Running Development Server

To load the local database:

cd workers-api
yarn build # Creates ../data/data/db.sql
wrangler d1 execute DB --file ../data/data/db.sql --local --config wrangler.local.toml

Run the development server:

cd ../dashboard
yarn dev

Then, open http://localhost:3000 with your browser to see the result.

Running tests

Run Jest unit tests:

yarn test:jest

Install playwright for end-to-end tests:

npx playwright install-deps
npx playwright install
npx playwright install msedge

Runs playwright end-to-end tests:

yarn test:e2e

Build & Deploy

Export environment variables:

export COKI_ENVIRONMENT=develop
export COKI_SITE_URL=https://develop.open.coki.ac
export COKI_API_URL=https://develop.api.coki.ac

Make sure that COKI_API_URL is pointing to a running version of the API, as it will be used during the build process.

Deploy Preview

Make sure that you are in the dashboard directory.

Pull preview project config:

vercel pull --environment=preview

Build project:

vercel build

Deploy preview:

vercel deploy --prebuilt

Deploy Production

Pull production project config:

vercel pull --environment=production

Build production project:

vercel build --prod

Deploy to production:

vercel deploy --prebuilt  --prod

3. REST API Development

See below for instructions specific to running the workers-api REST API.

Enter the workers-api folder:

cd ./workers-api

Running Development Server

Run the development server. This uses Miniflare to simulate Cloudflare Workers functionality.

yarn dev

Then, open http://127.0.0.1:8787/search/curtin with your browser or CURL to see the result of an API call.

Running tests

Runs Jest unit tests:

yarn test:e2e

Build & Deploy

Customise your wrangler.toml file, see wrangler.example.toml:

name = "coki-oa-web-api"
account_id = "my-cloudflare-account-id"
compatibility_date = "2022-02-03"

[build]
command = "yarn run build"

[site]
bucket = "./public"

[env.develop]
route = "my-develop-domain/*"

[env.staging]
route = "my-staging-domain/*"

[env.production]
route = "my-production-domain/*"

Make sure you are in the workers-api directory.

Build & deploy to develop:

yarn run wrangler deploy -e develop

Build & deploy to staging:

yarn run wrangler deploy -e staging

Build & deploy to production:

yarn run wrangler deploy -e production

Rendering Social Cards

This will be removed when card generation is performed by the workflow or the API.

To render the cards:

sudo apt-get install libgtk-3-dev libasound2
cd ./social-cards
npx puppeteer browsers install firefox
PUPPETEER_PRODUCT=firefox
yarn render

The social cards are saved in workers-images/public/social-cards.

• Make a zip file of the social-cards folder.
• Upload it to the coki-oa-web-data Google Cloud Storage Bucket. The cards will then be built into the latest.zip file by the Airflow workflow.

To manually view the social cards, start the local webserver and navigate to the following routes:

• /cards/country/[COUNTRY ID]
• /cards/institution/[ROR ID]

Making Icons

Download the Adobe Illustrator icons-final.ai file.

• Add your icon as a new artboard, it should be 64x64px in size.
• File > Export for Screens > Format SVG, Export Artboard.
• Open the IcoMoon App: https://icomoon.io/app/#/select.
• Click "Import Icons" and choose components/common/selection.json.
• Update or add your new icon.
• Select the icons you want to export.
• Click the "Generate Font" tab.
• Click Download.
• Get the selections.json file out of the downloaded file and replace the old selections.json file.

4. Configuring GitHub Actions

The following GitHub Secrets need to be created:

• GCP_CREDENTIALS: credentials for gsutil.
• BUCKET_NAME: the name of the Google Cloud Storage bucket that contains the data files.
• WRANGLER_CONFIG: the Cloudflare Wrangler configuration file.
• WRANGLER_CONFIG_API: the Cloudflare Wrangler configuration file for the workers-api Cloudflare Worker.
• CLOUDFLARE_API_TOKEN: the Cloudflare API Token.

See below for instructions on how to set these up.

wrangler.toml

Two wrangler.toml files need to be created, one for the Web App Cloudflare Worker and one for the API Cloudflare Worker.

For the web app, see the wrangler.example.toml file for an example of how to create the wrangler.toml file. Add the file to the WRANGLER_CONFIG GitHub Secret.

For the API, see workers-api/wrangler.example.toml file for an example of how to create the wrangler.toml file. Add the file to the WRANGLER_CONFIG_API GitHub Secret.

Cloudflare API Token

This connection contains Cloudflare API token that enables Wrangler to publish the Open Access Website to Cloudflare. See Cloudflare Creating API tokens for instructions on how to create a Cloudflare API token. The Cloudflare API token is already URL encoded. The required settings for the token are listed below.

Permissions:

• Account, Workers KV Storage:Edit
• Account, Workers Scripts:Edit
• Account, Account Settings:Read
• User, User Details:Read
• Zone, Workers Routes: Edit

Account Resources:

• Include, Zone Resources: Include, Specific zone:

cloudflare_api_token: http://:<cloudflare_api_token>@

External references

• Cloudflare Creating API tokens

5. REST API Endpoints

The COKI Open Access REST API offers a full text search endpoint for identifying countries and institutions based on their names. Additionally, the API provides filtering capabilities that allow users to find a range of countries and institutions that match specified parameters such as region, subregion, open access percentage, and number of outputs. Detailed information about individual countries and institutions can be retrieved using dedicated endpoints.

The data available through the REST API is copyrighted by Curtin University and licensed under the Creative Commons Attribution 4.0 International License, except the data in the description.text fields. These are licensed under the Creative Commons Attribution-ShareAlike 3.0 Unported License, as they are derived from Wikipedia.

Search

Search for countries and institutions by name.

GET /search/:text

Parameter	Description
:text	The url encoded name or partial name of a country or institution to search for.

Query Parameter	Description
page[number]	The page number starting at 0. Default value of 0.
limit[number]	The number of items per page.

Sample Request

curl https://api.coki.ac/search/curtin%20university

Sample Response

{
  "items": [
    {
      "id": "02n415q13",
      "name": "Curtin University",
      "logo_sm": "logos/institution/sm/02n415q13.jpg",
      "entity_type": "institution",
      "country_name": "Australia",
      "country_code": "AUS",
      "subregion": "Australia and New Zealand",
      "region": "Oceania",
      "institution_type": "Education",
      "stats": {
        "n_outputs": 42938,
        "n_outputs_open": 18388,
        "p_outputs_open": 42.824537705528904,
        "p_outputs_publisher_open_only": 8,
        "p_outputs_both": 16,
        "p_outputs_other_platform_open_only": 19,
        "p_outputs_closed": 57
      }
    }
  ],
  "nItems": 1,
  "page": 0,
  "limit": 18
}

Country

Get the full details for a country.

GET /country/:id

Parameter	Description
:id	A three letter country ISO code.

Sample Request

curl https://api.coki.ac/country/NZL

Sample Response

{
  "id": "NZL",
  "name": "New Zealand",
  "description": {
    "text": "New Zealand is an island country in the southwestern Pacific Ocean. It consists of two main landmasses\u2014the North Island and the South Island\u2014and over 700 smaller islands. It is the sixth-largest island country by area, covering 268,021 square kilometres.",
    "license": "https://en.wikipedia.org/wiki/Wikipedia:Text_of_Creative_Commons_Attribution-ShareAlike_3.0_Unported_License",
    "url": "https://en.wikipedia.org/wiki/New_Zealand"
  },
  "entity_type": "country",
  "logo_sm": "logos/country/sm/NZL.svg",
  "logo_md": "logos/country/md/NZL.svg",
  "logo_lg": "logos/country/md/NZL.svg",
  "wikipedia_url": "https://en.wikipedia.org/wiki/New_Zealand",
  "region": "Oceania",
  "subregion": "Australia and New Zealand",
  "start_year": 2000,
  "end_year": 2021,
  "stats": {
    "n_citations": 5634926,
    "n_outputs": 214124,
    "n_outputs_open": 75448,
    ...
  },
  "years": [
    {
      "year": 2000,
      "date": "2000-12-31",
      "stats": {
        "n_citations": 176810,
        "n_outputs": 3463,
        "n_outputs_open": 668,
        ...
      }
    },
    ...
  ],
  "repositories": [
    {
      "id": "PubMed Central",
      "total_outputs": 24570,
      "category": "Domain",
      "home_repo": false
    },
    {
      "id": "Europe PMC",
      "total_outputs": 19203,
      "category": "Domain",
      "home_repo": false
    },
    {
      "id": "Semantic Scholar",
      "total_outputs": 5851,
      "category": "Public",
      "home_repo": false
    },
    ...
  ]
}

Institution

Get the full details for an institution.

GET /institution/:id

Parameter	Description
:id	The ROR ID of the institution.

Sample Request

curl https://api.coki.ac/institution/030cszc07

Sample Response

{
  "id": "030cszc07",
  "name": "Australian Astronomical Observatory",
  "description": {
    "text": "The Australian Astronomical Observatory, formerly the Anglo-Australian Observatory, was an optical and near-infrared astronomy observatory with its headquarters in North Ryde in suburban Sydney, Australia.",
    "license": "https://en.wikipedia.org/wiki/Wikipedia:Text_of_Creative_Commons_Attribution-ShareAlike_3.0_Unported_License",
    "url": "https://en.wikipedia.org/wiki/Australian_Astronomical_Observatory"
  },
  "entity_type": "institution",
  "logo_sm": "logos/institution/sm/030cszc07.jpg",
  "logo_md": "logos/institution/md/030cszc07.jpg",
  "logo_lg": "logos/institution/lg/030cszc07.png",
  "url": "http://www.aao.gov.au/",
  "wikipedia_url": "https://en.wikipedia.org/wiki/Australian_Astronomical_Observatory",
  "region": "Oceania",
  "subregion": "Australia and New Zealand",
  "country_name": "Australia",
  "country_code": "AUS",
  "institution_type": "Facility",
  "start_year": 2000,
  "end_year": 2021,
  "stats": {
    "n_citations": 72555,
    "n_outputs": 1243,
    "n_outputs_open": 1115,
    ...
  },
  "years": [
    {
      "year": 2000,
      "date": "2000-12-31",
      "stats": {
        "n_citations": 171,
        "n_outputs": 3,
        "n_outputs_open": 2,
        ...
      }
    },
    ...
  ],
  "acronyms": [
    "AAO"
  ],
  "repositories": [
    {
      "id": "arXiv",
      "total_outputs": 1101,
      "category": "Preprint",
      "home_repo": false
    },
    {
      "id": "Australian National University - ANU Open Research",
      "total_outputs": 247,
      "category": "Institution",
      "home_repo": false
    },
    {
      "id": "University College London - UCL Discovery",
      "total_outputs": 194,
      "category": "Institution",
      "home_repo": false
    },
    ...
  ]
}

Countries

Return a paginated list of countries and optionally filter them.

GET /countries

The countries endpoint uses optional query parameters rather than path parameters.

Query Parameter	Description
page[number]	The page number starting at 0. Default value of 0.
limit[number]	The number of items per page. Default value of 18.
orderBy[string]	What field to order the results by, nested fields are supported. Default value of stats.p_outputs_open.
orderDir[string]	What direction to sort results, either ascending asc or descending dsc. Default value of dsc.
ids[string]	A list of 3 letter ISO country codes to return specific countries, separated by commas, e.g. ?ids=AUS,NZL. This parameter overrides all other parameters.
regions[string]	A list of regions to filter by. These must be URL encoded region names separated by commas, e.g. ?regions=Oceania,Americas.
subregions[string]	A list of subregions to filter by. These must be URL encoded subregion names separated by commas, e.g. ?subregions=Southern%20Asia,Latin%20America%20and%20the%20Caribbean
minNOutputs[number]	The minimum inclusive number of outputs a country must have to be returned. Default value of 1000.
maxNOutputs[number]	The maximum inclusive number of outputs a country must have to be returned. Default value of Number.MAX_VALUE.
minPOutputsOpen[number]	The minimum inclusive open access percentage a country must have to be returned. Default value of 0.
maxPOutputsOpen[number]	The maximum inclusive open access percentage a country must have to be returned. Default value of 100.

Query Parameter enumerations:

Query Parameter	Enumerations
orderDir	asc, dsc.
ids	3 letter ISO country codes.
regions	Africa, Americas, Asia, Europe and Oceania.
subregions	Australia and New Zealand, Central Asia, Eastern Asia, Eastern Europe, Latin America and the Caribbean, Melanesia, Micronesia, Northern Africa, Northern America, Northern Europe, Polynesia, South-eastern Asia, Southern Asia, Southern Europe, Sub-Saharan Africa, Western Asia, Western Europe.

Sample Request

curl "https://api.coki.ac/countries?page=0&regions=Oceania"

Sample Response

{
  "items": [
    {
      "id": "PNG",
      "name": "Papua New Guinea",
      "logo_sm": "logos/country/sm/PNG.svg",
      "entity_type": "country",
      "subregion": "Melanesia",
      "region": "Oceania",
      "stats": {
        "n_outputs": 1964,
        "n_outputs_open": 1022,
        "p_outputs_open": 52.0366598778004,
        "p_outputs_publisher_open_only": 12,
        "p_outputs_both": 29,
        "p_outputs_other_platform_open_only": 11,
        "p_outputs_closed": 48
      }
    },
    {
      "id": "AUS",
      "name": "Australia",
      "logo_sm": "logos/country/sm/AUS.svg",
      "entity_type": "country",
      "subregion": "Australia and New Zealand",
      "region": "Oceania",
      "stats": {
        "n_outputs": 1397124,
        "n_outputs_open": 556603,
        "p_outputs_open": 39.839198238667436,
        "p_outputs_publisher_open_only": 10,
        "p_outputs_both": 17,
        "p_outputs_other_platform_open_only": 13,
        "p_outputs_closed": 60
      }
    },
    {
      "id": "NZL",
      "name": "New Zealand",
      "logo_sm": "logos/country/sm/NZL.svg",
      "entity_type": "country",
      "subregion": "Australia and New Zealand",
      "region": "Oceania",
      "stats": {
        "n_outputs": 214072,
        "n_outputs_open": 75871,
        "p_outputs_open": 35.44181396913188,
        "p_outputs_publisher_open_only": 11,
        "p_outputs_both": 15,
        "p_outputs_other_platform_open_only": 10,
        "p_outputs_closed": 64
      }
    },
    ...
  ],
  "nItems": 4,
  "page": 0,
  "limit": 18,
  "orderBy": "stats.p_outputs_open",
  "orderDir": "dsc"
}

Institutions

Return a paginated list of institutions and optionally filter them.

GET /institutions

The institutions endpoint uses optional query parameters rather than path parameters.

Query Parameter	Description
page[number]	The page number starting at 0. Default value of 0.
limit[number]	The number of items per page. Default value of 18.
orderBy[string]	What field to order the results by, nested fields are supported. Default value of stats.p_outputs_open.
orderDir[string]	What direction to sort results, either ascending asc or descending dsc. Default value of dsc.
ids[string]	A list of ROR IDs to return specific institutions, separated by commas, e.g. ?ids=02n415q13,03b94tp07. ROR IDs should have the https://ror.org/ stripped. This parameter overrides all other parameters.
regions[string]	A list of regions to filter by. These must be URL encoded region names separated by commas, e.g. ?regions=Oceania,Americas.
subregions[string]	A list of subregions to filter by. These must be URL encoded subregion names separated by commas, e.g. ?subregions=Southern%20Asia,Latin%20America%20and%20the%20Caribbean
institutionTypes[string]	A list of institution types to filter by. These must be institution types separated by commas, e.g. ?institutionTypes=Education,Facility
countries[string]	A list of countries that institutions must belong to. These must 3 letter ISO country codes separated by commas, e.g. ?countries=NZL,AUS
minNOutputs[number]	The minimum inclusive number of outputs an institution must have to be returned. Default value of 1000.
maxNOutputs[number]	The maximum inclusive number of outputs an institution must have to be returned. Default value of Number.MAX_VALUE.
minPOutputsOpen[number]	The minimum inclusive open access percentage an institution must have to be returned. Default value of 0.
maxPOutputsOpen[number]	The maximum inclusive open access percentage an institution must have to be returned. Default value of 100.

Query Parameter enumerations:

Query Parameter	Enumerations
orderDir	asc, dsc.
ids	Any ROR id from: https://ror.org/ with "https://ror.org/" removed from the ID.
institutionTypes	Education, Healthcare, Company, Archive, Nonprofit, Government, Facility, Other.
regions	Africa, Americas, Asia, Europe and Oceania.
subregions	Australia and New Zealand, Central Asia, Eastern Asia, Eastern Europe, Latin America and the Caribbean, Melanesia, Micronesia, Northern Africa, Northern America, Northern Europe, Polynesia, South-eastern Asia, Southern Asia, Southern Europe, Sub-Saharan Africa, Western Asia, Western Europe.
countries	3 letter ISO country codes.

Sample Request

curl "https://api.coki.ac/institutions?page=0&regions=Oceania&institutionTypes=Facility"

Sample Response

{
  "items": [
    {
      "id": "030cszc07",
      "name": "Australian Astronomical Observatory",
      "logo_sm": "logos/institution/sm/030cszc07.jpg",
      "entity_type": "institution",
      "country_name": "Australia",
      "country_code": "AUS",
      "subregion": "Australia and New Zealand",
      "region": "Oceania",
      "institution_type": "Facility",
      "stats": {
        "n_outputs": 1243,
        "n_outputs_open": 1113,
        "p_outputs_open": 89.54143201930812,
        "p_outputs_publisher_open_only": 3,
        "p_outputs_both": 48,
        "p_outputs_other_platform_open_only": 39,
        "p_outputs_closed": 10
      }
    },
    {
      "id": "05qajvd42",
      "name": "Australia Telescope National Facility",
      "logo_sm": "logos/institution/sm/05qajvd42.jpg",
      "entity_type": "institution",
      "country_name": "Australia",
      "country_code": "AUS",
      "subregion": "Australia and New Zealand",
      "region": "Oceania",
      "institution_type": "Facility",
      "stats": {
        "n_outputs": 1742,
        "n_outputs_open": 1493,
        "p_outputs_open": 85.7060849598163,
        "p_outputs_publisher_open_only": 9,
        "p_outputs_both": 42,
        "p_outputs_other_platform_open_only": 35,
        "p_outputs_closed": 14
      }
    },
    {
      "id": "03dsbfb14",
      "name": "Australian Institute of Tropical Health and Medicine",
      "logo_sm": "logos/institution/sm/03dsbfb14.jpg",
      "entity_type": "institution",
      "country_name": "Australia",
      "country_code": "AUS",
      "subregion": "Australia and New Zealand",
      "region": "Oceania",
      "institution_type": "Facility",
      "stats": {
        "n_outputs": 1011,
        "n_outputs_open": 741,
        "p_outputs_open": 73.29376854599407,
        "p_outputs_publisher_open_only": 7,
        "p_outputs_both": 55,
        "p_outputs_other_platform_open_only": 11,
        "p_outputs_closed": 27
      }
    },
    ...
  ],
  "nItems": 21,
  "page": 0,
  "limit": 18,
  "orderBy": "stats.p_outputs_open",
  "orderDir": "dsc"
}

6. Contributors

Conceptualization: James P. Diprose, Lucy Montgomery, Cameron Neylon, and Richard Hosking.
Graphic design: Richard Rigoni.
Data curation: James P. Diprose and Richard Hosking.
Formal analysis: James P. Diprose, Cameron Neylon, and Richard Hosking.
Funding acquisition: Lucy Montgomery and Cameron Neylon.
Investigation: James P. Diprose.
Methodology: James P. Diprose.
Project administration: Lucy Montgomery, Cameron Neylon, and Kathryn R. Napier.
Resources: James P. Diprose.
Software: James P. Diprose and Aniek Roelofs.
Supervision: James P. Diprose, Lucy Montgomery, Cameron Neylon, and Kathryn R. Napier.
Visualization: James P. Diprose, Cameron Neylon and Richard Regoni.
Writing - original draft: James P. Diprose, Lucy Montgomery, Cameron Neylon, and Richard Hosking.
Writing - review & editing: Cameron Neylon, Kathryn R. Napier, Richard Hosking, Katie S. Wilson, and Tuan-Yow Chien.

7. License

Apache 2.0 License