nnhjy / Spine-Samples

Small samples of using Spine tools

Overview

This repository consists of 4 Spine-Toolbox projects.

1. Spine Demo

A project containing several Spine database to illustrate SpineOpt.jl functionalities (demo SpineOpt models) and SpineInterface.jl utilities (incl. improvising Julia JuMP models using SpineInterface to interact with SpineDB)

2. SpinePython

A demo project of a workflow to run a Python Tool and exchange data with other tools: Data Connection, Julia Tool, and Importer for JSON and CSV files. The project workflow contains:

1. A dummy input CSV file in the "Input_files" Data Connection;

2. A Julia script wrapped in the "Julia" Tool that reads the given input CSV and writes to an output CSV file; the output CSV file is passed over to the "ImportCSV" Importer and other tools (see 3);

3. A Python script wrapped in the "Python" Tool that reads the given input CSV and the output csv from the Julia script and writes to a JSON file and another CSV file; the JSON output file is passed to the "ImportJSON" Importer.

3. BuildTimeseries project

A demo workflow to attach "DateTime" index to TimeSeries data

4. Hydro reservoir example

A sample SpineOpt.jl model for hydro reservoir generation

Set up working environment for Spine toolkits

Python (virtual) environment setup

1. Download and install python interpreter
2. Create a virtual environment: "X:\path\to\python\interpretor -m venv Y:\path\to\virtual_env_folder"
3. In the Terminal, switch to the corresponding virtual environment
   • Run "deactivate" to exit current virenv
   • Run "cd path\to\the\virenv\folder"
   • Run ".\Scripts\activate" to activate the target virenv
   • Run "pip install -e package" in terminal under the activated virtual environment (the "-e" option is only needed for installing editable package, see pip install options) When the ipython and ipykernel (maybe only ipykernel needed) packages are installed, command "pip install -e local\package" can be used in the Python console where the new virtual environment is launched.

Julia environment setup

1. Install julia

2. Modify the environment variable "julia" to the latest Julia installed

   • System environment variable (if you install Julia for all users): "Control Panel\All Control Panel Items\System --> Advanced system settings --> Advanced --> Environmental Variables --> Edit in Path"
     • under "User variables for administrator_username" (only for the administrator, under "System variables" for all users)
   • User environment variable (if you install Julia for the current user): Search environment variables in the Windows taskbar --> "Edit environment variable for your account" --> Edit in Path under "User variables for the_current_username"
   • Edit in Path: Click New to add X:\directory\to\Julia-x.x.x\bin and delete the directory for old versions
   • In case the Jupyter notebook kernel needs updating, do:

      Pkg.update(); Pkg.build("IJulia")

   • Renew the path of julia executable in VSCode
     • Settings --> Search "julia" --> Julia: Environment Path / Julia: Executable Path

3. Install Spine related jl package:

   • pkg> dev "...path to SpineOpt folder...", pkg> dev "...path to SpineInterface folder..."

4. Build PyCall environment

   There are 3 ways by which PyCall.jl could work for the SpineInterface.jl call use spinedb-api.

   • pkg> add PyCall

   • Option 1 Install spinedb_api to the default conda python interpreter of Julia.

     • Confirm the default conda python is linked:

      julia> ENV["PYTHON"]=""
      julia> using PyCall, Pkg
      julia> Pkg.build("PyCall")

     • Find the conda python
     • "X:\path\to\the\conda\python\folder\Scripts\pip.exe install -e path/to/local/spinedb_api"

   • Option 2 Install spinedb_api to the parent python interpreter on which the virtual env is built (ref. PyCall documentation and PyCall issue with virtual environment)

     • Run (with ipython and ipykernel installed) pip install -e path/to/local/spinedb_api
     • Otherwise, X:\path\to\python\folder\Scripts\pip.exe install -e path/to/local/spinedb_api

   • Build customised python interpreter:

   • In Julia after activating the desired working environment:

      julia> ENV["PYTHON"] = raw"C:\path\to\the\parent\python.exe"
      julia> import Pkg; Pkg.build("PyCall")

   • Option 3 (recommended) For a dedicated virtual environment created by venv and virtualenv, PyCall could work provided that the Python executable used in the virtual environment is linked against the same libpython used by PyCall (conda environment not supported). See the explanation here.

     • In command line: virtual_env_folder\Scripts\Activate.ps1
     • In Julia after activating the desired working environment:

      julia> ENV["PYTHON"] = Sys.which("python")
      julia> ENV["PYCALL_JL_RUNTIME_PYTHON"] = Sys.which("python")
      julia> import Pkg; Pkg.build("PyCall")

   • Relaunch Julia and check which python the PyCall is using: PyCall.pyprogramname or PyCall.python

   • Check packages that are available for the PyCall used python:

      julia> using PyCall; pyimport("sys").path
      py"""
      import pkgutil
     
      def get_available_packages():
      	available_packages = [name for _, name, _ in pkgutil.iter_modules()]
      	return available_packages
      """
      available_packages = py"get_available_packages"()
      println(available_packages)

     When spinedb_api package is in the list, it is good to go.

(WIP) New approach to set up Spine working/development environment

Warning!!!: Under this configuration, only running the spinetoolbox requires activating the conda environment. Once the PyCall.jl is configured to use the conda python, there is no need to activate the conda environment for julia related tasks, e.g. updating environment packages (doing this with the conda activated causes unnecessary error).

Step 1 option 1. Build worry-free spinetoolbox in conda environment with git:

• prerequisites:

  • python 3.11 from miniconda or Anaconda
  • git

• In OS terminal (cmd or PowerShell):

   conda create -n spine-tools python=3.11
   coda activate spine-tools

• Install the active dev version spinetoolbox (need git installed):

  pip install git+https://github.com/spine-tools/spinetoolbox-dev

• Update package to the latest commit: reinstall the package using the same command.

Step 1 option 2. Conda environment for spine development

• prequisites:

  • python 3.11 from miniconda
  • git

• Create a new conda virtual environment:

   conda create -n spine-dev python=3.11

• install the python packages of spine-tools:

   conda activate spine-dev
   pip install -e .\path\to\local\Spine-Database-API
   pip install -e .\path\to\local\spine-engine
   pip install -e .\path\to\local\spine-items
   pip install -e .\path\to\local\Spine-Toolbox

  Update packages to the latest commits:

  1. Pull from the GitHub repositories
  2. Rerun the above commands

Step 2. Assign the configured conda Python to PyCall in the Julia environment where SpineOpt.jl and SpineInterface.jl are installed

This step is required for both options (the spine-dev and spine-tools conda env) unless you install SpineOpt.jl and SpineInterface.jl through the default channel of Spine toolbox.

• In OS terminal (cmd or PowerShell)

   conda activate spine-dev 
   cd path\to\the\working\julia\enviroment

• In Julia console

   # activate the working environment
   cd("path\\to\\the\\working\\julia\\enviroment")
   using Pkg; Pkg.activate(".")
   
   # the python of current console environment, also works with non-conda python
   ENV["PYTHON"] = Sys.which("python")
   # or when a conda environment is activated
   # ENV["PYTHON"] = ENV["CONDA_PREFIX"] * "\\python.exe"
   using Pkg; Pkg.build("PyCall")

• Relaunch Julia and check which python the PyCall is using: PyCall.pyprogramname or PyCall.python