zhgzhg / Drizzle

Dependency manager plugin for Arduino IDE

Drizzle

Declarative package manager plugin for Arduino IDE v1.8.x.

Drizzle installs libraries, platforms, boards, picks board-specific settings, and even installs other Arduino IDE tools with a single click. The preferences are described into the sketch's comments or in a separate JSON file. Switching between Arduino projects with different settings or compiling them on different environments has never been easier!

Drizzle operates on top of Arduino IDE's library and board manager. It enables them to provide components, which now can be described in text form. Because the only change in the sketch is the addition of a few comments or editing a separate JSON file, the sketch will remain compatible with Arduino IDEs which don't have Drizzle installed.

How to Use

1. Create a comment inside the main file of your Arduino sketch (preferably at the beginning).

2. Use markers like @DependsOn, @BoardManager, @BoardSettings, @Board, @Preferences, and @ArduinoTool to describe the sketch's requirements.

   1. You may use some of the Tools -> Drizzle -> Auto-generate ... marker UI options to let Drizzle fill them as a C-style comment at the beginning of your main sketch file, based on IDE's current settings. Depending on the environment the result might require adjustments.

3. Save your sketch.

An example:

/*
 * Hello World Arduino sketch.
 *
 * This sketch depends on the libraries and settings described below. You can either manually download
 * them via the Library Manager and select the needed options by hand, or do all of the above
 * via the "Tools -> Drizzle -> Apply Markers" menu command.
 *
 * @BoardManager esp8266::^2.6.3::https://arduino.esp8266.com/stable/package_esp8266com_index.json
 * @Board esp8266::esp8266::NodeMCU 1.0 (ESP-12E Module)
 * @BoardSettings esp8266::NodeMCU 1.0 (ESP-12E Module)::Flash Frequency->40MHz||Flash Mode->QIO
 *
 * @DependsOn Arduino_CRC32::1.0.0
 * @DependsOn Arduino Cloud Provider Examples::*
 * @DependsOn BMP280_DEV::(>= 1.0.8 && < 1.0.16)
 *
 * @Preferences esp8266::esp8266::NodeMCU 1.0 (ESP-12E Module)::compiler.cpp.extra_flags=-std=gnu++14 -DTEST123
 *
 * @ArduinoTool Drizzle::(<0.16.1)::https://github.com/zhgzhg/Drizzle/releases/download/0.16.1/drizzle-0.16.1-dist.zip
 */

// Your sample code follows below just as usual

void setup() {
  pinMode(LED_BUILTIN, OUTPUT);
}

void loop() {
  digitalWrite(LED_BUILTIN, HIGH);
  delay(1000);
  digitalWrite(LED_BUILTIN, LOW);
  delay(1000);
}

Like that the code self-explains its dependencies, so anyone interested in compiling it can do that with 2 clicks:

• Tools -> Drizzle -> Apply Markers menu
• Verify / Upload button

Using and combining Drizzle's markers is always optional.

Alternative Usage

The drizzle.json file can be used instead of the annotated comments in the main sketch file, if it's placed next to it, for describing dependencies and settings. In this case drizzle.json becomes the only settings/dependencies source, and any Drizzle markers found in the source code will be ignored.

An example for drizzle.json can read in the CLI Extras section.

How to Install

1. Download the "-dist" ZIP file from the Releases' assets.
2. Unzip it inside Arduino's installation directory / tools
3. Restart Arduino IDE

How to Update

When Arduino IDE is opened, Drizzle will automatically check for newer version, and will offer a semi-automatic update process - once downloaded, the IDE will be closed. The next time you open it, the latest version of Drizzle will be fully installed.

For a manual update follow the steps in How to Install .

Supported Markers

• @BoardManager platform_name::platform_version[::url_of_the_board_manager_json]

  • Install platforms a.k.a boards
  • Respects the first valid marker that's found. The rest will be ignored.
  • Examples:
    • @BoardManager Arduino AVR Boards::1.6.4
    • @BoardManager esp8266::^2.6.3::https://arduino.esp8266.com/stable/package_esp8266com_index.json
    • @BoardManager esp32::>1.0.3::https://dl.espressif.com/dl/package_esp32_index.json
    • @BoardManager Raspberry Pi Pico/RP2040::~1.6::https://github.com/earlephilhower/arduino-pico/releases/download/global/package_rp2040_index.json

• @Board [provider_package_name::]platform_name::board_id | board_name

  • Selects the id/name of the target board. The ID matching is with priority!
  • Respects the first valid marker that's found. The rest will be ignored.
  • Not specifying provider package name would make Drizzle to attempt resolving it automatically.
  • Examples:
    • @Board arduino::Arduino AVR Boards::Arduino Nano
    • @Board esp8266::NodeMCU 1.0 (ESP-12E Module)
    • @Board esp32::ESP32 Wrover Module
    • @Board esp32::esp32::esp32wrover
    • @Board rp2040::Raspberry Pi Pico/RP2040::Raspberry Pi Pico

• @DependsOn library_name::(library_version | library_uri)

  • Downloads libraries or installs local ones.
  • More than 1 marker can be used.
  • library_name must be unique. In case of duplicated library names, the first one will be respected.
  • The version can be either specified directly or via conditional expressions.
    • Full list of the supported expressions at: https://github.com/npm/node-semver#ranges
    • In case library_uri is specified instead of library_version:
      • library_name must be unique, but is not required to precisely match the actual library name
      • HTTP or HTTPS protocol is supported for .ZIP files directly specified in the URI, or GIT repositories
      • HTTP(S) URLs for git can end with #tag_or_commit_or_branch_name reference - see the examples below
      • file:/// prefix can be used to point to local directory containing the library, or to a concrete .ZIP file
  • To achieve better control any transitive dependencies won't be automatically installed, but will be listed in the logs.
  • Examples:
    • @DependsOn Arduino_CRC32::1.0.0
    • @DependsOn Arduino Cloud Provider Examples::*
    • @DependsOn BMP280_DEV::(>= 1.0.8 && < 1.0.16)
  • Examples with files:
    • @DependsOn Local BMP280_DEV::file:///C:/Users/John/Desktop/BMP280_DEV-1.0.15.zip
    • @DependsOn Local BMP280_DEV::file:///C:/Users/John/Desktop/BMP280_DEV_DIRECTORY
    • @DependsOn Github's BMP280_DEV::https://github.com/MartinL1/BMP280_DEV/archive/master.zip
    • @DependsOn Github's BMP280_DEV::https://github.com/MartinL1/BMP280_DEV.git
    • @DependsOn Github's BMP280_DEV::https://github.com/MartinL1/BMP280_DEV.git#V1.0.21

• @BoardSettings platform_name::board_id | board_name::menu_path[->option][||another_menu_path->option...]

  • Clicks on the UI options provided by a particular board (and platform)
  • Board ID or name can be used. Board ID matching is attempted first.
  • More than 1 marker can be used.
  • To match all platforms and/or board ids or board names a * can be used
  • To describe the path to the particular option -> can be used
  • To separate multiple menu paths || can be used
  • Menu matching is case-sensitive
  • Menu matching will be performed in the order of definition and will stop immediately once a match is found. Always define the concrete rules in the beginning, and the less concrete at the end.
  • Examples:
    • @BoardSettings esp32::esp32wrover::Flash Frequency->40MHz
    • @BoardSettings esp32::ESP32 Wrover Module::Flash Frequency->40MHz
    • @BoardSettings esp32::*::Flash Frequency->40MHz||PSRAM->Disabled
    • @BoardSettings *::*::Upload Speed->115200

• @Preferences [provider_package_name::]platform_name::board_id | board_name::definition=value[||another_definition=another_value...]

  • At compile time automatically appends arbitrary runtime preference definitions
  • Board ID or name can be used. Board ID matching is attempted first.
  • More than 1 marker can be used.
  • All the markers are attempted to be applied while matching against the current selected board.
  • To match all provider package, platforms and/or board ids or board names a * can be used
  • The matching is case-sensitive
  • The matching will be performed in the order of definition, from left to right, and will stop immediately once a match is found. Always define the concrete rules in the beginning, and the less concrete at the end.
  • Examples:
    • @Preferences esp32::esp32::esp32wrover::compiler.cpp.extra_flags=-std=gnu++17 -DMYDEF||compiler.c.extra_flags=-DMYDEFINITION2
    • @Preferences *::esp32::*::compiler.cpp.extra_flags=-std=gnu++14
    • @Preferences *::*::compiler.cpp.extra_flags=-std=gnu++14

• @ArduinoTool tool_name::version::tool_zip_url

  • Installs an Arduino tool from ZIP archive into the IDE. The tool is unzipped inside Arduino IDE's installation directory / tools, and will require restarting the IDE in order to be activated.
  • This marker works a bit different compared to @DependsOn. It performs supervising work rather than actual dependency version management. It should be used with caution as it has the potential of installing unwanted software.
  • The version field serves as a condition when tool_zip_url to be installed.
  • Version information of the already installed tool is obtained from the meta information inside the JAR file. If it doesn't contain MANIFEST.MF or it's lacking an Implementation-Version entry no actions will be taken. Otherwise, Drizzle will attempt installing the tool from tool_zip_url.
  • The tool_name must match the name of the directory containing the actual tool. It has to be unique. In the case of several duplicating names the first one will be respected.
  • Examples:
    • @ArduinoTool Drizzle::(<0.16.1)::https://github.com/zhgzhg/Drizzle/releases/download/0.16.1/drizzle-0.16.1-dist.zip
    • @ArduinoTool Drizzle::*::file:///C:/Users/John/Drizzle/drizzle.zip
    • @ArduinoTool EspExceptionDecoder::(<=1.0.0)::https://github.com/me-no-dev/EspExceptionDecoder/releases/download/2.0.2/EspExceptionDecoder-2.0.2.zip

CLI Extras

Drizzle offers CLI parsing of any Arduino sketch file, printing the recognized marker settings in JSON format. The reverse operation, where from JSON file Drizzle markers will be produced is also supported.

For e.g. java -jar drizzle-0.16.1-with-deps.jar --parse hello-world.ino will produce:

{
  "board_manager": {
    "platform": "esp8266",
    "version": "^2.6.3",
    "url": "https://arduino.esp8266.com/stable/package_esp8266com_index.json"
  },
  "board": {
    "providerPackage": "esp8266",
    "platform": "esp8266",
    "name": "NodeMCU 1.0 (ESP-12E Module)"
  },
  "board_settings": [
    {
      "board": {
        "platform": "esp8266",
        "name": "NodeMCU 1.0 (ESP-12E Module)"
      },
      "clickable_options": [
        [
          "Flash Frequency",
          "40MHz"
        ],
        [
          "Flash Mode",
          "QIO"
        ]
      ]
    }
  ],
  "libraries": {
    "BMP280_DEV": {
      "version": "(>= 1.0.8 && < 1.0.16)",
      "arduinoCliFmt": "BMP280_DEV@1.0.8"
    },
    "Arduino_CRC32": {
      "version": "1.0.0",
      "arduinoCliFmt": "Arduino_CRC32@1.0.0"
    },
    "Arduino Cloud Provider Examples": {
      "version": "*",
      "arduinoCliFmt": "Arduino Cloud Provider Examples"
    }
  },
  "preferences": [
    {
      "board": {
        "providerPackage": "esp8266",
        "platform": "esp8266",
        "name": "NodeMCU 1.0 (ESP-12E Module)"
      },
      "preferences": {
        "compiler.cpp.extra_flags": "-std=gnu++14 -DTEST123"
      }
    }
  ],
  "arduino_ide_tools": {
    "Drizzle": {
      "version": "(<0.16.1)",
      "url": "https://github.com/zhgzhg/Drizzle/releases/download/0.16.1/drizzle-0.16.1-dist.zip"
    }
  }
}

Executing on the above JSON java -jar drizzle-0.16.1-with-deps.jar --rev-parse hello-world.json will produce:

@BoardManager esp8266::^2.6.3::https://arduino.esp8266.com/stable/package_esp8266com_index.json
@Board esp8266::esp8266::NodeMCU 1.0 (ESP-12E Module)
@BoardSettings esp8266::NodeMCU 1.0 (ESP-12E Module)::Flash Frequency->40MHz||Flash Mode->QIO
@DependsOn BMP280_DEV::(>= 1.0.8 && < 1.0.16)
@DependsOn Arduino_CRC32::1.0.0
@DependsOn Arduino Cloud Provider Examples::*
@Preferences esp8266::esp8266::NodeMCU 1.0 (ESP-12E Module)::compiler.cpp.extra_flags=-std=gnu++14 -DTEST123
@ArduinoTool Drizzle::(<0.16.1)::https://github.com/zhgzhg/Drizzle/releases/download/0.16.1/drizzle-0.16.1-dist.zip

Using in Automated CI/CD Environments

With the help Drizzle's JSON output, jq, and adruino-cli (or its precessor arduino-builder) an automatic compilation from the command line can be achieved. A couple of examples are given below:

Assuming the execution happens inside arduinoci/ci-arduino-cli container.

Install Drizzle:

DRIZZLE_VER="0.16.1"

set -x
if [[ ! -f "Drizzle/tool/drizzle-$DRIZZLE_VER-with-deps.jar" ]]; then
  apt update && apt -y install wget unzip jq openjdk-17-jdk-headless
  rm -fr Drizzle
  wget https://github.com/zhgzhg/Drizzle/releases/download/$DRIZZLE_VER/drizzle-$DRIZZLE_VER-dist.zip
  unzip drizzle-$DRIZZLE_VER-dist.zip
fi

Produce Drizzle JSON file:

java -jar Drizzle/tool/drizzle-$DRIZZLE_VER-with-deps.jar --parse ./my-project >my-project.json

Install custom boards from URL, update indexes:

arduino-cli core update-index --additional-urls $(jq -r .board_manager.url ./my-project.json)
arduino-cli update

Extract FQBN, install libraries, extract compile preferences, and run a compilation:

set -x

ARDUINO_LIBRARY_ENABLE_UNSAFE_INSTALL="true" # allows installation from external HTTPS / GIT locations

FQBN=$(jq -r '[.board.providerPackage,.board.platform,.board.name] | "\(.[0]):\(.[1]):\(.[2])"' ./my-project.json)

PP=$(jq -r '[.board.providerPackage] | "\(.[0])"' ./my-project.json)
PF=$(jq -r '[.board.platform] | "\(.[0])"' ./my-project.json)
BN=$(jq -r '[.board.name] | "\(.[0])"' ./my-project.json)

while IFS= read -r lib ;
do
  arduino-cli lib install -v $lib
done <<< $(jq -r '.libraries | map(.arduinoCliFmt) | .[]' ./my-project.json)
BPS=$(jq -r 'first(.preferences | to_entries | .[] | select(((.value.board.providerPackage == "'"$PP"'" and .value.board.platform == "'"$PF"'") or (.value.board.providerPackage == "'*'" and .value.board.platform == "'*'")) and .value.board.name == "'"$BN"'")) | .value.preferences | to_entries | map("--build-property \"" + .key + "=" + .value + "\"") | join(" ")' ./my-project.json)
eval "BPSE=($BPS)"


arduino-cli compile -v --fqbn "$FQBN" "${BPSE[@]}" ./my-project

How To Compile

1. mvnw com.googlecode.maven-download-plugin:download-maven-plugin:wget@install-arduino-libs
2. mvnw clean package