denoflow / denoflow

Configuration as Code, use YAML to write automated workflows that run on Deno, with any Deno modules, Typescript/Javascript codes

Denoflow

简体中文文档 · Deno Land

Table of Contents

• About
• Getting Started
• Usage
• Faq
• Contributing

About

With denoflow, you'll use yaml to write automated workflow that runs on Deno, with any Deno modules, Typescript/Javascript code in your workflow. Denoflow will execute your workflow as you want. You can think of it as configuration-as-code.

Consider it an alternative to Zapier or IFTTT, for example fetching articles at regular intervals via RSS and then performing some tasks for non-repeating articles.

It's still at a very early stage, use with care!

If we need some GUI features, the ideal result is having some app schemas, using rjsf to render a web gui, gui can help us to generate yaml configs.

The ideal runtime is using cloud serverless platform, or CI platform, like Github Actions, Gitlab CI, self-hosted with Deno, or any Docker runtime.

Deno Deploy is not supported yet, because it doesn't support Code generation from strings, Error Detail: Code generation from strings disallowed for this context See Github Actions Example: test.yml

Now we can only write yaml by ourself, and actually, it's not that hard.

Join our Discord chat channel to discuss about Denoflow!

Stable deno land version see Denoflow

Playground

Try and exploring denoflow with the Online Playground

Why Deno?

• Deno uses URLs to import modules, which gives us very flexible and convenient workflow scripting capabilities, and you can use any module you want very easily.
• Deno is based on modern features of the JavaScript language, and officially provides a standard library that covers most of the commonly used features in workflows.
• Deno's default zero permissions design, which is important for workflow security, so that you can give your workflows only the permissions they need.

I have written actionsflow before, it must run in github actions, or local Docker, for workflow is too heavy, I found Deno features make it more suitable for doing flexible workflow based on yaml configuration, hope Denoflow can become a simple but powerful workflow assistant.

Prerequisites

Install Deno first.

Getting Started

Fetch from Hacker News API to webhook example:

mkdir workflows

touch workflows/fetch.yml

sources:
  - from: https://deno.land/x/axiod@0.24/mod.ts
    use: get
    args:
      - https://test.owenyoung.com/slim.json
    itemsPath: data.hits
    key: objectID
    limit: 1
steps: 
  - run: console.log('item', ctx.item)
  # Open: <https://requestbin.com/r/enyvb91j5zjv9/23eNPamD4DK4YK1rfEB1FAQOKIj> , See live webhook request.
  - from: https://deno.land/x/axiod@0.24/mod.ts
    use: post
    args:
      - https://enyvb91j5zjv9.x.pipedream.net/
      -  ${{ctx.item}}
      - headers:
          'Content-Type': 'application/json'

Open: https://requestbin.com/r/enyvb91j5zjv9/23eNPamD4DK4YK1rfEB1FAQOKIj , See live webhook request.

deno run --allow-read --allow-net --allow-write --allow-env --allow-run https://deno.land/x/denoflow/cli.ts run

Or simplly with all permissions: deno run -A https://deno.land/x/denoflow/cli.ts run

Denoflow will scan the workflows directory and run all valid .yml files.

latest version: https://denopkg.com/denoflow/denoflow@main/cli.ts

Try it at Online Playground

If you prefer to use fetch:

sources:
  - use: fetch
    args:
      - https://test.owenyoung.com/slim.json
    run: return ctx.result.json()
    itemsPath: hits
    key: objectID
    limit: 1
steps: 
  - use: fetch
    args:
      - https://enyvb91j5zjv9.x.pipedream.net/
      - method: POST
        headers:
          Content-Type: application/json
        body: ${{JSON.stringify(ctx.item)}}

Try it at Online Playground

RSS Feed to Discord Webhook Message

touch workflows/rss.yml

sources:
  - from: https://deno.land/x/denoflow@0.0.19/sources/rss.ts
    args:
      - https://actionsflow.github.io/test-page/hn-rss.xml
    limit: 1
steps:
  - use: fetch
    args:
      - ${{env.DISCORD_WEBHOOK}}
      - method: POST
        headers:
          Content-Type: application/json
        body: ${{ JSON.stringify({content:ctx.item.title.value}) }}

Try it at Online Playground

Or, if you prefer more raw way:

sources:
  - use: fetch
    args:
      - https://actionsflow.github.io/test-page/hn-rss.xml
    run: |
      const rss = await import("https://deno.land/x/rss/mod.ts");
      const xml = await ctx.result.text();
      const feed = await rss.parseFeed(xml);
      return feed.entries;
    limit: 1
steps:
  - use: fetch
    args:
      - ${{ctx.env.DISCORD_WEBHOOK}}
      - method: POST
        headers:
          'Content-Type': 'application/json'
        body: ${{ JSON.stringify({content:ctx.item.title.value}) }}

deno run --allow-read --allow-net --allow-write --allow-run --allow-env https://deno.land/x/denoflow/cli.ts run

Try it at Online Playground

A simple script generated exmaple:

sources:
  - run: return [{id:"1"}]
    force: true
steps: 
  - run: console.log("item",ctx.item);

Try it at Online Playground

More examples are in workflows directory, you can submit your awesome workflows.

Try Online Playground to explore workflow.

Life Cycle

1. sources?: where to fetch the data, Source[], can be one or more sources. Every source should return an array of items. e.g. [{"id":"1"}], the item key can be specified in key field.
   1. from?: import ts/js script from url or file path
   2. use?: run moduleName from above from , or if from is not provided, run global function like fetch, args will be passed to the function, the return value will be attached to ctx.result and ctx.sources[index].result , if use is a class, then ctx.result will be the instance of the class. use can also be Deno.cwd things, to call Deno functions.
   3. run?: run ts/js code, you can handle use result here. Return a result that can be stringified to json. The return value will be attached to ctx.result and ctx.sources[index].result
   4. itemsPath?: the path to the items in the result, like hits in https://test.owenyoung.com/slim.json
   5. key?: the key to identify the item, like objectID in https://test.owenyoung.com/slim.json, if not provided, will use id, denoflow will hash the id, then the same item with id will be skipped.
   6. reverse?, boolean, reverse the items
   7. filter?, string, script code, should handle ctx.item -> return true or false
   8. cmd: string, exec a shell command after all other task, the return value will be attached to ctx.cmdResult and ctx.sources[index].cmdResult
   9. post?: post script code, you can do some check, clean, things here, change ctx.state
2. filter? filter from all sources items, handle ctx.items, expected return a new boolean[],
   1. from?: import ts/js script from url or file path
   2. use?: run moduleName from above from , or if from is not provided, run global function like fetch, args will be passed to the function, the return value will be attached to ctx.result and ctx.sources[index].result , if use is a class, then ctx.result will be the instance of the class. use can also be Deno.cwd things, to call Deno functions.
   3. run?: run ts/js code, you can handle use result here.handle ctx.items, expected return a new boolean[], flag which item will be used. e.g. run: return ctx.items.map(item => item.title.value.includes('test'))
   4. cmd?: string, exec a shell command after all other task, the return value will be attached to ctx.cmdResult and filter.cmdResult
   5. post?: post script code, you can do some check, clean, things here, change ctx.state
3. steps? the steps to run, Step[], can be one or more steps.
   1. from?: import script from url or file path
   2. use?: run moduleName from above from , or if from is not provided, run global function like fetch, args will be passed to the function, the return value will be attached to ctx.result and ctx.sources[index].result , if use is a class, then ctx.result will be the instance of the class. use can also be Deno.cwd things, to call Deno functions.
   3. run?: run ts/js code, you can handle use result here. Return a result that can be stringified to json. the result will be attached to the ctx.steps[index].result
   4. cmd?: exec shell commands, will be run after run, the result will be attached to the ctx.steps[index].cmdResult
   5. post?: post script code, you can do some check, clean, things here, change ctx.state
4. post? final post script code, run after all steps done, you can do some check, clean, things here. You can use all steps params here.

Installing

deno install -n denoflow --allow-read --allow-net --allow-write --allow-run --allow-env  https://deno.land/x/denoflow/cli.ts

Then, you can run it with denoflow run , or denoflow run <files>

Update to latest version

deno cache --reload https://deno.land/x/denoflow/cli.ts

Usage

denoflow/0.0.17

Usage:
  $ denoflow run [...files or url]

Options:
  --force     Force run workflow files, if true, will ignore to read/save state
  --debug     Debug mode, will print more info
  --database  Database uri, default json://data
  --limit     max items for workflow every runs
  --sleep     sleep time between sources, filter, steps, unit seconds
  --stdin     read yaml file from stdin, e.g. cat test.yml | denoflow run --stdin
  -h, --help  Display this message

YAML Syntax

If you are not yet familiar with YAML syntax, take 5 minutes to familiarize yourself with.

You can use ${{variable}} in any fields to inject variables into your workflow, we inject ctx variable in template and script. For example:

Expressions

steps:
  - if: ${{ctx.items.lengh>10}}
    run: console.log(ctx.item);

All ctx variables, See Interface

State

You can simply use ctx.state to get or set state, for example:

let currentState = ctx.state || {};

let sent = ctx.state.sent || [];

if(sent.includes(ctx.item.id)){
  sent.push(ctx.item.id);
}

ctx.state = {
  sent
};

// deno flow will save the state for you , next you can read it.
return;

The state will be saved to data folder in json format. You can also use sqlite to store the state. Just set database: sqlite://data.sqlite in your workflow config file.

Faq

How to schedule a workflow?

Cause denoflow designed for serverless, simple, so it self can't schedule a workflow. You can use cron or other trigger to schedule a workflow. For example:

*/15 * * * * deno run --allow-read --allow-net --allow-write --allow-env --allow-run https://deno.land/x/denoflow/cli.ts run workflows/schedule15.yml

How to handle a webhook?

Like above, denoflow can't handle webhook directly, you can forward the webhook to denoflow, For github actions example:

Webhook.yml:

sources:
  - run: return [ctx.env.event]
    force: true
steps: 
  - run: console.log("item",ctx.item);

.github/workflows/webhook.yml:

name: Webhook
on:
  repository_dispatch:
  workflow_dispatch:
jobs:
  denoflow:
    runs-on: ubuntu-latest
    concurrency: denoflow
    steps:
      - name: Check out repository code
        uses: actions/checkout@v2
      - uses: denoland/setup-deno@v1
        with:
          deno-version: v1.x
      - env:
          event: ${{toJSON(github.event)}}
        run: deno run --allow-read --allow-net --allow-write --allow-env --allow-run https://deno.land/x/denoflow/cli.ts run workflows/webhook.yml

Why configuration is better than script?

If you write scripts directly, it will make maintenance very confusing, each script has to handle its own de-duplication and process management, using configuration files to write more readable, maintainable, to achieve a certain degree of low code.

Todo

• Support Sleep Option
• Support GUI generated workflow
• Support clean command
• denoflow playground
• support new instance