Giters

LightAndLight / ipso

A functional scripting language.

Home Page:https://ipso.dev

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Translate `uploadToCache[Macos]` to ipso

LightAndLight opened this issue · comments

commented

Branch: https://github.com/LightAndLight/ipso/tree/271-translate-uploadtocachemacos-to-ipso

Files:

#! /usr/bin/env ipso

binaryCacheBucket : String
binaryCacheBucket = "ipso-binary-cache"

binaryCacheEndpoint : String
binaryCacheEndpoint = "7065dc7f7d1813a29036535b4c4f4014.r2.cloudflarestorage.com"

main : IO ()
main =
  comp
    bind mOutPaths <- env.getvar "OUT_PATHS"
    bind outPaths <-
      case mOutPaths of
        None x ->
          comp
            println "missing environment variable: OUT_PATHS"
            exit.failure
        Some outPaths ->
          io.pure outPaths
    
    println "Uploading paths $outPaths"
    let binaryCacheUrl = "s3://$binaryCacheBucket?scheme=https&endpoint=$binaryCacheEndpoint&secret-key=/run/nix-signing-key"
    cmd.run `/nix/var/nix/profiles/default/bin/nix copy --to $binaryCacheUrl ${string.partsc ' ' outPaths}`

Tasks

  • Measure the performance difference between bash and ipso
  • Implement if ipso is as fast as bash
commented

I compared the two versions of uploadToCache with no environment variables set. callgrind's instruction count reported ~2 million for ipso and ~7.7 million for bash.

The callgrind report shows that bash spends most of its time (~6.4 million instructions) inside mblen, which seems to parse some locale files (/nix/store/scd5n7xsn0hh0lvhhnycr9gx0h8xfzsl-glibc-2.34-210/lib/gconv/gconv-modules.d/gconv-modules-extra.conf). The slow bash function is here.

If we ignore the cost of set_default_locale, we get ~1.3 million instructions for the rest of the bash program.

commented

When running with OUT_PATHS="aaa", I get ~2.5 million instructions for ipso and ~9.9 million instructions for bash.

set_default_locale still dominates bash's execution, at ~6.4 million instructions. Subtracting this we get ~3.5 million instructions for bash.

commented

For reference, the Python version has ~89 million instructions:

#! /usr/bin/env python3

import os
import subprocess

BINARY_CACHE_BUCKET = "ipso-binary-cache"
BINARY_CACHE_ENDPOINT = "7065dc7f7d1813a29036535b4c4f4014.r2.cloudflarestorage.com"

OUT_PATHS = os.environ["OUT_PATHS"]

print("Uploading paths", OUT_PATHS)
subprocess.run([
    "nix",
    "copy",
    "--to",
    f"s3://{BINARY_CACHE_BUCKET}?scheme=https&endpoint={BINARY_CACHE_ENDPOINT}&secret-key=/run/nix-signing-key",
    *OUT_PATHS.split(' ')
    ],
    check=True
)
commented

Blocked until I can find a nice way to provide ipso to the post-build-hook script. The Nix daemon runs the script and doesn't have easy access to a Nix-installed version of ipso. Example failure: https://github.com/LightAndLight/ipso/actions/runs/3501094293/jobs/5864384130#step:9:88.

commented

After using pueue to queue Nix cache uploads (#291), it was easy to migrate the scripts. I installed ipso in the runner's user, and because pueue is also running in the runner's user, it has access to the Nix-installed program.