dk solves the problem of README-itis: you give your users a long README document, your users fail to setup your software, and you lose a user forever.
If you have a Linux application you could choose nix or Docker, but outside of that domain you have limited options. dk reliably builds software comparable to nix and Docker, but is designed for non-techie end-users (esp. Windows users) and has a few security controls and a spec for easy adoption into language build tools.
We'll use the software behind dk itself as an example of easy-to-fail, average-complexity software. dk's scripting functionality requires a runtime environment and platform development kits are downloaded on-demand. We want our users (ex. you) to install and get started quickly. Users copy-and-paste the text blocks below on Windows. Go ahead and copy-and-paste yourself if you want to try the dk cross-compiling scripting environment, or skip past it to learn about the dk build tool:
# Install a standalone executable. And available for macOS and Linux.
$ winget install -e --id Diskuv.dk# THIS EXAMPLE: The `dk` software stack has scripting.
# Here's a script to download and print a page to the screen.
# IN GENERAL: Your users copy-and-paste your first example ...
$ dk -S "
module Uri = Tr1Uri_Std.Uri
" -U "
Tr1Stdlib_V414Io.StdIo.print_endline @@
Lwt_main.run @@
DkNet_Std.Http.fetch_url ~max_sz:4096 @@
Uri.of_string {|https://jigsaw.w3.org/HTTP/h-content-md5.html|}
" -O ReleaseSmall Exe
... THIS EXAMPLE: `dk` will install a scripting environment, download
... development kits on-demand for different platforms, and do a
... cross-compile. It *takes time* to download massive development kits.
... You (and your user) will do something else in another window, but as
... long as you don't have to type anything, you'll probably stick around!
... ⟳ ⟳ ⟳ software installed automatically ⟳ ⟳ ⟳
... ⟳ ⟳ ⟳ example executed ⟳ ⟳ ⟳
# THIS EXAMPLE: `dk` scripting makes standalone executables.
# IN GENERAL: Your users get the results they want ...
$ file target/ZzZz_Zz.Adhoc-*
target/ZzZz_Zz.Adhoc-android_arm32v7a: ELF 32-bit LSB pie executabl...
target/ZzZz_Zz.Adhoc-android_arm64v8a: ELF 64-bit LSB pie executabl...
target/ZzZz_Zz.Adhoc-android_x86_64: ELF 64-bit LSB pie executabl...
target/ZzZz_Zz.Adhoc-darwin_arm64: Mach-O 64-bit executable arm...
target/ZzZz_Zz.Adhoc-darwin_x86_64: Mach-O 64-bit executable x86...
target/ZzZz_Zz.Adhoc-linux_x86: ELF 32-bit LSB pie executabl...
target/ZzZz_Zz.Adhoc-linux_x86_64: ELF 64-bit LSB pie executabl...
target/ZzZz_Zz.Adhoc-windows_x86_64.exe: PE32+ executable (console) x...
target/ZzZz_Zz.Adhoc-windows_x86.exe: PE32 executable (console) In...You still need that long README for your complex software, but your software setup is boring and in your users' hands quickly.
dk solves README-itis in two ways:
- You model your actions (all that stuff you would put into a README) with scripts that
dkwill cross-compile for your users' platforms. - All required actions are executed as needed on your users' machines with dk's build tool.
Keep your setup boring: no interesting knobs, and no typing beyond the initial copy-and-paste.
Skip down to Comparisons for how dk fits with other tools.
The build tool is quite new and has not yet been integrated into the script runner. But it has a dk0/mlfront-shell reference implementation which are documented in the next sections, and specifications are at docs/SPECIFICATION.md.
Separately, a Quick Start for Scripting is further below, and the main documentation site for the script runner is https://diskuv.com/dk/help/latest/.
But we'll start with a walk-through of the build tool by unpackaging the popular zip compression software "7zip". (dk has no affliation with 7zip.)
- dk - A terribly uninteresting build and scripting system
- Introduction
- Concepts and Theory
- Using the Build Tool to create a multi-platform package
- 7zip assets
- 7zip extraction plan
- 7zip step 1 - download 7zr.exe
- 7zip step 1b - hide 7zr.exe assets
- 7zip step 1c - add all assets
- 7zip step 2 - extract 7z.exe from Windows installers
- 7zip step 3 - uncompress Unix tar from Unix compressed archives
- 7zip step 4 - extract 7zz from Unix tar
- 7zip - package interface
- 7zip - building in GitHub Actions
- 7zip - summary
- Using the Build Tool to debug a failed form
- Quick Start - Scripting
- Comparisons
- Licenses
- Open-Source
In the dk build system, you submit forms that produce objects created from assets.
The assets are input materials. These are files and folders, and they can be local or remote: source code, data files, audio, image and video files.
A form is a document with fields and a submit button. At some point ... not now ... there will be a graphical user interface (GUI) for these forms. But the command line interface (CLI) fallback is available today and forever: you use the DOS or Unix terminal, and the document is a command line in your terminal.
An object is a folder that the form produces.
Again: You submit forms that produce objects created from assets.
Finally, we introduce our fourth (4th) word: value. A value is any asset, form or object.
Let's relate these concepts to a build activity: our task is to unpack a 7zip executable (7zz on Unix or 7z.exe on Windows) from official, platform-specific 7zip installers.
This is not a toy problem! We would need the 7zip executable if our second task were to unpack a git executable from a Portable Git Installer.
For unpacking the 7zip executables, we'll submit a form several times (each time with different parameters):
$ git clone https://github.com/diskuv/dk.git dk0
Cloning into 'dk0'...
$ dk0/mlfront-shell -I dk0/pkgs/include -- get-object 'CommonsZip_Std.S7z.S7zExe@25.1.0' -s File.Darwin_x86_64 -m ./7zz.exe -f target/Darwin_x86_64.7zz.exe
[up-to-date] CommonsZip_Std.S7z.S7zExe@25.1.0+bn-20250101000000 -s File.Darwin_x86_64which builds the desired files into our desired "target" directory:
Directory: Y:\a\target
Mode LastWriteTime Length Name
---- ------------- ------ ----
-a---- 9/17/2025 11:36 AM 5885696 Darwin_arm64.7zz.exe
-a---- 9/17/2025 11:36 AM 5885696 Darwin_x86_64.7zz.exe
-a---- 9/17/2025 11:36 AM 1548480 Linux_arm.7zz.exe
-a---- 9/17/2025 11:36 AM 2456896 Linux_arm64.7zz.exe
-a---- 9/17/2025 11:36 AM 3192444 Linux_x86.7zz.exe
-a---- 9/17/2025 11:36 AM 2878000 Linux_x86_64.7zz.exe
-a---- 9/17/2025 11:36 AM 575488 Windows_x86.7zz.exe
-a---- 9/17/2025 11:36 AM 575488 Windows_x86_64.7zz.exe
so you can see the concepts in action:
| Concept | Examples |
|---|---|
| Assets |  |
| Submitting Forms |  |
| Generated Objects | Y:\a\target\pid\500\bnrtauvvz5kvhwd5\out\File.Windows_x86: |
Y:\a\target\pid\500\wtqhxi4flrfk6sab\out\File.Darwin_arm64: |
The objects you see above are intermediate folders, so this might be the first and only time you see them. Normally you pull files out of objects into your own target directories, like was done when we submitted the form with dk0/mlfront-shell ... -f target/Darwin_x86_64.7zz.exe.
Values have unique identifiers. We used one when we submitted the form: CommonsZip_Std.S7z.S7zExe@25.1.0.
These identifiers contain versions like 25.1.0. Making a change to a value means creating a new value with the same name but with an increased version. For example, if the text of your 2025-09-04 privacy policy is in the asset YourOrg_Std.StringsForWebSiteAndPrograms.PrivacyPolicy@1.0.20250904, an end-of-year update to the privacy policy could be YourOrg_Std.StringsForWebSiteAndPrograms.PrivacyPolicy@1.0.20251231. These semantic versions offer a lot of flexibility and are industry-standard: external link: semver 2.0. The important point is that values do not change; versions do.
The direct inspiration for dk is a little known build system called gg by Stanford. The paper is
"From Laptop to Lambda: Outsourcing Everyday Jobs to Thousands of Transient Functional Containers," in 2019 USENIX Annual Technical Conference (USENIX ATC '19).
The implementation of dk is fairly faithful to the Build systems à la carte: Theory and practice paper. dk models the Cloud Shake build system from that paper: constructive traces with a suspending scheduler. The reference implementation was initially a direct translation of the paper; see https://discuss.ocaml.org/t/ocaml-version-of-the-build-systems-a-la-carte-paper/17042 if you want to implement your dk compatible build system using the reference implementation code.
At core dk is a build tool and a package manager. At the end of the day dk needs to compile arbitrary code, execute arbitrary code, and place that code in (mostly) arbitrary locations on a file system.
But there is a core dk goal: don't be stupid.
The don't be stupid goal, combined with avoiding README-itis, are the reasons for the following security controls:
-
SHA-256 checksums on all assets, including source code, except local file assets may optionally have SHA-1 checksums. SHA-1 checksums provide interoperability with tools like
gitand Meta'swatchman. With OpenBSD signify signing of the SHA-256 checksums enforced for thedkregistry (the 3rd control), we gain an assertion that a trusted human authored a source code change. That will not stop social engineering attacks like the xz backdoor, but it is an improvement over blindly trusting git commmits. -
No tiny packages; that improves asset quality and eases auditing. Initially it may be that you can't submit a package whose binary size is under 1MB.
-
OpenBSD signify-based keys to identify software. The reference implementation generates build keys on first build, and they can be cached in CI if you trust your CI vendor:
$ dk0/mlfront-shell ... [signify] New build key pair in xxx/build.pub and xxx/build.sec ... [signify] Distribute key pair among trusted coworkers only!
The reference implementation does allow sharing data between machines, and only uses the build key pair to stop sharing the following critical data:
- parsed abstract syntax trees. While avoid parsing costs is important for performance, direct object-to-byte serializations like OCaml's Marshal, Python's pickle, etc. are vulnerable to injection attacks. Parsed ASTs are signed with a build key; any parsed AST that is not signed by your build key will be rejected and re-parsed.
The
dkregistry is vaporware at the moment, but the following three pieces will be linked together:- public key for a major.minor version (ex. 2.4.x)
- public key for the next major.minor version (ex. 2.5.x)
- a real name or organization (see next control)
The dual-key mechanism is used internally in
dkand documented at https://dkml.gitlab.io/build-tools/MlFront/MlFront_Signify/MlFront_Signify/Signify/index.html. -
Real names and countries to know who to trust. The simplest way in 2025 to do this is to require a micropayment (ex. $1/year). But to seed packages to spur adoption in the initial stages I (Jonah Beckford) will manually vet package authors.
NEXT STEPS: The next section goes over how to specify assets and forms, and how to submit forms.
Sorry macOS users, today the build tool
dk0/mlfront-shelldownloads an unsigned standalone binary. It will be signed later, oncedk0/mlfront-shellis merged intodk. Your mac probably won't like it. If you are adventurous, you can runxattr -d com.apple.quarantine ~/.local/share/mlfront-shell/mlfrontshellexe-2.4.*-darwin_arm64/mlfront-shelland try again.
We will work from scratch through a complete implementation of the 7zip package discussed in Concepts and Theory.
Let's start with the assets: .
The complete list is:
| Filename | Category |
|---|---|
7z2501-arm.exe |
Windows 7zip self-extracting installer |
7z2501-arm64.exe |
Windows 7zip self-extracting installer |
7z2501-extra.7z |
Extras |
7z2501-linux-arm.tar.xz |
Unix xz compressed archive |
7z2501-linux-arm64.tar.xz |
Unix xz compressed archive |
7z2501-linux-x64.tar.xz |
Unix xz compressed archive |
7z2501-linux-x86.tar.xz |
Unix xz compressed archive |
7z2501-mac.tar.xz |
Unix xz compressed archive |
7z2501-src.7z |
Unix xz compressed source archive |
7z2501-src.tar.xz |
Unix xz compressed archive |
7z2501-x64.exe |
Windows 7zip self-extracting installer |
7z2501-x64.msi |
Windows 7zip self-extracting installer |
7z2501.exe |
Windows 7zip self-extracting installer |
7z2501.msi |
Windows 7zip self-extracting installer |
7zr.exe |
Lightweight 7zip unzipper |
lzma2501.7z |
SDK |
All four (4) of the "Windows 7zip self-extracting installers" have 7z.exe inside.
And all six (6) of the "Unix xz compressed archives" have 7zz inside.
Both the four (4) Windows 7z.exe executables and the six (6) Unix 7zz executables have similar functionality.
To make it easy for other developers to use our package, we have decided on 7zz.exe as a common filename for both Windows and Unix. The .exe suffix is so that the file runs on Windows.
And now we need a plan to get 7z.exe and 7zz from the installers and compressed archives.
The file 7zr.exe can extract files from the four "Windows 7zip self-extracting installers". That means 7zr.exe is a dependency (requirement) to get the four (4) Windows 7z.exe executables out of the four "Windows 7zip self-extracting installers".
Getting 7zz is harder. The 7z.exe executables know how to uncompress the "Unix xz compressed archives". So with 7z.exe we can make 7z2501-linux-arm.tar.xz into 7z2501-linux-arm.tar. 7z.exe is a dependency of 7z2501-linux-arm.tar.
But 7z2501-linux-arm.tar is not enough; we need inside the tar archive. Good thing that 7z.exe also knows how to extract a member file from a .tar file. So with 7z.exe we can make 7z2501-linux-arm.tar into 7zz; that is, 7z.exe is a dependency of 7zz.
Whew!
Our plan is now concrete. We need the following five (5) steps.
- Download
7zr.exe. - Use
7zr.exeto extract7z.exefrom7z2501-arm.exeand7z2501-arm64.exe(etc.). - Use
7z.exeto uncompress7z2501-linux-x86.tarfrom7z2501-linux-x86.tar.xz(etc.). - Use
7z.exeto extract7zzfrom7z2501-linux-x86.tar(etc.). - Rename
7z.exeand7zzinto7zz.exe.
Unfortunately, using 7zr.exe or 7z.exe requires Windows hardware.
That means parts of the five step build plan must run on Windows hardware.
However, once the 7zip package has been built, all operating systems (Linux, macOS and Windows) can use the 7zip package. We'll see how as we enact the five step build plan.
We'll assume you are using VS Code. You can use other IDEs, but make sure you register the JSON schema so you have auto-complete.
With the dk build system, you use JSON build files. JSONC, an extension of JSON allowing /* ... */ and // ... end of line comments, is supported as well.
The JSON build file must either be named one of:
values.jsonvalues.jsonc*.values.json*.values.jsonc
We'll make a JSON file to download 7zr.exe.
Save the following file as 7zip-project/CommonsZip_Std.S7z1a.S7zr.values.jsonc.
Make a folder 7zip-project/ and save the file as 7zip-project/CommonsZip_Std.S7z.S7zr0.values.jsonc.
Some of JSON file is fairly straightforward, but let's go through six (6) fields that deserve mention:
-
The
$schemais a reserved field for VS Code to store the location of the JSON schema. You get auto-complete and documentation from the JSON schema. Use it! -
The
assets.listing_unencrypted.spec_versionmust be 2 unless there is a change to the specification. -
The
assets.listing_unencrypted.namefield gives part of a unique identifier for the7zr.exeasset file. It has syntax borrowed and hacked from the OCaml programming language.The first part of the name, before the first period, is the library identifier. For example,
CommonsZip_Stdis the library identifier forCommonsZip_Std.S7z.Assets. The library id visually has at least three bumps, with an underscore separating the second and third bump. It was designed to be visually recognizable (and recognizable from a lexer) while different enough from other identifiers that there was no accidental overlap. The following picture may help you remember theBumpBump_Bumpshape:
The parts after the first period, like
S7zandAssetsinCommonsZip_Std.S7z.Assetsare called namespace terms. They must start with a capital letter, and contain only the charactersA-Za-z0-9_. -
The
assets.listing_unencrypted.versionfield is not the original 7zip version "25.01". Instead,dkuses the semver 2.0 versioning specification. So "25.01" was translated into "25.1.0". -
The
assets.files.originfield must be the name of one of theassets.listing.origins. -
The
assets.files.checksum.sha256field should be the real hex-encoded checksum. But you may find it tedious figuring out the checksum. We have a better way; more on that later.
Now we are ready to run it. Let's review the command line options:
| Argument | What |
|---|---|
dk0/mlfront-shell |
The build tool. Eventually it will just be dk |
-I 7zip-project |
The folders containing *.values.jsonc files |
-x 7zip-org:subpath: |
Invalidate all files with the origin: "7zip-org" |
-- |
Separate dk0/mlfront-shell options from the command after |
get-asset-file |
Command to get the named asset file |
CommonsZip_Std.S7z1a.Assets@25.1.0 |
The name and version in .values.json |
-p 7zr.exe |
Identifies the file in files:[{path:...},...] |
-f target/7zr.exe |
Send command output into the file |
and with that we do:
# You should have done this already in earlier steps. If not, do it now:
# $ git clone https://github.com/diskuv/dk.git dk0
$ dk0/mlfront-shell -I 7zip-project -x 7zip-org:subpath: -- get-asset-file 'CommonsZip_Std.S7z1a.Assets@25.1.0' -p 7zr.exe -f target/7zr.exe
[error 215565e4]: Could not get asset file.
╭──▶ 7zip-project/CommonsZip_Std.S7z.S7zr0.values.jsonc:25.24-25.40
│
25 │ "sha256": "fill-me-in-later"
• ┬───────────────
• ╰╸ Could not find asset file `CommonsZip_Std.S7z1a.Assets@25.1.0+bn-20250101000000 -p 7zr.exe` because the computed SHA256 checksum `27cbe3d5804ad09e90bbcaa916da0d5c3b0be9462d0e0fb6cb54be5ed9030875` was unexpected.
•
│ Hint: This may have been a corrupted download, or an attempt to compromise security. Check the source `https://github.com/ip7z/7zip/releases/download/25.01`. If unsure how to proceed, contact your system or security administrator.
│ Hint: [autofix] Replace `fill-me-in-later` with `27cbe3d5804ad09e90bbcaa916da0d5c3b0be9462d0e0fb6cb54be5ed9030875` only if the checksum is valid; use `--autofix` to automatically correct it.
────╯Oops. We have an error! If you mess up with the JSON, you should get a specific error with a precise location inside in the JSON, and perhaps hints. If you don't find this to be the case, file an issue.
Let's rerun it with a new option:
| Argument | What |
|---|---|
--autofix |
Fix checksums (+ sizes in the future) |
and we get:
$ dk0/mlfront-shell --autofix -I 7zip-project -x 7zip-org:subpath: -- get-asset-file 'CommonsZip_Std.S7z1a.Assets@25.1.0' -p 7zr.exe -f target/7zr.exe
...
autofix applied to `7zip-project/CommonsZip_Std.S7z1a.S7zr.values.jsonc`$ dk0/mlfront-shell -I 7zip-project -x 7zip-org:subpath: -- get-asset-file 'CommonsZip_Std.S7z1a.Assets@25.1.0' -p 7zr.exe -f target/7zr.exe
[up-to-date] CommonsZip_Std.S7z1a.Assets@25.1.0+bn-20250101000000 -p 7zr.exeLet inspect the file. The file commands work only on Linux, but on Windows you can do a 7zr.exe --help.
$ file target/7zr.exe
target/7zr.exe: PE32 executable (console) Intel 80386, for MS Windows, 6 sectionsWe saw in the Concepts and Theory section that there are complex dependencies between the 7zip assets.
If a future version of 7zip changed those dependencies or changed the contents of a 7zip asset, any users of
the 7zip package who used a get-asset-file command could have a broken command.
In this step 1b, we'll use a form so that users of the 7zip package have a consistent interface to get 7zr.exe,
even if the 7zip assets change. A consistent interface is a best practice for all packages that use assets.
We'll create a new JSON file containing the form.
Save the following file as 7zip-project/CommonsZip_Std.S7z1b.S7zr.values.jsonc.
{
"$schema": "https://github.com/diskuv/dk/raw/refs/heads/V2_4/etc/jsonschema/mlfront-values.json",
"schema_version": { "major": 1, "minor": 0 },
"forms": [
{
"id": {
"package": "CommonsZip_Std.S7z1b",
"name": "S7zr",
"version": "25.1.0"
},
"precommands": {
"private": [
"get-asset-file CommonsZip_Std.S7z1a.Assets@25.1.0 -p 7zr.exe -f ${SLOT.File.Windows_arm}/7zr.exe",
"get-asset-file CommonsZip_Std.S7z1a.Assets@25.1.0 -p 7zr.exe -f ${SLOT.File.Windows_arm64}/7zr.exe",
"get-asset-file CommonsZip_Std.S7z1a.Assets@25.1.0 -p 7zr.exe -f ${SLOT.File.Windows_x86}/7zr.exe",
"get-asset-file CommonsZip_Std.S7z1a.Assets@25.1.0 -p 7zr.exe -f ${SLOT.File.Windows_x86_64}/7zr.exe"
]
},
"outputs": {
"files": [
{
"paths": ["7zr.exe"],
"slots": [
"File.Windows_arm",
"File.Windows_arm64",
"File.Windows_x86",
"File.Windows_x86_64"
]
}
]
}
}
]
}The form identifier CommonsZip_Std.S7z.S7zr@25.1.0 is split into three fields (package, name and version):
- The library identifier
CommonsZip_Stdand all the namespace terms except the last (S7z) belong in thepackagefield. - The last namespace term is the
namefield - The version is in the
versionfield.
(The last section described what a library identifier and namespace terms meant.)
If you are using the JSON schema to help create your JSON document, you will find that the package is optional.
This optionality is a convenience when embedding dk into a host language; some host languages like Java and the dk scripting system
can infer what the "package" is based on the file location of the JSON document.
Since we are using the reference implementation, we explicitly write the full package.
We also see the word slot used a few times ("slots":... and ${SLOT....}).
A slot is a parameter to a form. Whenever a form is used, the slot must be specified.
The name of a slot is one or more dot-separated namespace terms, but otherwise can be anything you want.
Each slot makes a unique instance of the form. Since we chose to have the four (4) slots, there are four (4) instances of the form created during a build.
We chose:
File.Windows_armFile.Windows_arm64File.Windows_x86File.Windows_x86_64
because 7zip has four different Windows installers.
Said another way, we have chosen in our interface API design to allow our users to ask for a 7zip executable specific to the operating system architectures that 7zip provides.
Choose slots that correspond to the natural split among your assets:
- Executables are naturally split by what platforms (ABI, operating system architecture, etc.) the executables run on.
- Multimedia assets are naturally split by dimension (high, medium, low res) and by media type (PNG, JPEG, etc.).
If you only have one asset, your one slot is conventionally called File.Agnostic.
What are the precommands? Precommands are a set of commands that run when the form is submitted.
You have already seen the get-asset-file command that you ran from the dk0/mlfront-shell command line in the
download 7zr.exe section. You also saw the get-object command that submitted
the 7zip form in Concepts and Theory. Commands we can execute from the dk0/mlfront-shell
command line can also run inside your form. Later sections will introduce more commands.
The JSON field is called precommands because a form has an optional "function" that gets called when the
form is submitted. We don't have a function in this example, but precommands always run before the optional function.
Key Point: The get-object precommand is the basic unit of function composition in the dk build system.
You can submit a form from the command line dk0/mlfront-shell ... -- get-object FORM ..., which runs the form's
precommands, and if one or more of those precommands is a get-object that precommand will submit another form.
And that form can then run precommands, which can then submit more forms. And so on.
In the JSON document, we also see our first variables:
${SLOT.File.Windows_arm}${SLOT.File.Windows_arm64}${SLOT.File.Windows_x86}${SLOT.File.Windows_x86_64}
When a form is submitted with get-object FORM -s SLOT, these variables are output directories specific to the named slot.
The Precommands section of the specification describes some optimizations
that are allowed. Most important is that if you ask for get-object FORM -s Windows_arm, the precommand for the unrelated slot get-asset-file ... -f ${SLOT.File.Windows_x86_64}/7zr.exe can be skipped.
The final piece of the form are the outputs. You must declare which files your form in the ${SLOT.*}
output directories. Don't worry ... the dk build system will give you an error and tell you a hint to fix it
when you forgot to declare files or you declare too many files.
We'll use get-object to submit our new form. With that we get:
$ dk0/mlfront-shell -I 7zip-project -x 7zip-org:subpath: --autofix -- get-object 'CommonsZip_Std.S7z1b.S7zr@25.1.0' -s File.Windows_x86_64 -d target/7zr-win64
[up-to-date] CommonsZip_Std.S7z1b.S7zr@25.1.0+bn-20250101000000 -s File.Windows_x86_64If we inspect the target directory, we see all the files from the object in Unix:
$ ls target/7zr-win64 | sort
7zr.exeand on Windows:
$ dir target/7zr-win64
Directory: target\7zr-win64
Mode LastWriteTime Length Name
---- ------------- ------ ----
-a--- 1/1/1980 12:00 AM 601088 7zr.exeNotice that all of the dates are set to Jan 1, 1980 for reproducibility.
Now we have indirection when the asset files change! We change the get-asset-file ... commands
in the precommands once, and none of our users need to change their get-object ... commands.
Now that we have seen how to specify one asset file, let's do all of the Windows installer and Unix compress archive asset files.
Save the following file as 7zip-project/CommonsZip_Std.S7z1c.S7zr.values.jsonc.
{
"$schema": "https://github.com/diskuv/dk/raw/refs/heads/V2_4/etc/jsonschema/mlfront-values.json",
"schema_version": {
"major": 1,
"minor": 0
},
"assets": [
{
"listing_unencrypted": {
"name": "CommonsZip_Std.S7z1c.Assets",
"spec_version": 2,
"version": "25.1.0"
},
"listing": {
"origins": [
{
"name": "7zip-org",
"mirrors": [
"https://github.com/ip7z/7zip/releases/download/25.01"
]
}
]
},
"files": [
{
"origin": "7zip-org",
"path": "7zr.exe",
"checksum": {
"sha256": "27cbe3d5804ad09e90bbcaa916da0d5c3b0be9462d0e0fb6cb54be5ed9030875"
},
"size": 601088
},
{
"origin": "7zip-org",
"path": "7z2501-arm.exe",
"checksum": {
"sha256": "c14b51d97ff644c49dcfc83ab000537ba806ebc7003359d7afc362e2af83f890"
},
"size": 16547840
},
{
"origin": "7zip-org",
"path": "7z2501-arm64.exe",
"checksum": {
"sha256": "6365c7c44e217b9c1009e065daf9f9aa37454e64315b4aaa263f7f8f060755dc"
},
"size": 15728640
},
{
"origin": "7zip-org",
"path": "7z2501-linux-arm.tar.xz",
"checksum": {
"sha256": "f5b498a55ab1bb2adb2690f7ad629022b5e52f159a95e7d71be6c5049db0696e"
},
"size": 12124160
},
{
"origin": "7zip-org",
"path": "7z2501-linux-arm64.tar.xz",
"checksum": {
"sha256": "39c5140f02ce4436599303c59a149f654cb1bbc47cdc105a942120d747ae040d"
},
"size": 13271040
},
{
"origin": "7zip-org",
"path": "7z2501-linux-x64.tar.xz",
"checksum": {
"sha256": "4ca3b7c6f2f67866b92622818b58233dc70367be2f36b498eb0bdeaaa44b53f4"
},
"size": 15308800
},
{
"origin": "7zip-org",
"path": "7z2501-linux-x86.tar.xz",
"checksum": {
"sha256": "62a47626eb9ce2886de3fec6f201f6a4210f14ba61b30a16141b953aed705207"
},
"size": 16793600
},
{
"origin": "7zip-org",
"path": "7z2501-mac.tar.xz",
"checksum": {
"sha256": "26aa75bc262bb10bf0805617b95569c3035c2c590a99f7db55c7e9607b2685e0"
},
"size": 18841600
},
{
"origin": "7zip-org",
"path": "7z2501-x64.exe",
"checksum": {
"sha256": "78afa2a1c773caf3cf7edf62f857d2a8a5da55fb0fff5da416074c0d28b2b55f"
},
"size": 15728640
}
]
}
]
}Previously we had used get-asset-file. Since we have so many asset files, it is easier
if we checked them all at once.
We'll be using a new command get-asset.
| Argument | What |
|---|---|
dk0/mlfront-shell |
The build tool. Eventually it will just be dk |
-I 7zip-project |
The folders containing *.values.jsonc files |
-x 7zip-org:subpath: |
Invalidate all files with the origin: "7zip-org" |
--autofix |
Fix any invalid checksums |
-- |
Separate dk0/mlfront-shell options from the command after |
get-asset |
Command to get all members of the named asset |
CommonsZip_Std.S7z1a.Assets@25.1.0 |
The name and version in .values.json |
-d target/7zr-assets |
Send command output into the directory |
With that we get:
$ dk0/mlfront-shell -I 7zip-project -x 7zip-org:subpath: --autofix -- get-asset 'CommonsZip_Std.S7z1c.Assets@25.1.0' -d target/7zr-assets
[up-to-date] CommonsZip_Std.S7z1c.Assets@25.1.0+bn-20250101000000If we inspect the target directory, we see all the files from the asset in Unix:
$ ls target/7zr-assets | sort
7z2501-arm.exe
7z2501-arm64.exe
7z2501-linux-arm.tar.xz
7z2501-linux-arm64.tar.xz
7z2501-linux-x64.tar.xz
7z2501-linux-x86.tar.xz
7z2501-mac.tar.xz
7z2501-x64.exe
7zr.exeand on Windows:
$ dir target/7zr-assets
Directory: target\7zr-assets
Mode LastWriteTime Length Name
---- ------------- ------ ----
-a--- 1/1/1980 12:00 AM 1658315 7z2501-arm.exe
-a--- 1/1/1980 12:00 AM 1588203 7z2501-arm64.exe
-a--- 1/1/1980 12:00 AM 1212764 7z2501-linux-arm.tar.xz
-a--- 1/1/1980 12:00 AM 1326736 7z2501-linux-arm64.tar.xz
-a--- 1/1/1980 12:00 AM 1571044 7z2501-linux-x64.tar.xz
-a--- 1/1/1980 12:00 AM 1721924 7z2501-linux-x86.tar.xz
-a--- 1/1/1980 12:00 AM 1876264 7z2501-mac.tar.xz
-a--- 1/1/1980 12:00 AM 1643509 7z2501-x64.exe
-a--- 1/1/1980 12:00 AM 601088 7zr.exeNotice that all of the dates are set to Jan 1, 1980 for reproducibility.
What happens if we output the asset to a file rather than a directory.
That is, what if we replaced -d target/7zr-assets with -f target/7zr-file?
$ dk0/mlfront-shell -I 7zip-project -x 7zip-org:subpath: --autofix -- get-asset 'CommonsZip_Std.S7z1c.Assets@25.1.0' -f target/7zr-file
[up-to-date] CommonsZip_Std.S7z1c.Assets@25.1.0+bn-20250101000000The file is a zip archive per the Saving Assets section of the specification.
$ unzip -l target/7zr-file
Archive: target/7zr-file
Length Date Time Name
--------- ---------- ----- ----
1658315 1980-01-01 00:00 ./7z2501-arm.exe
1588203 1980-01-01 00:00 ./7z2501-arm64.exe
1212764 1980-01-01 00:00 ./7z2501-linux-arm.tar.xz
1326736 1980-01-01 00:00 ./7z2501-linux-arm64.tar.xz
1571044 1980-01-01 00:00 ./7z2501-linux-x64.tar.xz
1721924 1980-01-01 00:00 ./7z2501-linux-x86.tar.xz
1876264 1980-01-01 00:00 ./7z2501-mac.tar.xz
1643509 1980-01-01 00:00 ./7z2501-x64.exe
601088 1980-01-01 00:00 ./7zr.exe
--------- -------
13199847 9 filesHowever, for performance, you should use get-asset-file when you need individual asset files.
In this section we want to run 7zr.exe (the lightweight 7zip unzipper)
to extract 7z.exe (the full featured 7zip unzip/zipper) from the Windows installers.
Whenever we run a command we need a form with a function.
Save the following file as 7zip-project/CommonsZip_Std.S7z2.Windows7zExe.values.jsonc.
{
"$schema": "https://github.com/diskuv/dk/raw/refs/heads/V2_4/etc/jsonschema/mlfront-values.json",
"schema_version": {
"major": 1,
"minor": 0
},
"forms": [
{
"id": {
"package": "CommonsZip_Std.S7z2",
"name": "Windows7zExe",
"version": "25.1.0"
},
"precommands": {
"private": [
// need bin/<architecture>/7zr.exe
"get-object CommonsZip_Std.S7z1b.S7zr@25.1.0 -s File.Windows_arm -d bin/File.Windows_arm",
"get-object CommonsZip_Std.S7z1b.S7zr@25.1.0 -s File.Windows_arm64 -d bin/File.Windows_arm64",
"get-object CommonsZip_Std.S7z1b.S7zr@25.1.0 -s File.Windows_x86 -d bin/File.Windows_x86",
"get-object CommonsZip_Std.S7z1b.S7zr@25.1.0 -s File.Windows_x86_64 -d bin/File.Windows_x86_64",
// need Windows installers
"get-asset-file CommonsZip_Std.S7z1c.Assets@25.1.0 -p 7z2501-arm.exe -f 7z-File.Windows_arm.exe",
"get-asset-file CommonsZip_Std.S7z1c.Assets@25.1.0 -p 7z2501-arm64.exe -f 7z-File.Windows_arm64.exe",
"get-asset-file CommonsZip_Std.S7z1c.Assets@25.1.0 -p 7z2501-x64.exe -f 7z-File.Windows_x86.exe",
"get-asset-file CommonsZip_Std.S7z1c.Assets@25.1.0 -p 7z2501-x64.exe -f 7z-File.Windows_x86_64.exe"
]
},
"function": {
"execution": [
// 7zr.exe only runs on Windows
{
"name": "OSFamily",
"value": "windows"
}
],
"args": [
// extract assetfile [7z2501-arm.exe] with [7zr.exe] to the output directory
// for slot [File.Windows_arm], etc.
"bin/${SLOTNAME.request}/7zr.exe",
"x",
"-o${SLOT.request}",
"7z-${SLOTNAME.request}.exe",
// extract only the '7z.exe' executable
"7z.exe"
]
},
"outputs": {
"files": [
{
"paths": ["7z.exe"],
"slots": [
"File.Windows_arm",
"File.Windows_arm64",
"File.Windows_x86",
"File.Windows_x86_64",
"File.Agnostic",
"File.Darwin_arm64",
"File.Darwin_x86_64",
"File.Linux_arm",
"File.Linux_arm64",
"File.Linux_x86",
"File.Linux_x86_64"
]
}
]
}
}
]
}The precommands should be familiar to you from the previous two sections.
The function runs the command:
bin/${SLOTNAME.request}/7zr.exe x -o${SLOT.request} 7z-${SLOTNAME.request}.exe 7z.exeBut remember we defined a slot as a parameter to a form.
So if we were to submit this form with get-object FORM -s File.Windows_x86_64, the
following command would be run:
bin/File.Windows_x86_64/7zr.exe x -o${SLOT.File.Windows_x86_64} 7z-File.Windows_x86_64.exe 7z.exe${SLOTNAME.request} expanded to File.Windows_x86_64, and ${SLOT.request}
expanded to ${SLOT.File.Windows_x86_64}.
All that means is that regardless of which slot is submitted with the form, the correct Windows installer is extracted.
We do have a problem:
7zr.exeonly runs on Windows, but we know that users will submit non-Windows slots for the 7zip package.${SLOTNAME.request}will expand to whatever the user requests ... it does not know anything about Windows.
We need to do three things to solve that problem:
- Give a
executionrequirement that only theOSFamilynamedwindowscan run. This is language that Google's Bazel build system uses, and we keep the same language indk. There are several providers of cloud-based "runners" for Bazel compatible build systems, that let you scale and run your builds on cloud hardware.dkwill support these runners in the future. - Provide the full list of slots in the
outputssection. - Make sure (in a later form) that we don't submit this Windows-only form on non-Windows hardware.
But for now we can submit the form if we are on Windows. We'll submit the File.Windows_arm64 slot:
$ dk0/mlfront-shell -I 7zip-project -x 7zip-org:subpath: --autofix -- get-object 'CommonsZip_Std.S7z2.Windows7zExe@25.1.0' -s File.Windows_arm64 -d target/7zexe-winarm64
[up-to-date] CommonsZip_Std.S7z2.Windows7zExe@25.1.0+bn-20250101000000 -s File.Windows_arm64If we inspect the target directory, we see all the files from the object on Windows:
$ dir target/7zexe-winarm64
Directory: target\7zexe-winarm64
Mode LastWriteTime Length Name
---- ------------- ------ ----
-a--- 1/1/1980 12:00 AM 575488 7z.exeAnd if we inspected that 7z.exe with the UNIX command file, we'd see:
$ file target/7zexe-winarm64/7z.exe
target/7zexe-winarm64/7z.exe: PE32+ executable (console) Aarch64, for MS Windows, 6 sectionsIn this section, we will use 7z.exe to uncompress 7z2501-linux-x86.tar from 7z2501-linux-x86.tar.xz (etc.).
This is a new form with a function that will call 7z.exe with the right parameters.
Save the following file as 7zip-project/CommonsZip_Std.S7z3.MacLinux7zTar.values.jsonc.
{
"$schema": "https://github.com/diskuv/dk/raw/refs/heads/V2_4/etc/jsonschema/mlfront-values.json",
"schema_version": {
"major": 1,
"minor": 0
},
"forms": [
{
"id": {
"package": "CommonsZip_Std.S7z3",
"name": "MacLinux7zTar",
"version": "25.1.0"
},
"precommands": {
"private": [
// need bin/<architecture>/7zr.exe
"get-object CommonsZip_Std.S7z1b.S7zr@25.1.0 -s File.Windows_x86 -d bin/File.Windows_x86",
// need macOS and Linux installers
"get-asset-file CommonsZip_Std.S7z1c.Assets@25.1.0 -p 7z2501-linux-arm.tar.xz -f 7z-File.Linux_arm.tar.xz",
"get-asset-file CommonsZip_Std.S7z1c.Assets@25.1.0 -p 7z2501-linux-arm64.tar.xz -f 7z-File.Linux_arm64.tar.xz",
"get-asset-file CommonsZip_Std.S7z1c.Assets@25.1.0 -p 7z2501-linux-x86.tar.xz -f 7z-File.Linux_x86.tar.xz",
"get-asset-file CommonsZip_Std.S7z1c.Assets@25.1.0 -p 7z2501-linux-x64.tar.xz -f 7z-File.Linux_x86_64.tar.xz",
"get-asset-file CommonsZip_Std.S7z1c.Assets@25.1.0 -p 7z2501-mac.tar.xz -f 7z-File.Darwin_x86_64.tar.xz",
"get-asset-file CommonsZip_Std.S7z1c.Assets@25.1.0 -p 7z2501-mac.tar.xz -f 7z-File.Darwin_arm64.tar.xz"
]
},
"function": {
"execution": [
// 7zr.exe only runs on Windows
{
"name": "OSFamily",
"value": "windows"
}
],
"args": [
// uncompress assetfile [7z2501-mac.tar.xz] with [7zr.exe] to the output directory
// for slot [File.Darwin_arm64], etc.
"bin/File.Windows_x86/7zr.exe",
"x",
"-o${SLOT.request}",
"7z-${SLOTNAME.request}.tar.xz",
"7z-${SLOTNAME.request}.tar"
]
},
"outputs": {
"files": [
{
"paths": ["7z-File.Darwin_arm64.tar"],
"slots": ["File.Darwin_arm64"]
},
{
"paths": ["7z-File.Darwin_x86_64.tar"],
"slots": ["File.Darwin_x86_64"]
},
{
"paths": ["7z-File.Linux_arm.tar"],
"slots": ["File.Linux_arm"]
},
{
"paths": ["7z-File.Linux_arm64.tar"],
"slots": ["File.Linux_arm64"]
},
{
"paths": ["7z-File.Linux_x86.tar"],
"slots": ["File.Linux_x86"]
},
{
"paths": ["7z-File.Linux_x86_64.tar"],
"slots": ["File.Linux_x86_64"]
}
]
}
}
]
}The use of 7zr.exe means we can only run this step on Windows hardware, even though we produce output for non-Windows slots like File.Linux_arm64:
$ dk0/mlfront-shell -I 7zip-project -x 7zip-org:subpath: --autofix -- get-object 'CommonsZip_Std.S7z3.MacLinux7zTar@25.1.0' -s File.Linux_arm64 -d target/7ztar-linuxarm64
[up-to-date] CommonsZip_Std.S7z3.MacLinux7zTar@25.1.0+bn-20250101000000 -s File.Linux_arm64and the target directory has:
Directory: target\7ztar-linuxarm64
Mode LastWriteTime Length Name
---- ------------- ------ ----
-a--- 1/1/1980 12:00 AM 5863936 7z-File.Linux_arm64.tar
In this section, we will use 7z.exe to extract 7zz from 7z2501-linux-x86.tar (etc.).
This is a new form with a function that will call 7z.exe with the right parameters.
Save the following file as 7zip-project/CommonsZip_Std.S7z4.MacLinux7zExe.values.jsonc.
{
"$schema": "https://github.com/diskuv/dk/raw/refs/heads/V2_4/etc/jsonschema/mlfront-values.json",
"schema_version": {
"major": 1,
"minor": 0
},
"forms": [
{
"id": {
"package": "CommonsZip_Std.S7z4",
"name": "MacLinux7zExe",
"version": "25.1.0"
},
"precommands": {
"private": [
// need bin/<architecture>/7z.exe (7zr does not extract tar files)
"get-object CommonsZip_Std.S7z2.Windows7zExe@25.1.0 -s File.Windows_x86 -d bin/File.Windows_x86",
// need macOS and Linux tar files. since same destination directory need [install-object]
"install-object CommonsZip_Std.S7z3.MacLinux7zTar@25.1.0 -s File.Linux_arm -d tarballs",
"install-object CommonsZip_Std.S7z3.MacLinux7zTar@25.1.0 -s File.Linux_arm64 -d tarballs",
"install-object CommonsZip_Std.S7z3.MacLinux7zTar@25.1.0 -s File.Linux_x86 -d tarballs",
"install-object CommonsZip_Std.S7z3.MacLinux7zTar@25.1.0 -s File.Linux_x86_64 -d tarballs",
"install-object CommonsZip_Std.S7z3.MacLinux7zTar@25.1.0 -s File.Darwin_arm64 -d tarballs",
"install-object CommonsZip_Std.S7z3.MacLinux7zTar@25.1.0 -s File.Darwin_x86_64 -d tarballs"
]
},
"function": {
"execution": [
// 7z.exe only runs on Windows
{
"name": "OSFamily",
"value": "windows"
}
],
"args": [
// extract object [7z2501-mac.tar] with [7z.exe] to the output directory
// for slot [File.Darwin_arm64], etc.
"bin/File.Windows_x86/7z.exe",
"x",
"-o${SLOT.request}",
"tarballs/7z-${SLOTNAME.request}.tar",
// extract only the '7zz' executable
"7zz"
]
},
"outputs": {
"files": [
{
"paths": ["7zz"],
"slots": ["File.Darwin_arm64"]
},
{
"paths": ["7zz"],
"slots": ["File.Darwin_x86_64"]
},
{
"paths": ["7zz"],
"slots": ["File.Linux_arm"]
},
{
"paths": ["7zz"],
"slots": ["File.Linux_arm64"]
},
{
"paths": ["7zz"],
"slots": ["File.Linux_x86"]
},
{
"paths": ["7zz"],
"slots": ["File.Linux_x86_64"]
}
]
}
}
]
}The use of 7z.exe means we can only run this step on Windows hardware, even though we produce output for non-Windows slots like File.Darwin_x86_64:
$ dk0/mlfront-shell -I 7zip-project -x 7zip-org:subpath: --autofix -- get-object 'CommonsZip_Std.S7z4.MacLinux7zExe@25.1.0' -s File.Darwin_x86_64 -d target/7zexe-macintel
[up-to-date] CommonsZip_Std.S7z4.MacLinux7zExe@25.1.0+bn-20250101000000 -s File.Darwin_x86_64and the target directory has:
Directory: target\7zexe-macintel
Mode LastWriteTime Length Name
---- ------------- ------ ----
-a--- 1/1/1980 12:00 AM 5885696 7zz
Using the UNIX command file we see that the CPU architecture matches what we requested:
$ file target/7zexe-macintel/7zz
target/7zexe-macintel/7zz: Mach-O universal binary with 2 architectures: [x86_64:\012- Mach-O 64-bit x86_64 executable, flags:<NOUNDEFS|DYLDLINK|TWOLEVEL|WEAK_DEFINES|BINDS_TO_WEAK|PIE>] [\012- arm64:\012- Mach-O 64-bit arm64 executable, flags:<NOUNDEFS|DYLDLINK|TWOLEVEL|WEAK_DEFINES|BINDS_TO_WEAK|PIE>]We have not yet provided an overall interface for the 7zip package. Let's do this right now!
Save the following file as 7zip-project/CommonsZip_Std.S7z5.values.jsonc.
{
"$schema": "https://github.com/diskuv/dk/raw/refs/heads/V2_4/etc/jsonschema/mlfront-values.json",
"schema_version": {
"major": 1,
"minor": 0
},
"forms": [
{
"id": {
"package": "CommonsZip_Std.S7z5",
"name": "S7zExe",
"version": "25.1.0"
},
"precommands": {
"private": [
// We'll make the executable name '7zz.exe' on all platforms so that
// end-users don't have to have complicated logic based on the platform.
// Windows. 7z.exe -> 7zz.exe
"get-object CommonsZip_Std.S7z.Windows7zExe@25.1.0 -s File.Windows_arm -m ./7z.exe -f ${SLOT.File.Windows_arm}/7zz.exe",
"get-object CommonsZip_Std.S7z.Windows7zExe@25.1.0 -s File.Windows_arm64 -m ./7z.exe -f ${SLOT.File.Windows_arm64}/7zz.exe",
"get-object CommonsZip_Std.S7z.Windows7zExe@25.1.0 -s File.Windows_x86 -m ./7z.exe -f ${SLOT.File.Windows_x86}/7zz.exe",
"get-object CommonsZip_Std.S7z.Windows7zExe@25.1.0 -s File.Windows_x86_64 -m ./7z.exe -f ${SLOT.File.Windows_x86_64}/7zz.exe",
// Unix. 7zz -> 7zz.exe
"get-object CommonsZip_Std.S7z.MacLinux7zExe@25.1.0 -s File.Darwin_arm64 -m ./7zz -f ${SLOT.File.Darwin_arm64}/7zz.exe",
"get-object CommonsZip_Std.S7z.MacLinux7zExe@25.1.0 -s File.Darwin_x86_64 -m ./7zz -f ${SLOT.File.Darwin_x86_64}/7zz.exe",
"get-object CommonsZip_Std.S7z.MacLinux7zExe@25.1.0 -s File.Linux_arm -m ./7zz -f ${SLOT.File.Linux_arm}/7zz.exe",
"get-object CommonsZip_Std.S7z.MacLinux7zExe@25.1.0 -s File.Linux_arm64 -m ./7zz -f ${SLOT.File.Linux_arm64}/7zz.exe",
"get-object CommonsZip_Std.S7z.MacLinux7zExe@25.1.0 -s File.Linux_x86 -m ./7zz -f ${SLOT.File.Linux_x86}/7zz.exe",
"get-object CommonsZip_Std.S7z.MacLinux7zExe@25.1.0 -s File.Linux_x86_64 -m ./7zz -f ${SLOT.File.Linux_x86_64}/7zz.exe"
]
},
"outputs": {
"files": [
{
"paths": ["7zz.exe"],
"slots": [
"File.Windows_arm",
"File.Windows_arm64",
"File.Windows_x86",
"File.Windows_x86_64",
"File.Darwin_arm64",
"File.Darwin_x86_64",
"File.Linux_arm",
"File.Linux_arm64",
"File.Linux_x86",
"File.Linux_x86_64"
]
}
]
}
}
]
}With that our users can grab the 7zz.exe executable for any operating system and CPU architecture:
$ dk0/mlfront-shell -I 7zip-project -x 7zip-org:subpath: --autofix -- get-object 'CommonsZip_Std.S7z5.S7zExe@25.1.0' -s File.Linux_x86_64 -d target/7z-linux64
[up-to-date] CommonsZip_Std.S7z5.S7zExe@25.1.0+bn-20250101000000 -s File.Linux_x86_64$ ls target/7z-linux64
7zz.exe
$ file target/7z-linux64/7zz.exe
target/7z-linux64/7zz.exe: ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, BuildID[sha1]=7a6c7a136fc4e7df4ddcd80d2aae72bc658ef822, for GNU/Linux 3.2.0, strippedSince 7zz.exe is a standalone executable that doesn't need any DLLs or .so (except standard GLIBC system libraries on Linux), we can use the -m MEMBER option to directly fetch the 7zz.exe executable:
$ dk0/mlfront-shell -I 7zip-project -x 7zip-org:subpath: --autofix -- get-object 'CommonsZip_Std.S7z5.S7zExe@25.1.0' -s File.Linux_x86_64 -m ./7zz.exe -f target/7zz.exe
[up-to-date] CommonsZip_Std.S7z5.S7zExe@25.1.0+bn-20250101000000 -s File.Linux_x86_64
$ file target/7zz.exe
target/7zz.exe: ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, BuildID[sha1]=7a6c7a136fc4e7df4ddcd80d2aae72bc658ef822, for GNU/Linux 3.2.0, strippedWe danced around the need for a Windows machine. Many developers don't have one.
Now we are ready to use GitHub Actions to provide that Windows machine.
Here is a GitHub Actions workflow; we have saved it as .github/workflows/CommonsZip_Std.S7z.S7zExe.build.yml:
name: CommonsZip_Std.S7z.S7zExe
on:
push: # build on every commit
workflow_dispatch: # allow manual triggering from GitHub page
jobs:
build:
# Your job will be different, but this job needs Windows since CommonsZip_Std.S7z.MacLinux7zExe@25.1.0 has OSFamily=windows.
runs-on: windows-2022
steps:
- name: Checkout
uses: actions/checkout@v4
- name: Cache dk data stores and build keys
uses: actions/cache@v4
with:
# The data stores and build keys are safe to share between Windows, macOS and Linux
enableCrossOsArchive: true
path: |
target/data/dk/val.1
target/data/dk/cts.1.*
target/config/dk/build.pub
target/config/dk/build.sec
restore-keys: dk-stores-and-keys # use latest cache
key: dk-stores-and-keys-${{ github.run_id }} # update cache on every run
- name: Build pkgs/include/CommonsZip_Std.S7z.S7zExe
env:
# Cross-platform CI caches behave best when the cached data is under the project directory
XDG_CONFIG_HOME: ${{ github.workspace }}/target/config
XDG_DATA_HOME: ${{ github.workspace }}/target/data
run: |
git clone --branch V2_4 https://github.com/diskuv/dk.git dk0
dk0/mlfront-shell -I dk0/pkgs/include --verbose -- get-object 'CommonsZip_Std.S7z.S7zExe@25.1.0' -s File.Darwin_arm64 -m ./7zz.exe -f target/Darwin_arm64.7zz.exe
dk0/mlfront-shell -I dk0/pkgs/include --verbose -- get-object 'CommonsZip_Std.S7z.S7zExe@25.1.0' -s File.Darwin_x86_64 -m ./7zz.exe -f target/Darwin_x86_64.7zz.exe
dk0/mlfront-shell -I dk0/pkgs/include --verbose -- get-object 'CommonsZip_Std.S7z.S7zExe@25.1.0' -s File.Linux_arm -m ./7zz.exe -f target/Linux_arm.7zz.exe
dk0/mlfront-shell -I dk0/pkgs/include --verbose -- get-object 'CommonsZip_Std.S7z.S7zExe@25.1.0' -s File.Linux_arm64 -m ./7zz.exe -f target/Linux_arm64.7zz.exe
dk0/mlfront-shell -I dk0/pkgs/include --verbose -- get-object 'CommonsZip_Std.S7z.S7zExe@25.1.0' -s File.Linux_x86 -m ./7zz.exe -f target/Linux_x86.7zz.exe
dk0/mlfront-shell -I dk0/pkgs/include --verbose -- get-object 'CommonsZip_Std.S7z.S7zExe@25.1.0' -s File.Linux_x86_64 -m ./7zz.exe -f target/Linux_x86_64.7zz.exe
dk0/mlfront-shell -I dk0/pkgs/include --verbose -- get-object 'CommonsZip_Std.S7z.S7zExe@25.1.0' -s File.Windows_x86 -m ./7zz.exe -f target/Windows_x86.7zz.exe
dk0/mlfront-shell -I dk0/pkgs/include --verbose -- get-object 'CommonsZip_Std.S7z.S7zExe@25.1.0' -s File.Windows_x86_64 -m ./7zz.exe -f target/Windows_x86_64.7zz.exe
- name: Test 7zz.exe
run: target/Windows_x86_64.7zz.exe --help
# Create release if and only if tag is pushed
- name: Release all slots
uses: softprops/action-gh-release@v2
if: github.ref_type == 'tag'
with:
files: |
target/Darwin_arm64.7zz.exe
target/Darwin_x86_64.7zz.exe
target/Linux_arm.7zz.exe
target/Linux_arm64.7zz.exe
target/Linux_x86.7zz.exe
target/Linux_x86_64.7zz.exe
target/Windows_x86.7zz.exe
target/Windows_x86_64.7zz.exeIt uses the community 7zip package (links provided in the next section) rather than the one we built in this tutorial.
You can see it in action at https://github.com/diskuv/dk/actions/workflows/CommonsZip_Std.S7z.S7zExe.build.yml
Today, there is no mechanism to pull the data from GitHub Actions into your local machine. That feature can be added; file a issue to express interest.
We introduced the following commands:
get-asset-fileget-assetget-object
and explained how to specify:
"assets": ..."forms": ...
and explained how to run the reference implementation's dk0/mlfront-shell.
We talked through how to create a build plan for a non-simple package like 7zip. Many of your packages will have easier build plans.
We saw how to present a consistent interface with get-object for our package that is resilient to changes in underlying assets.
And we saw how to leverage an operating system (in our example Windows) to provide a build environment for other operating systems.
Finally, the community 7zip package is available for comparison at:
- pkgs/include/CommonsZip_Std.S7z.S7zr.values.jsonc
- pkgs/include/CommonsZip_Std.S7z.Windows7zExe.values.jsonc
- pkgs/include/CommonsZip_Std.S7z.MacLinux7zTar.values.jsonc
- pkgs/include/CommonsZip_Std.S7z.MacLinux7zExe.values.jsonc
- pkgs/include/CommonsZip_Std.S7z.values.jsonc
dk has a mechanism to "enter" an incomplete object that has failed to build.
Once an object has successfully built, it is no longer possible to "enter" the object since all values are unchangeable.
Entering an incomplete object means you get a shell (your Unix shell, PowerShell, etc.) where you can see the results of the form's precommand and the results of the form's function, if it has a function.
We'll create a form that always fails, with precommands for the 7zip package we discussed last chapter. With that failed form we'll be able to inspect the 7zip packages.
The always-failing form is the following (you don't need to create your own copy):
{
"$schema": "https://github.com/diskuv/dk/raw/refs/heads/V2_4/etc/jsonschema/mlfront-values.json",
"schema_version": {
"major": 1,
"minor": 0
},
"forms": [
{
"id": {
"package": "CommonsZip_Std.S7z9",
"name": "Debug",
"version": "25.1.0"
},
"precommands": {
"private": [
// we'll use the community 7zip package, not the tutorial 7zip package
"get-object CommonsZip_Std.S7z.S7zExe@25.1.0 -s File.Linux_arm -d linuxarm_stuff",
"get-object CommonsZip_Std.S7z.S7zExe@25.1.0 -s File.Windows_x86 -d win32_stuff",
// all forms, even ones designed to fail, must have at least one slot
"get-object CommonsZip_Std.S7z.S7zExe@25.1.0 -s File.Linux_arm64 -d ${SLOT.File.Linux_arm64}"
]
},
"function": {
"args": [
// run something that will always fail. the precommands are what we want to test
"false"
]
},
"outputs": {
"files": [
{
"paths": ["7zz.exe"],
"slots": ["File.Linux_arm64"]
}
]
}
}
]
}Enter the form with:
$ dk0/mlfront-shell -I dk0/docs/7zip-tutorial -x 7zip-org:subpath: -- enter-object 'CommonsZip_Std.S7z9.Debug@25.1.0' -s File.Linux_arm64
PS CommonsZip_Std.S7z9.Debug@25.1.0+bn-20250101000000 -s File.Linux_arm64>
PS CommonsZip_Std.S7z9.Debug@25.1.0+bn-20250101000000 -s File.Linux_arm64> dir -recurse
Directory: target\pid\480\lsgbf44nuuwajzjh\fn\File.Linux_arm64
Mode LastWriteTime Length Name
---- ------------- ------ ----
d---- 9/18/2025 8:00 AM linuxarm_stuff
d---- 9/18/2025 8:00 AM win32_stuff
Directory: target\pid\480\lsgbf44nuuwajzjh\fn\File.Linux_arm64\linuxarm_stuff
Mode LastWriteTime Length Name
---- ------------- ------ ----
-a--- 1/1/1980 12:00 AM 1548480 7zz.exe
Directory: target\pid\480\lsgbf44nuuwajzjh\fn\File.Linux_arm64\win32_stuff
Mode LastWriteTime Length Name
---- ------------- ------ ----
-a--- 1/1/1980 12:00 AM 575488 7zz.exe
PS CommonsZip_Std.S7z9.Debug@25.1.0+bn-20250101000000 -s File.Linux_arm64> dir $env:SHELL_SLOT
Directory: target\pid\480\lsgbf44nuuwajzjh\out\File.Linux_arm64
Mode LastWriteTime Length Name
---- ------------- ------ ----
-a--- 1/1/1980 12:00 AM 2456896 7zz.exe
PS CommonsZip_Std.S7z9.Debug@25.1.0+bn-20250101000000 -s File.Linux_arm64> exitThe Unix shell is similar.
The $env:SHELL_SLOT (PowerShell), %SHELL_SLOT% (Windows Batch) or $SHELL_SLOT have the SLOT output directory for enter-object OBJECT -s SLOT.
Install on Windows:
winget install -e --id Diskuv.dkor Apple/Silicon:
sudo curl -o /usr/local/bin/dk https://diskuv.com/a/dk-exe/2.4.202508302258-signed/dk-darwin_arm64
sudo chmod +x /usr/local/bin/dkor Apple/Intel:
sudo curl -o /usr/local/bin/dk https://diskuv.com/a/dk-exe/2.4.202508302258-signed/dk-darwin_x86_64
sudo chmod +x /usr/local/bin/dkor Linux with glibc and libstdc++ (Debian, Ubuntu, etc. but not Alpine):
sudo curl -o /usr/local/bin/dk https://diskuv.com/a/dk-exe/2.4.202508302258-signed/dk-linux_x86_64
sudo chmod +x /usr/local/bin/dk
[ -x /usr/bin/dnf ] && sudo dnf install -y libstdc++Then cross-compile a script to standalone Windows, Linux, Android executables (and to a macOS executable if you are on a macOS machine):
$ dk -S "
module Http = DkNet_Std.Http
module Uri = Tr1Uri_Std.Uri
let print_endline = Tr1Stdlib_V414Io.StdIo.print_endline
" -U "
print_endline @@
Lwt_main.run @@
Http.fetch_url ~max_sz:4096 @@
Uri.of_string {|https://jigsaw.w3.org/HTTP/h-content-md5.html|}
" -O ReleaseSmall Exe
[INFO ] /StdStd_Std.Run/
Assembling adhoc script ZzZz_Zz.Adhoc:
module Http = DkNet_Std.Http
module Uri = Tr1Uri_Std.Uri
let print_endline = Tr1Stdlib_V414Io.StdIo.print_endline
let __init (context : DkCoder_Std.Context.t) =
print_endline @@ Lwt_main.run
@@ Http.fetch_url ~max_sz:4096
@@ Uri.of_string {|https://jigsaw.w3.org/HTTP/h-content-md5.html|};
__init context
[@@warning "-unused-var-strict"]
[INFO ][2025-09-17T23:18:56Z] /StdStd_Std.Exe/
<linux_x86_64> target/ZzZz_Zz.Adhoc-linux_x86_64
[INFO ][2025-09-17T23:18:57Z] /StdStd_Std.Exe/
<linux_x86> target/ZzZz_Zz.Adhoc-linux_x86
[INFO ][2025-09-17T23:19:01Z] /StdStd_Std.Exe/
<android_x86_64> target/ZzZz_Zz.Adhoc-android_x86_64
[INFO ][2025-09-17T23:19:04Z] /StdStd_Std.Exe/
<android_arm64v8a> target/ZzZz_Zz.Adhoc-android_arm64v8a
[INFO ][2025-09-17T23:19:05Z] /StdStd_Std.Exe/
<android_arm32v7a> target/ZzZz_Zz.Adhoc-android_arm32v7a
[INFO ][2025-09-17T23:19:05Z] /StdStd_Std.Exe/
Skipped darwin_x86_64 executable since that requires macOS machines for codesigning and Xcode for licensed MacOSX SDK
[INFO ][2025-09-17T23:19:05Z] /StdStd_Std.Exe/
Skipped darwin_arm64 executable since that requires macOS machines for codesigning and Xcode for licensed MacOSX SDK
[INFO ][2025-09-17T23:19:08Z] /StdStd_Std.Exe/
<windows_x86_64> target/ZzZz_Zz.Adhoc-windows_x86_64.exe
[INFO ][2025-09-17T23:19:11Z] /StdStd_Std.Exe/
<windows_x86> target/ZzZz_Zz.Adhoc-windows_x86.exeThe executables will be available in the target/ folder:
$ file target/ZzZz_Zz.Adhoc-* | cut -c1-69 | awk '{print $0 "..."}'
target/ZzZz_Zz.Adhoc-android_arm32v7a: ELF 32-bit LSB pie executabl...
target/ZzZz_Zz.Adhoc-android_arm64v8a: ELF 64-bit LSB pie executabl...
target/ZzZz_Zz.Adhoc-android_x86_64: ELF 64-bit LSB pie executabl...
target/ZzZz_Zz.Adhoc-darwin_arm64: Mach-O 64-bit executable arm...
target/ZzZz_Zz.Adhoc-darwin_x86_64: Mach-O 64-bit executable x86...
target/ZzZz_Zz.Adhoc-linux_x86: ELF 32-bit LSB pie executabl...
target/ZzZz_Zz.Adhoc-linux_x86_64: ELF 64-bit LSB pie executabl...
target/ZzZz_Zz.Adhoc-windows_x86.exe: PE32 executable (console) In...
target/ZzZz_Zz.Adhoc-windows_x86_64.pdb: MSVC program database ver 7....
target/ZzZz_Zz.Adhoc-windows_x86_64.exe: PE32+ executable (console) x...
target/ZzZz_Zz.Adhoc-windows_x86.pdb: MSVC program database ver 7....Italics are a pending feature.
| Tool | Features better with the tool | Features better with dk |
|---|---|---|
| Nix | Huge set of packages | Works on Windows. Static types. |
| (contd.) | Signify-backed supply chain | |
| Buck2 + | Scales to millions of files | Works well outside of monorepo |
| ... Bazel | Backed by Big Tech | Easier to adopt |
| Ninja | Integrated with CMake. Fast | Cloud-friendly, sharable artifacts |
| (contd.) | Rules to simplify build files | Pending integration in scripts |
| (contd.) | Multithreading | Pending integration in dk |
| Docker | Huge set of images | Works well on Windows |
| (contd.) | Works very well in CI | Works in CI and during installs |
The following are tools specific to the OCaml language, and dk should not replace them.
Skip this table if you are not an OCaml-er.
| Tool | Features better with the tool | Features better with dk |
|---|---|---|
| opam | Thousands of packages | Immutable storage. Binary artifacts. |
| dune | Watch mode. Fast | Extensible. Not tied to OCaml. |
Copyright 2023 Diskuv, Inc.
The ./dk and ./dk.cmd build scripts ("dk") are
available under the Open Software License version 3.0,
https://opensource.org/license/osl-3-0-php/.
A guide to the Open Software License version 3.0 is available at
https://rosenlaw.com/OSL3.0-explained.htm.
dk.cmd downloads parts of the 7-Zip program. 7-Zip is licensed under the GNU LGPL license.
The source code for 7-Zip can be found at <www.7-zip.org>. Attribute requirements are available at https://www.7-zip.org/faq.html.
"dk" downloads OCaml, codept and other binaries at first run and on each version upgrade. OCaml has a LPGL2.1 license with Static Linking Exceptions. codept has a LPGL2.1 license with Static Linking Exceptions. The other binaries are DkSDK Coder Runtime Binaries © 2023 by Diskuv, Inc. These DkSDK Coder Runtime Binaries are licensed under Attribution-NoDerivatives 4.0 International. To view a copy of this license, visit http://creativecommons.org/licenses/by-nd/4.0/.
"dk" acts as a package manager; you run ./dk and tell it what packages you want to download
and run. These packages have independent licenses and you may be prompted to accept a license.
Those licenses include but are not limited to:
The significant parts of dk that are open-source and downloaded:
- DkML compiler: https://github.com/diskuv/dkml-compiler and https://gitlab.com/dkml/distributions/dkml
- MlFront: https://gitlab.com/dkml/build-tools/MlFront
- Tr1 libraries: to be published
{ "$schema": "https://github.com/diskuv/dk/raw/refs/heads/V2_4/etc/jsonschema/mlfront-values.json", "schema_version": { "major": 1, "minor": 0 }, "assets": [ { "listing_unencrypted": { "name": "CommonsZip_Std.S7z1a.Assets", "spec_version": 2, // 25.01 "version": "25.1.0" }, "listing": { "origins": [ { "name": "7zip-org", "mirrors": [ "https://github.com/ip7z/7zip/releases/download/25.01" ] } ] }, "files": [ { "origin": "7zip-org", "path": "7zr.exe", "checksum": { "sha256": "fill-me-in" }, "size": 601088 } ] } ] }