jbaber / usv

Unicode Separated Values (USV) data markup for units, records, groups, files, streaming, and more.

Unicode Separated Values (USV)

Unicode Separated Values (USV) is a data format that uses Unicode characters to mark parts.

USV builds on ASCII separated values (ASV) and contrasts with comma separated values (CSV).

USV offers pragmatic ways to edit data in text editors by using visual symbols and line breaks.

USV has capabilities for spreadsheet folios and sheets, databases schemas and tables, and more.

FAQ • RFC • Code • Comparisons • Criticisms • TODO • XKCD

USV characters

Separators:

• Unit Separator (US) is U+001F or U+241F ␟

• Record Separator (RS) is U+001E or U+241E ␞

• Group Separator (GS) is U+001D or U+241D ␝

• File Separator (FS) is U+001C or U+241C ␜

Modifiers:

• Escape (ESC) is U+001B or U+241B ␛

• End of Transmission (EOT) is U+0004 or U+2404 ␄

Liners:

• Carriage Return (CR) is U+000D

• Line Feed (LF) is U+000A

Hello World

The USV unit "hello" and USV unit "world":

hello␟world␟

Liners can prettify the visual display layout:

hello␟
world␟

Parsing can use libraries such as the USV Rust crate:

use usv::*;
let input = "hello␟world␟";
let units = input.units().collect();

Comparisons to text data formats

Capability	USV	ASV	CSV
Units / Cells / Fields	✅	✅	✅
Records / Lines / Rows	✅	✅	✅
Groups / Sheets / Tables	✅	✅	⛔
Files / Folios / Schemas	✅	✅	⛔
Visible separators	✅	⛔	✅
Separator line spacing	✅	⛔	🟡
IETF.org standards-track	✅	⛔	🟡
Unicode UTF-8 default	✅	⛔	⛔

Comparisons to spreadsheets and databases

USV semantics are units, records, groups, files.

Spreadsheet semantics are cells, lines, sheets, folios.

Databases semantics are fields, rows, tables, schemas.

Documentation

Commentary:

• Frequently Asked Questions (FAQ)

• Criticisms and replies

• TODO list

Specification:

• Request For Comments (RFC)

• Augmented Backus–Naur Form (ABNF)

• Code examples and production crates

• Command line argument parsing

Character details:

• Escape (ESC)

• End of Transmission (EOT)

• Liners (CR|LF)

How to:

• How to type Unicode characters

• How to use split and regex

Context:

• Converters for ASV, CSV, JSON, XLSX

• Comparisons with ASV, CSV, TSV, RSV, JSON

• History of ASCII separated values (ASV)

Editor notes:

• vim notes

• emacs notes

Example files:

• hello-world.usv versus hello-world.csv

• zen-koans.usv versus zen-koans.csv

• blog-posts.usv versus blog-posts.csv

• end-of-transmission.usv

Examples

USV with 2 units by 2 records by 2 groups by 2 files:

a␟b␟␞c␟d␟␞␝e␟f␟␞g␟h␟␞␝␜i␟j␟␞k␟l␟␞␝m␟n␟␞o␟p␟␞␝␜

Liners can prettify the visual display layout:

a␟b␟␞
c␟d␟␞
␝
e␟f␟␞
g␟h␟␞
␝␜
i␟j␟␞
k␟l␟␞
␝
m␟n␟␞
o␟p␟␞
␝␜

Parsing example with the USV Rust crate and its iterators:

use usv::*;
let input = "a␟b␟␞c␟d␟␞␝e␟f␟␞g␟h␟␞␝␜i␟j␟␞k␟l␟␞␝m␟n␟␞o␟p␟␞␝␜";
for f in input.files() {
    for g in file.groups() {
        for r in group.records() {
            for u in r.units() {
                println!(&u);
            }
        }
    }
}

Why use USV?

USV can handle data that contains commas, semicolons, quotes, tabs, newlines, and other special characters, all without escaping.

USV can format units/columns/cells and records/rows/lines and groups/tables/grids and files/schemas/folios.

USV aims to be an international standard, and has a official IETF RFCXML Internet Draft submitted.

USV uses Unicode characters that are semantically meaningful.

USV works well with any typical modern editor, font, terminal, shell, search, and language.

USV uses visible letter-width characters, and these are easy to view, select, copy, paste, search.

USV is easy and friendly

USV is intended to be easy to use and friendly to try.

USV works with many kinds of data, and many kinds of editors. Any editor that can render the USV characters will work. We use vim, emacs, helix, Zed, VS Code, JEOTrains IDEs, Nova, TextMate, Sublime, Notepad++, etc.

USV works with many kinds of tools. Any tool that can parse the USV characters will work. We use awk, sed, grep, rg, miller, etc.

USV works with many kinds of languages. Any language that can handle UTF-8 character encoding and rendering should work. We use C, C++, C#, Elixir, Erlang, Go, Java, JavaScript, Julia, Kotlin, Perl, PHP, Python, R, Ruby, Rust, Swift, TypeScript, etc.

Legal protection for standardization

The USV project aims to become a free open source IETF standard and IANA standard, much like the standards for CSV and TDF.

Until the standardization happens, the terms "USV" and "Unicode Separated Values" are trademarks of this project, and this repository is copyright 2022-2024. When IETF and IANA approve the submissions as a standard, then the trademarks and copyrights will go to a free libre open source software advocacy foundation.

Conclusion

USV is helping us with data projects. We hope USV may help you too.

We welcome constructive feedback about USV, as well as git issues, pull requests, and standardization help.

FAQ • RFC • Code • Comparisons • Criticisms • TODO • XKCD