stdlib: organize the standard library modules (`lib`)

[Saem Ghani]

The lib directory, which contains the standard library, is a mess. it's difficult to find the module you're looking for without a "go to definition", or resorting to grepping file names. Instead the top level lib directory should strive to expose all public modules in a flat fashion, with private and implementation specific bits hidden behind sub-directories.

This ties in with the issue where the tests/stdlib use yet another "organization" scheme, see: #537.

This should be done either:

1. piece by piece in conjunction with tests/stdlib organizing
2. after tests/stdlib mirrors the current lib directory structure

As part of working through this issue, please note any observations useful for future projects (e.g.: many implementations of atomics).

[Clay Sweetser]

Some thoughts/comments, which may or may not be entirely related to this issue:

• Something the Python and Go standard libraries do is use target-agnostic 'front-end' modules that conditionally import target-specific 'back-end' modules. The front-end modules expose public APIs and include general implementation logic, and utilize back-end modules for target-specific tasks. For example, Go's subprocess module, exec, has exec.go, exec_windows.go, and exec_linux.go (among others). Adopting a similar design would get rid of the gigantic when statements that populate certain standard library modules and make it easier to determine what needs to be implemented to target a specific platform.
• I would argue that while a flat structure is ok in general, there should still be some folders for well-known categories. For example, "random" for random number generation implementations, "containers" for generic container data types, etc.

[Saem Ghani]

Some thoughts/comments, which may or may not be entirely related to this issue:

Thanks for sharing.

• Something the Python and Go standard libraries do is use target-agnostic 'front-end' modules that conditionally import target-specific 'back-end' modules. The front-end modules expose public APIs and include general implementation logic, and utilize back-end modules for target-specific tasks. For example, Go's subprocess module, exec, has exec.go, exec_windows.go, and exec_linux.go (among others). Adopting a similar design would get rid of the gigantic when statements that populate certain standard library modules and make it easier to determine what needs to be implemented to target a specific platform.

I at least somewhat agree, we can do some of this where it makes sense, which would likely be top level when statements, for smaller cases (i.e. inside routine definitions) when is fine. Put another way:

when for internal API bad, when for working around minor platform differences good.

• I would argue that while a flat structure is ok in general, there should still be some folders for well-known categories. For example, "random" for random number generation implementations, "containers" for generic container data types, etc.

Strongly disagree, if the canonical import is std/foo, then foo must have a public module that sits flatly in the directory. Once we've fixed module lookups with package prefixes we can then start rearranging things. Until that time, we should not add any name or organizational 'indirection' that obfuscates things.

[Clay Sweetser]

So, some open questions:

• Where should "private" (unlisted/not documented) modules used by both multiple standard library modules and multiple system modules go?
• Should private modules use any sort of naming convention to indicate that they should not be used?

[Saem Ghani]

So, some open questions:

Great questions.

• Where should "private" (unlisted/not documented) modules used by both multiple standard library modules and multiple system modules go?

If used by system, and by others, then putting them under the system directory isn't all bad, though I don't care for relative imports. Otherwise, if a sibling directory to system works, we've used the word common to describe common things, perhaps private might also work. Really does make the case for having package private. 😬

• Should private modules use any sort of naming convention to indicate that they should not be used?

Not right now, but maybe something will occur to us in the future as we make these changes and learn from them. Fewer changes in this regard will at least be kinder in terms of git history.