Design core framework configuration system

[knoepfel]

DUNE US S&C R&D item 076

[knoepfel]

See comments by @brettviren re. Jsonnet TLAs:

• https://github.com/orgs/Framework-R-D/discussions/506#discussioncomment-16658615
• https://github.com/orgs/Framework-R-D/discussions/506#discussioncomment-16700045

[brettviren]

Another big issue for using Jsonnet is to use go-jsonnet instead of C jsonnet (which is behind the C++ bindings in libjsonnet++.so)

The Go Jsonnet library has optimizations that the Jsonnet team stated they will not back port to the C library. These optimizations can change Jsonnet compile times from hours to minutes, minutes to seconds. They are really worth the effort.

Switching from using libjsonnet.so to libgojsonnet.so is one part of the solution. The other part of the solution involves the C++ bindings which are currently only provided by the C/C++ "jsonnet" package and not (currently) by the go-jsonnet package. Wire-Cell Toolkit used to use the C++ bindings but in the end they were more hassle than they are worth and for a while now WCT has used the C API/ABI directly. Since the Go library provides these interfaces, it's trivial to swap to using go-jsonnet.

So, my suggestion is for Phlex to support go-jsonnet in one of two ways:

1. Drop the C++ interface and use the C API/ABI.

2. Keep the C++ interface and rely on a compatible C++ interface with go-jsonnet (tbd).

Option 1 does require a reworking of Phlex, probably just phlex.cpp's main().

Option 2 requires no change to Phlex and almost no change to Phlex-related Spack but does require inventing a C++ interface to go-jsonnet and introduction of a virtual "libjsonnet" to the Spack ecosystem.

If one or the other option here sounds acceptable, I can of course provide more details and perhaps PRs to the relevant phlex and/or spack related packages.

[brettviren]

FYI, I have started some packages that touch on this.

• https://github.com/brettviren/boost-config

This provides mechanisms to serialize between C++ struct and JSON or Jsonnet (or others) at the cost of calling BOOST_CONFIG_*(MyStruct) macros. It has some pure Python example script to dig out boost-config schema info from shared libraries. I have CMake call this so a package build can produce a family of .libjsonnet files that define functions that act as validating constructors of objects that match the schema defined by the C++ struct.

• phlex-config (not yet pushed to github)

This is my attempt to apply boost-schema to solidify the part of configuration that is common to algorithms such as specifying input/output products

• https://github.com/brettviren/dune-config

A pure-jsonnet package to provide a basis for configuring DUNE jobs using detector-independent job level configs that are parameterized by a per-detector, common-schema object.

These three things are being done in the context of wire-cell-phlex integration package. They are still rather in flux and may reflect diverging opinion. Let me bake them a bit longer. For now, just know there's some activity going on in this space. At some point I will seek some critical eyeballs.