This release introduces various improvements to the cue command, including a streamlined CLI interface and improved interoperability between different file formats.
Many of the command-line flags were previously rather arbitrarily implemented for some commands and not others. This release fixes most of that. This release also makes it possible to convert between different formats, such as YAML, JSON, JSON Schema (input only), OpenAPI, and Protbuf (input only).
This release also starts a new versioning strategy. CUE exists of several fairly independent parts. Each next milestone focusses on a one of these parts. This round it was the cue command-line tool. Next round will be the evaluator. Each next minor version will correspond to one such milestone. This is not to say that upgrades and bug fixes cannot happen to other parts within a cycle—they can—but it helps focus.
The new strategy will also make it easier to do intermittent releases. We will use the minor releases to signify big additions and backwards-incompatible changes. Patch releases may include backwards-incompatible changes that include bug fixes or disabling previously deprecated features. We intent to follow this trategy until providing a general backwards-compatibility guarantee for v1.0.0. As usual, we intend to provide a smooth migration path for each incompatible language change.
As of this release, a Docker image is generated as part of the release process.
Define file types
A frequently asked feature was to be able to interpret files as anything. CUE now allows qualifiers to "cast" a file to any supported format. For instance,
$ cue eval json: foo.data
interprets foo.data as JSON and then prints it as CUE. This also shows that the input for eval no longer needs to be CUE but can be any format that is convertible to CUE.
This means that CUE can now also be used between non-CUE formats. For instance,
$ cue export foo.json -o foo.yaml
will convert JSON to YAML.
Packages can now be qualified by package name (for instance ./foo:bar), not just directory.
Run cue help inputs and cue help filetypes for more information.
Streamlined flags
The import command already had a set of flags that maps JSON and YAML files to a prescribed location within a CUE configuration. These flags are now also available for many other commands so that JSON and YAML files, as well as other formats, can be processed by the cue command without first having to convert them to CUE.
For instance,
$ cue eval -l 'strings.ToLower(kind)' -l name foo.yaml
puts any document in foo.yaml at a path based on its kind and name fields.
Also the output flags (file and format), among others, have been harmonized across commands. This sometimes led to backwards-incompatible changes.
Run cue help flags for more information.
CUE as a data format
The qualifier mechanism can also be used to restrict what language constructions can be used in .cue files. Most notably,
$ cue eval data: foo.cue
specifies that the file foo.cue may only contain data fields, but no references, definitions, builtin invocations, or any other CUE construct other than simple data. In other words, this allows CUE to be used as an alternative JSON or YAML format.
In the future we may add a file extension to indicate this CUE dialect.
New cue def command
The cue eval command was configured to generate cue that seemed pleasing to read to humans, but it would generate something that is neither useful as data nor as a schema. cue export could already be used to generate data, but there was nothing to conveniently generate a schema.
cue def is also the command of choice converting schema-based formats to CUE.
Value injection
Many users have requested a convenient way to defining environment-specific variants of configurations and ways to inject values based on environemt variables and command line flags.
We have added support for environment variables at various points in the tooling layer. Due to their non-hermetic nature, though, environment variables are not an appropriate mechanism for CUE configurations in general.
An important criterion for allowing such injection was retaining the ability to analyze configurations and make intent of how to use a configuration explicit. Simultaneously, we did not want to impose on the structure of a configuration.
Our solution is to use attributes to explicitly mark injection points. A field that with a "tag" attribute of the following form:
environment: "prod" | "staging" @tag(env)
can be set to "prod" on the command line using -t env=prod. The tag attribute may also explicitly define some shorthands. For instance, by specifying
environment: "prod" | "staging" @tag(env,short=prod|staging)
the same can be accomplished using -t prod.
This mechanism makes it very clear to both the user and the cue tooling which non-concrete values may be set by the user and which should be resolved by values within a configuration.
In the future we may consider to also allow whole-file injection analogous to Go build tags.
This release does not include support for command-line flags in the tooling layer. This is still being considered for a future release.
Run cue help injection for more information.
OpenAPI support
The cue tool now has basic support for converting OpenAPI definitions to CUE. This is a first stab, does not yet support external references, and is not as feature rich as the CUE-to-OpenAPI Go API that is used by Istio to generate its OpenAPI. But it should be a good start.
OpenAPI definitions can be imported using cue import or just used on the fly using cue eval or cue def. OpenAPI can be defined in JSON or YAML. For instance:
$ cue def openapi: foo.yaml
will interpret the foo.yaml file as OpenAPI and generate corresponding CUE.
It is also possible to convert CUE to OpenAPI. OpenAPI can be represented in JSON, YAML, or even CUE. For instance,
$ cue def ./pkg -out openapi+yaml
writes the evaluated CUE as OpenAPI in YAML format to stdout.
JSON Schema support
The cue tool has very basic support for mapping JSON Schema to CUE. This is still experimental and the converse (mapping CUE to JSON Schema) is not yet supported. The typical conventions followed in OpenAPI make it easy to find a mapping from OpenAPI to CUE. The same does not hold for JSON Schema. This is partly complicated by fields and definitions sharing the same namespace in CUE. Also, the top-level schema of JSON Schema is unnamed, making it hard to find an appropriate destination.
There is a proposal for CUE to have definitions and fields live in a separate namespace. This will make such a mapping considerably easier. For now, consider the mapping experimental.
As with OpenAPI, external references and anchors are not yet supported.
Auto detection of file types
The cue tool now interprets files with a .json, .yaml and .yml extension as either data, OpenAPI, or JSON Schema based on markers in the data. For JSON Schema this is a $schema field that refers to a schema on json-schema.org. OpenAPI is selected if a file contains a top-level openapi field with major version 3, an info.title, and an info.version field.
The use of qualifiers disables auto detection. So auto detection for a JSON file, for instance, can be disabled using a json: or data: qualifier.
cue import modes
The cue import tool now has import modes that specify which kinds of files to import. The default is the old default: JSON and YAML files, but with auto detection enabled.
Run cue help import for more details.
Protobuf support
The new cue import proto command can import Protobuf definitions.
A successful run imports all transitive dependencies of a .proto file into a CUE module: .proto files located within the module are converted to a CUE package at that location. The package name is derived from the go_package option, if present. Imported .proto files that can be found in a given include directory are put in the corresponding path in the cue.mod/gen directory.
The existing Protobuf API has been modified to generate CUE types hierarchically (instead of mapping all files to the top level) to make them closely correspond to the Protobuf definitions.
Protobuf definitions can be annotated with CUE-specific options to specify additional constraints for fields.
Command and Task changes
The task section in commands is now optional. A task can be included anywhere in a command. So commands with a single task can now be written as
import "tool/cli"
command: echo: cli.Print & {
text: "Hello world!"
}
The cue tool will look for tasks anywhere nested within the run command where each task is considered a leaf node in this nesting.
Tasks can also be defined at the top of the file. These will only be run if they are referenced by a task defined in an executed command
Sometimes task must run after another task, even though it does not depend on any of its outputs. The $after field can be used in that case to explicitly mark a dependency.
See https://pkg.go.dev/cuelang.org/go@v0.1.0/pkg/tool?tab=doc for more information.
Vet changes
In addition to the flag restructuring, it is now possible to check a data file against a CUE package.
A backwards-incompatible change is that one now needs to use the -d flag, instead of the -e flag, to select a schema against which to compare. The -e flag was used by other commands to mean something different and this was cleaned up for consistency sake.
Trim safeguard
The cue trim command now checks that the resulting rewrite results in the same output when exported. This prevents a bug in the trim algorithm from inadventently compromising a configuration.
Run the command with the -i flag to bypass this check.
API additions
Go API:
cue/ast: Many new AST construction helperscue/literal: Added CUE number parsing utilitiescue: AddedLookupDef,Dereference,IsClosedand various other methods.cue:Fillnow acceptsValue.
CUE API
various additions to
pkg/encoding/yamlpkg/filepkg/listpkg/mathpkg/regexppkg/tool/execpkg/tool/os
Language Changes
- Added attribute declarations (
@foo(bar)can be used as a declaration.)
Backwards-incompatible changes
Changes in flags:
- The
--attributesflag has been renamed to--show-attributesto be consistent with the--show-hiddenand--show-optionalflags. - The
-eflag forcue vethas been renamed to-d. The-eflag was already used for other commands and means something similar, but different. This allows-eand-dto be simulatenously supported. - The
-oflag now always defines an output file and the-outflag now always defines an output format. - Block comment support was already deprecated for a while, but support has now been removed from the language.
API changes:
cue:Instance.Namewas changed toInstance.PkgName.cue:Iteratoris now returned as a pointer.cue:Syntaxnow includes a package clause if a package is defined.encoding/protobuf:- proper encoding of
oneOf, which previously required at least one of the one-of fields to be specified. - generated CUE types are now declared as definitions.
- generated CUE types are now organized hierarchically, mapping one-to-one to the hierarchy defined in the
.protofiles. It previously adopted the Go convention, mapping all types to the top-level.
- proper encoding of
Bug fixes and performance enhancements
There were several bug fixes. Some of them might break your configuration. For instance, a regression was introduced that allowed required imports to be omitted in *_tool.cue files.
Changelog
e6c7748 .github: small improvements to the bug template
58cdb77 .github: update issue templates
fb54e1b Add help text to show yaml export option
0227ac0 Fix typos in 'cue mod'
066ede6 all: gofmt -s
30a4bee all: make goimports happy
52cc7f1 ci: rebuild tip.cuelang.org on master commits
0846b93 cmd/cue/cmd: add --tags flag
d10e59f cmd/cue/cmd: add --with-context flag for import
116e9fd cmd/cue/cmd: add def command
af71ef2 cmd/cue/cmd: add injection help topic
5208956 cmd/cue/cmd: add safeguard to trim operation
67d084a cmd/cue/cmd: allow commands to be documented by comments
f89aa48 cmd/cue/cmd: allow decoding JSON Schema and OpenAPI
a920a11 cmd/cue/cmd: allow mix of data and package files
694303f cmd/cue/cmd: allow specifying field type for path flag
4d5f5c7 cmd/cue/cmd: allow task hierarchy
511584e cmd/cue/cmd: allow top-level tasks
38a114d cmd/cue/cmd: bail out from errors earlier
4d6e269 cmd/cue/cmd: consolidate placement logic to single file
e3a7a72 cmd/cue/cmd: control task dependency using before and after fields
4616a3b cmd/cue/cmd: enable placement logic for vet, export, and eval
3b17f6a cmd/cue/cmd: err on space-separated syntax
67e25d4 cmd/cue/cmd: eval flag changes
eb7c51d cmd/cue/cmd: factor out common flag code
7a02899 cmd/cue/cmd: fail in case flag evaluation fails
5a8279f cmd/cue/cmd: fix Windows compatibility of test
00757a9 cmd/cue/cmd: fix crash
8af2efe cmd/cue/cmd: fix description for --show-optional flag
27b9ca3 cmd/cue/cmd: fix export status and output
485f389 cmd/cue/cmd: fix stdin usage for import
6eed1e9 cmd/cue/cmd: harmonize output flags
e7e0139 cmd/cue/cmd: hoist expression logic into iterator
8fbf67e cmd/cue/cmd: improve $after semantics and errors
1974705 cmd/cue/cmd: introduce config struct
2beaa9c cmd/cue/cmd: introduce help topics for new CLI
de9cc69 cmd/cue/cmd: make error messages identical on Windows
6dfecc3 cmd/cue/cmd: make eval use output flags
dd87908 cmd/cue/cmd: make task references absolute
ed5689f cmd/cue/cmd: move export to filetypes
337e2d1 cmd/cue/cmd: move fmt to new infrastructure
6e48383 cmd/cue/cmd: move get to to use the ast
abb5401 cmd/cue/cmd: move import to filetypes
01fbfa1 cmd/cue/cmd: move vet to filetypes
bfdc423 cmd/cue/cmd: prevent infinite loops while building dependency graph
c82a793 cmd/cue/cmd: require imports for tooling
f8afaed cmd/cue/cmd: simplify usage of --path flag
f733746 cmd/cue/cmd: streaming instance iteration
dcffaef cmd/cue/cmd: support -e for export
d4ce8ef cmd/cue/cmd: support import of proto and openapi
77ffe9d cmd/cue/cmd: support openapi out and proto in
a32d5dd cmd/cue/cmd: track dependencies in definitions
d8757db cmd/cue/cmd: unshadow type names in get go
882d88d cmd/cue/cmd: update cmd documentation
88b9bd5 cmd/cue/cme: fix import error message regression
bd5faf9 cmd/cue: fix cue import -p
3db1509 cmd/cue: make fmt work on tool files
6ad04b3 cmd/cue: use module information if no -X version information specified
2b93e11 cue/ast/astutil: catch cycle for label expressions
e076cd1 cue/ast/astutil: improve LabelName logic
8c70185 cue/ast: add NewBinExpr helper
e2650e0 cue/ast: add NewBool helper
e803ab8 cue/ast: add NewLit helper
b236d4a cue/ast: add NewStruct helper
d4faf4b cue/ast: add PackageName method
e2e548b cue/ast: allow CommentGroups in NewStruct
8e11671 cue/build: remove unnecessary duplicate initialized
6409c68 cue/cmd/cue: hoist placeOrphan from import
33c7705 cue/cmd/cue: hoist placement logic from import
442518f cue/errors: make compile again with Go 1.12
a96b3bb cue/errors: remove unnecessary first slice length on list.Reset()
e7f46c0 cue/format: always remove space before ':'
d05ea95 cue/format: fix simplification bug
af31654 cue/format: fix string label rewriting
78c24c5 cue/format: only format valid identifiers
ffd21e6 cue/format: print empty struct on one line by default
9b2f6b0 cue/format: replace bulk labels with ellipsis if possible
cb1baf4 cue/literal: factor out number parsing code
fcdf3fe cue/load: allow mixing anonymous packages in files mode
28d5c83 cue/load: allow using path:pkgname on command line
1288f89 cue/load: expose error types
4fd96d9 cue/load: factor out addFiles.
86a67bc cue/load: fix test breakage for Windows
f6f7791 cue/load: remove unnecessary colon check
104d266 cue/load: remove unnecessary import alias
334313c cue/load: use build.File to support "-"
a16973a cue/parser: fix parsing bug
9b301e4 cue/scanner: completely remove block comment support
83fa8f4 cue: add '-local cuelang.org/go' flag for buildins.go
024f1f5 cue: add Dereference and fix value paths
53ca13f cue: add IsClosed method
c7c70a6 cue: add Subsume to replace Subsumes
060a3c8 cue: add __foo variants for predeclared identifiers
2ca34b3 cue: add a test case for list.Equals
d56457f cue: add fallback mode for unresolved disjunctions
d9e4b53 cue: allow a Value to be passed to Fill
2540e8a cue: allow alias reuse in subscopes
72d8a92 cue: allow export of schema and other options
b9d22d6 cue: allow schema mode in subsumption
58ada0e cue: define Fill on Value, LookupDef and bug fixes
1fecca8 cue: detect incomplete value for value method
cc60dee cue: do not evaluate list elem for evalPartial
1e76e22 cue: fail when converting invalid UTF-8 to a CUE string
847d37b cue: fix Issue 289
76a47bc cue: fix Value.Subsumes example to correct function naming convention
5119440 cue: fix bug that dropped optional fields in def
e084198 cue: fix compiler bug and improve debug print
5ad8510 cue: fix crash if yielded value is a disjunction
ab2c7e5 cue: fix detection of incomplete value
8108ef2 cue: fix export bug
8398107 cue: fix handling of incomplete calls
76252f4 cue: fix import usage detection for partially resolved files
72dba3d cue: fix marshaling bug
d06f2d4 cue: fix number output
aa5e0c7 cue: fix struct instance matching of references
0a6f7c9 cue: implement attribute declarations
442bf2d cue: interpret nil as _ or top
c3c1624 cue: keep path in API error value
1ffc615 cue: limit number of position errors
b992abe cue: prioritize missing field over incomplete errors
fe9722f cue: rename Name to PkgName and add DisplayName
31206eb cue: return pointer from Struct.Fields.
b0678ab cue: simplify numLit
f4649b2 cue: support export package clause and docs
a805655 cue: track incompleteness across marshal errors
4ec09a4 deps: upgrade cobra to v0.0.7 to use new SetIn/SetOut
0cb140e doc/ref/spec.md: reformat to deal with Hugo bug
4dd9630 doc/ref/spec: attribute changes
107863a doc/ref: fix a go-ism in the spec
6320089 doc/tutorial/basics: change title of section
0e4130f doc/tutorial/basics: remove reference to % operator
275105a doc/tutorial/kubernetes: a few fixes
7e3bdda doc/tutorial/kubernetes: fix eval command typos
3171634 doc/tutorials/basics: remove duplicate text
7739e72 doc/tutorials/basics: typo
d64abbe docker: add Dockerfile and .dockerignore
0308a53 encoding/gocode: make work for definitions
b756712 encoding/json: don't simplify hidden fields
e752347 encoding/json: fix Doc comments for Extract and Decode
f81c40b encoding/json: fix JSON import for numeric keys
671b956 encoding/jsonschema: add features to support OpenAPI
47d9870 encoding/jsonschema: expose API of JSON Schema decoder
60b4d91 encoding/jsonschema: fix Windows path issue
6cb0878 encoding/jsonschema: initial JSON Schema to CUE conversion
bd98b13 encoding/jsonschema: require explicit package name
3c437bd encoding/openapi: encode title, version, and contact as fields
f51c879 encoding/openapi: generate CUE expressions
64b39e6 encoding/openapi: implement Extract
5c2b086 encoding/openapi: make API backwards compatible for Istio
eac3ea2 encoding/openapi: more accurate handling of disjunctions
56509a5 encoding/openapi: sort schema alphabetically
ed90d00 encoding/openapi: support nested defintitions
439f857 encoding/openapi: supported nested definitions
09ad0d2 encoding/protobuf: allow specifying package
f9fa4d8 encoding/protobuf: define messages as closed
0797e46 encoding/protobuf: don't rely on verbatim Ident formatting
c3efeb0 encoding/protobuf: generate nested definitions
0f03725 encoding/protobuf: proper encoding of oneOf
08c8a8f encoding/protobuf: remove dependency on load
cf72796 encoding/yaml: eliminate use of ghodss yaml pkg
ede1e7d github: cache ~/go/pkg/mod
d9ca690 github: enable docker feature
2e67318 github: enable run goreleaser automatically when bump new tag
c24f394 github: fix lack of v prefix on build_args
d0a6eef go.sum: run go tidy
aae80a0 internal/cli: hoist cli parsing code from tool/env
7988b62 internal/cmd/qgo: update Copyright year to 2020
29e9d00 internal/diff: avoid cycle when diffing
5c3a17b internal/diff: fix bug that caused hang
1d1295b internal/encoding/json: implement Encode
40cc12c internal/encoding/yaml: fix go1.12 breakage
975776e internal/encoding: always format with tabs
6e8fdd4 internal/encoding: support cue file type
81a20d5 internal/encoding: support input validation
f9f8e62 internal/filetypes: add auto interpretation
9b1dff7 internal/filetypes: add options and mode defaults
38362f8 internal/filetypes: better error message for files without extension
bd2173e internal/filetypes: expose package test internally
26e63ec internal/filetypes: fix Windows breakage
429bd0f internal/filetypes: package for interpreting command line args
c1b5e62 internal: add MakeInstance
4805941 internal: add ToFile and ToExpr
b8cad1a internal: hoist attribute parsing logic
a494f2e internal: switch defaullt GenPath to be in cue.mod
0198c89 module: update google/go-cmp to v0.4.0
9e71832 pkg/encoding/yaml: request concrete data for syntax
a9b38db pkg/encoding/yaml: validate using instance check
9242bf5 pkg/file: Fix file.Glob to generate correct CUE structure
c441dd3 pkg/list: add sort functionality
df7da7b pkg/list: documentation fix.
0afa377 pkg/list: fix typo
fe7a58a pkg/list: pick default before flattening
7008195 pkg/math/bits: reduce maximum bit index
2db865d pkg/regexp: add Valid function
72f8e8a pkg/tool/exec: allow passing env vars in exec.Run
a3a4b71 pkg/tool/exec: fix stdin processing
05a3f7d pkg/tool/exec: use os.Stdin on null only
42346df pkg/tool/os: add tasks for environment variables
c22d46d pkg/tool/os: remove env manualipulate tasks
a7b4ca8 pkg/tool: improve error reporting
2d8560d pkg/tool: update documentation and remove $before
44df1a8 pkg/tool: use $id instead of kind to identify task types
d1cfa57 tools/trim: fix build breakage
c242505 tools/trim: fix trimming bug
fd90d12 tools/trim: undo faulty optimization