wader/fq v0.0.5 on GitHub

Improved binary slicing and bit reading, avro_ocf decoder (thanks @xentripetal), asn1_ber decoder, renamed display aliases, new grep_by and paste function.

Changes

Big internal bit reader refactor. Now much more consistent code and fixes some issues reading and decoding of binary arrays and binary slices. #123
- Bit reading and IO have been moved to a bitio package.
- Non-simple bit reading have been move out of bitio to decode package.
- [0,1,1,0,0,1,1,0,0,1,1,1,0,0,0,1 | tobits] | tobytes | tostring returns "fq".
- [.frames[0], .frames[-1]] | mp3 decode mp3 based on first and last frame of other mp3.
Add grep_by that recursively selects using a filter condition and ignores errors. #102
- grep_by(.type == "trak") finds all objects where .type is "trak" (all mp4 track boxes).
- grep_by(tonumber | . >= 40 and . <= 100) find all numbers between 40 and 100.
- grep_by(format == "jpeg") find all jpegs.
Add paste function to read string from stdin util ^D (EOF). Can be used to paste in REPl etc. #143
- paste | frompem | asn1_ber | repl wait for PEM encoded text on stdin (paste and press ^D), decode it with asn1_ber and start a sub-REPL with the result.
- paste | fromjson decode pasted JSON.
- eval(paste) eval pasted jq expression.
Cleanup display aliases. Remove v and f, add da, dd, dv and ddv. #112
- d/d($opts) display value and truncate long arrays and buffers
- da/da($opts) display value and don't truncate arrays
- dd/dd($opts) display value and don't truncate arrays or buffers
- dv/dv($opts) verbosely display value and don't truncate arrays but truncate buffers
- ddv/ddv($opts) verbosely display value and don't truncate arrays or buffers
Refactor radix into toradix($base)/fromradix($base). #139
Remove number_to_bytes. Can be done with tobytes. #139
Change tobytes to zero pad most significant bits to byte alignment if needed. #133
Add tobytes/tobits variant that takes an argument to add extra padding. #133
- 0xf | tobytes 8 bit binary with last 4 bits set
- 0xf | tobytes(4) 32 bit binary with last 4 bits set
- 0xf | tobits(12) 12 bit binary with last 4 bits set
Rename fq type buffer to binary as it makes more sense. #133
Add topem/frompem to work with PEM encoding. #92
Add Windows scoop install. #137 Thanks @thushan
Add display, decode value, binary and binary array documentation. #118 #136 #133
Add decode API documentation. #149
Improved REPL completion for keys. #144
Add -o force=<bool> option that sets force decode option. Same as mp4({force: true}). #126

Decoder changes

avro_ocf Add decoder. #38 Thanks @xentripetal
- Full avro OCF support. Handles all primitive, complex, and logical types besides decimals.
- Able to handle deflate, snappy, and null codecs for blocks.
asn1_ber Add decoder. #92
- Also decodes CER and DER (X.690) but with no additional validation at the moment.
- Support all types but real type is currently limited to range for 64 bit interger/float.
- Has torepr support.
- No schema support.
aac_frame Only decode object types we know about. #130
mp3 Shorter sync find heuristics. #104
mp4 Add stz2 support
mp4 Add pnot (preview container) and jP (JPEG 2000) signature. #125

Also thanks to @Doctor-love for keeping things tidy.

Changelog

6fc1efd Add test case with all data types
ae4a624 Adds Windows Scoop instructions for fq.
4b809a7 Change avro codec to funcs
66ca1f1 Change tests to use new verbose syntax
7345b8c Cleanup
07ddf36 Cleanup for linting
4508241 Cleanup snappy
0909fb6 Comment on snappy decompression
21cfc70 Dates need to specify UTC too
75b8496 Fix lint
7a8e3ca Hook into registry, add codecs
251053e Initial pass on logical types
2605bce Lint and add basic doc
ee18407 Parse header using avro decoders. Still not certain this is the best idea. Will get opinions before finalizing.
ab50088 Polish of problem template and clarifying questions
27789f2 Regenerate docs
5a1d35e Remove redudant question and fix typo
31c4c0d Support snappy and deflate codecs
0300c95 Take heading off doc to match make doc format
6f57cdb Timestamps should be UTC
06085a2 Undo change to doc/file.mp4. I have no idea how this got changed in the first place? Maybe some macos shenanigans.
d137a72 Update docker-golang from 1.17.6 to 1.17.7
267e30e Update github-go-version from 1.17.6, 1.17.6 to 1.17.7
1e859cd Update github-golangci-lint from 1.43.0 to 1.44.0
16849c8 Update linting
d02d896 Update make-golangci-lint from 1.43.0 to 1.44.0
68e85a2 Use existing scalar description helper
3bab3d6 aac_frame: Only try decode object types we know about
0829c16 asn1_ber: Add decoder
0312c92 asn1_ber: Add more doc and multiple outputs for frompem
06245d1 binary,decode,doc: Rename buffer to binary and add some documentation
7c52153 bitio,decode: Refactor bitio usage and make buffer slicing more correct
0d74e87 bitio,doc: Even more cleanup
d854ed5 bitio: Cleanup documentation a bit
82aeb35 bitio: More doc cleanup
01ecde6 bump: Add snappy config
de64a99 cleanup some docs, change enum to mapper, error zigzag on more than 8 bytes
6cd1c38 decode,scalar: Add scalar.Str{Uint/Int/F}ToSym to parse numbers
4ab6381 decode: Add scalars args to FieldRootBitBuf
be71eb0 decode: Rename LenFn() to LimitedFn, add FramedFn and document
7bc2521 doc,interp: Add some example usages to cli help
8e47fb1 doc,matroska: Fix filesname in example
c15f528 doc: Add format links to format table
b86da7a doc: Add inital decoder API documentation
49c90f8 doc: Add macOS security notes and move supported format up a bit
06b67e4 doc: Add more license details
a8664ed doc: Add per format documentation
0955262 doc: Add snappy license
3630785 doc: Cleanup and add more decode value and binary documentation
710c29b doc: Color edges in diagram based on dest
f0ce717 doc: Document display and some more jq hints
b350468 doc: More decode API details and polish
6b51b06 doc: More display alias leftover fixes
dd3e40f doc: Unbreak formats_digaram.jq since radix change
c52a1a2 doc: Use f($a; $b) instead of jq f/2 notation
233d86a fq: Add arch and os to --version
b8efd8e fuzz: Fuzz all formats
e1bdfdf fuzz: List seed numbers and make it build again
6090b65 fuzz: Make it compile again and run one format per fuzz
aea4884 github: Add basic issue template
b55ca2c gojq: Rebase fq branch
47c978e goreleaser: Use zip for macos
8537117 id3v2: Should assert not validate magic
d6ca481 initial work for avro OCF files
ca68e6a interp: Add Platform() method to OS interface
e792598 interp: Add grep_by/1 to recursively match using a filter
0a1a561 interp: Add missing default opts for tovalue
48a19cb interp: Add paste function to allow pasting text into REPL etc
fc0aacb interp: Cleanup display aliases, now: d, da, dd, dv, ddv
bf7a483 interp: Fix handling of group decode error from stdin
26d9650 interp: Refactor radix* into toradix($base)/fromradix($base)
366f6b1 interp: Support force decode as -o force=true
77ab667 interp: Use absolute path in errors
c31ec2a interp: Use correct sym color
898dfec lint: Fix typeassert and case exhaustive warnings
d540116 make doc
bf170be make: Cleanup some not very used targets
8d2d88f mp3: Decrease max sync seek length between frames to 4k
d555c32 mp4,fuzz: Fatal error on infinite sgpd box entries
45b00aa mp4: Add stz2 support
092609b mp4: Add video preview (pnot) and JPEG 2000 (jP) signatures
febce5a mpeg_spu: Fatal error on infinite loop
c58ba28 mpeg_spu: Fatal error on unknown cmd
d1943da pcapng,fuzz: Fix infinite loop by fatal error on block length <= 0
2ab395a protobuf: Add note about sub message decoding
af05381 repl,interp: Make stdio work during completion
bd9be2c repl: Fix completion of non-underscore extkeys
69c745d simplify scalar usage
778a1a4 zip: Assert signature not validate

(Some commits have been removed from list for clarity)