wader/fq v0.0.10 on GitHub

Changes

Add bplist Apple Binary Property List decoder. Thanks David McDonald @dgmcdona #427
Add markdown decoder. #422
Fix panic when interrupting (ctrl-c) JSON output (fq tovalue file etc), #440
Fix issue using debug (and some other native go iterator functions) inside path(...), which is used by assign (... = ...) expressions etc. #439
Fix issue and also make toactual and tosym work more similar to tovalue. #432
Fix issue with unknown fields (gaps found after decoding) where one continuous gap could end up split into two of more unknown fields. #431

More format documentation and also nicer help output. Also now all documentation is in markdown format. #430 #422

# or help(matroska) in the REPL
$ fq -h matroska
matroska: Matroska file decoder

Decode examples
===============

  # Decode file as matroska
  $ fq -d matroska . file
  # Decode value as matroska
  ... | matroska

Lookup element using path
=========================

  $ fq 'matroska_path(".Segment.Tracks[0)")' file.mkv

Get path to element
===================

  $ fq 'grep_by(.id == "Tracks") | matroska_path' file.mkv

References
==========
- https://tools.ietf.org/html/draft-ietf-cellar-ebml-00
- https://matroska.org/technical/specs/index.html
- https://www.matroska.org/technical/basics.html
- https://www.matroska.org/technical/codec_specs.html
- https://wiki.xiph.org/MatroskaOpus

Decoder changes

ar Allow empty integer strings. For example owner id can be an empty string. #428
bitcoin_blkdat Assert that there is a header. As the format is part of the probe group this speeds up probing. #402

bplist Add Apple Binary Property List decoder.

$ fq '.objects.entries[0] | .key, .value' Info.plist
    │00 01 02 03 04 05 06 07 08 09│0123456789│.objects.entries[0].key{}:
0x32│               5c            │     \    │  type: "ascii_string" (5) (ASCII encoded string)
0x32│               5c            │     \    │  size_bits: 12
    │                             │          │  size: 12
0x32│                  43 46 42 75│      CFBu│  value: "CFBundleName"
0x3c│6e 64 6c 65 4e 61 6d 65      │ndleName  │
    │00 01 02 03 04 05 06 07 08 09│0123456789│.objects.entries[0].value{}:
0x1ea│         5f                  │   _      │  type: "ascii_string" (5) (ASCII encoded string)
0x1ea│         5f                  │   _      │  size_bits: 15
0x1ea│            10               │    .     │  large_size_marker: 1 (valid)
0x1ea│            10               │    .     │  exponent: 0
0x1ea│               18            │     .    │  size_bigint: 24
    │                             │          │  size: 24
0x1ea│                  41 70 70 6c│      Appl│  value: "AppleProResCodecEmbedded"
0x1f4│65 50 72 6f 52 65 73 43 6f 64│eProResCod│
0x1fe│65 63 45 6d 62 65 64 64 65 64│ecEmbedded│

Supports torepr

$ fq torepr.CFBundleName Info.plist
"AppleProResCodecEmbedded"

elf

More robust decoding when string tables are missing. #417

# extract entry opcodes and disassemble with ndisasm
$ fq -n '"f0VMRgIBAQAAAAAAAAAAAAIAPgABAAAAeABAAAAAAABAAAAAAAAAAAAAAAAAAAAAAAAAAEAAOAABAAAAAAAAAAEAAAAFAAAAAAAAAAAAAAAAAEAAAAAAAAAAQAAAAAAAAAAAAAEAAAAAAAAAAQAAAAAAIAAAAAAAsDxmvwYADwU=" | frombase64 | . as $b | elf | $b[.header.entry-.program_headers[0].vaddr:]' \
| ndisasm -b 64 -
00000000  B03C              mov al,0x3c
00000002  66BF0600          mov di,0x6
00000006  0F05              syscall

Now decodes program header notes. #421

markdown Add decoder. Is used in fq to render CLI help. #422

# array with all level 1 and 2 headers
$ fq -d markdown '[.. | select(.type=="heading" and .level<=2)?.children[0]]' README.md
[
  "fq",
  "Usage",
  "Presentations",
  "Install",
  "TODO and ideas",
  "Development and adding a new decoder",
  "Thanks and related projects",
  "License"
]

matroska Add support for sample lacing. Used by FLAC samples etc. #404

mp4 More codec names and also use official names from mp4ra.org. #424

# show details of first two track in file
$ fq -o line_bytes=10 '.tracks[0,1]' big_buck_bunny.mp4
        │00 01 02 03 04 05 06 07 08 09│0123456789│.tracks[0]{}: track
0x00910a│20 68 10 01 a0 40 0e 20 8c 1b│ h...@. ..│  samples[0:1295]:
0x009114│c2 2b 99 09 84 42 60 a8 c4 60│.+...B`..`│
*       │until 0x541697.7 (5473678)   │          │
        │                             │          │  id: 1
        │                             │          │  data_format: "mp4a" (MPEG-4 Audio)
        │00 01 02 03 04 05 06 07 08 09│0123456789│.tracks[1]{}: track
0x00a5e6│                           00│         .│  samples[0:1440]:
0x00a5f0│00 00 0c 06 00 07 8b 71 b0 00│.......q..│
0x00a5fa│00 03 00 40 80 00 00 00 15 06│...@......│
*       │until 0x540959.7 (5464939)   │          │
        │                             │          │  id: 2
        │                             │          │  data_format: "avc1" (Advanced Video Coding)

html Handle leading doc type and processing directives. #414

Changelog

a77cec9 Added documentation and tests, fixed bad date parsing
d784db6 Adds support for Apple Binary Plist, version 00
5711f29 Code fixes from PR, still need to add tests and testdata
6b04f2d Documentation cleanup
bcccde2 Fixes and embeds documentation
ebae938 Fixes bug in integer parsing
368d183 Size check on nBits to save memory
84ca101 Update docker-golang from 1.19.0 to 1.19.1
c47c386 Update github-go-version from 1.19.0, 1.19.0, 1.19.0 to 1.19.1
816169b Update github-golangci-lint to 1.50.0 from 1.49.0
21f2980 Update make-golangci-lint to 1.50.0 from 1.49.0
5f61994 adds function for decoding fixed sized arrays
f08f44f ar: Integer strings might be empty
004406d bitcoin_blkdat,bitcoin_block: Make sure there is a header if blkdat
421b2b3 bplist: Fix unknown field for singletons and add torepr tests
16b0121 bplist: Make torepr convert to values
fe64530 csv: Add tsv and header example
cb3dc80 decode,tar: Add scalar description and Try* helpers
a6429ff decode: Remove RangeSorted flag as we can decide on array/struct instead
a468684 deps: Manual update ones not using bump
a7a101c doc,help: Nicer format help and move help tests into each format
725ab1b doc,html,xml: Add more documentation and examples
abd19ed doc: Fix format sections a bit
0fdc03a doc: Fix some incorrect example prompts
5382d46 elf: Basic program header notes decoding
12105d8 elf: Treat missing string tables as empty to be more robust
3deceee fixes from PR comments
226a9a3 generics: Use more from x/exp
404b170 gojq: Update fq fork
376f0eb gojq: Update rebased fq fork
87b2c6c help,doc: Use markdown for format documentation again
8016352 html: Handle html with leading doctype etc
768df30 interp,decode: For struct use map to lookup field
c4219d6 interp: Fix interrupt panic for cli eval
00ee10a interp: Make to{actual,sym} behave similar to tovalue
00a5066 markdown: Add decoder
7749e1b matroska: Add proper lacing support
20a1537 mp4: Fix data_format typo
2655ba0 mp4: More codec names (from mp4ra.org)
7cd43b4 perfomance: increase performance by map usage
6a6fec5 range,decode: Use own range sort impl to speed up a bit
0f35fe4 ranges,decode: Correctly skip empty ranges when adding unknown fields
ea81efe readline: Update rebased fq fork
369f401 removed unneccessary type conversions
3198602 removed unused return type
7d86534 sortex: Package with type safe sort helpers
808202f test: Skip go test with -race by default
12836ab updates fqtest
1e47f4f updates tests post integer-bug fix
3d8ea1d updates torepr for data type
1385b5d wasm: Add some documentation
d6316d5 wav: Decode smpl chunk