github wader/fq v0.0.8
on GitHub

latest releases: v0.11.0, v0.10.0, v0.9.0...
22 months ago

Changes

  • Add support for some common structured serialization formats: #284 #335
    • XML, toxml, fromxml options for indent, jq mapping variants (object or array) and order preservation
    • HTML, fromhtml options for indent, jq mapping variants (object or array) and order preservation
    • TOML, totoml, fromtoml
    • YAML, toyaml, fromyaml
    • jq-flavored JSON (optional key quotes and trailing comma) tojq, fromjq options for indent #284 
      # query a YAML file
$ fq '...' file.yml

# convert YAML to JSON
# note -r for raw string output, without a JSON string with JSON would outputted
$ fq -r 'tojson({indent:2})' file.yml

$ fq -nr '{hello: {world: "test"}} | toyaml, totoml, toxml, tojq({indent: 2})'
hello:
    world: test

[hello]
  world = "test"

<hello>
  <world>test</world>
</hello>
{
  hello: {
    world: "test"
  }
}
$ echo '<doc><element a="b"></doc>' | fq -r '.doc.element."-a"'
b
$ echo '<doc><element a="b"></doc>' | fq -r '.doc.element."-a" = "<test>" | toxml({indent: 2})'
<doc>
  <element a="&lt;test&gt;"></element>
</doc>
    • CSV, tocsv, fromcsv options for separator and comment character 
      $ echo -e  '1,2\n3,4' | fq -rRs 'fromcsv | . + [["a","b"]] | tocsv'
1,2
3,4
a,b
  • Add support for binary encodings
    • Base64. tobase64, frombase64 options for encoding variants. 
      $ echo -n hello | base64 | fq -rRs 'frombase64 | tostring'
hello
    • Hex string. tohex, fromhex
  • Add support for text formats
    • XML entities toxmlentities, fromxmlentities
    • URL tourl, fromurl 
      $ echo -n 'https://host/path/?key=value#fragment' | fq -Rs 'fromurl | ., (.host = "changed" | tourl)'
{
  "fragment": "fragment",
  "host": "host",
  "path": "/path/",
  "query": {
    "key": "value"
  },
  "rawquery": "key=value",
  "scheme": "https"
}
"https://changed/path/?key=value#fragment"
    • URL path encoding tourlpath, fromurlpath
    • URL encoding tourlencode, fromurlencode
    • URL query tourlquery, fromurlquery
  • Add support for common hash functions:
    • MD4 tomd4
    • MD5 tomd5 
      $ echo -n hello | fq -rRs 'tomd5 | tohex'
5d41402abc4b2a76b9719d911017c592
    • SHA1 tosha1
    • SHA256 tosha256
    • SHA512 tosha512
    • SHA3 224 tosha3_224
    • SHA3 256 tosha3_256
    • SHA3 384 tosha3_384
    • SHA3 512 tosha3_512
  • Add support for common text encodings:
    • ISO8859-1 toiso8859_1, fromiso8859_1
    • UTF8 tutf8, fromutf8
    • UTF16 toutf16, fromutf16
    • UTF16LE toutf16le, fromutf16le
    • UTF16BE toutf16be, fromutf16be 
      $ echo -n 00680065006c006c006f | fq -rRs 'fromhex | fromutf16be'
hello
  • Add group function, same as group_by(.) #299
  • Update/rebase readline dependency (based on @tpodowd chzyer/readline#207) #305 #308
    • Less blinking/redraw in REPL
    • Lots of small bug fixes
  • Update/rebase gojq dependency #247
    • Fixes JQValue destructing issue (ex: <some object JQValue> as {$key})
  • Major rewrite/refactor how native function are implemented. Less verbose and less error-prone as now shared code takes care of type casting and some argument errors. #316
  • Add tojson($opts) that support indent option. tojson still works as before (no indent). 
    $ echo '{a: 1}' | fq -r 'tojson({indent: 2})'
{
  "a": 1
}
  • Rename --decode-file (will still work) to --argdecode be be more consistent with existing --arg* arguments. #309
  • On some decode error cases fq can now keep more of partial tree making it easier to know where it stopped #245
  • Build with go 1.18 #272

Decoder changes

  • bitcoin Add Bitcoin blkdat, block, transcation and script decoders #239
  • elf Use correct offset to dynamic linking string table #304
  • tcp Restructure into separate client/server objects and add skipped_bytes (number of bytes with known missing ACK), has_start (has first byte in stream) and has_end (has last byte in stream) per direction #251
    • Old: 
          │00 01 02 03 04 05 06 07│01234567│.tcp_connections[0]{}: tcp_connection
      │                       │        │  source_ip: "192.168.69.2"
      │                       │        │  source_port: 34059
      │                       │        │  destination_ip: "192.168.69.1"
      │                       │        │  destination_port: "http" (80) (World Wide Web HTTP)
      │                       │        │  has_start: true
      │                       │        │  has_end: true
 0x000│47 45 54 20 2f 74 65 73│GET /tes│  client_stream: raw bits
 0x008│74 2f 65 74 68 65 72 65│t/ethere│
 *    │until 0x1bc.7 (end) (44│        │
 0x000│48 54 54 50 2f 31 2e 31│HTTP/1.1│  server_stream: raw bits
 0x008│20 32 30 30 20 4f 4b 0d│ 200 OK.│
 *    │until 0x191.7 (end) (40│        │
    • New: 
          │00 01 02 03 04 05 06 07│01234567│.tcp_connections[0]{}: tcp_connection
      │                       │        │  client{}:
      │                       │        │    ip: "192.168.69.2"
      │                       │        │    port: 34059
      │                       │        │    has_start: true
      │                       │        │    has_end: true
      │                       │        │    skipped_bytes: 0
 0x000│47 45 54 20 2f 74 65 73│GET /tes│    stream: raw bits
 0x008│74 2f 65 74 68 65 72 65│t/ethere│
 *    │until 0x1bc.7 (end) (44│        │
      │                       │        │  server{}:
      │                       │        │    ip: "192.168.69.1"
      │                       │        │    port: "http" (80) (World Wide Web HTTP)
      │                       │        │    has_start: true
      │                       │        │    has_end: true
      │                       │        │    skipped_bytes: 0
 0x000│48 54 54 50 2f 31 2e 31│HTTP/1.1│    stream: raw bits
 0x008│20 32 30 30 20 4f 4b 0d│ 200 OK.│
 *    │until 0x191.7 (end) (40│        │
  • zip Add 64-bit support and add uncompress option #278
  • matroska Update and regenerate based on latest spec and also handle unknown ids better #291
  • mp4 Changes:
    • Fix PSSH decode issue #283
    • Add track for track_id references without tfhd box
    • Makes it possible to see samples in fragments without having an init segment.
      Note it is possible to decode samples in a fragment file by concatenating the init and fragment file ex: cat init frag | fq ....
    • Add senc box support #290
    • Don't decode encrypted samples #311
    • Add track_id to tracks #254
    • Add fairplay PSSH system ID #310
    • Properly handle trun data offset #294
    • Skip decoding of individual PCM samples for now #268
    • Add mvhd, tkhd, mdhd and mehd version 1 support #258
    • Make sure to preserve sample table order #330
  • fairplay_spc Add basic FairPlay Server Playback Context decoder #310
  • avc_pps Correctly check for more rbsp data

Changelog

  • 210940a Update docker-golang from 1.18.1 to 1.18.2
  • fbeabdc Update docker-golang from 1.18.2 to 1.18.3
  • 51a414d Update docker-golang from 1.18.3 to 1.18.4
  • 3017e8b Update github-go-version from 1.18.1, 1.18.1, 1.18.1 to 1.18.2
  • c597f7f Update github-go-version from 1.18.2, 1.18.2, 1.18.2 to 1.18.3
  • dd28392 Update github-go-version from 1.18.3, 1.18.3, 1.18.3 to 1.18.4
  • d10a361 Update github-golangci-lint from 1.45.2 to 1.46.0
  • 75b5946 Update github-golangci-lint from 1.46.0 to 1.46.1
  • 3ffa9ef Update github-golangci-lint from 1.46.1 to 1.46.2
  • 4be8cb9 Update github-golangci-lint from 1.46.2 to 1.47.0
  • 1b8f4be Update github-golangci-lint from 1.47.0 to 1.47.1
  • fc596a7 Update github-golangci-lint from 1.47.1 to 1.47.2
  • 62be922 Update gomod-BurntSushi/toml from 1.1.0 to 1.2.0
  • 5db7397 Update make-golangci-lint from 1.45.2 to 1.46.0
  • 456742e Update make-golangci-lint from 1.46.0 to 1.46.1
  • 0675711 Update make-golangci-lint from 1.46.1 to 1.46.2
  • 3d69e9d Update make-golangci-lint from 1.46.2 to 1.47.0
  • 2170925 Update make-golangci-lint from 1.47.0 to 1.47.1
  • c4199c0 Update make-golangci-lint from 1.47.1 to 1.47.2
  • 02f00be Update usage.md
  • 75169a6 asn1: Add regression test for range decode fix ##330
  • b0096bc avc_pps: Correct check if there is more rbsp data
  • 5d67df4 avro_ocf: Fix panic on missing meta schema
  • 417255b bitcoin: Add blkdat, block, transcation and script decoder
  • a6a9713 decode: Cleanup Try/ pairs
  • 3ce660a decode: Keep decode tree on RangeFn error
  • c4dd518 decode: Make compound range sort optional
  • 8bb4a6d decode: Range decode with new decoder to preserve bit reader
  • 342612e dev: Cleanup linters and fix some unused args
  • 78aa96b dev: Cleanup some code to fix a bunch of new linter warnings
  • 3570f1f doc: Add more related tools
  • 7aff654 doc: Clarify decode, slurp and spew args
  • 0863374 doc: Correct bencode spec URL
  • 10cc551 doc: Improve and cleanup text formats
  • b100611 doc: Typos and add note about Try* functions
  • c27646a doc: Update and shorten README.md a bit
  • b038872 doc: Use singular jq value to refer to jq value
  • a980656 doc: go 1.18 and improve intro text a bit
  • a64c28d dump: Skip JQValueEx if there are not options
  • 40481f6 elf,fuzz: Error on too large string table
  • f66a359 elf: Use correct offset to dynamic linking string table
  • 64f3e5c fairplay: Add basic SPC decoder and PSSH system id
  • cae288e format,intepr: Refactor json, yaml, etc into formats also move out related functions
  • e9d9f8a fq: Use go 1.18
  • 377af13 fqtest: Cleanup path usage
  • 2464ebc fuzz: Replace built tag with FUZZTEST env and use new interp api
  • 0f78687 gojq: Fix JQValue index and destructuring issue and rebase fq fork
  • 59c7d0d gojq: Rebase fq fork
  • c57dc17 gojq: Rebase fq fork
  • 9a7ce14 gojq: Update rebased fq fork
  • c1a0cda gojq: Update rebased fq fork
  • 32361de gojqextra: Cleanup gojq type cast code
  • 9b2e474 gojqextra: Simplify function type helpers
  • fd30209 hevc_vps,fuzz: Error on too many vps layers
  • efa5e23 icc_profile: Correctly clamp align padding on EOF
  • 1ddea1a interp,format: Refactor registry usage and use function helpers
  • a3c33fc interp: Add group/0
  • 95e6196 interp: Add internal is helpers
  • 3b717c3 interp: Add to/from for some common serialzations, encodings and hashes
  • 6b08800 interp: Cast jq value to go value properly for encoding functions
  • f5be518 interp: Cleanup and clarify some format naming
  • c770185 interp: Extract to/from map/struct to own package
  • 8dde3ef interp: Fix crash when including relatve path when no search paths are set
  • 735c443 interp: Improve type normalization and use it for toyaml and totoml
  • 81a014c interp: Make empty _finally fin error on error
  • 2dc509a interp: Refactor dump and revert #259 for now
  • ab8c728 interp: Rename --decode-file to --argdecode to be more consistent
  • dff3cc1 interp: dump: Fix column truncate issue with unicode bars
  • 5109df4 interp: dump: Show address bar for nested roots
  • 8021492 interp: help: Fix incorrect options example
  • 7671434 mapstruct: Handle nested values when converting to camel case
  • c92f4f1 matroska: Update ebml_matroska.xml and allow unknown ids
  • c2a359b mod: Update golang.org/x/{crypto,net}
  • 3780375 mp3: Use d.FieldValueU and some cleanup
  • 7b27e50 mp4,bitio: Fix broken pssh decoding and add proper reader cloning to bitio
  • 6b00297 mp4,senc: Refactor current track/moof tracking and add senc box support
  • 8228eca mp4: Add track id field and add track for tfhd with unseen track_id
  • ea2cc3c mp4: Don't decode encrypted samples
  • c6d0d89 mp4: Don't range sort samples, keep sample table order
  • 7d25fbf mp4: Properly use trun data offset
  • ba844eb mp4: Skip fields for pcm samples for now
  • 0e02bb6 mp4: iinf: Only assume sub boxes for version 0
  • 2e32818 mp4: mvhd,tkhd,mdhd,mehd: Add version 1 support
  • 44bab27 readline: Rebase on top of tpodowd's redraw/completion fixes PR
  • a512269 readline: Rebase on top of tpodowd's update PR
  • 54dcdce readline: Update fq fork
  • 6e7267d readme: add MacPorts install details
  • 76161a1 scalar,mp4,gzip,tar: Add timestamp to description
  • 9133f0e scalar: Add *Fn type to map value and clearer naming
  • 34cf544 tcp: Split into client/server structs and add skipped_bytes and has_start/end per direction
  • 1aaaefb wav,bencode,mpeg_ps_packet,id3v1: Random fixes
  • 47350e4 zip: Add uncompress=false test and some docs
  • e641274 zip: Add zip64 support and uncompress option
  • aa694e3 zip: s/Decompress/Uncompress/
Check out latest releases or
releases around wader/fq v0.0.8

Don't miss a new fq release

NewReleases is sending notifications on new releases.

Get notifications