neilotoole/sq v0.48.5

on GitHub

Fixed

• #446: A bufio.ErrTooLong was being returned by bufio.Scanner, when splitting lines from input that was too long (larger than bufio.MaxScanTokenSize, i.e. 64KB). This meant that sq wasn't able to parse large JSON files, amongst other problems. The maximum buffer size is now configurable via the new tuning.scan-buffer-limit option. Note that the buffer will start small and grow as needed, up to the limit.

  $ sq config set tuning.scan-buffer-limit 64MB   # or 1024B, 64KB, 1GB, etc.
  

  A more useful error message is also now returned when the buffer limit is exceeded (the error suggests adjusting tuning.scan-buffer-limit).

Changed

• Renamed config option tuning.buffer-mem-limit to tuning.buffer-spill-limit. The new name better reflects the purpose of the option.