Added
- Added an
offsetsargument tochunk()andChunker.__call__()that specifies whether to return the start and end offsets of each chunk (#9). The argument defaults toFalse. - Added an
overlapargument tochunk()andChunker.__call__()that specifies the proportion of the chunk size, or, if >=1, the number of tokens, by which chunks should overlap (#1). The argument defaults toNone, in which case no overlapping occurs. - Added an undocumented, private
_make_chunk_function()method to theChunkerclass that constructs chunking functions with call-level arguments passed. - Added more unit tests for new features as well as for multiple token counters and for ensuring there are no chunks comprised entirely of whitespace characters.
Changed
- Began removing chunks comprised entirely of whitespace characters from the output of
chunk(). - Updated
semchunk's description from 'A fast and lightweight Python library for splitting text into semantically meaningful chunks.' and 'A fast, lightweight and easy-to-use Python library for splitting text into semantically meaningful chunks.'.
Fixed
- Fixed a typo in the docstring for the
__call__()method of theChunkerclass returned bychunkerify()where most of the documentation for the arguments were listed under the section for the method's returns.
Removed
- Removed undocumented, private
chunk()method from theChunkerclass returned bychunkerify(). - Removed undocumented, private
_reattach_whitespace_splittersargument ofchunk()that was introduced to experiment with potentially adding support for overlap ratios.