gotd/td v0.164.0

on GitHub

This release is about clients that run for weeks: archivers, userbots and services with many accounts in one process. Most changes came from running gotd in such an archiver and from comparing it with Telegram Desktop on the wire. Release post: https://gotd.dev/blog/v0.164.0

Highlights

Lifecycle and bots

• Client.Start, Stop, Shutdown, Close, Wait, Done and telegram.Connect: use a client without wrapping everything in Run (#1948).
• telegram/bot: a batteries-included bot bootstrap with token validation (auth.ParseBotToken), a session and updates state in a directory (updates.FileStorage), missed updates after restart, and FLOOD_WAIT on login (#1950).

Knowing when something is wrong

• OnAuthLost when an authorized session is terminated, with a typed *AuthLostError from Run (#1906, #1924, #1935); banned, deleted and frozen accounts are detected (#1934).
• Connection events with DC and kind (#1918), per-request deadlines with resend (RequestTimeout, PingInterval, PingTimeout; #1907).
• OpenTelemetry metrics and debug logs for connections, requests, pools, update recovery and downloads (#1930–#1933).
• Wire tracer hooks (Options.WireTracer, #1937) and the wiretrace package and CLI to decode, analyze and diff traces, including Telegram Desktop traces with UI attribution (#1953).

Updates that don't go missing

• other_updates of updates.difference and channelDifference are applied instead of dropped; no recursion on unknown peers (#1941, with ports of #1854 by @igitur and #1864, #1866, #1867 by @vzd3v).
• StopOnApplyError (#1942, port of #1870 by @vzd3v), TrackChannel (#1898), BeforeRun buffer/drop (#1928), ErrStopped after Run (#1929), channelDifferenceTooLong messages passed to OnChannelGap (#1897).
• UpdateDispatcher copies share handlers (#1895); OnAny* and OnBefore (#1899, #1915).

Connections

• Acks for messages inside received containers (#1938); a clean session after in-place key regeneration (#1946); no DC-transfer deadlock (#1939).
• Run stops on permanent initConnection errors with *ConfigError (#1945); initConnection is re-sent after CONNECTION_NOT_INITED/CONNECTION_LAYER_INVALID (#1954).
• Slow answers: after the retry limit the engine keeps waiting for the result instead of failing (#1955).
• Client.FileDC per-DC pools (#1905), sub-connections without a global lock (#1894), home DC via the primary session (#1893).

Downloads

• Resume, known size and progress (#1901), Verify (#1902, #1952), file reference refresh (#1903, #1925), adaptive Scheduler and per-DC schedulers (#1904, #1917, #1920), CDN through any invoker (#1900, #1923).

Messages, queries, auth

• Layer 233 (#1872), HTML widget messages (#1947), entity order like TDLib (#1943), EntitiesFrom (#1896).
• Iterators: batch size is the server's page size and short pages no longer stop iteration (#1940); cursors, min/max id and entities (#1911); stories (#1912, #1926); min peers (#1910, #1916, #1921).
• QR login: 2FA via Options.Password (#1909), latch and refresh fixes (#1944), migration when the token is accepted on another DC (#1951). Takeout resume and TAKEOUT_INIT_DELAY (#1908).

Upgrading

Most changes are additions. Behaviour changes to check:

• Iterators (#1940) request the server's page size instead of 1 and don't stop on a short page; set BatchSize explicitly if you relied on the old value.
• Differences (#1941): other_updates are now applied, and updatePtsChanged triggers one extra getDifference.
• Run (#1945) returns *ConfigError on permanent initConnection errors (API_ID_INVALID and similar) instead of reconnecting forever.
• rpc.Engine.Do (#1955) no longer returns RetryLimitReachedErr; it waits for the result until the context is done or the connection closes.
• ExtractPeer returns peer.ErrMinPeer for min users and channels (#1910).
• updates.Manager.Handle and TrackChannel return updates.ErrStopped after Run stops (#1929).

Thanks to @igitur and @vzd3v for the updates fixes, and to everyone who reported issues.

All changes

• fix(telegram): create sub-connections per DC without global lock by @ernado in #1894
• feat(tg): add EntitiesFrom and EntitiesFromResult by @ernado in #1896
• fix(telegram): do not drop ready state when request completes first by @ernado in #1913
• feat(downloader): allow CDN downloads through any invoker by @ernado in #1900
• feat(downloader): refresh file reference on FILE_REFERENCE_* errors by @ernado in #1903
• feat(query): add DialogKey.InputPeer and min-aware ExtractPeer by @ernado in #1910
• fix(tg): share UpdateDispatcher handlers between copies by @ernado in #1895
• feat(tg): add unified OnAny* update handlers and OnBefore hook by @ernado in #1899
• feat(updates): add Manager.TrackChannel and track channels on ChannelTooLong by @ernado in #1898
• feat(telegram): report lost authorization via OnAuthLost by @ernado in #1906
• fix(telegram): use primary session for Client.DC to the home DC by @ernado in #1893
• feat(telegram): add Client.FileDC with cached per-DC file pools by @ernado in #1905
• feat(downloader): resume from offset, known size and progress by @ernado in #1901
• feat(downloader): verify local file against upload.getFileHashes by @ernado in #1902
• feat(takeout): resume saved sessions and handle TAKEOUT_INIT_DELAY by @ernado in #1908
• fix(updates): deliver channelDifferenceTooLong messages and report gap by @ernado in #1897
• feat(downloader): add adaptive Scheduler shared by downloads by @ernado in #1904
• feat(telegram): add request deadlines with resend and ping options by @ernado in #1907
• feat(qrlogin): complete 2FA via Options.Password by @ernado in #1909
• feat(query): add min/max id, resumable cursors and entities to iterators by @ernado in #1911
• feat(query): add stories query helpers by @ernado in #1912
• style: gofmt files missed by formatting by @ernado in #1914
• ci: run macOS tests natively on arm64 by @ernado in #1919
• fix(gen): generate update dispatcher state with handlers by @ernado in #1922
• feat(tg): pass topic IDs to OnAnyTyping and OnAnyRead* handlers by @ernado in #1915
• fix(peer): merge min users and channels into full ones in Fill by @ernado in #1916
• fix(downloader): report CDN-internal retries to Scheduler by @ernado in #1917
• feat(telegram): add per-DC download schedulers via Options.DownloadScheduler by @ernado in #1920
• test(telegram): cover CDN redirect through DownloadVia by @ernado in #1923
• fix(fileref): use channels.getMessages for channel from message by @ernado in #1925
• fix(query): return initial stories state on ForEach error by @ernado in #1926
• feat(telegram): report DC and connection kind via OnConnDead by @ernado in #1918
• fix(telegram): call OnAuthLost on its own goroutine by @ernado in #1924
• fix(telegram): recreate FILE_MIGRATE pool after failed auth transfer by @ernado in #1927
• fix(updates): return ErrStopped instead of blocking after Run stops by @ernado in #1929
• feat(updates): buffer or drop updates received before Run by @ernado in #1928
• fix(query): keep offset peer in sync for min peers by @ernado in #1921
• feat(downloader): add metrics and debug logs by @ernado in #1930
• feat(updates): add metrics and debug logs of gap recovery by @ernado in #1931
• feat(telegram): add OpenTelemetry metrics and debug logs by @ernado in #1932
• feat(telegram): pass client metrics and logger to downloads and schedulers by @ernado in #1933
• feat(telegram): detect banned, deleted and frozen accounts by @ernado in #1934
• chore(deps): bump golang.org/x/tools from 0.48.0 to 0.52.0 in /_tools in the golang group across 1 directory by @dependabot[bot] in #1848
• chore(deps): bump github.com/rogpeppe/go-internal from 1.15.0 to 1.16.0 by @dependabot[bot] in #1841
• chore(deps): bump github.com/pion/ice/v4 from 4.4.0 to 4.4.1 by @dependabot[bot] in #1846
• chore(deps): bump github.com/klauspost/compress from 1.19.1 to 1.19.2 by @dependabot[bot] in #1847
• fix(telegram): keep OnAuthLost stop reason when Run group is stopping by @ernado in #1935
• fix(html): use correct format verbs for user ID errors by @ernado in #1936
• feat(mtproto): add wire tracer hooks by @ernado in #1937
• fix(mtproto): ack content messages inside containers by @ernado in #1938
• fix(telegram): do not deadlock DC transfer setup on unready connection by @ernado in #1939
• fix(query): default iterator batch size to the server page size by @ernado in #1940
• fix(updates): apply other_updates from differences by @ernado in #1941
• feat(updates): allow stopping on apply errors by @ernado in #1942
• Update Telegram schema to the latest layer by @gotd-bot[bot] in #1872
• fix(entity): sort entities by offset, length and type like tdlib by @ernado in #1943
• fix(qrlogin): keep authorization from loginTokenSuccess, latch updates, retry refresh by @ernado in #1944
• fix(telegram): stop Run on permanent initConnection errors by @ernado in #1945
• fix(mtproto): start a clean session after in-place key regeneration by @ernado in #1946
• feat(message): support HTML widget messages by @ernado in #1947
• ci: temporarily disable macOS jobs by @ernado in #1949
• feat(telegram): add Start, Stop, Shutdown and Wait lifecycle by @ernado in #1948
• feat(bot): add batteries-included bot bootstrap by @ernado in #1950
• fix(qrlogin): migrate when the token is accepted on another DC by @ernado in #1951
• fix(downloader): stop hash requests at the end of the file by @ernado in #1952
• feat(wiretrace): add wire trace format, decoder and CLI by @ernado in #1953
• fix(manager): re-send initConnection after the server loses connection state by @ernado in #1954
• fix(rpc): keep waiting for the result after the retry limit by @ernado in #1955

Full Changelog: v0.163.0...v0.164.0