container-runtime: New feature: ID compression for DataStores & DDSs
Key changes
- A new API IContainerRuntimeBase.generateDocumentUniqueId() is exposed. This API will opportunistically generate IDs in short format (non-negative numbers). If it can't achieve that, it will return UUID strings. UUIDs generated will have low entropy in groups and will compress well. It can be leveraged anywhere in container where container unique IDs are required. I.e. any place that uses uuid() and stores data in container is likely candidate to start leveraging this API.
- Data store internal IDs (IDs that are auto generated by FF system) will opportunistically be generated in shorter form. Data stores created in detached container will always have short IDs, data stores created in attached container will opportunistically be short (by using newly added IContainerRuntimeBase.generateDocumentUniqueId() capability)
- Similar DDS names will be opportunistically short (same considerations for detached DDS vs. attached DDS)
Implementation details
- Container level ID Compressor can now be enabled with delay. With such setting, only new IContainerRuntimeBase.generateDocumentUniqueId() is exposed (ID Compressor is not exposed in such case, as leveraging any of its other capabilities requires future container sessions to load ID Compressor on container load, for correctness reasons). Once Container establishes connection and any changes are made in container, newly added API will start generating more compact IDs (in most cases).
Breaking changes
- DDS names can no longer start with "_" symbol - this is reserved for FF needs. I've validated that's not an issue for AzureClient (it only creates root object by name, everything else is referred by handle). Our main internal partners almost never use named DDSs (I can find only 4 instances in Loop).
Backward compatibility considerations
- Data store internal IDs could collide with earlier used names data stores. Earlier versions of FF framework (before DataStore aliasing feature was added) allowed customers to supply IDs for data stores. And thus, files created with earlier versions of framework could have data store IDs that will be similar to names FF will use for newly created data stores ("A", ... "Z", "a"..."z", "AA", etc.). While such collision is possible, it's very unlikely (almost impossible) if user-provided names were at least 4-5 characters long.
- If application runs to these problems, or wants to reduce risks, consider disabling ID compressor via IContainerRuntimeOptions.enableRuntimeIdCompressor = "off".
Minor changes
- IContainerRuntime.createDetachedRootDataStore() is removed. Please use IContainerRuntime.createDetachedDataStore and IDataStore.trySetAlias() instead
- IContainerRuntimeOptions.enableRuntimeIdCompressor has been changes from boolean to tri-state.
fluid-framework: SharedObject classes are no longer exported as public
SharedObject and SharedObjectCore are intended for authoring DDSes, and thus are only intended for use within the Fluid Framework client packages. They is no longer publicly exported: any users should fine their own solution or be upstreamed. SharedObject and SharedObjectCore are available for now as @alpha to make this migration less disrupting for any existing users.
fluid-framework: ContainerSchema is now readonly
The ContainerSchema type is intended for defining input to these packages. This should make the APIs more tolerant and thus be non-breaking, however its possible for some users of ContainerSchema to use it in ways where this could be a breaking change: any such users should remove their mutations and/or use a different type.
fluid-framework: EventEmitterWithErrorHandling is no longer publicly exported
EventEmitterWithErrorHandling is intended for authoring DDSes, and thus is only intended for use within the Fluid Framework client packages. It is no longer publicly exported: any users should fine their own solution or be upstreamed. EventEmitterWithErrorHandling is available for now as @alpha to make this migration less disrupting for any existing users.
aqueduct: Deprecated PureDataObjectFactory.createRootInstance and replaced with PureDataObjectFactory.createInstanceWithDataStore
Deprecated: PureDataObjectFactory.createRootInstance
This was deprecated because PureDataObjectFactory.createRootInstance has an issue at scale. PureDataObjectFactory.createRootInstance used the old method of creating PureDataObjects with names. The issue was that simultaneous creations could happen, and the old api had no good way of dealing with those types of collisions. This version slightly improved it by resolving those collisions by assuming whatever datastore was created with the alias or rootDataStoreId would just return that datastore. This will work for developers who expect the same type of PureDataObject to be returned from the createRootInstance api, but if a potentially different PureDataObject would be returned, then this api would give you the wrong typing.
For a replacement api see PureDataObjectFactory.createInstanceWithDataStore.
New method PureDataObjectFactory.createInstanceWithDataStore
This was done as a replacement of PureDataObjectFactory.createRootInstance. This exposes the IDataStore interface in the form of [PureDataObject, IDataStore]. IDataStore provides the opportunity for developers to use the IDataStore.trySetAlias method. This can return 3 different scenarios Success, Conflict, or AlreadyAliased. These scenarios can allow the developer to handle conflicts as they wish.
aqueduct: PureDataObjectFactory.instantiateDataStore now returns IFluidDataStoreChannel
The return type of PureDataObjectFactory.instantiateDataStore was changed from FluidDataStoreRuntime to
IFluidDataStoreChannel.
container-definitions: Deprecate IDeltaManager.inbound and IDeltaManager.outbound
IDeltaManager.inbound was deprecated because it was not very useful to the customer and there are pieces of functionality that can break the core runtime if used improperly. For example, summarization and processing batches. Do not use the apis on this if possible. Data loss/corruption may occur in these scenarios in which IDeltaManger.inbound.pause() or IDeltaManager.inbound.resume() get called.
Deprecated IDeltaManager.outbound as it was not very useful to the customer and there are pieces of functionality that can break the core runtime if used improperly. For example, generation of batches and chunking. Op batching and chunking can be broken. Data loss/corruption may occur in these scenarios in which IDeltaManger.inbound.pause() or IDeltaManager.inbound.resume() get called.
Alternatives
- Alternatives to
IDeltaManager.inbound.on("op", ...)areIDeltaManager.on("op", ...) - Alternatives to calling
IDeltaManager.inbound.pause,IDeltaManager.outbound.pauseforIContainerdisconnect useIContainer.disconnect. - Alternatives to calling
IDeltaManager.inbound.resume,IDeltaManager.outbound.resumeforIContainerreconnect useIContainer.connect.
container-definitions: ILoaderOptions no longer accepts arbitrary key/value pairs
ILoaderOptions has been narrowed to the specific set of supported loader options, and may no longer be used to pass arbitrary key/value pairs through to the runtime.
container-definitions: Added containerMetadata prop on IContainer interface
Added containerMetadata prop on IContainer interface.
container-loader: Behavior change: IContainer.attach will be made retriable in the next release
The attach function on IContainer has been modified such that the container stay open on non-fatal errors. On failure of attach the developer should inspect IContainer.closed to see if the container has been closed. If not closed, the developer can retry calling attach.
The functionality is currently behind a configuration Fluid.Container.RetryOnAttachFailure which can be set to true to enable the new functionality.
In the next release we will default to the new behavior, and it will be possible to disable this behavior by setting Fluid.Container.RetryOnAttachFailure to false
container-loader: IParsedUrl does not accept null version
IParsedUrl previously claimed to accept null version to indicate that we should not load from a snapshot, but this was internally converted into undefined (thereby loading from latest snapshot). The typing has been updated to reflect this reality.
container-loader: Internal format of the string returned by container.serialize has changed.
serialize is being changed to align format with similar APIs. There are no changes in external behaviour.
data-object-base: LazyLoadedDataObject and LazyLoadedDataObjectFactory removed
LazyLoadedDataObject and LazyLoadedDataObjectFactory are not recommended for use and have been removed.
runtime-definitions: ITelemetryContext: Functions get and serialize are now deprecated
ITelemetryContext is to be used only for instrumentation, not for attempting to read the values already set by other code. This is important because this public interface may soon use FF's should-be internal logging instrumentation types, which we reserve the right to expand (to support richer instrumentation). In that case, we would not be able to do so in a minor release if they're used as an "out" type like the return type for get.
There is no replacement given in terms of immediate programmatic access to this data. The expected use pattern is something like this:
- Some code creates a concrete implementation of
ITelemetryContextand passes it around - Callers use the "write" functions on the interface to build up the context
- The originator uses a function like
serialize(on the concrete impl, not exposed on the interface any longer) and passes the result to a logger - The data is inspected along with other logs in whatever telemetry pipeline is used by the application (or Debug Tools, etc)
id-compressor: Deprecated ID compressor class has been removed from the public API.
This change should be a no-op for consumers, as there were already better static creation/deserialization functions for use and compressor types are generally unused outside the runtime.
counter: SharedCounter.create now returns ISharedCounter
SharedCounter.create now returns ISharedCounter instead of SharedCounter.
telemetry-utils: logIfFalse() is removed
The deprecated logIfFalse() function has been removed. Consumers who want to keep using it can reimplement in in their codebase by copying it from a previous version of the @fluidframework/telemetry-utils package.
replay-driver: Remove OpStorage export
The OpStorage class was non-functional, and has been removed.
core-interfaces: Removed ITelemetryProperties, TelemetryEventCategory, TelemetryEventPropertyType, and ITaggedTelemetryPropertyType
The ITelemetryProperties interface was deprecated and has been removed. Use the identical ITelemetryBaseProperties instead.
The TelemetryEventCategory type was deprecated and has been removed from @fluidframework/core-interfaces, since it had moved to @fluidframework/telemetry-utils in the past.
The TelemetryEventPropertyType type alias was deprecated and has been removed. Use the identical TelemetryBaseEventPropertyType instead.
The ITaggedTelemetryPropertyType interface was deprecated and has been removed. Use Tagged<TelemetryBaseEventPropertyType> instead.
API tightening
The Fluid Framework API has been clarified with tags applied to package exports. As we are working toward a clear, safe, and stable API surface, some build settings and imports may need to be adjusted.
Now: Most packages are specifying "exports" - import specifierss like @fluidframework/foo/lib/internals will become build errors. The fix is to use only public APIs from @fluidframework/foo.
Coming soon: Build resolutions (moduleResolution in tsconfig compilerOptions) will need to be resolved with Node16, NodeNext, or a bundler that supports resolution of named import/export paths. Internally, some FF packages will use @fluidframework/foo/internal import paths that allow packages to talk to each other using non-public APIs.
Final stage: APIs that are not tagged @public will be removed from @fluidframework/foo imports.
Resolved URLs no longer use non-standard protocols
Previously, IResolvedUrl.url could use a non-standard protocol like fluid://, fluid-odsp://, or fluid-test://. These have been replaced with https:// to permit standards-compliant URL parsing.
driver-definitions: Deprecate ISnapshotContents
ISnapshotContents is deprecated. It has been replaced with ISnapshot.
driver-definitions: repositoryUrl removed from IDocumentStorageService
The repositoryUrl member of IDocumentStorageService was unused and always equal to the empty string. It has been removed.
tree: Minor API fixes for "@fluidframework/tree" package.
Rename IterableTreeListContent to IterableTreeArrayContent, inline TreeMapNodeBase into TreeMapNode, rename TreeArrayNode.spread to TreeArrayNode.spread and remove create which was not supposed to be public (use TreeArrayNode.spread instead).
Deprecated error-related enums have been removed
Error-related enums ContainerErrorType, DriverErrorType, OdspErrorType and RouterliciousErrorType were previously deprecated and are now removed. There are replacement object-based enumerations of ContainerErrorTypes, DriverErrorTypes, OdspErrorTypes and RouterliciousErrorTypes. Refer to the release notes of Fluid Framework version 2.0.0-internal.7.0.0 for details on the replacements.
map, tree: DDS classes are no longer publicly exported
SharedMap and SharedTree now only export their factories and the interface types. The actual concrete classes which leak implementation details are no longer exported. Users of the SharedMap type should use ISharedMap. Users of the SharedTree type should use ISharedTree.
routerlicious-driver: Ephemeral containers now controlled in attach() call rather than as driver policy
Previously, ephemeral containers were created by adding an isEphemeralContainer flag in IRouterliciousDriverPolicies. Now, it is controlled by a createAsEphemeral flag on the resolved URL. See #19544 for an example of how to set this flag via your URL resolver.
runtime-definitions: FlushMode.Immediate is deprecated
FlushMode.Immediate is deprecated and will be removed in the next major version. It should not be used. Use FlushMode.TurnBased instead, which is the default. See https://github.com/microsoft/FluidFramework/tree/main/packages/runtime/container-runtime/src/opLifecycle#how-batching-works for more information
datastore-definitions: Add TChannel type parameter to IChannelFactory.
Add TChannel type parameter (which defaults to IFluidLoadable) to IChannelFactory. When left at its default this preserves the old behavior, however packages exporting IChannelFactory will now reference IFluidLoadable if not providing a different parameter and thus will implicitly depend on @fluidframework/core-interfaces.
datastore-definitions: IFluidDataStoreRuntime.logger is now an ITelemetryBaseLogger
IFluidDataStoreRuntime.logger is now an ITelemetryBaseLogger instead of the deprecated ITelemetryLogger. The sendTelemetryEvent(), sendErrorEvent(), or sendPerformanceEvent() methods were not intended for users of IFluidDataStoreRuntime. You can keep using the logger's send() method to generate telemetry.
odsp-driver: createNavParam removed from odspDriverUrlResolverForShareLink
Remove deprecated method: createNavParam from odsp-driver's odspDriverUrlResolverForShareLink.
odsp-driver: Removed deprecated implementation of SingleRT feature
Removed the deprecated logic of creating sharing-links with container attach (called SingleRT) which was enabled via enableShareLinkWithCreate flag in HostStoragePolicy. This change removes SharingLinkTypes interface definition, removes other deprecated properties from the odsp-driver's resolvedUrl object and also removes the enableShareLinkWithCreate flag. The newer version of SingleRT feature continues to exist, which can be enabled via enableSingleRequestForShareLinkWithCreate feature flag in HostStoragePolicy.
odsp-driver: Added a new method to odspDriverUrlResolverForShareLink class
Added a new method called appendLocatorParams to odspDriverUrlResolverForShareLink class which appends locator params to the base URL provided.
runtime-definitions: Deprecated ID compressor related types have been removed.
This change should be a no-op for consumers, as these types were almost certainly unused and are also available in the standalone package id-compressor (see #18749).
runtime-definitions: Moved ISignalEnvelope interface to core-interfaces
The ISignalEnvelope interface has been moved to the @fluidframework/core-interfaces package.
@fluidframework/data-object-base demoted to experimental status
@fluidframework/data-object-base is now @fluid-experimental/data-object-base, and is not recommended for use in production. Prefer to use the data object classes from @fluidframework/aqueduct.
core-interfaces: Removed deprecated telemetry event types
The deprecated ITelemetryErrorEvent, ITelemetryGenericEvent, and ITelemetryPerformanceEvent interfaces, which represented different kinds of telemetry events, were not intended for consumers of Fluid Framework and have thus been removed. ITelemetryBaseEvent is the only telemetry event interface that should be used in/by consuming code.
ITelemetryLogger was not intended for consumers of Fluid Framework and has been removed. Consumers should use the simpler ITelemetryBaseLogger instead.
@fluidframework/ink package demoted to @fluid-experimental/ink
The ink package has always been in an experimental state, but this has not been clearly communicated (primarly just through notes in TODO.md). It has now been moved to the @fluid-experimental scope to more clearly indicate its development state.
@fluidframework/view-interfaces package removed
The view-interfaces package has been removed without replacement. The mountable view interfaces have been moved to the example-utils directory of the FluidFramework repo and may be used as a reference if needed, though this pattern is not recommended.
@fluid-experimental/react-inputs removed
This package was experimental and has been removed. No replacement is provided, but the patterns from the example packages can be used to instruct binding to a view.
@fluid-tools/webpack-fluid-loader no longer published
The @fluid-tools/webpack-fluid-loader package is no longer published as it is not a recommended pattern. Please consult fluidframework.com for current example patterns.
Remove deprecated package @fluid-tools/fluidapp-odsp-urlresolver
The FluidAppOdspUrlResolver class is now incorporated into the @fluidframework/odsp-urlresolver package.
@fluidframework/view-adapters package removed
The view-adapters package has been removed without replacement. The MountableView class has been moved to the example-utils directory of the FluidFramework repo and may be used as a reference if needed, though this pattern is not recommended.
@fluidframework/mocha-test-setup moved to @fluid-internal/mocha-test-setup
The mocha-test-setup package is intended to aid in testing internal to the FluidFramework repo, and should not be used outside of the repo.
data-object-base: LazyLoadedDataObject and LazyLoadedDataObjectFactory deprecated
LazyLoadedDataObject and LazyLoadedDataObjectFactory have been deprecated and are not recommended for use. For lazy loading of data objects, prefer to defer dereferencing their handles.
@fluidframework/dds-interceptions is now considered experimental
The DDS interception pattern was intended to support attribution scenarios but is now in process of being replaced by a different attributor approach. It is not recommended for use.
Updated @fluidframework/protocol-definitions
The @fluidframework/protocol-definitions dependency has been upgraded to v3.2.0.
Updated server dependencies
The following Fluid server dependencies have been updated to the latest version, 4.0.0. See the full changelog.
- @fluidframework/gitresources
- @fluidframework/server-kafka-orderer
- @fluidframework/server-lambdas
- @fluidframework/server-lambdas-driver
- @fluidframework/server-local-server
- @fluidframework/server-memory-orderer
- @fluidframework/protocol-base
- @fluidframework/server-routerlicious
- @fluidframework/server-routerlicious-base
- @fluidframework/server-services
- @fluidframework/server-services-client
- @fluidframework/server-services-core
- @fluidframework/server-services-ordering-kafkanode
- @fluidframework/server-services-ordering-rdkafka
- @fluidframework/server-services-ordering-zookeeper
- @fluidframework/server-services-shared
- @fluidframework/server-services-telemetry
- @fluidframework/server-services-utils
- @fluidframework/server-test-utils
- tinylicious