NEST & Elasticsearch.Net 7.0: Now GA!
After many months of work, two alphas and a beta, we are pleased to announce the GA release of the NEST and Elasticsearch.Net 7.0 clients.
The overall themes of this release have been based around faster serialization, performance improvements, codebase simplification, and ensuring parity with the many new features available in Elasticsearch 7.0.
Types removal
Specifying types within the .NET clients is now deprecated in 7.0, in line with the overall Elasticsearch type removal strategy.
In instances where your index contains type information and you need to preserve this information, one recommendation is to introduce a property to describe the document type (similar to a table per class with discriminator field in the ORM world) and then implement a custom serialization / deserialization implementation for that class.
This Elasticsearch page details some other approaches.
Faster Serialization
After internalizing the serialization routines, and IL-merging the Newtonsoft.Json package in 6.x, we are pleased to announce that the next stage of serialization improvements have been completed in 7.0.
Both SimpleJson and Newtonsoft.Json have been completely removed and replaced with an implementation of Utf8Json, a fast serializer that works directly with UTF-8 binary. This has yielded a significant performance improvement, which we will be sharing in more detail in a later blog post.
With the move to Utf8Json, we have removed some features that were available in the previous JSON libraries that have proven too onerous to carry forward at this stage.
-
JSON in the request is never indented, even if
SerializationFormatting.Indented
is specified. The serialization routines generated by Utf8Json never generate anIJsonFormatter<T>
that will indent JSON, for performance reasons. We are considering options for exposing indented JSON for development and debugging purposes. -
NEST types cannot be extended by inheritance. With NEST 6.x, additional properties can be included for a type by deriving from that type and annotating these new properties. With the current implementation of serialization with Utf8Json, this approach will not work.
-
Serializer uses
Reflection.Emit
. Utf8Json usesReflection.Emit
to generate efficient formatters for serializing types that it sees.Reflection.Emit
is not supported on all platforms, for example, UWP, Xamarin.iOS, and Xamarin.Android. -
Elasticsearch.Net.DynamicResponse
deserializes JSON arrays toList<object>
. SimpleJson deserialized JSON arrays toobject[]
, but Utf8Json deserializes them toList<object>
. This change is preferred for allocation and performance reasons. -
Utf8Json is much stricter when deserializing JSON object field names to C# POCO properties. With the internal Json.NET serializer in 6.x, JSON object field names would attempt to be matched with C# POCO property names first by an exact match, falling back to a case insensitive match. With Utf8Json in 7.x however, JSON object field names must match exactly the name configured for the C# POCO property name.
We believe that the trade-off of features vs. GA release has been worthwhile at this stage. We hold a view to address some of these missing features in a later release.
High to Low level client dispatch changes
In 6.x, the process of an API call within NEST looked roughly like this
client.Search()
=> Dispatch()
=> LowLevelDispatch.SearchDispatch()
=> lowlevelClient.Search()
=> lowlevelClient.DoRequest()
With 7.x, this process has been changed to remove dispatching to the low-level client methods. The new process looks like this
client.Search()
=> lowlevelClient.DoRequest()
This means that in the high-level client IRequest
now builds its own URLs, with the upside that the call chain is shorter and allocates fewer closures. The downside is that there are now two URL building mechanisms, one in the low-level client and a new one in the high-level client. In practice, this area of the codebase is kept up to date via code generation, so it does not place any additional burden on development.
Given the simplified call chain and debugging experience, we believe this is an improvement worth making.
Namespaced API methods and Upgrade Assistant
As the API surface of Elasticsearch has grown to well over 200 endpoints, so has the number of client methods exposed, leading to an almost overwhelming number to navigate and explore through in an IDE. This is further exacerbated by the fact that the .NET client exposes both synchronous and asynchronous API methods for both the fluent API syntax as well as the object initializer syntax.
To address this, the APIs are now accessible through sub-properties on the client instance.
For example, in 6.x, to create a machine learning job
var putJobResponse = client.PutJob<Metric>("id", c => c
.Description("Lab 1 - Simple example")
.ResultsIndexName("server-metrics")
.AnalysisConfig(a => a
.BucketSpan("30m")
.Latency("0s")
.Detectors(d => d.Sum(c => c.FieldName(r => r.Total)))
)
.DataDescription(d => d.TimeField(r => r.Timestamp))
);
This has changed to
var putJobResponse = client.MachineLearning.PutJob<Metric>("id", c => c
.Description("Lab 1 - Simple example")
.ResultsIndexName("server-metrics")
.AnalysisConfig(a => a
.BucketSpan("30m")
.Latency("0s")
.Detectors(d => d.Sum(c => c.FieldName(r => r.Total)))
)
.DataDescription(d => d.TimeField(r => r.Timestamp))
);
Notice the client.MachineLearning.PutJob
method call in 7.0, as opposed to client.PutJob
in 6.x.
We believe this grouping of functionality leads to a better discoverability experience when writing your code, and improved readability when reviewing somebody else's.
The Upgrade Assistant
To assist developers in migrating from 6.x, we have published the Nest.7xUpgradeAssistant
Nuget package. When included in your project and the using Nest.ElasticClientExtensions;
directive is added, calls will be redirected from the old API method names to the new API method names in 7.0. The result is that your project will compile and you won't need to immediately update your code to use the namespaced methods; instead you'll see compiler warnings indicating the location of the new API methods in 7.0.
This package is to assist developers migrating from 6.x to 7.0 and is limited in scope to this purpose. It is recommended that you observe the compiler warnings and adjust your code as indicated.
Observability and DiagnosticSource
7.0 introduces emitting System.Diagnostics.DiagnosticSource
information from the client, during a request. The goal is to enable rich information exchange with the Elastic APM .NET agent and other monitoring libraries.
We emit DiagnosticSource information for key parts of a client request via an Activity
event, shipping support for Id
, ParentId
, RootId
, as well as request Duration
.
To facilitate wiring this up to DiagnosticListener.AllListeners
, we ship both with static access to the publisher names and events through Elasticsearch.Net.Diagnostics.DiagnosticSources
as well as strongly typed listeners, removing the need to cast the object passed to activity start/stop events.
An example listener implementation that writes events to the console is given below
private class ListenerObserver : IObserver<DiagnosticListener>
{
public void OnCompleted() => Console.WriteLine("Completed");
public void OnError(Exception error) => Console.Error.WriteLine(error.Message);
public void OnNext(DiagnosticListener value)
{
void WriteToConsole<T>(string eventName, T data)
{
var a = Activity.Current;
Console.WriteLine($"{eventName?.PadRight(30)} {a.Id?.PadRight(32)} {a.ParentId?.PadRight(32)} {data?.ToString().PadRight(10)}");
}
if (value.Name == DiagnosticSources.AuditTrailEvents.SourceName)
value.Subscribe(new AuditDiagnosticObserver(v => WriteToConsole(v.Key, v.Value)));
if (value.Name == DiagnosticSources.RequestPipeline.SourceName)
value.Subscribe(new RequestPipelineDiagnosticObserver(
v => WriteToConsole(v.Key, v.Value),
v => WriteToConsole(v.Key, v.Value))
);
if (value.Name == DiagnosticSources.HttpConnection.SourceName)
value.Subscribe(new HttpConnectionDiagnosticObserver(
v => WriteToConsole(v.Key, v.Value),
v => WriteToConsole(v.Key, v.Value)
));
if (value.Name == DiagnosticSources.Serializer.SourceName)
value.Subscribe(new SerializerDiagnosticObserver(v => WriteToConsole(v.Key, v.Value)));
}
}
Using the following example
var pool = new SniffingConnectionPool(new[] { node.NodesUris().First() });
var settings = new ConnectionSettings(pool).SniffOnStartup();
var client = new ElasticClient(settings);
var x = client.Search<object>(s=>s.AllIndices());
Console.WriteLine(new string('-', Console.WindowWidth - 1));
var y = client.Search<object>(s=>s.Index("does-not-exist"));
Emits the following console output:
SniffOnStartup.Start |59e275e-4f9c835d189eb14a. Event: SniffOnStartup
Sniff.Start |59e275e-4f9c835d189eb14a.1. |59e275e-4f9c835d189eb14a. GET _nodes/http,settings
Sniff.Start |59e275e-4f9c835d189eb14a.1.1. |59e275e-4f9c835d189eb14a.1. GET _nodes/http,settings
SendAndReceiveHeaders.Start |59e275e-4f9c835d189eb14a.1.1.1. |59e275e-4f9c835d189eb14a.1.1. GET _nodes/http,settings
SendAndReceiveHeaders.Stop |59e275e-4f9c835d189eb14a.1.1.1. |59e275e-4f9c835d189eb14a.1.1. 200
ReceiveBody.Start |59e275e-4f9c835d189eb14a.1.1.2. |59e275e-4f9c835d189eb14a.1.1. GET _nodes/http,settings
Deserialize.Start |59e275e-4f9c835d189eb14a.1.1.2.1. |59e275e-4f9c835d189eb14a.1.1.2. request/response: Nest.DefaultHighLevelSerializer
Deserialize.Stop |59e275e-4f9c835d189eb14a.1.1.2.1. |59e275e-4f9c835d189eb14a.1.1.2. request/response: Nest.DefaultHighLevelSerializer
ReceiveBody.Stop |59e275e-4f9c835d189eb14a.1.1.2. |59e275e-4f9c835d189eb14a.1.1. 200
Sniff.Stop |59e275e-4f9c835d189eb14a.1.1. |59e275e-4f9c835d189eb14a.1. GET _nodes/http,settings
Sniff.Stop |59e275e-4f9c835d189eb14a.1. |59e275e-4f9c835d189eb14a. Successful low level call on GET: /_nodes/http,settings?timeout=2s&flat_settings=true
SniffOnStartup.Stop |59e275e-4f9c835d189eb14a. Event: SniffOnStartup Took: 00:00:00.1872459
Ping.Start |59e275f-4f9c835d189eb14a. HEAD /
SendAndReceiveHeaders.Start |59e275f-4f9c835d189eb14a.1. |59e275f-4f9c835d189eb14a. HEAD /
SendAndReceiveHeaders.Stop |59e275f-4f9c835d189eb14a.1. |59e275f-4f9c835d189eb14a. 200
ReceiveBody.Start |59e275f-4f9c835d189eb14a.2. |59e275f-4f9c835d189eb14a. HEAD /
ReceiveBody.Stop |59e275f-4f9c835d189eb14a.2. |59e275f-4f9c835d189eb14a. 200
Ping.Stop |59e275f-4f9c835d189eb14a. Successful low level call on HEAD: /
CallElasticsearch.Start |59e2760-4f9c835d189eb14a. POST _all/_search
SendAndReceiveHeaders.Start |59e2760-4f9c835d189eb14a.1. |59e2760-4f9c835d189eb14a. POST _all/_search
Serialize.Start |59e2760-4f9c835d189eb14a.1.1. |59e2760-4f9c835d189eb14a.1. request/response: Nest.DefaultHighLevelSerializer
Serialize.Stop |59e2760-4f9c835d189eb14a.1.1. |59e2760-4f9c835d189eb14a.1. request/response: Nest.DefaultHighLevelSerializer
SendAndReceiveHeaders.Stop |59e2760-4f9c835d189eb14a.1. |59e2760-4f9c835d189eb14a. 200
ReceiveBody.Start |59e2760-4f9c835d189eb14a.2. |59e2760-4f9c835d189eb14a. POST _all/_search
Deserialize.Start |59e2760-4f9c835d189eb14a.2.1. |59e2760-4f9c835d189eb14a.2. request/response: Nest.DefaultHighLevelSerializer
Deserialize.Stop |59e2760-4f9c835d189eb14a.2.1. |59e2760-4f9c835d189eb14a.2. request/response: Nest.DefaultHighLevelSerializer
ReceiveBody.Stop |59e2760-4f9c835d189eb14a.2. |59e2760-4f9c835d189eb14a. 200
CallElasticsearch.Stop |59e2760-4f9c835d189eb14a. Successful low level call on POST: /_all/_search?typed_keys=true
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
CallElasticsearch.Start |59e2761-4f9c835d189eb14a. POST does-not-exist/_search
SendAndReceiveHeaders.Start |59e2761-4f9c835d189eb14a.1. |59e2761-4f9c835d189eb14a. POST does-not-exist/_search
Serialize.Start |59e2761-4f9c835d189eb14a.1.1. |59e2761-4f9c835d189eb14a.1. request/response: Nest.DefaultHighLevelSerializer
Serialize.Stop |59e2761-4f9c835d189eb14a.1.1. |59e2761-4f9c835d189eb14a.1. request/response: Nest.DefaultHighLevelSerializer
SendAndReceiveHeaders.Stop |59e2761-4f9c835d189eb14a.1. |59e2761-4f9c835d189eb14a. 404
ReceiveBody.Start |59e2761-4f9c835d189eb14a.2. |59e2761-4f9c835d189eb14a. POST does-not-exist/_search
Deserialize.Start |59e2761-4f9c835d189eb14a.2.1. |59e2761-4f9c835d189eb14a.2. request/response: Nest.DefaultHighLevelSerializer
Deserialize.Stop |59e2761-4f9c835d189eb14a.2.1. |59e2761-4f9c835d189eb14a.2. request/response: Nest.DefaultHighLevelSerializer
ReceiveBody.Stop |59e2761-4f9c835d189eb14a.2. |59e2761-4f9c835d189eb14a. 404
CallElasticsearch.Stop |59e2761-4f9c835d189eb14a. Unsuccessful low level call on POST: /does-not-exist/_search?typed_keys=true
Response Interfaces removed
Most API methods now return classes and not interfaces; for example, the client method client.Cat.Help
now returns a CatResponse<CatAliasesRecord>
as opposed to an interface named ICatResponse<CatAliasesRecord>
.
In instances where methods can benefit from returning an interface, these have been left intact, for example, ISearchResponse<T>
.
So why make the change?
Firstly, this significantly reduces the number of types in the library, reducing the overall download size, improving assembly load times and eventually the execution.
Secondly, it removes the need for us to manage the conversion of a Task<Response>
to Task<IResponse>
, a somewhat awkward part of the request pipeline.
The downside is that it does make it somewhat more difficult to create mocks / stubs of responses in the client.
After lengthy discussion we decided that users can achieve a similar result using a JSON string and the InMemoryConnection
available in Elasticsearch.Net. We use this technique extensively in the Tests.Reproduce project.
Another alternative would be to introduce an intermediate layer in your application, and conceal the client calls and objects within that layer so they can be mocked.
Response.IsValid semantics
IApiCallDetails.Success
and ResponseBase.IsValid
have been simplified, making it easier to inspect if a request to Elasticsearch was indeed successful or not.
Low Level Client
If the status code from Elasticsearch is 2xx
then .Success
will be true
. In instances where a 404
status code is received, for example if a GET
request results in a missing document, then .Success
will be false
. This is also the case for HEAD
requests that result in a 404
.
This is controlled via IConnectionConfiguration.StatusCodeToResponseSuccess
, which currently has no public setter.
High Level Client
The NEST high level client overrides StatusCodeToResponseSuccess
, whereby 404
status codes now sets .Success
as true
.
The reasoning here is that because NEST is in full control of url and path building the only instances where a 404
is received is in the case of a missing document, never from a missing endpoint.
However, in the case of a 404
the ResponseBase.IsValid
property will be false
.
It has the nice side effect that if you set .ThrowExceptions()
and perform an action on an entity that does not exist it won't throw as .ThrowExceptions()
only inspects .Success
on ApiCallDetails
.
Breaking changes
Please refer to the documentation for a full list of public API breaking changes between 6.8.0 and 7.0.0 GA releases.
Bug Fixes
- #3852 Change
HighlightFieldDictionary
to use formatter - #3224 Change
GenericProperty.Index
to `bool?`` - #3673 Exception when aggregating dates, issue in deserialisation
- #3679 Fix utf8json escape characters, particularly quotes
- #3694 NEST removes null value from
after_key
Dictionary when aggregating - #3743 Fix deserialization of special characters
- #3819, #3820 Use
CultureInfo.InvariantCulture`` when writing values in
GeoWKTWriter`` - #3825 Use
Marshal.SecureStringToGlobalAllocUnicode
- #3883 Fix
CatHelp
andNodeHotThreads
integration tests due to mimetype mismatch - #3887
ResponseBuilder
mimeType check is too strict on net461 framework - #3891 Ephemeral Cluster for tests now uses a SHA has for its cached folder name
- #3892 Remove
ValueTuple
andValueTask
dependencies - #3834 Sum on stats aggregation always returns a value
- #3833 Advance JSON eader when not expected JSON token
- #3815 Collection deserialisation needs to check for null tokens
- #3805 Routing must be specified when
doc
used - #3794
PostData
contained serializable routines instead ofSerializableData
- #3786 Re-instantiate custom deserialisation for nodes hot threads
- #3783 Some Audit event do not carry timing information, now removed from audit log
- #3779 Use a custom convertor for
GeoOrientation
to tolerate alternative server options - #3707 6.x sets
IsValid
=false
for404
response - #3667 Deserialize relation on
TotalHits
- #3666
SynchronizedCollection
was removed - #3657
double
values should always be serialized with a decimal point - #3656 Low level client exception structure in
utf8json
should matchSimpleJson
- #3650
IMemoryStreamFactory
was being side stepped in some places - #3648 Use
HttpMessageHandler
onHttpConnection
- #3646 Consistently name sort types
- #3622 Consistent naming of Timezone/TimeZone
- #3568 Remove
IAcknowledgedResponse
implementation fromIRevertModelSnapshotResponse
- #3454 Change
GeoUnit
toUnit
onSortGeoDistance
- #3415 Cannot use
Xamarin.iOS
NSUrlSessionHandler
for HTTP - #3229 Fixes certutil command name in the documentation
- #3777, #3664 Refactor code generator
- #3871 Treat compiler warnings as errors
- #3866 Update check for unmapped API endpoints
- #3790 Resolve Resharper warnings
- #3683 Resolve issues logged in TODO comments
- #3668 Improve fields resolver performance
- #3589 Fix #3578 Add support for disabling id inference
- #3624 Integration tests
Features & Enhancements
- #3670 Remove response interfaces
- #3213, #3230 Refactor geo shapes and geo shape queries, better
geo_shape
support - #3493 Use
utf8json
as the internal serializer - #3508, #3628 Refactor
Script
query to useIScript
property, andIScript
onScriptQuery
- #3647 Add
IpRangeBucket
forIpRangeAggregation
- #3550 Introduce
IPRangeBucket
type forIpRange
aggregation - #3649 Refactor test configuration to use YAML file
- #3661 Use
float?
for Suggester fractional properties - #3672 Consolidate
Name
andId
parameters - #3677 Refactor
ServerError
,Error
andErrorCause
- #3685 Expose typed expressions
- #3793 Custom response deserialization wiring
- #3807 Use indexers to set default mapping values
- #3806 User
SecureString
forBasic
andProxy
authentication passwords - #3809 Add overloads of
SerializeToString
andSerializeToBytes
- #3812, #3780, #3824 Treat cluster state as key/value response
- #3827 Add index name to
ICreateIndexResponse
- #3835 Support deprecated
delimited_payload_filter
name in deserialization - #3839 Remove deleted Migration Assistance and Upgrade APIs
- #3840 Update documentation for default number of shards
- #3856 Change
Similarity
fromenum
tostring
- #3864 Remove
HighlightDocumentDictionary
andHighlightHit
- #3865, #3854 Delete Indices Upgrade APIs
- #3867 Clean up type extensions
- #3868 Rename
ICovariantSearchRequest
toISearchTypeInformation
- #3869
DocCount
no longer nullable on matrix stats aggregation - #3872 Lazy document now uses source serializer and has async overloads
- #3878 Dictionary serialization tests
- #3879 Make
Should
implementation explicit - #3893 Add
System.DiagnosticSource
dependancy to nuspec file - #3583,#3608 Replace internal serializer with
utf8json
- #3680 Integrate interval query functionality
- #3686 Add support for
is_write_index
- #3694 NEST removes null value from
after_key
Dictionary when aggregating - #3711 Implement Intervals query
- #3714 Add
SequenceNumber
andPrimaryTerm
toHits
,Get
andMulitGetHit
- #3814 Expose deprecated paths in low level client
- #3822 Upgrade assistant
- #3831 Return of net461
- #3837 Add
index.analyze.max_token_count
to updateable index settings - #3838 Add updateable max* index settings
- #3841 Adding
took
time to_msearch
- #3842 Add updateable index settings for nested fields and nested objects
- #3844 Align
Cat
Threadpool
properties - #3853 Dictionary serialisation tests
- #3862 Add DiagnosticSource / Activity support
- #3876 Clean up unused code
- #3888 Ensure Searchresponse uses an interface