NEST & Elasticsearch.Net 7.0: Now GA!

After many months of work, two alphas and a beta, we are pleased to announce the GA release of the NEST and Elasticsearch.Net 7.0 clients.

The overall themes of this release have been based around faster serialization, performance improvements, codebase simplification, and ensuring parity with the many new features available in Elasticsearch 7.0.

Types removal

Specifying types within the .NET clients is now deprecated in 7.0, in line with the overall Elasticsearch type removal strategy.

In instances where your index contains type information and you need to preserve this information, one recommendation is to introduce a property to describe the document type (similar to a table per class with discriminator field in the ORM world) and then implement a custom serialization / deserialization implementation for that class.

This Elasticsearch page details some other approaches.

Faster Serialization

After internalizing the serialization routines, and IL-merging the Newtonsoft.Json package in 6.x, we are pleased to announce that the next stage of serialization improvements have been completed in 7.0.

Both SimpleJson and Newtonsoft.Json have been completely removed and replaced with an implementation of Utf8Json, a fast serializer that works directly with UTF-8 binary. This has yielded a significant performance improvement, which we will be sharing in more detail in a later blog post.

With the move to Utf8Json, we have removed some features that were available in the previous JSON libraries that have proven too onerous to carry forward at this stage.

JSON in the request is never indented, even if SerializationFormatting.Indented is specified. The serialization routines generated by Utf8Json never generate an IJsonFormatter<T> that will indent JSON, for performance reasons. We are considering options for exposing indented JSON for development and debugging purposes.
NEST types cannot be extended by inheritance. With NEST 6.x, additional properties can be included for a type by deriving from that type and annotating these new properties. With the current implementation of serialization with Utf8Json, this approach will not work.
Serializer uses Reflection.Emit. Utf8Json uses Reflection.Emit to generate efficient formatters for serializing types that it sees. Reflection.Emit is not supported on all platforms, for example, UWP, Xamarin.iOS, and Xamarin.Android.
Elasticsearch.Net.DynamicResponse deserializes JSON arrays to List<object>. SimpleJson deserialized JSON arrays to object[], but Utf8Json deserializes them to List<object>. This change is preferred for allocation and performance reasons.
Utf8Json is much stricter when deserializing JSON object field names to C# POCO properties. With the internal Json.NET serializer in 6.x, JSON object field names would attempt to be matched with C# POCO property names first by an exact match, falling back to a case insensitive match. With Utf8Json in 7.x however, JSON object field names must match exactly the name configured for the C# POCO property name.

We believe that the trade-off of features vs. GA release has been worthwhile at this stage. We hold a view to address some of these missing features in a later release.

High to Low level client dispatch changes

In 6.x, the process of an API call within NEST looked roughly like this

client.Search()
  => Dispatch()
    => LowLevelDispatch.SearchDispatch()
      => lowlevelClient.Search()
        => lowlevelClient.DoRequest()

With 7.x, this process has been changed to remove dispatching to the low-level client methods. The new process looks like this

client.Search()
  => lowlevelClient.DoRequest()

This means that in the high-level client IRequest now builds its own URLs, with the upside that the call chain is shorter and allocates fewer closures. The downside is that there are now two URL building mechanisms, one in the low-level client and a new one in the high-level client. In practice, this area of the codebase is kept up to date via code generation, so it does not place any additional burden on development.

Given the simplified call chain and debugging experience, we believe this is an improvement worth making.

Namespaced API methods and Upgrade Assistant

As the API surface of Elasticsearch has grown to well over 200 endpoints, so has the number of client methods exposed, leading to an almost overwhelming number to navigate and explore through in an IDE. This is further exacerbated by the fact that the .NET client exposes both synchronous and asynchronous API methods for both the fluent API syntax as well as the object initializer syntax.

To address this, the APIs are now accessible through sub-properties on the client instance.

For example, in 6.x, to create a machine learning job

var putJobResponse = client.PutJob<Metric>("id", c => c
    .Description("Lab 1 - Simple example")
    .ResultsIndexName("server-metrics")
	.AnalysisConfig(a => a
        .BucketSpan("30m")
        .Latency("0s")
    	.Detectors(d => d.Sum(c => c.FieldName(r => r.Total)))
	)
	.DataDescription(d => d.TimeField(r => r.Timestamp))
);

This has changed to

var putJobResponse = client.MachineLearning.PutJob<Metric>("id", c => c
    .Description("Lab 1 - Simple example")
    .ResultsIndexName("server-metrics")
	.AnalysisConfig(a => a
        .BucketSpan("30m")
        .Latency("0s")
    	.Detectors(d => d.Sum(c => c.FieldName(r => r.Total)))
	)
	.DataDescription(d => d.TimeField(r => r.Timestamp))
);

Notice the client.MachineLearning.PutJob method call in 7.0, as opposed to client.PutJob in 6.x.

We believe this grouping of functionality leads to a better discoverability experience when writing your code, and improved readability when reviewing somebody else's.

The Upgrade Assistant

To assist developers in migrating from 6.x, we have published the Nest.7xUpgradeAssistant Nuget package. When included in your project and the using Nest.ElasticClientExtensions; directive is added, calls will be redirected from the old API method names to the new API method names in 7.0. The result is that your project will compile and you won't need to immediately update your code to use the namespaced methods; instead you'll see compiler warnings indicating the location of the new API methods in 7.0.

This package is to assist developers migrating from 6.x to 7.0 and is limited in scope to this purpose. It is recommended that you observe the compiler warnings and adjust your code as indicated.

Observability and DiagnosticSource

7.0 introduces emitting System.Diagnostics.DiagnosticSource information from the client, during a request. The goal is to enable rich information exchange with the Elastic APM .NET agent and other monitoring libraries.

We emit DiagnosticSource information for key parts of a client request via an Activity event, shipping support for Id, ParentId, RootId, as well as request Duration.

To facilitate wiring this up to DiagnosticListener.AllListeners, we ship both with static access to the publisher names and events through Elasticsearch.Net.Diagnostics.DiagnosticSources as well as strongly typed listeners, removing the need to cast the object passed to activity start/stop events.

An example listener implementation that writes events to the console is given below

private class ListenerObserver : IObserver<DiagnosticListener>
{
  public void OnCompleted() => Console.WriteLine("Completed");
  
  public void OnError(Exception error) => Console.Error.WriteLine(error.Message);
	
  public void OnNext(DiagnosticListener value)
  {
		void WriteToConsole<T>(string eventName, T data)
		{
			var a = Activity.Current;
			Console.WriteLine($"{eventName?.PadRight(30)} {a.Id?.PadRight(32)} {a.ParentId?.PadRight(32)} {data?.ToString().PadRight(10)}");
		}
		if (value.Name == DiagnosticSources.AuditTrailEvents.SourceName)
			value.Subscribe(new AuditDiagnosticObserver(v => WriteToConsole(v.Key, v.Value)));

		if (value.Name == DiagnosticSources.RequestPipeline.SourceName)
			value.Subscribe(new RequestPipelineDiagnosticObserver(
				v => WriteToConsole(v.Key, v.Value),
				v => WriteToConsole(v.Key, v.Value))
			);

		if (value.Name == DiagnosticSources.HttpConnection.SourceName)
			value.Subscribe(new HttpConnectionDiagnosticObserver(
				v => WriteToConsole(v.Key, v.Value),
				v => WriteToConsole(v.Key, v.Value)
			));

		if (value.Name == DiagnosticSources.Serializer.SourceName)
			value.Subscribe(new SerializerDiagnosticObserver(v => WriteToConsole(v.Key, v.Value)));
  }
}

Using the following example

var pool = new SniffingConnectionPool(new[] { node.NodesUris().First() });
var settings = new ConnectionSettings(pool).SniffOnStartup();
var client = new ElasticClient(settings);
var x = client.Search<object>(s=>s.AllIndices());
               	
Console.WriteLine(new string('-', Console.WindowWidth - 1));
                                  	
var y = client.Search<object>(s=>s.Index("does-not-exist"));

Emits the following console output:

SniffOnStartup.Start       	|59e275e-4f9c835d189eb14a.    	Event: SniffOnStartup
Sniff.Start                	|59e275e-4f9c835d189eb14a.1. 	|59e275e-4f9c835d189eb14a.   	GET _nodes/http,settings
Sniff.Start                	|59e275e-4f9c835d189eb14a.1.1.   |59e275e-4f9c835d189eb14a.1. 	GET _nodes/http,settings
SendAndReceiveHeaders.Start	|59e275e-4f9c835d189eb14a.1.1.1. |59e275e-4f9c835d189eb14a.1.1.   GET _nodes/http,settings
SendAndReceiveHeaders.Stop 	|59e275e-4f9c835d189eb14a.1.1.1. |59e275e-4f9c835d189eb14a.1.1.   200   	
ReceiveBody.Start          	|59e275e-4f9c835d189eb14a.1.1.2. |59e275e-4f9c835d189eb14a.1.1.   GET _nodes/http,settings
Deserialize.Start          	|59e275e-4f9c835d189eb14a.1.1.2.1. |59e275e-4f9c835d189eb14a.1.1.2. request/response: Nest.DefaultHighLevelSerializer
Deserialize.Stop           	|59e275e-4f9c835d189eb14a.1.1.2.1. |59e275e-4f9c835d189eb14a.1.1.2. request/response: Nest.DefaultHighLevelSerializer
ReceiveBody.Stop           	|59e275e-4f9c835d189eb14a.1.1.2. |59e275e-4f9c835d189eb14a.1.1.   200   	
Sniff.Stop                 	|59e275e-4f9c835d189eb14a.1.1.   |59e275e-4f9c835d189eb14a.1. 	GET _nodes/http,settings
Sniff.Stop                 	|59e275e-4f9c835d189eb14a.1. 	|59e275e-4f9c835d189eb14a.   	Successful low level call on GET: /_nodes/http,settings?timeout=2s&flat_settings=true
SniffOnStartup.Stop        	|59e275e-4f9c835d189eb14a.    	Event: SniffOnStartup Took: 00:00:00.1872459
Ping.Start                 	|59e275f-4f9c835d189eb14a.    	HEAD /	
SendAndReceiveHeaders.Start	|59e275f-4f9c835d189eb14a.1. 	|59e275f-4f9c835d189eb14a.   	HEAD /	
SendAndReceiveHeaders.Stop 	|59e275f-4f9c835d189eb14a.1. 	|59e275f-4f9c835d189eb14a.   	200   	
ReceiveBody.Start          	|59e275f-4f9c835d189eb14a.2. 	|59e275f-4f9c835d189eb14a.   	HEAD /	
ReceiveBody.Stop           	|59e275f-4f9c835d189eb14a.2. 	|59e275f-4f9c835d189eb14a.   	200   	
Ping.Stop                  	|59e275f-4f9c835d189eb14a.    	Successful low level call on HEAD: /
CallElasticsearch.Start    	|59e2760-4f9c835d189eb14a.    	POST _all/_search
SendAndReceiveHeaders.Start	|59e2760-4f9c835d189eb14a.1. 	|59e2760-4f9c835d189eb14a.   	POST _all/_search
Serialize.Start            	|59e2760-4f9c835d189eb14a.1.1.   |59e2760-4f9c835d189eb14a.1. 	request/response: Nest.DefaultHighLevelSerializer
Serialize.Stop             	|59e2760-4f9c835d189eb14a.1.1.   |59e2760-4f9c835d189eb14a.1. 	request/response: Nest.DefaultHighLevelSerializer
SendAndReceiveHeaders.Stop 	|59e2760-4f9c835d189eb14a.1. 	|59e2760-4f9c835d189eb14a.   	200   	
ReceiveBody.Start          	|59e2760-4f9c835d189eb14a.2. 	|59e2760-4f9c835d189eb14a.   	POST _all/_search
Deserialize.Start          	|59e2760-4f9c835d189eb14a.2.1.   |59e2760-4f9c835d189eb14a.2. 	request/response: Nest.DefaultHighLevelSerializer
Deserialize.Stop           	|59e2760-4f9c835d189eb14a.2.1.   |59e2760-4f9c835d189eb14a.2. 	request/response: Nest.DefaultHighLevelSerializer
ReceiveBody.Stop           	|59e2760-4f9c835d189eb14a.2. 	|59e2760-4f9c835d189eb14a.   	200       
CallElasticsearch.Stop     	|59e2760-4f9c835d189eb14a.    	Successful low level call on POST: /_all/_search?typed_keys=true
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
CallElasticsearch.Start    	|59e2761-4f9c835d189eb14a.    	POST does-not-exist/_search
SendAndReceiveHeaders.Start    |59e2761-4f9c835d189eb14a.1. 	|59e2761-4f9c835d189eb14a.   	POST does-not-exist/_search
Serialize.Start            	|59e2761-4f9c835d189eb14a.1.1.   |59e2761-4f9c835d189eb14a.1. 	request/response: Nest.DefaultHighLevelSerializer
Serialize.Stop             	|59e2761-4f9c835d189eb14a.1.1.   |59e2761-4f9c835d189eb14a.1. 	request/response: Nest.DefaultHighLevelSerializer
SendAndReceiveHeaders.Stop 	|59e2761-4f9c835d189eb14a.1. 	|59e2761-4f9c835d189eb14a.   	404   	
ReceiveBody.Start          	|59e2761-4f9c835d189eb14a.2. 	|59e2761-4f9c835d189eb14a.   	POST does-not-exist/_search
Deserialize.Start          	|59e2761-4f9c835d189eb14a.2.1.   |59e2761-4f9c835d189eb14a.2. 	request/response: Nest.DefaultHighLevelSerializer
Deserialize.Stop           	|59e2761-4f9c835d189eb14a.2.1.   |59e2761-4f9c835d189eb14a.2. 	request/response: Nest.DefaultHighLevelSerializer
ReceiveBody.Stop           	|59e2761-4f9c835d189eb14a.2. 	|59e2761-4f9c835d189eb14a.   	404   	
CallElasticsearch.Stop     	|59e2761-4f9c835d189eb14a.    	Unsuccessful low level call on POST: /does-not-exist/_search?typed_keys=true

Response Interfaces removed

Most API methods now return classes and not interfaces; for example, the client method client.Cat.Help now returns a CatResponse<CatAliasesRecord> as opposed to an interface named ICatResponse<CatAliasesRecord>.

In instances where methods can benefit from returning an interface, these have been left intact, for example, ISearchResponse<T>.

So why make the change?

Firstly, this significantly reduces the number of types in the library, reducing the overall download size, improving assembly load times and eventually the execution.

Secondly, it removes the need for us to manage the conversion of a Task<Response> to Task<IResponse>, a somewhat awkward part of the request pipeline.

The downside is that it does make it somewhat more difficult to create mocks / stubs of responses in the client.

After lengthy discussion we decided that users can achieve a similar result using a JSON string and the InMemoryConnection available in Elasticsearch.Net. We use this technique extensively in the Tests.Reproduce project.

Another alternative would be to introduce an intermediate layer in your application, and conceal the client calls and objects within that layer so they can be mocked.

Response.IsValid semantics

IApiCallDetails.Success and ResponseBase.IsValid have been simplified, making it easier to inspect if a request to Elasticsearch was indeed successful or not.

Low Level Client

If the status code from Elasticsearch is 2xx then .Success will be true. In instances where a 404 status code is received, for example if a GET request results in a missing document, then .Success will be false. This is also the case for HEAD requests that result in a 404.

This is controlled via IConnectionConfiguration.StatusCodeToResponseSuccess, which currently has no public setter.

High Level Client

The NEST high level client overrides StatusCodeToResponseSuccess, whereby 404 status codes now sets .Success as true.

The reasoning here is that because NEST is in full control of url and path building the only instances where a 404 is received is in the case of a missing document, never from a missing endpoint.

However, in the case of a 404 the ResponseBase.IsValid property will be false.

It has the nice side effect that if you set .ThrowExceptions() and perform an action on an entity that does not exist it won't throw as .ThrowExceptions() only inspects .Success on ApiCallDetails.

Breaking changes

Please refer to the documentation for a full list of public API breaking changes between 6.8.0 and 7.0.0 GA releases.

Bug Fixes

#3852 Change HighlightFieldDictionary to use formatter
#3224 Change GenericProperty.Index to `bool?``
#3673 Exception when aggregating dates, issue in deserialisation
#3679 Fix utf8json escape characters, particularly quotes
#3694 NEST removes null value from after_key Dictionary when aggregating
#3743 Fix deserialization of special characters
#3819, #3820 Use CultureInfo.InvariantCulture`` when writing values in GeoWKTWriter``
#3825 Use Marshal.SecureStringToGlobalAllocUnicode
#3883 Fix CatHelp and NodeHotThreads integration tests due to mimetype mismatch
#3887 ResponseBuilder mimeType check is too strict on net461 framework
#3891 Ephemeral Cluster for tests now uses a SHA has for its cached folder name
#3892 Remove ValueTuple and ValueTask dependencies
#3834 Sum on stats aggregation always returns a value
#3833 Advance JSON eader when not expected JSON token
#3815 Collection deserialisation needs to check for null tokens
#3805 Routing must be specified when doc used
#3794 PostData contained serializable routines instead of SerializableData
#3786 Re-instantiate custom deserialisation for nodes hot threads
#3783 Some Audit event do not carry timing information, now removed from audit log
#3779 Use a custom convertor for GeoOrientation to tolerate alternative server options
#3707 6.x sets IsValid=false for 404 response
#3667 Deserialize relation on TotalHits
#3666 SynchronizedCollection was removed
#3657 double values should always be serialized with a decimal point
#3656 Low level client exception structure in utf8json should match SimpleJson
#3650 IMemoryStreamFactory was being side stepped in some places
#3648 Use HttpMessageHandler on HttpConnection
#3646 Consistently name sort types
#3622 Consistent naming of Timezone/TimeZone
#3568 Remove IAcknowledgedResponse implementation from IRevertModelSnapshotResponse
#3454 Change GeoUnit to Unit on SortGeoDistance
#3415 Cannot use Xamarin.iOS NSUrlSessionHandler for HTTP
#3229 Fixes certutil command name in the documentation
#3777, #3664 Refactor code generator
#3871 Treat compiler warnings as errors
#3866 Update check for unmapped API endpoints
#3790 Resolve Resharper warnings
#3683 Resolve issues logged in TODO comments
#3668 Improve fields resolver performance
#3589 Fix #3578 Add support for disabling id inference
#3624 Integration tests

Features & Enhancements

#3670 Remove response interfaces
#3213, #3230 Refactor geo shapes and geo shape queries, better geo_shape support
#3493 Use utf8json as the internal serializer
#3508, #3628 Refactor Script query to use IScript property, and IScript on ScriptQuery
#3647 Add IpRangeBucket for IpRangeAggregation
#3550 Introduce IPRangeBucket type for IpRange aggregation
#3649 Refactor test configuration to use YAML file
#3661 Use float? for Suggester fractional properties
#3672 Consolidate Name and Id parameters
#3677 Refactor ServerError, Error and ErrorCause
#3685 Expose typed expressions
#3793 Custom response deserialization wiring
#3807 Use indexers to set default mapping values
#3806 User SecureString for Basic and Proxy authentication passwords
#3809 Add overloads of SerializeToString and SerializeToBytes
#3812, #3780, #3824 Treat cluster state as key/value response
#3827 Add index name to ICreateIndexResponse
#3835 Support deprecated delimited_payload_filter name in deserialization
#3839 Remove deleted Migration Assistance and Upgrade APIs
#3840 Update documentation for default number of shards
#3856 Change Similarity from enum to string
#3864 Remove HighlightDocumentDictionary and HighlightHit
#3865, #3854 Delete Indices Upgrade APIs
#3867 Clean up type extensions
#3868 Rename ICovariantSearchRequest to ISearchTypeInformation
#3869 DocCount no longer nullable on matrix stats aggregation
#3872 Lazy document now uses source serializer and has async overloads
#3878 Dictionary serialization tests
#3879 Make Should implementation explicit
#3893 Add System.DiagnosticSource dependancy to nuspec file
#3583,#3608 Replace internal serializer with utf8json
#3680 Integrate interval query functionality
#3686 Add support for is_write_index
#3694 NEST removes null value from after_key Dictionary when aggregating
#3711 Implement Intervals query
#3714 Add SequenceNumber and PrimaryTerm to Hits, Get and MulitGetHit
#3814 Expose deprecated paths in low level client
#3822 Upgrade assistant
#3831 Return of net461
#3837 Add index.analyze.max_token_count to updateable index settings
#3838 Add updateable max* index settings
#3841 Adding took time to _msearch
#3842 Add updateable index settings for nested fields and nested objects
#3844 Align Cat Threadpool properties
#3853 Dictionary serialisation tests
#3862 Add DiagnosticSource / Activity support
#3876 Clean up unused code
#3888 Ensure Searchresponse uses an interface

elastic/elasticsearch-net 7.0.0
on GitHub

NEST & Elasticsearch.Net 7.0: Now GA!

Types removal

Faster Serialization

High to Low level client dispatch changes

Namespaced API methods and Upgrade Assistant

The Upgrade Assistant

Observability and DiagnosticSource

Response Interfaces removed

Response.IsValid semantics

Low Level Client

High Level Client

Breaking changes

Bug Fixes

Features & Enhancements

View the full list of issues and PRs

elastic/elasticsearch-net 7.0.0 on GitHub

NEST & Elasticsearch.Net 7.0: Now GA!

Types removal

Faster Serialization

High to Low level client dispatch changes

Namespaced API methods and Upgrade Assistant

The Upgrade Assistant

Observability and DiagnosticSource

Response Interfaces removed

Response.IsValid semantics

Low Level Client

High Level Client

Breaking changes

Bug Fixes

Features & Enhancements

View the full list of issues and PRs

elastic/elasticsearch-net 7.0.0
on GitHub