Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,8 @@ v0.3.1 (Unreleased)

### Bug fixes
* **Code-only driver settings on `UseClickHouse(DbConnection)` and `UseClickHouse(DbDataSource)`** are no longer lost. When the provider added `join_use_nulls` on open, and when it made the master connection for `EnsureCreated`, `EnsureDeleted` and migrations, it rebuilt the driver settings from the connection string. That dropped every setting with no connection-string key, such as `SkipServerCertificateValidation`, `BearerToken`, `CustomHeaders`, `HttpClient` and `LoggerFactory`, so a server with a self-signed certificate failed with an untrusted-root error. The provider now copies the `ClickHouseClientSettings` object instead. ([#50](https://github.com/ClickHouse/ClickHouse.EntityFrameworkCore/issues/50))
* **`SaveChanges` and `BulkInsertAsync` on `UseClickHouse(DbConnection)`** no longer throw `InvalidOperationException` ("Cannot obtain IClickHouseClient"). `GetClickHouseClient()` and `CreateMasterConnection()` follow the current `DbConnection`. They use the `ClickHouseDataSource` only for a connection that the provider got from it and that still has its database. For another `ClickHouseConnection`, the driver keeps the connection client internal, so the provider creates a `ClickHouseClient` from a copy of `ClickHouseConnection.Settings`, with `Database` set to `ClickHouseConnection.Database`. Queries send `ClickHouseConnection.Database`, which includes `ChangeDatabase`, and `InsertBinaryAsync` sends the client database, so inserts and queries use the same database. This also applies to `ChangeDatabase` on a connection that the provider created from a connection string or data source. The relational connection keeps the client, creates a new one when the connection settings object or database changes, and disposes it with the context. The client has its own HTTP connection pool unless the settings carry an `HttpClient` or `HttpClientFactory`. A `DbConnection` that is not a `ClickHouseConnection` can insert only through a `ClickHouseDataSource`; otherwise the error now names its type.
* **`SetConnectionString` and `SetDbConnection` at runtime** now apply to inserts and the master connection. `ClickHouseRelationalConnection` kept the data source from the options, so after `context.Database.SetConnectionString(...)` or `SetDbConnection(...)`, `SaveChanges`, `BulkInsertAsync` and the master connection (`EnsureCreated`, `EnsureDeleted`, migrations) used the original target. On the connection-string path, a `SetConnectionString` before the first query had no effect, because `CreateDbConnection()` used the cached data source. `dotnet ef database update --connection` and `dotnet ef migrations list --connection` call `SetConnectionString` before they connect, so they connected with the `DbContext` connection string. The `ConnectionString` setter now gets the provider-owned data source for the new string from `ClickHouseDataSourceManager.GetOrCreateDataSource`, and `SetDbConnection` with a connection that is not null clears the data source. Both dispose the cached insert client. As in Npgsql, `SetConnectionString` on a context configured with `UseClickHouse(DbDataSource)` drops that data source.
* `ToStartOfWeek` now rejects row-dependent week modes during query translation, and `ToStartOfInterval` likewise rejects row-dependent interval sizes. ClickHouse requires these operands to be constant for the query; literals and captured query parameters remain supported.
* `Sum`/`SumAsync` over a `double` or `float` column no longer throws `InvalidCastException`. EF Core wraps a top-level aggregate so the empty case returns `0`, supplying that fallback as a boxed `Int32` carrying the `Float64`/`Float32` mapping; the literal generators now convert rather than unbox. The `Float32` read path also converts, since ClickHouse widens `sum(Float32)` to `Float64` (which the driver's `GetFloat()` refuses to downcast). ([#46](https://github.com/ClickHouse/ClickHouse.EntityFrameworkCore/issues/46))
* **SummingMergeTree with multiple sum columns**: `HasSummingMergeTreeEngine("A", "B")` now generates valid DDL (`SummingMergeTree((A, B))`). Previously it emitted a comma-separated argument list (`SummingMergeTree(A, B)`), which ClickHouse rejects with `NUMBER_OF_ARGUMENTS_DOESNT_MATCH`. Single-column and no-column usage are unaffected.
Expand Down
2 changes: 2 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -186,6 +186,8 @@ long rowsInserted = await ctx.BulkInsertAsync(events);

This calls `InsertBinaryAsync` directly, bypassing EF Core's change tracker entirely. Entities are **not** tracked after insert.

`SaveChanges` and `BulkInsertAsync` work with a connection string, a `ClickHouseDataSource` or a `ClickHouseConnection`, and insert into the server and database of the current connection. With a `ClickHouseConnection`, inserts use a separate client with a copy of the connection settings and the database of the connection (including `ChangeDatabase`). That client has its own HTTP connection pool, unless the settings have an `HttpClient` or `HttpClientFactory`. After `context.Database.SetConnectionString(...)` or `SetDbConnection(...)`, inserts go to the new target.

### JSON Columns

The provider supports ClickHouse's `Json` column type, mapping to `System.Text.Json.Nodes.JsonNode` or `string`.
Expand Down
2 changes: 2 additions & 0 deletions RELEASENOTES.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,8 @@ v0.3.1 (Unreleased)

### Bug fixes
* **Settings you set in code now apply when you pass a `ClickHouseConnection` or `ClickHouseDataSource` to `UseClickHouse`.** Settings that have no connection-string key, such as `SkipServerCertificateValidation`, `BearerToken`, `CustomHeaders` or a custom `HttpClient`, were lost when the provider opened the connection or created the database. For example, a server with a self-signed certificate failed with an untrusted-root error even with `SkipServerCertificateValidation = true`. They are now kept on every path, including `EnsureCreated`, `EnsureDeleted` and migrations. ([#50](https://github.com/ClickHouse/ClickHouse.EntityFrameworkCore/issues/50))
* **`SaveChanges` and `BulkInsertAsync` now work when you pass a `ClickHouseConnection` to `UseClickHouse`.** Before, they threw `InvalidOperationException`. Inserts use the same server, settings and database as your queries, including a database you selected with `ChangeDatabase`. The same is true when you call `ChangeDatabase` on `context.Database.GetDbConnection()` with a connection string or a data source. The provider uses a separate client for these inserts, so it opens its own HTTP connections unless your settings have an `HttpClient` or `HttpClientFactory`.
* **`context.Database.SetConnectionString(...)` and `SetDbConnection(...)` now apply to inserts and to database creation.** Before, `SaveChanges`, `BulkInsertAsync`, `EnsureCreated`, `EnsureDeleted` and migrations kept the original connection, and a `SetConnectionString` call before the first query had no effect. For example, `dotnet ef database update --connection "..."` applied the migrations to the connection string in your `DbContext`, not to the one you gave. If you configure the context with a `DbDataSource` and then call `SetConnectionString`, the context now uses the new connection string.
* `ToStartOfWeek` now rejects row-dependent week modes during query translation, and `ToStartOfInterval` likewise rejects row-dependent interval sizes. ClickHouse requires these operands to be constant for the query; literals and captured query parameters remain supported.
* Summing a `double` or `float` column (`.SumAsync(x => x.Value)`) no longer throws `InvalidCastException`. Two ClickHouse-specific mismatches were biting: EF Core hands the float literal generator a boxed `Int32` `0` as the empty-result fallback, and ClickHouse widens `sum(Float32)` to `Float64` so the driver couldn't read it back as a `float`. Both the literal generation and the `Float32` read path now convert instead of hard-casting. ([#46](https://github.com/ClickHouse/ClickHouse.EntityFrameworkCore/issues/46)) (Thanks to @HotTotem!)
* **SummingMergeTree with multiple sum columns** now produces valid DDL. Configuring more than one sum column (`HasSummingMergeTreeEngine("A", "B")`) previously emitted `SummingMergeTree(A, B)`, which ClickHouse rejects with `NUMBER_OF_ARGUMENTS_DOESNT_MATCH` — the engine takes a single optional parameter that must be a tuple of columns. Multiple columns are now wrapped in a tuple (`SummingMergeTree((A, B))`); single-column and no-column usage are unchanged.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,11 @@ public class ClickHouseDataSourceManager : IDisposable, IAsyncDisposable
_ => null
};

private ClickHouseDataSource GetOrCreateDataSource(string connectionString, bool joinNullSemanticsDisabled)
/// <summary>
/// Returns the provider-owned data source for a connection string. There is one data source,
/// and one HTTP connection pool, for each effective connection string.
/// </summary>
public ClickHouseDataSource GetOrCreateDataSource(string connectionString, bool joinNullSemanticsDisabled)
{
var effectiveConnectionString = joinNullSemanticsDisabled
? connectionString
Expand Down
154 changes: 136 additions & 18 deletions src/EFCore.ClickHouse/Storage/Internal/ClickHouseRelationalConnection.cs
Original file line number Diff line number Diff line change
Expand Up @@ -10,37 +10,102 @@ namespace ClickHouse.EntityFrameworkCore.Storage.Internal;

public class ClickHouseRelationalConnection : RelationalConnection, IClickHouseRelationalConnection
{
private readonly DbDataSource? _dataSource;
private readonly ClickHouseDataSourceManager? _dataSourceManager;
private readonly bool _joinNullSemanticsDisabled;
private DbDataSource? _dataSource;

// The connection that CreateDbConnection got from _dataSource. Only this connection can use
// the data source client and settings.
private DbConnection? _dataSourceConnection;

// Insert client for a ClickHouseConnection that does not use the data source client, and the
// connection settings object that it copies.
private ClickHouseClient? _connectionClient;
private ClickHouseClientSettings? _connectionClientSource;

public ClickHouseRelationalConnection(
RelationalConnectionDependencies dependencies,
ClickHouseDataSourceManager dataSourceManager)
: base(dependencies)
{
var extension = dependencies.ContextOptions.FindExtension<ClickHouseOptionsExtension>();
_dataSourceManager = dataSourceManager;
_dataSource = dataSourceManager.GetDataSource(extension);
_joinNullSemanticsDisabled = extension?.JoinNullSemanticsDisabled ?? false;
}

// Used by CreateMasterConnection only
private ClickHouseRelationalConnection(
RelationalConnectionDependencies dependencies,
DbDataSource? dataSource,
bool joinNullSemanticsDisabled)
: base(dependencies)
{
_dataSource = dataSource;
_joinNullSemanticsDisabled = joinNullSemanticsDisabled;
}

/// <summary>
/// Sets the connection string, and uses the provider-owned data source for it. This removes a
/// <see cref="DbDataSource"/> that was configured with <c>UseClickHouse(DbDataSource)</c>.
/// </summary>
public override string? ConnectionString
{
get => base.ConnectionString;
set
{
var dataSource = string.IsNullOrWhiteSpace(value)
? null
: _dataSourceManager?.GetOrCreateDataSource(value, _joinNullSemanticsDisabled);

// The base setter also applies the string to the current connection, if there is one.
// That connection then does not come from the new data source.
base.ConnectionString = value;
_dataSource = dataSource;
_dataSourceConnection = null;
DisposeConnectionClient();
}
}

/// <summary>
/// Sets the connection. A connection that is not null replaces the configured data source.
/// </summary>
public override void SetDbConnection(DbConnection? value, bool contextOwnsConnection)
{
base.SetDbConnection(value, contextOwnsConnection);

if (value is not null)
_dataSource = null;

_dataSourceConnection = null;
DisposeConnectionClient();
}

// Dispose and DisposeAsync call these with disposeDbConnection: true. A pooled context keeps
// its connection between leases, so it keeps the insert client too.
protected override void ResetState(bool disposeDbConnection)
{
base.ResetState(disposeDbConnection);
if (disposeDbConnection)
DisposeConnectionClient();
}

protected override async ValueTask ResetStateAsync(bool disposeDbConnection)
{
await base.ResetStateAsync(disposeDbConnection).ConfigureAwait(false);
if (disposeDbConnection)
DisposeConnectionClient();
}

protected override DbConnection CreateDbConnection()
=> _dataSource?.CreateConnection()
?? new ClickHouseConnection(
ClickHouseClientIdentity.CreateSettings(
_joinNullSemanticsDisabled
? ConnectionString!
: ClickHouseDataSourceManager.EnsureDefaultSettings(ConnectionString!)));
{
if (_dataSource is not null)
return _dataSourceConnection = _dataSource.CreateConnection();

return new ClickHouseConnection(
ClickHouseClientIdentity.CreateSettings(
_joinNullSemanticsDisabled
? ConnectionString!
: ClickHouseDataSourceManager.EnsureDefaultSettings(ConnectionString!)));
}

protected override bool SupportsAmbientTransactions => false;

Expand Down Expand Up @@ -120,19 +185,19 @@ public IClickHouseRelationalConnection CreateMasterConnection()

return new ClickHouseRelationalConnection(
Dependencies with { ContextOptions = optionsBuilder.Options },
dataSource: null,
_joinNullSemanticsDisabled);
}

private bool TryGetClickHouseClientSettings(out ClickHouseClientSettings settings)
{
if (_dataSource is ClickHouseDataSource clickHouseDataSource)
var connection = DbConnection;
if (GetDataSourceOfConnection(connection) is { } dataSource)
{
settings = clickHouseDataSource.Settings;
settings = dataSource.Settings;
return true;
}

if (DbConnection is ClickHouseConnection clickHouseConnection)
if (connection is ClickHouseConnection clickHouseConnection)
{
settings = clickHouseConnection.Settings;
return true;
Expand All @@ -142,6 +207,27 @@ private bool TryGetClickHouseClientSettings(out ClickHouseClientSettings setting
return false;
}

/// <summary>
/// Returns the <see cref="ClickHouseDataSource"/> that has the same target as the connection, or
/// null. A <see cref="ClickHouseConnection"/> must come from the data source and still have the
/// data source database, because <c>ChangeDatabase</c> and a new connection string change the
/// target of the connection only. Another connection type (for example, a connection that an
/// interceptor wraps) cannot give its settings, so it uses the data source.
/// </summary>
private ClickHouseDataSource? GetDataSourceOfConnection(DbConnection connection)
{
if (_dataSource is not ClickHouseDataSource dataSource)
return null;

if (connection is not ClickHouseConnection)
return dataSource;

return ReferenceEquals(connection, _dataSourceConnection)
&& connection.Database == dataSource.Settings.Database
? dataSource
: null;
}

public override IDbContextTransaction BeginTransaction(IsolationLevel isolationLevel)
=> new ClickHouseTransaction();

Expand All @@ -157,13 +243,45 @@ public override Task<IDbContextTransaction> BeginTransactionAsync(
CancellationToken cancellationToken = default)
=> Task.FromResult<IDbContextTransaction>(new ClickHouseTransaction());

/// <summary>
/// Returns the client for inserts, with the same target as the current connection. Inserts go to
/// the client database and queries go to <see cref="System.Data.Common.DbConnection.Database"/>.
/// For a connection from the <see cref="ClickHouseDataSource"/>, this is the data source client.
/// For another <see cref="ClickHouseConnection"/>, the driver does not make the connection client
/// public, so this object owns a client with a copy of the connection settings and database.
/// </summary>
public IClickHouseClient GetClickHouseClient()
{
if (_dataSource is ClickHouseDataSource clickHouseDataSource)
return clickHouseDataSource.GetClient();
var connection = DbConnection;
if (GetDataSourceOfConnection(connection) is { } dataSource)
return dataSource.GetClient();

if (connection is not ClickHouseConnection clickHouseConnection)
{
throw new InvalidOperationException(
$"Inserts need a {nameof(ClickHouseConnection)}, but the connection is a " +
$"'{connection.GetType().Name}'. Configure the context with a connection string, a " +
$"{nameof(ClickHouseDataSource)} or a {nameof(ClickHouseConnection)}.");
}

throw new InvalidOperationException(
"Cannot obtain IClickHouseClient. The connection must be configured with a connection string " +
"or ClickHouseDataSource, not a raw DbConnection.");
var settings = clickHouseConnection.Settings;
var database = clickHouseConnection.Database;
if (_connectionClient is null
|| !ReferenceEquals(_connectionClientSource, settings)
|| _connectionClient.Settings.Database != database)
{
DisposeConnectionClient();
_connectionClient = new ClickHouseClient(new ClickHouseClientSettings(settings) { Database = database });
_connectionClientSource = settings;
}

return _connectionClient;
}

private void DisposeConnectionClient()
{
_connectionClient?.Dispose();
_connectionClient = null;
_connectionClientSource = null;
}
}
Loading