diff --git a/CHANGELOG.md b/CHANGELOG.md index 5062f10..bd6e687 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -5,6 +5,8 @@ v0.3.1 (Unreleased) ### Bug fixes * **Code-only driver settings on `UseClickHouse(DbConnection)` and `UseClickHouse(DbDataSource)`** are no longer lost. When the provider added `join_use_nulls` on open, and when it made the master connection for `EnsureCreated`, `EnsureDeleted` and migrations, it rebuilt the driver settings from the connection string. That dropped every setting with no connection-string key, such as `SkipServerCertificateValidation`, `BearerToken`, `CustomHeaders`, `HttpClient` and `LoggerFactory`, so a server with a self-signed certificate failed with an untrusted-root error. The provider now copies the `ClickHouseClientSettings` object instead. ([#50](https://github.com/ClickHouse/ClickHouse.EntityFrameworkCore/issues/50)) +* **`SaveChanges` and `BulkInsertAsync` on `UseClickHouse(DbConnection)`** no longer throw `InvalidOperationException` ("Cannot obtain IClickHouseClient"). `GetClickHouseClient()` and `CreateMasterConnection()` follow the current `DbConnection`. They use the `ClickHouseDataSource` only for a connection that the provider got from it and that still has its database. For another `ClickHouseConnection`, the driver keeps the connection client internal, so the provider creates a `ClickHouseClient` from a copy of `ClickHouseConnection.Settings`, with `Database` set to `ClickHouseConnection.Database`. Queries send `ClickHouseConnection.Database`, which includes `ChangeDatabase`, and `InsertBinaryAsync` sends the client database, so inserts and queries use the same database. This also applies to `ChangeDatabase` on a connection that the provider created from a connection string or data source. The relational connection keeps the client, creates a new one when the connection settings object or database changes, and disposes it with the context. The client has its own HTTP connection pool unless the settings carry an `HttpClient` or `HttpClientFactory`. A `DbConnection` that is not a `ClickHouseConnection` can insert only through a `ClickHouseDataSource`; otherwise the error now names its type. +* **`SetConnectionString` and `SetDbConnection` at runtime** now apply to inserts and the master connection. `ClickHouseRelationalConnection` kept the data source from the options, so after `context.Database.SetConnectionString(...)` or `SetDbConnection(...)`, `SaveChanges`, `BulkInsertAsync` and the master connection (`EnsureCreated`, `EnsureDeleted`, migrations) used the original target. On the connection-string path, a `SetConnectionString` before the first query had no effect, because `CreateDbConnection()` used the cached data source. `dotnet ef database update --connection` and `dotnet ef migrations list --connection` call `SetConnectionString` before they connect, so they connected with the `DbContext` connection string. The `ConnectionString` setter now gets the provider-owned data source for the new string from `ClickHouseDataSourceManager.GetOrCreateDataSource`, and `SetDbConnection` with a connection that is not null clears the data source. Both dispose the cached insert client. As in Npgsql, `SetConnectionString` on a context configured with `UseClickHouse(DbDataSource)` drops that data source. * `ToStartOfWeek` now rejects row-dependent week modes during query translation, and `ToStartOfInterval` likewise rejects row-dependent interval sizes. ClickHouse requires these operands to be constant for the query; literals and captured query parameters remain supported. * `Sum`/`SumAsync` over a `double` or `float` column no longer throws `InvalidCastException`. EF Core wraps a top-level aggregate so the empty case returns `0`, supplying that fallback as a boxed `Int32` carrying the `Float64`/`Float32` mapping; the literal generators now convert rather than unbox. The `Float32` read path also converts, since ClickHouse widens `sum(Float32)` to `Float64` (which the driver's `GetFloat()` refuses to downcast). ([#46](https://github.com/ClickHouse/ClickHouse.EntityFrameworkCore/issues/46)) * **SummingMergeTree with multiple sum columns**: `HasSummingMergeTreeEngine("A", "B")` now generates valid DDL (`SummingMergeTree((A, B))`). Previously it emitted a comma-separated argument list (`SummingMergeTree(A, B)`), which ClickHouse rejects with `NUMBER_OF_ARGUMENTS_DOESNT_MATCH`. Single-column and no-column usage are unaffected. diff --git a/README.md b/README.md index dc7d9c8..abc90b5 100644 --- a/README.md +++ b/README.md @@ -186,6 +186,8 @@ long rowsInserted = await ctx.BulkInsertAsync(events); This calls `InsertBinaryAsync` directly, bypassing EF Core's change tracker entirely. Entities are **not** tracked after insert. +`SaveChanges` and `BulkInsertAsync` work with a connection string, a `ClickHouseDataSource` or a `ClickHouseConnection`, and insert into the server and database of the current connection. With a `ClickHouseConnection`, inserts use a separate client with a copy of the connection settings and the database of the connection (including `ChangeDatabase`). That client has its own HTTP connection pool, unless the settings have an `HttpClient` or `HttpClientFactory`. After `context.Database.SetConnectionString(...)` or `SetDbConnection(...)`, inserts go to the new target. + ### JSON Columns The provider supports ClickHouse's `Json` column type, mapping to `System.Text.Json.Nodes.JsonNode` or `string`. diff --git a/RELEASENOTES.md b/RELEASENOTES.md index bdbb752..9ffe0e3 100644 --- a/RELEASENOTES.md +++ b/RELEASENOTES.md @@ -5,6 +5,8 @@ v0.3.1 (Unreleased) ### Bug fixes * **Settings you set in code now apply when you pass a `ClickHouseConnection` or `ClickHouseDataSource` to `UseClickHouse`.** Settings that have no connection-string key, such as `SkipServerCertificateValidation`, `BearerToken`, `CustomHeaders` or a custom `HttpClient`, were lost when the provider opened the connection or created the database. For example, a server with a self-signed certificate failed with an untrusted-root error even with `SkipServerCertificateValidation = true`. They are now kept on every path, including `EnsureCreated`, `EnsureDeleted` and migrations. ([#50](https://github.com/ClickHouse/ClickHouse.EntityFrameworkCore/issues/50)) +* **`SaveChanges` and `BulkInsertAsync` now work when you pass a `ClickHouseConnection` to `UseClickHouse`.** Before, they threw `InvalidOperationException`. Inserts use the same server, settings and database as your queries, including a database you selected with `ChangeDatabase`. The same is true when you call `ChangeDatabase` on `context.Database.GetDbConnection()` with a connection string or a data source. The provider uses a separate client for these inserts, so it opens its own HTTP connections unless your settings have an `HttpClient` or `HttpClientFactory`. +* **`context.Database.SetConnectionString(...)` and `SetDbConnection(...)` now apply to inserts and to database creation.** Before, `SaveChanges`, `BulkInsertAsync`, `EnsureCreated`, `EnsureDeleted` and migrations kept the original connection, and a `SetConnectionString` call before the first query had no effect. For example, `dotnet ef database update --connection "..."` applied the migrations to the connection string in your `DbContext`, not to the one you gave. If you configure the context with a `DbDataSource` and then call `SetConnectionString`, the context now uses the new connection string. * `ToStartOfWeek` now rejects row-dependent week modes during query translation, and `ToStartOfInterval` likewise rejects row-dependent interval sizes. ClickHouse requires these operands to be constant for the query; literals and captured query parameters remain supported. * Summing a `double` or `float` column (`.SumAsync(x => x.Value)`) no longer throws `InvalidCastException`. Two ClickHouse-specific mismatches were biting: EF Core hands the float literal generator a boxed `Int32` `0` as the empty-result fallback, and ClickHouse widens `sum(Float32)` to `Float64` so the driver couldn't read it back as a `float`. Both the literal generation and the `Float32` read path now convert instead of hard-casting. ([#46](https://github.com/ClickHouse/ClickHouse.EntityFrameworkCore/issues/46)) (Thanks to @HotTotem!) * **SummingMergeTree with multiple sum columns** now produces valid DDL. Configuring more than one sum column (`HasSummingMergeTreeEngine("A", "B")`) previously emitted `SummingMergeTree(A, B)`, which ClickHouse rejects with `NUMBER_OF_ARGUMENTS_DOESNT_MATCH` — the engine takes a single optional parameter that must be a tuple of columns. Multiple columns are now wrapped in a tuple (`SummingMergeTree((A, B))`); single-column and no-column usage are unchanged. diff --git a/src/EFCore.ClickHouse/Storage/Internal/ClickHouseDataSourceManager.cs b/src/EFCore.ClickHouse/Storage/Internal/ClickHouseDataSourceManager.cs index 6d68c90..52ff9b8 100644 --- a/src/EFCore.ClickHouse/Storage/Internal/ClickHouseDataSourceManager.cs +++ b/src/EFCore.ClickHouse/Storage/Internal/ClickHouseDataSourceManager.cs @@ -20,7 +20,11 @@ public class ClickHouseDataSourceManager : IDisposable, IAsyncDisposable _ => null }; - private ClickHouseDataSource GetOrCreateDataSource(string connectionString, bool joinNullSemanticsDisabled) + /// + /// Returns the provider-owned data source for a connection string. There is one data source, + /// and one HTTP connection pool, for each effective connection string. + /// + public ClickHouseDataSource GetOrCreateDataSource(string connectionString, bool joinNullSemanticsDisabled) { var effectiveConnectionString = joinNullSemanticsDisabled ? connectionString diff --git a/src/EFCore.ClickHouse/Storage/Internal/ClickHouseRelationalConnection.cs b/src/EFCore.ClickHouse/Storage/Internal/ClickHouseRelationalConnection.cs index 274bc62..350456e 100644 --- a/src/EFCore.ClickHouse/Storage/Internal/ClickHouseRelationalConnection.cs +++ b/src/EFCore.ClickHouse/Storage/Internal/ClickHouseRelationalConnection.cs @@ -10,8 +10,18 @@ namespace ClickHouse.EntityFrameworkCore.Storage.Internal; public class ClickHouseRelationalConnection : RelationalConnection, IClickHouseRelationalConnection { - private readonly DbDataSource? _dataSource; + private readonly ClickHouseDataSourceManager? _dataSourceManager; private readonly bool _joinNullSemanticsDisabled; + private DbDataSource? _dataSource; + + // The connection that CreateDbConnection got from _dataSource. Only this connection can use + // the data source client and settings. + private DbConnection? _dataSourceConnection; + + // Insert client for a ClickHouseConnection that does not use the data source client, and the + // connection settings object that it copies. + private ClickHouseClient? _connectionClient; + private ClickHouseClientSettings? _connectionClientSource; public ClickHouseRelationalConnection( RelationalConnectionDependencies dependencies, @@ -19,6 +29,7 @@ public ClickHouseRelationalConnection( : base(dependencies) { var extension = dependencies.ContextOptions.FindExtension(); + _dataSourceManager = dataSourceManager; _dataSource = dataSourceManager.GetDataSource(extension); _joinNullSemanticsDisabled = extension?.JoinNullSemanticsDisabled ?? false; } @@ -26,21 +37,75 @@ public ClickHouseRelationalConnection( // Used by CreateMasterConnection only private ClickHouseRelationalConnection( RelationalConnectionDependencies dependencies, - DbDataSource? dataSource, bool joinNullSemanticsDisabled) : base(dependencies) { - _dataSource = dataSource; _joinNullSemanticsDisabled = joinNullSemanticsDisabled; } + /// + /// Sets the connection string, and uses the provider-owned data source for it. This removes a + /// that was configured with UseClickHouse(DbDataSource). + /// + public override string? ConnectionString + { + get => base.ConnectionString; + set + { + var dataSource = string.IsNullOrWhiteSpace(value) + ? null + : _dataSourceManager?.GetOrCreateDataSource(value, _joinNullSemanticsDisabled); + + // The base setter also applies the string to the current connection, if there is one. + // That connection then does not come from the new data source. + base.ConnectionString = value; + _dataSource = dataSource; + _dataSourceConnection = null; + DisposeConnectionClient(); + } + } + + /// + /// Sets the connection. A connection that is not null replaces the configured data source. + /// + public override void SetDbConnection(DbConnection? value, bool contextOwnsConnection) + { + base.SetDbConnection(value, contextOwnsConnection); + + if (value is not null) + _dataSource = null; + + _dataSourceConnection = null; + DisposeConnectionClient(); + } + + // Dispose and DisposeAsync call these with disposeDbConnection: true. A pooled context keeps + // its connection between leases, so it keeps the insert client too. + protected override void ResetState(bool disposeDbConnection) + { + base.ResetState(disposeDbConnection); + if (disposeDbConnection) + DisposeConnectionClient(); + } + + protected override async ValueTask ResetStateAsync(bool disposeDbConnection) + { + await base.ResetStateAsync(disposeDbConnection).ConfigureAwait(false); + if (disposeDbConnection) + DisposeConnectionClient(); + } + protected override DbConnection CreateDbConnection() - => _dataSource?.CreateConnection() - ?? new ClickHouseConnection( - ClickHouseClientIdentity.CreateSettings( - _joinNullSemanticsDisabled - ? ConnectionString! - : ClickHouseDataSourceManager.EnsureDefaultSettings(ConnectionString!))); + { + if (_dataSource is not null) + return _dataSourceConnection = _dataSource.CreateConnection(); + + return new ClickHouseConnection( + ClickHouseClientIdentity.CreateSettings( + _joinNullSemanticsDisabled + ? ConnectionString! + : ClickHouseDataSourceManager.EnsureDefaultSettings(ConnectionString!))); + } protected override bool SupportsAmbientTransactions => false; @@ -120,19 +185,19 @@ public IClickHouseRelationalConnection CreateMasterConnection() return new ClickHouseRelationalConnection( Dependencies with { ContextOptions = optionsBuilder.Options }, - dataSource: null, _joinNullSemanticsDisabled); } private bool TryGetClickHouseClientSettings(out ClickHouseClientSettings settings) { - if (_dataSource is ClickHouseDataSource clickHouseDataSource) + var connection = DbConnection; + if (GetDataSourceOfConnection(connection) is { } dataSource) { - settings = clickHouseDataSource.Settings; + settings = dataSource.Settings; return true; } - if (DbConnection is ClickHouseConnection clickHouseConnection) + if (connection is ClickHouseConnection clickHouseConnection) { settings = clickHouseConnection.Settings; return true; @@ -142,6 +207,27 @@ private bool TryGetClickHouseClientSettings(out ClickHouseClientSettings setting return false; } + /// + /// Returns the that has the same target as the connection, or + /// null. A must come from the data source and still have the + /// data source database, because ChangeDatabase and a new connection string change the + /// target of the connection only. Another connection type (for example, a connection that an + /// interceptor wraps) cannot give its settings, so it uses the data source. + /// + private ClickHouseDataSource? GetDataSourceOfConnection(DbConnection connection) + { + if (_dataSource is not ClickHouseDataSource dataSource) + return null; + + if (connection is not ClickHouseConnection) + return dataSource; + + return ReferenceEquals(connection, _dataSourceConnection) + && connection.Database == dataSource.Settings.Database + ? dataSource + : null; + } + public override IDbContextTransaction BeginTransaction(IsolationLevel isolationLevel) => new ClickHouseTransaction(); @@ -157,13 +243,45 @@ public override Task BeginTransactionAsync( CancellationToken cancellationToken = default) => Task.FromResult(new ClickHouseTransaction()); + /// + /// Returns the client for inserts, with the same target as the current connection. Inserts go to + /// the client database and queries go to . + /// For a connection from the , this is the data source client. + /// For another , the driver does not make the connection client + /// public, so this object owns a client with a copy of the connection settings and database. + /// public IClickHouseClient GetClickHouseClient() { - if (_dataSource is ClickHouseDataSource clickHouseDataSource) - return clickHouseDataSource.GetClient(); + var connection = DbConnection; + if (GetDataSourceOfConnection(connection) is { } dataSource) + return dataSource.GetClient(); + + if (connection is not ClickHouseConnection clickHouseConnection) + { + throw new InvalidOperationException( + $"Inserts need a {nameof(ClickHouseConnection)}, but the connection is a " + + $"'{connection.GetType().Name}'. Configure the context with a connection string, a " + + $"{nameof(ClickHouseDataSource)} or a {nameof(ClickHouseConnection)}."); + } - throw new InvalidOperationException( - "Cannot obtain IClickHouseClient. The connection must be configured with a connection string " + - "or ClickHouseDataSource, not a raw DbConnection."); + var settings = clickHouseConnection.Settings; + var database = clickHouseConnection.Database; + if (_connectionClient is null + || !ReferenceEquals(_connectionClientSource, settings) + || _connectionClient.Settings.Database != database) + { + DisposeConnectionClient(); + _connectionClient = new ClickHouseClient(new ClickHouseClientSettings(settings) { Database = database }); + _connectionClientSource = settings; + } + + return _connectionClient; + } + + private void DisposeConnectionClient() + { + _connectionClient?.Dispose(); + _connectionClient = null; + _connectionClientSource = null; } } diff --git a/test/EFCore.ClickHouse.Tests/ConnectionTargetTests.cs b/test/EFCore.ClickHouse.Tests/ConnectionTargetTests.cs new file mode 100644 index 0000000..2bd1609 --- /dev/null +++ b/test/EFCore.ClickHouse.Tests/ConnectionTargetTests.cs @@ -0,0 +1,415 @@ +using System.Data.Common; +using ClickHouse.Driver; +using ClickHouse.Driver.ADO; +using ClickHouse.EntityFrameworkCore.Storage.Internal; +using Microsoft.EntityFrameworkCore; +using Microsoft.EntityFrameworkCore.Diagnostics; +using Microsoft.EntityFrameworkCore.Infrastructure; +using Microsoft.EntityFrameworkCore.Storage; +using Xunit; + +namespace EFCore.ClickHouse.Tests; + +/// +/// Checks which server and database the read connection, the write client and the master +/// connection resolve to. These tests do not connect to a server: creating and opening a +/// or a client does not send requests. +/// +public class ConnectionTargetTests +{ + private const string TargetA = "Host=host-a;Port=8123;Database=db_a"; + private const string TargetB = "Host=host-b;Port=8124;Database=db_b"; + + private sealed class TargetContext : DbContext + { + private readonly Action _configure; + + public TargetContext(Action configure) + { + _configure = configure; + } + + protected override void OnConfiguring(DbContextOptionsBuilder optionsBuilder) + => _configure(optionsBuilder); + } + + private sealed class PooledTargetContext : DbContext + { + public PooledTargetContext(DbContextOptions options) + : base(options) + { + } + } + + private sealed class WrappingInterceptor : DbConnectionInterceptor + { + public override InterceptionResult ConnectionCreating( + ConnectionCreatingEventData eventData, + InterceptionResult result) + => InterceptionResult.SuppressWithResult(new FakeDbConnection(TargetA)); + } + + private static IClickHouseClient WriteClient(DbContext ctx) + => ctx.GetService().GetClickHouseClient(); + + private static ClickHouseClientSettings ReadSettings(DbContext ctx) + => ((ClickHouseConnection)ctx.Database.GetDbConnection()).Settings; + + private static ClickHouseClientSettings WriteSettings(DbContext ctx) + => WriteClient(ctx).Settings; + + private static ClickHouseClientSettings MasterSettings(DbContext ctx) + { + using var master = ctx.GetService().CreateMasterConnection(); + return ((ClickHouseConnection)master.DbConnection).Settings; + } + + [Fact] + public void GetClickHouseClient_DbConnection_CopiesConnectionSettings() + { + var settings = new ClickHouseClientSettings(TargetA) + { + SkipServerCertificateValidation = true, + BearerToken = "token-abc", + }; + using var connection = new ClickHouseConnection(settings); + using var ctx = new TargetContext(o => o.UseClickHouse(connection)); + + var client = WriteClient(ctx); + + Assert.NotSame(connection.Settings, client.Settings); + Assert.Equal("host-a", client.Settings.Host); + Assert.Equal(8123, client.Settings.Port); + Assert.Equal("db_a", client.Settings.Database); + Assert.True(client.Settings.SkipServerCertificateValidation); + Assert.Equal("token-abc", client.Settings.BearerToken); + } + + [Fact] + public void GetClickHouseClient_DbConnection_ReusesClient() + { + using var connection = new ClickHouseConnection(TargetA); + using var ctx = new TargetContext(o => o.UseClickHouse(connection)); + + var first = WriteClient(ctx); + var second = WriteClient(ctx); + + Assert.Same(first, second); + } + + [Fact] + public void GetClickHouseClient_DbConnection_UsesDatabaseOfConnection() + { + // Reads send ClickHouseConnection.Database, which includes ChangeDatabase. + // Inserts send the client database, so the two must agree. + using var connection = new ClickHouseConnection(TargetA); + connection.ChangeDatabase("db_changed"); + using var ctx = new TargetContext(o => o.UseClickHouse(connection)); + + var first = WriteClient(ctx); + Assert.Equal("db_changed", first.Settings.Database); + + connection.ChangeDatabase("db_changed_again"); + var second = WriteClient(ctx); + + Assert.NotSame(first, second); + Assert.Equal("db_changed_again", second.Settings.Database); + } + + [Fact] + public void GetClickHouseClient_DbConnection_FollowsReplacedSettings() + { + using var connection = new ClickHouseConnection(TargetA); + using var ctx = new TargetContext(o => o.UseClickHouse(connection)); + + var first = WriteClient(ctx); + connection.Settings = new ClickHouseClientSettings(TargetB); + var second = WriteClient(ctx); + + Assert.NotSame(first, second); + Assert.Equal("host-b", second.Settings.Host); + } + + [Fact] + public async Task GetClickHouseClient_DbConnection_CreatesNewClientAfterDispose() + { + using var connection = new ClickHouseConnection(TargetA); + using var ctx = new TargetContext(o => o.UseClickHouse(connection)); + var relational = ctx.GetService(); + + var first = WriteClient(ctx); + relational.Dispose(); + var second = WriteClient(ctx); + await relational.DisposeAsync(); + var third = WriteClient(ctx); + + Assert.NotSame(first, second); + Assert.NotSame(second, third); + } + + [Fact] + public void GetClickHouseClient_DbConnection_KeepsClientAfterPoolReset() + { + using var connection = new ClickHouseConnection(TargetA); + using var ctx = new TargetContext(o => o.UseClickHouse(connection)); + + var first = WriteClient(ctx); + ((IResettableService)ctx.GetService()).ResetState(); + + Assert.Same(first, WriteClient(ctx)); + } + + [Fact] + public void GetClickHouseClient_ConnectionString_UsesDataSourceClient() + { + using var ctx = new TargetContext(o => o.UseClickHouse(TargetA)); + + var client = WriteClient(ctx); + + Assert.Same(client, WriteClient(ctx)); + Assert.Equal("host-a", client.Settings.Host); + } + + [Fact] + public void GetClickHouseClient_ConnectionString_UsesChangedDatabase() + { + using var ctx = new TargetContext(o => o.UseClickHouse(TargetA)); + var dataSourceClient = WriteClient(ctx); + + ctx.Database.GetDbConnection().ChangeDatabase("db_changed"); + + Assert.NotSame(dataSourceClient, WriteClient(ctx)); + Assert.Equal("db_changed", WriteSettings(ctx).Database); + Assert.Equal("host-a", WriteSettings(ctx).Host); + } + + [Fact] + public void GetClickHouseClient_DataSource_KeepsDataSourceClientAfterOpen() + { + // Open adds join_use_nulls to a copy of the connection settings. The connection still + // comes from the data source, so inserts still use the data source client. + using var dataSource = new ClickHouseDataSource(TargetA); + using var ctx = new TargetContext(o => o.UseClickHouse(dataSource)); + + ctx.Database.OpenConnection(); + ctx.Database.CloseConnection(); + + Assert.NotSame(dataSource.Settings, ReadSettings(ctx)); + Assert.Same(dataSource.GetClient(), WriteClient(ctx)); + Assert.Equal("host-a", MasterSettings(ctx).Host); + } + + [Fact] + public void GetClickHouseClient_WrappedConnection_UsesDataSourceClient() + { + using var ctx = new TargetContext(o => o + .UseClickHouse(TargetA) + .AddInterceptors(new WrappingInterceptor())); + + Assert.IsType(ctx.Database.GetDbConnection()); + Assert.Equal("host-a", WriteSettings(ctx).Host); + } + + [Fact] + public void GetClickHouseClient_NonClickHouseConnection_Throws() + { + using var connection = new FakeDbConnection(TargetA); + using var ctx = new TargetContext(o => o.UseClickHouse(connection)); + + var ex = Assert.Throws(() => WriteClient(ctx)); + Assert.Contains(nameof(FakeDbConnection), ex.Message); + } + + [Fact] + public void GetClickHouseClient_NonClickHouseDataSource_Throws() + { + using var dataSource = new FakeDbDataSource(TargetA); + using var ctx = new TargetContext(o => o.UseClickHouse(dataSource)); + + var ex = Assert.Throws(() => WriteClient(ctx)); + Assert.Contains(nameof(FakeDbConnection), ex.Message); + } + + [Fact] + public void SetConnectionString_BeforeFirstUse_ConnectionUsesNewTarget() + { + using var ctx = new TargetContext(o => o.UseClickHouse(TargetA)); + + ctx.Database.SetConnectionString(TargetB); + + Assert.Equal("host-b", ReadSettings(ctx).Host); + Assert.Equal("host-b", WriteSettings(ctx).Host); + Assert.Equal("host-b", MasterSettings(ctx).Host); + } + + [Fact] + public void SetConnectionString_AfterFirstUse_WritesAndMasterUseNewTarget() + { + using var ctx = new TargetContext(o => o.UseClickHouse(TargetA)); + Assert.Equal("host-a", ReadSettings(ctx).Host); + Assert.Equal("host-a", WriteSettings(ctx).Host); + + ctx.Database.SetConnectionString(TargetB); + + Assert.Equal("host-b", ReadSettings(ctx).Host); + Assert.Equal("host-b", WriteSettings(ctx).Host); + Assert.Equal("db_b", WriteSettings(ctx).Database); + Assert.Equal("host-b", MasterSettings(ctx).Host); + } + + [Fact] + public void SetConnectionString_UsesSharedDataSourceForNewTarget() + { + using var first = new TargetContext(o => o.UseClickHouse(TargetA)); + using var second = new TargetContext(o => o.UseClickHouse(TargetB)); + + first.Database.SetConnectionString(TargetB); + + Assert.Same(WriteClient(second), WriteClient(first)); + } + + [Fact] + public void SetConnectionString_DataSource_DropsDataSource() + { + using var dataSource = new ClickHouseDataSource(TargetA); + using var ctx = new TargetContext(o => o.UseClickHouse(dataSource)); + + ctx.Database.SetConnectionString(TargetB); + + Assert.Equal("host-b", ReadSettings(ctx).Host); + Assert.Equal("host-b", WriteSettings(ctx).Host); + Assert.NotSame(dataSource.GetClient(), WriteClient(ctx)); + Assert.Equal("host-b", MasterSettings(ctx).Host); + } + + [Fact] + public void SetConnectionString_DbConnection_WritesAndMasterUseNewTarget() + { + using var connection = new ClickHouseConnection(TargetA); + using var ctx = new TargetContext(o => o.UseClickHouse(connection)); + Assert.Equal("host-a", WriteSettings(ctx).Host); + + ctx.Database.SetConnectionString(TargetB); + + Assert.Equal("host-b", connection.Settings.Host); + Assert.Equal("host-b", WriteSettings(ctx).Host); + Assert.Equal("host-b", MasterSettings(ctx).Host); + } + + [Fact] + public void SetConnectionString_DbConnection_KeepsChangedDatabase() + { + // The driver keeps the ChangeDatabase value when the connection string changes. + using var connection = new ClickHouseConnection(TargetA); + connection.ChangeDatabase("db_changed"); + using var ctx = new TargetContext(o => o.UseClickHouse(connection)); + + ctx.Database.SetConnectionString(TargetB); + + Assert.Equal("db_changed", connection.Database); + Assert.Equal("host-b", WriteSettings(ctx).Host); + Assert.Equal("db_changed", WriteSettings(ctx).Database); + } + + [Fact] + public void SetConnectionString_DbConnection_SameString_KeepsCodeOnlySettings() + { + // The base setter does not change a connection that already has the string, so the + // connection keeps its code-only settings, and inserts and the master connection must too. + var settings = new ClickHouseClientSettings(TargetA) + { + SkipServerCertificateValidation = true, + BearerToken = "token-abc", + }; + using var connection = new ClickHouseConnection(settings); + using var ctx = new TargetContext(o => o.UseClickHouse(connection)); + + ctx.Database.SetConnectionString(connection.ConnectionString); + + Assert.Equal("token-abc", connection.Settings.BearerToken); + Assert.Equal("token-abc", WriteSettings(ctx).BearerToken); + Assert.True(WriteSettings(ctx).SkipServerCertificateValidation); + Assert.Equal("token-abc", MasterSettings(ctx).BearerToken); + Assert.True(MasterSettings(ctx).SkipServerCertificateValidation); + } + + [Fact] + public void SetDbConnection_ConnectionString_WritesAndMasterUseNewConnection() + { + using var ctx = new TargetContext(o => o.UseClickHouse(TargetA)); + Assert.Equal("host-a", WriteSettings(ctx).Host); + + using var connection = new ClickHouseConnection(TargetB); + ctx.Database.SetDbConnection(connection); + + Assert.Equal("host-b", ReadSettings(ctx).Host); + Assert.Equal("host-b", WriteSettings(ctx).Host); + Assert.Equal("host-b", MasterSettings(ctx).Host); + } + + [Fact] + public void SetDbConnection_DataSource_WritesAndMasterUseNewConnection() + { + using var dataSource = new ClickHouseDataSource(TargetA); + using var ctx = new TargetContext(o => o.UseClickHouse(dataSource)); + Assert.Equal("host-a", WriteSettings(ctx).Host); + + using var connection = new ClickHouseConnection(TargetB); + ctx.Database.SetDbConnection(connection); + + Assert.Equal("host-b", ReadSettings(ctx).Host); + Assert.Equal("host-b", WriteSettings(ctx).Host); + Assert.Equal("host-b", MasterSettings(ctx).Host); + } + + [Fact] + public void SetDbConnection_DbConnection_ReplacesWriteClient() + { + using var first = new ClickHouseConnection(TargetA); + using var ctx = new TargetContext(o => o.UseClickHouse(first)); + var firstClient = WriteClient(ctx); + + using var second = new ClickHouseConnection(TargetB); + ctx.Database.SetDbConnection(second); + + var secondClient = WriteClient(ctx); + Assert.NotSame(firstClient, secondClient); + Assert.Equal("host-b", secondClient.Settings.Host); + } + + [Fact] + public void SetDbConnection_Null_ThenSetConnectionString_UsesNewTarget() + { + using var ctx = new TargetContext(o => o.UseClickHouse(TargetA)); + Assert.Equal("host-a", ReadSettings(ctx).Host); + + ctx.Database.SetDbConnection(null); + ctx.Database.SetConnectionString(TargetB); + + Assert.Equal("host-b", ReadSettings(ctx).Host); + Assert.Equal("host-b", WriteSettings(ctx).Host); + Assert.Equal("host-b", MasterSettings(ctx).Host); + } + + [Fact] + public void PooledContext_KeepsReadAndWriteTargetsTogetherAfterReset() + { + var options = new DbContextOptionsBuilder().UseClickHouse(TargetA).Options; + var factory = new PooledDbContextFactory(options); + + PooledTargetContext leased; + using (var ctx = factory.CreateDbContext()) + { + leased = ctx; + Assert.Equal("host-a", ReadSettings(ctx).Host); + ctx.Database.SetConnectionString(TargetB); + } + + using (var ctx = factory.CreateDbContext()) + { + Assert.Same(leased, ctx); + Assert.Equal(ReadSettings(ctx).Host, WriteSettings(ctx).Host); + Assert.Equal(ReadSettings(ctx).Host, MasterSettings(ctx).Host); + } + } +} diff --git a/test/EFCore.ClickHouse.Tests/ConnectionWriteTargetTests.cs b/test/EFCore.ClickHouse.Tests/ConnectionWriteTargetTests.cs new file mode 100644 index 0000000..ec8607d --- /dev/null +++ b/test/EFCore.ClickHouse.Tests/ConnectionWriteTargetTests.cs @@ -0,0 +1,337 @@ +using ClickHouse.Driver.ADO; +using ClickHouse.EntityFrameworkCore.Extensions; +using ClickHouse.EntityFrameworkCore.Storage.Internal; +using Microsoft.EntityFrameworkCore; +using Microsoft.EntityFrameworkCore.Infrastructure; +using Xunit; + +namespace EFCore.ClickHouse.Tests; + +/// +/// Inserts through UseClickHouse(ClickHouseConnection), and checks that inserts, queries and the +/// master connection use the new target after SetConnectionString or SetDbConnection. +/// Target A and target B are two databases on the same server. +/// +public class ConnectionWriteTargetTests : IClassFixture +{ + private readonly ConnectionWriteTargetFixture _fixture; + + public ConnectionWriteTargetTests(ConnectionWriteTargetFixture fixture) + { + _fixture = fixture; + } + + [Fact] + public async Task SaveChanges_DbConnection_InsertsRows() + { + const string marker = nameof(SaveChanges_DbConnection_InsertsRows); + await using var connection = new ClickHouseConnection(_fixture.ConnectionStringA); + await using var ctx = new WriteTargetContext(o => o.UseClickHouse(connection)); + + ctx.Rows.AddRange(Row(1, marker), Row(2, marker)); + await ctx.SaveChangesAsync(); + + Assert.Equal(2, await ctx.Rows.CountAsync(r => r.Marker == marker)); + Assert.Equal(2, await CountAsync(_fixture.ConnectionStringA, marker)); + } + + [Fact] + public async Task BulkInsertAsync_DbConnection_InsertsRows() + { + const string marker = nameof(BulkInsertAsync_DbConnection_InsertsRows); + await using var connection = new ClickHouseConnection(_fixture.ConnectionStringA); + await using var ctx = new WriteTargetContext(o => o.UseClickHouse(connection)); + + var inserted = await ctx.BulkInsertAsync([Row(1, marker), Row(2, marker), Row(3, marker)]); + + Assert.Equal(3, inserted); + Assert.Equal(3, await ctx.Rows.CountAsync(r => r.Marker == marker)); + Assert.Equal(3, await CountAsync(_fixture.ConnectionStringA, marker)); + } + + [Fact] + public async Task SaveChanges_DbConnection_UsesChangedDatabase() + { + // Queries use ClickHouseConnection.Database, which includes ChangeDatabase. Inserts must too. + const string marker = nameof(SaveChanges_DbConnection_UsesChangedDatabase); + await using var connection = new ClickHouseConnection(_fixture.ConnectionStringA); + connection.ChangeDatabase(_fixture.DatabaseB); + await using var ctx = new WriteTargetContext(o => o.UseClickHouse(connection)); + + ctx.Rows.Add(Row(1, marker)); + await ctx.SaveChangesAsync(); + await ctx.BulkInsertAsync([Row(2, marker)]); + + Assert.Equal(2, await ctx.Rows.CountAsync(r => r.Marker == marker)); + await AssertRowsInTargetBAsync(marker, 2); + } + + [Fact] + public async Task SaveChanges_ConnectionString_UsesChangedDatabase() + { + const string marker = nameof(SaveChanges_ConnectionString_UsesChangedDatabase); + await using var ctx = new WriteTargetContext(o => o.UseClickHouse(_fixture.ConnectionStringA)); + ctx.Database.GetDbConnection().ChangeDatabase(_fixture.DatabaseB); + + ctx.Rows.Add(Row(1, marker)); + await ctx.SaveChangesAsync(); + await ctx.BulkInsertAsync([Row(2, marker)]); + + Assert.Equal(2, await ctx.Rows.CountAsync(r => r.Marker == marker)); + await AssertRowsInTargetBAsync(marker, 2); + } + + [Fact] + public async Task SetConnectionString_DbConnection_KeepsChangedDatabase() + { + // The driver keeps the ChangeDatabase value when the connection string changes. + const string marker = nameof(SetConnectionString_DbConnection_KeepsChangedDatabase); + await using var connection = new ClickHouseConnection(_fixture.ConnectionStringA); + connection.ChangeDatabase(_fixture.DatabaseB); + await using var ctx = new WriteTargetContext(o => o.UseClickHouse(connection)); + + ctx.Database.SetConnectionString(_fixture.ConnectionStringA); + ctx.Rows.Add(Row(1, marker)); + await ctx.SaveChangesAsync(); + + Assert.Equal(1, await ctx.Rows.CountAsync(r => r.Marker == marker)); + await AssertRowsInTargetBAsync(marker, 1); + } + + [Fact] + public async Task SetConnectionString_BeforeFirstUse_WritesToNewTarget() + { + const string marker = nameof(SetConnectionString_BeforeFirstUse_WritesToNewTarget); + await using var ctx = new WriteTargetContext(o => o.UseClickHouse(_fixture.ConnectionStringA)); + + ctx.Database.SetConnectionString(_fixture.ConnectionStringB); + ctx.Rows.Add(Row(1, marker)); + await ctx.SaveChangesAsync(); + + Assert.Equal(1, await ctx.Rows.CountAsync(r => r.Marker == marker)); + await AssertRowsInTargetBAsync(marker, 1); + } + + [Fact] + public async Task SetConnectionString_AfterFirstUse_WritesToNewTarget() + { + const string marker = nameof(SetConnectionString_AfterFirstUse_WritesToNewTarget); + await using var ctx = new WriteTargetContext(o => o.UseClickHouse(_fixture.ConnectionStringA)); + await ctx.Rows.CountAsync(); + + ctx.Database.SetConnectionString(_fixture.ConnectionStringB); + ctx.Rows.Add(Row(1, marker)); + await ctx.SaveChangesAsync(); + await ctx.BulkInsertAsync([Row(2, marker)]); + + Assert.Equal(2, await ctx.Rows.CountAsync(r => r.Marker == marker)); + await AssertRowsInTargetBAsync(marker, 2); + } + + [Fact] + public async Task SetConnectionString_DataSource_WritesToNewTarget() + { + const string marker = nameof(SetConnectionString_DataSource_WritesToNewTarget); + await using var dataSource = new ClickHouseDataSource(_fixture.ConnectionStringA); + await using var ctx = new WriteTargetContext(o => o.UseClickHouse(dataSource)); + + ctx.Database.SetConnectionString(_fixture.ConnectionStringB); + ctx.Rows.Add(Row(1, marker)); + await ctx.SaveChangesAsync(); + + Assert.Equal(1, await ctx.Rows.CountAsync(r => r.Marker == marker)); + await AssertRowsInTargetBAsync(marker, 1); + } + + [Fact] + public async Task SetConnectionString_BeforeFirstUse_EnsureCreatedUsesNewTarget() + { + var database = _fixture.DatabaseA + "_created"; + var connectionString = new ClickHouseConnectionStringBuilder(_fixture.ConnectionStringA) + { + Database = database + }.ConnectionString; + await using var ctx = new WriteTargetContext(o => o.UseClickHouse(_fixture.ConnectionStringA)); + + ctx.Database.SetConnectionString(connectionString); + try + { + Assert.True(await ctx.Database.EnsureCreatedAsync()); + + Assert.Equal(1, await ScalarAsync( + _fixture.ConnectionStringA, + $"SELECT count() FROM system.tables WHERE database = '{database}' AND name = 'write_target_rows'")); + } + finally + { + await ctx.Database.EnsureDeletedAsync(); + } + } + + [Fact] + public async Task SetDbConnection_ConnectionString_WritesToNewConnection() + { + const string marker = nameof(SetDbConnection_ConnectionString_WritesToNewConnection); + await using var ctx = new WriteTargetContext(o => o.UseClickHouse(_fixture.ConnectionStringA)); + await ctx.Rows.CountAsync(); + + await using var connection = new ClickHouseConnection(_fixture.ConnectionStringB); + ctx.Database.SetDbConnection(connection); + ctx.Rows.Add(Row(1, marker)); + await ctx.SaveChangesAsync(); + await ctx.BulkInsertAsync([Row(2, marker)]); + + Assert.Equal(2, await ctx.Rows.CountAsync(r => r.Marker == marker)); + await AssertRowsInTargetBAsync(marker, 2); + } + + [Fact] + public async Task SetDbConnection_DataSource_WritesToNewConnection() + { + const string marker = nameof(SetDbConnection_DataSource_WritesToNewConnection); + await using var dataSource = new ClickHouseDataSource(_fixture.ConnectionStringA); + await using var ctx = new WriteTargetContext(o => o.UseClickHouse(dataSource)); + + await using var connection = new ClickHouseConnection(_fixture.ConnectionStringB); + ctx.Database.SetDbConnection(connection); + ctx.Rows.Add(Row(1, marker)); + await ctx.SaveChangesAsync(); + + Assert.Equal(1, await ctx.Rows.CountAsync(r => r.Marker == marker)); + await AssertRowsInTargetBAsync(marker, 1); + } + + [Fact] + public async Task SetDbConnection_DbConnection_WritesToNewConnection() + { + const string marker = nameof(SetDbConnection_DbConnection_WritesToNewConnection); + await using var first = new ClickHouseConnection(_fixture.ConnectionStringA); + await using var ctx = new WriteTargetContext(o => o.UseClickHouse(first)); + ctx.Rows.Add(Row(1, marker)); + await ctx.SaveChangesAsync(); + + await using var second = new ClickHouseConnection(_fixture.ConnectionStringB); + ctx.Database.SetDbConnection(second); + ctx.Rows.Add(Row(2, marker)); + await ctx.SaveChangesAsync(); + + Assert.Equal(1, await CountAsync(_fixture.ConnectionStringA, marker)); + Assert.Equal(1, await CountAsync(_fixture.ConnectionStringB, marker)); + } + + [Fact] + public async Task CreateMasterConnection_AfterSetConnectionString_UsesNewTarget() + { + // The databases are on the same server, so a session setting identifies the target. + await using var ctx = new WriteTargetContext(o => o.UseClickHouse(_fixture.ConnectionStringA)); + await ctx.Rows.CountAsync(); + + ctx.Database.SetConnectionString(_fixture.ConnectionStringB + ";set_log_comment=target-b"); + + Assert.Equal("target-b", await MasterLogCommentAsync(ctx)); + } + + [Fact] + public async Task CreateMasterConnection_AfterSetDbConnection_UsesNewTarget() + { + await using var dataSource = new ClickHouseDataSource(_fixture.ConnectionStringA); + await using var ctx = new WriteTargetContext(o => o.UseClickHouse(dataSource)); + + var settings = new ClickHouseClientSettings(_fixture.ConnectionStringB); + settings.CustomSettings["log_comment"] = "target-b"; + await using var connection = new ClickHouseConnection(settings); + ctx.Database.SetDbConnection(connection); + + Assert.Equal("target-b", await MasterLogCommentAsync(ctx)); + } + + private static WriteTargetRow Row(long id, string marker) => new() { Id = id, Marker = marker }; + + private async Task AssertRowsInTargetBAsync(string marker, long expected) + { + Assert.Equal(expected, await CountAsync(_fixture.ConnectionStringB, marker)); + Assert.Equal(0, await CountAsync(_fixture.ConnectionStringA, marker)); + } + + private static Task CountAsync(string connectionString, string marker) + => ScalarAsync(connectionString, $"SELECT count() FROM write_target_rows WHERE marker = '{marker}'"); + + private static async Task ScalarAsync(string connectionString, string sql) + { + await using var connection = new ClickHouseConnection(connectionString); + await connection.OpenAsync(); + await using var cmd = connection.CreateCommand(); + cmd.CommandText = sql; + return Convert.ToInt64(await cmd.ExecuteScalarAsync()); + } + + private static async Task MasterLogCommentAsync(DbContext ctx) + { + await using var master = ctx.GetService().CreateMasterConnection(); + await master.OpenAsync(CancellationToken.None); + await using var cmd = master.DbConnection.CreateCommand(); + cmd.CommandText = "SELECT getSetting('log_comment')"; + return (string?)await cmd.ExecuteScalarAsync(); + } +} + +public class WriteTargetRow +{ + public long Id { get; set; } + public string Marker { get; set; } = string.Empty; +} + +public class WriteTargetContext : DbContext +{ + private readonly Action _configure; + + public WriteTargetContext(Action configure) + { + _configure = configure; + } + + public DbSet Rows => Set(); + + protected override void OnConfiguring(DbContextOptionsBuilder optionsBuilder) + => _configure(optionsBuilder); + + protected override void OnModelCreating(ModelBuilder modelBuilder) + => modelBuilder.Entity(entity => + { + entity.ToTable("write_target_rows"); + entity.HasKey(e => e.Id); + entity.Property(e => e.Id).HasColumnName("id"); + entity.Property(e => e.Marker).HasColumnName("marker"); + }); +} + +public class ConnectionWriteTargetFixture : IAsyncLifetime +{ + public string ConnectionStringA { get; private set; } = string.Empty; + public string ConnectionStringB { get; private set; } = string.Empty; + public string DatabaseA => new ClickHouseConnectionStringBuilder(ConnectionStringA).Database; + public string DatabaseB => new ClickHouseConnectionStringBuilder(ConnectionStringB).Database; + + public async Task InitializeAsync() + { + ConnectionStringA = await SharedContainer.GetConnectionStringAsync(); + ConnectionStringB = await SharedContainer.GetConnectionStringAsync(); + + foreach (var connectionString in new[] { ConnectionStringA, ConnectionStringB }) + { + await using var connection = new ClickHouseConnection(connectionString); + await connection.OpenAsync(); + await using var cmd = connection.CreateCommand(); + cmd.CommandText = """ + CREATE TABLE write_target_rows ( + id Int64, + marker String + ) ENGINE = MergeTree() + ORDER BY id + """; + await cmd.ExecuteNonQueryAsync(); + } + } + + public Task DisposeAsync() => Task.CompletedTask; +} diff --git a/test/EFCore.ClickHouse.Tests/DotnetEfCliTests.cs b/test/EFCore.ClickHouse.Tests/DotnetEfCliTests.cs index f40aed2..9274fd8 100644 --- a/test/EFCore.ClickHouse.Tests/DotnetEfCliTests.cs +++ b/test/EFCore.ClickHouse.Tests/DotnetEfCliTests.cs @@ -99,6 +99,29 @@ public async Task Database_update_creates_correct_schema() Assert.True(indexCount > 0, "Expected at least one data skipping index"); } + [Fact] + public async Task Database_update_uses_connection_option() + { + if (!_dotnetEfAvailable) + return; // dotnet-ef not installed — skip gracefully + + // The context configures _connectionString. dotnet-ef applies --connection with + // SetConnectionString before the first query, and the migration must use it. + var otherConnectionString = await SharedContainer.GetConnectionStringAsync(); + + await RunDotnetEfSuccessfully("migrations", "add", "InitialCreate"); + await RunDotnetEfSuccessfully("database", "update", "--connection", otherConnectionString); + + using var other = new global::ClickHouse.Driver.ADO.ClickHouseConnection(otherConnectionString); + await other.OpenAsync(); + Assert.Equal(1UL, await QueryScalar(other, "SELECT count() FROM `__EFMigrationsHistory`")); + + using var configured = new global::ClickHouse.Driver.ADO.ClickHouseConnection(_connectionString); + await configured.OpenAsync(); + Assert.Equal(0UL, await QueryScalar(configured, + "SELECT count() FROM system.tables WHERE database = currentDatabase()")); + } + [Fact] public async Task Idempotent_script_is_rejected() {