Skip unsupported column data types during schema discovery - #3802
Joymax (joymaxnascimento) wants to merge 12 commits into
Conversation
|
Azure Pipelines: There may be pipelines that require an authorized user to comment /azp run to run. |
@microsoft-github-policy-service agree |
|
Azure Pipelines: There may be pipelines that require an authorized user to comment /azp run to run. |
There was a problem hiding this comment.
🟡 Changes recommended
Case-sensitive permission/primary-key filtering can incorrectly drop permitted columns or primary keys when config/schema casing differs, breaking metadata inference.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Pull request overview
This PR updates SQL metadata discovery to honor entity field-level permissions when building the schema-read projection, avoiding provider failures on unsupported CLR-mapped columns (e.g., SQL Server geometry) that are not readable per config.
Changes:
- Narrow
FillSchemaForTableAsyncfromSELECT *to a permission-derived projection where possible, with fallbacks to preserve current behavior. - Re-apply per-entity column restrictions when populating
SourceDefinition.Columns, accounting for the shared schemaDataTablecache. - Add an MSSQL test fixture/table/config entity to validate that a
geometrycolumn omitted fromfields.includeis not inferred.
File summaries
| File | Description |
|---|---|
| src/Core/Services/MetadataProviders/SqlMetadataProvider.cs | Builds a permission-aware column projection for schema discovery and filters inferred columns accordingly. |
| src/Service.Tests/UnitTests/SqlMetadataProviderUnitTests.cs | Adds an MSSQL test asserting excluded geometry column is not inferred. |
| src/Service.Tests/DatabaseSchema-MsSql.sql | Adds geometry_type_table and seed data for the new test case. |
| src/Service.Tests/dab-config.MsSql.json | Adds GeometryType entity with fields.include restricting readable columns. |
| src/Service.Tests/Snapshots/ConfigurationTests.TestReadingRuntimeConfigForMsSql.verified.txt | Updates snapshot to reflect the added MSSQL entity. |
| config-generators/mssql-commands.txt | Adds generator command to create the GeometryType entity with fields.include. |
Review details
Suppressed comments (1)
src/Core/Services/MetadataProviders/SqlMetadataProvider.cs:1904
- Permitted column resolution uses case-sensitive sets (StringComparer.Ordinal) for Included/Excluded, but SourceDefinition.Columns is case-insensitive (StringComparer.InvariantCultureIgnoreCase). If config uses different casing than the schema (common with SQL Server), IsColumnPermitted() can incorrectly treat a readable column as non-permitted and drop it from SourceDefinition.Columns, breaking runtime behavior.
HashSet<string> included = new(StringComparer.Ordinal);
HashSet<string>? excluded = null;
bool allColumns = false;
- Files reviewed: 6/6 changed files
- Comments generated: 1
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Good catch. Fixed: Included/Excluded and the exposed-to-backing map now use |
There was a problem hiding this comment.
🟡 Changes recommended
The new permission-based schema projection can omit primary key columns when PKs are inferred (not configured), causing initialization failures for valid configs that restrict fields.include without including PKs.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Review details
- Files reviewed: 6/6 changed files
- Comments generated: 1
- Review effort level: Lite
There was a problem hiding this comment.
🟡 Changes recommended
The unconfigured-primary-key guard leaves the reported geometry scenario on SELECT *, while shared metadata and case-sensitive identifiers introduce additional correctness issues.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Review details
Suppressed comments (1)
src/Core/Services/MetadataProviders/SqlMetadataProvider.cs:2026
- This guard disables the new behavior for the exact regression fixture:
GeometryTypehas neitherfields[].primary-keynorsource.key-fields, soconfiguredPrimaryKeyis empty, this returns unrestricted, andBuildSchemaProjectionAsyncproducesSELECT *.FillSchemawill therefore still encountergeomand fail before the new test reaches its assertions. The primary key needs to be obtained without materializing every column (for example from catalog metadata) and unioned into the restricted projection; requiring it in configuration would not fix the reported configuration.
if (entity.Permissions is null || entity.Permissions.Length == 0 || configuredPrimaryKey.Count == 0)
{
return new(AllColumns: true, included, new HashSet<string>(StringComparer.OrdinalIgnoreCase));
- Files reviewed: 6/6 changed files
- Comments generated: 2
- Review effort level: Balanced
…ring field permissions
There was a problem hiding this comment.
🟡 Changes recommended
Nested database connections can deadlock under a one-connection pool, and cache keys collide for supported dotted identifiers.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Review details
Suppressed comments (1)
Previously missed (1) — in code that hasn't changed since the last review.
src/Core/Services/MetadataProviders/SqlMetadataProvider.cs:1847
BuildSchemaProjectionAsyncopens another connection throughGetColumnsAsync, but the outer connection is already open here. With the valid SQL Server settingMax Pool Size=1, the innerOpenAsyncwaits for this connection until timeout, so schema discovery fails; larger pools also consume two simultaneous connections per object. Build the projection before opening the adapter connection, or reuse the same connection for the catalog query.
- Files reviewed: 7/7 changed files
- Comments generated: 2
- Review effort level: Balanced
…biguate the cache key
Correct, and thank you for catching it — this one was mine to see. The projection is now resolved |
There was a problem hiding this comment.
🔵 Needs a closer look
Skipped configured fields remain resolvable, and case-sensitive object names can collide in the metadata cache.
Review details
Suppressed comments (3)
Previously missed (3) — in code that hasn't changed since the last review.
src/Core/Services/MetadataProviders/SqlMetadataProvider.cs:104
- This case-insensitive cache key aliases distinct SQL Server objects in a case-sensitive database. For example,
dbo.Fooanddbo.foocan both be exposed, but after metadata for the first is cached the second reuses those catalog rows, so its projection and column definitions can silently come from the wrong table. Use an ordinal case-sensitive comparer for this per-object catalog cache; case-insensitive databases remain unaffected.
src/Core/Services/MetadataProviders/SqlMetadataProvider.cs:1946 - Configured
fieldsand legacymappingsare still copied into the exposed/backing maps even when their backing column was skipped here, and bothTryGetBackingColumn/TryGetExposedColumnNamealso fall back directly toEntity.Fields. As a result, an entity that explicitly listsgeomstill reports it as valid to callers such as MCP aggregate validation, which checks onlyTryGetBackingColumn, and can issue a runtime query against the supposedly omitted UDT. Build these maps only from columns present inSourceDefinition.Columnsand prevent the fallback from resolving skipped configured fields; add a regression case withgeominfieldsormappings.
src/Core/Services/MetadataProviders/SqlMetadataProvider.cs:1972 - The configured-primary-key rejection has no regression coverage: the new tests exercise a mixed table with an inferred supported key and an all-unsupported view, so neither reaches this branch. Add an in-memory MSSQL entity over
geometry_type_tablewithkey-fields: ["geom"]and assert the initialization status, substatus, column name, and type; otherwise this consistency guard can regress unnoticed.
- Files reviewed: 7/7 changed files
- Comments generated: 0 new
- Review effort level: Balanced
Two of the three are fixed in the latest commit. The cache comparer is now ordinal, with the reason documented on both fields — you are right that The primary-key guard now has coverage: ValidatePrimaryKeyOnUnsupportedColumnFailsInitialization On the third, I agree with the finding but would rather not fix it here. Building the The exposure is bounded: it requires explicitly naming the unsupported column in fields or |
| } | ||
| else | ||
| { | ||
| readableColumns.Add(columnName); |
There was a problem hiding this comment.
The catalog lookup happens indirectly through existing code:
BuildSchemaProjectionAsync() → GetCachedColumnsAsync() → GetColumnsAsync() → conn.GetSchemaAsync("Columns", columnRestrictions).
GetColumnsAsync() already exists in the base SqlMetadataProvider class, so the GetSchemaAsync() call is not a newly added line in this diff. For MSSQL, conn is a SqlConnection.
Previously, this catalog supplemented metadata obtained through FillSchema; it did not determine the SELECT projection. This PR now uses the catalog's column names to construct that projection.
The distinction matters because the catalog includes SQL Server columns declared HIDDEN, whereas SELECT * omits them. Adding every non-spatial catalog column to readableColumns therefore exposes hidden temporal period columns.
Reproduced configuration:
- A temporal table with
Id,Name, nullableLocation geometry, andValidFrom/ValidTodeclaredGENERATED ALWAYS AS ROW START/END HIDDEN. - Permissions include
*and excludeLocation. - Both revisions use
Type System Version=SQL Server 2005in the connection string. This allows the merge-base to discover the spatial column through the driver's legacy type representation, establishing a previously working configuration. It is a test precondition, not a proposed general workaround.
Observed difference:
- Merge-base: REST/GraphQL expose only
IdandName; PUT with onlyNamereturns 200. - PR head: both period columns become exposed; the same PUT returns 500:
Cannot update GENERATED ALWAYS columns.
DAB's PUT uses overwrite semantics. BaseSqlQueryStructure.AddNullifiedUnspecifiedFields() assigns null to writable columns absent from the request. The newly discovered period columns reach that path because the existing read-only classification does not recognize them as generated-always temporal columns.
Please preserve the columns that SELECT * would expose, then subtract the unsupported types. This requires accounting for database hidden-column metadata, not applying permission filters. Add regression tests for both the exposed schema and supported-field PUT.
There was a problem hiding this comment.
Confirmed and fixed. The projection now reproduces what SELECT * exposes before subtracting the unsupported types — hidden columns are excluded through sys.columns.is_hidden, and they are not reported as skipped, since they were never part of the object's shape as the engine sees it. Covered by ValidateHiddenPeriodColumnsAreNotInferred, over a new temporal_geometry_type_table fixture with HIDDEN period columns and a spatial column. Measured run is in the summary comment.
| subStatusCode: DataApiBuilderException.SubStatusCodes.ErrorInInitialization); | ||
| } | ||
|
|
||
| _skippedColumnsByObject[GetObjectCacheKey(schemaName, tableName)] = skippedColumns; |
There was a problem hiding this comment.
There are several separate representations of an entity's fields in DAB:
SourceDefinition.Columns: discovered database-column metadata used by query generation and validation.- Exposed/backing maps: translations between database column names and API aliases.
- Authorization metadata: fields permitted for each role and operation.
This PR removes unsupported columns from the first representation, but the other representations can retain them.
The existing GenerateExposedToBackingColumnMapUtil() adds entries from Entity.Fields and legacy Entity.Mappings without requiring those columns to exist in SourceDefinition.Columns. The existing AuthorizationResolver.SetEntityPermissionMap() also accepts explicitly included field names. PopulateAllowedExposedColumns() then resolves those names through the maps.
Consequently, an explicitly permitted alias can remain usable even though its column metadata was removed. Those helpers are not new in this diff, but allowing startup makes this inconsistent state reachable.
Reproduced configuration:
- Database column
Location geometry. - API alias
Position, configured through eitherfieldsormappings. - Permissions explicitly include the backing names
Id,Name, andLocation. - For the mutation reproduction, create/update permissions include those fields and
request-body-strictis false.
Observed results:
- Startup succeeds and logs that
Locationwas skipped. - Default REST GET returns 500:
FOR JSON cannot serialize CLR objects. The permission-derived default projection includes the stale alias and selects the spatial column. - POST/PATCH supplying
Positionreturn 500 withKeyNotFoundException. - MCP create/update encounter the same missing-column error.
- With the
fieldsalias format, MCPdescribe_entitiesadvertisesPosition, although GraphQL omits it and explicit MCP reads reject it.
For the mutation failure, successful TryGetBackingColumn() resolution leads SqlMutationEngine.PopulateParamsFromRestRequest() to index SourceDefinition.Columns["Location"], which no longer exists. Non-strict mode allows the input past the earlier unexpected-field validation.
Aliases with wildcard permissions alone were rejected correctly; the confirmed failure requires the explicit permission reference above. This is not a demonstrated authorization bypass.
Please either reject incompatible configured references during initialization or consistently reconcile mappings, permission-derived projections, MCP metadata, and name-resolution fallbacks with the usable columns. Add tests for both alias formats, default reads, and non-strict mutations.
There was a problem hiding this comment.
Agreed, and I am taking the first of your two options: initialization now fails when mappings, fields or a permission's fields.include name a column left out of the projection. exclude is left alone, since it asks for what already happened. Covered by ValidateConfiguredReferenceToUnsupportedColumnFailsInitialization. I did not do the full reconciliation here — reasoning in the summary comment — and I am happy to take it as a follow-up, or in this PR if you prefer.
| Connection = conn | ||
| Connection = conn, | ||
| CommandText | ||
| = $"SELECT {projection} FROM {tableNameWithSchemaPrefix}" |
There was a problem hiding this comment.
The filtered SELECT list does not fully control the columns examined by FillSchema.
DbDataAdapter.FillSchema() internally requests schema and key information using CommandBehavior.KeyInfo. That behavior comes from the framework implementation; there does not need to be an explicit KeyInfo reference in this PR.
SQL Server/SqlClient can consequently append primary or unique-key columns that were omitted from the SELECT list as hidden reader columns. These provider-added hidden columns are distinct from SQL table columns declared HIDDEN.
Reproduced cases:
- A table whose database primary key is
Location hierarchyid, selecting onlyName. - A composite database primary key of
(TenantId, Location hierarchyid), selecting onlyTenantIdandName. - A table with no database primary key, a non-null
Location hierarchyid UNIQUE, and a supportedIdconfigured as DAB's key throughsource.key-fields: ["Id"], selecting onlyIdandName.
In all three cases, the reader still contains Location with IsHidden = true and an unresolved CLR type. FillSchema() therefore throws the original DataReader.GetFieldType(N) returned null error.
The new RejectPrimaryKeyOnUnsupportedColumn() guard cannot handle this because it runs in PopulateSourceDefinitionAsync() only after GetTableWithSchemaFromDataSetAsync() has completed. The adapter throws inside that earlier call.
The third case is important: configuring a supported DAB key does not change the adapter's independent database key discovery, so it still retrieves the unsupported unique-key column and prevents startup.
Please account for provider-added key metadata when collecting schema. Diagnose unsupported actual keys before adapter execution where rejection is necessary, and avoid reintroducing excluded unique-key columns when a supported configured key is available. Preserve ordinary primary-key inference when changing this path.
Add tests using real hierarchyid primary/composite keys and the unique-key scenario above. The existing test that configures geometry as a key on a table whose actual database primary key is int does not exercise this framework behavior.
There was a problem hiding this comment.
You were right, and this was the one that decided the approach: the narrowed projection was not fixing #3801 wherever the unsupported type participates in a key. When the projection is narrowed, discovery now reads the shape with CommandBehavior.SchemaOnly instead of FillSchema and takes the primary key from the catalog. When nothing is skipped, FillSchema still runs with KeyInfo unchanged, so ordinary primary-key inference is untouched. Your three cases are covered by ValidateUnsupportedDatabasePrimaryKeyFailsInitialization, ValidateUnsupportedColumnInCompositeDatabasePrimaryKeyFailsInitialization and ValidateConfiguredKeyIsUsedWhenUniqueIndexColumnIsUnsupported.
…d references Schema discovery reproduces the columns SELECT * exposes before subtracting unsupported data types, so HIDDEN period columns stay out of the exposed contract. When the projection is narrowed it reads the shape with CommandBehavior.SchemaOnly instead of DbDataAdapter.FillSchema, whose KeyInfo behaviour reintroduced excluded key columns as hidden reader columns, and takes the primary key from the catalog. Ordinary primary-key inference is untouched when nothing is skipped. Configuration naming a column left out of the projection now fails initialization. Addresses the three review findings on Azure#3802.
|
All three findings are addressed and pushed. I replied in each thread with what changed and which I also rewrote the description. It described only the projection narrowing, and its compatibility One correction to the record: the description previously said the automated test had not been run, MSSQL only — DWSQL, MySQL and PostgreSQL need engines I do not have locally, so CI remains the check The one design question still open is on your second finding. I rejected incompatible configured |
There was a problem hiding this comment.
🟡 Changes recommended
Identifier handling and reconstructed key/type metadata can still reject valid entities or expose incorrect schemas.
Get a fresh assessment by requesting another Copilot review.
Review details
- Files reviewed: 7/7 changed files
- Comments generated: 4
- Review effort level: Balanced
…he narrowed path object_id() now delimits both name parts through QUOTENAME, so a schema or table needing quoting resolves instead of returning no rows and silently leaving the projection at SELECT *. Identity no longer travels through DataColumn.AutoIncrement, whose setter coerces a DataType it cannot increment to Int32 — SQL Server allows identity on tinyint, numeric and decimal. It comes from sys.columns.is_identity and is applied after the columns are populated, so the provider-reported DataType is preserved. Absent a database primary key, the first non-nullable unique index whose key columns are all readable is inferred, in index order. The data adapter does this on the unnarrowed path, and dropping it made the narrowed path demand source.key-fields for objects the other path resolves on its own. The catalog read is deferred until an object is known to hold an unsupported type, so objects that need no narrowing no longer pay an extra query per startup. Adds unique_key_geometry_table and ValidateUniqueKeyIsInferredWhenNoDatabasePrimaryKeyExists. Addresses the Copilot review on Azure#3802.
All four are addressed. Two of them changed the approach, so noting what moved rather than only
Unique key inference when there is no primary key. Restored. The catalog read now returns the Covered by Catalog query on every object. Reordered as suggested. Classification now runs first over the Nine tests pass against SQL Server 2025 (RTM-CU8-GDR) 17.0.4085.5 Developer Edition in a local |
There was a problem hiding this comment.
🟡 Changes recommended
Database policies can still reference omitted columns and consequently fail every request at runtime.
Get a fresh assessment by requesting another Copilot review.
Review details
- Files reviewed: 7/7 changed files
- Comments generated: 2
- Review effort level: Balanced
…data A database policy is parsed per request against the OData model, which EdmModelBuilder builds from SourceDefinition.Columns, so one naming a column left out of the projection returns 400 on every request for that role. Configured references in mappings, fields, fields.include and policy.database are now all rejected at initialization. Policy field references are scanned rather than parsed, because the model the parser needs does not exist yet at that point. Adds identity assertions to ValidateUnsupportedColumnTypeIsNotInferred, plus decimal_identity_geometry_table and ValidateIdentityTypeIsPreservedOnTheNarrowedPath, which is what verifies that carrying identity from the catalog preserves a DataType that DataColumn.AutoIncrement would coerce to Int32. Addresses the second Copilot review on Azure#3802.
Why make this change?
A table holding a
geometrycolumn cannot be exposed at all.FillSchemaForTableAsyncreads the object shape withSELECT *, andDbDataAdapter.FillSchemaneeds a CLR type for every column in the projection. For a SQL Server CLR user-defined type the reader reports no CLR type, soSchemaMappingfails withDataReader.GetFieldType(N) returned nulland takes the whole object down. DAB does not referenceMicrosoft.SqlServer.Types. The entity never loads, and there is no configuration-side workaround — excluding the column through permissions does not help, because discovery runs before authorization is consulted.What is this change?
Schema discovery leaves out columns whose data type the provider cannot map to a CLR type, so the rest of the object stays reachable.
SqlMetadataProvidergains a virtualUnsupportedColumnDataTypes, empty by default. When it is empty the projection staysSELECT *, so PostgreSQL, MySQL and Cosmos are untouched.MsSqlMetadataProvideroverrides it with the three SQL Server CLR user-defined types:geometry,geography,hierarchyid.When a provider declares such types, the projection is built from the catalog and names the remaining columns. Identifiers come from the catalog's own
COLUMN_NAME, so casing and quoting match the database rather than the config. The catalog is read once per object and shared with column definition population, instead of being queried twice.The projection reproduces the set of columns
SELECT *exposes before subtracting the unsupported types. This matters because the catalog also reports columns declaredGENERATED ALWAYS AS ROW START/END HIDDEN, whichSELECT *does not return: naming them would widen the exposed contract rather than preserve it, and since the read-only classification does not recognize generated-always period columns, an overwriting PUT would target them. They are excluded throughsys.columns.is_hidden, and they are not reported as skipped — they were never part of the object's shape as the engine sees it.When — and only when — the projection is narrowed, the shape is read with
CommandBehavior.SchemaOnlyrather than throughDbDataAdapter.FillSchema.FillSchemaruns withKeyInfo, under which the provider performs its own key discovery and returns key columns absent from theSELECTlist as hidden reader columns; an unsupported type among them reintroducesDataReader.GetFieldType(N) returned nulleven though the projection excluded it. That is reachable through the object's own primary key and through any unique index, including when a supported key is configured throughsource.key-fields. WithoutKeyInfothe projection is authoritative, and the primary key comes from the catalog instead. When nothing is skipped,FillSchemastill runs withKeyInfoexactly as before, so ordinary primary-key inference is untouched for every object that works today.The primary key comes from the object's own key when it has one. Absent a primary key, the first unique index whose every key column is non-nullable and readable is used, in index order — which is what the data adapter reports as
DataTable.PrimaryKeyon the unnarrowed path, so the two paths agree and the narrowed one does not start demandingsource.key-fieldsfor an object the other resolves on its own.Identity columns are carried from
sys.columns.is_identityrather than throughDataColumn.AutoIncrement. That setter coerces a DataType it cannot increment toInt32, and SQL Server allows identity ontinyint,numericanddecimal, so using it as the transport would report the wrongSystemTypeand reach parameter typing and the generated API schemas.A primary key that cannot be read is rejected at initialization, with the column and its data type named. This covers a key configured over such a column, and the object's own database key when no key is configured — single column or one member of a composite key. Omitting it would leave the key in
SourceDefinition.PrimaryKeywhile absent fromSourceDefinition.Columns, and that inconsistency surfaces much later as a lookup failure. The generic "primary key not configured" error is not used for these, because it reads as something the user forgot to configure even though no configuration can express that key.Configuration naming a column left out of the projection also fails initialization. The exposed and backing column maps are built from entity fields and mappings without requiring the column to exist in
SourceDefinition.Columns, and the authorization resolver accepts explicitly included field names, so such a reference keeps resolving after the column is gone and then fails per request rather than at startup.mappings,fields, a permission'sfields.includeand a database policy are all rejected;excludeis left alone, since it asks for what already happened. The policy case matters for the same reason the field-permission approach was withdrawn: a policy is parsed per request against the OData model, whichEdmModelBuilderbuilds fromSourceDefinition.Columns, so one naming an absent column returns 400 on every request for that role.The catalog facts that the
Columnsschema collection does not carry — which columns the database hides fromSELECT *, and which columns form the object's primary key — come from a new provider hook,GetObjectCatalogMetadataAsync, which returns null in the base class. OnlyMsSqlMetadataProviderimplements it, readingsys.columnsjoined tosys.indexes/sys.index_columns, cached once per object for the duration of initialization. Providers that declare no unsupported types never call it, and neither does an object that holds none: the classification runs first over theColumnsschema collection, which is read for every object anyway, so only an object whose projection must be narrowed pays the additional query.If nothing unsupported is present, or the catalog cannot be read, or it does not report a usable type name, the projection falls back to
SELECT *and behavior is exactly what it is today. That last case includes an engine withoutsys.columns.is_hidden, which exists from SQL Server 2016 on, and a login withoutVIEW DEFINITION. When every column of an object is unsupported it fails with a clear initialization error instead, since falling back there would re-issue the projection that cannot be read.Skipped columns are logged once per object at Warning level, naming each column and its type, so the omission is discoverable rather than silent.
Deliberately limited to three types
MsSqlQueryBuilder's autoentity discovery skips objects containinggeography,geometry,hierarchyid,sql_variant,xml,rowversionorvector. That list is intentionally not reused here:SqlTypeConstants.SupportedSqlDbTypesmarkstimestampandvectoras supported, and there are dedicatedvectortests, so excluding them would regress shipped behavior. This change is scoped to the CLR user-defined types, which are the ones whose CLR type the provider does not report.Why not honor
fields.includeinsteadAn earlier revision of this PR narrowed the projection to the columns the permissions allow to be read. I withdrew it: read permission and physical existence are different things, and several paths need a column to be in
SourceDefinition.Columnswithout needing it to be readable. A database policy on an excluded column stops parsing against the EDM model (EdmModelBuilderiteratesColumns) and returns 400 on every request in that role; entities sharing asource.objectshare oneSourceDefinition, so a per-entity filter cannot isolate anything; multiple-create indexesColumnsby relationship column name; REST PUT stops nulling the hidden columns. Skipping by type has none of that blast radius, because a column the provider cannot type was never usable in any of those paths.Compatibility impact
Skipping is provider-wide and driven by the column's data type alone, so it does not consult permissions: a
geometry,geographyorhierarchyidcolumn is left out even for a role that grants unrestricted read, andfields.include/fields.excludeno longer narrow discovery. They are not ignored either — naming one of these columns infields,mappingsor a permission'sfields.includenow fails initialization, because the reference cannot be honored once the column is not part of the exposed contract.That is a deliberate widening, and it takes nothing away in practice. Those three types cannot be read today at all — the object fails discovery and the entity never loads, which is #3801. The observable change is that such an entity now loads with the column absent, instead of not loading. No column that previously reached
SourceDefinition.Columnsstops reaching it, so REST, GraphQL, OpenAPI, MCP and database policies see exactly what they see today for every object that works today.Objects that fail today now fail with a specific reason instead of the provider's opaque error: one whose configured or database primary key is of an unsupported type, and one whose every column is. Providers other than MSSQL and DWSQL declare no unsupported types and keep
SELECT *. DWSQL sharesMsSqlMetadataProviderand therefore inherits the override, which is inert there since Synapse has no spatial types.The one thing worth reviewing is discoverability: an omitted column is now a Warning in the log rather than a startup failure. I chose the Warning deliberately, but if you would rather the engine keep failing loudly and require an explicit opt-in per entity, that is a small change and I am happy to make it.
How was this tested?
Eleven tests in
src/Service.Tests/UnitTests/SqlMetadataProviderUnitTests.csinfer metadata against a live MSSQL instance:Run against SQL Server 2025 (RTM-CU8-GDR) 17.0.4085.5, Developer Edition, in a local Linux container, with
DatabaseSchema-MsSql.sqlapplied in full:MSSQL only. DWSQL, MySQL and PostgreSQL need engines I do not have locally, so CI remains the check for those suites — no fixture of theirs is touched by this change.
The fixtures follow the pattern already used for
vectorcolumns, insrc/Service.Tests/DatabaseSchema-MsSql.sql:geometry_type_tableandgeometry_only_view— a spatial column, and an object whose every column is spatial.temporal_geometry_type_table— system-versioned, withHIDDENperiod columns and a spatial column.hierarchyid_pk_tableandhierarchyid_composite_pk_table— an unsupported type as the database primary key, alone and as one member of a composite key.hierarchyid_unique_table— no database primary key, an unsupported column under a unique index, and a supported column to configure as the key.unique_key_geometry_table— no database primary key, a non-nullable unique index over a supported column, and a spatial column.decimal_identity_geometry_table— adecimal(18, 0)identity column, whose CLR typeDataColumn.AutoIncrementwould coerce toInt32, and a spatial column.config-generators/mssql-commands.txtandsrc/Service.Tests/dab-config.MsSql.jsoncarry theGeometryTypeentity, andsrc/Service.Tests/Snapshots/ConfigurationTests.TestReadingRuntimeConfigForMsSql.verified.txthas the matching block. I added only that block rather than accepting the full received file: on my machine the committeddab-config.MsSql.jsonand the committed snapshot already disagree on action ordering forBookandWebsiteUser, and I did not want to bake that local difference into the snapshot. Happy to regenerate both properly if you prefer.The entities for the objects added in the latest round are declared in memory inside the tests rather than in
dab-config.MsSql.json, since several of them fail by design and every MSSQL fixture initializes every configured entity. No further snapshot regeneration is involved.Sample Request(s)
Table:
Entity — no special configuration needed:
Before this change the engine fails while reading metadata for
Geom. After it, the entity loads and responds:with this in the log at startup:
A table holding a
geometrycolumn cannot be exposed at all.FillSchemaForTableAsyncreads theobject shape with
SELECT *, andDbDataAdapter.FillSchemaneeds a CLR type for every column inthe projection. For a SQL Server CLR user-defined type the reader reports no CLR type, so
SchemaMappingfails withDataReader.GetFieldType(N) returned nulland takes the whole objectdown. DAB does not reference
Microsoft.SqlServer.Types. The entity never loads, and there is noconfiguration-side workaround — excluding the column through permissions does not help, because
discovery runs before authorization is consulted.
What is this change?
Schema discovery leaves out columns whose data type the provider cannot map to a CLR type, so the
rest of the object stays reachable.
SqlMetadataProvidergains a virtualUnsupportedColumnDataTypes, empty by default. When it isempty the projection stays
SELECT *, so PostgreSQL, MySQL and Cosmos are untouched.MsSqlMetadataProvideroverrides it with the three SQL Server CLR user-defined types:geometry,geography,hierarchyid.When a provider declares such types, the projection is built from the catalog and names the
remaining columns. Identifiers come from the catalog's own
COLUMN_NAME, so casing and quotingmatch the database rather than the config. The catalog is read once per object and shared with
column definition population, instead of being queried twice.
The projection reproduces the set of columns
SELECT *exposes before subtracting theunsupported types. This matters because the catalog also reports columns declared
GENERATED ALWAYS AS ROW START/END HIDDEN, whichSELECT *does not return: naming them wouldwiden the exposed contract rather than preserve it, and since the read-only classification does
not recognize generated-always period columns, an overwriting PUT would target them. They are
excluded through
sys.columns.is_hidden, and they are not reported as skipped — they were neverpart of the object's shape as the engine sees it.
When — and only when — the projection is narrowed, the shape is read with
CommandBehavior.SchemaOnlyrather than throughDbDataAdapter.FillSchema.FillSchemarunswith
KeyInfo, under which the provider performs its own key discovery and returns key columnsabsent from the
SELECTlist as hidden reader columns; an unsupported type among themreintroduces
DataReader.GetFieldType(N) returned nulleven though the projection excluded it.That is reachable through the object's own primary key and through any unique index, including
when a supported key is configured through
source.key-fields. WithoutKeyInfothe projectionis authoritative, and the primary key comes from the catalog instead. When nothing is skipped,
FillSchemastill runs withKeyInfoexactly as before, so ordinary primary-key inference isuntouched for every object that works today.
The primary key comes from the object's own key when it has one. Absent a primary key, the first
unique index whose every key column is non-nullable and readable is used, in index order — which
is what the data adapter reports as
DataTable.PrimaryKeyon the unnarrowed path, so the twopaths agree and the narrowed one does not start demanding
source.key-fieldsfor an object theother resolves on its own.
Identity columns are carried from
sys.columns.is_identityrather than throughDataColumn.AutoIncrement. That setter coerces a DataType it cannot increment toInt32, and SQLServer allows identity on
tinyint,numericanddecimal, so using it as the transport wouldreport the wrong
SystemTypeand reach parameter typing and the generated API schemas.A primary key that cannot be read is rejected at initialization, with the column and its data
type named. This covers a key configured over such a column, and the object's own database key
when no key is configured — single column or one member of a composite key. Omitting it would
leave the key in
SourceDefinition.PrimaryKeywhile absent fromSourceDefinition.Columns, andthat inconsistency surfaces much later as a lookup failure. The generic "primary key not
configured" error is not used for these, because it reads as something the user forgot to
configure even though no configuration can express that key.
Configuration naming a column left out of the projection also fails initialization. The exposed
and backing column maps are built from entity fields and mappings without requiring the column to
exist in
SourceDefinition.Columns, and the authorization resolver accepts explicitly includedfield names, so such a reference keeps resolving after the column is gone and then fails per
request rather than at startup.
mappings,fields, a permission'sfields.includeand adatabase policy are all rejected;
excludeis left alone, since it asks for what alreadyhappened. The policy case matters for the same reason the field-permission approach was withdrawn:
a policy is parsed per request against the OData model, which
EdmModelBuilderbuilds fromSourceDefinition.Columns, so one naming an absent column returns 400 on every request for thatrole.
The catalog facts that the
Columnsschema collection does not carry — which columns thedatabase hides from
SELECT *, and which columns form the object's primary key — come from a newprovider hook,
GetObjectCatalogMetadataAsync, which returns null in the base class. OnlyMsSqlMetadataProviderimplements it, readingsys.columnsjoined tosys.indexes/sys.index_columns, cached once per object for the duration of initialization. Providers thatdeclare no unsupported types never call it, and neither does an object that holds none: the
classification runs first over the
Columnsschema collection, which is read for every objectanyway, so only an object whose projection must be narrowed pays the additional query.
If nothing unsupported is present, or the catalog cannot be read, or it does not report a usable
type name, the projection falls back to
SELECT *and behavior is exactly what it is today. Thatlast case includes an engine without
sys.columns.is_hidden, which exists from SQL Server 2016on, and a login without
VIEW DEFINITION. When every column of an object is unsupported it failswith a clear initialization error instead, since falling back there would re-issue the projection
that cannot be read.
Skipped columns are logged once per object at Warning level, naming each column and its type, so
the omission is discoverable rather than silent.
Deliberately limited to three types
MsSqlQueryBuilder's autoentity discovery skips objects containinggeography,geometry,hierarchyid,sql_variant,xml,rowversionorvector. That list is intentionally notreused here:
SqlTypeConstants.SupportedSqlDbTypesmarkstimestampandvectoras supported,and there are dedicated
vectortests, so excluding them would regress shipped behavior. Thischange is scoped to the CLR user-defined types, which are the ones whose CLR type the provider
does not report.
Why not honor
fields.includeinsteadAn earlier revision of this PR narrowed the projection to the columns the permissions allow to be
read. I withdrew it: read permission and physical existence are different things, and several
paths need a column to be in
SourceDefinition.Columnswithout needing it to be readable. Adatabase policy on an excluded column stops parsing against the EDM model (
EdmModelBuilderiterates
Columns) and returns 400 on every request in that role; entities sharing asource.objectshare oneSourceDefinition, so a per-entity filter cannot isolate anything;multiple-create indexes
Columnsby relationship column name; REST PUT stops nulling the hiddencolumns. Skipping by type has none of that blast radius, because a column the provider cannot
type was never usable in any of those paths.
Compatibility impact
Skipping is provider-wide and driven by the column's data type alone, so it does not consult
permissions: a
geometry,geographyorhierarchyidcolumn is left out even for a role thatgrants unrestricted read, and
fields.include/fields.excludeno longer narrow discovery.They are not ignored either — naming one of these columns in
fields,mappingsor apermission's
fields.includenow fails initialization, because the reference cannot be honoredonce the column is not part of the exposed contract.
That is a deliberate widening, and it takes nothing away in practice. Those three types cannot be
read today at all — the object fails discovery and the entity never loads, which is #3801. The
observable change is that such an entity now loads with the column absent, instead of not
loading. No column that previously reached
SourceDefinition.Columnsstops reaching it, so REST,GraphQL, OpenAPI, MCP and database policies see exactly what they see today for every object that
works today.
Objects that fail today now fail with a specific reason instead of the provider's opaque error:
one whose configured or database primary key is of an unsupported type, and one whose every
column is. Providers other than MSSQL and DWSQL declare no unsupported types and keep
SELECT *. DWSQL sharesMsSqlMetadataProviderand therefore inherits the override, which isinert there since Synapse has no spatial types.
The one thing worth reviewing is discoverability: an omitted column is now a Warning in the log
rather than a startup failure. I chose the Warning deliberately, but if you would rather the
engine keep failing loudly and require an explicit opt-in per entity, that is a small change and
I am happy to make it.
How was this tested?
Eleven tests in
src/Service.Tests/UnitTests/SqlMetadataProviderUnitTests.csinfer metadataagainst a live MSSQL instance:
ValidateUnsupportedColumnTypeIsNotInferredgeometrycolumn is absent from the inferredSourceDefinitionwhile the supported columns are present, on the strength of its type aloneValidateObjectWithOnlyUnsupportedColumnsFailsInitializationValidatePrimaryKeyOnUnsupportedColumnFailsInitializationValidateHiddenPeriodColumnsAreNotInferredHIDDENperiod columns of a temporal table stay out of the exposed contract, as does the spatial columnValidateUnsupportedDatabasePrimaryKeyFailsInitializationValidateUnsupportedColumnInCompositeDatabasePrimaryKeyFailsInitializationValidateConfiguredKeyIsUsedWhenUniqueIndexColumnIsUnsupportedsource.key-fieldsValidateUniqueKeyIsInferredWhenNoDatabasePrimaryKeyExistssource.key-fieldsconfiguredValidateConfiguredReferenceToUnsupportedColumnFailsInitializationmappingsentry over an unsupported column fails initializationValidateDatabasePolicyOverUnsupportedColumnFailsInitializationValidateIdentityTypeIsPreservedOnTheNarrowedPathdecimalidentity column keeps its reported type and is still marked auto-generatedRun against SQL Server 2025 (RTM-CU8-GDR) 17.0.4085.5, Developer Edition, in a local Linux
container, with
DatabaseSchema-MsSql.sqlapplied in full:MSSQL only. DWSQL, MySQL and PostgreSQL need engines I do not have locally, so CI remains the
check for those suites — no fixture of theirs is touched by this change.
The fixtures follow the pattern already used for
vectorcolumns, insrc/Service.Tests/DatabaseSchema-MsSql.sql:geometry_type_tableandgeometry_only_view— a spatial column, and an object whose everycolumn is spatial.
temporal_geometry_type_table— system-versioned, withHIDDENperiod columns and a spatialcolumn.
hierarchyid_pk_tableandhierarchyid_composite_pk_table— an unsupported type as thedatabase primary key, alone and as one member of a composite key.
hierarchyid_unique_table— no database primary key, an unsupported column under a uniqueindex, and a supported column to configure as the key.
unique_key_geometry_table— no database primary key, a non-nullable unique index over asupported column, and a spatial column.
decimal_identity_geometry_table— adecimal(18, 0)identity column, whose CLR typeDataColumn.AutoIncrementwould coerce toInt32, and a spatial column.config-generators/mssql-commands.txtandsrc/Service.Tests/dab-config.MsSql.jsoncarry theGeometryTypeentity, andsrc/Service.Tests/Snapshots/ConfigurationTests.TestReadingRuntimeConfigForMsSql.verified.txthas the matching block. I added only that block rather than accepting the full received file: on
my machine the committed
dab-config.MsSql.jsonand the committed snapshot already disagree onaction ordering for
BookandWebsiteUser, and I did not want to bake that local differenceinto the snapshot. Happy to regenerate both properly if you prefer.
The entities for the objects added in the latest round are declared in memory inside the tests
rather than in
dab-config.MsSql.json, since several of them fail by design and every MSSQLfixture initializes every configured entity. No further snapshot regeneration is involved.
Sample Request(s)
Table:
Entity — no special configuration needed:
Before this change the engine fails while reading metadata for
Geom. After it, the entity loadsand responds:
{ "value": [ { "Id": 1, "Name": "Shaft collar" } ] }with this in the log at startup: