Support non-primitive column types across SQL Server, Postgres and MySQL (supersedes #72)

.github/postgres/Dockerfile

@@ -0,0 +1,5 @@
+# PostgreSQL test image with the PostGIS and pgvector extensions (used to test the extension data types).
+FROM postgres:17
+RUN apt-get update \
+    && apt-get install -y --no-install-recommends postgresql-17-postgis-3 postgresql-17-pgvector \
+    && rm -rf /var/lib/apt/lists/*

.github/workflows/CI.yml

@@ -12,10 +12,10 @@ jobs:
 
     steps:
     - name: Checkout
-      uses: actions/checkout@v2
+      uses: actions/checkout@v7
 
     - name: Setup .NET
-      uses: actions/setup-dotnet@v1
+      uses: actions/setup-dotnet@v6
       with:
         dotnet-version: |
             8.0.x
@@ -28,26 +28,8 @@ jobs:
     - name: Build
       run: dotnet build --no-restore
 
-    - name: Pull SqlServer
-      run: docker pull mcr.microsoft.com/mssql/server
-
-    - name: Start SqlServer
-      run: docker run -e "ACCEPT_EULA=Y" -e "SA_PASSWORD=yourStrong(!)Password" --name sqlserver -p 1433:1433 -d mcr.microsoft.com/mssql/server:2019-latest
-
-    - name: Pull MySQL
-      run: docker pull mysql
-
-    - name: Start MySQL
-      run: docker run --name db-mysql -p 3306:3306 -e MYSQL_ROOT_PASSWORD=yourStrong#!Password -d mysql
-
-    - name: Pull Postgres
-      run: docker pull postgres
-
-    - name: Start Postgres
-      run: docker run --name db-postgres -p 5432:5432 -e POSTGRES_PASSWORD=yourStrong#!Password -d postgres
-
-    - name: Sleep (allow databases to complete startup)
-      run: sleep 10
+    - name: Start databases (SQL Server, MySQL, Postgres)
+      run: docker compose up -d --build --wait --wait-timeout 180
 
     - name: Set EnvVar for Test
       run: |

CHANGELOG.md

@@ -2,6 +2,16 @@
 
 Represents the **NuGet** versions.
 
+## v3.3.0
+- *Enhancement:* Non-primitive database types are now mapped to their correct .NET types (unqualified; a `global using` is expected) instead of throwing `Database data type 'x' does not have corresponding .NET type mapping defined`, which previously failed the `Data`, `Reset` and `Inspect` commands for the whole database. Originated from PR #72 by Wesley Teixeira.
+  - SQL Server: `geography`/`geometry` (`Geometry`), `hierarchyid` (`HierarchyId`), `vector` (`SqlVector<float>`), `xml`/`json` (`string`) and `sql_variant` (`object`).
+  - Postgres: network, text-search, geometric, bit-string, range/multirange, array, `hstore`, `ltree`, `citext`, enum, PostGIS and pgvector types.
+  - MySql: `mediumint`, `year`, `bit(n)`, `vector` and spatial types (a spatial value with an `SRID=n;` prefix is interpreted as longitude then latitude, consistent with Postgres and SQL Server).
+- *Enhancement:* Added `DbColumnSchema.DataParserType` (the primitive type the data parser uses to parse YAML/JSON values; `DotNetType` remains the true EF type), `IsDotNetTypeAClass` (generated EF models now emit `= default!` for non-nullable class types), `IsDataComparable`, `UdtName` and `NativeSqlType`. The provider `DatabaseSchemaConfig` has corresponding virtual methods.
+- *Fixed:* SQL Server `BINARY(n)` and `VECTOR(n)` columns are now formatted with their length/dimensions.
+- *Fixed:* SQL Server data merge no longer compares non-comparable columns (e.g. `xml`, `geography`) using `EXCEPT`.
+- *Note:* Test databases are now SQL Server 2025, MySQL 9 and Postgres 17 with PostGIS and pgvector (see `.github/postgres/Dockerfile`).
+
 ## v3.2.0
 - *Enhancement:* The `MigrationCommand.Execute` has been enhanced to support the execution of YAML/JSON data files. Raw SQL must _now_ be prefixed by `>` to explicitly differentiate from a file path.
 - *Enhancement:* The `MigrationCommand.Inspect` has been enhanced to also report whether each column has been identified as a JSON column (i.e. `DbColumnSchema.IsJson`).

CONTRIBUTING.md

@@ -51,6 +51,8 @@ We use [`NUnit`](https://github.com/nunit/nunit) for all unit testing.
 
 We understand there is more work to be performed in generating a higher level of code coverage; this technical debt is on the backlog.
 
+The tests are executed against live SQL Server, MySQL and Postgres (with PostGIS and pgvector) databases. These can be started using the [`docker-compose.yml`](./docker-compose.yml) (`docker compose up -d --build`); the `DbEx_ConnectionStrings__*` environment variables, as set within the [CI](./.github/workflows/CI.yml) workflow, must then be configured to reference them.
+
 <br/>
 
 ## Code reviews and checkins

Common.targets

@@ -1,6 +1,6 @@
 <Project>
   <PropertyGroup>
-    <Version>3.2.0</Version>
+    <Version>3.3.0</Version>
     <LangVersion>preview</LangVersion>
     <Authors>Avanade</Authors>
     <Company>Avanade</Company>

README.md

@@ -319,6 +319,31 @@ Within a code-generation, or other context, the database schema may need to be i
 
 The [`Database`](./src/DbEx/DatabaseExtensions.cs) class provides a `SelectSchemaAsync` method to return a [`DbTableSchema`](./src/DbEx/DbSchema/DbTableSchema.cs) list, including the respective columns for each table (see [`DbColumnSchema`](./src/DbEx/DbSchema/DbColumnSchema.cs)).
 
+### Data types
+
+Each [`DbColumnSchema`](./src/DbEx/DbSchema/DbColumnSchema.cs) exposes the inferred .NET type, and how data is parsed:
+
+Property | Description
+-|-
+`DotNetType` | The .NET type a column maps to, e.g. `string`, `int`, `Geometry` or `NpgsqlRange<int>`. Types are _not_ namespace-qualified; the consuming project is expected to have the required `global using` statements in place for the types used (e.g. NetTopologySuite, Npgsql and Pgvector types).
+`IsDotNetTypeAClass` | Indicates whether `DotNetType` is a reference type (e.g. a non-nullable property that needs initializing, such as `= default!`). Value types, including structs such as `NpgsqlCidr`, return `false`.
+`DataParserType` | The primitive .NET type used to parse a YAML/JSON data value for the column. For a type that has no primitive representation, e.g. spatial, vector and network types, this is `string` and the value is specified using its text form (see below); the provider then converts to the database type.
+`NativeSqlType` | The full native database type (including precision, length and dimensions) where available, e.g. `vector(3)`.
+`UdtName` | The underlying user-defined/extension type name where applicable (Postgres).
+`IsDataComparable` | Indicates whether the database can compare values (e.g. SQL Server `xml` and `geography` cannot). Where not, the SQL Server `$` merge updates matched rows without the changed-data check.
+
+Non-primitive types are supported by each provider, including (non-exhaustive):
+
+Provider | Types
+-|-
+SQL Server | `geography` and `geometry` (`Geometry`), `hierarchyid` (`HierarchyId`), `vector` (`SqlVector<float>`), `xml` and `json` (`string`), `sql_variant` (`object`).
+Postgres | network (`inet`, `cidr`, `macaddr`), text search, geometric, bit-string, range and multirange, array, `hstore`, `ltree`, `citext`, enum, plus PostGIS (`geometry`, `geography`) and pgvector (`vector`, `halfvec`, `sparsevec`) extensions.
+MySQL | `mediumint`, `year`, `bit(n)`, `vector`, and the spatial types (`geometry`, `point`, `linestring`, `polygon`, etc.).
+
+For the _data_ YAML/JSON, spatial values are specified as [well-known text](https://en.wikipedia.org/wiki/Well-known_text_representation_of_geometry) (WKT), with an optional [extended](https://postgis.net/docs/using_postgis_dbmanagement.html#EWKB_EWKT) `SRID=n;` prefix, e.g. `SRID=4326;POINT(-122.349 47.651)`. The coordinate order is _always_ longitude then latitude (x y); MySQL is explicitly instructed to use this order as it otherwise defaults to latitude then longitude for geographic coordinate systems. A vector is specified as `[1, 2, 3]`, and a multi-bit value as a bit string, e.g. `10101010`.
+
+> Note: where a database extension is required (e.g. PostGIS, pgvector) it must be enabled by a migration script before the data is loaded. See the [`DbEx.Test.PostgresConsole`](./tests/DbEx.Test.PostgresConsole/Migrations/006-create-test-extratypes-table.pgsql) test for an example.
+
 <br/>
 
 ## Other considerations

docker-compose.yml

@@ -2,23 +2,43 @@ name: DbEx
 
 services:
   db-sql-server:
-    image: mcr.microsoft.com/mssql/server:2019-latest
+    image: mcr.microsoft.com/mssql/server:2025-latest
     environment:
       SA_PASSWORD: "yourStrong(!)Password"
       ACCEPT_EULA: "Y"
     ports:
       - "1433:1433"
+    healthcheck:
+      test: ["CMD-SHELL", "/opt/mssql-tools18/bin/sqlcmd -C -S localhost -U sa -P \"$$SA_PASSWORD\" -Q \"SELECT 1\" -b -o /dev/null"]
+      interval: 5s
+      timeout: 5s
+      retries: 30
+      start_period: 10s
 
   db-mysql:
-    image: mysql
+    image: mysql:9
     environment:
       MYSQL_ROOT_PASSWORD: "yourStrong#!Password"
     ports:
       - "3306:3306"
+    healthcheck:
+      # TCP (not the socket) as the temporary initialization server only listens on the socket.
+      test: ["CMD-SHELL", "mysqladmin ping -h 127.0.0.1 -uroot -p\"$$MYSQL_ROOT_PASSWORD\" --silent"]
+      interval: 5s
+      timeout: 5s
+      retries: 30
+      start_period: 10s
 
   db-postgres:
-    image: postgres
+    build: ./.github/postgres
+    image: dbex-postgres-test
     environment:
       POSTGRES_PASSWORD: "yourStrong#!Password"
     ports:
-      - "5432:5432"
+      - "5432:5432"
+    healthcheck:
+      test: ["CMD-SHELL", "pg_isready -h 127.0.0.1 -U postgres"]
+      interval: 5s
+      timeout: 5s
+      retries: 30
+      start_period: 10s

src/DbEx.MySql/MySqlSchemaConfig.cs

@@ -81,10 +81,11 @@ public override DbColumnSchema CreateColumnFromInformationSchema(DbTableSchema t
             Scale = dr.GetValue<ulong?>("NUMERIC_SCALE"),
             DefaultValue = dr.GetValue<string>("COLUMN_DEFAULT"),
             IsDotNetDateOnly = RemovePrecisionFromDataType(dt).Equals("DATE", StringComparison.OrdinalIgnoreCase),
-            IsDotNetTimeOnly = RemovePrecisionFromDataType(dt).Equals("TIME", StringComparison.OrdinalIgnoreCase)
+            IsDotNetTimeOnly = RemovePrecisionFromDataType(dt).Equals("TIME", StringComparison.OrdinalIgnoreCase),
+            NativeSqlType = dr.GetValue<string?>("COLUMN_TYPE")
         };
 
-        c.IsJsonContent = c.Type.Equals("JSON", StringComparison.OrdinalIgnoreCase) || (c.DotNetType == "string" && c.Name.EndsWith(JsonColumnNameSuffix, StringComparison.Ordinal));
+        c.IsJsonContent = c.Type.Equals("JSON", StringComparison.OrdinalIgnoreCase) || (c.Name.EndsWith(JsonColumnNameSuffix, StringComparison.Ordinal) && c.DotNetType == "string");
         if (c.IsJsonContent && c.Name.EndsWith(JsonColumnNameSuffix, StringComparison.Ordinal))
             c.DotNetCleanedName = DbTableSchema.CreateDotNetName(c.Name[..^JsonColumnNameSuffix.Length]);
 
@@ -160,26 +161,43 @@ public override string ToDotNetTypeName(DbColumnSchema schema)
         else if (Migration.Args.EmitDotNetTimeOnly && schema.IsDotNetTimeOnly)
             return "TimeOnly";
 
+        // Source of truth: https://mysqlconnector.net/overview/mysql-to-dotnet-types/ and the Pomelo.EntityFrameworkCore.MySql type mappings (spatial types use NetTopologySuite).
         return dbType.ToUpperInvariant() switch
         {
             "CHAR" or "VARCHAR" or "TINYTEXT" or "TEXT" or "MEDIUMTEXT" or "LONGTEXT" or "SET" or "ENUM" or "NCHAR" or "NVARCHAR" or "JSON" => "string",
-            "DECIMAL" => "decimal",
+            "DECIMAL" or "NUMERIC" => "decimal",
             "DATETIME" or "TIMESTAMP" => "DateTime",
             "DATE" => "DateOnly",
             "TIME" => "TimeOnly",
-            "BINARY" or "VARBINARY" or "TINYBLOB" or "BLOB" or "MEDIUMBLOB" or "LONGBLOB" => "byte[]",
-            "BIT" or "BOOL" or "BOOLEAN" => "bool",
+            "BINARY" or "VARBINARY" or "TINYBLOB" or "BLOB" or "MEDIUMBLOB" or "LONGBLOB" or "VECTOR" => "byte[]",
+            "BOOL" or "BOOLEAN" => "bool",
+            "BIT" => schema.Precision is null or 1 ? "bool" : "ulong",
             "DOUBLE" => "double",
-            "INT" => "int",
+            "INT" or "INTEGER" or "MEDIUMINT" => "int",
             "BIGINT" => "long",
-            "SMALLINT" => "short",
+            "SMALLINT" or "YEAR" => "short",
             "TINYINT" => "byte",
             "FLOAT" => "float",
-            "UNIQUEIDENTIFIER" => "Guid",
+            "GEOMETRY" => "Geometry",
+            "POINT" => "Point",
+            "LINESTRING" => "LineString",
+            "POLYGON" => "Polygon",
+            "MULTIPOINT" => "MultiPoint",
+            "MULTILINESTRING" => "MultiLineString",
+            "MULTIPOLYGON" => "MultiPolygon",
+            "GEOMETRYCOLLECTION" or "GEOMCOLLECTION" => "GeometryCollection",
             _ => throw new InvalidOperationException($"Database data type '{dbType}' does not have corresponding .NET type mapping defined."),
         };
     }
 
+    /// <inheritdoc/>
+    /// <remarks>A <c>VECTOR</c> is a <c>byte[]</c> in .NET, but is parsed from (and written as) its text (string) representation (e.g. <c>[1,2,3]</c>); likewise a multi-bit <c>BIT</c> is a <c>ulong</c> in .NET (as per the MySql.Data and Pomelo EF providers), but is parsed from its bit-string (e.g. <c>10101010</c>).</remarks>
+    public override string ToDataParserTypeName(DbColumnSchema schema)
+    {
+        var type = schema.ThrowIfNull(nameof(schema)).Type.ToUpperInvariant();
+        return (type is "VECTOR" && schema.DotNetType == "byte[]") || (type is "BIT" && schema.DotNetType == "ulong") ? "string" : base.ToDataParserTypeName(schema);
+    }
+
     /// <inheritdoc/>
     public override string ToFormattedSqlType(DbColumnSchema schema, bool includeNullability = true)
     {
@@ -190,7 +208,9 @@ public override string ToFormattedSqlType(DbColumnSchema schema, bool includeNul
             "CHAR" or "VARCHAR" or "NCHAR" or "NVARCHAR" => schema.Length.HasValue && schema.Length.Value > 0 ? $"({schema.Length.Value})" : "(MAX)",
             "DECIMAL" => $"({schema.Precision}, {schema.Scale})",
             "NUMERIC" => $"({schema.Precision}, {schema.Scale})",
+            "BIT" => schema.Precision.HasValue ? $"({schema.Precision.Value})" : string.Empty,
             "TIME" or "DATETIME" or "TIMESTAMP" => schema.Scale.HasValue && schema.Scale.Value > 0 ? $"({schema.Scale})" : string.Empty,
+            "VECTOR" => ExtractTypeArguments(schema.NativeSqlType),
             _ => string.Empty
         });
 
@@ -200,19 +220,56 @@ public override string ToFormattedSqlType(DbColumnSchema schema, bool includeNul
         return sb.ToString();
     }
 
+    /// <summary>
+    /// Extracts the parenthesized type arguments (e.g. <c>(3)</c>) from a native SQL type (e.g. <c>vector(3)</c>).
+    /// </summary>
+    private static string ExtractTypeArguments(string? nativeSqlType)
+    {
+        var start = nativeSqlType?.IndexOf('(') ?? -1;
+        var end = nativeSqlType?.IndexOf(')') ?? -1;
+        return nativeSqlType is not null && start >= 0 && end > start ? nativeSqlType[start..(end + 1)] : string.Empty;
+    }
+
     /// <inheritdoc/>
-    public override string ToFormattedSqlStatementValue(DbColumnSchema dbColumnSchema, object? value) => value switch
+    public override string ToFormattedSqlStatementValue(DbColumnSchema dbColumnSchema, object? value)
     {
-        null => "NULL",
-        string str => $"'{str.Replace("'", "''", StringComparison.Ordinal)}'",
-        bool b => b ? "true" : "false",
-        Guid => $"'{value}'",
-        DateTime dt => $"'{dt.ToString(Migration.Args.DataParserArgs.DateTimeFormat, System.Globalization.CultureInfo.InvariantCulture)}'",
-        DateTimeOffset dto => $"'{dto.ToString(Migration.Args.DataParserArgs.DateTimeOffsetFormat, System.Globalization.CultureInfo.InvariantCulture)}'",
+        if (value is string text)
+        {
+            var type = dbColumnSchema.Type.ToUpperInvariant();
+            if (type is "VECTOR")
+                return $"STRING_TO_VECTOR('{text.Replace("'", "''", StringComparison.Ordinal)}')";
+
+            if (type is "BIT")
+                return $"b'{text.Replace("'", "''", StringComparison.Ordinal)}'";
+
+            if (type is "GEOMETRY" or "POINT" or "LINESTRING" or "POLYGON" or "MULTIPOINT" or "MULTILINESTRING" or "MULTIPOLYGON" or "GEOMETRYCOLLECTION" or "GEOMCOLLECTION")
+            {
+                // Supports the extended well-known text (EWKT) form of 'SRID=n;WKT' to specify the spatial reference system; the long-lat axis order is explicit as MySQL otherwise defaults to lat-long for geographic SRIDs (differs to PostGIS and SQL Server).
+                var srid = string.Empty;
+                if (text.StartsWith("SRID=", StringComparison.OrdinalIgnoreCase) && text.IndexOf(';') is var i and > 0 && uint.TryParse(text[5..i], out var s))
+                {
+                    srid = $", {s}, 'axis-order=long-lat'";
+                    text = text[(i + 1)..];
+                }
+
+                return $"ST_GeomFromText('{text.Replace("'", "''", StringComparison.Ordinal)}'{srid})";
+            }
+        }
+
+        return value switch
+        {
+            null => "NULL",
+            string str => $"'{str.Replace("'", "''", StringComparison.Ordinal)}'",
+            bool b when dbColumnSchema.Type.Equals("BIT", StringComparison.OrdinalIgnoreCase) => b ? "b'1'" : "b'0'",
+            bool b => b ? "true" : "false",
+            Guid => $"'{value}'",
+            DateTime dt => $"'{dt.ToString(Migration.Args.DataParserArgs.DateTimeFormat, System.Globalization.CultureInfo.InvariantCulture)}'",
+            DateTimeOffset dto => $"'{dto.ToString(Migration.Args.DataParserArgs.DateTimeOffsetFormat, System.Globalization.CultureInfo.InvariantCulture)}'",
 #if NET7_0_OR_GREATER
-        DateOnly d => $"'{d.ToString(Migration.Args.DataParserArgs.DateOnlyFormat, System.Globalization.CultureInfo.InvariantCulture)}'",
-        TimeOnly t => $"'{t.ToString(Migration.Args.DataParserArgs.TimeOnlyFormat, System.Globalization.CultureInfo.InvariantCulture)}'",
+            DateOnly d => $"'{d.ToString(Migration.Args.DataParserArgs.DateOnlyFormat, System.Globalization.CultureInfo.InvariantCulture)}'",
+            TimeOnly t => $"'{t.ToString(Migration.Args.DataParserArgs.TimeOnlyFormat, System.Globalization.CultureInfo.InvariantCulture)}'",
 #endif
-        _ => value.ToString()!
-    };
+            _ => value.ToString()!
+        };
+    }
 }