Kevin Chen SQL Server performance case study
Theme

SQL Server performance case study

Five common production problems on a fictional courier schema, reproduced, fixed and measured. Measured on Microsoft SQL Server 2019 (RTM-CU27-GDR) (KB5040948) - 15.0.4382.1 (X64), median of 7 runs with a warm cache, on 2026-11-04. Generated from the repo's results files.

Results

ScenarioReads beforeReads afterElapsed beforeElapsed after
01 non sargable date, one-day report 6,3992,532 108.4 ms2.1 ms
02 key lookups, zone despatch board 2,5283 1.4 ms0.3 ms
03 parameter sniffing, large customer, small customer ran first 360,1726,399 304.0 ms185.1 ms
03 parameter sniffing, small customer, large customer ran first 6,39957 37.7 ms1.3 ms
04 implicit conversion, lookup by consignment number 2,9186 66.2 ms0.3 ms

Logical reads came out the same on every run, so they are the stable measure. Elapsed times moved by up to about 50% between runs, so treat them as orders of magnitude. This is a developer machine with everything in memory, not a promise about your server.

Function on a date column

Problem. A daily report filters with CONVERT(CHAR(10), BookedUtc, 120) = '2025-03-14'. Wrapping the column in a function means SQL Server cannot seek the index on BookedUtc, so it scans all 600,000 jobs and converts each date to text.

Fix. Filter the bare column with a half-open range: BookedUtc >= '2025-03-14' AND BookedUtc < '2025-03-15'. No schema change, same 822 rows.

Measured. Logical reads went from 6,399 to 2,532. Elapsed time went from about 108 ms to about 2 ms.

Worth noticing. Reads fell by only 2.5 times but elapsed time fell by about 50 times. Most of the original cost was CPU spent converting 600,000 dates to strings, which a reads-only review would underrate. The 2,532 reads that remain are key lookups, and scenario 02 shows how to remove that kind of cost when a query runs often enough to justify an index.

One-day report 822 rows, same result before and after

Logical reads
Before6,399
After2,532
Elapsed, median of 7 runs
Before108.4 ms
After2.1 ms
CPU
Before109 ms
After0 ms

Plan before

Clustered Index Scan PK_Jobs  actual 822, est. 1
01-non-sargable-date-before-1.sqlplan

Plan after

Nested Loops  actual 822, est. 823
  Nested Loops  actual 822, est. 823
    Merge Interval  actual 1, est. 2
      Concatenation  actual 2, est. 2
        Compute Scalar  actual 1, est. 1
          Constant Scan  actual 1, est. 1
        Compute Scalar  actual 1, est. 1
          Constant Scan  actual 1, est. 1
    Index Seek IX_Jobs_BookedUtc  actual 822, est. 823
  Key Lookup PK_Jobs  actual 822, est. 1
01-non-sargable-date-after-1.sqlplan
The SQL, before and after

before.sql

-- Daily report written by converting the column to text. The index on BookedUtc cannot be used for a seek.
-- BENCH:MEASURE one-day report
SELECT JobId, CustomerId, BookedUtc, JobStatus
FROM dbo.Jobs
WHERE CONVERT(CHAR(10), BookedUtc, 120) = '2025-03-14';

after.sql

-- Same report as a half-open date range on the bare column. No schema change needed.
-- BENCH:MEASURE one-day report
SELECT JobId, CustomerId, BookedUtc, JobStatus
FROM dbo.Jobs
WHERE BookedUtc >= '2025-03-14' AND BookedUtc < '2025-03-15';

Key lookups from the wrong index

Problem. A despatch board asks for one pickup zone on one day. The only useful index is on BookedUtc, so SQL Server reads the day for all 60 zones and does a key lookup per row (822 lookups) to test the zone and fetch the other columns. Only 21 rows come back.

Fix. A covering index with the equality column first: (PickupZoneId, BookedUtc) INCLUDE (CustomerId, DriverId, JobStatus, ServiceLevel).

Measured. Logical reads went from 2,528 to 3. Elapsed time went from about 1.4 ms to about 0.3 ms.

Worth noticing. On one execution with a warm cache that time saving is tiny, and it would be dishonest to sell it as more. The value is in reads: a board refreshed by many users all day does about 800 times less page work, which leaves more buffer pool for everything else. Every index also slows writes and takes disk space, so a real review weighs this against the write load and looks for existing indexes it can replace.

cleanup.sql drops the index afterwards, because it also happens to cover the query in scenario 01 and would change that scenario's "before" numbers. The benchmark hit exactly that on its second run, which is why the cleanup exists.

Zone despatch board 21 rows, same result before and after

Logical reads
Before2,528
After3
Elapsed, median of 7 runs
Before1.4 ms
After0.3 ms
CPU
Before0 ms
After0 ms

Plan before

Nested Loops  actual 21, est. 823
  Nested Loops  actual 821, est. 823
    Merge Interval  actual 1, est. 2
      Concatenation  actual 2, est. 2
        Compute Scalar  actual 1, est. 1
          Constant Scan  actual 1, est. 1
        Compute Scalar  actual 1, est. 1
          Constant Scan  actual 1, est. 1
    Index Seek IX_Jobs_BookedUtc  actual 821, est. 823
  Key Lookup PK_Jobs  actual 21, est. 106
02-key-lookups-before-1.sqlplan

Plan after

Nested Loops  actual 21, est. 106
  Merge Interval  actual 1, est. 2
    Concatenation  actual 2, est. 2
      Compute Scalar  actual 1, est. 1
        Constant Scan  actual 1, est. 1
      Compute Scalar  actual 1, est. 1
        Constant Scan  actual 1, est. 1
  Index Seek IX_Jobs_PickupZone_BookedUtc  actual 21, est. 106
02-key-lookups-after-1.sqlplan
The SQL, before and after

before.sql

-- Despatch board for one pickup zone. Only the index on BookedUtc helps, so the query seeks a one-day
-- range for all zones and does a key lookup per row just to test PickupZoneId and fetch the other columns.
DROP INDEX IF EXISTS IX_Jobs_PickupZone_BookedUtc ON dbo.Jobs;
GO
-- BENCH:MEASURE zone despatch board
SELECT JobId, CustomerId, DriverId, JobStatus, ServiceLevel, BookedUtc
FROM dbo.Jobs
WHERE PickupZoneId = 17
  AND BookedUtc >= '2025-10-06' AND BookedUtc < '2025-10-07';

after.sql

-- Covering index: equality column first, then the range column, with the selected columns included.
DROP INDEX IF EXISTS IX_Jobs_PickupZone_BookedUtc ON dbo.Jobs;
GO
CREATE INDEX IX_Jobs_PickupZone_BookedUtc
    ON dbo.Jobs (PickupZoneId, BookedUtc)
    INCLUDE (CustomerId, DriverId, JobStatus, ServiceLevel);
GO
-- BENCH:MEASURE zone despatch board
SELECT JobId, CustomerId, DriverId, JobStatus, ServiceLevel, BookedUtc
FROM dbo.Jobs
WHERE PickupZoneId = 17
  AND BookedUtc >= '2025-10-06' AND BookedUtc < '2025-10-07';

Parameter sniffing on a stored procedure

Problem. dbo.GetCustomerJobs returns a customer's job history. Customer 1 has 119,987 jobs and customer 5000 has 18. SQL Server builds the cached plan for whichever value it sees first, so the same procedure is fast or slow depending on who ran first after the last recompile. That is why this kind of issue often looks random and "fixes itself" after a restart.

Fix. OPTION (RECOMPILE) on the statement, so each call gets a plan for its own value.

Measured.

Worth noticing. Both directions are bad. The large customer suffers from 120,000 key lookups, and the small customer pays for a full scan to get 18 rows. OPTION (RECOMPILE) is not free, since every call compiles, and that is acceptable here but not for a procedure called thousands of times a minute. In that case I would look at a covering index, OPTIMIZE FOR, or separate code paths for the few very large customers. Which one is right depends on call frequency, so the choice is made per procedure.

Large customer, small customer ran first 119,987 rows, same result before and after

Logical reads
Before360,172
After6,399
Elapsed, median of 7 runs
Before304.0 ms
After185.1 ms
CPU
Before219 ms
After78 ms

Plan before

Sort  actual 119,987, est. 21
  Nested Loops  actual 119,987, est. 21
    Index Seek IX_Jobs_CustomerId  actual 119,987, est. 21
    Key Lookup PK_Jobs  actual 119,987, est. 1
03-parameter-sniffing-before-1.sqlplan

Plan after

Sort  actual 119,987, est. 119,987
  Clustered Index Scan PK_Jobs  actual 119,987, est. 119,987
03-parameter-sniffing-after-1.sqlplan

Small customer, large customer ran first 18 rows, same result before and after

Logical reads
Before6,399
After57
Elapsed, median of 7 runs
Before37.7 ms
After1.3 ms
CPU
Before31 ms
After0 ms

Plan before

Sort  actual 18, est. 119,987
  Clustered Index Scan PK_Jobs  actual 18, est. 119,987
03-parameter-sniffing-before-2.sqlplan

Plan after

Sort  actual 18, est. 21
  Nested Loops  actual 18, est. 21
    Index Seek IX_Jobs_CustomerId  actual 18, est. 21
    Key Lookup PK_Jobs  actual 18, est. 1
03-parameter-sniffing-after-2.sqlplan
The SQL, before and after

before.sql

-- Customer job history. Customer 1 owns 20% of all jobs, customer 5000 has a few dozen.
-- Whichever customer runs first after a recompile decides the plan everyone else gets.
CREATE OR ALTER PROCEDURE dbo.GetCustomerJobs
    @CustomerId INT
AS
BEGIN
    SET NOCOUNT ON;
    SELECT JobId, BookedUtc, JobStatus, ServiceLevel, RatedAmount, Notes
    FROM dbo.Jobs
    WHERE CustomerId = @CustomerId
    ORDER BY BookedUtc DESC;
END
GO
ALTER DATABASE SCOPED CONFIGURATION CLEAR PROCEDURE_CACHE;
GO
-- Small customer runs first and caches a seek plus key lookups.
EXEC dbo.GetCustomerJobs @CustomerId = 5000;
GO
-- BENCH:MEASURE large customer, small customer ran first
EXEC dbo.GetCustomerJobs @CustomerId = 1;
-- BENCH:PREP
ALTER DATABASE SCOPED CONFIGURATION CLEAR PROCEDURE_CACHE;
GO
-- Large customer runs first and caches a full scan.
EXEC dbo.GetCustomerJobs @CustomerId = 1;
GO
-- BENCH:MEASURE small customer, large customer ran first
EXEC dbo.GetCustomerJobs @CustomerId = 5000;

after.sql

-- OPTION (RECOMPILE) gives each call a plan for its own parameter value, at the cost of a compile per call.
CREATE OR ALTER PROCEDURE dbo.GetCustomerJobs
    @CustomerId INT
AS
BEGIN
    SET NOCOUNT ON;
    SELECT JobId, BookedUtc, JobStatus, ServiceLevel, RatedAmount, Notes
    FROM dbo.Jobs
    WHERE CustomerId = @CustomerId
    ORDER BY BookedUtc DESC
    OPTION (RECOMPILE);
END
GO
ALTER DATABASE SCOPED CONFIGURATION CLEAR PROCEDURE_CACHE;
GO
EXEC dbo.GetCustomerJobs @CustomerId = 5000;
GO
-- BENCH:MEASURE large customer, small customer ran first
EXEC dbo.GetCustomerJobs @CustomerId = 1;
-- BENCH:PREP
ALTER DATABASE SCOPED CONFIGURATION CLEAR PROCEDURE_CACHE;
GO
EXEC dbo.GetCustomerJobs @CustomerId = 1;
GO
-- BENCH:MEASURE small customer, large customer ran first
EXEC dbo.GetCustomerJobs @CustomerId = 5000;

Implicit conversion, VARCHAR column and NVARCHAR parameter

Problem. ConsignmentNumber is VARCHAR(20) with a unique index. The calling code passes a .NET string through AddWithValue, which sends NVARCHAR. NVARCHAR has higher precedence, so SQL Server converts the column rather than the parameter, and a one-row lookup becomes a scan of the whole index.

Fix. Send the right type. Nothing changes in the database:

// after command.Parameters.Add("@Number", SqlDbType.VarChar, 20).Value = number; ```

Measured. Logical reads went from 2,918 to 6. Elapsed time went from about 66 ms to about 0.3 ms.

Worth noticing. The result depends on collation. This database uses SQL_Latin1_General_CP1_CI_AS, where the conversion forces a scan. With a Windows collation SQL Server can often still seek, so the same code would cost much less, which is why the setup script pins the collation. Plans with this problem carry a PlanAffectingConvert warning, so they can be searched for across the plan cache instead of found one complaint at a time.

Lookup by consignment number 1 rows, same result before and after

Logical reads
Before2,918
After6
Elapsed, median of 7 runs
Before66.2 ms
After0.3 ms
CPU
Before63 ms
After0 ms

Plan before

Nested Loops  actual 1, est. 1
  Index Scan UX_Consignments_Number  actual 1, est. 1
  Key Lookup PK_Consignments  actual 1, est. 1
04-implicit-conversion-before-1.sqlplan

Plan after

Nested Loops  actual 1, est. 1
  Index Seek UX_Consignments_Number  actual 1, est. 1
  Key Lookup PK_Consignments  actual 1, est. 1
04-implicit-conversion-after-1.sqlplan
The SQL, before and after

before.sql

-- ConsignmentNumber is VARCHAR(20). This is what an ADO.NET call sends when it uses
-- AddWithValue with a .NET string: an NVARCHAR parameter. The column gets converted, so no seek.
-- BENCH:MEASURE lookup by consignment number
EXEC sp_executesql
    N'SELECT ConsignmentId, JobId, WeightKg, Items FROM dbo.Consignments WHERE ConsignmentNumber = @Number',
    N'@Number NVARCHAR(20)',
    @Number = N'HF0000450000';

after.sql

-- The parameter type now matches the column. The only change is in the calling code.
-- BENCH:MEASURE lookup by consignment number
EXEC sp_executesql
    N'SELECT ConsignmentId, JobId, WeightKg, Items FROM dbo.Consignments WHERE ConsignmentNumber = @Number',
    N'@Number VARCHAR(20)',
    @Number = 'HF0000450000';

Deadlock between two concurrent batches

Problem. Two batches update the same jobs and consignments at the same time. dbo.RateJobs goes Jobs then Consignments, and dbo.ImportPods goes Consignments then Jobs. Each ends up holding what the other needs, SQL Server kills one as the deadlock victim (error 1205), and that batch's transaction rolls back.

Fix. Three things, in order of importance:

1. Consistent lock order: both procedures now touch Jobs first, then Consignments (after.sql). 2. Short transactions: 200 jobs per commit instead of 2,000, so locks are held for less time and a failure loses less work. 3. A retry on error 1205 with a short backoff, in DeadlockRepro/Program.cs, as a safety net.

Run it.

It runs two concurrent tasks through 20 rounds of 2,000 jobs in each of three modes, then checks the rows themselves: a job counts as processed only if both batches stamped it and all of its consignments.

Measured.

Worth noticing. Every round deadlocks before the fix because the repro holds each transaction open long enough to overlap, using a simulated work delay. Real systems usually deadlock less often, and that makes the problem harder to diagnose, not milder. Retry alone recovers the data but the deadlocks keep happening, so it hides the cause and still pays for the rollbacks. The elapsed times come from this deliberately conflicting workload and should not be read as a general throughput gain.

read-deadlock-graphs.sql pulls the graphs from the built-in system_health session. The XML includes host and login names, so review it before sharing it. The repro changes stamps on jobs 300,001 to 340,000; rerun setup/setup.sh for a pristine database.

20 rounds of 2,000 jobs, two concurrent batches per round, 40,000 jobs in total.

ModeDeadlocks (error 1205)RetriesBatches that gave upJobs fully processedJobs expectedElapsed (s)
before20020040,00016.4
retry-only2020040,00040,00013.4
fixed00040,00040,0006.9
The SQL, before and after

before.sql

-- Two nightly batches touch the same jobs and consignments in opposite table order.
-- RateJobs goes Jobs then Consignments, ImportPods goes Consignments then Jobs.
-- @WorkMs stands in for the application work done while the transaction is open.
CREATE OR ALTER PROCEDURE dbo.RateJobs
    @FromJobId INT, @ToJobId INT, @Stamp DATETIME2(0), @WorkMs INT
AS
BEGIN
    SET NOCOUNT ON;
    SET XACT_ABORT ON;
    DECLARE @Delay CHAR(12) = CONVERT(CHAR(12), DATEADD(MILLISECOND, @WorkMs, CAST('00:00:00' AS TIME(3))), 114);
    BEGIN TRANSACTION;
        UPDATE dbo.Jobs SET RatedUtc = @Stamp WHERE JobId BETWEEN @FromJobId AND @ToJobId;
        WAITFOR DELAY @Delay;
        UPDATE dbo.Consignments SET LastUpdatedUtc = @Stamp WHERE JobId BETWEEN @FromJobId AND @ToJobId;
    COMMIT;
END
GO
CREATE OR ALTER PROCEDURE dbo.ImportPods
    @FromJobId INT, @ToJobId INT, @Stamp DATETIME2(0), @WorkMs INT
AS
BEGIN
    SET NOCOUNT ON;
    SET XACT_ABORT ON;
    DECLARE @Delay CHAR(12) = CONVERT(CHAR(12), DATEADD(MILLISECOND, @WorkMs, CAST('00:00:00' AS TIME(3))), 114);
    BEGIN TRANSACTION;
        UPDATE dbo.Consignments SET PodUtc = @Stamp WHERE JobId BETWEEN @FromJobId AND @ToJobId;
        WAITFOR DELAY @Delay;
        UPDATE dbo.Jobs SET DeliveredUtc = @Stamp WHERE JobId BETWEEN @FromJobId AND @ToJobId;
    COMMIT;
END
GO

after.sql

-- The fix on the database side: both batches now lock Jobs first, then Consignments.
-- RateJobs already used that order and stays as it was. The client side of the fix (short batches
-- and a retry on error 1205) is in DeadlockRepro/Program.cs.
CREATE OR ALTER PROCEDURE dbo.RateJobs
    @FromJobId INT, @ToJobId INT, @Stamp DATETIME2(0), @WorkMs INT
AS
BEGIN
    SET NOCOUNT ON;
    SET XACT_ABORT ON;
    DECLARE @Delay CHAR(12) = CONVERT(CHAR(12), DATEADD(MILLISECOND, @WorkMs, CAST('00:00:00' AS TIME(3))), 114);
    BEGIN TRANSACTION;
        UPDATE dbo.Jobs SET RatedUtc = @Stamp WHERE JobId BETWEEN @FromJobId AND @ToJobId;
        WAITFOR DELAY @Delay;
        UPDATE dbo.Consignments SET LastUpdatedUtc = @Stamp WHERE JobId BETWEEN @FromJobId AND @ToJobId;
    COMMIT;
END
GO
CREATE OR ALTER PROCEDURE dbo.ImportPods
    @FromJobId INT, @ToJobId INT, @Stamp DATETIME2(0), @WorkMs INT
AS
BEGIN
    SET NOCOUNT ON;
    SET XACT_ABORT ON;
    DECLARE @Delay CHAR(12) = CONVERT(CHAR(12), DATEADD(MILLISECOND, @WorkMs, CAST('00:00:00' AS TIME(3))), 114);
    BEGIN TRANSACTION;
        UPDATE dbo.Jobs SET DeliveredUtc = @Stamp WHERE JobId BETWEEN @FromJobId AND @ToJobId;
        WAITFOR DELAY @Delay;
        UPDATE dbo.Consignments SET PodUtc = @Stamp WHERE JobId BETWEEN @FromJobId AND @ToJobId;
    COMMIT;
END
GO