Five common production problems on a fictional courier schema, reproduced, fixed and measured. Measured on Microsoft SQL Server 2019 (RTM-CU27-GDR) (KB5040948) - 15.0.4382.1 (X64), median of 7 runs with a warm cache, on 2026-11-04. Generated from the repo's results files.
| Scenario | Reads before | Reads after | Elapsed before | Elapsed after |
|---|---|---|---|---|
| 01 non sargable date, one-day report | 6,399 | 2,532 | 108.4 ms | 2.1 ms |
| 02 key lookups, zone despatch board | 2,528 | 3 | 1.4 ms | 0.3 ms |
| 03 parameter sniffing, large customer, small customer ran first | 360,172 | 6,399 | 304.0 ms | 185.1 ms |
| 03 parameter sniffing, small customer, large customer ran first | 6,399 | 57 | 37.7 ms | 1.3 ms |
| 04 implicit conversion, lookup by consignment number | 2,918 | 6 | 66.2 ms | 0.3 ms |
Logical reads came out the same on every run, so they are the stable measure. Elapsed times moved by up to about 50% between runs, so treat them as orders of magnitude. This is a developer machine with everything in memory, not a promise about your server.
Problem. A daily report filters with CONVERT(CHAR(10), BookedUtc, 120) = '2025-03-14'. Wrapping the column in a function means SQL Server cannot seek the index on BookedUtc, so it scans all 600,000 jobs and converts each date to text.
Fix. Filter the bare column with a half-open range: BookedUtc >= '2025-03-14' AND BookedUtc < '2025-03-15'. No schema change, same 822 rows.
Measured. Logical reads went from 6,399 to 2,532. Elapsed time went from about 108 ms to about 2 ms.
Worth noticing. Reads fell by only 2.5 times but elapsed time fell by about 50 times. Most of the original cost was CPU spent converting 600,000 dates to strings, which a reads-only review would underrate. The 2,532 reads that remain are key lookups, and scenario 02 shows how to remove that kind of cost when a query runs often enough to justify an index.
Nested Loops actual 822, est. 823 Nested Loops actual 822, est. 823 Merge Interval actual 1, est. 2 Concatenation actual 2, est. 2 Compute Scalar actual 1, est. 1 Constant Scan actual 1, est. 1 Compute Scalar actual 1, est. 1 Constant Scan actual 1, est. 1 Index Seek IX_Jobs_BookedUtc actual 822, est. 823 Key Lookup PK_Jobs actual 822, est. 101-non-sargable-date-after-1.sqlplan
-- Daily report written by converting the column to text. The index on BookedUtc cannot be used for a seek. -- BENCH:MEASURE one-day report SELECT JobId, CustomerId, BookedUtc, JobStatus FROM dbo.Jobs WHERE CONVERT(CHAR(10), BookedUtc, 120) = '2025-03-14';
-- Same report as a half-open date range on the bare column. No schema change needed. -- BENCH:MEASURE one-day report SELECT JobId, CustomerId, BookedUtc, JobStatus FROM dbo.Jobs WHERE BookedUtc >= '2025-03-14' AND BookedUtc < '2025-03-15';
Problem. A despatch board asks for one pickup zone on one day. The only useful index is on BookedUtc, so SQL Server reads the day for all 60 zones and does a key lookup per row (822 lookups) to test the zone and fetch the other columns. Only 21 rows come back.
Fix. A covering index with the equality column first: (PickupZoneId, BookedUtc) INCLUDE (CustomerId, DriverId, JobStatus, ServiceLevel).
Measured. Logical reads went from 2,528 to 3. Elapsed time went from about 1.4 ms to about 0.3 ms.
Worth noticing. On one execution with a warm cache that time saving is tiny, and it would be dishonest to sell it as more. The value is in reads: a board refreshed by many users all day does about 800 times less page work, which leaves more buffer pool for everything else. Every index also slows writes and takes disk space, so a real review weighs this against the write load and looks for existing indexes it can replace.
cleanup.sql drops the index afterwards, because it also happens to cover the query in scenario 01 and would change that scenario's "before" numbers. The benchmark hit exactly that on its second run, which is why the cleanup exists.
Nested Loops actual 21, est. 823 Nested Loops actual 821, est. 823 Merge Interval actual 1, est. 2 Concatenation actual 2, est. 2 Compute Scalar actual 1, est. 1 Constant Scan actual 1, est. 1 Compute Scalar actual 1, est. 1 Constant Scan actual 1, est. 1 Index Seek IX_Jobs_BookedUtc actual 821, est. 823 Key Lookup PK_Jobs actual 21, est. 10602-key-lookups-before-1.sqlplan
Nested Loops actual 21, est. 106 Merge Interval actual 1, est. 2 Concatenation actual 2, est. 2 Compute Scalar actual 1, est. 1 Constant Scan actual 1, est. 1 Compute Scalar actual 1, est. 1 Constant Scan actual 1, est. 1 Index Seek IX_Jobs_PickupZone_BookedUtc actual 21, est. 10602-key-lookups-after-1.sqlplan
-- Despatch board for one pickup zone. Only the index on BookedUtc helps, so the query seeks a one-day -- range for all zones and does a key lookup per row just to test PickupZoneId and fetch the other columns. DROP INDEX IF EXISTS IX_Jobs_PickupZone_BookedUtc ON dbo.Jobs; GO -- BENCH:MEASURE zone despatch board SELECT JobId, CustomerId, DriverId, JobStatus, ServiceLevel, BookedUtc FROM dbo.Jobs WHERE PickupZoneId = 17 AND BookedUtc >= '2025-10-06' AND BookedUtc < '2025-10-07';
-- Covering index: equality column first, then the range column, with the selected columns included.
DROP INDEX IF EXISTS IX_Jobs_PickupZone_BookedUtc ON dbo.Jobs;
GO
CREATE INDEX IX_Jobs_PickupZone_BookedUtc
ON dbo.Jobs (PickupZoneId, BookedUtc)
INCLUDE (CustomerId, DriverId, JobStatus, ServiceLevel);
GO
-- BENCH:MEASURE zone despatch board
SELECT JobId, CustomerId, DriverId, JobStatus, ServiceLevel, BookedUtc
FROM dbo.Jobs
WHERE PickupZoneId = 17
AND BookedUtc >= '2025-10-06' AND BookedUtc < '2025-10-07';Problem. dbo.GetCustomerJobs returns a customer's job history. Customer 1 has 119,987 jobs and customer 5000 has 18. SQL Server builds the cached plan for whichever value it sees first, so the same procedure is fast or slow depending on who ran first after the last recompile. That is why this kind of issue often looks random and "fixes itself" after a restart.
Fix. OPTION (RECOMPILE) on the statement, so each call gets a plan for its own value.
Measured.
Worth noticing. Both directions are bad. The large customer suffers from 120,000 key lookups, and the small customer pays for a full scan to get 18 rows. OPTION (RECOMPILE) is not free, since every call compiles, and that is acceptable here but not for a procedure called thousands of times a minute. In that case I would look at a covering index, OPTIMIZE FOR, or separate code paths for the few very large customers. Which one is right depends on call frequency, so the choice is made per procedure.
Sort actual 119,987, est. 21 Nested Loops actual 119,987, est. 21 Index Seek IX_Jobs_CustomerId actual 119,987, est. 21 Key Lookup PK_Jobs actual 119,987, est. 103-parameter-sniffing-before-1.sqlplan
Sort actual 119,987, est. 119,987 Clustered Index Scan PK_Jobs actual 119,987, est. 119,98703-parameter-sniffing-after-1.sqlplan
Sort actual 18, est. 119,987 Clustered Index Scan PK_Jobs actual 18, est. 119,98703-parameter-sniffing-before-2.sqlplan
Sort actual 18, est. 21 Nested Loops actual 18, est. 21 Index Seek IX_Jobs_CustomerId actual 18, est. 21 Key Lookup PK_Jobs actual 18, est. 103-parameter-sniffing-after-2.sqlplan
-- Customer job history. Customer 1 owns 20% of all jobs, customer 5000 has a few dozen.
-- Whichever customer runs first after a recompile decides the plan everyone else gets.
CREATE OR ALTER PROCEDURE dbo.GetCustomerJobs
@CustomerId INT
AS
BEGIN
SET NOCOUNT ON;
SELECT JobId, BookedUtc, JobStatus, ServiceLevel, RatedAmount, Notes
FROM dbo.Jobs
WHERE CustomerId = @CustomerId
ORDER BY BookedUtc DESC;
END
GO
ALTER DATABASE SCOPED CONFIGURATION CLEAR PROCEDURE_CACHE;
GO
-- Small customer runs first and caches a seek plus key lookups.
EXEC dbo.GetCustomerJobs @CustomerId = 5000;
GO
-- BENCH:MEASURE large customer, small customer ran first
EXEC dbo.GetCustomerJobs @CustomerId = 1;
-- BENCH:PREP
ALTER DATABASE SCOPED CONFIGURATION CLEAR PROCEDURE_CACHE;
GO
-- Large customer runs first and caches a full scan.
EXEC dbo.GetCustomerJobs @CustomerId = 1;
GO
-- BENCH:MEASURE small customer, large customer ran first
EXEC dbo.GetCustomerJobs @CustomerId = 5000;-- OPTION (RECOMPILE) gives each call a plan for its own parameter value, at the cost of a compile per call.
CREATE OR ALTER PROCEDURE dbo.GetCustomerJobs
@CustomerId INT
AS
BEGIN
SET NOCOUNT ON;
SELECT JobId, BookedUtc, JobStatus, ServiceLevel, RatedAmount, Notes
FROM dbo.Jobs
WHERE CustomerId = @CustomerId
ORDER BY BookedUtc DESC
OPTION (RECOMPILE);
END
GO
ALTER DATABASE SCOPED CONFIGURATION CLEAR PROCEDURE_CACHE;
GO
EXEC dbo.GetCustomerJobs @CustomerId = 5000;
GO
-- BENCH:MEASURE large customer, small customer ran first
EXEC dbo.GetCustomerJobs @CustomerId = 1;
-- BENCH:PREP
ALTER DATABASE SCOPED CONFIGURATION CLEAR PROCEDURE_CACHE;
GO
EXEC dbo.GetCustomerJobs @CustomerId = 1;
GO
-- BENCH:MEASURE small customer, large customer ran first
EXEC dbo.GetCustomerJobs @CustomerId = 5000;Problem. ConsignmentNumber is VARCHAR(20) with a unique index. The calling code passes a .NET string through AddWithValue, which sends NVARCHAR. NVARCHAR has higher precedence, so SQL Server converts the column rather than the parameter, and a one-row lookup becomes a scan of the whole index.
Fix. Send the right type. Nothing changes in the database:
// after command.Parameters.Add("@Number", SqlDbType.VarChar, 20).Value = number; ```
Measured. Logical reads went from 2,918 to 6. Elapsed time went from about 66 ms to about 0.3 ms.
Worth noticing. The result depends on collation. This database uses SQL_Latin1_General_CP1_CI_AS, where the conversion forces a scan. With a Windows collation SQL Server can often still seek, so the same code would cost much less, which is why the setup script pins the collation. Plans with this problem carry a PlanAffectingConvert warning, so they can be searched for across the plan cache instead of found one complaint at a time.
Nested Loops actual 1, est. 1 Index Scan UX_Consignments_Number actual 1, est. 1 Key Lookup PK_Consignments actual 1, est. 104-implicit-conversion-before-1.sqlplan
Nested Loops actual 1, est. 1 Index Seek UX_Consignments_Number actual 1, est. 1 Key Lookup PK_Consignments actual 1, est. 104-implicit-conversion-after-1.sqlplan
-- ConsignmentNumber is VARCHAR(20). This is what an ADO.NET call sends when it uses
-- AddWithValue with a .NET string: an NVARCHAR parameter. The column gets converted, so no seek.
-- BENCH:MEASURE lookup by consignment number
EXEC sp_executesql
N'SELECT ConsignmentId, JobId, WeightKg, Items FROM dbo.Consignments WHERE ConsignmentNumber = @Number',
N'@Number NVARCHAR(20)',
@Number = N'HF0000450000';-- The parameter type now matches the column. The only change is in the calling code.
-- BENCH:MEASURE lookup by consignment number
EXEC sp_executesql
N'SELECT ConsignmentId, JobId, WeightKg, Items FROM dbo.Consignments WHERE ConsignmentNumber = @Number',
N'@Number VARCHAR(20)',
@Number = 'HF0000450000';Problem. Two batches update the same jobs and consignments at the same time. dbo.RateJobs goes Jobs then Consignments, and dbo.ImportPods goes Consignments then Jobs. Each ends up holding what the other needs, SQL Server kills one as the deadlock victim (error 1205), and that batch's transaction rolls back.
Fix. Three things, in order of importance:
1. Consistent lock order: both procedures now touch Jobs first, then Consignments (after.sql). 2. Short transactions: 200 jobs per commit instead of 2,000, so locks are held for less time and a failure loses less work. 3. A retry on error 1205 with a short backoff, in DeadlockRepro/Program.cs, as a safety net.
Run it.
It runs two concurrent tasks through 20 rounds of 2,000 jobs in each of three modes, then checks the rows themselves: a job counts as processed only if both batches stamped it and all of its consignments.
Measured.
Worth noticing. Every round deadlocks before the fix because the repro holds each transaction open long enough to overlap, using a simulated work delay. Real systems usually deadlock less often, and that makes the problem harder to diagnose, not milder. Retry alone recovers the data but the deadlocks keep happening, so it hides the cause and still pays for the rollbacks. The elapsed times come from this deliberately conflicting workload and should not be read as a general throughput gain.
read-deadlock-graphs.sql pulls the graphs from the built-in system_health session. The XML includes host and login names, so review it before sharing it. The repro changes stamps on jobs 300,001 to 340,000; rerun setup/setup.sh for a pristine database.
| Mode | Deadlocks (error 1205) | Retries | Batches that gave up | Jobs fully processed | Jobs expected | Elapsed (s) |
|---|---|---|---|---|---|---|
| before | 20 | 0 | 20 | 0 | 40,000 | 16.4 |
| retry-only | 20 | 20 | 0 | 40,000 | 40,000 | 13.4 |
| fixed | 0 | 0 | 0 | 40,000 | 40,000 | 6.9 |
-- Two nightly batches touch the same jobs and consignments in opposite table order.
-- RateJobs goes Jobs then Consignments, ImportPods goes Consignments then Jobs.
-- @WorkMs stands in for the application work done while the transaction is open.
CREATE OR ALTER PROCEDURE dbo.RateJobs
@FromJobId INT, @ToJobId INT, @Stamp DATETIME2(0), @WorkMs INT
AS
BEGIN
SET NOCOUNT ON;
SET XACT_ABORT ON;
DECLARE @Delay CHAR(12) = CONVERT(CHAR(12), DATEADD(MILLISECOND, @WorkMs, CAST('00:00:00' AS TIME(3))), 114);
BEGIN TRANSACTION;
UPDATE dbo.Jobs SET RatedUtc = @Stamp WHERE JobId BETWEEN @FromJobId AND @ToJobId;
WAITFOR DELAY @Delay;
UPDATE dbo.Consignments SET LastUpdatedUtc = @Stamp WHERE JobId BETWEEN @FromJobId AND @ToJobId;
COMMIT;
END
GO
CREATE OR ALTER PROCEDURE dbo.ImportPods
@FromJobId INT, @ToJobId INT, @Stamp DATETIME2(0), @WorkMs INT
AS
BEGIN
SET NOCOUNT ON;
SET XACT_ABORT ON;
DECLARE @Delay CHAR(12) = CONVERT(CHAR(12), DATEADD(MILLISECOND, @WorkMs, CAST('00:00:00' AS TIME(3))), 114);
BEGIN TRANSACTION;
UPDATE dbo.Consignments SET PodUtc = @Stamp WHERE JobId BETWEEN @FromJobId AND @ToJobId;
WAITFOR DELAY @Delay;
UPDATE dbo.Jobs SET DeliveredUtc = @Stamp WHERE JobId BETWEEN @FromJobId AND @ToJobId;
COMMIT;
END
GO-- The fix on the database side: both batches now lock Jobs first, then Consignments.
-- RateJobs already used that order and stays as it was. The client side of the fix (short batches
-- and a retry on error 1205) is in DeadlockRepro/Program.cs.
CREATE OR ALTER PROCEDURE dbo.RateJobs
@FromJobId INT, @ToJobId INT, @Stamp DATETIME2(0), @WorkMs INT
AS
BEGIN
SET NOCOUNT ON;
SET XACT_ABORT ON;
DECLARE @Delay CHAR(12) = CONVERT(CHAR(12), DATEADD(MILLISECOND, @WorkMs, CAST('00:00:00' AS TIME(3))), 114);
BEGIN TRANSACTION;
UPDATE dbo.Jobs SET RatedUtc = @Stamp WHERE JobId BETWEEN @FromJobId AND @ToJobId;
WAITFOR DELAY @Delay;
UPDATE dbo.Consignments SET LastUpdatedUtc = @Stamp WHERE JobId BETWEEN @FromJobId AND @ToJobId;
COMMIT;
END
GO
CREATE OR ALTER PROCEDURE dbo.ImportPods
@FromJobId INT, @ToJobId INT, @Stamp DATETIME2(0), @WorkMs INT
AS
BEGIN
SET NOCOUNT ON;
SET XACT_ABORT ON;
DECLARE @Delay CHAR(12) = CONVERT(CHAR(12), DATEADD(MILLISECOND, @WorkMs, CAST('00:00:00' AS TIME(3))), 114);
BEGIN TRANSACTION;
UPDATE dbo.Jobs SET DeliveredUtc = @Stamp WHERE JobId BETWEEN @FromJobId AND @ToJobId;
WAITFOR DELAY @Delay;
UPDATE dbo.Consignments SET PodUtc = @Stamp WHERE JobId BETWEEN @FromJobId AND @ToJobId;
COMMIT;
END
GO