Redshift Sort Keys, WLM, and Spectrum: RA3 Tuning and the Concurrency Scaling Bill

The BI team turned on concurrency scaling because Monday’s stand-up was “the cluster is busy.” Four weeks later the Redshift bill had an $8,400 line that was not compute of the RA3 nodes we sized. Tableau had issued a few hundred concurrent SELECT * against a Spectrum table over a year of Parquet. Extra clusters spun, scanned S3, and spun down. The dashboard still took 40 seconds because nothing was sorted and nothing was partitioned.

I turned concurrency scaling off for the ETL queue, put the BI workload on its own WLM queue with a cap, and rewrote the Spectrum view to name columns and require dt. The $8,400 dropped. The 40 seconds dropped only after we fixed the sort key on the internal table the dashboard should have been using.

RA3 did not make this a Snowflake problem. The broader choice is in Redshift vs Snowflake. This post is the knobs that still exist after you pick Redshift.

flowchart TD
  q[Query] --> wlm[WLM queue]
  wlm --> local{Local table?}
  local -->|yes| sort[Sort key + dist key]
  local -->|Spectrum| spec[S3 scan]
  spec --> star{SELECT * ?}
  star -->|yes| bill[Bytes scanned]
  wlm --> burst{Queue saturated?}
  burst -->|concurrency scaling on| extra[Billed extra clusters]
  burst -->|off / cap| wait[Wait or timeout]

WLM decides who runs. Sort keys decide how much of a local table you read. Spectrum decides how much S3 you pay for. Concurrency scaling is a separate invoice.

RA3 still has a physics

Managed storage means you do not size DS2 disks. It does not mean every query is a metadata prune. Data is still distributed on a key (or AUTO, or EVEN) and still stored in sort order if you loaded it that way. A missing sort key on a 4 billion row fact is a full slice scan with nicer hardware.

create table analytics.fct_orders (
  order_id      varchar(32)  not null,
  customer_id   varchar(32)  not null,
  order_ts      timestamp    not null,
  amount_usd    numeric(12,2) not null,
  status        varchar(16)  not null
)
diststyle key
distkey (customer_id)
compound sortkey (order_ts, status);

-- Load in sort order when you can. COPY + unsorted region
-- is why VACUUM exists.
copy analytics.fct_orders
from 's3://acme-lake/out/fct_orders/dt=2026-08-21/'
iam_role 'arn:aws:iam::123456789012:role/RedshiftCopy'
format as parquet;

vacuum sort only analytics.fct_orders;
analyze analytics.fct_orders;

DISTKEY (customer_id) is for the join you run every hour against dim_customer. If the common join is something else, you picked a souvenir. EVEN is honest when there is no join key. ALL on a large fact is how you fill a node. AUTO is fine until you measure; then pin what you measured.

COMPOUND sort keys prefix-match. Filter on order_ts and you win. Filter only on status and the second column does not save you. Interleaved sort keys exist and I no longer use them: vacuum cost and planner surprises outran the benefit on every cluster I have owned.

VACUUM and ANALYZE are still jobs

RA3 reclaims deletes better than dense-compute nodes did. It does not keep sort order for you after a week of COPY into the unsorted region. Dashboards that were 2 seconds become 40 when 60% of the table sits unsorted.

VACUUM SORT ONLY on the hot facts after the load. ANALYZE so the planner sees the new stats. I run both in the ETL queue, not as a polite suggestion in a wiki. Skip them and you will “tune WLM” for a statistics problem.

Spectrum SELECT *

Spectrum is Redshift’s external table over S3. You pay for bytes scanned, with partition pruning only if the predicate is on a partition column the DDL declared. SELECT * from a 200-column wide Parquet landing table is an S3 scan dressed as SQL.

create external table spectrum.orders_raw (
  order_id     varchar(32),
  customer_id  varchar(32),
  amount_usd   double precision,
  order_ts     timestamp,
  status       varchar(16)
)
partitioned by (dt date)
stored as parquet
location 's3://acme-lake/landing/orders/';

-- Partition + columns. Anything else is a bill.
select order_id, amount_usd, status
from spectrum.orders_raw
where dt = date '2026-08-21'
  and status = 'paid';

Land with Spectrum. Serve from a local table that has a sort key. I do not point Tableau at spectrum.*. That is how concurrency scaling and Spectrum meet on the same invoice.

Concurrency scaling is a surprise because it works

When a queue is saturated, Redshift can run extra clusters and charge you for them by the second. BI bursts look great. A runaway dashboard refresh looks like a successful scale-out.

I disable concurrency scaling on ETL queues. ETL should wait or fail, not clone the cluster because a merge queued behind itself. For BI I set a WLM concurrency and a query timeout, and I alert on concurrency_scaling usage in the bill and in SVL_CONCURRENCY_SCALING_USAGE. “Make it faster” is a sort key. Extra clusters are a credit card.

Failure modes

AUTO everywhere, never measured. Fine at 100 GB. At 10 TB you own a mystery.

Sort key on a column nobody filters. Decorative.

VACUUM deferred for a month. Unsorted region is the table.

Spectrum as the semantic layer. Bytes × users.

Concurrency scaling as the default. ETL clones itself.

WLM queues I actually run

Three queues, not one default. ETL (concurrency 3–5, concurrency scaling off, timeout long enough for the merge). BI (concurrency 10–15, short timeout, scaling capped or off until you have a sort key). Ad-hoc (lowest memory, shortest timeout, so a notebook cannot evict the close).

-- Who is scanning what. Run after a slow tile.
SELECT
  q.query,
  TRIM(q.label) AS label,
  s.tbl,
  s.rows,
  s.bytes,
  s.is_rrscan
FROM stl_scan s
JOIN stl_query q ON q.query = s.query
WHERE q.starttime >= DATEADD(hour, -2, GETDATE())
  AND s.userid > 1
ORDER BY s.bytes DESC
LIMIT 20;

If bytes is the whole fact and is_rrscan is true, the sort key did not match the filter. If the table is external, you are in Spectrum — name columns and require dt. Hub: Redshift. The warehouse pick is still Redshift vs Snowflake.

FAQ

Does RA3 remove sort keys? No. Managed storage removes disk sizing. COMPOUND SORTKEY and DISTKEY still decide the scan.

When do I VACUUM? After large COPY into the unsorted region. VACUUM SORT ONLY on hot facts in the ETL queue, then ANALYZE. Skipping them is a 40-second dashboard.

Can Tableau read Spectrum? It can. It should not. Land with Spectrum, serve from a local sorted table. Spectrum SELECT * is an S3 bill.

Should concurrency scaling stay on? Off for ETL. Capped and alarmed for BI. Extra clusters are a credit card, not a sort key.

COMPOUND or interleaved? COMPOUND, prefix-matched to the WHERE you run. Interleaved vacuum cost outran the benefit on every cluster I have owned.

What to do Monday

For each slow dashboard, read STL_SCAN and SVL_QUERY_SUMMARY. If you scanned the fact, fix SORTKEY and VACUUM. If the scan is Spectrum, name columns and require a partition predicate. Turn concurrency scaling off on ETL. Cap it on BI and put a budget alarm on the usage view. RA3 bought you managed storage. It did not buy you a warehouse that sorts itself.