en-US/about_Mgx_Tuning.help.txt
|
TOPIC
about_Mgx_Tuning SHORT DESCRIPTION Default settings, hard ceilings, and tuning guidance for large tenants. LONG DESCRIPTION Mgx ships with conservative defaults to prevent accidental self-DoS against your tenant. Every tunable has a hard ceiling enforced at the property level -- invalid values throw rather than silently apply. Use Set-MgxOption to change settings at runtime. Only explicitly passed parameters are modified; all others retain their current values. Use Set-MgxOption -Reset to restore all defaults. CLIENT-SIDE RATE LIMITING (ON BY DEFAULT) Setting Default Hard Max Purpose ------- ------- -------- ------- RateLimitBurst 200 10,000 Token bucket burst capacity RateLimitPerSecond 50 10,000 Sustained request rate RateLimitQueueLimit 500 100,000 Queued requests before rejection Disable explicitly with Set-MgxOption -NoRateLimit. ADAPTIVE PACING (ON BY DEFAULT) A proactive gate ahead of the token bucket. The bucket enforces a hard request-rate ceiling; the pacer slows down BEFORE Graph throttles, based on what the tenant is signaling. Independent of -NoRateLimit; disable explicitly with Set-MgxOption -NoAdaptivePacing. State is kept per workload (drive/SharePoint, directory, other), so a throttle on one service never slows an unrelated workload in the same session. Three mechanisms, all inactive (zero delay) in steady state: - AIMD back-off. A 429 caps the workload's request rate (4 req/sec on the first throttle, halving on repeat, floor 2). The cap climbs back a tenth of the ceiling per clean second and expires outright after five quiet minutes. A server Retry-After additionally holds ALL requests for that workload, not just the throttled one. - Slow start. A workload that is cold (first use, or idle five minutes) opens with a 4 req/sec rate cap that doubles each clean second until it reaches the ceiling. The first request is never delayed; the cap only bites when demand exceeds it, i.e. fan-outs -- the pattern that otherwise throttles before completing a single item. - Throttle-proximity damping. When Graph reports x-ms-throttle-limit-percentage (documented to appear above 80% of budget), requests are spaced with a delay that ramps from 0ms at 0.8 to 2000ms at 1.2, while the report stays fresh (30s). Batch outer /$batch POSTs are exempt from the gate -- batch throughput is owned by BatchItemsPerSecond and its item-level pacing above -- but their responses still feed the pacer's signals. Watch it work: Get-MgxTelemetry shows total pacing wait, activation count, the last reported throttle percentage, and a per-workload state line (active caps, slow start, latency vs. baseline). Run any cmdlet with -Verbose to see individual pacing waits. RETRY BUDGET CAPS Setting Default Hard Max Purpose ------- ------- -------- ------- MaxRetryAttempts 7 50 Retries per request MaxRetryAfterSeconds 120s 600s Caps server Retry-After delays TotalTimeoutSeconds 300s 3,600s Total time across all retries AttemptTimeoutSeconds 30s 300s Per-request timeout CIRCUIT BREAKER Setting Default Hard Max Purpose ------- ------- -------- ------- CircuitBreakerFailureRatio 10% 100% Failure ratio to trip CircuitBreakerMinThroughput 40 1,000 Min requests before evaluating CircuitBreakerSamplingDurationSeconds 30s 300s Evaluation window CircuitBreakerDurationSeconds 15s 300s Open-state duration Trips on 500/502/503/504 (server errors). Does NOT trip on 429 -- that is expected throttling, not failure. BATCH CONSTRAINTS Setting Default Hard Max Purpose ------- ------- -------- ------- BatchChunkConcurrency 1 10 Concurrent batch chunks BatchItemsPerSecond 20 1,000 Inter-chunk pacing (items/sec) Batch size 20 20 Items per /$batch POST (Graph limit) Per-item retries 3 3 Retries for failed items in a batch Sequential chunk execution (default) paces at 20 items/sec to stay under Graph's write throttle ceiling. Pacing adapts to what the tenant actually tolerates. A 429 halves the write rate, down to a floor of 2 items/sec, and the reduced rate carries across Invoke-MgxBatchRequest calls so a loop of small batches does not immediately get throttled again. It also climbs back: every two consecutive chunks without a 429 add a tenth of the configured rate, and once no throttling has been seen for five minutes the configured rate is restored outright. Run Invoke-MgxBatchRequest with -Verbose to see each adjustment. WRITE COST: PACE BY WRITES, NOT ITEMS BatchItemsPerSecond paces batch items, but Graph throttles directory writes. Those are the same thing only when each item performs one write. A compound item multiplies the true rate: Item shape Writes per item ---------- --------------- POST /users (plain create) 1 PATCH /users/{id} 1 POST /groups with 20 members@odata.bind ~21 At the default 20 items/sec, plain creates put ~20 writes/sec on the tenant; group creates with 20 bound members put ~420 writes/sec on it. The trap: heavy-write batches look fine at first. Graph's write throttle is a sliding budget, not an instant rate check -- a burst allowance absorbs the early chunks, then sustained throttle waves begin. Observed on a live ~100k-object tenant: 57,000 plain user creates ran at 20 items/sec for 48 minutes with zero visible throttling, while group creates with 20-member binds at the same item rate ran clean for several thousand items and then drew 429 waves with multi-minute Retry-After backoff. Adaptive pacing recovers on its own (rates halve on 429, climb back on clean chunks, restore fully after five clean minutes) and no items are lost -- but recovery time is wall-clock time. For compound items, set the rate for the write cost up front: # Starting point: divide the default budget by writes-per-item # e.g. ~21-write group creates: Set-MgxOption -BatchItemsPerSecond 2 Then let adaptive pacing find the tenant's real ceiling from below, instead of starting above it and paying for the discovery in Retry-After waits. Run with -Verbose to watch the rate adjustments. POST SAFETY Non-idempotent methods (POST) only retry on 429 -- not on 5xx. A 500 on POST may mean the request was partially processed; blindly retrying could create duplicates. LARGE TENANT CONFIGURATION Default settings are tuned for tenants up to ~10k objects. Larger tenants should adjust timeouts, rate limits, and batch pacing. Observed throughput (v1.0.1, live tenant): Operation Objects Throughput Notes --------- ------- ---------- ----- User creation (batch POST) 25,000 20 items/sec Zero 429s with default pacing User enumerate (-All) 47,000 2,653/sec Streaming pagination Batch GET Mixed 107/sec Default settings 50k+ tenants (read-heavy): Set-MgxOption -TotalTimeoutSeconds 600 ` -AttemptTimeoutSeconds 60 ` -RateLimitBurst 300 ` -RateLimitQueueLimit 2000 TotalTimeoutSeconds 300 -> 600 Full enumeration takes > 5 min AttemptTimeoutSeconds 30 -> 60 Large pages can exceed 30s RateLimitBurst 200 -> 300 Absorbs pagination spikes RateLimitQueueLimit 500 -> 2,000 Fan-out queues more requests 100k+ write operations: Set-MgxOption -BatchItemsPerSecond 15 ` -TotalTimeoutSeconds 3600 ` -AttemptTimeoutSeconds 180 BatchItemsPerSecond 20 -> 15 Stay under write throttle TotalTimeoutSeconds 300 -> 3,600 100k writes takes ~110 min AttemptTimeoutSeconds 30 -> 180 Batch POST slows under load SEE ALSO Set-MgxOption Get-MgxOption Get-MgxTelemetry about_Mgx |