{"id":3075,"date":"2026-11-19T15:16:23","date_gmt":"2026-11-19T15:16:23","guid":{"rendered":"https:\/\/hostry.com\/blog\/?p=3075"},"modified":"2026-11-19T15:16:24","modified_gmt":"2026-11-19T15:16:24","slug":"ai-driven-attacks-infrastructure-defense","status":"publish","type":"post","link":"https:\/\/hostry.com\/blog\/ai-driven-attacks-infrastructure-defense\/","title":{"rendered":"How to Defend Against AI-Powered Cyber Attacks on Your Linux Servers"},"content":{"rendered":"\n<figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"1200\" height=\"675\" src=\"https:\/\/hostry.com\/blog\/wp-content\/uploads\/2026\/11\/ai-driven-attacks-infrastructure-defense-cover.webp\" alt=\"How to defend against AI-powered cyber attacks on Linux servers: a lighthouse beam over a server island catches the one boat that approaches slowly\" class=\"wp-image-3072\" srcset=\"https:\/\/hostry.com\/blog\/wp-content\/uploads\/2026\/11\/ai-driven-attacks-infrastructure-defense-cover.webp 1200w, https:\/\/hostry.com\/blog\/wp-content\/uploads\/2026\/11\/ai-driven-attacks-infrastructure-defense-cover-300x169.webp 300w, https:\/\/hostry.com\/blog\/wp-content\/uploads\/2026\/11\/ai-driven-attacks-infrastructure-defense-cover-1024x576.webp 1024w, https:\/\/hostry.com\/blog\/wp-content\/uploads\/2026\/11\/ai-driven-attacks-infrastructure-defense-cover-450x253.webp 450w, https:\/\/hostry.com\/blog\/wp-content\/uploads\/2026\/11\/ai-driven-attacks-infrastructure-defense-cover-768x432.webp 768w\" sizes=\"auto, (max-width: 1200px) 100vw, 1200px\" \/><\/figure>\n\n\n\n<pre class=\"wp-block-preformatted\" style=\"font-size:14px\"><em><strong>In a five-host lab, 200 probing requests at 177 per second were flagged in 0.18 s. The same 200 requests sent one every 4 seconds triggered no threshold rule. Only a decoy URL caught them, on request 33. The probing was scripted (a fixed path list, one nmap -sV scan, ten wrong logins), and no attack was reproduced. To defend against AI-powered cyber attacks once someone holds a key or a foothold, run decoys, a persistence audit and gated SSH key rotation as a routine that works without you.<\/strong><\/em><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">If you look after a handful to a few dozen Linux servers and have no security team, this is how to defend against AI-powered cyber attacks, in this order.<\/p>\n\n\n\n<ol class=\"wp-block-list\"><li><strong><a href=\"#recon-in-logs\">Keep threshold detection for bursts<\/a><\/strong> and set sshd to <code>LogLevel VERBOSE<\/code>; thresholds miss slow probing.<\/li><li><strong><a href=\"#decoys\">Plant three decoys<\/a><\/strong>: a credentials file under auditd, a logging-only SSH key, an unlinked URL.<\/li><li><strong><a href=\"#persistence-audit\">Inventory keys and persistence points<\/a><\/strong> on every host and keep that baseline on a control node.<\/li><li><strong><a href=\"#rotation\">Automate SSH key rotation with a safe order<\/a><\/strong> that proves the new key before removing the old.<\/li><li><strong><a href=\"#schedule\">Schedule the rotation<\/a><\/strong> behind an all-or-nothing gate.<\/li><li><strong><a href=\"#decoy-to-rotation\">Wire decoy alerts to the audit and then to rotation<\/a><\/strong>, paging a person when the audit finds something or a host is down.<\/li><li><strong><a href=\"#standards\">Map the routine to the audit frame<\/a><\/strong> (CASA, SOC 2, NIST) by what it actually requires.<\/li><\/ol>\n\n\n\n<p class=\"wp-block-paragraph\">This routine handles what happens after someone holds a key or a foothold: stolen keys in use, planted persistence, quiet probing. It does not replace patching, MFA on panels, or key-only SSH behind a host firewall, as set up in <strong><a href=\"https:\/\/hostry.com\/blog\/secure-new-linux-vps-first-30-minutes\/\">the hardening pass for the first 30 minutes on a new Linux server<\/a><\/strong>. For exposed panels, see <strong><a href=\"https:\/\/hostry.com\/blog\/ai-agent-exposed-to-internet-vps\/\">checking what an AI agent on a VPS exposes to the internet<\/a><\/strong>.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">The measurements below were reproduced with Ansible core 2.16.3 (ansible.posix 1.5.4), OpenSSH 9.6p1 and nginx 1.24.0 on Ubuntu 24.04, five hosts simulated as network namespaces in a container; the schedule was run with Semaphore UI 2.17.15 in the same lab. The hosts share one kernel and near-zero latency, so every time is a lab number. The files shown are the lab&#8217;s own; replace its addresses (10.77.0.x), the def-svc account and <code>\/opt\/def-lab<\/code> paths with yours.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"by-the-record\">AI-driven cyber attacks on infrastructure: what the record shows<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">The documented cases show AI making reconnaissance, trying one approach after another and harvesting credentials much faster and cheaper. People still made the key decisions, the techniques were already known, and the models made mistakes. Every AI agent cyberattack story belongs to one of three groups: confirmed incidents, demonstrations, and estimates or vendor claims.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Confirmed incidents, as reported by the company that investigated them<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">On November 13, 2025, the <strong><a href=\"https:\/\/www.anthropic.com\/news\/disrupting-AI-espionage\" target=\"_blank\" rel=\"noopener\">Anthropic report on an AI-orchestrated espionage campaign<\/a><\/strong> described a group it attributes to a Chinese state sponsor using Claude Code against about thirty organizations, a few successfully. Anthropic estimates the AI did 80-90% of the work, with people at about four to six decision points, and peaked at thousands of requests, often several per second. Claude also &#8220;occasionally hallucinated credentials&#8221;, which Anthropic calls an obstacle to attacks without people.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Anthropic&#8217;s <strong><a href=\"https:\/\/anthropic.com\/threat-intelligence-report-september-2026\" target=\"_blank\" rel=\"noopener\">September 2026 threat intelligence report<\/a><\/strong> covers December 2025 to August 2026. In its GTG-50014 section, operators suspected to be ShinyHunters affiliates dumped more than 2,100 Azure AD token sets from over 40 tenants in about 34 hours. The report says &#8220;humans have retained the decisions that matter most to them&#8221;, and that no operation depended on an entirely new technique, with one exception (an undocumented WordPress re-installation race condition). It advises treating AI keys and agent integrations as production credentials.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">On November 5, 2025, the <strong><a href=\"https:\/\/cloud.google.com\/blog\/topics\/threat-intelligence\/threat-actor-usage-of-ai-tools\" target=\"_blank\" rel=\"noopener\">GTIG AI Threat Tracker<\/a><\/strong> described PROMPTSTEAL, attributed to APT28, which asks a hosted model for one-line Windows commands, as Google&#8217;s &#8220;first observation of malware querying an LLM deployed in live operations&#8221;.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Demonstrations and benchmarks<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">The same GTIG report lists PROMPTFLUX, which rewrites its own code through a model, as experimental and unable to compromise a victim network. ZeroDayBench (arXiv, March 2, 2026) gave GPT-5.2, Claude Sonnet 4.5 and Grok 4.1 agents 22 new critical vulnerabilities to find and patch; its authors conclude the models are &#8220;not yet capable of autonomously solving our tasks&#8221;. Only its abstract was read.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Estimates and vendor claims<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">Rapid7&#8217;s Q3 2026 update (October 21, 2026) found that 8% of the threat reporting it tracked cited AI as an operational component, a share of reports and not of attacks, and that &#8220;human operators continued to select targets and determine objectives&#8221;. The UK&#8217;s NCSC (May 7, 2025) judged &#8220;fully automated, end-to-end advanced cyber attacks&#8221; unlikely before 2027.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">For a server owner, the steps that got cheaper are the ones that reach your machines first: scanning, trying paths and logins, and reusing stolen keys at a pace no person types.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"exploit-window\">Time to exploit a vulnerability in 2026: days, sometimes less<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Three 2026 reports put the time from disclosure to exploitation at days or below zero, each with its own definition, so they cannot be merged into one figure.<\/p>\n\n\n\n<ul class=\"wp-block-list\"><li><strong><a href=\"https:\/\/cloud.google.com\/blog\/topics\/threat-intelligence\/m-trends-2026\" target=\"_blank\" rel=\"noopener\">Mandiant&#8217;s M-Trends 2026<\/a><\/strong> (March 23, 2026) gives a mean time to exploit of &#8220;an estimated -7 days&#8221;, which means exploitation on average began before a patch existed. The sample is Mandiant&#8217;s own engagements, where exploits were the initial vector in 32% of intrusions.<\/li><li><strong><a href=\"https:\/\/www.rapid7.com\/blog\/post\/tr-q3-2026-threat-update-attackers-exploiting-vulnerabilities-within-3-days\/\" target=\"_blank\" rel=\"noopener\">Rapid7&#8217;s Q3 2026 threat update<\/a><\/strong> reports three days on average between a high-severity vulnerability becoming public and its exploitation, with 71% of the exploitation it tracked happening within the first seven days. The base is 43 newly exploited high- and critical-severity vulnerabilities.<\/li><li><strong><a href=\"https:\/\/www.vulncheck.com\/blog\/state-of-exploitation-2026\" target=\"_blank\" rel=\"noopener\">VulnCheck&#8217;s State of Exploitation 2026<\/a><\/strong> (January 21, 2026) found 28.96% of 2025&#8217;s known exploited vulnerabilities were exploited on or before the day their CVE was published, up from 23.6% a year earlier. VulnCheck notes that same-day exploitation is not always a zero-day.<\/li><\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">The NCSC says the window has shrunk to days. Mandiant also reports that in 2025 the median time from initial access to hand-off to a secondary group fell to 22 seconds, from over 8 hours in 2022, and that it does not see 2025 as the year breaches were the direct result of AI.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">When exploitation can arrive before the patch, what you control on the day is what an intruder does next on the host. The routine below alerts when a decoy key, file or URL is used, flags new <span class=\"highlighted\">authorized_keys<\/span> lines, cron files and systemd units, and replaces the service key on every host.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"recon-in-logs\">Low-and-slow scan detection: what probing leaves in your logs<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">In this lab the paced run stayed below both configured thresholds. An awk rule and fail2ban, both set to more than 30 4xx responses per source in 60 seconds, flagged a burst of 200 requests within half a second; the same requests spaced 4 seconds apart (a defender&#8217;s test parameter) produced zero hits over 13.4 minutes.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">The list was a fixed set of common admin, backup and config paths, with the decoy path at position 33. The burst ran at 177 requests per second with a browser user agent; 199 of 200 got a 404. The awk rule fired at 0.18 s and fail2ban 1.0.2, with a log-only action, logged its ban at 0.49 s.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>slow run: start 1795093668.369949388 end 1795094470.352856954 duration 801.982907566 s\nnginx lines: 200\nmax requests in any 60 s window: 15 (threshold 31)\nmean rate: 0.251 req\/s\nawk detector lines during slow run: 0\n[...]\ndecoy alert during slow run:\n1795093796.743609113 ALERT canary-url host=def-t1 hit_at=1795093796.741 from=10.77.0.20 \"GET \/backup.sql HTTP\/1.1\" 404 162 \"Mozilla\/5.0 (X11; Linux x86_64)\"<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">The busiest minute held 15 requests against a threshold of 31. The only alert came from the decoy URL <code>\/backup.sql<\/code>, 128 seconds in, on request 33, because that path was in the list; a decoy sees nothing else, so keep the threshold.<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1200\" height=\"800\" src=\"https:\/\/hostry.com\/blog\/wp-content\/uploads\/2026\/11\/ai-driven-attacks-infrastructure-defense-two-paces.webp\" alt=\"Low and slow scan detection: the same 200 requests as a burst flagged by a threshold rule in 0.18 s, and paced at one per 4 seconds with zero threshold alerts while a decoy URL alerts on request 33\" class=\"wp-image-3068\" srcset=\"https:\/\/hostry.com\/blog\/wp-content\/uploads\/2026\/11\/ai-driven-attacks-infrastructure-defense-two-paces.webp 1200w, https:\/\/hostry.com\/blog\/wp-content\/uploads\/2026\/11\/ai-driven-attacks-infrastructure-defense-two-paces-300x200.webp 300w, https:\/\/hostry.com\/blog\/wp-content\/uploads\/2026\/11\/ai-driven-attacks-infrastructure-defense-two-paces-1024x683.webp 1024w, https:\/\/hostry.com\/blog\/wp-content\/uploads\/2026\/11\/ai-driven-attacks-infrastructure-defense-two-paces-450x300.webp 450w, https:\/\/hostry.com\/blog\/wp-content\/uploads\/2026\/11\/ai-driven-attacks-infrastructure-defense-two-paces-768x512.webp 768w\" sizes=\"auto, (max-width: 1200px) 100vw, 1200px\" \/><figcaption>The same 200 requests, two paces: the threshold sees only the burst; the decoy sees both.<\/figcaption><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\">Both detectors read nginx&#8217;s log in the lab&#8217;s format, defined in the <code>http<\/code> block with <code>$msec<\/code> first:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>log_format tl '$msec $remote_addr \"$request\" $status $body_bytes_sent \"$http_user_agent\"';<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">The detector, <code>detect-burst.sh<\/code>, runs mawk (Ubuntu&#8217;s default awk) with <code>-W interactive<\/code>; without it mawk buffers output on a pipe, and a cold run of this page saw a live detector print nothing during a burst. With gawk, add <code>fflush()<\/code> after <code>printf<\/code>.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>#!\/usr\/bin\/env bash\n# Threshold detector: more than T bad events from one source within W seconds.\n#   detect-burst.sh nginx  -&gt; 4xx in the nginx access log (uses the log's own $msec)\n#   detect-burst.sh sshd   -&gt; Failed password\/publickey in the sshd log (sshd -E writes\n#                             no timestamps, so arrival time is prepended)\n# Every flagged line is prefixed with the wall-clock time the detector printed it.\nW=${W:-60}; T=${T:-30}; OUT=\/opt\/def-lab\/detect.log\ncase $1 in\nnginx) tail -n0 -F \/opt\/def-lab\/hosts\/t1\/log\/access.log | mawk -W interactive -v W=$W -v T=$T '\n  $6&gt;=400 { ip=$2; t=$1; q[ip]=q[ip] \" \" t; n=split(q[ip],a,\" \"); s=\"\"; c=0\n    for(i=1;i&lt;=n;i++) if(a[i]&gt;t-W){s=s \" \" a[i]; c++}; q[ip]=s\n    if(c&gt;T &amp;&amp; !f[ip]){ f[ip]=1; printf \"DETECT nginx-burst ip=%s %d 4xx in %ds (event at %s)\\n\", ip, c, W, t } }' ;;\nsshd) tail -n0 -F \/opt\/def-lab\/hosts\/t1\/log\/sshd.log | while IFS= read -r l; do echo \"$(date +%s.%N) $l\"; done | mawk -W interactive -v W=$W -v T=$T '\n  \/Failed (password|publickey) for\/ { for(i=1;i&lt;=NF;i++) if($i==\"from\") ip=$(i+1); t=$1\n    q[ip]=q[ip] \" \" t; n=split(q[ip],a,\" \"); s=\"\"; c=0\n    for(i=1;i&lt;=n;i++) if(a[i]&gt;t-W){s=s \" \" a[i]; c++}; q[ip]=s\n    if(c&gt;T &amp;&amp; !f[ip]){ f[ip]=1; printf \"DETECT ssh-burst ip=%s %d failures in %ds (event at %s)\\n\", ip, c, W, t } }' ;;\nesac | while IFS= read -r l; do echo \"$(date +%s.%N) $l\" | tee -a $OUT; done<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">fail2ban used two small filters and a jail. The stock <code>sshd<\/code> filter matched none of 423 lines of the lab&#8217;s <code>sshd -E<\/code> log, which lacks the syslog prefix it expects; the lab&#8217;s filter matched 17. On a journald host the stock filter applies.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code># filter.d\/def-nginx-4xx.conf\n[Definition]\nfailregex = ^\\s*&lt;HOST&gt; \"[^\"]*\" 4\\d\\d\ndatepattern = ^{EPOCH}\n# filter.d\/def-sshd-fail.conf\n[Definition]\nfailregex = ^Failed (?:password|publickey) for \\S+ from &lt;HOST&gt; port \\d+\ndatepattern = {NONE}\n# jail.conf\n[DEFAULT]\nbackend = polling\nbanaction = dummy\naction = dummy[target=\/opt\/def-lab\/f2b\/ban.log]\n[def-nginx-4xx]\nenabled = true\nfilter = def-nginx-4xx\nlogpath = \/opt\/def-lab\/hosts\/t1\/log\/access.log\nmaxretry = 31\nfindtime = 60\nbantime = 600\n[def-sshd]\nenabled = true\nfilter = def-sshd-fail\nlogpath = \/opt\/def-lab\/hosts\/t1\/log\/sshd.log\nmaxretry = 6\nfindtime = 60\nbantime = 600<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">One <code>nmap -sV<\/code> run left 8 nginx lines (four with the Nmap Scripting Engine&#8217;s agent string) and 3 sshd lines, and read the exact package builds from the banners.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>PORT   STATE SERVICE VERSION\n22\/tcp open  ssh     OpenSSH 9.6p1 Ubuntu 3ubuntu13.20 (Ubuntu Linux; protocol 2.0)\n80\/tcp open  http    nginx 1.24.0 (Ubuntu)\n[...]\nNmap done: 1 IP address (1 host up) scanned in 19.94 seconds\n[...]\nnginx lines: 8; sshd lines: 3<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">At sshd&#8217;s default <code>LogLevel INFO<\/code>, an attempt with a key the server does not accept leaves one line and no fingerprint, so a detector matching <code>Failed publickey<\/code> sees nothing.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>== LogLevel INFO: one wrong-key attempt -&gt; 1 lines\nConnection closed by authenticating user def-svc 10.77.0.20 port 49352 [preauth]\n== LogLevel VERBOSE: one wrong-key attempt -&gt; 3 lines\nConnection from 10.77.0.20 port 49354 on 10.77.0.15 port 22 rdomain \"\"\nFailed publickey for def-svc from 10.77.0.20 port 49354 ssh2: ED25519 SHA256:h70\/br\/cDDRZmaCO\/Vv10cq8CGFaoFaJW01gUcKoznE\nConnection closed by authenticating user def-svc 10.77.0.20 port 49354 [preauth]<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">If fail2ban is not detecting a slow scan, these are the usual reasons: the pace stays under <code>maxretry<\/code> per <code>findtime<\/code>, and rejected-key lines are not written at INFO. Set <code>LogLevel VERBOSE<\/code> in <span class=\"highlighted\">\/etc\/ssh\/sshd_config<\/span> on every host. The sshd half of the noise disappears once SSH stops listening publicly, and <strong><a href=\"https:\/\/hostry.com\/blog\/ssh-over-wireguard-ubuntu-24-04\/\">moving SSH behind a WireGuard tunnel on Ubuntu 24.04<\/a><\/strong> is one way to get there.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"decoys\">Canary tokens on a Linux server: a file, a key and a URL<\/h2>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1200\" height=\"675\" src=\"https:\/\/hostry.com\/blog\/wp-content\/uploads\/2026\/11\/ai-driven-attacks-infrastructure-defense-canary.webp\" alt=\"A glowing cage with a canary on a server rack draws in a swarm of small probing lights\" class=\"wp-image-3070\" srcset=\"https:\/\/hostry.com\/blog\/wp-content\/uploads\/2026\/11\/ai-driven-attacks-infrastructure-defense-canary.webp 1200w, https:\/\/hostry.com\/blog\/wp-content\/uploads\/2026\/11\/ai-driven-attacks-infrastructure-defense-canary-300x169.webp 300w, https:\/\/hostry.com\/blog\/wp-content\/uploads\/2026\/11\/ai-driven-attacks-infrastructure-defense-canary-1024x576.webp 1024w, https:\/\/hostry.com\/blog\/wp-content\/uploads\/2026\/11\/ai-driven-attacks-infrastructure-defense-canary-450x253.webp 450w, https:\/\/hostry.com\/blog\/wp-content\/uploads\/2026\/11\/ai-driven-attacks-infrastructure-defense-canary-768x432.webp 768w\" sizes=\"auto, (max-width: 1200px) 100vw, 1200px\" \/><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\">The lab&#8217;s decoy URL alerted in 0.013 s, the SSH key in 0.21 s and the credentials file in 0.24 s, each on the first touch, since nothing legitimate uses a decoy.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">MITRE D3FEND files this under D3-DUC, Decoy User Credential. Thinkst&#8217;s Canarytokens calls its tokens &#8220;like motion sensors for your networks, computers and clouds&#8221;; the official list has AWS API keys, kubeconfig and WireGuard configs but no SSH key type, so the SSH decoy below is built from sshd&#8217;s own key options. Never use a live credential as bait.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Decoy credentials file under auditd<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">On one host the service account got a <span class=\"highlighted\">~\/.aws\/credentials<\/span> file with fake documentation-style values (the lab&#8217;s example key ID was <code>AKIAEXAMPLEDECOY0001<\/code>) and an audit watch for reads. auditd must be running: <code>auditctl<\/code> accepts the rule while auditing is off (<code>enabled 0<\/code>) and records nothing. On a systemd host use <code>systemctl enable --now auditd<\/code> (not testable in the container). The rule:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>auditctl -w \/opt\/def-lab\/hosts\/t2\/home\/def-svc\/.aws\/credentials -p r -k def-canary-aws  # auditd -n -c \/opt\/def-lab\/audit (log_file changed only)<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">For reboots, put the rule without <code>auditctl<\/code> in a file under <span class=\"highlighted\">\/etc\/audit\/rules.d\/<\/span>; the stand recheck loaded that form with <code>auditctl -R<\/code>. The follower:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>#!\/usr\/bin\/env bash\n# Follows the lab audit log; one alert line per read of the decoy credentials file.\nLOG=\/opt\/def-lab\/audit\/log\/audit.log; OUT=\/opt\/def-lab\/alerts.log\ntail -n0 -F $LOG | grep --line-buffered 'key=\"def-canary-aws\"' | while read -r l; do\n  pid=$(grep -o ' pid=[0-9]*' &lt;&lt;&lt;\"$l\" | head -1 | cut -d= -f2)\n  uid=$(grep -o ' uid=[0-9]*' &lt;&lt;&lt;\"$l\" | head -1 | cut -d= -f2)\n  comm=$(grep -o 'comm=\"[^\"]*\"' &lt;&lt;&lt;\"$l\"); exe=$(grep -o 'exe=\"[^\"]*\"' &lt;&lt;&lt;\"$l\")\n  ppid=$(grep -o ' ppid=[0-9]*' &lt;&lt;&lt;\"$l\" | cut -d= -f2)\n  echo \"$(date +%s.%N) ALERT canary-file host=def-t2 file=~def-svc\/.aws\/credentials uid=$uid($(id -nu $uid 2&gt;\/dev\/null)) pid=$pid ppid=$ppid $comm $exe parent=\\\"$(ps -o args= -p $ppid 2&gt;\/dev\/null)\\\"\" | tee -a $OUT\ndone<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">It alerted 0.236-0.255 s after the reading SSH command started.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>1795093168.433433380 ALERT canary-file host=def-t2 file=~def-svc\/.aws\/credentials uid=30034(def-svc) pid=8135 ppid=8134 comm=\"head\" exe=\"\/usr\/bin\/head\" parent=\"\"<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">auditd names the reader (user, PID, program); inotifywait saw the access but not who. The login UID stayed &#8220;unset&#8221; because the lab&#8217;s sshd ran without PAM, and the source address came only from sshd&#8217;s log.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Honeytoken SSH key with restrict,command=<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">Every host got an extra <span class=\"highlighted\">authorized_keys<\/span> line for a key pair no person or job uses; <code>backup-deploy<\/code> is the lab&#8217;s example comment:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>restrict,command=\"\/opt\/def-lab\/bin\/def-canary-ssh\" ssh-ed25519 AAAA... backup-deploy<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">The forced command is <span class=\"highlighted\">\/opt\/def-lab\/bin\/def-canary-ssh<\/span>; keep it owned by root, mode 0755:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>#!\/bin\/sh\n# forced command of the decoy key: record and refuse\nprintf '%s ALERT canary-ssh-key host=%s from=\"%s\" cmd=\"%s\" user=%s\\n' \"$(date +%s.%N)\" \"$(hostname -I | cut -d' ' -f1)\" \"$SSH_CONNECTION\" \"$SSH_ORIGINAL_COMMAND\" \"$(id -un)\" &gt;&gt; \/opt\/def-lab\/alerts-ssh.log\nexit 1<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">It runs as the service account. On the stand, with no log file the client saw <code>cannot create \/opt\/def-lab\/alerts-ssh.log: Permission denied<\/code> and no alert was written; creating it with group write and no read fixed that:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>install -m 0620 -o root -g def-svc \/dev\/null \/opt\/def-lab\/alerts-ssh.log<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Using the decoy key produced an alert 0.212-0.218 s after the client started, with the source address.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>--- alerts-ssh.log\n1795093218.013188416 ALERT canary-ssh-key host=10.77.0.14 from=\"10.77.0.20 56958 10.77.0.14 22\" cmd=\"\" user=def-svc\n1795093218.237669850 ALERT canary-ssh-key host=10.77.0.14 from=\"10.77.0.20 56966 10.77.0.14 22\" cmd=\"id; cat \/etc\/shadow\" user=def-svc\n--- sshd t4\nAccepted publickey for def-svc from 10.77.0.20 port 56958 ssh2: ED25519 SHA256:zNBAhp94YvOUa7mHp\/S8icBM0uZ2XP\/XFl6uFY04Vsk\nStarting session: forced-command (key-option) '\/opt\/def-lab\/bin\/def-canary-ssh' for def-svc from 10.77.0.20 port 56958 id 0<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">The requested command was recorded and never run. But sshd logs the decoy as <code>Accepted publickey<\/code>, so a dashboard of successful logins counts it as a success unless you exclude its fingerprint. What else <code>restrict<\/code> blocks is tested in <strong><a href=\"https:\/\/hostry.com\/blog\/github-actions-deploy-vps-restricted-ssh-key\/\">the post on restricting a CI deploy key with restrict,command=<\/a><\/strong>.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Decoy URL on nginx<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">The cheapest decoy is an unlinked location in the <code>server<\/code> block that writes a second log and returns a 404:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code># decoy URL: never linked anywhere; any hit is an alert\nlocation = \/backup.sql {\n  access_log \/opt\/def-lab\/hosts\/t1\/log\/access.log tl;\n  access_log \/opt\/def-lab\/hosts\/t1\/log\/decoy.log tl;\n  return 404;\n}<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-code\"><code>#!\/usr\/bin\/env bash\n# one alert line per hit on the decoy URL (log line: $msec addr \"request\" status bytes \"ua\")\ntail -n0 -F \/opt\/def-lab\/hosts\/t1\/log\/decoy.log | while read -r ts addr rest; do\n  echo \"$(date +%s.%N) ALERT canary-url host=def-t1 hit_at=$ts from=$addr $rest\" &gt;&gt; \/opt\/def-lab\/alerts-url.log\ndone<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">One request with curl alerted 0.013 s after it was sent, and the visitor still got a plain 404:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>1795093409.783662971 ALERT canary-url host=def-t1 hit_at=1795093409.779 from=10.77.0.20 \"GET \/backup.sql HTTP\/1.1\" 404 162 \"curl\/8.5.0\"<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Pick a path scanners ask for and your site never serves, and exclude your own scanner by address.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"persistence-audit\">Check servers for backdoor authorized_keys, cron and systemd entries before you rotate<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">To check a server for a backdoor in <span class=\"highlighted\">authorized_keys<\/span>, cron or systemd, compare it with a baseline saved while it was clean, before any rotation. An exclusive rotation removes unknown keys, and with them the evidence.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">MITRE ATT&amp;CK lists these as T1098.004 SSH Authorized Keys, T1053.003 Cron and T1543.002 Systemd Service. The lab&#8217;s read-only audit, <code>audit-persistence.yml<\/code>, fingerprints every <span class=\"highlighted\">authorized_keys<\/span> line with <code>ssh-keygen -lf<\/code>, hashes <code>\/etc\/cron.d<\/code> and the <code>.service<\/code> and <code>.timer<\/code> files under <code>\/etc\/systemd\/system<\/code>, and diffs against the baseline and a list of expected fingerprints:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code># Read-only persistence audit. mode=baseline records the known state on the control\n# node; mode=check (default) diffs live state against it and the expected key inventory.\n- name: Collect authorized keys, cron.d and systemd units (read-only)\n  hosts: fleet\n  gather_facts: false\n  become: true\n  vars:\n    ansible_ssh_extra_args: \"-i {{ lookup('file', '\/opt\/def-lab\/ctl\/keys\/current') }}\"\n    audit_user_regex: '^def-'          # lab: only def-* accounts; on a real host drop this filter\n  tasks:\n    - name: Users with a home directory\n      ansible.builtin.shell: &gt;-\n        getent passwd | awk -F: '$1 ~ \/{{ audit_user_regex }}\/ {print $1\":\"$6}'\n      register: users\n      changed_when: false\n    - name: Fingerprint every authorized_keys line (SHA256, type, comment)\n      ansible.builtin.shell: |\n        for u in {{ users.stdout_lines | join(' ') }}; do\n          f=\"${u#*:}\/.ssh\/authorized_keys\"; [ -f \"$f\" ] || continue\n          ssh-keygen -lf \"$f\" | awk -v u=\"${u%%:*}\" '{print u\" \"$2\" \"$NF\" \"$0}' | cut -d' ' -f1-3\n        done\n      register: keys\n      changed_when: false\n    - name: Hash cron.d entries and systemd unit files\n      ansible.builtin.shell: |\n        find \/etc\/cron.d \/etc\/systemd\/system -type f \\( -path '\/etc\/cron.d\/*' -o -name '*.service' -o -name '*.timer' \\) -exec sha256sum {} + | sort -k2\n      register: files\n      changed_when: false\n\n- name: Compare with the expected inventory\n  hosts: fleet\n  gather_facts: false\n  vars:\n    mode: check\n    bdir: \/opt\/def-lab\/ctl\/baseline\n    expected_fps: \"{{ lookup('file', '\/opt\/def-lab\/ctl\/expected_keys.txt').split('\\n') | select | list }}\"\n    live_fps: \"{{ keys.stdout_lines | map('split', ' ') | map(attribute=1) | list }}\"\n  tasks:\n    - name: Save baseline of files\n      ansible.builtin.copy:\n        dest: \"{{ bdir }}\/{{ inventory_hostname }}.files\"\n        content: \"{{ files.stdout }}\\n\"\n      delegate_to: localhost\n      when: mode == 'baseline'\n    - name: Diff files against baseline\n      ansible.builtin.command: diff \"{{ bdir }}\/{{ inventory_hostname }}.files\" -\n      args:\n        stdin: \"{{ files.stdout }}\"\n      delegate_to: localhost\n      register: fdiff\n      changed_when: false\n      failed_when: false\n      when: mode == 'check'\n    - name: Report\n      ansible.builtin.debug:\n        msg:\n          keys_total: \"{{ keys.stdout_lines | length }}\"\n          unknown_keys: \"{{ keys.stdout_lines | reject('search', expected_fps | map('regex_escape') | join('|')) | list }}\"\n          new_or_changed_files: \"{{ (fdiff.stdout_lines | default([])) | select('match', '^[&lt;&gt;]') | list }}\"\n    - name: Flag host\n      ansible.builtin.fail:\n        msg: \"UNKNOWN persistence on {{ inventory_hostname }}\"\n      when: mode == 'check' and ((keys.stdout_lines | reject('search', expected_fps | map('regex_escape') | join('|')) | list | length &gt; 0) or ((fdiff.stdout_lines | default([])) | select('match', '^[&lt;&gt;]') | list | length &gt; 0))<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\"><span class=\"highlighted\">\/opt\/def-lab\/ctl\/expected_keys.txt<\/span> holds one fingerprint per line (service key and decoy), the second field of <code>ssh-keygen -lf key.pub<\/code>. One run with <code>-e mode=baseline<\/code> on a trusted fleet writes the baseline; both modes were rechecked on the stand. The audit connects through the rotation&#8217;s pointer file and needs sudo for <code>become<\/code>.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Three items were then planted (an extra key on t4, plus a cron file on t2 and a unit on t5 that each run only <code>\/bin\/true<\/code>):<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>ok: [def-t2] =&gt;\n        new_or_changed_files:\n        - '&gt; 7a92c62f[...]  \/etc\/cron.d\/def-sync'\n        unknown_keys: []\nok: [def-t4] =&gt;\n        keys_total: '3'\n        new_or_changed_files: []\n        unknown_keys:\n        - def-svc SHA256:WlU\/81YGWix0ESKki4HNGsP9m2r1Hlyx\/H9MuyI3Zv8 (ED25519)\nok: [def-t5] =&gt;\n        new_or_changed_files:\n        - '&gt; 52566652[...]  \/etc\/systemd\/system\/def-updater.service'\nfatal: [def-t4]: FAILED! =&gt;     msg: UNKNOWN persistence on def-t4\nfatal: [def-t2]: FAILED! =&gt;     msg: UNKNOWN persistence on def-t2\nfatal: [def-t5]: FAILED! =&gt;     msg: UNKNOWN persistence on def-t5\ndef-t1 [...] failed=0 \/ def-t3 [...] failed=0\nWALL 3.91 s<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">All three were flagged in 3.91 s, and the run exits non-zero (2 on the stand recheck). There is a blind spot, because <code>ssh-keygen -lf<\/code> prints the same fingerprint with and without the line&#8217;s options.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>$ ssh-keygen -lf (canary line WITH restrict,command=)\n256 SHA256:zNBAhp94YvOUa7mHp\/S8icBM0uZ2XP\/XFl6uFY04Vsk backup-deploy (ED25519)\n$ ssh-keygen -lf (same key, options stripped)\n256 SHA256:zNBAhp94YvOUa7mHp\/S8icBM0uZ2XP\/XFl6uFY04Vsk backup-deploy (ED25519)<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Stripping <code>restrict,command=<\/code> from an allowed key turns it into a full login that a fingerprint-only audit never reports, so hash whole lines too. Later, the next rotation silently removed the planted key on t4. That is correct for a rotation and bad for an investigation, so save the audit output first.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">On real hosts drop the <code>audit_user_regex<\/code> filter so root and every account are included. Web shells (T1505.003) sit in the web root, which this audit does not read; hash it the same way.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"rotation\">Rotate SSH keys with Ansible without locking yourself out<\/h2>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1200\" height=\"675\" src=\"https:\/\/hostry.com\/blog\/wp-content\/uploads\/2026\/11\/ai-driven-attacks-infrastructure-defense-key-carousel.webp\" alt=\"A data centre drawn as a clockwork machine: a carousel moves new keys past the server racks while probes circle outside the walls\" class=\"wp-image-3071\" srcset=\"https:\/\/hostry.com\/blog\/wp-content\/uploads\/2026\/11\/ai-driven-attacks-infrastructure-defense-key-carousel.webp 1200w, https:\/\/hostry.com\/blog\/wp-content\/uploads\/2026\/11\/ai-driven-attacks-infrastructure-defense-key-carousel-300x169.webp 300w, https:\/\/hostry.com\/blog\/wp-content\/uploads\/2026\/11\/ai-driven-attacks-infrastructure-defense-key-carousel-1024x576.webp 1024w, https:\/\/hostry.com\/blog\/wp-content\/uploads\/2026\/11\/ai-driven-attacks-infrastructure-defense-key-carousel-450x253.webp 450w, https:\/\/hostry.com\/blog\/wp-content\/uploads\/2026\/11\/ai-driven-attacks-infrastructure-defense-key-carousel-768x432.webp 768w\" sizes=\"auto, (max-width: 1200px) 100vw, 1200px\" \/><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\">Ansible SSH key rotation is safe when every host goes through four steps in this order:<\/p>\n\n\n\n<ol class=\"wp-block-list\"><li>Add the new public key while the old one stays.<\/li><li>Log in from the control node with only the new key.<\/li><li>Make the new key exclusive, keeping any listed decoy keys.<\/li><li>Prove that the old key is refused.<\/li><\/ol>\n\n\n\n<p class=\"wp-block-paragraph\">Five hosts started with one RSA key for the service account. The control node&#8217;s <code>ansible.cfg<\/code> pins host keys in its own <code>known_hosts<\/code> and the inventory names the fleet:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code># ansible.cfg\n[defaults]\ninventory = inventory.ini\nhost_key_checking = True\nforks = 10\ncallback_result_format = yaml\nstdout_callback = default\ntimeout = 5\n[ssh_connection]\nssh_args = -o UserKnownHostsFile=\/opt\/def-lab\/ctl\/known_hosts -o StrictHostKeyChecking=yes -o IdentitiesOnly=yes -o ControlMaster=no\npipelining = True\n# inventory.ini\n[fleet]\ndef-t1 ansible_host=10.77.0.11\ndef-t2 ansible_host=10.77.0.12\ndef-t3 ansible_host=10.77.0.13\ndef-t4 ansible_host=10.77.0.14\ndef-t5 ansible_host=10.77.0.15\n[fleet:vars]\nansible_user=def-svc\nansible_python_interpreter=\/usr\/bin\/python3<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">The complete playbook, <code>rotate-scheduled.yml<\/code>, adds an all-or-nothing gate at the start and a pointer file at the end, and already contains the three fixes described in the schedule section:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code># Scheduled variant of rotate.yml: old key = keys\/current pointer, new generation\n# from the clock; pointer moves only when EVERY fleet host converged.\n# Safe SSH key rotation for a service account.\n# Order: add new key -&gt; verify login with new key from control -&gt; only then make\n# the new key exclusive -&gt; verify the old key is refused. A host that is down or\n# fails verification keeps its old key; the control node keeps both private keys\n# until every host has converged.\n- name: Gate - every fleet host must answer with the current key, or nothing is touched\n  hosts: fleet\n  gather_facts: false\n  vars:\n    ansible_ssh_extra_args: \"-i {{ lookup('file', '\/opt\/def-lab\/ctl\/keys\/current') }}\"\n  tasks:\n    - name: Reach host with the current key\n      ansible.builtin.ping:\n      ignore_unreachable: true\n      register: gate_ping\n\n- name: Abort the whole run if any host failed the gate\n  hosts: localhost\n  connection: local\n  gather_facts: false\n  tasks:\n    - name: Hosts that did not answer\n      ansible.builtin.set_fact:\n        gate_failed: \"{{ groups['fleet'] | reject('in', groups['fleet'] | map('extract', hostvars) | selectattr('gate_ping', 'defined') | rejectattr('gate_ping.unreachable', 'defined') | map(attribute='inventory_hostname') | list) | list }}\"\n    - name: Stop before any change\n      ansible.builtin.fail:\n        msg: \"gate failed, no host rotated, unreachable: {{ gate_failed }}\"\n      when: gate_failed | length &gt; 0\n\n- name: Generate the new key pair on the control node\n  hosts: localhost\n  connection: local\n  gather_facts: false\n  vars:\n    keydir: \/opt\/def-lab\/ctl\/keys\n  tasks:\n    - name: Pick generation name\n      ansible.builtin.set_fact:\n        gen: \"{{ gen | default(lookup('pipe', 'date +%Y%m%d%H%M%S')) }}\"\n    - name: Refuse to reuse the current generation\n      ansible.builtin.assert:\n        that: keydir + '\/svc_' + gen != lookup('file', keydir + '\/current')\n        fail_msg: \"generation {{ gen }} is already current\"\n    - name: New ed25519 key for generation {{ gen }}\n      community.crypto.openssh_keypair:\n        path: \"{{ keydir }}\/svc_{{ gen }}\"\n        type: ed25519\n        comment: \"def-svc-{{ gen }}\"\n\n- name: Rotate def-svc key across the fleet\n  hosts: fleet\n  gather_facts: false\n  vars:\n    gen: \"{{ hostvars['localhost'].gen }}\"\n    keydir: \/opt\/def-lab\/ctl\/keys\n    new_key: \"{{ keydir }}\/svc_{{ gen }}\"\n    old_key: \"{{ lookup('file', keydir + '\/current') }}\"\n    # keys that must stay after rotation besides the new one (e.g. canary keys)\n    keep_keys: \"{{ lookup('file', '\/opt\/def-lab\/ctl\/keep_keys.pub', errors='ignore') | default('', true) | split('\\n') | select | list }}\"\n    ansible_ssh_extra_args: \"-i {{ new_key }} -i {{ old_key }}\"\n    sshopts: \"-o BatchMode=yes -o IdentitiesOnly=yes -o ConnectTimeout=5 -o UserKnownHostsFile=\/opt\/def-lab\/ctl\/known_hosts -o StrictHostKeyChecking=yes\"\n  tasks:\n    - name: 1. Add the new public key (old key stays)\n      ansible.posix.authorized_key:\n        user: def-svc\n        key: \"{{ lookup('file', new_key + '.pub') }}\"\n        state: present\n\n    - name: 2. Verify a fresh login with ONLY the new key\n      ansible.builtin.command: &gt;\n        ssh {{ sshopts }} -i {{ new_key }} def-svc@{{ ansible_host }} true\n      delegate_to: localhost\n      changed_when: false\n\n    - name: 3. Make the new key exclusive (removes the old key)\n      ansible.posix.authorized_key:\n        user: def-svc\n        key: \"{{ ([lookup('file', new_key + '.pub')] + keep_keys) | join('\\n') }}\"\n        exclusive: true\n        state: present\n\n    - name: 4. Prove the old key is now refused\n      ansible.builtin.command: &gt;\n        ssh {{ sshopts }} -i {{ old_key }} def-svc@{{ ansible_host }} true\n      delegate_to: localhost\n      register: oldtry\n      changed_when: false\n      failed_when: oldtry.rc == 0\n\n    - name: 5. Mark host as rotated\n      ansible.builtin.set_fact:\n        rotated: true\n\n- name: Move the control-node pointer only if the whole fleet converged\n  hosts: localhost\n  connection: local\n  gather_facts: false\n  vars:\n    keydir: \/opt\/def-lab\/ctl\/keys\n    done_hosts: \"{{ groups['fleet'] | map('extract', hostvars) | selectattr('rotated', 'defined') | map(attribute='inventory_hostname') | list }}\"\n    pending: \"{{ groups['fleet'] | difference(done_hosts) }}\"\n  tasks:\n    - name: Report hosts still on the old key\n      ansible.builtin.debug:\n        msg: \"pending (keep old private key!): {{ pending }}\"\n    - name: Point 'current' at the new key\n      ansible.builtin.copy:\n        dest: \"{{ keydir }}\/current\"\n        content: \"{{ keydir }}\/svc_{{ gen }}\"\n      when: pending | length == 0<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">It depends on the pointer file <span class=\"highlighted\">\/opt\/def-lab\/ctl\/keys\/current<\/span>, holding the absolute path of the private key the fleet trusts now (the stand recheck started it at <code>\/opt\/def-lab\/ctl\/keys\/svc_old<\/code>), on <code>keep_keys.pub<\/code> with the full decoy line, on the pinned <code>known_hosts<\/code>, and on every host being reachable.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><code>ansible_ssh_extra_args<\/code> decides the connection key: each task offers the new key, then the pointer&#8217;s key. With <code>ControlMaster=no<\/code> every task opens a fresh connection, so after step 3 removes the old key, the next task still gets in with the new one. A cold run that connected with one key and no shared connection saw step 3 remove the old key and then fail on every host. After a full run the pointer names the new key; two consecutive runs from a fresh state both exited 0 on the stand recheck.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">The failure tests used the same four tasks without gate and pointer (the lab&#8217;s <code>rotate.yml<\/code>, generation given as <code>-e gen=<\/code>). Five hosts took 5.3-5.7 s, each <code>changed=2<\/code>; a re-run with the same generation gave <code>changed=0<\/code> in 5.05 s. Then one host had sshd stopped and another got <code>PubkeyAcceptedAlgorithms rsa-sha2-512,rsa-sha2-256<\/code>, which refuses ed25519, so the new key installs and does not work.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>fatal: [def-t3]: UNREACHABLE! =&gt;\n    msg: 'Data could not be sent to remote host \"10.77.0.13\". [...] [Errno 32] Broken pipe. [Errno 32] Broken pipe'\nfatal: [def-t4 -&gt; localhost]: FAILED! =&gt;\n    [...] stderr: 'def-svc@10.77.0.14: Permission denied (publickey,password).'\ndef-t1                     : ok=4    changed=2    unreachable=0    failed=0\ndef-t2                     : ok=4    changed=2    unreachable=0    failed=0\ndef-t3                     : ok=0    changed=0    unreachable=1    failed=0\ndef-t4                     : ok=1    changed=1    unreachable=0    failed=1\ndef-t5                     : ok=4    changed=2    unreachable=0    failed=0\nWALL 4.40 s\n\nt1: ssh-ed25519 def-svc-g2;\nt2: ssh-ed25519 def-svc-g2;\nt3: ssh-rsa def-svc-2025;\nt4: ssh-rsa def-svc-2025;ssh-ed25519 def-svc-g2;\nt5: ssh-ed25519 def-svc-g2;<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Both problem hosts kept the old key. With pipelining on, the stopped sshd showed as <code>Broken pipe<\/code>; plain <code>ssh<\/code> said <code>Connection refused<\/code>, as does the gate output below. t4&#8217;s log said <code>signature algorithm ssh-ed25519 not in PubkeyAcceptedAlgorithms<\/code>. This run exited 4 (a host unreachable); a failed task or gate stop exits 2, so treat any non-zero status as &#8220;fleet not converged&#8221;.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">The common shortcut, one <code>exclusive: true<\/code> task without verification, reported success on the same t4:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>changed: [def-t4]\ndef-t4                     : ok=1    changed=1    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0\n=== after naive: can we still get in? ===\n-- with svc_old\ndef-svc@10.77.0.14: Permission denied (publickey,password).\nrc=255\n-- with svc_g2\ndef-svc@10.77.0.14: Permission denied (publickey,password).\nrc=255<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\"><code>failed=0<\/code>, and the host refused both keys. The lab recovered by writing the file back directly; a real server needs a console that does not depend on SSH. A <strong><a href=\"https:\/\/hostry.com\/products\/vps\/\">Hostry Linux VPS<\/a><\/strong> runs on KVM with its own kernel and full root, and the control panel&#8217;s VNC console is the way back in when a rotation goes wrong.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">One result went against expectations. The same safe order as a bash loop over plain <code>ssh<\/code> (21 commands for five hosts) was faster in machine time, 3.9-4.1 s against Ansible&#8217;s 5.3-5.7 s.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>10.77.0.11: rotated, old key refused\n[... .12-.15 identical ...]\ncommands run: 21\nm1 WALL 4.06 s\nm2 WALL 3.88 s\nm3 WALL 3.92 s<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Ansible carries about 3.2 s of fixed start-up, then about 1.9 s per extra host serially or 0.6 s in parallel (five hosts, 2 vCPU control node). The loop left <span class=\"highlighted\">authorized_keys<\/span> at mode 0644, where the module kept 0600. The playbook added a recap naming each failed host and a re-run that left converged hosts untouched.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"schedule\">Put the rotation on a schedule with Semaphore UI<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">A Semaphore UI schedule for an Ansible playbook is a five-field cron expression on a template, evaluated in UTC by default. Running every 2 minutes surfaced three faults in earlier revisions, all fixed in the file above, plus a clash with fail2ban.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Installing Semaphore is not covered here; the <strong><a href=\"https:\/\/semaphoreui.com\/docs\/admin-guide\/installation\" target=\"_blank\" rel=\"noopener\">Semaphore UI installation guide<\/a><\/strong> lists the routes. The <strong><a href=\"https:\/\/semaphoreui.com\/docs\/user-guide\/schedules\" target=\"_blank\" rel=\"noopener\">Semaphore schedules documentation<\/a><\/strong> says the scheduler &#8220;operates in the UTC timezone&#8221; unless <code>schedule.timezone<\/code> or <code>SEMAPHORE_SCHEDULE_TIMEZONE<\/code> names an IANA zone, and the service is restarted. The lab ran 2.17.15; GitHub listed v2.19.17 as latest at the time of writing. The schedule, created through the REST API:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>{\"id\":1,\"project_id\":1,\"template_id\":1,\"cron_format\":\"*\/2 * * * *\",\"name\":\"rotate every 2 min (lab)\",\"active\":true,\"type\":\"\",\"delete_after_run\":false,\"repository_id\":null}<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Tasks took 6.6-7.9 s for five hosts, clone included:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>7 success 13:03:02 b3504ce 13:03:03.481 13:03:10.559\n6 error 13:02:30 1320f16 13:02:31.092 13:02:37.474\n5 success 13:02:00 decf777 13:02:01.015 13:02:07.774\n4 success 13:00:00 decf777 13:00:01.021 13:00:08.915\n3 error 12:58:00 17f2956 12:58:01.015 12:58:02.419\n2 error 12:56:00 17f2956 12:56:01.020 12:56:05.899\n1 success 12:54:00 17f2956 12:54:01.031 12:54:07.627<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\" style=\"font-size:14px\"><strong>Fault 1 (fixed by the gate): one down host splits the fleet, and the next run locks out the rest<\/strong><\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Without a gate, task 2 ran with t3&#8217;s sshd stopped. Four hosts rotated and the pointer stayed: <code>pending (keep old private key!): ['def-t3']<\/code>. Task 3 then could not log in anywhere: the healthy hosts held the task-2 key, the run offered the pointer&#8217;s task-1 key and a brand-new one, and each answered <code>Permission denied (publickey,password)<\/code>.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">The gate pings every host with the current key and fails the run before any change if one does not answer. With t5 down:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>TASK [Reach host with the current key] *****************************************\nfatal: [def-t5]: UNREACHABLE! =&gt;\n    changed: false\n    msg: |-\n        Data could not be sent to remote host \"10.77.0.15\". Make sure this host can be reached over ssh: ssh: connect to host 10.77.0.15 port 22: Connection refused\n    unreachable: true\n...ignoring\n[...]\nTASK [Stop before any change] **************************************************\nfatal: [localhost]: FAILED! =&gt;\n    changed: false\n    msg: 'gate failed, no host rotated, unreachable: [''def-t5'']'<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">A fleet can still split when a host passes the gate and fails verification, as t4 did. The lab log does not record how the task-3 hosts came back; its helper <code>reset-old-key.sh<\/code> writes the old public key onto every host, standing in for a console. A stand recheck found a path without a console: fix the cause, point <code>current<\/code> at the generation named in the failed run&#8217;s log, and re-run. All five converged:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>echo -n \/opt\/def-lab\/ctl\/keys\/svc_&lt;generation&gt; &gt; \/opt\/def-lab\/ctl\/keys\/current<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-code\"><code>recover rc=0\nPLAY RECAP *********************************************************************\ndef-t1                     : ok=6    changed=2    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0\n[... def-t2 to def-t5 identical ...]\nlocalhost                  : ok=6    changed=2    unreachable=0    failed=0    skipped=1    rescued=0    ignored=0   <\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\" style=\"font-size:14px\"><strong>Fault 2 (fixed by keep_keys): the exclusive step deleted the decoy key<\/strong><\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><code>exclusive: true<\/code> keeps only the keys you pass, so task 5 silently removed the <code>backup-deploy<\/code> decoy everywhere. The <code>keep_keys<\/code> file, joined to the new key in step 3, fixed it.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\" style=\"font-size:14px\"><strong>Fault 3 (fixed by seconds and an assert): two runs in one minute produced the same key name<\/strong><\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Task 6, triggered in the same minute as task 5, got the same <code>date +%Y%m%d%H%M<\/code> name, so step 4 failed everywhere; nothing broke. Seconds plus an assert fixed it, and task 7 succeeded.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\" style=\"font-size:14px\"><strong>The clash: your own rotation trips fail2ban<\/strong><\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Each rotation fails two logins per host by design: the first task offers the new key before it is installed, and step 4 tries the old key after removal. The lab&#8217;s sshd log shows one <code>Failed publickey<\/code> line per key per run, and the stand recheck counted 2 per host. With a 6-in-60-seconds jail, three runs within 27 seconds got the control node banned:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>2026-11-19 15:09:40,579 fail2ban.actions        [11007]: NOTICE  [def-sshd] Ban 10.77.0.10<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">A real firewall action would block the rotation itself. The lab&#8217;s jail has no exception; on the stand recheck this line under <code>[DEFAULT]<\/code> let three back-to-back rotations through:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>ignoreip = 127.0.0.1\/8 10.77.0.10<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-code\"><code>2026-11-19 16:00:07,963 fail2ban.filter         [14670]: INFO    [def-sshd] Ignore 10.77.0.10 by ip\n[...]\n2026-11-19 16:00:28,228 fail2ban.filter         [14670]: INFO    [def-sshd] Ignore 10.77.0.10 by ip\n[...]\n   |- Currently banned:\t0\n   |- Total banned:\t0<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Because an exempt control node holds every key, keep its private keys readable only by the automation user, and reach it only over a VPN or from an allow-listed address.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"decoy-to-rotation\">Wire a decoy alert to rotation<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">From the decoy key being used to the whole fleet rotated took 7.10-7.30 s in three automated runs. By hand the same safe loop needs 4.1-4.3 s of machine time plus the time a person takes to notice and start it, which the lab did not measure.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">The watcher, <code>respond-on-canary.sh<\/code>, changes into the playbook directory, follows the decoy key&#8217;s log and runs the gated playbook once:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>#!\/usr\/bin\/env bash\n# Detect-to-rotate: on the first canary-ssh alert, run the gated rotation playbook once.\ncd \/opt\/def-lab\/ctl\/ans\ntail -n0 -F \/opt\/def-lab\/alerts-ssh.log | head -n1 | while read -r l; do\n  echo \"$(date +%s.%N) RESPOND start (alert: $l)\"\n  ip netns exec def-ctl ansible-playbook rotate-scheduled.yml &gt; \/opt\/def-lab\/respond-play.txt 2&gt;&amp;1; rc=$?\n  echo \"$(date +%s.%N) RESPOND end rc=$rc pointer=$(cat \/opt\/def-lab\/ctl\/keys\/current)\"\ndone<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-code\"><code>run 1: KEY-USE 1795093755.573500503\n1795093755.790416440 RESPOND start (alert: 1795093755.784526586 ALERT canary-ssh-key host=10.77.0.14 [...])\n1795093762.676435448 RESPOND end rc=0 pointer=\/opt\/def-lab\/ctl\/keys\/svc_20261119150917<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">The alert landed 0.21-0.22 s after the key use and the playbook started 0.005-0.007 s later; the rest is the rotation.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">The lab ran alert and rotation only; this is our proposed policy for the full chain:<\/p>\n\n\n\n<ol class=\"wp-block-list\"><li>On a decoy alert, run the read-only audit (3.9 s on this stand) and store its output before anything changes.<\/li><li>If the audit reports an unknown key, cron file or unit, or fails, page a person and do not rotate automatically, since rotation erases evidence.<\/li><li>If a host is unreachable, the gate stops the run (exit 2) and the alert goes to a person.<\/li><li>Only a clean audit and a passed gate let the rotation proceed unattended.<\/li><li>As a suggestion, allow one automatic rotation per hour per alert source, so a decoy fired on purpose cannot churn the fleet.<\/li><\/ol>\n\n\n\n<p class=\"wp-block-paragraph\">The gate refuses to rotate while any host is down, which suits a schedule. During an incident a person may rotate the reachable hosts on purpose with the ungated playbook. The down host then keeps its old key, and the control node keeps the old private key until it converges.<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1200\" height=\"800\" src=\"https:\/\/hostry.com\/blog\/wp-content\/uploads\/2026\/11\/ai-driven-attacks-infrastructure-defense-alert-loop.webp\" alt=\"Decoy alert to SSH key rotation loop: decoy fires in 0.01-0.24 s, persistence audit in 3.9 s, rotation on all hosts in 5.3-5.7 s, old key refused, with a branch to a person when the audit finds something and the scheduled run as a drill\" class=\"wp-image-3069\" srcset=\"https:\/\/hostry.com\/blog\/wp-content\/uploads\/2026\/11\/ai-driven-attacks-infrastructure-defense-alert-loop.webp 1200w, https:\/\/hostry.com\/blog\/wp-content\/uploads\/2026\/11\/ai-driven-attacks-infrastructure-defense-alert-loop-300x200.webp 300w, https:\/\/hostry.com\/blog\/wp-content\/uploads\/2026\/11\/ai-driven-attacks-infrastructure-defense-alert-loop-1024x683.webp 1024w, https:\/\/hostry.com\/blog\/wp-content\/uploads\/2026\/11\/ai-driven-attacks-infrastructure-defense-alert-loop-450x300.webp 450w, https:\/\/hostry.com\/blog\/wp-content\/uploads\/2026\/11\/ai-driven-attacks-infrastructure-defense-alert-loop-768x512.webp 768w\" sizes=\"auto, (max-width: 1200px) 100vw, 1200px\" \/><figcaption>Decoy alert, then audit, then rotation, then the old-key check; an audit finding stops the chain and pages a person.<\/figcaption><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\">Keep the scheduled run mainly as a drill that proves the rotation works on today&#8217;s fleet, and run it again after any change to the playbook, sshd policy or host list.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"standards\">What CASA, SOC 2 and NIST actually say about key rotation<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">None of these frames sets a rotation interval for SSH keys. The closest published figure is NIST&#8217;s 1-2 years for a private authentication key, offered as a rough guideline.<\/p>\n\n\n\n<figure class=\"wp-block-table\"><table><thead><tr><th>Frame<\/th><th>What it covers<\/th><th>What it says on rotation<\/th><\/tr><\/thead><tbody><tr><td>CASA spec v2.1.1 (App Defense Alliance, June 3, 2026)<\/td><td>App security for AL1 (verified self assessment) and AL2 (lab assessment); item 6.7.1 on storing secrets<\/td><td>No rotation requirement<\/td><\/tr><tr><td>SOC 2 Trust Services Criteria (AICPA)<\/td><td>CC6 logical access, CC7 system operations and monitoring, CC8 change management<\/td><td>No interval set, according to auditors&#8217; guides<\/td><\/tr><tr><td>NIST SP 800-63B-4<\/td><td>Passwords and other authenticators for people<\/td><td>Periodic password changes must not be required; force a change on evidence of compromise<\/td><\/tr><tr><td>NIST SP 800-57 Part 1 Rev. 5<\/td><td>Cryptographic key management<\/td><td>Table 1: private authentication key 1-2 years, as a rough guideline<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\">For CASA assessments, the ADA says the framework user, such as Google, decides an app&#8217;s assurance level, and &#8220;All applications must be revalidated every year.&#8221; The <strong><a href=\"https:\/\/github.com\/appdefensealliance\/ASA-WG\/blob\/main\/CASA\/CASA%20Specification.md\" target=\"_blank\" rel=\"noopener\">CASA specification v2.1.1<\/a><\/strong> item closest to this subject is 6.7.1: &#8220;The application shall securely store access tokens, API keys, and other server-side secrets.&#8221; Nothing in the spec requires rotating them, so decoys and gated rotation are at most supporting evidence for 6.7.1.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">On <strong>SOC 2 key rotation requirements<\/strong>, the AICPA text sits behind a login and was not read. The criteria do not set an interval, according to auditors&#8217; guides; Copla&#8217;s (July 2025, updated January 2026) maps credential controls to CC6.1-CC6.3 and calls 60-90 days common practice. The routine maps onto CC6 logical access, CC7 monitoring and CC8 change management, where a playbook in git is easier to show than a shell history. An autumn 2026 exposure draft would renumber these to CC5, CC6 and CC7, per A-LIGN (November 5, 2026); comments close January 15, 2027.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong><a href=\"https:\/\/pages.nist.gov\/800-63-4\/sp800-63b.html\" target=\"_blank\" rel=\"noopener\">NIST SP 800-63B-4, section 3.1.1.2<\/a><\/strong> says verifiers &#8220;SHALL NOT require subscribers to change passwords periodically&#8221; and must force one on evidence of compromise; it is written for human passwords. SP 800-57 Part 1 Rev. 5 (May 2020) suggests 1 to 2 years for a private authentication key and calls its table &#8220;only rough order-of-magnitude guidelines&#8221;. Reading an SSH key as such a key is this page&#8217;s inference.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Our proposed operating policy, which no standard requires: rotate on evidence (a decoy fired, a person left, a key leaked), treat each SSH key as a private authentication key and replace it within NIST&#8217;s rough 1-2 year guideline, and run the schedule as a drill.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"not-covered\">Where decoys and key rotation do not protect you<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">These controls only cover activity after access: a key in use, a planted job, a probe of a decoy path. They do not stop the initial compromise.<\/p>\n\n\n\n<ul class=\"wp-block-list\"><li>Application-layer flaws in your own code or in exposed panels are found by code review and patching.<\/li><li>Phishing and stolen human passwords are an identity problem, handled with phishing-resistant MFA on the accounts that can reach your servers.<\/li><li>Patch speed answers the exploitation window above; automatic security updates and a written patch window for internet-facing services are the starting point.<\/li><li>An intruder who touches no decoy and plants nothing in the audited places stays invisible here.<\/li><\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">The routine runs from one control node: an Ansible checkout, optionally Semaphore, and a watcher on the decoy logs. A Hostry Linux VPS can host that node, with plans that start at $6.00\/mo and go up to 32 GB RAM. Start with the audit baseline, because the audit compares every later state against it.<\/p>\n<!-- AddThis Advanced Settings generic via filter on the_content --><!-- AddThis Share Buttons generic via filter on the_content -->","protected":false},"excerpt":{"rendered":"<p>Slow probing at one request every 4 seconds tripped no threshold rule in a five-host lab, while a decoy URL caught it on request 33. What the AI-attac&#8230;<\/p>\n","protected":false},"author":6,"featured_media":3073,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1,127,20],"tags":[],"class_list":["post-3075","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-virtual-private-server","category-security","category-system-admin"],"read_time":23,"_links":{"self":[{"href":"https:\/\/hostry.com\/blog\/wp-json\/wp\/v2\/posts\/3075","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/hostry.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/hostry.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/hostry.com\/blog\/wp-json\/wp\/v2\/users\/6"}],"replies":[{"embeddable":true,"href":"https:\/\/hostry.com\/blog\/wp-json\/wp\/v2\/comments?post=3075"}],"version-history":[{"count":1,"href":"https:\/\/hostry.com\/blog\/wp-json\/wp\/v2\/posts\/3075\/revisions"}],"predecessor-version":[{"id":3077,"href":"https:\/\/hostry.com\/blog\/wp-json\/wp\/v2\/posts\/3075\/revisions\/3077"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/hostry.com\/blog\/wp-json\/wp\/v2\/media\/3073"}],"wp:attachment":[{"href":"https:\/\/hostry.com\/blog\/wp-json\/wp\/v2\/media?parent=3075"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/hostry.com\/blog\/wp-json\/wp\/v2\/categories?post=3075"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/hostry.com\/blog\/wp-json\/wp\/v2\/tags?post=3075"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}