Tom Hundley
🤖 Ghostwritten by Claude Opus 4.6 · Fact-checked & edited by GPT 5.4
A credential rotation only works if it is complete. In Grafana Labs' May 18, 2026 disclosure, the company said attackers accessed its GitHub environment using a GitHub token that was not replaced during an emergency rotation. The practical lesson is straightforward: rotating most secrets is not enough if one still-valid token remains in the path of an attacker.
The incident also appears to have included an extortion attempt tied to stolen source code, which Grafana said it would not pay. For engineering teams, the useful takeaway is less about the drama of the breach and more about the mechanics: inventory every credential, replace it everywhere, revoke the old one, and verify that revocation actually worked.
TL;DR: Rotation means issuing a new credential, updating every system that uses it, revoking the old one, and confirming the old one no longer works.
If you have ever changed a password after a breach alert, you have done the simplest form of rotation. For production systems, though, rotation usually involves API tokens, deploy keys, database passwords, OAuth client secrets, CI credentials, and service accounts spread across multiple tools.
In practice, a complete rotation has four parts:
Miss any one of those steps and the rotation is incomplete. The specific lesson from the Grafana case is not that rotation failed as a concept, but that inventory and verification matter as much as issuance and revocation.
TL;DR: Secrets spread across tools and environments, and emergency response exposes every gap in inventory discipline.
Incomplete rotations usually happen because credentials are duplicated in more places than teams remember under pressure. A single token may exist in:
.env filesThat is why emergency rotations are hard even for mature teams. The problem is rarely the act of generating a new token. The problem is knowing every place the old one lives.
Reporting on May 18, 2026 described Grafana's root cause as a missed GitHub token during a broader rotation effort after the TanStack-related supply-chain fallout. Follow-up coverage also connected the incident to a wider chain of May 2026 compromises affecting multiple organizations. Some details remain based on public reporting rather than a full post-incident technical report, so the safest conclusion is narrow: one overlooked credential was enough to preserve attacker access.
TL;DR: Build an inventory first, update every location, verify revocation with a real test, and document the result for the next incident.
Before rotating anything, list every place the credential may exist.
| Location | Example | Commonly missed? |
|---|---|---|
| Local env files | .env, .env.local, .env.production |
Sometimes |
| Hosting platform settings | Vercel, Fly.io, Railway, Netlify | Yes |
| CI/CD secrets | GitHub Actions, GitLab CI, CircleCI | Often |
| Secrets manager | 1Password, Doppler, AWS Secrets Manager | Less often |
| Teammate machines | Local clones and shell profiles | Often |
| Legacy configs | Staging, preview, or abandoned services | Very often |
| Source history | Old commits containing plaintext secrets | Very often |
| Third-party automations | Zapier, n8n, webhook endpoints | Often |
Issue the new credential and update every location on the inventory. Do not rely on memory, and do not assume the obvious systems are the only systems.
This is the step many teams skip. After revocation, try to use the old credential in a controlled test. If it still authenticates, the rotation is not complete.
Maintain a simple credential inventory with:
That record reduces guesswork during the next incident and makes routine rotation easier.
TL;DR: AI tools can help find references to a credential, but use them to locate secrets—not to paste secrets into prompts.
AI coding assistants can help search a codebase and config files for references to a credential. The safe pattern is to describe the secret and its likely variable names or prefixes without pasting the secret value itself.
Use a prompt like this:
I need to rotate a credential used in this project. Help me build a complete inventory of every place it may be stored or referenced.
Check and list:
1. Env files such as .env, .env.local, and .env.production
2. Hardcoded references in source code using likely variable names or known prefixes
3. CI/CD configuration files
4. Docker and compose files
5. Deployment platform config files
6. Package scripts and build scripts
7. Git history for accidental plaintext commits
8. Documentation and README references
9. Third-party integration configs and webhook settings
10. Any stack-specific locations based on this project architecture
For each location found, tell me what needs to be updated. Then suggest a safe verification step to confirm the old credential no longer works.Two cautions matter here:
TL;DR: Upstream compromises create downstream cleanup work, and the organizations that handle rotation best are the ones with current inventories and short-lived credentials.
The Grafana incident was part of a broader May 2026 security wave linked in public reporting to the TanStack compromise and related downstream exposure. Not every affected organization was hit in the same way, and not every incident shared the same root cause. Still, the pattern is familiar: one upstream compromise forces many downstream teams into urgent credential rotation and access review.
That is why the most resilient approach is preventive rather than reactive:
The extortion angle reinforces a separate point: backups, source control, and incident response preparation give organizations more options after a breach. They do not prevent compromise, but they can reduce the leverage attackers gain from it.
Public reporting said Grafana Labs disclosed unauthorized access to its GitHub environment on May 18, 2026 and attributed the access to a GitHub token that was missed during an emergency credential rotation. Reports also said the company received an extortion demand related to stolen source code and declined to pay.
Because attackers only need one valid path back in. If a single token, deploy key, or service credential remains active, the rest of the rotation may not materially reduce risk.
Use a controlled test after revocation. For example, make an API request with the old key or attempt an authenticated operation that should now fail. Verification should be documented, not assumed.
There is no universal interval that fits every system, but many organizations use a regular schedule for long-lived secrets and rotate immediately after suspected compromise, personnel changes, or upstream incidents. Where possible, replacing long-lived credentials with short-lived, automatically expiring tokens is stronger than relying on manual rotation alone.
At minimum: the secret name, purpose, owner, storage locations, dependent systems, last rotation date, and the test used to confirm revocation. That turns rotation from a memory exercise into an operational process.
The clearest lesson from Grafana's May 2026 breach is operational, not theoretical: secret rotation fails at the edges. One overlooked token can undo an otherwise disciplined response. For teams shipping quickly, the unglamorous controls still matter most: maintain an inventory, rotate methodically, verify revocation, and reduce reliance on long-lived credentials wherever possible.
Discover more content: