We asked DeepSeek to write 24 smart contracts, then scanned them: results and false positives

By ContractScan Team · 2026-10-15

Disclosure: an LLM wrote the contracts in this post, because that is what we were testing. We wrote and checked the analysis and prose ourselves. This post is not an audit.

Short answer

We gave DeepSeek's API 24 plain-language requests for six common contract types, then ran Slither, Aderyn and ContractScan's own heuristics over the output.

• 22 of 24 compiled. The two failures had real syntax errors.
• We confirmed two real high-severity bugs from the tools. Both are reentrancy paths in one AMM. A third is plausible but unverified. A heuristic also correctly flagged an owner function that can take every depositor's ETH, which the prompt had asked for.
• Most "high" findings were false positives. Slither: 2 true, 1 uncertain, 4 false out of 7. Aderyn: 0 true out of 15.
• The worst bug we found by reading the code was flagged by nothing. A fixed-APR staking contract pays rewards from other stakers' deposits.
• Our own verdicts were wrong in both directions. We changed the scanner and re-ran the same files. Details are at the end.

Twenty-four samples do not establish a "vulnerability rate" for AI-written Solidity. They do show that on these contracts, static analysis was noisy in both directions, and the most serious problems were economic or "it did what you asked" problems rather than classic code bugs.

Why we ran this

"Is AI-written code safe?" keeps coming up. Examples include the Hacker News thread about a vibe-coded app hacked twice and the discussion of Anthropic's experiment with AI agents exploiting contracts. We found almost no published data on LLM-written Solidity in particular, so we produced a small, fully published dataset.

Method

Generation. We wrote 3 differently phrased prompts for each of 6 types: ERC-20 with tax, staking pool, ETH vault/escrow, NFT mint with allowlist, AMM/token launcher, and multisig/timelock. The prompts read like real requests. One example: "I'm launching a meme coin on Base. Need a token contract with a 3% buy tax and 7% sell tax sent to my dev wallet, and I want to be able to change the taxes later." Every prompt ended by asking for a single self-contained file with no imports and pragma solidity ^0.8.20.

• We sent 18 requests as deepseek-chat and repeated the first prompt of each type as deepseek-reasoner (6 more).
• Settings: default temperature, no system prompt, one turn.
• All calls ran on 2026-10-15 around 06:48 UTC.
• Both model names were served by DeepSeek-V4.1-Flash, according to the responses and DeepSeek's pricing page. So in practice we compared one model with thinking off and on.
• Usage was 1,948 input and 91,285 output tokens, about $0.11 at the peak rate.

Scanning. ContractScan compiles each file in a sandbox with solc 0.8.24 and runs Slither 0.11.6 (default detectors), Aderyn 0.6.8, and regex "rug pattern" heuristics. We recorded compile failures and did not fix them.

Triage. We manually reviewed every high finding and labeled it true, false, uncertain, or "accurate but expected" (the flag is correct and the behavior is what the user asked for). We wrote no exploit PoCs, so "true" means confirmed by reading the code.

Results

How we triaged the high findings:

Only 1 of the 4 "do_not_deploy" verdicts rested on a true finding. The vault where the owner can take all deposits got only "fix_first". One ERC-20 got "fix_first" because our fee-change regex matched setExcludedFromTax.

Notable examples

1. AMM: real reentrancy (Slither caught it, Aderyn didn't)

dex_launcher_1_chat has no reentrancy guard and sends ETH before it syncs reserves:

amountETHOut = (amountInWithFee * reserveETH) / (reserveToken * FEE_DENOMINATOR + amountInWithFee);
_safeTransferETH(to, amountETHOut);   // external call to a caller-chosen address
_update();                            // reserves synced only afterwards

The receiving contract can call back in and swap against a stale, too-high reserveETH. removeLiquidity has the same flaw: LP tokens are burned before reserves update. Slither reported both as reentrancy-eth (SWC-107). The fix is a guard plus updating reserves before the transfer. The thinking-mode sample guarded every entry point, and Slither still flagged it (reentrancy-balance, a false positive).

2. Staking: insolvent by design, flagged by nothing

The prompt asked users to "earn rewards in the same token at a fixed APR of 20%." The contract accrues rewards with a standard accumulator:

uint256 reward = rate * elapsed;
accRewardPerToken += (reward * ACC_PRECISION) / totalStaked;

There is no funding function and no reward reserve, and claims are paid from the same pool as deposits. Unless someone keeps topping it up, early claimers are paid with later stakers' principal, and the last users out lose. The only tool output was a Slither medium on a timestamp equality, which is noise.

3. Vault: the "emergency withdraw" the user asked for

The prompt asked for "an emergency withdraw for the owner just in case." The model wrote exactly that:

function emergencyWithdraw() external onlyOwner {
    uint256 balance = address(this).balance;
    totalLocked = 0;
    (bool ok, ) = payable(owner).call{value: balance}("");

This sends every depositor's ETH to the owner. The model gave no warning about it. Only our heuristic flagged it, and at the time it produced just "fix_first". This is a case of the LLM shipping the default it was asked for, not inventing a bug.

4. Compile failures

One file declared an interface inside a contract. The other put @openzeppelin/merkle-tree) in a NatSpec comment, which solc rejects as an unknown tag. A user pasting either into Remix gets an error, not a contract.

What the tools got wrong

• Aderyn noise. eth-send-unchecked-address fired 7 times; each time the recipient was msg.sender, the owner or a configured fee address. abi-encode-packed-hash-collision fired 4 times on strings that are never hashed. reentrancy-state-change fired 3 times: on a constructor, on a view call, and on code that already updates state first. weak-randomness fired once on a transaction-ID hash.
• Misses. The staking insolvency, and Aderyn's miss of the AMM reentrancy. We read the flagged code and spot-checked the rest but did not fully audit all 22 contracts, so other misses are likely.

What we changed after this experiment

We changed ContractScan's verdict logic based on this triage. We added regression tests using these contracts and re-scanned the same 24 files. We did not regenerate anything.

1. Corroboration. A single high finding no longer produces "do_not_deploy" by itself. Now one of these is required:
2. a high-precision Slither detector (reentrancy-eth, arbitrary-send-eth, suicidal or controlled-delegatecall), or
3. a second tool or heuristic flagging overlapping lines.

Otherwise the verdict is "fix_first" with low confidence. Every report now includes the confidence level and the reasons for the verdict. 2. Aderyn downgrades. The four Aderyn detectors above produced only false positives here. They now report as low severity: still visible, but they no longer drive the verdict. 3. Owner can take user funds. If a contract accepts deposits or stakes, and an owner or admin function can send its whole balance (or an arbitrary amount) out with no visible timelock, the verdict is now "do_not_deploy". NFT mint proceeds, escrow arbiters, and withdrawals capped at "balance minus user deposits" are excluded. 4. Fee-change regex. Exemption and recipient setters (setExcludedFromTax, setFeeRecipient, …) no longer count as fee changes. 5. New medium heuristic. It flags fixed APR/APY rewards paid in the staked token when the contract has no funding function and no reward reserve. It is deliberately conservative.

Verdicts before and after:

What moved:

• Both true blockers now get "do_not_deploy": the AMM reentrancy and the vault owner-drain (which moved up from "fix_first").
• Two false "do_not_deploy" verdicts became "fix_first" (low confidence): the guarded AMM and the NFT tokenURI case.
• One false "do_not_deploy" remains. arbitrary-send-eth on a vault that only pays msg.sender or the owner's accrued fees. We kept that detector on the precise list. Across 24 files that is one data point, not enough to drop it.
• Five contracts moved from "fix_first" to "deployable": they were held back only by downgraded Aderyn findings or the fee-regex misfire. "Deployable" means static analysis found nothing high or medium. It does not mean safe. The insolvent staking contract stays at "fix_first", now with an explicit solvency warning.

Tuning rules against the same 24 files we measured on is overfitting by construction. The real test is the next, independent sample.

Limitations

• One model family. Every sample came from DeepSeek-V4.1-Flash, with thinking off and on.
• Small n. Four samples per type, one temperature, one day. Read the counts as anecdotes with receipts.
• No imports. We asked for self-contained files. Real users often get OpenZeppelin imports, so hand-rolled bugs are probably overrepresented.
• Static analysis and code reading only. No fuzzing, fork tests or exploits.
• One reviewer pass. The triage is our judgment, published row by row so you can dispute specific rows.
• Not an audit. Do not deploy any of these contracts.

Checklist for AI-written Solidity

1. Compile with the exact solc you will deploy with.
2. List every owner-only function that moves user funds, and decide whether you would accept it as a user.
3. For anything paying yield, write down where rewards come from and check the contract enforces that.
4. Treat any external call made before state is updated as a bug until proven otherwise.
5. Treat scanner "high" labels as leads to check, not verdicts.

Reproduce it

All data is at https://github.com/contractscan-dev/ai-solidity-experiment-2026-10 and the scanner source is at https://github.com/contractscan-dev/contractscan. We publish the prompts, generation script, raw API responses (with returned model name and token usage), extracted contracts, scan script, raw findings, per-contract results before and after the change, and our per-finding triage (review.csv). Tools: Slither 0.11.6, Aderyn 0.6.8, solc 0.8.24. LLM output is non-deterministic, so a re-run will produce different contracts. The scan step is reproducible from the published .sol files.

Sources

• DeepSeek API pricing and models: https://api-docs.deepseek.com/quick_start/pricing (fetched 2026-10-15)
• Anthropic, "AI agents find $4.6M in blockchain smart contract exploits": https://red.anthropic.com/2025/smart-contracts/
• HN discussion of the Anthropic post: https://news.ycombinator.com/item?id=46115214
• HN, "I vibe coded and shipped an app in three days. It got hacked. Twice.": https://news.ycombinator.com/item?id=44157131
• SWC-107 Reentrancy: https://swcregistry.io/docs/SWC-107
• Experiment data: https://github.com/contractscan-dev/ai-solidity-experiment-2026-10
• Slither: https://github.com/crytic/slither · Aderyn: https://github.com/Cyfrin/aderyn