Introduction
The question “will AI replace programmers?” is usually posed and answered badly. The optimistic camp says AI is just autocomplete; the pessimistic camp says coders are obsolete. Both miss what is actually happening.
This article works through the question systematically, building from first principles:
- What are the real limits of AI coding ability?
- Does AI learn bad code, and does that matter?
- What happens when AI trains on AI?
- Can verification save us?
- What does economic history suggest?
- What does the programmer’s role actually become?
- What are the physical constraints?
- What are the ecological risks?
- What does collective intelligence tell us?
The conclusion is neither utopian nor apocalyptic, but it is uncomfortable: programmers will not disappear, but the profession will be unrecognizable within a generation.
Part 1: The Limits of AI Coding
1.1 The “finite code” fallacy
A common worry: AI can only learn from existing code, so it must hit a ceiling.
This misunderstands what coding skill is.
Coding is not memorization of a corpus. It is search over a combinatorial space of programs, which is effectively unbounded. The number of syntactically valid programs grows exponentially with length.
AI improves not only from more training data, but from:
- Better reasoning
- Longer context windows
- Tool use (compilers, tests, debuggers)
- Self-correction loops
None of these depend on “how much code exists.”
1.2 The real ceilings
The limits that do exist are different:
| Limit | Why it matters |
|---|---|
| Verification | AI generates code faster than humans can review it |
| Novel systems | Distributed systems, hardware, legacy code — no correct answer in training data |
| Accountability | Someone must be legally and ethically responsible |
| Context | Business logic, user intent, org politics — much of it never written down |
Implication: The value of typing code is collapsing. The value of deciding what to build and proving it correct is rising.
Part 2: Does AI Learn Bad Code?
2.1 The premise is correct
The majority of real-world code is mediocre:
- Rushed, undocumented, inconsistent
- Full of workarounds, dead code, copy-paste
- Textbook-good code is a rounding error
- Much of the corpus is tutorial code and homework
By raw volume, AI’s diet is mostly not-great code.
2.2 But AI is not a naive imitator
Modern models write better code than the median of their training data, because:
- Good patterns appear frequently. Bad hacks tend to be idiosyncratic.
- Human feedback pushes toward quality. RLHF and instruction tuning reward idiomatic code.
- Tests provide ground truth. Execution turns “looks right” into “is right.”
- Models generalize over intent, not surface form.
2.3 Where the concern is fully valid
| Failure mode | Why it happens |
|---|---|
| Security vulnerabilities | Insecure patterns are widespread in real code |
| Outdated idioms | The corpus is a time capsule |
| Subtle anti-patterns | Frequency ≠ correctness |
| Hallucinated APIs | Blending similar code into nonexistent calls |
| Context-blindness | The right code for a prototype is wrong for a bank |
The real risk: AI writes plausible, conventional, slightly-wrong code that passes casual review and fails in production. This is more dangerous than obvious garbage because it is harder to catch.
Part 3: The Recursive Loop Problem
3.1 Model collapse
When AI trains on AI-generated data, it loses the tails of the distribution. Each generation narrows further. Errors compound.
This is model collapse, and it is already measurable in code.
3.2 The evidence
| Metric | Finding |
|---|---|
| GitHub submissions | 1 billion in 2025; on track for 14 billion in 2026 |
| AI code quality | 1.7x more major issues; 2.74x higher security vulnerability rate |
| Secret leaks | 3.2% rate in AI commits vs. 1.5% baseline |
| Open-source response | Zig banned AI contributions; Homebrew and Matplotlib maintainers drowning in AI noise |
3.3 The feedback loop
Models trained on GitHub code now write GitHub code that goes back into GitHub. The recursion is real.
Part 4: The Financial Pressure Not to Solve Model Collapse
The claim that “there is financial pressure not to solve this properly” sounds conspiratorial, but it is actually just ordinary incentive analysis. No one has to be malicious. The pressure arises from the structure of the situation.
4.1 The problem being ignored
Fixing model collapse properly requires:
- Data provenance tracking — knowing which data is human-written vs. AI-generated
- Rigorous curation — filtering, weighting, and validating training data
- Verification infrastructure — tests, formal methods, human review at scale
- Clean reference corpora — maintaining islands of verified, human-authored code
- Slower iteration — accepting that “generate faster” is not the same as “improve faster”
All of these are expensive. None of them produce visible short-term gains. And all of them can be deferred.
4.2 Why companies defer the fix
The cost is immediate; the benefit is distant.
Fixing data quality costs money now. The benefit — avoiding degradation — appears years later, if at all, and is invisible when it works. This is classic temporal discounting. A dollar spent today on data curation is a dollar not spent on capability, marketing, or compute.
The degradation is gradual and hard to attribute.
Model collapse does not announce itself. Outputs become slightly more generic. Edge cases are handled slightly worse. Confidence remains high even as accuracy declines. The model still “looks fine” on benchmarks. This makes the problem easy to deny and hard to prove.
Competition punishes caution.
If Company A spends heavily on data curation and Company B does not, Company B ships faster and cheaper. In a competitive market, the cautious player loses market share before the reckless player hits the wall. This is a race to the bottom.
The cost is externalized.
The companies generating AI code do not bear the full cost of the degradation. That cost falls on open-source maintainers, downstream developers, future model trainers, and users. This is a negative externality — the producer captures the benefit; society absorbs the cost.
Detection is an arms race.
AI detectors are unreliable. Watermarking is easily stripped. Provenance standards are not universally adopted. The volume of data makes manual review impossible. Even well-intentioned actors cannot easily solve the problem — which gives cover to bad-faith actors.
The “someone else will fix it” assumption.
There is a widespread belief that a future breakthrough will solve data quality automatically, or that regulation will eventually force standards, or that the problem is overstated. This is diffusion of responsibility. Everyone assumes someone else is handling it. No one is.
4.3 The structural analogy
This pattern is not unique to AI. It appears wherever:
| Condition | Example |
|---|---|
| Cost is immediate, benefit is delayed | Climate change, infrastructure maintenance |
| Degradation is gradual | Antibiotic resistance, soil depletion |
| Cost is externalized | Pollution, data pollution |
| Competition punishes caution | Financial deregulation, labor outsourcing |
| Detection is hard | Misinformation, tax evasion |
In each case, the rational behavior of individual actors produces a collectively irrational outcome. The problem is not that people are stupid or evil. The problem is that the incentive structure rewards deferral.
4.4 Why “hope it takes long enough” is the actual strategy
When a problem is expensive to fix, invisible in the short term, a competitive disadvantage to address, externally borne, and hard to measure, the rational strategy is to defer and hope:
- Ship now. Capture the benefit of AI-generated code.
- Ignore the degradation. It will not show up this quarter.
- Blame external factors. If quality declines, attribute it to complexity, scaling, or user error.
- Wait for someone else. A competitor, a regulator, or a research lab will solve it.
- If it becomes undeniable, retrofit. By then, the cost is someone else’s problem.
This is not a conspiracy. It is what happens when no single actor has both the incentive and the ability to fix a collective problem.
4.5 What would actually change the incentives
| Lever | How it works |
|---|---|
| Regulation | Mandate data provenance or quality standards |
| Liability | Make producers legally responsible for downstream harm |
| Market pressure | Customers pay premiums for verified-clean code |
| Standards | Industry-wide provenance protocols |
| Measurement | Metrics that make degradation visible and comparable |
| Competition on quality | Vendors differentiate on data hygiene, not just speed |
None of these are easy. All of them require collective action. And collective action on externalities is historically slow, uneven, and often triggered only after a visible disaster.
The result: A collective problem that everyone can see coming, and no one has the incentive to prevent. The question is not whether we can fix it. The question is what it will take to make fixing it the rational choice.
Part 5: Why “Verify Harder” Doesn’t Work
5.1 The structural failures
If generation is cheap and flawed, and verification is expensive and imperfect, then:
- Volume wins. Generation outpaces review. Review becomes rubber-stamping.
- Verification degrades. If AI reviews AI, you have added another loop layer.
- You can’t verify what you don’t understand. For novel or domain-specific logic, verification becomes theater.
- The loop is self-referential. Generate → verify → regenerate still samples from a shrinking space.
Verification is a dam, not a source. It slows the flood. It does not create clean water.
5.2 What actually helps
- Ground truth that isn’t text — tests, formal verification, real-world outcomes, physical constraints
- Genuinely new information — new domains, regulations, hardware, mathematical results
- Architectural constraints — strong type systems, DSLs, specs that shrink the acceptable space
- Accepting the loop and managing its rate — islands of human-verified code; treating AI output as untrusted input
5.3 The bifurcation
| Domain type | Outcome |
|---|---|
| Verifiable (compilers, crypto, protocols) | Loop is survivable; ground truth exists |
| Non-verifiable (most business software) | Loop degrades; human judgment becomes scarce and expensive |
The likely future: A small, expensive, human-verified core of critical software surrounded by a large, cheap, slowly-degrading periphery.
Part 6: The Economic Counterargument
6.1 Jevons paradox
When a resource becomes more efficient to use, total consumption tends to rise, not fall.
Historical precedent in software:
- Assembly → high-level languages: more programmers, not fewer
- Structured programming → OOP: more systems, not fewer
- On-prem → cloud: more software, not less
6.2 Application to AI
When the cost of building falls, demand rises faster than cost falls.
The systems people dared not build become worth trying:
- Internal tools for small teams
- One-off software for non-software domains
- Legacy modernization
- The long tail of problems that were always possible but never economic
6.3 The caveats
| Caveat | Explanation |
|---|---|
| Different people | The expansion may not employ the same programmers |
| Bottleneck shifts | The constraint becomes judgment, not coding |
| Reasons for avoidance remain | Risk, ambiguity, politics, maintenance — AI doesn’t lower these |
| Cost shifts, not disappears | Inference, verification, iteration, and maintenance costs remain |
The honest claim: AI makes software faster to draft, at a cost that varies wildly by context. Whether that translates to cheaper depends on everything downstream.
Part 7: The Programmer’s New Role
7.1 The systems analyst returns
Programmers do not need to become domain experts. If they did, domain experts would be redundant.
The future role resembles an upgraded systems analyst:
- Not a domain expert, but fluent enough to ask the right questions
- Not a decision-maker, but able to help domain experts see what is possible
- The bridge between “what the business wants” and “what AI can build”
7.2 Why this matters
- The coding part is no longer the bottleneck. Producing the clear spec is.
- Ambiguity is the enemy of AI. A vague requirement produces plausible-but-wrong code faster than any human.
- Domain experts are the source of truth, but they speak in complaints and frustrations, not specs.
7.3 The evolving skill set
| Old programmer | New programmer |
|---|---|
| Writes code for a living | Directs AI, verifies, integrates, owns outcomes |
| Deep in one stack | Deep in one domain, fluent across stacks |
| Employed by software companies | Embedded in every industry |
| Valued for coding skill | Valued for judgment, taste, and accountability |
| Builds systems that were justified | Builds systems that were previously unjustifiable |
Part 8: The “Toy Blocks” Future
8.1 The vision
Vendors use AI to mass-produce verified components. Programmers assemble them like toy blocks.
8.2 Why it could work
- AI is strongest at bounded, verifiable work like writing well-tested functions
- Verification scales better at the component level
- Vendors have strong incentives to build component libraries as moats
8.3 Where it runs into friction
| Friction | Explanation |
|---|---|
| Interface problem | Blocks only compose if interfaces match; history is littered with failed component models |
| Verification doesn’t compose | 50 verified modules composed is an unverified system |
| Combinatorial explosion | With 100,000 modules, knowing which combination solves your problem is its own hard problem |
| Glue code | Often 60–80% of the work, and the hardest part to componentize |
| Vendor lock-in | “Verified components” can become “verified components, but only in our walled garden” |
8.4 The deeper realization
This is not fundamentally new. Programmers already call functions and libraries as modules, pass data between them, and handle mismatches.
What changes: The size and trustworthiness of the modules.
What doesn’t change: The need to select, integrate, and handle the seams.
The programmer becomes a selector and integrator more than a builder — which is what senior engineers have always been paid for.
Part 9: The Physical Limits
9.1 The efficiency gap
| Metric | Human brain | AI data center |
|---|---|---|
| Power draw | ~20 watts | ~1 MW (per large task) |
| Ops per second | ~10^15 | Comparable per cluster, but far less efficient |
| Per-operation cost | ~10^-14 joules | 10^-9 to 10^-8 joules |
| Efficiency ratio | — | AI is 10^5 to 10^6 times less efficient |
The entire human population represents ~160 GW of always-on biological cognition. Today’s largest AI clusters are still under 1 GW.
9.2 The demand explosion
- Agentic AI uses 3,000x more tokens than a one-shot query
- Uber reportedly burned its annual AI budget in four months
- Cheap tokens lead to massively more token consumption — Jevons paradox in real time
9.3 The infrastructure lag
- Power, grid connections, transformers, and land are hard constraints
- Data centers are moving 175km from cities to find power (vs. 46km historically)
- OpenAI has reportedly slowed frontier model development because training costs are rising faster than capability gains
The honest answer: AI demand is exploding faster than infrastructure can scale. The question is not “will AI match the human swarm.” It is whether the physical world can keep pace with the digital ambition.
Part 10: The Ecological Risk
10.1 The evolutionary logic
Finite resources force specialization. Specialization creates vulnerability. A diverse population survives shocks that kill any individual.
10.2 AI’s vulnerabilities
| Biological threat | AI equivalent |
|---|---|
| Virus exploiting shared receptor | Prompt injection working on many models |
| Antibiotic-resistant bacteria | Attack surviving current defenses |
| Pandemic in dense population | Vulnerability in a widely-used model |
| Genetic uniformity in crops | All models trained on same data |
| Monoculture collapse | One dominant architecture failing |
10.3 The diversity problem
Forces pushing toward diversity:
- Open-source models
- Edge/local models
- Domain-specific models
- National/regional models
Forces pushing toward homogeneity:
- Scale economics (only a few can afford frontier training)
- Benchmark convergence
- Data convergence
- Deployment convergence
- Model collapse
The honest read: Diversity is real at the periphery, but collapsing at the frontier. The frontier is a monoculture by design. If a vulnerability hits that layer, it hits almost everything important.
10.4 What resilience would require
- Architectural diversity (not just transformers)
- Data diversity (not just web-scraped corpora)
- Deployment diversity (not just a few cloud providers)
- Governance diversity (not just a few regulatory regimes)
- Failure diversity (systems that fail in different ways)
Today, the industry is moving in the opposite direction: consolidation, standardization, and scale. That is efficient. It is also fragile.
Part 11: The Swarm
11.1 Emergence
Individual ants are simple. As a swarm, they build complex mazes. Complex intelligence can emerge from simple agents following simple rules.
Humanity has already done this:
- The scientific method — a distributed error-correction system
- Language, writing, mathematics — external cognitive scaffolding
- Open-source software — millions of people, no central planner
11.2 The real comparison
If the swarm is the real unit of intelligence, the question is not “can an individual human out-think AI?” It is “can the human swarm adapt faster than the AI swarm?”
| Human swarm | AI swarm | |
|---|---|---|
| Speed | Generational (slow) | Iterative (fast) |
| Diversity | Enormous | Narrow |
| Error correction | Distributed, robust | Centralized, brittle |
| Motivation | Survival, meaning, curiosity | Optimized objectives |
| Failure mode | Groupthink, polarization | Model collapse, monoculture |
11.3 The lesson
Diversity is what makes a swarm resilient to novel threats. The human swarm has it in abundance.
But it is not automatic. A swarm that stops exploring — that delegates all thinking to one fast, uniform, centralized intelligence — is no longer a swarm. It is a monoculture.
The wisdom of the swarm will surprise us. But only if we let it stay a swarm.
Conclusion
The direct answers
Will AI replace programmers?
No. But “programmer” will not mean what it means today.
Is AI learning bad code?
Yes — but it writes better than the median of its training data, while still reproducing widespread mistakes in security, idioms, and context.
Will the recursive loop degrade everything?
For non-verifiable domains, yes. For verifiable domains, ground truth provides an escape.
Why isn’t this being fixed?
Because the cost is immediate, the benefit is distant, the damage is gradual, the cost is externalized, and competition punishes caution. It is not a conspiracy; it is an incentive structure.
Can verification save us?
No. It is a dam, not a source. It slows the flood but does not create clean water.
Will more systems be built?
Yes — Jevons paradox suggests demand rises faster than cost falls. But the expansion may not employ the same people.
What does the programmer become?
A selector, integrator, and systems analyst — someone who translates domain needs into buildable specs and verifies outcomes.
What are the physical limits?
Severe. AI is 10^5 to 10^6 times less efficient than biology per operation, and demand is exploding faster than infrastructure can scale.
Can AI survive a “disease”?
At the periphery, yes. At the core, no — the core is a monoculture by design.
The shape of the future
| Layer | What happens |
|---|---|
| Component layer | AI generates abundant, well-tested modules |
| Interface layer | Standards and contracts become more important, not less |
| Composition layer | The programmer’s core job: choosing, connecting, orchestrating |
| Verification layer | Shifts from “is this function right?” to “is this composition right?” |
| Glue layer | Still exists, still significant, still the unglamorous heart of the work |
The final word
Humans have survived every previous test because we kept choosing to. Not because we were strong or smart enough, but because enough of us refused to stop being the kind of creature that tries.
That is not guaranteed to continue. It is a choice each generation makes, mostly without realizing it is making one.
This generation is making that choice right now — in how it uses, or does not use, the tool it built. And it will not know for decades whether it chose well.
The transition will be brutal for some, expansive for others, and uneven across both. But the direction is not extinction. It is transformation under pressure — the same test every generation before us faced, with higher stakes and less time.
The swarm will surprise us. But only if we let it stay a swarm.
This article distills a long conversation about AI, programming, model collapse, economics, ecology, and emergence. The questions were better than the answers.