TL;DR
- AI-generated code works, but it often lacks structure and clarity.
- Asking for test-driven development (TDD) improves readability, maintainability and modularity.
- Even AI is cautious about refactoring without tests, just like a human developer.
- AI can produce functional code. Applying TDD makes it significantly better.
Here is the detail.
The experiment: can AI write good code?
Curious about AI's ability to generate quality code, I asked ChatGPT to build a PHP plugin, a language and stack I am not especially familiar with. The goal was to see whether AI-generated code was viable.
Initial observations:
- The code did not work first time.
- Debugging took longer than expected.
- The code was hard to understand.
- It was one large, monolithic class with bloated methods.
That made me wonder how AI-generated code would fare in a structured exercise such as a simple kata.
The questions:
- What would the AI-generated code look like?
- Would it resemble code developed with TDD?
- Would it show the problems we see from developers new to TDD: functional, but riddled with code smells?
In coding dojos, when participants solve a problem without TDD the result usually works but is not clean. Even in simple katas you see duplication, smells and poor structure.
So I ran a quick experiment with a simple exercise.
Step 1: AI-generated code without TDD
I prompted ChatGPT:
"We're running a coding dojo in C#, focusing on the Prime Factors kata. Write a static method generate(int N) that returns a list of prime factors of N in ascending order."
ChatGPT's first attempt:
What I found:
- The function works as intended.
- There are no tests.
- There is unnecessary duplication.
- The loops are more complex than they need to be.
The function solves the problem, but it is not test-driven, clean or maintainable.
Step 2: applying TDD to AI-generated code
Next I asked:
"Write the code, this time using a TDD approach."
This time the approach was different.
The AI's TDD approach:
- Took small, incremental steps.
- Wrote failing tests first.
- Implemented the minimum to pass each test.
- Was noticeably hesitant to refactor without tests.
The TDD-driven final code:
Improvements:
- Tests now drive the implementation.
- Better structure, less complexity.
- Some duplication in the number handling remains.
At first glance the AI appeared to follow TDD. That raised a critical question:
Did the AI actually apply TDD principles, or did it execute predefined steps without understanding their purpose?
It wrote tests first and iterated, but it did not fully embrace the incremental nature of TDD. It skipped the essential baby steps, the small guiding tests that evolve a solution one step at a time.
Instead of progressing gradually it jumped ahead, testing larger cases (30, 56, 97 and 101) before validating foundational ones like 3, 6 or 9. A true test-driven approach takes small, controlled iterations, so each step builds on the last while keeping clarity and correctness.
Is AI truly applying test-driven development?
On the surface the AI follows the TDD process: write tests, implement, iterate. But does it understand TDD, or is it running a script?
The observations that raise doubt:
- Delayed refactoring. The AI only refactored after completing the implementation, rather than continuously improving the code with each test cycle as TDD encourages.
- No clean-code principles by default. It did not apply craftsmanship principles such as meaningful naming, single responsibility or reducing duplication unless explicitly told to.
- Skipped steps. It jumped to complex solutions too quickly. Instead of testing 2, then 3, then 9 before 30, it jumped from 4 to 40, violating the Transformation Priority Premise.
- Step size. It iterates, but it does not always take the smallest possible step, and so it misses the learning moments.
The Transformation Priority Premise, as defined by Robert C. Martin, is a structured way to evolve code incrementally. You introduce the simplest possible change before a more complex one, so the code stays clean, adaptable and well tested throughout.
The AI violated this by taking large leaps in logic and skipping the validation steps that drive incremental improvement in real TDD.
Building high-performing teams
The experiment was insightful but also raised concerns. Even with AI, code quality is never guaranteed. AI can reinforce good practice or amplify bad practice. It can help developers build products faster, but at what cost?
Perhaps this is why test-first and quality-first thinking matters more than ever. It is what keeps software adaptable, maintainable and valuable.
Anyone can write code. Heck, even my gran could. Not everyone can write high-quality code that is extensible, maintainable and resilient over time. That is why coding is more than writing code. It is about aligning with the engineering and quality principles of Extreme Programming (XP) and software craftsmanship, so you build products that evolve with the market and keep delivering value to customers.
AI can generate functional code. Would you trust it to build a mission-critical system, like the software in a modern car that receives regular updates from the manufacturer, without human oversight, test validation and quality principles?
The experiment reinforces that AI is a powerful tool, but it currently lacks the judgement, intent and expertise of true software craftsmanship.
As we coach teams on what good looks like, AI may help accelerate the journey to high-quality products. For now, it still takes human expertise to make sure we are crafting scalable, adaptable software that truly serves customers.
See TDD in action, without AI
There is no substitute for hands-on experience. Watch our video demonstration of a test-driven approach to the Prime Factors kata, with no AI involved.
Key takeaways
- TDD is not just about writing tests.
- It is about designing better software.
- It is about learning to take small, safe steps.
We ran the experiment again, this time prompting the AI with the transformation premise and small incremental test steps before the dojo prompt. The result was noticeably closer to what we would write ourselves.
But getting it right in the end does not mean the code was high quality from the start. It only improved with extra guidance. Even minimal principle-based checks are essential for clean, maintainable code.
AI is a powerful tool. How we apply our expertise is what makes the difference.
Your thoughts?
- Have you used AI to help with your coding? If so, how?
- Would you trust AI to generate code from written instructions alone?
- Have you ever hesitated to refactor without tests? Would you trust AI to change code without altering its behaviour?
If you want to strengthen your team's technical practices, talk to us about coaching.