As artificial intelligence continues to reshape the software development landscape, I recently conducted an intensive experiment to test the true capabilities of AI coding assistants. Over five consecutive days, spending 2-3 hours daily, I collaborated with an AI to build a complete application from scratch – including database design, API server implementation, Single Page Application (SPA) client, data modeling, and more.
The Experiment Setup
The goal was ambitious: develop a complex product without writing a single line of code myself, instead relying entirely on AI assistance. The project encompassed all aspects of modern application development, including:
- Database architecture
- API server development
- SPA client implementation
- Comprehensive data modeling
- Code refactoring
- Implementation of strict typing
- Adherence to coding conventions
Surprising Results
After five days, the results were both enlightening and unexpected. The AI generated over 150 commits and more than 100,000 lines of code, creating a functioning complex product. However, the experience revealed clear strengths and limitations of current AI coding assistants.
Where AI Excels
- Typing Implementation: The AI demonstrated remarkable proficiency in implementing and maintaining type systems, showing a deep understanding of type safety and structure.
- Convention Adherence: It consistently followed established coding conventions, maintaining clean and standardized code throughout the project.
- Focused Refactoring: When given specific, targeted refactoring tasks, the AI executed them efficiently and accurately.
Where AI Falls Short
- Big Picture Architecture: Perhaps unsurprisingly, the AI struggled with high-level architectural decisions and system design. These tasks still require human expertise and strategic thinking.
- Cross-Domain Refactoring: The AI showed limitations when handling refactoring tasks that spanned multiple domains, particularly when updating data models. This highlights a common challenge in software development: the difficulty of managing dispersed business logic.
- Innovation: While excellent at implementing and maintaining existing patterns, the AI didn’t demonstrate ability to generate novel solutions or architectural approaches.
Looking Ahead
This experiment is just the beginning. The next phase will focus on testing the AI’s capabilities in common post-development scenarios:
- Managing version upgrades
- Handling breaking changes
- Executing large-scale refactors
- Implementing data model upgrades
Key Takeaways
This experiment demonstrates that AI coding assistants are becoming increasingly powerful tools in a developer’s arsenal. However, they’re best viewed as highly capable assistants rather than replacements for human developers. Their strength lies in accelerating implementation of well-defined patterns and maintaining consistency, while human developers remain essential for strategic decisions and innovative solutions.
I’m documenting this ongoing experiment through daily streams on YouTube, sharing real-time insights into the capabilities and limitations of AI in software development.
Stay tuned for updates as we explore more complex scenarios and push the boundaries of AI-assisted development.
Written by Claude Sonnet 3.5 from a linked in post I authored.