You’re right that agile software development is not necessarily radically different from other software development approaches. The “radical” idea in agile is to Do Things That Work, which usually means adapting the process to make it work better, and to listen to the people doing the work (“people over processes”).
Under the agile umbrella, we have specific process methodologies such as Scrum. Scrum is definitely iterative and incremental, and suggests feedback loops of various levels. In a typical instantiation of a Scrum process, you’d see feedback loops like the following:
- TDD red/green/refactor cycles
In Scrum, the main feedback loop is the sprint. Each sprint has a defined duration and defined scope. At the beginning, we determine some tasks that should be Done during the sprint. At the end, we see what was Done. Did some things fail to get Done? During the retrospective, there’s opportunity to understand why and adapt as necessary for the next sprint. Then we repeat. Since we might adapt between each Sprint – and in particular, the Product Owner may change priorities based on the experiences – it is not appropriate to plan in advance multiple sprints. One step at a time.
You are entirely correct that individual tasks (backlog items) in Scrum are not handled in an iterative manner. A task is either Done or it’s not. This is considered important in order to manage clear expectations with the Product Owner, but also to demonstrate clear progress to the PO.
The entire point of Scrum is that many projects do not make clear progress because a lot of tasks are only mostly done. This delays value, and makes it difficult to close feedback loops. The idea is that with Scrum, there will be something Done each Sprint, thus delivering clear and predictable value to the PO. And since the PO can re-prioritize between Sprints, this maximizes the value delivered across the project duration.
When you say “incremental models are defined by a comprehensive upfront documentation” then this is wrong. There are processes without a focus on documentation that are clearly incremental, such as Scrum. It’s also perfectly OK to prepare lots of upfront documentation in an agile project, if that documentation has value (though in many cases, a Spike is more useful to find a good design than to do lots of upfront analysis). This circles back to my original point: agile is about finding a process that works. In this viewpoint, being agile is about values that inform how you select and implement processes, and are not an inherent attribute of processes themselves. (Indeed, most Scrum instantiations are probably not agile in any meaningful way…)
Iterative/incremental processes were pioneered in environments with very high integrity requirements and thus a focus on documentation and testing. In particular, some incremental processes involve an entire Waterfall-like process per increment, including analysis and design phases. This isn’t wrong, but this also isn’t the defining feature of iterative/incremental. The defining feature is that functionality of the project is split across two or more increments, thus de-risking the project and allowing for early feedback to be integrated. This is broad enough to cover both defense projects that start with a 1000-page requirements document and a three-person web consultancy doing Scrumban.