The Future of Project Management

PM is evolving

Project management (PM) and the way software is developed has radically changed over the last few decades.

Agile has become the norm, with a mix of “Agile” and “lean” practices being implemented at the enterprise level. Why have these buzzwords become so popular and why have current methodologies proven to be so successful? Software development has been going on for quite some time, so what changed?

This article will answer those questions by explaining the history of software development and what it means for the future of project management.

“Waterfall”

At first glance, one might say that we are more Agile now and that waterfall is a thing of the past. However when you work with any large organization, you see that their software development plans, roadmaps, and what managers are showing their bosses looks a lot like the waterfall model. As far as I can tell, this is how most projects have been planned for decades: sequentially. So why does everyone call this “waterfall?”

In a 1970 paper that Winston Royce wrote while working for Lockheed, he described the waterfall model almost as a caricature of the sequential style of planning and managing a project. Yet the idea stuck around and the name waterfall stuck too. This sequential style of PM has been around for a while and seems to be the most simple way to describe to someone a software development plan. It’s just easy to say that A leads to B leads to C.

Lean

After World War II, America was riding an economic boom and was the greatest manufacturing power in the world. American managers were comfortable and quite successful at planning and managing manufacturing projects that resembled a waterfall model and embedded that process into America’s management culture. Across the Pacific, Japan was developing a new style of management. In the resource-constrained environment of post WWII Japan, Toyota developed the foundation of what we now call lean management. Toyota’s engineers popularized the idea of Kaizen, or continuous improvement. They also came up with a brilliant way of managing tasks and resources through Kanban. This was while the US military industrial complex was growing into a behemoth and hammering waterfall-style management thinking into the brains of leaders. West Germany also joined in on the manufacturing boom and developed what is now referred to as the V-shaped model (basically an extension of the waterfall model that allows for good old German discipline with lots of testing). It would take until the 1980s for these management philosophies to collide through a long list of corporate mergers and software project implementations.

Iterative (aka Agile)

Despite folks saying that Waterfall is the way that projects have always been done, iterative processes have actually been around for a long time too. Still people continue to act like the principles from Agile are something brand new that we need to learn and implement, even though Agile-like ideas have always been common among high performing teams. As Henry Ford was managing and growing operations at his Highland Park plant, he was constantly refining not only designs for new Ford models but also continuously improving the speed and quality of the assembly line. He definitely used an iterative approach. These iterations were displayed through the nearly continuous improvement in the company’s manufacturing line throughout its history.

Another example of iterative management that is often not recognized is in the construction industry. Many construction teams will meet every morning to assign daily tasks for a project. Sure, the foreman or PM may be telling higher level management that they are following a GANTT chart and the critical path method to accomplish the project on time. However, in reality small construction teams follow an iterative process where they build as much as they can one day, a supervisor makes note of the progress, and then they assign tasks the following day. This cycle repeats until the construction project is completed. The Scrum approach to software development - where software engineers get to pick their tasks or user stories - is similar to how equipment operators or tradesmen pick the parts of the projects which are best suited to their expertise. This iterative style of management is not new, however its formalization into higher levels of management is new.

Enterprise

We have witnessed an explosion in management science since the 1990s. A lot of the advances in PM are due to the US Department of Defense formalizing management and improvement processes. The personal and team software processes (PSP and TSP) along with capability maturity model integration (CMMI) were some of the first few methodologies to get formalized and brought into the private sector. IBM’s formalization of processes both at the team and enterprise level eventually evolved into what is called Disciplined Agile today. PMI’s acquisition of Discipline Agile in 2019 possibly marks the complete convergence (and commercialization by consultants) of traditional and iterative methodologies. Scaled Agile’s success with its prolific Scaled Agile Framework (SAFe) in corporate America further shows that the rate of change in the search for effective PM methodologies has plateaued. The 2022 State of Agile Report showed that 53% of respondents were using SAFe at the enterprise level and about 40% were using a variation of Scrum adapted for the enterprise. The market for enterprise agile coaching seems mature at this point, but there is one more trend that has yet to stabilize.

Automation

The practice of merging work from developers at a faster and faster pace laid the groundwork for the development of continuous integration, Git and Extreme Programming (XP) in the 1990s. After a couple more decades of internet growth and an explosion in the demand for software, development and operations (DEVOPS) converged along with continuous integration and continuous delivery (CI/CD). Since then, we have continued to see more and more sophisticated layers of automation. Now it is common for professionals in all kinds of industries to say they are using artificial intelligence and machine learning (AI/ML) as tools. So what is next? How do we manage projects now and as we approach artificial general intelligence (AGI)?

More Automation

IBM developed the process of Cleanroom Software Engineering (CSE) in the 1980s. This highly mathematical and formal methodology produces extremely reliable software with very few defects. However, the time and cost required to use this methodology is way too high for almost all consumer software. So the rise of Agile methodologies left CSE behind in the mainstream web application development part of the “tech” industry. Yet similar methodologies continue to be used in the aerospace industry, healthcare/pharmaceuticals, the food industry and other areas that demand high quality engineering. As AI/ML is applied to solve more problems, it is likely that we will see a resurgence in processes like CSE for more uses, probably combined with some automated A/B testing. We are getting to the point where no-code solutions and programming copilots are giving software developers the leverage to act as product managers and vice versa. These roles are changing with the advancement of the automated copilots as well. Eventually, an automated product manager that can code will build our software and test it in the blink of an eye.

Where we will continue to struggle for years to come will be managing projects at the corporate level. Quickly developing web applications is one thing. However, gathering the required data and communicating it effectively between different roles, organizations, or even companies is far more time consuming. Difficulty communicating will always be the most challenging thing in PM. We have a lot of work to do before AI can solve a complex problem that requires coordination among distinct organizations.

For more information on Software PM, see this timeline.