Productivity in Drupal’s Development (part 1 of n)

Some of you may recall that a couple of years ago (wow, that long?) I wrote an article on this blog entitled “Artificial Intelligence and Repetitive Tasks in a CMS” describing some of the biggest causes of grief I had when using Drupal (and other online presence/communication creators). I felt it was time for another one of these posts in the same style to critique and give suggestions again. This time, not on Drupal as software, but on the Drupal development process. Also note, I love Drupal and make my living from it, all of this is meant to be constructive.

“Drupal” isn’t Drupal.

If you’ve used Drupal for any length of time you know this is true — namely, the “Drupal” you talk about, use and recommend isn’t the Drupal people go to http://drupal.org and download on their server. I’ve been there before when I made the mistake of touting how wonderful Drupal was, and seen the inevitable conclusion by the person I told this to.

They downloaded the Drupal archive, extracted on their web server, set it up, and came back to me. They thought it was crap. Heck, the out of box experience for Drupal *IS* horrible. Of course I explained that “Oh, no, you want to disable overlay, install admin_menu, install views, grab a theme from here,….” My friend, heard this:

Called "the four subspaces" on Wikipedia commons, I don't want to know what it means.

See, the Drupal that people love and enjoy isn’t easy to get. Even though we all tout Drupal as everything but a sugar substitute, getting your hands on that “Drupal” joy is nearly impossible without having done it before. Oh, and, profiles are wonderful, but not all that useful because of how the rest of the development cycle works:

Month 0: Drupal is released
Month 3: Most modules become stable
Month 6-eternity: Views, Chaos Tools and Panels become stable
Month 10: Oh, Drupal has BlahShiney API that is AWESOME!
Month 12: A useful profile is actually released in a beta
Month 14: Useful profile’s author forgets about it due to work/classes.
Month 18: Drupal <next version> Beta/RC is out!
Month 18: “You should wait for/use the beta of Drupal , which has KitchenSink API by instead of using ”
Month 20: Useful profiles for <current version> actually become stable.
Month 22-24: Drupal <next version> released, see Month 0.

An editorial design on toxic water by Yesenia Rivero
I think, bearing in mind a few objections here and there and the odd “but we totally fixed this for real this time, again, honest” comment, everyone can relate to this timeline. There is a simple solution for this; but it would need Dries to backtrack his State of Drupal speech and give up on rapid core overhauls.

You see, Mozilla has had this problem before. Mozilla browsers are simply too big for any one person to handle entirely. Because of that, they are split into a series of modular components with a manager (or maybe several co-managers) overseeing patches and the development cycle for that component. These people are called “drivers” and it’s their job to keep development moving, organize with each other and the higher overall project leaders on how to keep their teams moving in the right direction, and to all freeze their codebases in preparation for major launch events.

Some of that terminology probably feels familiar to those who use Drupal. Now, let’s think of how we could apply this logic to how Drupal develops.

Drupal [Distribution] becomes a stable distribution that has Drupal Core and the top 10 or so most common modules that the community needs to actually launch a good Drupal site. Drupal is a launching pad to great web applications (yeah, yeah; it can be a framework, but let’s focus on the idea of a launching pad distribution that is Drupal’s bread and butter). Drupal core would be split into the required components, with all the optional components separated out, along with these 10 or so critical contrib modules to become this base distribution.

Luxurious blue screwdriver from Amazon

Each of these components has a driver, someone who looks after their component, as well as co-drivers who might only be focused on improvements for a certain version or adding new features. This group of people talk regularly to coordinate efforts and are steered by the distribution’s (Drupal’s) driver (Dries or someone else who manages a specific release).

Minor changes and improvements can be signed off on completely by the component driver without going up the food-chain. Major changes or features get pushed to the distribution driver, both for a “yeah, this is a good idea” and a “yeah, this is ready to commit.” This lets key people stay focused on their component and improving it, and keeps being a Drupal release coordinator from turning into a full-time job/life (btw, Angie, you’re awesome to have done that and making it look easy).

This distribution driver sets the timeline for a release, and all the componet drivers also enforce that timeline for their branch. As critical points in the timeline draw close, they will switch new features to a new version, and only focus on improvements for the current (we’re familiar with this now in core).

When version freezes occur, they occur across all components and the result is a coordinated release of Drupal that it actually “Drupal,” the cool experience we want to share and not “Drupal” the “oh, this is crappy” at-first-look framework. Effectively, this means having active “maintainers” with commit authority across all parts of Drupal core. This removes strain from distributions drivers (maintainers) and allows people to be specialized. Also, synchronicity between Drupal core and the most critical modules so Drupal the framework and Drupal the handy distribution people depend on can be both generally expected at predictable times.

This organized and controlled distribution could be considered, to steal from operating systems, ring 0. Everything in this is critical to Drupal Distribution. The next most popular modules in contrib, as well as sponsored projects could form the basis for components in ring 1; less controlled but still given some attention to have ready for releases and security. Ring 2 would be what we all consider the jungle of contrib as it stands now, with only perhaps some reputation changes that folks are already talking about anyways, and I won’t get into.

Feedback and conversation is very welcome, but before you comment please note: I know parts of this have been discussed, and parts of it may even be attempted as due course; so please don’t comment with, “but we’re already doing x part of this, so y and z are invalid.” Also, any assumptions that I know what happens on all of groups.drupal.org or in back-channel developer discussions should be reconsidered before commenting.

P.S.: Remember Mozilla? They became so caught up in looking busy with new version numbers that they started to release minor fixes and improvements as major releases (6 and 7 instead of 5.1, 5.2). Let’s avoid that, shall we?

6 thoughts on “Productivity in Drupal’s Development (part 1 of n)

  1. “There is a simple solution for this; but it would need Dries to backtrack his State of Drupal speech and give up on rapid core overhauls.”

    That’s already been backtracked, we’re looking at a 2-3 year release cycle for Drupal 8 now.

    http://buytaert.net/how-i-think-about-drupal-release-date-planning

    To have components separated out and managed independently, you’d need to actually have separate components, which we don’t currently. So before anything like that could even be considered, there is a tonne of clean up to do, i.e. http://drupal.org/node/1224666

    1. Thanks for taking the time to comment catch! Glad to hear the 12-18 month cycles are out the window — it was perhaps the worst idea I’d ever heard for Drupal in it’s current state (see my Drupal lifespan “figure” above on why). That said, my overarching statement through this blog post is the “Drupal” we hype isn’t the Drupal people get their hands on, and that needs to change. My development scenario is just one possible path to making the two Drupal’s mesh in a nicer and better way, but is far from the only solution.

      http://drupal.org/node/1224666 sounds like it could be a great step in making a transition to a development process like I’m suggesting easier; however, I do worry it will turn the mentality of Drupal’s development cycle to one more focused on Drupal as a framework (which suddenly means Drupal is competing with many older and, in some cases better frameworks) rather than a web application/content management system (which doesn’t have nearly the choice, at least in the high end, in free alternatives).

  2. Even though you said not to, I’m still going to say that you’re bringing this up about 6-8 months too late. Dries put out a public call for suggestions on how the development process should be changed for D8 at that time and those suggestions were formed into what we have now.

    Secondly the structure that you describe would have made it difficult to do at least three of the current Core Initiatives (HTML5, i18n, & Configuration Management) since these all span every subsystem in Core.

    1. Thanks for replying dalin! Yeah, I was in over my head with development projects then and totally missed “the call”. Thankfully, Drupal tends to reorganize things very frequently since it has a habit of them not really working out, so I’m sure it isn’t too late for these ideas to come into consideration. Not to mention the idea I really want to convey in this article is the different between what “Drupal” is in use, and in our hype and the Drupal tarball folks download. (This isn’t just a newcomer either, everyone has that list of modules that have to become stable before they can upgrade to Drupal 7/8).

      As to core initiatives, those end up being placed more on the shoulders of distribution drivers — with the component drivers also watching for quality and giving input, of course. A change in authority does not mean we can’t break it and bring everyone together on initiatives. The Gecko rendering engine @ Mozilla often has to do this with major changes.

  3. I think dalin’s second point is the important one.Every core initiative, and many patches, span several subsystems in core – it’s impossible not to when everything is so closely coupled. So before anything like this could even be considered, it needs to be made possible by removing inter-dependencies in the first place.

  4. “There is a simple solution for this; but it would need Dries to backtrack his State of Drupal speech and give up on rapid core overhauls.”

    I see that a good point. But IMO Drupal would not be healthy without that, it’s the nature of developer driven community which need challenged. And the world is accelerating at incredible rate, 3 years is very long and almost unpredictable, only moderned technology survived.

    To my interpretion, You’re pointing to frustration of drastic changes. I ageed that we’re not yet have enough tools to deals with them, but how about if we set the goal like ‘Make Drupal drastic changes manageable to developers and everyone’?

    I see those configuraion initiative is the first step.

    Richard

Leave a Reply