A Culture of Open Development at Adobe
We’ve talked in the past about the thinking behind the Open Development model at Adobe. But how does it work in practice?
Let’s start with a story about a challenge.
No service or system is ever perfect, or finished. It should constantly be adapting and evolving to meet the needs of its users, and how they work. There’s always something to be done.
And, not too long ago, the Adobe I/O team had some specific changes they wanted to make to improve the product for their users.
“The logging subsystem we previously had inside Runtime was tricky for our team to administer, and needed a bit of improvement”, said senior computer scientist Dan McWeeney.
“As an example, a function logging a large number of log lines could only retrieve them via the command line interface in batches of ten. This obviously made for a tedious experience. And the highly distributed nature of the original architecture made it hard to monitor for failures.”
In this case, Adobe’s own Developer Experience (DevX) team was willing to help.
“They kindly offered to help us make this subsystem fully pluggable, so we could deliver log lines to external systems configured by customers,” said McWeeney.
“So instead of seeing logs in batches of ten at a time, the customers could get them in something like Splunk.”
So far, so good. But how does a collaboration between two such teams work? How would the work be split up? What sort of solution would emerge, and when?
The answer to this lies in Adobe’s commitment to an approach known as “Open Development.”
What is Open Development?
When trying to understand Open Development, it can be helpful to think about it as a way of doing things.
In 2017, Adobe Digital Experience group’s principal scientist Bertrand Delacretaz laid out an “Open Development Manifesto”. This manifesto highlighted some of the key elements of a methodology that had defined Adobe’s culture for several years.
The manifesto included statements such as:
- “I’m not afraid of others respectfully challenging my technical decisions and/or my code…”
- “I thrive for very high quality in my code and in our products, and won’t be shy to respectfully express my disagreement if something puts that at risk”
- “I work in an open way, commit early and often so that others can see what I’m doing while that evolves…”
Essentially, it’s a method that prizes collaboration, communication, and openness. Not someone attempting to solve a problem without sharing how they did it, or why they did it that way.
“Open Dev is a concept that’s been knocking around for a while at Adobe,” said Dan.
“We contribute across the company to several Apache projects, and have drawn a lot from their model internally.”
These ideas are familiar to anyone that’s worked in Open Source.
Indeed, they’ve been adopted by many corporate organisations over the years. The term “Inner Source” was coined by Tim O’Reilly in 2000 to describe the way that an organisation might follow Open Source practices, but within its own walls. It’s something that’s been adopted by major companies including HP, Microsoft, Google and IBM.
Open Development shares many characteristics with Inner Source, except — as Bertrand points out:
“Inner Source is more specific, because it’s specifically about [applying open source principles] inside your company, whereas you could say that we also do Open Development when we do Open Source, except that the final product is Open Source.”
So think of it less as the end product, and more the way of getting there.
So how does it work, internally?
“Open Development is a great way for a project to get visibility, recognition and contribution”, said Felix Delval, an architect at Magento Commerce.
“It’s as much a technical adventure as a human one. Generally, you’re coming to a project you don’t yet have a clear understanding about, with the aim of performing an atomic change.
Felix recommends starting off by making sure that the project is well structured and documented and — if not — taking the time to create that documentation and a basic suite of tests. This has the added benefit of helping out the next batch of developers who jump into the project.
“Once that technical side is under control, you begin the human journey of discussion and definition of what the problem is, and how it should be solved. You should never underestimate how problematic it can be to make sure that everyone understands the problem. Most of the time, a group of people working on something will arrive with their own terminologies and meanings for common words, so really understanding those terms and learning the project’s jargon will greatly improve everyone’s chances of success.”
Adobe is a big company, with several teams. But — undeniably — every team has skills, insights, and expertise that could benefit other teams. The key is ensuring they’re aware of that, and empowered to do something about it.
“As a dedicated Dev Ex team, we can focus on the developer experience full-time, so we learn what other Dev Ex teams are doing, and what happens beyond Adobe’s walls”, said Olga Kopylova, observability architect at Adobe.
“As a result, we can bring good insights into what developer experience teams can implement, and we can also help with that implementation. So, one benefit is having cross-cutting areas like observability covered better. And another is having extra hands for the lower price of reviewing pull requests, and helping us out in return here and there.”
So how does this sort of collaboration work?
“Initially, we discuss with a team what they think can be improved in the dev ex area of their product. We try the product ourselves, and come up with a list of potential items to work on.
“As both teams agree on a project we can help them with, we start the collaboration. In this case, I/O Runtime had a dedicated project manager and developer who helps us as we need it. And, thanks to this approach, we managed to build a detailed delivery plan and follow it to production. And our team followed the delivery requirements of the I/O Runtime team to make sure our delivery was aligned to their standards.”
What makes this possible is that the process is transparent. Individuals can see what exists, and what has changed, and there’s clear documentation that indicates the thinking behind the decisions. It’s a culture in which contributors can make “polite” requests to solve problems and amend code, on the understanding that any changes are open to feedback and pushback. Here, Olga breaks down the typical workflow she and the team follow:
- We planned a regular meeting, where Dev Ex engineers and product managers/engineers could align (Dev Ex engineers also met regularly, just themselves). This ensured each engineer was always working as an active project manager and mediator between teams.
- Regular, helpful insights frequently came from these meetings. The Dev Ex team kept a document with a backlog for the project, divided into meaningful, actionable items. Everything is mutually agreed upon (with the PM having the “last word”). Olga stresses everything was discussible, but no decision is forced if the “project team” collectively does not agree to it.
The team relied heavily on Slack to stay connected on everything. “Slack and dedicated people resolve everything, quickly,” she said.
As the Dev Ex and product teams worked closer and closer together, this process became even more effective. Trust grew, with the Dev Ex team getting more access and involvement (which becomes even more important as you move from dev, to stage, to prod).
As Bertrand said in a previous post:
“Open Development fosters a communications style where everybody is aware of what’s going on in a project, even if they’re not participating in all discussions.”
The Runtime logging subsystem was a perfect candidate for an Open Development approach, because of how it was set up.
“It can be hard to work with external teams on ‘rangy’ changes, because they have tendrils into many parts of the system,” said Dan.
In this case, the team could focus on one area of the system, and become experts in it, and therefore provide the required changes. This had dual benefits. Firstly, it created a new expertise where one didn’t exist before. Secondly, it’s excellent fodder for future Open Dev projects.
“As the old adage goes: ‘many eyes make all bugs shallow.’ So new features and adjacent requirements can also be sussed out during this process. The Dev Ex team is already working hard with us on more features in this area of developer ergonomics.”
Learning the ropes
But while Open Development can — in many ways — be a beneficial process, it’s also a “learned” process. Teams won’t necessarily adopt the right mindset overnight, and can require work and patience to reach the point that you feel comfortable with the approach.
Olga admits that it was initially hard to adjust to the collaborative work, and that it took time for the teams to make time for each other, by answering questions in Slack, reviewing pull requests promptly, and scheduling meetings.
“With time, we have built a process that works for us,” she said. “We feel like a part of I/O Runtime now, and we know what to expect.”
There was also the issue of new technologies. Or — more accurately — technologies that were new to one team, but not to another.
“Onboarding might take some time, and involve some silly questions,” Olga said. “But as a Dev Ex team, working with different teams, we should be ready to learn new things every time we start a new project. We actually like to learn things.
“Not saying I loved Ruby, but k8s and Ethos were great things to get additional experience with.”
Indeed, that “learning process” is a crucial reason why many companies like the Inner Source approach, which gives employees space to acclimatize to the collaborative and open approach, rather than immediately launching into the sprawling Open Source ecosystem.
“Open Source would need more commitment in advance, certainly”, said Olga. “If the product were opened to Open Source, but not ready to process pull requests, the community would quickly stop contributing.
“Also, the Open Source community can be overwhelming due to its size, and also the diversity of opinions. Most people contributing to OS projects do it to solve their own issues. This may be a common scenario or a one-off thing, but it all needs to be processed, and processed fairly.
“With the Dev Ex team being under the same umbrella as a product team, it’s easier to cut bureaucracy corners and focus on common goals.”
What skills does Open Development require?
So what does it mean to “do” Open Development? Olga believes it comes down to three key things:
Firstly, the approach encouraged the teams to focus on deliverables. Team members learned to be very clear and open about what needed to be delivered to production, and when tasks needed to be done.
“It’s easy just to do things, research, try, and wait. Instead, we had to be proactive”, said Olga. “We pinged people when we needed a pull request to be reviewed. And we made sure we were goal-oriented and not drowning in discussions.”
The second key aspect came down to communication. Olga said that it was vital that both teams were open to ideas and suggestions, and aware of everyone’s else’s priorities and other work.
“We were new to the I/O Runtime team, and they had their own schedules and tasks”, she said. “We tried to keep a balance between pinging them for help and working on our own, and to have an open and productive discussion rather than just pushing our own ideas. They know their product better.
“I/O Runtime did a great job in this area as well. They were always open to our ideas, and helped us when we needed it. They trusted us, and we trusted them.”
The third element was transparency. At every stage, the teams made sure they clearly documented what needed to be delivered. And that became even more important when priorities changed.
“It’s really important to keep context and goals documented, and easily available to everyone”, she said. “This really helped us to see what was left to be done. And nobody complained about the list of tasks changing, as this is how the world works.
“It’s less important to deliver exactly what was promised half a year ago. What’s vital is to make sure that everyone’s on the same page at any given point in time.”
“It’s really helped us to create the documentation and processes we need to manage all this,” added Dan. “These engagements go a long way when it comes to patching holes in our documentation, and streamlining our onboarding of new engineering.”
When done properly, Open Development also has the benefit of increasing understanding and empathy between teams, and giving your team a more rounded perspective on what’s going on in their organization.
“Being able to participate in different projects enables both the contributing team and the receiving team to know about what both groups are doing, and to get to know each other on a human basis,” said Felix.
“Spending the past year on an OpenDev project has given me the opportunity to meet dozens of new colleagues in different business units, in locations all over the world. I really appreciated that, especially during a pandemic when opportunities to meet new people were thin on the ground.
“On top of that, discussing my ideas with such a variety of people helped me grow as a professional and a person. Which is absolutely invaluable.”
