ArgoCon Community Learnings from the Adobe IDP Workshop
Co-Author: Aya Ivtsan.
Last month (Sept. 19–21), industry leaders converged in Mountain View, California to host ArgoCon 2022. The Argo project is being leveraged by more than 190 companies including Google, Intuit, Tesla, and Adobe. This year, Adobe was a diamond sponsor of ArgoCon and hosted a workshop on September 21, 2022, to brainstorm and formalize the definition and capabilities of an Internal Developer Platform (IDP) with the community’s feedback. More than fifty members of the Argo community, including CNCF industry leaders, came to the Adobe offices and joined with Adobe engineers and leaders to talk about IDP. This workshop also featured a panel discussion which included industry leaders from AWS, Intuit, and Upbound (covered in detail below).
In this workshop, which was in the spirit of community open discussion, we shared our IDP journey, and brainstormed with other community thought-leaders on making IDP a reality, based on Argo and other open-source projects. In preparation for the workshop, we had shared the following GitHub document where we outlined the key capabilities of an IDP and gathered the community’s thoughts and feedback. Here is a visual we came up with to formalize key capabilities of an IDP. We divided them into three phases of the development lifecycle as explained below:
· Discover and Create: Onboarding, training, code bootstrapping, local development, etc.
· Integrate and Deploy: Deployment, integrate distributed systems, configuration of resource.
· Operate and Improve: Management of the service/app at runtime, observability, monitoring, etc.
Please see this GitHub document to learn more about each capability.
Below are key highlights from the IDP workshop:
IDP panel discussion
Aya Ivtsan from Adobe moderated the IDP panel.
The panelists were:
· Viktor Farcic — Upbound
· Nima Kaviani — AWS
· Edward Lee — Intuit
Here is a picture from the panel discussion:
Panel summary:
· What is your Twitter version of what an IDP is, and what is it not? Do you feel there are any misconceptions on what IDP should be and do?
Viktor:
· It’s not about wrapping things in a layer but treating every user as a customer. Problems come from tools that developers are handed, the ones they aren’t choosing.
Nima:
· IDP should be less about tools, but more about practices. We have all the tools, pieces and portals, but we need an overarching set of practices to improve productivity including consistency across teams.
· There’s a big appeal in giving teams autonomy, freedom & responsibility, but this leads to sprawl and duplicated effort, and IDP should remove that.
Ed:
· IDP is a platform abstraction to remove dependencies and friction. A company must be aligned on dev process. IDP is not just for developers, it’s for the whole company. It should be “development”, not “developer”. Operations, security, product management and all the data they produce with executive views & reporting built in.
· It’s not just the product, but the standards and processes for how development gets done. The UI Portal is just the tip of the iceberg and should only be used when starting a project or when something goes wrong. It should be invisible to developers.
· Large companies will have multiple development processes and getting them all to agree is a big part of building an IDP. Shared standards with buy-in are required. Document decisions that get made, standards agreed on, that platform team can then implement.
· IDP is about making developers more productive. How are you measuring developer productivity? What are the top KPIs you use to measure the success of your IDP?
Viktor:
· The key KPI is the time to deliver to prod (as waits between activities are removed).
Nima:
· The key KPIs are DORA metrics. How much time developers spend on core features vs non-functional challenges.
Ed:
· DORA metrics — frequency of deployments, time from commit to prod, MTTR, what percentage of changes cause incidents.
· When teams can see their metrics, it generates discussions about how to improve them, which in turn brings improvements.
· What are your thoughts on self-service in the context of IDP? What is the role GitOps should play in enabling the IDP?
Viktor:
· Git should be the only writable place — controlled and reviewed. Single interface recording all changes. All other tooling should be read-only which eliminates “who did what” discussions.
· GitOps shifts the focus from how to do things, to just defining what those things should be and having background processes translate that desired state to reality. Developers don’t have to master all the tooling.
Nima:
· Give users tools and practices and then stay out of their way. Set up agreed patterns for bootstrapping, then let teams fly free.
Ed:
· Follow the open-source development model. Since platform teams can’t give white-glove support to everyone using the platform, they need to invest in documentation and let users come to the repo and understand it, install it and run it. Five minutes to “wow” should be the goal, or they won’t come back.
· Internal projects must be the same way and shouldn’t require meetings, white-glove, etc. Don’t abuse your captive audience (ie internal developers), inner-source the same way open-source is done and measure cross-group contributions.
· GitOps is great for developer experiences as it saves UI work, but don’t use it for everything or abuse it.
IDP community brainstorming and breakouts
During the workshop, the group collectively went through a white-boarding exercise (using Miro), where attendees shared and voted for their top pain-points in the development lifecycle. Six key themes emerged, out of which we reduced the final list for deep-dives to two (see below):
· Disjointed experiences across tools and workflows (winner by votes, see details below)
· Discovering existing services, libraries, templates, and documentation (winner by votes, see details below)
· Continuous code improvements to address concerns like security and performance
· What is the right level of abstraction?
· How do we measure success?
· Creating a new service with production readiness built-in
Here’s a snapshot of the Miro board the group worked on:
Here were a few highlights of the breakout discussions for these two topics:
Disjointed experiences across tools and workflows:
Pain points shared:
· Different entry points and flows to kickstart a project and multiple infrastructure-as-code tools being leveraged.
· Lack of a common unified dashboard that shows data around cost, APIs and observability in general.
· Vendor lock-in.
· Scaling issues and keeping with versions of plug-ins.
· Onboarding for new employees/grads and lack of K8s knowledge.
Potential solutions discussed:
· A common onboarding API-first wizard which is easily scalable and maintainable.
· Consistent authentication model across toolsets and better interoperability between core components.
· Bring in college graduates and train them on enterprise-ready software within the first 90 days to start creating power users early.
· Don’t over-automate if there’s no payoff, get user feedback and don’t have a single team/bottleneck creating the onboarding flow.
· From a technology perspective, leverage technologies that encourage open development, like micro frontends, and harness Argo Events for useful data extrapolation from artifacts and developer related tasks.
Discovering existing services, libraries, templates, and documentation:
Pain points shared:
· Discoverability of services, templates, APIs, and libraries.
· How should teams handle new vs. established entities.
· Problems with maintaining and updating generated code/templates.
· When searching for existing services/libraries/tools within the company, it’s difficult to tell what is actively being developed and the availability for those solutions.
Potential solutions discussed:
· Establish consistent naming conventions and/or metadata with frictionless enforcement.
· Build tools on top of the established patterns and ensure documentation does not make assumptions of what is known/needed.
· Templates/marketplaces/portals that help people know what solutions are out there.
· Need solid examples of all the pieces that are needed to go from repo to production and have tool chains (Portals, CLI) to guide through processes.
· From a technology perspective, capture GitHub information to identify active projects while moving to an internal open-source model.
· Utilize observability metrics to establish usage patterns.
· Use things like API Gateway to create a consistent methodology.
· Leverage Artifactory for images, libraries for baselines, documentation platforms.
Wrap up and references
ArgoCon 2022 was a great success and the community’s participation during Adobe’s IDP workshop was very active, which shows that we all face similar pain points and challenges. We look forward to continued discussion and collaboration with the community as we continue to improve the developer experience and development lifecycle.
Below are some materials you could reference to learn more about the workshop and how Adobe is leveraging the Argo projects.
· CNCF Slack channel: #adobe-idp (Join to stay tuned for updates): o This channel contains the PowerPoint deck we presented in the workshop and information about the Miro board we used. We will keep you updated on IDP related updates and share future sessions Adobe will be presenting during KubeCon or other conferences.
· How Ethos Powers a Cloud-Native Transformation at Adobe: https://adobe.ly/ethosblog
· ArgoCon21- Adobe CI/CD Presentation: https://adobe.ly/adobeargocon21
· Adobe’s IDP Journey and Lessons: https://adobe.ly/adobeidpjourney
· ArgoCon22- Harder, Better, Faster, Stronger Pipelines @ Adobe session: https://www.youtube.com/watch?v=AD3isYT6M1c
· ArgoCon22- From PR to Artifactory in 60 Secs: https://www.youtube.com/watch?v=JD3dRElwXMM
· ArgoCon22- Keynote: Adobe’s Internal Developer Platform Journey and Lessons: https://www.youtube.com/watch?v=NJF6bBU6Duw
