Use Your Own Tools

Ambassador Labs's Developer Relations team recently sat down with colleagues from throughout the Ambassador organization to talk about how we use our own tools, or "drink our own champagne"..

As the first in a series of articles sharing the insights from these conversations, Senior Engineering Manager Steve Barlow talked about his professional life, his journey to Ambassador Labs and Kubernetes, and how his passion for "figuring things out" has led to creating a better developer experience for the 99% and the 1% developer alike, particularly through using and improving Ambassador tools.

Empathy for the user, whether an external customer or an internal developer, and understanding how and why they will use an application or piece of software, is at the heart of the work. While all of Steve's experience informed his approach, a previous role as a program manager clarified some of the important questions to understand why things fail, why tools are not being used and how organizational culture can affect products/tools.

"At some point I realized that the software I had helped build during my career did not exist in the wild any more – none of the projects had worked out," Steve explained, "These projects did not fail for technical reasons. Software did not ship because the way we managed it was bad. It occurred to me that I had to work as a technical program manager to figure out the concerns that happen on that side of the house to understand what was going wrong."

Reduce friction: Bridge the technology-business gap

Steve elaborated, "Being in a position to bridge the technical and organizational issues contributes to reducing the artificial stress and potentially lower quality that can result. Internally, I see that it is important to have a kind of mediator helping to figure out what is driving the friction and make people from both the technical and business sides understand the source of the tension to make it more sane."

This "voice of reason" helps influence and shape how tools are developed, improved and shared throughout the org, whether it is individual tools or the entire developer experience through a developer control plane/paved path for developers to follow. This generally starts with onboarding, which can be one of the biggest challenges for developers joining a cloud-native team.

The Kubernetes journey and using Ambassador Labs tools internally

"When I started at Ambassador," Steve explained, "I did everything as though I were onboarding as a developer myself. I tried to run Quick Starts, tried to get my dev environment up and running, and so on. This immediately exposed a lot of friction, seeing what developers need to do – but might not have any easy time doing. I should be able to get a dev environment running within a day, and if I can't, something is going on. As a manager, I am looking for those points of friction."

Prior to Ambassador, Steve had some experience with Kubernetes and containers, but had not tried out Ambassador tools, like Edge Stack API Gateway and Telepresence. He hadn't worked in roles that posed the kinds of challenges that lead teams to experiment with and adopt tools like Telepresence. Part of leading the development and evolution of Ambassador tools meant not only getting hands-on with the products but understanding better the problems they were trying to address, including creating a product experience to ease developer toil.

"We were just spinning up Ambassador Cloud when I joined. The idea behind Ambassador Cloud was making the path clearer for people migrating to Kubernetes. A lot of developers, in my experience, want to know enough to get their job done but do not necessarily want to be Kubernetes experts. They have their primary responsibilities and full plates," Steve shared. "In my own life, I dealt primarily with checkout services, using SAP systems and a bunch of other stuff. But I also know my stuff has to run and run well, but I don't want to know the arcana of how that happens."

Joining the dots to reduce developer toil

With disparate pieces like Telepresence and Ambassador Edge Stack, both helping teams code, ship and run at scale, and developers increasingly being told they have to run everything themselves. In these cases, developers who aren't invested in knowing all the ins and outs want to figure out, "How can I make this NOT my full-time job since I already have one?"

"The idea behind the cloud solution is that we have these pieces that work well in the problem space they are designed for but there is a life cycle to address," Steve stated. "I have to build the thing, deploy the thing, run the thing, and scale the thing. With a cloud solution, we could integrate with great tools to solve these specific problems while also providing a view for somebody that needs to see

everything end to end."

The cloud solution and tools ensure that a team and its developers do not need to be Kubernetes experts to perform the code-ship-run equation. How much does a developer need to know? This is where Steve sees the value of Ambassador tools.

Measuring success: How do you know your tools are helping you achieve aims?

Success metrics fall into two categories. One is operational, and one is business related.

"As an engineering manager, there are operational metrics I care about, like how well the system is running, how many incidents we have and what caused those incidents. We are trying to optimize for both the operational and business aims. We don’t want to ship 100 features one week if doing so makes the workplace miserable and damages the quality of what we deliver. This doesn't help the product, the business, our teams, or developers. We want to keep improving as an engineering org and continuously reflect on the things that worked and didn’t work."

What makes a better developer experience?

Improving as an engineering organization and creating a better developer experience means, for example, measuring things that affect the speed of feedback and productivity and enabling teams with the right tools to achieve these aims. For example, Steve explained how reducing the time to approval for pull requests (PRs) and preventing PRs from getting stale improves time to feedback, and specific tools, like Telepresence, can help in several ways.

Tool time: How Telepresence delivers productivity gains and success

"Time to feedback can be critical in organizations focused on speed, like ours," Steve shared. "Even as someone working with Telepresence at Ambassador, I had a fairly narrow view of what Telepresence does, using it to intercept a service I was developing. The classic use case, that is, I develop, and then there is a cluster that has 12 other services that mine interacts with. I want to understand them without running them all on my laptop, or in our case, where we have shared staging. Some orgs spin up whole clusters for each developer, and things can get both expensive and out of date. During my own actual code loop, I ran Telepresence and could see the responses and work through them, which is efficient and productive.

But Telepresence can do more than that – at different stages of development. One is in active coding. Another is when I reach a stage where you can test whether something is good. You can proactively put it up against the real thing, where you can easily see where the issues are before actually submitting a PR. A kind of pre-validation PR to reassure yourself, 'Do I feel clean about posting this PR?' When you're posting the PR and sharing it for others to look at, Telepresence provides a preview link so the reviewer can click and see as they look at the code, providing helpful context."

Expanding the scope of tool usage

Steve described how he works with two internal teams. First, there is a code team focused on developing Telepresence and parts of the cloud that integrate with it. The second team is the growth engineering team, which works closely with cross-functional Ambassador teams, including marketing, developer relations, and sales. These teams are focused on running experiments to work toward business goals and how we can continuously improve.

As an example, we worked with growth engineering to focus on the Telepresence Quick Start (QS) guide. We knew, for example, that 100 people looked at the QS, but only two people ever successfully created an intercept (which was how we defined “success” for the QS). Somehow, we lost 98 people and needed to figure out why. Is it poorly written or confusing? We needed to experiment to find out if we could make immediate improvements. For example, what if we change the text this way, what if we automate a piece of this?"

Telepresence is an invaluable tool for enabling these cross-functional experiments. One of the engineers might mock up a page, but it is not quite ready for staging. Telepresence can run against whatever the code bridge is on the local machine, and a developer can share a URL to gauge with team members whether development is going in the right direction or not. Telepresence helps provide more context to enable better feedback, content, user flow and so on. While this does not, as Steve explained, directly shorten the process of PR closure overall, it does reduce the overall time to shipping one of these experiments.

Use your own tools: Benefits of “drinking your own champagne”

"Drinking our own champagne allows us to have confidence in what solutions we put into the world. When we use our own tools, we may be using the most common use case, which is not as complicated as the use cases of many of our customers," Steve explained. "When the baseline use case is solid, we are very sure that when someone has a problem with Telepresence, like 'is this a bug?', 'is this a missing feature?', 'is this a strange interaction with a load balancer we haven't encountered before?', we can more easily and quickly identify fixes and alleviate points of friction for users."

One such point of friction that led to the saved intercepts feature in Telepresence came from internal discussion on how Telepresence was being used. Developers who use Telepresence daily weren't typing intercept commands themselves – they used predefined intercept commands that included all the necessary parameters, ports, etc. It turned out to be a valuable shortcut that developers used to relieve them from having to think about all these important but irrelevant-to-developers parameters.

"Based on this experience," Steve shared, "we wanted to provide native support for these shortcuts. Saved intercepts will be a part of Ambassador Cloud so that when you are logged in, Ambassador Cloud stores the history of your successful intercept commands. Someone who doesn't know anything about configuring intercept commands can use saved intercepts to save time."

Beyond tools: Spreading best practices and culture internally

"Drinking your own champagne" isn't only about the use of your own tools internally. It is also about using tools as enablers for spreading best practices and culture. Tools are only as good as how they are deployed within a well-functioning development culture.

Culturally, it's important to be able to answer the question as to why external organizations and developers should use your tools. To do this, you need to enable and ensure your internal teams are prepared to use your tools. And if they aren't, why not? Are you building something that is not applicable for you, and if so, are you the best organization to build it… and how do you know? What is involved in drinking your own champagne – both in terms of the tools themselves and in how you make them available internally?

"Some of this is about onboarding. When someone new starts, we always buddy up new starters with other developers. In the past, it always used to be the most senior person, but we have discovered that while new developers want to talk to the most senior person, they want to be paired up day to day with someone closer to their own level is better," Steve explained. "We help people jump right in with both the tools and the culture. I want new devs to get a PR right away, even if it is just fixing a typo. It does not matter what it is. Because the point is that you have your dev environment set up, you understand the pattern of how this stuff works as soon as possible."

Conclusion: Raise a glass to drinking your own champagne

"A part of drinking our own champagne is living our company culture, which includes a real sense of empowerment," Steve shared. "One thing I personally like about working at Ambassador is the direction you get to go figure things out on your own. The culture is not directive; it is open ended with specific goals in mind. I like feeling like I can make decisions together with the team I am working with and improve our own productivity, and by extension, our products, getting them shipped faster.

It is a philosophy that filters down – that’s why we work in cycles – we don’t know what we don’t know, and we could be wrong. And if we are, we don’t end up wasting a whole year or six months – at worst, we lose a month or six weeks, but then we can adjust. Because we have that cultural framework, this social contract, as long as we keep delivering at least every six weeks, the business supports our freedom."

The Ambassador team is always happy to talk more about our tools, your plans, and how our tools can fit with what you're doing. To learn more, don't forget to join the Ambassador Community Slack channel.