Overview of the Brightcove Architecture Review Process

We believe in continuous improvement at Brightcove. That applies to more than just ourselves and our software: it also applies to our processes. Recently, a group of us got together to talk about how our architecture review process was working for us and decided to give it a refresh. We want an architecture review process that is:

Iterative
Comfortable
Adaptable to different types of projects
Respectful to the expertise of the participants

The end result was the following document that describes our updated process.

What is architecture?

Software architecture is about making fundamental structural choices that are costly to change once implemented (source).

What is the process?

Designing software is a continuous process. If it was possible to know everything we need to know to make perfect decisions up front, we wouldn’t need agile. To reflect this, our architecture review process has several steps that can be adapted to the needs of each project.

It may not be necessary to perform all these steps. It may be helpful to repeat steps. Generally, a review and a retrospective should be considered the minimum. Do whatever makes the most sense to achieve the following goals:

Make the best software architecture decisions possible
Produce useful engineering documentation
Learn along the way and adapt your plans to what you learn
Keep your peers informed of what you’ve decided and what you’ve learned

The following sections describe each step in the process.

Architecture Workshop

Maybe you're planning a new project, but just don’t know how to implement it yet. Maybe you have a challenge that isn't fully defined yet. Maybe you have three different perspectives on a challenge and are not sure how to pick one. Maybe you have an idea for an architectural change and would like to connect it to a problem statement. Rather than hunkering down and trying to answer these questions in a silo, it’s good to start talking to other experts early. This makes it easier to incorporate feedback and consider big changes before the plan gets too established. It also helps the architecture review run smoothly, since the architecture will be more ready.

Checklist

Invite subject matter experts (eg: leads of impacted teams)
Invite the architecture reviews Slack channel
Suggest 3-8 attendees

Suggestions

Start by fully providing the context and explaining the problem that needs to be solved. Inviting someone who has no pre-existing knowledge can help hone the problem statement and challenge assumptions.
Architecture workshops can be speculative. Maybe the idea won’t end up moving forward, or it could become a hackweek project instead. No problem! If all you’ve done is gotten some experts to discuss and learn more about the challenges we face, it was a win.
If there are too many unknowns to move forward, consider some research activities then try again. Prototypes, research spikes, and reading are useful tools in this phase.
Invite non-experts too. This is a good opportunity to give people more experience in architecture design. The best way to learn how to design good architecture is to either try it yourself and fail a few times or watch other people do it.

Architecture Review

An architecture review is the presentation of your architecture to a broad audience. Here, we want as much participation as possible. You’re both soliciting additional feedback and educating the audience about your project. This should be done before development begins if possible.

Checklist

Invite the main engineering Slack channel
Remind #engineering right before the meeting starts
Create engineering documentation before the architecture review and share it with participants in advance
In the meeting, walk through the documentation, explain it in detail, and solicit questions and feedback.

Suggestions

Start by fully providing the context and explaining the problem that needs to be solved. Remember: your audience probably includes people who are new to Brightcove and people who have no context about your engineering area. Avoid saying “as you already know…” as it can alienate newcomers and reduce participation.
Make engineering documentation that will be updated going forward, not just a one-off document. At the end of the project, you will ideally have up-to-date documentation of what was actually built. This documentation can be used for reference, to share with stakeholders, to ramp-up new team members, and so on.
Be visual in your documentation. Diagrams and photos of whiteboards can really help readers take in your ideas efficiently.
It’s possible to feel like this is a defense rather than a feedback cycle. If you feel like you’re on the defense, remember you don’t need to have all the answers now. It’s ok to say “Thank you for bringing that up, we’ll get back to you after we’ve had some time to think about it.”
There is no expectation that you do everything someone says you should do. Disagreements are ok. Overlooking something is not as great. As people learn from each other, they will gravitate toward the best ideas.

Documentation

Here are some suggestions of what to include in your documentation. Not all of these will be relevant to every project.

System interactions
Interfaces
Domain modeling
Deployment plan
Technologies used
Where will the hosts live
Monitoring plan
COGs
Billing
User experience
Maintainability
Accepted compromises (technical debt, etc)
Security
Potential patterns of abuse
Secure coding practices
Authentication / authorization
Auditing / logging

Participant Guidelines

Presenting new ideas to a big group can be pretty stressful. Please help them have a good experience!
Avoid advocating for your ideas by raising the stakes, eg “If you don’t do this, the project will fail.” Focus on explaining the specific ramifications that you believe are being overlooked.
Avoid saying “Why don’t you just…” This suggests that the presenter is making an obvious oversight. This is usually a matter of perspective, so approaching from a perspective of curiosity and exploration can lead to a more constructive conversation.
If you have a feeling something won’t work, but you’re not sure why, it might be better to give it some more thought before bringing it up. Or, if you want to raise a gut feeling, call it what it is. eg “I have a gut feeling there is a problem with using this tool we’re not thinking of, so I want to revisit this again later.”

Architecture Update

For long-running projects, it is useful to get together regularly to talk about the progress of the project. This acts as a retro for the progress so far, a review of the updated architecture, and a chance to change plans according to what was learned.

Checklist

Invite everyone involved in the project
Invite the architecture reviews Slack channel
Update the engineering documentation you created in the review so it reflects the latest state of the software.
Solicit retrospective notes ahead of the meeting. Provide a place for people to write what is working, what isn’t working, and suggested actions.
In the meeting, walk through the updated engineering documentation, call attention to the changes, and solicit questions and feedback
In the meeting, have everyone read their retro notes and encourage discussion

Suggestions

For long-running projects, it may be wise to do this at least once a quarter. This can result in finding hidden issues or novel new ideas, even if the project is going well.
If you feel like it’s time to have this meeting, you’re probably right. Don’t wait until the end of the quarter, end of the project, etc.

Architecture Retro

Have this meeting after the software is shipped. Essentially, this is the same as the architecture update, except it’s the last one and it should include a wider audience.

Checklist

Invite everyone involved in the project
Invite the main engineering Slack channel
Update the engineering documentation you created in the review so it reflects the latest state of the software.
Solicit retrospective notes ahead of the meeting. Provide a place for people to write what worked, what didn't work and suggested changes.
In the meeting, walk through the updated engineering documentation, call attention to the changes, and solicit questions and feedback
In the meeting, have everyone read their retro notes and encourage discussion

Suggestions

Retros are one meeting where sticking to the agenda is not always best. If people want to discuss a particular challenge, make room for it. You can always schedule more time if needed.