Microservices, constant iteration, and the shelf life of good solutions, according to Diederik van Liere

Welcome to Level-Up, an exclusive interview series with standout engineering leaders who share what’s top of mind for them. This interview puts the spotlight on Diederik Van Liere, VP of Data & Engineering at Wealthsimple. Let us know who we should talk to next!

Today’s companies are inundated with data–data about how their users behave, what products they buy, and how they interact with your software.

Diederik van Liere, VP of Data & Engineering at Wealthsimple (previously Shopify and Wikipedia) is fascinated by how this data can be leveraged to make better decisions and to create more user-friendly products and services.

At Wealthsimple, Diederik is responsible for everything with data: data warehouse, data platform, data science. He’s also responsible for the company’s Book of Records, which keeps track of client assets, securities, and cash that they can invest and trade.

“I’m curious about how we can use data to make better decisions,” he said. “In particular, how can we use data for automated decision-making using machine learning and AI? These are the challenges I wrestle with on a daily basis.”

I recently got to chat with Diederik about the architecture at Wealthsimple. He shared details about Wealthsimple’s stack, as well as his perspective on microservices and the future of software development. Diederik has also watched companies grow and has insights into how platform architecture must  adjust as business objectives evolve and companies process more data.

Diving into Wealthsimple’s architecture

Diederik describes Wealthsimple as a matrixed organization. That is, the company’s products have their own set of microservices, as well as those that spam or are re-used across the products.

“We have four core products and each one has its own dedicated product engineering groups. They’re very feature-focused, and they have their own set of microservices,” said Diederik. “But we also have a few domains that offer functionality across all of our products. For example, a client [i.e. user] domain, a money movement domain, or a trading domain has functionality that can be re-used as well as functionality that’s specific to the product. As a result, we have a matrixed organization.”

Wealthsimple’s approach to service ownership has changed as the company has grown. “As your business evolves, you have to redraw the boundaries of service ownership,” said Diederik.

He suggested that while many assume service ownership is static–believing that because Team A built a certain system, Team A will own it forever–they’ve become far more pragmatic and flexible at Weathsimple. As a business grows, it’s engineering organization needs to figure out how to manage the inevitable service ownership changes in order to evolve with the business.

The pros and cons of microservices

Diederik shared a number of interesting pros and cons with microservices architecture. Of course, it’s much easier to get familiar with a microservices codebase because the scope is significantly smaller. At the same time, it’s much harder to understand how these services fit together and interact within a larger architecture.

“When you’re using microservices, you’re naturally going to have dependencies on other services. To really understand those–and to debug them when issues arise–it becomes significantly more complex,” said Diederik.

But Diederik doesn’t see this as a reason to use a monolith rather than microservices. Instead, he sees the need for investing much more in tooling and logging. “In order to be effective with microservices, you need to invest way more in tooling and logging to help you understand how your module is actually working in detail,” he said.

Since this tooling is so foundational, Diederik advocated for opinionated platform teams that take decisions about monitoring and logging libraries out of the hands of individual developers. Choice and flexibility aren’t appropriate everywhere.

Diederik also believes that some of the challenges in microservices can be ameliorated by being savvy about creating API gateways. “People need to move away from an architecture where ‘everyone is allowed to talk to everyone,’” he said. “Rather than talk to a service directly, they should communicate through an API gateway that routes to the right microservice.”

Engineers and data scientists want to move as quickly as they can to meet the business needs, but they may never be able to respond instantly. “But it’s important to realize that the microservice philosophy is a way to think about what can we do to fundamentally move faster to support the business better,” said Diederik.

A constant iteration as businesses grow and change

As businesses grow, they collect more and more data. But collecting data is the first step–you need to make it discoverable and interpretable to make it useful. And, what worked well when you first started may not suffice as you grow.

“What typically tends to happen is that the solutions that worked really well have a limited shelf life. So you keep chasing, or improving I should say. You keep improving your architecture for the next scale, next challenge.” he said.

Diederik recommends developing a dynamic perspective. After all, the scope and domain of a particular service might be right for a certain moment in time, but inevitably the business keeps changing, and your scope and domain need to change again.

 “When it comes to architecture, we need to think of things in terms of a dynamic, constant cycle of iterations. We get it more right, and then something happens and things are less right,” he said. “We go through this over and over again.”

For example, Wealthsimple operated its invest product in the US market for a while. They had one service with an abstraction layer on different brokerages. But after ceasing to offer the product in the US, that abstraction layer became unnecessary. At one point, it was very useful and served a clear purpose. But the whole purpose of the unifying API was no longer a business goal.

“When it comes to architecture, we need to think of things in terms of a dynamic, constant cycle of iterations. We get it more right, and then something happens and things are less right,” he said. “We go through this over and over again.”

A third wave of software development - monoliths to microservices to something new

Diederik believes that the cutting edge of software development is in microservices, but that the future may bring something new. He’s been impressed with the leaps that no code services have taken and believes there may be a third wave of software development where new possibilities emerge.

“No code platforms have come a long way in the past 10 years,” said Diederik. “I think we will get more and more powerful abstractions in the next phase with more expressive blocks. I’m speculating here–but I think we’ll see this more and more”