December 31, 2020

Lessons from freshman year

One year into my tech journey as a full-time software engineer - what an amazing ride! I have grown much more than I expected and taken away way too many lessons from the year. I often wonder if it ever slows down but my feeling is that it never does. Given it’s the end of the year, it’s a good time to write about my experience. This is a long post. To make it more palatable, I’ve split into two sections - the first is core lessons and the second can be thought of as an addendum. I write for you but also for me so consider this a blog post and a journal entry ✍🏾. Time for some reflections!

Software is all about trade-offs

Software has many camps full of evangelists. Just start a conversation about using Vim and you’ll encounter some of them. For example, I’m sure this guy is a Vim user.

The honest truth is that there is no one way to build software. There is no truth. There is no best way that is generally applicable. Everything in software is contextual. All decisions come down to trade-offs. Do you need fast writes or fast reads? Do you need to stream or batch process data? Are you concerned about latency and/or throughput? All these questions will guide you to the final set of technologies you use. The more complex your existing system, the more difficult questions you’ll have to answer and the more trade-offs you’ll have to make.

It’s annoying to see such a strong hype culture in tech (though I’m sure it’s not local to us). Nonetheless, once “revolutionary” new technology comes out, some proponents think it is the only way to build software and will consider your technological choices inferior if you diverge. This is undeniably false. I can guarantee you that your 50 person startup does not need microservices and Kubernetes. You can get extremely far with something simpler and/or design your applications in a way that can easily be split out when needed. Unfortunately, you’ll hear a chorus of developers pushing that microservices is the only way to build software in the modern era. What is important is not the use of any specific technology but the applicability of it to your context. You’re not Netflix, Google, Facebook, Stripe or Uber. You are X and you have Y problems that can be solved with Z technologies. Understanding how problems are solved in various companies is important as it improves your mental models. You’ll be able to improve your designs and draw inspiration from multiple sources, tailored to your context. Think deeply about your problems, understand your trade-offs and design from there.

Building products is hard

I don’t think you can appreciate how hard it is to build a product until you are in the thick of it. Depending on the nature of the product, the technical aspects of it can even be the easiest. There are several dimensions to the process that make it extremely difficult. Some of the ones I have no good answers for:

  • How do we get relevant and accurate customer feedback - It is fairly straightforward to get some feedback. The problem is getting unbiased and accurate feedback. For example, the phrasing of a question can bias the answer. “Did you have a positive experience with this aspect of the product”, “Did you have a negative experience with this aspect of the product”, “How was your experience of this aspect of the product” are all asking the same underlying question but could swing the pendulum of answers in one direction or the other. Being able to get accurate feedback is difficult but essential for the development of a product.
  • What is an effective way of marketing the product - It’s easy to believe that marketing just requires exposure. The more visible it is and the more hype you create, the more people will use your product. This is most definitely false, nothing is that easy. You have to know your customer and meet them where they are. This looks different for every company. Tailoring your marketing strategy to your context is necessary for it to work.
  • Prioritising feature development - There are too many features to build and not enough time and people to build them. In my opinion, this is usually a good thing as it forces you to sift through the noise. Nonetheless, prioritising what to work on is a really big headache. Every single feature feels high priority. It’s as if not building all the features right now will sink your product. There is also the question of whether the features you do decide to build are the right features. There is nothing worse than building the wrong thing. Validating that you are working on the right features requires good customer feedback and as we already know that is non-trivial too.
  • Balancing speed vs quality - In the startup realm, your aim is to minimize time to market. This presents the common trade-off between speed and quality. Balancing them adequately is difficult. Too much time and you risk becoming redundant. Too low quality and you risk poor reception due to negative user experience. Understanding where and how much you can trade-off one for the other is an important part of product development.

All these above points just highlight how difficult it is to build products. It is by no means an insurmountable challenge nor is every mistake equally as costly. For me, the realisation was that building products is just not straightforward and many factors to its success are outside the realm of tech.

Value is king

As developers, it’s easy to get caught up in the technology you’re using. There’s a whole website dedicated to searching tech stacks of several companies. Everybody wants to use kubernetes, service meshes, distributed databases, Golang or Rust, VueJS, deep learning. You can go on forever. Companies often use this to market themselves - “we use machine learning to power our next-generation platform” blah blah blah. This does matter for marketing purposes in the case where your consumers are likely to understand the value of the technology. However, the technology is not the reason they use your product/service, it’s the value. Value is king. It is the only thing that matters. How you go about attaining that value is up to you. If that comes from technology from the 1800s or is cutting-edge research from weeks ago is actually irrelevant. If I as the consumer derive the same value from a product with old technology vs new technology, I will just choose the cheaper one. This is all to say that focus on providing value to your customer. Use whatever technology will add that value and make your lives easier as your product becomes more complex but there is no point in getting caught up in it.

Scaling startups is difficult

As a developer, when we think scale, we think about X fold increases in the use of our product. This is a common concern because everything becomes more difficult when you introduce scale. In the same way scale is difficult for technology, so it is for a startup. Anyone that’s ever been part of a fast-growing startup will tell you that during those growth periods, it feels like chaos. The most glaring difficulty during this time is scaling processes. How do you go from managing a team of 20 to a team of 100 in a short period of time? Exactly. At this stage in a startup’s life, the processes won’t have caught up to reality. The business will have to define and redefine its organizational structure, introduce processes to mitigate lack of communication, ensure transparency across the whole team, figure out how to handle hiring for different teams, how to do performance reviews, their promotion structures, their compensation structures. It is a lot. Scaling a startup is a journey, one that anyone joining a fast-growing startup should be ready for. It’s a great time to be part of a company and see it put through its paces.

Communities matter

Building welcoming and authentic communities matters, possibly more so than the topic they form around. When learning Rust, I quickly came to a standstill. It is a big language and it does not hide many details from you. This makes it fairly difficult to learn and to continuously progress. Around that time, I came across the website, Awesome Rust Mentors where you can find someone to mentor you along your Rust journey. That led me to meeting the wonderful Ana Hobden. Next thing I knew, I was on a Discord server with a whole bunch of software developers from around the world with many of them being heavy Rust users. It is an amazing community - everyone is welcoming, kind and willing to help. Arguably, the reason I’ve managed to continue on with Rust is the community. This is a sentiment you often hear in other communities too such as the Golang and Kubernetes communities. I think community is of the utmost importance and it’s value cannot be underestimated. Oftentimes, the success of a technology (especially in open source) is, at least in part, a result of the community that is built around it.

Great developers are great designers

One glaring difference I’ve noticed between myself and senior developers is their ability to reason about and design code, libraries and systems. I think one of the hallmarks of great developers is their ability to design. Why? Well, the process of writing the code is often not the most difficult part. There are several difficult design choices:

  • How do we design the right abstractions so they do not leak (too much)?
  • How do we structure the code so it is easy to read while being easily extendible?
  • How do we design the architecture of the system so it is as simple as it can be for the given complexity while being easy to maintain and extend?
  • What are the right technologies given our requirements?

These are all things that do not actually have to do with writing code. Look at the RFC process for any language and you’ll understand what I mean by design. Great developers are great designers.


Those are the core lessons that really stuck with me this year. Read on for the rest not so core but pretty interesting lessons.

Coming in as a junior developer with limited knowledge, the sheer amount of tech to learn is overwhelming. Purely out of curiosity, I spent most of my year in exploration mode. I managed to get a feel for different technologies, programming languages, architectural patterns, etc. In hindsight, it was a great “decision”. Early in your career is the best time to explore because you are not expected to be an expert in anything. Exploring also gives you the chance to discover what you really enjoy. I currently work as a machine learning engineer. My interests have gone from machine learning research -> machine learning systems -> software infrastructure and distributed systems (as a more general field instead of machine learning specific). If I didn’t spend time reading and surveying the tech landscape, I probably would not have found that I really enjoyed other subfields. Another thing I did was play around with several languages: Clojure, Golang and Rust. I found it to be really valuable. From Clojure, I learned about the functional programming paradigm (that’s why I chose to learn it) and about the Lisp family of languages. I didn’t spend that much time with Golang and did not even mess around with their highly touted concurrency story. To all the Gophers, I’m sorry. However, from Go I learned that a language that keeps it simple can reduce friction during development, being opinionated can be beneficial and that you can live without exceptions. Rust is currently my favourite language. Memory safety, ergonomic error handling, strong type systems, pattern matching. The list goes on. Anyway, the point is that, spending time exploring is advantageous. It helps build a solid foundation and shapes your thinking.

Fundamentals come first

Tech is a huge, fast-evolving field. It is literally impossible to keep up. The best way forward is to focus on fundamentals. You often hear this song from the frontend community where they have extremely powerful frameworks. These are great but they are abstractions - plenty of detail is hidden. Many developers spend time only learning frameworks and not the core principles behind them. This makes much of your knowledge irrelevant when a new framework rises to popularity. A focus on fundamentals will ensure you have a small, well-refined and good personal “standard library”. This sets a solid foundation and allows you to onboard information much faster. It can be fairly boring focusing on fundamentals though. I’ve found a top-down approach to learning to be most beneficial. Starting at a high-level gives you a feel for what is going on and peaks your interest. Over time, I successively go lower-level until I’m satisfied.

Seniors are leverage

As a junior, you come in wide-eyed, fresh, hungry to learn and literally know nothing. In comparison to you, seniors seem all-knowing. While in truth they are also learning and growing into their seniority, they contain a wealth of knowledge. They have wisdom you can only get from paying your dues. Lean on them for technical advice and knowledge. Listen to them when they are having open discussions. Ask them tons of difficult questions. I certainly have and have found it to be one of the most valuable ways of learning. It’s helped shape and refine my thinking on different aspects of this profession. They are leverage when it comes to technical growth.

Seniority is (sometimes) overrated

There is no consensus on what a senior truly is. It’s a recurring story that seniors from a startup who join big tech firms often start as intermediates. As you become more senior, the expectations on your ability increase which in turn makes it easier to categorise your level. Depending on the density of talent, levels start lower or higher than in your current firm. This range of ability can be quite deceiving. It’s entirely possible that intermediates/seniors are not actually that much better technically than their junior/early intermediate counterparts. There are a number of dimensions that make a senior a senior. Obviously depth and breadth of knowledge is one of them but more importantly, in my humble opinion, is their ability to reason. Understanding how changes will affect the system, what trade-offs they are making by adopting a certain technology, being able to model the domain well, creating good abstractions, applying architectural patterns when they are a natural fit, etc. These are all things that come with experience (and being burnt several times πŸ˜†). Not all seniors are actually good at them and in cases where they are not, seniority is overrated. Outside of your own personal incentives (i.e money, status), it’s important not to give too much weight to seniority. A focus on technical mastery will take you where you need to go.

Titles matter

There’s a growing narrative that titles do not matter. I guess this is more specific to smaller companies where there is a flatter organizational structure. Regardless, individuals and companies alike are pushing a message that they are not as important as before, that everyone’s voice is heard and that if you have great ideas, you are empowered to take them forward. While this is often true, it doesn’t take into account that we behave in accordance with our position in the hierarchy. Let’s look at a contrived example. Imagine there is a tech team and they are struggling with silos between product teams. A junior is passionate about knowledge sharing, has good ideas and wants to push for it to become ingrained in the team’s culture. She comes up with a plan, the CTO loves it and leaves it in her hands to drive forward. Now comes the challenge. You are a junior and therefore inherently don’t command the same level of power and influence over the team. If your seniors are not as invested in the idea, the power dynamic is not in your favour. Given the hierarchy, you’re likely to stop pushing your agenda as you’re now “imposing” something upon your seniors and intermediates. This can be taken negatively quite easily, depending on the personality of your team. Alternatively, it could be that people won’t take it as seriously because it’s not coming from the top; if it’s a junior’s agenda, it’s surely dismissable.

Titles give you access to certain conversations and platforms. The higher you go, the more knowledge you’ll have of the state of affairs of the company. You’ll be invited to conversations that determine the direction of individual teams and the company. It’s a powerful position to be in. To me, it’s clear that titles really do matter. I don’t think it worth chasing titles at the expense of everything else but it is naive to completely dismiss their importance.

Overcoming silos needs good systems

As the need for more teams emerges and the number of staff increases, communication becomes one of the biggest bottlenecks. Without good communication, silos begin to emerge. Soon enough, tech has no idea what marketing is doing, marketing has no idea what finance is doing, finance has no idea what sales is doing and everyone is just confused. Nobody knows what any other team is doing. Worst off are teams that are re-solving problems because they don’t know other teams have solved them already. This tends to happen in product teams. Product team A will go about building some feature and solve a whole host of technical problems. Later in the year, product team B will encounter and solve a portion of those problems as well, not knowing product team A has done so already. Overcoming silos requires a good system that enables effective communication. A good example of this is AirBnB. They wanted to overcome silos amongst their data teams. They call the process of ensuring insights uncovered by one person/team are effectively transferred to the rest of the company, scaling knowledge. In this particular case, they built an internal product called Knowledge Repo which is used by data scientists to share their work. It’s a centralised space so people can look up all work done by various teams. This is a good way to scale communication for this particular use case. Silos are a massive issue. Miscommunication causes tension, misunderstandings and reduces the productivity of the company as a whole. Continously ensuring communication is good is important.

Processes increase velocity

I read somewhere that processes are not supposed to introduce overhead. When scaling a company, processes are introduced in order to manage teams effectively. Often, process is seen as a negative as it adds multiple steps between idea and action and promotes a highly bureaucratic structure. This is true, especially at massive firms. Process, however, is not meant to slow you down. They are introduced precisely because things are in a mess and getting anything done effectively takes too long. Processes increase velocity when done right. Getting it right is the challenge and requires a team/company to be flexible. Processes should be changed, added and removed when needed in order to maximize efficiency. In that framework, they are really your best friend and help everyone move faster.

Progress is misleading

The hallmark of any high functioning team is efficiency. Very high efficiency. Unfortunately, becoming efficient is something you have to pay for with time. Imagine we have teams A and B. Team A spends 40% of their sprint working on their infrastructure and operations. This includes things like build pipelines, tooling for managing cloud environments, automating repeatable processes and the works. Team B spends 10% of their time doing this kind of work. Both teams have 10 days to build out a feature. Due to team B’s policy, by day 3, they are already halfway through building their new feature whereas team A only gets halfway on day 6. Fortunately for team A, their time spent on operations has paid itself back, allowing them to finish and deploy by day 10. Team B finishes on day 7 but their deployment story isn’t nearly as sleek, adding 4 days. They finish on day 11. This is a contrived example but I’m trying to highlight that progress can be misleading. Because team B was already halfway on day 3, they felt like they were making rapid progress. Even if they are significantly slowed down later in the process, it does not feel nearly as bad because they are still making progress, albeit more slowly. What does not feel like progress is addressing tooling and infrastructure as it is adjacent to feature development. Team A’s policy is based on the belief that they will become faster over time by making it simpler and faster to get work done. The next time they build features, team A can finish in 4 days while team B might finish in 9. They are paying in time today so that they move faster tomorrow.

Moving slow in machine learning

In my experience, there are two things (amongst others) that slow down machine learning development:

  • Poor data management
  • Poor end-to-end (e2e) testing

Data management

Machine learning is all about data. Models come and go but data is here to stay. Without good infrastructure, tooling and management of your data, you’re going to spend ample amounts of time shouting at your screen in agony. Yes, I know you do already but it will be worse πŸ™ƒ. Just finding and gathering all the necessary data becomes painful, only for you to find out that the data is not as pretty and complete as you had assumed. Without the right data management, the work is pushed onto developers at the time of feature development. Data usually resides in disparate places, is not clean and is not easily queryable. Time is sunk into collecting this data but is not integrated into a system. This means unless that code is shared, anyone working with the same or similar data is going to repeat this process. Worst of all is if that process takes a number of hours to complete - extremely common with batch jobs as data scales. The last thing you want in this process is a high inertia in just retrieving that data every single time. Hours and hours of dev time lost. Addressing this is, once again, one of those trade-offs between feature development and operations/infrastructure work.

E2E testing

In many use cases, machine learning models form part of a greater system. To test whether an update has resulted in an improvement, you can’t test the machine learning model in isolation. You have to test the entire system’s performance. It’s entirely possible that an improvement in the model leads to a degradation in performance of the system. Why? Well let’s take an example scenario. You have a machine learning model A. You have downstream components, alpha and beta. Both alpha and beta are tuned to the current performance of model A. When you make an update to model A, your tuning becomes outdated. The entire system now performs worse than before. It’s easy to see the remedy - retune alpha and beta. Without e2e testing, you would have not discovered the performance is worse. In fact, you may not even be able to confirm the performance has actually improved. Trying to do it manually is a laborious process and prone to error. Good e2e testing will give you confidence in your updates and trust in your deployments.

Talking is pretty useless

I am a talker. I love to discuss new topics and really dive into them, tear them apart, turn them upside-down, shake them, spin them. You get the point πŸ˜†. I love retrospectives (as you can probably tell from this post), ideation sessions and strategy sessions. Over time I’ve realised one of my biggest flaws is that I can talk forever but often don’t translate that into action. That is a very dangerous world to exist in. Words are powerful when followed by action but are otherwise just vibrations of air. I am now concerned with talk leading to actionable outcomes which are acted upon. Without that, issues never truly get resolved and the problem will just come up again later, only for the cycle to repeat. As an individual, team or company, meetings and discussions should usually lead to practical, actionable outcomes and a plan to action them.

The End

If you’ve made it this far, I’m so sorry πŸ˜† but thank you for rocking with me. Sophomore year, let’s go!

Powered by Hugo & Kiss.