How to solve coding problems

Every single line of code ever written was ultimately made with one purpose in mind: to solve problems. No matter what you do, you are solving problems on several scales at once.

A small one-liner solves a problem that makes a function work. The function is needed for a data processing pipeline. The pipeline is integrated into a platform that enables a machine learning-driven solution for its users.

Problems are everywhere. Their magnitude and impact might be different, but the general problem-solving strategies are the same.

As an engineer, developer, or data scientist, being effective in problem-solving can supercharge your results and put you before your peers. Some can do this instinctively after years of practice, and some have to put conscious effort to learn it. However, no matter who you are, you can, and you must improve your problem-solving skills.

Having a background in research-level mathematics, I had the opportunity to practice problem-solving and observe the process. Surprisingly, this is not something which you have to improvise each time. Rather, a successful problem-solver has several standard tools and a general plan under their belt, adapting as they go.

In this post, I aim to give an overview of these tools and use them to create a process you can follow at any time. Let's place ourselves into the following scenario to make the situation realistic: we are deep learning engineers working on an object detection model. Our data is limited, so we need to provide a solution for image augmentation.

Image augmentations. Source: albumentations README

Augmentation generates new data from the available images by applying random transformations like crops, blurs, brightness changes, etc. See the figure above from the readme of the fantastic albumentations library.

You need to deliver the feature by next week, so you need to start working on it right away. How to approach the problem? (As a mathematician myself, my thinking process is heavily influenced by the book How to Solve It by George Pólya. Although a mathematical problem is different from real-life coding problems, this is a must-read for anyone who wishes to get good in problem-solving.)

Step 0: Understanding the problem

Before attempting to solve whatever problem you have in mind, some questions need to be answered. Not understanding the details can lead to wasted time, and you don't want that. For instance, it is good to be clear about the following.

What is the scale of the problem? In our image augmentation example, will you need to process thousands of images per second in production, or is it just for you to experiment with some methods? If a production-grade solution is required, you should be aware of this in advance.
Will other people use your solution? If people will work with your code extensively, you must put significant effort into code quality and documentation. However, you don't need to invest a lot of time if the solution is for your use only. (I already see people disagreeing with me :) However, I firmly believe in minimizing the amount of work. So, if you only need to quickly try out an idea and experiment, feel free not to consider code quality.)
Do you need a general or a special solution? You can waste a lot of time implementing features no one will ever use, including you. In our example, do you need a wide range of image augmentation methods or just vertical and horizontal flips? In the latter case, flipping the images in advance and adding them to your training set can also work, which requires minimal work.

A good gauge of your degree of understanding is your ability to explain and discuss the problem with others. Discussion is also a great way to discover unexpected approaches and edge cases.

When you understand your constraints and have a somewhat precise problem specification, it is time to get to work.

Step 1. Is there an existing solution?

The first thing you must always do is to look for existing solutions. Unless you are pushing the very boundaries of human knowledge, someone else had already encountered this issue, created a thread on Stack Overflow, and possibly wrote an open source library around it.

Take advantage of this. There are several benefits of using well-established tools instead of creating your own ones.

You save a tremendous amount of time and work. This is essential when operating under tight deadlines. (One of my teachers used to ironically say that "you can save an hour of Google search with two months of work". Spot on.)
Established tools are more likely to be correct. Open source tools are constantly validated and checked by the community, and thus, they are less likely to contain bugs. (Of course, this is not a guarantee.)
Less code for you to maintain. Again, we should always strive to reduce complexity, and preferably the amount of code. If you use an external tool, you don't have to worry about its maintenance, which is a great deal. Every line of code has a hidden cost of maintenance, to be paid later. (Often when it is the most inconvenient.)

Junior developers and data scientists often overlook these and prefer always to write everything from scratch. (I certainly did, but I quickly learned to know better.) The most extreme case I have seen was a developer who wrote his own deep learning framework. You should never do that unless you are a deep learning researcher and know how to do significantly better than the existing frameworks.

Of course, not all problems require an entire framework. Maybe you are just looking for a one-liner. Looking for existing solutions can be certainly beneficial, though you need to be careful in this case. Finding and using code snippets from Stack Overflow is only acceptable if you take the time to understand how and why it works. In the worst case, not doing so may result in unpleasant debugging sessions later or even severe security vulnerabilities.

For these smaller problems, looking for an existing solution consists of browsing tutorials and best practices. In general, there is a balance between ruthless pragmatism and outside-the-box thinking. When you implement something using established best practices, you are doing a favor for the developers who will use and maintain that piece of code. (Often including you.)

There is an existing solution. What next?

Suppose that you have followed my advice on your path towards delivering image augmentation for your data preprocessing pipeline, looked for existing solutions, and found the awesome albumentations library. Great! What next?

As always, there is a wide range of things to consider. Unfortunately, just because you have identified an external tool that can be a potential solution, it doesn't mean that it will suit your purposes.

Is it working well and supported properly? There is one thing worse than not using external code: using buggy and unmaintained external code. If a project is not well documented and not maintained, you should avoid it. For smaller problems, where answers generally can be found on Stack Overflow, the working well part is essential. (See the post I have linked above.)
Is it adaptable directly? For example, if you use an image processing library that is not compatible with albumentations, you must do additional work. Sometimes, this can be too much, and you have to look for another solution.
Does it perform adequately? If you need to process thousands of images per second, performance is a factor. A library might be convenient to use, but if it fails to perform, it has to go. This might not be a problem for all cases (for instance, if you are looking for a quick solution to do experiments), but if it is, it should be discovered early, before putting much work into it.
Do you understand how it works and what are its underlying assumptions? This is especially true for using Stack Overflow code snippets for the reasons I have mentioned above. For more complex issues like the image augmentation problem, you don't need to understand every piece of external code line by line. However, you need to be aware of the requirements of the library, for instance, the format of the input images.

This, of course, is applicable only if you can find an external solution. Read on to see what to do when this is not the case.

What if there are no existing solutions?

Sometimes you have to develop a solution on your own. The smaller the problem is, the more frequently it happens. These are great opportunities for learning and building. This is the actual problem-solving part, the one that makes many of us most excited.

There are several strategies to employ, and all of them should be in your toolkit. If you read carefully, you'll notice that there is a common pattern.

Can you simplify? Sometimes, it is enough to solve only a special case. For instance, if you know that the inputs for your image augmentation pipeline will always have the same format, there is no need to spend time processing the input for several cases.
Isolate the components of the problem. Solving one problem can be difficult, let alone two at the same time. You should always make things easy for yourself. When I was younger, I thought that solving hard problems was The Way to get dev points. Soon, I have realized that the people who solve hard problems always do it by solving many small ones.
Can you solve for special cases? Before you go and implement an abstract interface for image augmentation, you should work on a single method to add into your pipeline. Once you discover the finer details and map out the exact requirements, you can devise a more general solution.

In essence, problem solving is an iterative process where you pick the problem apart step by step, eventually reducing it to easily solvable pieces.

Step 2. Break the solution (Optional)

There is a common trait which I have noticed in many excellent mathematicians and developers: they enjoy picking apart a solution, analyzing what makes them work. This is how you learn and how you build robust yet simple code.

Breaking things can be part of the problem solving process. Going from a special case to a general one, you usually discover solutions by breaking what you have.

When it is done

Depending on the magnitude of the problem itself, you should consider open-sourcing it if you are allowed. Solving problems for other developers is a great way to contribute to the community.

For instance, this is how I have built modAL, one of the most popular active learning libraries for Python. I started from a particular problem: building active learning pipelines for bioinformatics. Since building complex methods always require experimentation, I needed a tool that enabled rapid experimentation. This wasn't easy to achieve with the available frameworks at the time, so I slowly transformed my code into a tool that others could easily adopt.

What used to be "just" a solution became a library with thousands of users.

Conclusion

Contrary to popular belief, effective problem solving is not the same as coming up with brilliant ideas all the time. Instead, it is a thinking process with some well-defined and easy-to-use tools that anyone can learn. Smart developers use these instinctively, making them look like magic.

You can improve problem-solving skills with deliberate practice and awareness of thinking habits. There are several platforms where you can find problems to work on, like Project Euler or HackerRank. However, even if you start applying these methods to issues you encounter during your work, you'll see your skills improve rapidly.