"When you first start off trying to solve a problem, the first solutions you come up with are very complex, and most people stop there. But if you keep going, and live with the problem and peel more layers of the onion off, you can often times arrive at some very elegant and simple solutions. Most people just don't put in the time or energy to get there."
Steve Jobs points to a pitfall that many of us can fall into. Particularly PhD's who are used to operating in an environment where you breath in complexity with a group of peers on a daily basis. But in industry this is not why you are getting paid. You are getting paid to solve a problem and offer a solution that people can take it and run with it!
ISO 42010, has helped me solve problems in a more systematic way. The idea is that every organisation, project, problem , software, you name it has an architecture regardless of us being aware of it or not.
My problem solving flow is following:
STAKEHOLDERS: First start by identifying the stakeholders in the problem that you are trying to solve. Who is going to use your product or results of your analysis. What is it that they are doing? Having a clear understanding of the receiving end leads you to the next step, "concerns".
CONCERNS: Things that keep your stakeholders awake at night is their concern! Why is the sales down? Is there any anomalies in our service usage? Whether you are doing a descriptive or prescriptive analysis you have to answer your stakeholders questions. Identifying their explicit and implicit questions and addressing them in your solution and analysis will make your solution more usable and reduces the number of iterations that are necessary to get to the final outcome. The golden question you can ask yourself is: What does the stakeholder need to make a decision? Your analysis should answer this within the constraints of your data.
ARCHITECTURE VIEWPOINT: Once you know the concerns and have envisioned the end result, you can trace back from the questions and delineate your solution. Your solution can be very complex or very simple. This is the step where you do the correct transformation to the data to extract your answers and test your hypothesis, and validate your results. But always have in mind that the value is not driven from the complexity of your solution but the integrity of the outcome.
ARCHITECTURE VIEW: Imagine you have built a marvelous skyscraper (aka your solution) and I want to buy a unit in it. Do I need to know where you bought the concrete and how much per ton? You have to decide what is the write angle for the buyer (read stakeholder) to look at this magnificent edifice! So present your result by showing the stakeholder the answer to their questions with enough of the complexity of your solution that help them play with the results and try to poke holes in it on their own time. This way the stakeholder can check the foundations of your solution and understand it better before making a decision based on it?
When faced with a new problem, what are your ways of solving it?