Why Consider a Hadoop to Databricks Lakehouse Migration?
Hadoop offers the option to maintain huge on-prem workloads but enterprises need to migrate this data into cloud-based managed services...
The hardest engineering, scientific and statistical problems of our times are being tackled by means of mathematical optimization. Operating at the core of most statistical techniques and machine learning methods, optimization is what allows algorithms to transform large numbers of random variables into understandable and actionable observations.
However, the optimization concept is by no means only limited to computer scientists, engineers, and statisticians. On the contrary, we are all exposed on a daily basis to different kinds of optimization. Search engines trying to figure out the best responses to search queries? Heavily relying on optimization. Location-based driving assistants trying to find the best route across town? You guessed it, all powered by optimization.
In the business world, optimization models also play a crucial role: they are a proven way to improve strategic, tactical, and operational processes. They give human decision-makers the opportunity to tackle complex problems and answer questions that are impacted by a large number of factors. Building mathematical models based on historical and real-time data provides an objective and scientific foundation for day-to-day plans, predictions, and decisions.
A large variety of business problems can be formulated as optimization problems. However, this requires an appropriate problem formulation, that identifies the sources of uncertainty (known as random variables), appropriate modeling methods, and deployment solutions. In order to get started, it is necessary to identify the following elements, common to all optimization problems:
The objective function specifies what needs to be accomplished after solving the optimization problem. What are the quantitative goals and how are the success measures calculated? In the case of simple problems, only a single objective function is specified, whereas more complex problems require identifying multiple objective functions.
The decision variables specify what data points are available for tackling the problem. These can be of various types (categorical, continuous, binary) and they have a direct impact on the result of the objective function. The number of decision variables has a direct impact on the complexity of the problem.
The constraints correspond to conditions that must be satisfied when solving the problem. They represent business restrictions and can take many shapes (legal, economic, physical), according to the nature of the problem. Naturally, optimization models are successful when they reach the objective function and also satisfy all constraints.
All these elements are necessary for optimization models. They can be seen as the translation into mathematical formulas of the key characteristics of a business problem. There are many use cases for optimization models since they can be employed to improve multiple aspects of a business: improving processes, increasing automation, enabling efficient resource management.
Let us take a closer look at an optimization use case in an industry that does not run short of optimization problems.
Supply chain management is built entirely around the concept of optimization. From fleet management to distribution networks and production management, many operational problems in the supply chain can be seen as an optimization challenge.
One modern challenge for the supply chain is the implementation of the Lean and Green paradigm. With corporations seeing increased benefits (economic, environmental, and social) when accounting for their environmental impact, lean and green practices have become an indispensable part of modern supply chain strategies.
As part of an automotive corporate group, a complex supply chain network includes a variety of companies. There are raw material suppliers, plants, assemblers, collection, and disassembly centers. Each is responsible for specific kinds of processing all throughout the transformation from raw materials to finished products. Similarly, each of the different entities has a different capacity, processing, and delivery constraints. Ensuring that such a complex ecosystem stays functional requires the synchronization of all parties involved.
The common objective is to identify and eliminate activities that do not add value to the ecosystem and to remove those that result in unnecessary spending and waste. Also known as non-value activities, they consist of “overproduction, waiting, transportation, inappropriate processing, unnecessary inventory, unnecessary motion and defects in manufacturing”.
Understanding the intricate relations in such a complex ecosystem and achieving the lean and green objective would be impossible without a way to mathematically formulate and tackle the problem. This is where optimization modeling comes into play.
Without going into too much mathematical detail, let us take a look at the main optimization components discussed above. The model chosen to represent the network above is a linear programming model, in which the requirements are represented by linear relationships. Similarly, the model also uses a linear objective function, which is subject to linear equality and inequality constraints.
The objective function is a weighted sum of six other functions, which need to be minimized by the optimization model. The list includes:
Each function is in turn specified as a linear function of a variety of factors. Here is the formula for the CO2 emission costs, just to feed your curiosity:
We know, we promised not to go into detail. But this looks amazing, doesn’t it? That’s why our team is here for you: so you don’t have to deal with it.
The decision variables and model parameters include a few hundred factors, specifying anything from distances, prices, transportation duration & costs to opening hours. The following are a few examples:
The constraints can refer to one of the following categories: capacity, demand, transportation. More complex constraints are known as balance constraints (Kirchoff Law), which ensure that the sum of flows coming into a node is equal to the sum of flows going out of that node. Once more, a few hundred formulas represent these constraints. Listing them all here is beyond the scope of this article.
The use case presented above is by no means purely theoretical. A multitude of industry players is implementing an adapted version of the Lean and Green optimization model. While the big players in the automotive industry are leading research efforts to better optimize their supply chains, companies of all sizes engage in optimization modeling across a variety of business areas. From capacity planning to network design, human resource management, inventory, and transportation management, mathematical optimization provides a way to better understand complex, intricate ecosystems.
Optimization is by no means restricted to specific industries. Real-world examples can be found in a variety of sectors: from internet search to telecommunications, chemical processing, and even water resource management. Optimization-based solutions help model uncertainty and bring a competitive advantage to decision-makers. They also offer a solid foundation for an objective, scientifically-proven decision-making process.
Getting started with mathematical optimization is possible for businesses of all sizes. Get in touch with our team of engineers and PhDs to better understand your business problems. We’ll pin down those that can be tackled by means of optimization and we’ll assist you on every step of the journey.
Schedule 15-min with a Blue Orange Digital Solution Architect to discuss which option is right for your data sources and future goals.
Josh Miramant is the CEO and founder of Blue Orange Digital, a data science and machine learning agency with offices in New York City and Washington DC.
Miramant is a popular speaker, futurist, and a strategic business & technology advisor to enterprise companies and startups. As an example of thought leadership, Miramant has been featured in IBM ThinkLeaders, Dell Technologies, Global Banking & Finance Review, the IoT Council of Europe, among others. He can be reached at email@example.com.
Blue Orange Digital is recognized as a “Top AI Development and Consultant Agency,” by Clutch and YahooFinance, for innovations in predictive analytics, automation, and optimization with machine learning in NYC.
They help organizations optimize and automate their businesses, implement data-driven analytic techniques, and understand the implications of new technologies such as artificial intelligence, big data, and the Internet of Things.