Audience requirements: high school math; previous blog post. Programming knowledge is not needed; it will just help you appreciate the code example.

# Review of previous example

This is the simplistic example from the previous post^{1}.

To get as close as possible to a target (a.k.a. ideal) portfolio of 10% x, 50% y, 40% z,

minimize^{2}:

- |10 – x| + |50 – y| + |40 – z| (
*distance from ideal portfolio*)

subject to:

- 0 <= x <= 100 (
*an asset class can be between 0 and 100% of the portfolio*) - 0 <= y <= 100
- 0 <= z <= 100
- x + y + z = 100 (
*all holdings sum up to the entire portfolio*)

Similar constraints are color-coded into groups, for clarity. If the mathy notation puts you off, you can just read the text descriptions in *italics*, which are only mentioned once per group.

# New requirements

We will now augment the original example to illustrate a new point.

Assume that we also want to prevent too much trading (buying or selling)^{3} of y or z (but not x) – e.g. no more than 30% of the total portfolio value.

This requires rewriting the linear program, because the variables are currently too coarse; there’s nothing that represents buying or selling yet.

# Case 1: without using our infrastructure

Rewrite the final amount for x (and similarly for y and z) as

x_{final} = x_{initial} + x_{buy_amount} – x_{sell_amount}

Either buy or sell amount will be positive, but not both; we can’t both buy and sell something. They can both be 0, however, if we don’t trade.

So we’ll now have 9 variables instead of 3.

- x
_{f}= x_{i}+ x_{b}– x_{s} - y
_{f}= y_{i}+ y_{b}– y_{s} - z
_{f}= z_{i}+ z_{b}– z_{s}

In practice, the initial amounts are known ahead of time, so they can be constant numbers. Using these 6 variables (x_{b}, x_{s}, y_{b}, y_{s}, z_{b}, z_{s}) instead of the original 3 (i.e. without using x_{f}, y_{f}, or z_{f}), we can rewrite our LP as

Minimize:

|10 – (7 + x_{b} – x_{s})| + |50 – (52 + y_{b} – y_{s})| + |40 – (41 + z_{b} – z_{s})| (*distance from ideal portfolio*)

subject to:

- 0 <= 7 + x
_{b}– x_{s}<= 100 (*an asset class can be between 0 and 100% of the portfolio*) - 0 <= 52 + y
_{b}– y_{s}<= 100 - 0 <= 41 + z
_{b}– z_{s}<= 100 - (7 + x
_{b}– x_{s}) + (52 + y_{b}– y_{s}) + (41 + z_{b}– z_{s}) = 100 (*all holdings sum up to 100%*) - y
_{b}+ y_{s}+ z_{b}+ z_{s}<= 30 (*prevent too much trading, i.e. more than 30% of portfolio*) - x
_{b}>= 0 (*buy and sell amounts are non-negative*) - x
_{s}>= 0 - y
_{b}>= 0 - y
_{s}>= 0 - z
_{b}>= 0 - z
_{s}>= 0

This looks quite complicated.

It is possible to simplify these a bit. E.g.

0 <= 7 + x_{b} – x_{s} <= 100

can be rewritten as

-7 <= x_{b} – x_{s} <= 93

However, the point here isn’t to make this simpler to the external solver __software__ that will solve this. It doesn’t care – computers are good at this stuff. It will still come up with the same solution.

The real concern is that the system is now more complicated, so such simplifications won’t help. Any code that refers to the final amount – such as the code that builds the objective function – will now need to be aware of (and updated to use) the new “initial + buy – sell” way of representing the final amount. Since code is written by humans, the extra complication makes it harder to understand, fix, and improve on that code.

# Case 2: without our infrastructure, but slightly better

We could make things simpler by including and actually using x_{f}, y_{f}, and z_{f} (instead of their expanded forms), but we need additional constraints._{ }

Minimize

|10 – x_{f}| + |50 – y_{f}| + |40 – z_{f}| (*distance from ideal portfolio*)

subject to:

- 0 <= x
_{f}<= 100 (*an asset class can be between 0 and 100% of the portfolio*) - 0 <= y
_{f}<= 100 - 0 <= z
_{f}<= 100 - x
_{f }+ y_{f }+ z_{f}= 100 (*all holdings sum up to 100%*) - y
_{b}+ y_{s}+ z_{b}+ z_{s}<= 30 (*prevent too much trading, i.e. more than 30% of portfolio*) - x
_{b}>= 0 (*buy and sell amounts are non-negative*) - x
_{s}>= 0 - y
_{b}>= 0 - y
_{s}>= 0 - z
_{b}>= 0 - z
_{s}>= 0 - x
_{f}= x_{i}+ x_{b}– x_{s }(*final amount = initial + buy – sell*) - y
_{f}= y_{i}+ y_{b}– y_{s} - z
_{f}= z_{i}+ z_{b}– z_{s}

Constraints #12 – #14 are now new. However, now we have 14 constraints instead of 11. We just introduced 2 new problems, which are immaterial in this small example, but matter once the problem gets bigger.

## Problem 1: performance / speed

The underlying software we use to find the best solution may be smart enough to understand that the problem is no more complicated now than before, but we can’t count on it. In most cases, those extra variables will slow things down.

## Problem 2: simplicity

The programmer needs to be aware of all these variables; nothing is hidden from her. For example, she does not care in this example about the buy or sell amounts per se; she only cares about gross trade amounts (buy + sell). But she can’t refer to the trade amount without knowing that it consists of the sum of 2 variables, buy + sell.

A fundamental engineering principle is for components to expose only the necessary details. For instance, an HVAC contractor can install an a/c system and the right ducts without knowing the details of how the a/c system works internally. Similarly, hiding details from individual software components makes building the whole system easier.

# Case 3: Using our infrastructure

A simpler way is to use x_{f}, y_{f}, and z_{f }when describing the problem, but not make the underlying solver aware of this^{4}.

Minimize

|10 – x_{f}| + |50 – y_{f}| + |40 – z_{f}| (*distance from ideal portfolio*)

subject to:

- 0 <= x
_{f}<= 100 (*an asset class can be between 0 and 100% of the portfolio*) - 0 <= y
_{f}<= 100 - 0 <= z
_{f}<= 100 - x
_{f }+ y_{f }+ z_{f}= 100 (*all holdings sum up to 100%*) - y
_{b}+ y_{s}+ z_{b}+ z_{s}<= 30 (*prevent too much trading, i.e. more than 30% of portfolio*) - x
_{b}>= 0 (*buy and sell amounts are non-negative*) - x
_{s}>= 0 - y
_{b}>= 0 - y
_{s}>= 0 - z
_{b}>= 0 - z
_{s}>= 0

The improvement is very subtle, but important. Case #3 combines the benefits of case #1 (only 11 constraints, vs 14) and case #2 (easier to understand). The point is that the necessary translations of “high-level variables” (e.g. x_{f}) are done automatically and hidden from the programmer.

For non-programmers: the least bad analogy we can think of is asking someone to cook the best meal possible, subject to what’s in the fridge and pantry. This is a high-level optimization; the cook has to solve the low-level optimization based on using a maximum of 1 lb of chicken, 2 lbs of potatoes, etc.

For programmers: think of it as a compiler; the underlying 3rd party optimization solver sees something closer to case #1 – but with 11 constraints instead of 14, which is more efficient. If you read this far and are wondering how, you may be a great fit for our company, so shoot us an email!

# Why this all matters

A central tenet of the Rowboat Advisors approach is to evaluate intelligently the tradeoffs between the different improvements that can be made to a portfolio. As more and more of these “sub-objectives” come into play (reduced taxes, reduced fees, tracking a target portfolio, etc.), the extra complexity becomes a real problem – and, in practice, far more so than in this simple example.

For programmers: think of writing a parser in java vs. bash. For a simple parser (e.g. for csv), bash will be worse than java, but will at least suffice. However, for a more complicated parser (e.g. for xml), bash is not just worse, but is effectively unusable.

# Actual code example

This is the Java code that will tell us the cheapest way to make an acceptable 200 ml rum-and-coke drink.

@Test public void rumAndCoke() { HighLevelLPBuilder builder = makeBuilder().withHumanReadableLabel(label("minimize cost of drink")); HighLevelVariablesBuilder varBuilder = builder.getHighLevelVariablesBuilder(); // Amounts of rum and coke that we will use in the cocktail (in liters). // Customers will complain if there's not enough rum in the cocktail; use at least 50 ml // We can use any amount of coke, but of course it has to be positive. Variable rum = varBuilder.addConstrainedRawVariable("rum", Range.atLeast(0.05)); Variable coke = varBuilder.addConstrainedRawVariable("coke", Range.atLeast(0.0)); SuperVar cost = varBuilder.addSuperVar(generalSuperVarWithoutAddedConstraints( label("cost of drink (rum costs $4.5 / liter, and coke $1.5)"), disjointHighLevelVarExpression(4.5, rum, 1.5, coke))); SuperVar sugar = varBuilder.addSuperVar(generalSuperVarWithoutAddedConstraints( label("total sugar in grams (rum is 25 / liter; coke about 40)"), disjointHighLevelVarExpression(25, rum, 40, coke))); LinearOptimizationProgram lp = builder .setObjectiveFunction(simpleLinearObjectiveFunction(singleVarExpression(cost))) .addConstraint(highLevelVarConstraint(label("drink must be exactly 200 ml"), sumOfDisjointHighLevelVars(rum, coke), EQUAL_TO_SCALAR, constantTerm(0.2))) .addConstraint(highLevelVarConstraint(label("drink can't be too sweet, so no more than 7 grams of sugar"), sugar.getHighLevelVarExpression(), LESS_THAN_SCALAR, constantTerm(7))) .addConstraint(highLevelVarConstraint(label("drink can't be too cheap, so no less than $0.60"), cost.getHighLevelVarExpression(), GREATER_THAN_SCALAR, constantTerm(0.60))) .build(); System.out.println(lp); AllVariablesAndDoubles solution = optimizer.minimize(lp).get().getVariablesAndOptimalValues(); BiConsumer asserter = (expected, var) -> assertEquals(expected, evaluator.evaluateHighLevelVar(var, solution), 1e-8); asserter.accept(0.1, coke); // 0.1 = 100 ml asserter.accept(0.1, rum); asserter.accept(doubleExplained(6.5, 25 * 0.1 + 40 * 0.1), sugar); // This solution makes sense. The % of coke // * can't go below 50%; we can't have a cheaper price, since a 50-50 mix already gives $0.60 (min allowed) // * shouldn't go above 50%: we want to minimize cost, and less coke (=> more rum) increases cost asserter.accept(doubleExplained(0.60, 4.5 * 0.1 + 1.5 * 0.1), cost); }

The code also prints out the LP after the infrastructure does the translation. Notice how there are only 2 variables (x0 for *rum*, x1 for *coke*), even though the source code mentions *cost* and *sugar* in the objective and constraints.

[LOPI minimize cost of drink supervars: [[x0:rum], [x1:coke]] minimize: 4.5 [x0:rum] + 1.5 [x1:coke] subject to constraints: drink must be exactly 200 ml : 0.2000000000 == + [x0:rum] + [x1:coke] drink can't be too sweet, so no more than 7 grams of sugar : 7.0000000000 > + 25.0000000000*[x0:rum] + 40.0000000000*[x1:coke] drink can't be too cheap, so no less than $0.60 : 0.6000000000 < + 4.5000000000*[x0:rum] + 1.5000000000*[x1:coke] variable ranges: rum [0.05‥+∞) coke [0.0‥+∞) Initial point: Optional.empty LOPI]

This is great! Our description of the problem used *cost* and *sugar*, which are easy to understand, while the actual LP that will get solved only mentions 2 variables (*rum*, *coke*) instead of 4 (*rum*, *coke*, *cost*, *sugar*), which will be faster to solve^{5}.

# Summary

Our infrastructure allows us to build sophisticated investment logic in a simple fashion. We described one of the many features: high-level variables. This allows for investing logic that’s easier to understand and faster to execute.

# Notes

- We will not use c, s, b for cash / stocks / bonds, because b and s may be confused with ‘buy’ and ‘sell’.
- Absolute values are not directly supported by linear optimization. A future post will explain how we do this.
- In practice, we can’t always do that; we need to allow full investment from an all-cash account (a lot of buying), and withdrawals (a lot of selling)
- AMPL is a very well established modeling language, designed by AT&T Bell Labs in the 80s by (among others) the famous Brian Kernighan. It allows the concept of a high-level variable as described in this post – plus a ton more, though not relevant to us. However, it does not have some features we need (described in future posts). There are also several other reasons why it’s not a good fit for us, which are beyond the scope of this post. Our infrastructure was actually built before finding out that AMPL exists. The downside is that we missed out on a potential opportunity to use something existing (although, in the end, we can’t); the upside is the validation that the infrastructure we built was in the right direction.
- The difference in this example will be insignificant, but with hundreds or thousands of variables, it matters.

## One thought on “Rowboat Advisors optimization infrastructure, part 1: high level variables”