Claimed vs Measured

One thing first: you didn't see it because of a lack of fund knowledge.

Of the places where these projects went wrong, not one was about funds. They were all about "measurement" — how to make judgments on a finite dataset without deceiving yourself. That is an independent craft, nearly unrelated to knowing funds. Professional quant teams routinely crash in exactly these places; academia has named these errors, written papers about them, and invented dedicated tests.

Worse, these errors look like success while they are happening. Backtest returns rise, champions refresh, the logs fill with "found a better strategy." Every feedback signal is positive. By the time live trading tells you the truth, a long time has passed.

The 11 lessons below each teach one judgment tool you can use independently. After finishing them, you should be able to take any backtest report and ask the questions that reveal it for what it is.

Every strategy you try spends a little of the information in your data. Data is finite, so hypotheses cost money. The first step is always to compute your budget.

Three numbers to compute first

Plug in the real numbers from Fund_strategy_X:

Does the champion strategy pass? Put its t-statistic next to the threshold:

"Every day is a fresh judgment" — why that fails

The most natural rebuttal: You make a choice every day, each has an outcome 8 days later — isn't that 663 independent judgments?

This claim can be tested directly — if each judgment were independent, adjacent trades' returns should be uncorrelated. Measured:

Overfitting can be recognized without statistics. It has a very stable appearance. All eight signs below come from Fund_strategy_X, and each can be checked in five minutes.

Split the data in two: the training segment selects strategies; the validation segment is only reported on, never used in selection. This is the simplest and most powerful step in the whole methodology.

Split Fund_strategy_X at 2026-05-01 and run the same batch of strategies on both segments:

Why does this happen? Spread the champion's returns by quarter and it becomes obvious — all its returns come from a few bull-market quarters:

Both projects report a metric called oracle efficiency: your return ÷ the return of perfectly knowing each day's best fund. It answers "how far from the upper bound," but cannot answer "is there any skill at all" — in a bull market even a pure-beta strategy has a decent efficiency.

What you need is a zero-skill baseline: buy all qualifying funds equal-weight every day. The difference is skill.

The words "those same trading days" are the key. Selective strategies and daily strategies trade on different days; comparing against the full-period pool mean credits it with timing it never expressed.

Fund_strategy_X's objective function is cum_profit, defined as:

This definition contains no risk at all. Overlapping positions are summed as if independent trades, so "putting all the money on one fund at the same time" goes entirely unpunished.

So the search's champion grew into this:

More ironic still: both projects' oracle/engine.py already contain a real portfolio engine — simulate_strategy() with cash, T+2 settlement, a max position count, and per-fund exposure caps.

In both projects it was never called once.

Everything so far was bad news. This lesson is good news, and the only return source in this handbook that requires no forecasting skill.

When the signal is weak (such systems' information coefficients are typically 0.02~0.05), the risk reduction comes mainly from diversification, not from picking precision. Diversification is math, not judgment — it does not stop working when the market changes.

Below are results actually run on the Pareto project:

One side effect deserves its own paragraph. Lesson 5's pathology of "65 consecutive days buying the same fund" is eliminated outright by the single constraint "at most 2 simultaneous trades per fund" — and this constraint requires no judgment about the future at all.

When live trading loses money, there are only two possibilities, with completely different remedies:

There is only one way to tell them apart: reconcile. Take every live trade, replay it with the backtest engine on the same day, same strategy, and compare the returns on both sides.

The Pareto project designed three validation layers. Understanding their differences is the core of the whole methodology.

The L2 trap: a fixed calendar smuggles in seasonality

Holding out the fixed months of January, May, and September looks reasonable — the rule is explicit and auditable. But the "out-of-sample" carved out this way is collinear with seasonality.

The QDII pool's measured results expose this:

Two consequences: first, the "holdout-month return > 0" threshold is passed by everyone on QDII — zero screening power; second, worse — it rewards strategies that happen to load more heavily on those three months.。

The fix is to replace the fixed calendar with random/rotating blocks (combinatorial purged cross-validation). Pareto already uses exactly this blocking mechanism for PBO — port it over.

These two metrics do not evaluate strategies; they evaluate the process by which you selected them. That is what makes them special.

PBO — probability of backtest overfitting

Slice time into many blocks and repeatedly "pick the best strategy on half the blocks, then see its rank on the other half." PBO = the probability that the picked champion ranks below the median on the other half.

DSR — deflated Sharpe ratio

Folds "how many hypotheses you tried" into the significance. The more you try, the less the same Sharpe is worth. DSR is a probability: the probability that this strategy's true Sharpe exceeds 0. Convention requires > 0.95.0.95。

If compute were free, you would of course want to search without limit. This lesson explains why that is entirely possible, and what conditions it requires.

The key: DSR and PBO do not punish "how much you searched," but "how many selections you made on the same data." These two things can be separated.

Companion: turning logs from "things tried" into "hypotheses tested"

One more way to make unlimited search and meaningful statistics coexist: do inference at the "mechanism" level, not the "parameter variant" level.

Require every candidate to declare a mechanism_id from a controlled vocabulary — adding a new mechanism is a slow action requiring a written economic rationale. Then:

Within a family, variation can be unlimited (free, no cost); statistical inference happens at the family level (bounded, meaningful). One precondition: family representatives must not be picked by training performance, or within-family multiple testing returns — use equal weight within the family, or let L3 performance decide.

The four things every hypothesis should state clearly: economic mechanism / expected direction / expected magnitude / failure condition. No more than 2 parameters. Write it down and file it before running the data.

The last lesson comes from an easily overlooked observation: what exactly is the "8-day holding period"?

It is the redemption-fee-free eligibility day, not a forced exit day. At maturity the position becomes eligible to be reclaimed, not required to be. If the strategy still wants this fund today, you do nothing at all — just keep holding it.

But the current system deleted this decision. In the backtest, future_return is a fixed 8-day forward return — every candidate is a forced 8-day round trip. Nowhere in the candidate space is there a "continue holding or switch" choice.

This explains the 65-day mystery

Back to Lesson 5's "65 consecutive days buying the same fund." We previously classified it as a concentration pathology. But viewed through the correct framework —

Conversely: if continuing a hold does not consume new quota, then each day's ¥4,000 naturally goes elsewhere — diversification appears for free, with no enforcement needed.

Now look again at the two worst live trades: 5.4055 → 4.0572 (−24.9%), 5.0748 → 3.9466 (−22.2%). Forced to sell into the hole on day 8. With a "continue holding or switch" decision, that would have been optional.

Ask these, in order, of any backtest report handed to you. The ones it cannot answer are where the problems are.

After all this, the most important adjustment may be expectations.

An honest goal: beat the equal-weight fund pool by 2~5%/year over the long run, with a significantly smaller max drawdown. That is already a solid result. Not 90% annualized.

Fund_strategy_X's root cause on that path was its objective function assuming from day one that "90% annualized is reachable" — so all its search headed toward the noisiest directions.

What you can reliably get, ranked by reliability

1. Diversification — pure math, no forecasting needed. Measured to cut drawdown by 42%~54%.
2. Not deploying overfit things — requires wiring PBO/DSR/L3 into position sizing.
3. Zero fee rates — already yours; don't waste them with high turnover.
4. Fund-picking alpha — fourth, and the least reliable.