Skip to content

Portfolios

The portfolios concept owns portfolio construction workflows. It connects assets, signals, rebalance strategies, portfolio weights, portfolio metadata, and portfolio value time series.

Position-aware cash flows and valuation

For the complete engine contract, source schemas, configuration, ledger fields, restart semantics, extension hooks, and operational limitations, read Position-Aware Portfolio Accounting.

The existing PortfoliosUpdater remains the default weight-return engine. Omitting PortfolioBuildConfiguration.accounting_configuration, or setting it to None, does not add dependencies or change the serialized legacy configuration. Supplying accounting configuration selects a separate history; passing that configuration to PortfoliosUpdater is rejected instead of falling back to weight-only valuation.

The opt-in implementation follows ADR 0042:

signal weights + execution-market observations
                         |
                         v
        RebalanceStrategy + execution/cost models
                         |
                         v
lifecycle observations -> PortfolioEngine <- price/FX observations
                         |
                         v
             PortfolioAccounting (pure reducer)
                         |
                         v
         PortfolioEventLedgerStorage (authority)
                         |
                         v
      state, cash-flow, and portfolio-value projections

PortfolioAccounting is intentionally usable in offline tests and examples. PortfolioEngine is the TimeIndexTableUpdater boundary: it declares the signal, strategy observation, lifecycle, valuation, and FX sources and publishes the long event ledger. Portfolio execution is simulated only by the configured RebalanceStrategy; no broker/account execution source is accepted. The ledger grain is (time_index, portfolio_identifier, event_identifier, event_revision, record_identifier). The migration provider creates that table plus additive state and completed-cash-flow projection schemas. The pure reducer can also reconstruct positions, cash, obligations, applied event revisions, and the latest NAV from a complete active ledger after validating its digest and state chain.

DividendCashFlowModel recognizes a receivable using the position at the entitlement time. Payment later reduces that receivable and increases settled cash. It therefore works when the originating shares have already been sold, and settlement does not recognize the dividend a second time.

MarketPriceValuationModel values signed positions, cash, and obligations in the configured valuation Asset. Foreign values require an explicit observation with grain (time_index, base_asset_identifier, quote_asset_identifier) and a strictly positive rate; it does not infer an inverse pair or silently replace a missing rate. Price and FX selection is bounded by maximum_staleness.

Use the public extension boundary for unusual economics:

from msm_portfolios.accounting import LifecycleEventModel, PositionCashFlowModel

A module-level directly injected model selects source-backed candidates and returns a flat, columnar EventBatch. The engine owns canonical identifiers, validation, state application, NAV reconciliation, and persistence. A custom model cannot replace those operations or write a projection directly.

Three offline examples require no backend writes:

uv run --extra portfolios python \
  examples/msm_portfolios/portfolio_cashflows_and_fx_valuation_example.py
uv run --extra portfolios python \
  examples/msm_portfolios/portfolio_custom_cashflow_model_example.py

uv run --extra portfolios python \
  examples/msm_portfolios/portfolio_perpetual_funding_example.py

The first buys and later sells a EUR-denominated stock in a USD portfolio, recognizes a EUR dividend before the sale, and settles it afterward. The second defines and injects a user-owned vectorized EUR usage royalty model. The third models a variation-margined linear perpetual whose same-time funding event is applied before target sizing.

Scope

Portfolios answer these questions:

  • Which assets are eligible for a portfolio?
  • Which valuation source provides the asset values used for returns?
  • Which signals produce target weights?
  • Which rebalance strategy converts signals into portfolio weights?
  • Which updaters store canonical portfolio values, signal weights, and portfolio weights?
  • Which metadata identifies portfolios, signals, and rebalance strategies?

Primary Modules

  • msm_portfolios.configuration: portfolio configuration models.
  • msm_portfolios.time_index: canonical updaters for portfolios, portfolio weights, signal weights, storage initialization, and identity helpers.
  • msm_portfolios.time_index.contract_instances: factories that build a provider-owned instance of the PortfoliosTS, PortfolioWeightsTS and SignalWeightsTS contracts (ADR 0047).
  • msm_portfolios.services.portfolio_datasets: the Markets catalog of which instance holds each portfolio's datasets, with register, reconcile and list services.
  • msm_portfolios.rebalance_strategy: rebalance strategy base classes and built-in strategies.
  • msm.models.portfolios: core SQLAlchemy MetaTable declaration for portfolio identity/reference data.
  • msm_portfolios.models.portfolios: SQLAlchemy MetaTable declarations for portfolio descriptive metadata.
  • msm_portfolios.models.rebalancing and msm_portfolios.models.signals: SQLAlchemy MetaTable declarations for rebalance strategy metadata and signal metadata.
  • msm.api.portfolios: typed row API for core Portfolio identity rows.
  • msm_portfolios.api.portfolios: typed row API for PortfolioMetadata.
  • msm_portfolios.api.market_metadata: typed row APIs for SignalMetadata and RebalanceStrategyMetadata.
  • msm.services.portfolios: service helpers for portfolio identity rows.
  • msm_portfolios.contrib: contributed price and signal updaters.
  • msm_portfolios.utils: small shared logging and time constants only.
  • msm_portfolios.contrib.signals.regression_utils: regression helpers used by contributed replicator-style signals.

Key Contracts

Portfolio updaters use canonical time-indexed frames. Portfolio identity should be deterministic: configuration hashes and signal/rebalance UIDs must be stable for equivalent configuration payloads.

Canonical markets updaters derive their published identifiers from the same rule as MetaTables: the default markets namespace keeps bare logical identifiers, while a non-default MSM_AUTO_REGISTER_NAMESPACE prefixes them. That namespace also becomes the default TimeIndexTableUpdater hash_namespace. Pass an explicit namespace only for isolated tests or experiments.

Signal observation, rebalance decision, execution, valuation, and analytical time are separate concepts. Every canonical timestamp must come from a signal, persisted calendar event, execution bar, or valuation observation. Job time and generic frequency strings are not economic clocks.

Use the typed row API for registry records:

import msm

from msm.api.calendars import Calendar
from msm.api.portfolios import Portfolio

msm.start_engine(models=["Calendar", "CalendarDate", "CalendarSession", "Portfolio"])

calendar = Calendar.create_from_pandas_calendar(
    source_identifier="24/7",
    unique_identifier="CRYPTO_24_7",
    display_name="Crypto 24/7",
    valid_from="2026-05-25",
    valid_to="2026-05-25",
    timezone="UTC",
)

portfolio = Portfolio.upsert(
    unique_identifier="btc-eth-target",
    calendar_uid=calendar.uid,
)

PortfolioCalendarEvents publishes persisted CalendarSession opens or closes at their actual UTC timestamps, including early closes and DST shifts. CalendarEventSignal declares that published table as an observed dependency. ImmediateSignal instead selects original signal observations and declares no additional event source. Event selection and state transitions occur in the strategy below PortfolioRebalance; neither PortfolioWeights nor PortfoliosUpdater builds a rebalance index.

calendar_identifier and calendar_events_instance are required for CalendarEventSignal. The event producer requires the canonical persisted Calendar.unique_identifier; a source_identifier alias is rejected because published rows carry a foreign key to that canonical value. Missing and failed calendar lookups raise an error; neither the producer nor the strategy substitutes a local pandas or synthetic calendar. Materialize the calendar horizon before running the portfolio graph.

Portfolio.upsert(...) writes only the portfolio identity row. Portfolio constituents, weights, values, and optional index publication are separate portfolio workflows.

General Rebalance Architecture

ADR 0040 records the implemented general strategy boundary. ImmediateSignal, CalendarEventSignal, TimeWeighted, VolumeParticipation, and LiquidityConstrained are concrete strategies below that boundary; none of them defines the architecture itself.

SignalWeights target intent --------------------------+
                                                       |
strategy-declared observed inputs ---------------------+--> PortfolioRebalance
  calendar events, bars, volume, quotes, book depth,          generic stateful updater
  available liquidity, schedules, or fills                         |
                                                                   v
                                                     PortfolioRebalanceStateStorage
                                                     active intent, partial progress,
                                                     remaining target, provenance
                                                                   |
                                                                   v
                                                        PortfolioWeights
                                                        executed-weight projection
                                                                   |
                                                                   v
                                                        PortfolioWeightsStorage
                                                                   |
valuation observations --------------------------------------------+--> PortfoliosUpdater
                                                                        valuation only
                                                                            |
                                                                            v
                                                                     PortfoliosStorage
                                                                            |
                                                                            v
                                                                     PortfolioAnalytics

The portfolio composes the rebalance strategy and includes its serialized policy and declared dependency identities in portfolio/update hashing. It does not subclass the strategy and does not copy or generate strategy dates.

The strategy contract owns dependency declaration, input validation, target cutoff and succession, source-event selection, partial state transitions, and completion. The generic rebalance updater contains no branch for immediate, calendar, time, volume, or liquidity strategies. A missing future bar, quote, liquidity observation, or fill leaves the target unfinished; it never forces a completion timestamp.

PortfolioTable.calendar_uid remains required portfolio reference metadata; it does not implicitly schedule rebalances. Calendar-aware strategies declare their published calendar-event input explicitly. Strategies driven by bars, volume, quotes, order-book depth, available liquidity, schedules, or fills do not gain calendar events merely because the portfolio has a reference calendar.

PortfolioRebalanceStateStorage is a portfolio-level state ledger, not a broker order/fill ledger. PortfolioWeights projects only state transitions that actually changed executed weights, and PortfoliosUpdater continues to value those weights at independent valuation-source observations. Missing future bars or liquidity leave a target pending or partial; they do not create a completion timestamp.

Trailing daily-liquidity participation

TrailingAverageDailyVolumeParticipation separates capacity estimation from execution pricing. It declares two independent asset-indexed sources:

completed daily liquidity bars (daily VWAP, daily volume)
  -> trailing average daily notional
  -> per-asset session cap

observable intraday execution bars (execution price, bar volume)
  -> per-bar cap
  -> actual execution price and quantity

For each asset, completed historical daily notional is daily_vwap * daily_volume. The strategy averages the most recent configured number of completed observations that were available strictly before the session execution window began. A current-session or future daily VWAP is never eligible because VWAP is an ex-post statistic. Daily input timestamps must therefore represent the right edge/availability time of completed bars.

At each intraday bar, executable notional is the minimum of the remaining target notional, remaining per-session daily cap, and the configured fraction of the current observed bar's notional. The stored execution_price is always the configured field from the intraday execution source, such as close, mid_price, or an arrival-price observation. It is never the historical daily VWAP. Daily capacity consumed is persisted per asset in strategy_state, so a restart or a same-session target supersession cannot reset the participation limit.

The history read is bounded by history_lookback_days. Rolling daily capacity, signal targets, and timestamp-to-observation groups are prepared once per run; the transition loop does not rescan the full daily history for each intraday bar. Apart from sorting unsorted source frames, preparation is linear in the daily and intraday input rows, followed by the unavoidable emitted state transitions.

Use examples/msm_portfolios/portfolio_trailing_adv_participation_preview.py to preview this state machine against registered daily-liquidity and intraday-bar tables. The example filters both source reads to the requested assets, applies the strategy-owned bounded history window, and prints quantities and cap consumption without writing PortfolioRebalanceStateStorage. Production portfolio workflows inject the same strategy into BacktestingWeightsConfig.rebalance_strategy_instance and let PortfolioRebalance persist the resulting transitions.

Portfolio Read Services

Reusable portfolio output reads live under src/msm_portfolios/services/portfolio_reads.py and are exported from msm_portfolios.services:

from msm_portfolios.services import (
    latest_portfolio_weights,
    portfolio_values,
    signal_weights,
)

weights = latest_portfolio_weights(
    ["btc-eth-target"],
    weights_date="2026-05-25T00:00:00Z",
    as_of=True,
    repository_context=runtime.context,
)
values = portfolio_values(
    ["equity-tracker", "income-fund", "btc-eth-target"],
    start="2026-05-01T00:00:00Z",
    end="2026-05-31T00:00:00Z",
    repository_context=runtime.context,
)
signal = signal_weights(
    "signal-uid",
    start="2026-05-01T00:00:00Z",
    repository_context=runtime.context,
)

Every read first resolves, in one statement, which table holds each portfolio's rows: PortfolioTable left-joined to its PortfolioDatasetAvailabilityTable row and to SignalMetadataTable (resolve_portfolio_dataset_tables(...)). The requested portfolios are then grouped by table uid:

  • a group with a catalogued table uid is read once by uid through metatables.TimeIndexTableRef, filtered by portfolio_identifier (or signal_uid), on the contract columns only; extra provider columns are ignored;
  • the group with no catalog row reads the Markets-owned core table (PortfoliosStorage, PortfolioWeightsStorage, SignalWeightsStorage) through compiled SQL, as before.

The result is one ordered list of row dictionaries with portfolio_uid attached, however many instances served it. The fan-out is bounded by the number of distinct tables, not of portfolios.

latest_portfolio_weights(...) reads the PortfolioWeightsTS instance by PortfolioTable.unique_identifier, which is the storage-facing portfolio_identifier. Pass as_of=True for latest-at-or-before snapshot selection, as_of=False for exact timestamp matching, and order="asc" for the earliest snapshot when no weights_date is given. On a catalogued table the latest snapshot time comes from get_last_observation, and the snapshot rows from one bounded get_df_between_dates call.

portfolio_values(...) reads value rows for one or more portfolios, with optional start, end, latest_only and limit. signal_weights(...) reads one signal's rows; the instance is the caller's meta_table_uid (from the portfolio's catalog row), else SignalMetadataTable.meta_table_uid, else the core table. These helpers return market-domain row dictionaries. Command Center tabular frames and dashboard formatting are consumer concerns.

Use an explicit repository_context for live platform reads. Tests and downstream adapters may pass an executor callable for the SQL statements and a dataset_reader callable (uid to TimeIndexTableRef-like reader) for the catalogued tables, as tests/msm_portfolios/services/test_portfolio_reads.py does. See examples/msm_portfolios/portfolio_read_services.py for an offline example with injected executors.

Contract Instances And The Dataset Catalog

ADR 0047 makes PortfoliosTS, PortfolioWeightsTS and SignalWeightsTS storage contracts with any number of instances. ms-markets owns the contract: columns, dtypes, nullability, time index and index names. The core tables are the Markets-owned instance for portfolios Markets computes itself. A provider owns its instances, in its own MetaTables namespace, migrated by its own Alembic provider and written by its own nodes with output_table=.

from sqlalchemy import MetaData
from sqlalchemy.orm import DeclarativeBase

from metatables import sqlalchemy_naming_convention
from msm_portfolios.time_index import PortfoliosUpdater
from msm_portfolios.time_index.contract_instances import (
    configured_portfolio_rebalance_state_storage,
    configured_portfolio_weights_storage,
    configured_portfolios_storage,
    configured_signal_weights_storage,
)


class ProviderBase(DeclarativeBase):
    metadata = MetaData(naming_convention=sqlalchemy_naming_convention())


ProviderPortfoliosStorage = configured_portfolios_storage(
    instance_key="main",
    namespace="my-provider",
    storage_app="my_provider",
    base=ProviderBase,
)
ProviderPortfolioWeightsStorage = configured_portfolio_weights_storage(
    instance_key="main", namespace="my-provider",
    storage_app="my_provider", base=ProviderBase,
)
ProviderSignalWeightsStorage = configured_signal_weights_storage(
    instance_key="main", namespace="my-provider",
    storage_app="my_provider", base=ProviderBase,
)

Each factory returns a PlatformTimeIndexMetaTable class whose table is <storage_app>__<contract>__<instance_key> (here my_provider__portfoliosts__main), whose MetaTable identifier is <Contract>.<instance_key> prefixed with the namespace when it is not the default Markets one (my-provider.PortfoliosTS.main), and whose columns, dtypes, foreign keys to the core registries (Portfolio, Asset, SignalMetadata) and unique grain index equal the contract's. With base=, the referenced core tables are copied into the provider's metadata so the foreign keys resolve without importing the core classes into the provider's migration scope. Without base= the instance is registered in the Markets registry; the Markets migration provider still lists only the core tables. The provider may add columns beyond the contract; Markets reads the contract columns and ignores the rest. PortfolioCanonicalUpdater subclasses accept the instance through output_table=.

The engine outputs are contracts too: configured_portfolio_rebalance_state_storage, configured_portfolio_calendar_events_storage and configured_portfolio_event_ledger_storage build PortfolioRebalanceStateTS, PortfolioCalendarEventsTS and PortfolioEventLedgerTS instances with the same arguments. PortfoliosUpdater builds its own PortfolioWeights and PortfolioRebalance updaters, so it takes their instances as well:

ProviderRebalanceStateStorage = configured_portfolio_rebalance_state_storage(
    instance_key="main", namespace="my-provider",
    storage_app="my_provider", base=ProviderBase,
)

updater = PortfoliosUpdater(
    portfolio_configuration=portfolio_configuration,
    output_table=ProviderPortfoliosStorage,
    weights_output_table=ProviderPortfolioWeightsStorage,
    rebalance_output_table=ProviderRebalanceStateStorage,
)

Do not pass namespace= to the updater: on an updater it is the MetaTables hash namespace, which changes update_hash. The instance's namespace comes from its factory.

PortfolioWeights alone takes output_table= and rebalance_output_table=. InterpolatedPrices takes storage_namespace=, storage_app= and storage_base= to keep its derived table in the provider's namespace; the table keeps its storage identity and identifier suffix, and none of the three enters update_hash.

An application on the remote row backend registers and reconciles the catalog with msm_portfolios.api.datasets.PortfolioDatasets.register(...), .reconcile(...) and .get(...), which run the services below on the Markets API. register binds the portfolio's current signal; PortfolioDatasets.register_signal({"signal_uid": ..., "signal_meta_table_uid": ...}) binds any other signal, such as the earlier signals of a producer whose signal uid changes each run, after verifying the table against the signal weights contract. PortfolioDatasets.unregister(portfolio_uid=...) removes the catalog row (and the signal binding to the catalogued signal instance) to roll a move back; reads return to the core tables. PortfolioDatasets.unregister_signal(signal_uid=...) clears any other signal's binding, undoing register_signal. Instance tables change where rows land, not identities: none of them enters update_hash, and MetaTables keys each update record by update hash and output table.

Markets keeps the catalog in PortfolioDatasetAvailabilityTable (msm.models.portfolio_dataset_availability, identifier PortfolioDatasetAvailability, mirroring IndexDatasetAvailabilityTable). One row per portfolio records the values, weights and signal instance uids and the reconciled population state of the values instance: populated, compatible_empty or unavailable, with row count, earliest and latest time index, reconciled time and error fields. SignalMetadataTable.meta_table_uid records the signal's instance so a signal can be read without its portfolio.

from msm_portfolios.services import (
    list_portfolio_dataset_states,
    reconcile_portfolio_dataset_availability,
    register_portfolio_datasets,
)

state = register_portfolio_datasets(
    runtime.context,
    {
        "portfolio_uid": portfolio.uid,
        "values_meta_table_uid": ProviderPortfoliosStorage.get_meta_table_uid(),
        "weights_meta_table_uid": ProviderPortfolioWeightsStorage.get_meta_table_uid(),
        "signal_meta_table_uid": ProviderSignalWeightsStorage.get_meta_table_uid(),
    },
)
result = reconcile_portfolio_dataset_availability(
    runtime.context, portfolio_uids=[portfolio.uid]
)

register_portfolio_datasets checks that each declared table is a visible TimeIndexMetaTable that exposes the contract columns and index names, upserts the catalog row, binds SignalMetadata.meta_table_uid when the portfolio has a signal, and reconciles. reconcile_portfolio_dataset_availability groups the requested portfolios by values table uid and aggregates each instance once with one count, min and max statement grouped by portfolio_identifier; an instance the Markets workload cannot read yields unavailable rows carrying the error. Portfolios without a catalog row are reported in skipped_portfolio_uids and keep reading the core tables. The provider calls these through the Markets API after creating the portfolio and after each run.

Who may read is a separate, narrow grant: the Markets workload needs data_viewer on each provider's time-series tables, or the call is requester-bound and the person's own grants apply.

Moving Existing History

A portfolio that was already built into the core tables keeps its history there until it is copied: portfolio and signal history is not recomputed. msm_portfolios.services.series_moves moves it and rolls it back the same way for every provider:

from msm_portfolios.services.series_moves import (
    PortfolioSeriesMove,
    PortfolioSeriesTables,
    SignalSeriesMove,
)
from msm.storage_copies import run_series_moves

instances = PortfolioSeriesTables(
    values=ProviderPortfoliosStorage,
    weights=ProviderPortfolioWeightsStorage,
    rebalance_state=ProviderPortfolioRebalanceStateStorage,
    signal_weights=ProviderSignalWeightsStorage,
)
moves = [
    PortfolioSeriesMove(
        portfolio.unique_identifier,
        instances,
        portfolio_uid=portfolio.uid,
        signal_uids=earlier_signal_uids,
        with_current_signal=True,
        on_copied=lambda: use_provider_tables(portfolio),
        on_rolled_back=lambda: use_core_tables(portfolio),
    ),
]
report = run_series_moves(moves, dry_run=True)  # counts only
report = run_series_moves(moves, dry_run=False)
report = run_series_moves(moves, rollback=True, dry_run=False)

A move copies, per key, the signal weights of the signals that move with the portfolio, then its rebalance state, weights, values and event ledger, writing only the rows the instance lacks (msm.storage_copies.copy_time_index_rows). When every key is complete it calls on_copied, where the provider points its updaters at the instances, binds the moved signals that are not the portfolio's current one with register_signal, and writes the catalog row last with register. If a key is still missing rows the move stops as incomplete and nothing in Markets changes; running it again copies only the gaps. Before on_copied it checks that every signal it will bind has a SignalMetadata row, so a signal that never ran fails the move before the provider switches.

A rollback copies the rows the instances gained since the move back into the core tables, removes the catalog row (unregister(..., unbind_signal=False)), clears the bindings of the signals that moved with the portfolio (unregister_signal) and calls on_rolled_back; a signal moved on its own keeps its binding until its own SignalSeriesMove rolls back. The instances keep their rows. Every move takes keys_per_read and deadline_seconds for its reads. SignalSeriesMove does the same for signals moved on their own; run_series_moves runs signals before portfolios on a move and portfolios first on a rollback. Calendar events are not copied: the engine rebuilds them.

msm series move module:callable [--execute] and msm series rollback ... run the moves a provider callable returns (it receives rollback= when it accepts that argument); without --execute they only count. The operator needs data_viewer on the tables read, write access to the tables written (a rollback writes the core tables); pause the provider's runs while a move or rollback runs.

Repair legacy midnight-indexed portfolio values

Portfolio values written by the former daily resampling path may carry a UTC midnight time_index while close_time records the real exchange close. Do not rewrite those indexed coordinates directly. Plan a portfolio-scoped tail rollback against the persisted calendar first:

PYTHONPATH=src:. python \
  examples/msm_portfolios/portfolio_midnight_timestamp_repair.py \
  --portfolio-identifier <portfolio-identifier> \
  --start 2026-01-01T00:00:00Z \
  --end <latest-portfolio-value-timestamp>

The default is a read-only dry run. It checks Portfolio.calendar_uid, persisted CalendarSession.closes_at, the historical close_time, destination conflicts, and whether --end reaches the latest stored value. Any uncertainty is a blocking issue. Review the JSON plan, then apply exactly one portfolio:

PYTHONPATH=src:. python \
  examples/msm_portfolios/portfolio_midnight_timestamp_repair.py \
  --portfolio-identifier <portfolio-identifier> \
  --start 2026-01-01T00:00:00Z \
  --end <latest-portfolio-value-timestamp> \
  --apply

Apply performs an inclusive PortfoliosStorage tail delete scoped by portfolio_identifier; it does not finish the repair by itself. Pause any scheduled writer before apply, and keep it paused while you immediately rerun the portfolio workflow with the migrated configuration to replay that tail from canonical valuation observations. The result compares the deleted count with the dry-run count and exits nonzero if they differ. Run the dry plan again after replay: it should contain no rollback. Process another portfolio only after the previous portfolio has been replayed and verified.

Portfolio Registry Tables

Portfolio registry tables are regular platform-managed MetaTables. They describe portfolio identity and relationships; they do not store historical portfolio values. Historical values, weights, and signal outputs live in time-index-table output tables.

Portfolio identity is core reference data and lives under:

src/msm/models/portfolios/
├── __init__.py
├── core.py       PortfolioTable
├── groups.py     PortfolioGroupTable, PortfolioGroupMembershipTable
└── signals.py    SignalMetadataTable
src/msm/models/portfolio_dataset_availability.py
                  PortfolioDatasetAvailabilityTable

PortfolioTable is the canonical portfolio identity row. It is keyed by unique_identifier and stores optional published_index_uid linkage to IndexTable, an optional signal_uid linkage to SignalMetadataTable, plus TimeIndexTableUpdater UIDs for canonical portfolio outputs. A portfolio is not an asset. The optional published index link is metadata for workflows that want to expose the portfolio as an index-like observable; core portfolio weights, values, account expansion, and virtual-fund allocation use PortfolioTable.uid / PortfolioTable.unique_identifier.

Portfolio groups are core reference data too. PortfolioGroupTable stores group identity and PortfolioGroupMembershipTable stores the many-to-many relationship between groups and portfolios. There is no portfolio_group_uid column on PortfolioTable; a portfolio can belong to several groups without duplicating or changing the portfolio identity row.

Portfolio descriptive metadata remains in msm_portfolios:

src/msm_portfolios/models/portfolios/
├── __init__.py
└── metadata.py   PortfolioMetadataTable

PortfolioMetadataTable is descriptive metadata keyed by portfolio unique_identifier. It is intentionally not a foreign-key extension of PortfolioTable; it is human-facing metadata that can be managed without changing the portfolio identity row.

Rebalance strategy calendar keys resolve persisted core CalendarTable rows first. Durable portfolio records must use PortfolioTable.calendar_uid; this field is required and cannot be null. Legacy pandas-market calendar keys remain a fallback path only for runtime calendar resolution, not for persisted portfolio identity.

Table Relationships

Portfolio identity, required calendar linkage, and optional published-index linkage:

+-----------------------------+        optional published index  +-----------------------------+
| PortfolioTable              |--------------------------------->| IndexTable                  |
|-----------------------------| published_index_uid             |-----------------------------|
| uid PK                      |                                  | uid PK                      |
| unique_identifier unique    |                                  | unique_identifier unique    |
| calendar_uid FK NOT NULL ---+----+                             | index_type                  |
| portfolio_weights_table_update_uid |--> PortfolioWeights       +-----------------------------+
| signal_weights_table_update_uid    |--> SignalWeights
| signal_uid FK nullable ---------+----------------------------+
| portfolio_table_update_uid         |--> PortfoliosUpdater
| backtest_table_price_column_name|
+-----------------------------+
                                  |
                                  | required durable calendar relationship
                                  v
                         +-----------------------------+
                         | CalendarTable               |
                         |-----------------------------|
                         | uid PK                      |
                         | unique_identifier unique    |
                         | valid_from / valid_to       |
                         +-----------------------------+

                                  optional signal pointer
                                  v
                         +-----------------------------+
                         | SignalMetadataTable         |
                         |-----------------------------|
                         | uid PK                      |
                         | signal_uid unique           |
                         | signal_description          |
                         +-----------------------------+

Portfolio metadata is a separate descriptive table:

+-----------------------------+              same logical key              +-----------------------------+
| PortfolioTable              |------------------------------------------->| PortfolioMetadataTable      |
|-----------------------------| unique_identifier by convention           |-----------------------------|
| uid PK                      |              no database FK                | uid PK                      |
| unique_identifier unique    |                                           | unique_identifier unique    |
| registry/config fields      |                                           | description                 |
+-----------------------------+                                           +-----------------------------+

Portfolio groups are many-to-many classification metadata:

+-----------------------------+        1..*        +-----------------------------------+
| PortfolioGroupTable         |------------------->| PortfolioGroupMembershipTable     |
|-----------------------------|                    |-----------------------------------|
| uid PK                      |                    | uid PK                            |
| unique_identifier unique    |                    | portfolio_group_uid FK cascade    |
| display_name                |                    | portfolio_uid FK cascade          |
| description                 |                    | unique(group, portfolio)          |
+-----------------------------+                    +------------------+----------------+
                                                                   |
                                                                   | *..1
                                                                   v
                                                        +-----------------------------+
                                                        | PortfolioTable              |
                                                        |-----------------------------|
                                                        | uid PK                      |
                                                        | unique_identifier unique    |
                                                        | calendar_uid FK NOT NULL    |
                                                        +-----------------------------+

Deleting a portfolio group removes only membership rows through cascade. Deleting a portfolio removes only its membership rows through cascade. Neither operation deletes the other side of the relationship.

Portfolio time-index-table output is separate from registry MetaTables. These storage classes are registered through the same catalog bootstrap, after their FK target MetaTables:

+-----------------------------+             writes             +--------------------------------------+
| PortfolioCalendarEvents     |------------------------------->| PortfolioCalendarEventsStorage       |
| PortfolioRebalance          |------------------------------->| PortfolioRebalanceStateStorage       |
| PortfolioWeights            |------------------------------->| PortfolioWeightsStorage              |
| SignalWeights               |------------------------------->| SignalWeightsStorage                 |
| PortfoliosUpdater           |------------------------------->| PortfoliosStorage                    |
| PortfolioAnalytics         |------------------------------->| PortfolioAnalyticsStorage           |
| External price updaters     |------------------------------->| ExternalPricesStorage                |
| InterpolatedPrices          |------------------------------->| configured InterpolatedPricesStorage |
+-----------------------------+                                +--------------------------------------+
          TimeIndexTableUpdater update logic                                  PlatformTimeIndexMetaTable

+-----------------------------+        required parent          +--------------------------------------+
| SignalMetadataTable         |<-------------------------------| SignalWeightsStorage                 |
|-----------------------------| signal_uid FK                  |--------------------------------------|
| uid PK                      |                                | time_index                           |
| signal_uid unique           |                                | signal_uid                           |
| signal_description          |                                | asset_identifier                     |
+-----------------------------+                                +--------------------------------------+

SignalMetadataTable.signal_description is descriptive text for humans. Store plain text or Markdown, not HTML tags; rendering belongs to the consuming UI. PortfolioTable.signal_uid is nullable because portfolio rows can be registered before a signal workflow runs. Once PortfoliosUpdater.run(..., update_pointers=True) completes, the portfolio row should store the same signal_uid that identifies the signal metadata and signal-weight rows. API reads for portfolio signal weights must use this pointer; they must not infer a signal by scanning the shared SignalWeightsStorage table.

PortfoliosStorage.portfolio_identifier, PortfolioRebalanceStateStorage.portfolio_identifier, and PortfolioWeightsStorage.portfolio_identifier reference PortfolioTable.unique_identifier; portfolio value and weight rows must be written for a real portfolio identity. Rebalance state and portfolio weights also reference AssetTable.unique_identifier, so neither can point to unknown assets. PortfolioCalendarEventsStorage.calendar_identifier references CalendarTable.unique_identifier. PortfoliosUpdater resolves the portfolio identifier from the attached PortfolioTable row or from the explicit runtime identifier before normalizing rows.

PortfolioRebalanceStateStorage.time_index is the strategy-selected observed event timestamp, including partial or pending events. PortfolioWeightsStorage.time_index is an execution timestamp copied from a state transition that changed executed weight. PortfoliosStorage.time_index is the valuation observation timestamp. They do not need to be equal: weekly execution can feed daily valuation. Optional PortfolioAnalyticsStorage.time_index is the actual source observation chosen for an analytical period; period_start and period_end carry bucket boundaries without relabelling the source event.

Portfolio construction depends on a real valuation source, but portfolio logic does not own valuation ingestion. Example workflows publish normalized OHLCV bars to ExternalPricesStorage only so the example is self-contained. Production users can point portfolio configurations at any registered compatible valuation storage table, including one produced by another library, vendor connector, model valuation process, or project TimeIndexTableUpdater.

Valuation Source Resolution

Portfolio valuation inputs are not stored on PortfolioTable. They are provided by the portfolio build configuration and consumed through TimeIndexTableUpdater dependencies.

The current portfolio path keeps execution and valuation dependencies explicit:

SignalWeights --------------------------+
strategy-declared observations ---------+--> PortfolioRebalance
  bars, volume, liquidity, fills, events       | writes/restarts from
                                              v
                                  PortfolioRebalanceStateStorage
                                              |
                                              v
                                      PortfolioWeights
                                              |
                                              v
ValuationSource --------------------> PortfoliosUpdater
  bars, fair value, NAV                    |
                                          v
                                  canonical portfolio values

PortfolioBuildConfiguration.valuation_source_instance receives the valuation source that portfolio construction consumes. The valuation source may be an InterpolatedPrices instance, another TimeIndexTableUpdater, or an TimeIndexTableRef pointing at compatible registered storage. The valuation source must expose rows keyed by (time_index, asset_identifier) and include the configured numeric valuation_column, for example close, fair_value, nav, or mark_price. Execution observations are strategy inputs, not portfolio valuation inputs. ImmediateSignal requires no execution market-data dependency. TimeWeighted declares price bars, VolumeParticipation declares price and volume bars, TrailingAverageDailyVolumeParticipation declares completed daily liquidity plus observable intraday execution bars, and LiquidityConstrained declares price plus available liquidity. Each uses actual observations from its declared source; historical daily VWAP is capacity input only.

This producer boundary is intentional. Price collection, valuation modeling, normalization, vendor mapping, and connector-specific scheduling are separate concerns from portfolio construction. Portfolio extensions should focus on universe selection, signals, rebalancing, execution assumptions, and portfolio output storage. They should consume a registered valuation storage contract instead of importing or constructing the producer that wrote it.

If persistent interpolation is needed, InterpolatedPrices is built before the portfolio and then passed into PortfolioBuildConfiguration like any other dependency. InterpolatedPricesConfig accepts either source_price_instance or source_time_index_meta_table_uid. Use source_price_instance when the source price TimeIndexTableUpdater or TimeIndexTableRef is already part of the graph. Use source_time_index_meta_table_uid when attaching an already registered compatible source table through TimeIndexTableRef.from_uid(...). InterpolatedPrices validates the registered source cadence, exposes the resolved source from dependencies(), and writes the configured interpolation output.

Timestamp-valued bar payloads such as open_time, first_trade_time, and last_trade_time remain timezone-aware datetimes throughout interpolation and are normalized to datetime64[ns, UTC] before publication. They are never round-tripped through unit-ambiguous integers. This matters with pandas 3, where parsed datetimes commonly retain microsecond resolution and an integer cast therefore represents microseconds rather than nanoseconds.

Upgrading the library does not rewrite previously published interpolated rows. If inspection finds corrupt timestamp payloads, install the corrected package first, determine the earliest affected observation per asset, apply an inclusive asset_identifier-scoped tail delete through TimeIndexMetaTable.delete_after_date(...), and immediately replay the interpolation updater and affected downstream graph. Do not update persisted time coordinates in place.

The interpolation policy is storage identity, not row metadata. InterpolatedPrices builds a configured storage class whose __metatable_extra_hash_components__ include the source TimeIndexMetaTable UID, the source table cadence, upsample_frequency_id, and intraday_bar_interpolation_rule; those components determine the configured physical table identity. The rows keep the normal price-bar grain (time_index, asset_identifier). The policy values are not repeated on every price row.

Configured interpolation storage is a dynamic schema artifact. It is derived from a real registered source price storage table and a concrete interpolation policy, so it is not part of the package-wide static start_engine(...) model list. Prepare it before the normal portfolio run:

python examples/msm_portfolios/portfolio_equal_weights_prepare_schema.py
python examples/msm_portfolios/portfolio_equal_weights_run.py

The preparation step attaches the static schema, reads the registered ExternalPricesStorage UID and cadence metadata, builds the configured InterpolatedPricesStorage class, and uses the active migration namespace from the MetaTables migration provider to find or generate the real dynamic Alembic revision. It then runs the dynamic provider upgrade before any portfolio TimeIndexTableUpdater writes, even when a stale TimeIndexMetaTable metadata row already exists. Metadata alone is not considered schema preparation; the physical table must be created by the migration flow. If an older registered ExternalPricesStorage row is missing cadence metadata, the preparation script patches that metadata to the model-declared cadence before deriving the dynamic table. Runtime portfolio code then uses the registered table; it does not create or migrate dynamic storage.

PortfolioBuildConfiguration.valuation_column chooses which numeric column from the explicit valuation source drives portfolio returns. It is a string, not an OHLC enum. Bar-based workflows can use valuation_column="close", while model or vendor workflows can use fields such as fair_value, nav, or settlement_price. Portfolio valuation performs bounded per-asset as-of selection under ValuationAlignmentPolicy; alignment targets only explicit valuation observations and never creates a timestamp. Execution-side event selection and capacity are owned by the configured rebalance strategy.

For a custom valuation source, register or migrate the source storage outside portfolio core, then pass the registered TimeIndexMetaTable UID through TimeIndexTableRef.from_uid(...). The source table must keep its real value column name; it should not be reshaped to close just to satisfy portfolio core. See examples/msm_portfolios/portfolio_custom_valuation_column_example.py for the fair_value configuration path. The portfolio update start is always the latest PortfoliosStorage timestamp for the resolved PortfolioTable.unique_identifier, stored as PortfoliosStorage.portfolio_identifier. It is not derived from signal progress, valuation-source progress, or the table-wide maximum of the shared PortfoliosStorage table.

After resolving that portfolio-scoped start, PortfoliosUpdater reads canonical executed weights and actual valuation observations. It selects the latest weights as-of each eligible observation, validates per-asset staleness, and writes one canonical value row per eligible source timestamp. Before the window, it retrieves all required per-asset seed observations in one set-based request. Shared multidimensional sources use complete coordinates; for signal weights that means both signal_uid and asset_identifier. A rerun before new source data arrives is an empty update.

Executed-weight seed rows are normalized and retained only when their time_index is strictly earlier than the calculation-window start, even if an upstream latest-observation response violates the requested < start bound. The current window, retained seed, and combined frame must each be unique on (time_index, asset_identifier); a genuine duplicate fails with an explicit invariant error before weights are pivoted.

Strict valuation coverage is exposure-aware at each asset and timestamp. A valuation is required when either the current or immediately preceding executed weight is nonzero: the current weight protects entries, and the preceding weight protects exits. A zero-to-zero asset is optional, and after a completed exit it becomes optional on the following valuation timestamp. The complete signal and executed-weight snapshots are not filtered, so explicit zero rows remain available for state reconstruction and auditing.

In code, the important wiring is:

source_bars_node = ExampleDailyBars(asset_identifiers=["asset-btc", "asset-eth"])
source_bars_node.run(update_tree=False)

price_source = InterpolatedPrices(
    interpolation_config=InterpolatedPricesConfig(
        asset_list=["asset-btc", "asset-eth"],
        intraday_bar_interpolation_rule="ffill",
        source_price_instance=source_bars_node,
        upsample_frequency_id="1d",
    )
)

signal_weights = FixedWeights.from_signal_configuration(...)
calendar_events = PortfolioCalendarEvents(
    config=PortfolioCalendarEventsConfiguration(
        calendar_identifier="CRYPTO_24_7",
        event_types=("market_close",),
    )
)

portfolio_configuration = PortfolioConfiguration(
    portfolio_build_configuration=PortfolioBuildConfiguration(
        valuation_source_instance=price_source,
        valuation_column="close",
        execution_configuration=PortfolioExecutionConfiguration(...),
        backtesting_weights_configuration=BacktestingWeightsConfig(
            signal_weights_instance=signal_weights,
            rebalance_strategy_instance=CalendarEventSignal(
                calendar_events_instance=calendar_events,
                calendar_identifier="CRYPTO_24_7",
                rebalance_event="market_close",
            ),
        ),
    ),
    portfolio_markets_configuration=PortfolioMarketsConfig(...),
)

For a user-owned fair-value source, the same portfolio configuration uses the source UID directly and keeps the valuation column explicit:

valuation_source = TimeIndexTableRef.from_uid(fair_value_table_uid)

portfolio_configuration = PortfolioConfiguration(
    portfolio_build_configuration=PortfolioBuildConfiguration(
        valuation_source_instance=valuation_source,
        valuation_column="fair_value",
        execution_configuration=PortfolioExecutionConfiguration(...),
        backtesting_weights_configuration=BacktestingWeightsConfig(...),
    ),
    portfolio_markets_configuration=PortfolioMarketsConfig(...),
)

PortfolioRebalance.dependencies() merges signal weights with arbitrary typed sources declared by the strategy, validates those observations against the strategy's required grain and fields, and persists the resulting state. PortfolioWeights.dependencies() exposes only that rebalance-state updater. PortfoliosUpdater.dependencies() continues to expose canonical executed weights and its independent valuation source. Valuation sources may contain extra assets; required holdings are aligned independently using the latest observation at or before the target within maximum_staleness.

portfolio_prices_frequency and PriceAlignmentPolicy were removed. Use ValuationAlignmentPolicy for source freshness and PortfolioAnalytics for daily, weekly, monthly, or other reporting frequency conversion.

Existing portfolio output progress is scoped by portfolio_identifier because PortfoliosStorage is keyed by (time_index, portfolio_identifier). A later row for another portfolio in the shared storage table must not move this portfolio's start date; if this portfolio has no progress entry, the workflow treats it as a fresh portfolio rather than using the table-wide maximum.

Signal output progress is scoped by signal_uid because SignalWeightsStorage is keyed by (time_index, signal_uid, asset_identifier). signal_uid is a required reference to SignalMetadataTable.signal_uid, and asset_identifier is a required reference to AssetTable.unique_identifier, so signal metadata and assets must exist before signal-weight rows are published. Contributed signal nodes must read progress under their own signal_uid; a later row from another signal in the shared table must not shorten this signal's source-data window.

Account Target-Position Exposure To Portfolios

Account allocation registry rows remain core msm account concepts: AccountAllocationModelTable, AccountTargetAllocationTable, and PositionSetTable. The timestamped target exposure rows that can reference a constructed portfolio are also core account allocation storage. They live in msm.time_index.accounts.storage.TargetPositionsStorage; portfolio workflows may read or expand them, but they do not own the table.

+-----------------------------+       position_set_uid       +-----------------------------+
| PositionSetTable            |<-----------------------------| TargetPositionsStorage      |
| owner: msm                  |                              | owner: msm                  |
|-----------------------------|                              |-----------------------------|
| uid PK                      |                              | time_index                  |
| account_target_allocation_uid|                              | target_type                 |
| position_set_time UTC       |                              | target_uid                  |
+-----------------------------+                              | asset_uid nullable FK       |
                                                               | portfolio_uid nullable FK   |
                                                               | exposure columns            |
                                                               +-------------+---------------+
                                                                             |
                                                                             | portfolio_uid
                                                                             v
                                                               +-----------------------------+
                                                               | PortfolioTable              |
                                                               | owner: msm                  |
                                                               | uid PK                      |
                                                               | unique_identifier unique    |
                                                               +-----------------------------+

A target row has exactly one target:

target_type = asset
  target_uid = asset_uid
  asset_uid -> AssetTable.uid

target_type = portfolio
  target_uid = portfolio_uid
  portfolio_uid -> PortfolioTable.uid

Portfolio target rows are mandate exposure, not custody holdings and not portfolio indices. They are expanded into asset-level exposure only when a downstream workflow explicitly calls the portfolio expansion service and provides a resolver for current portfolio weights.

Portfolio Construction And Account Virtual-Fund Allocation Boundary

Portfolio construction produces portfolio artifacts. It does not own virtual-fund identity, and it does not write virtual-fund allocation rows. Virtual funds are account-owned allocation views that target a portfolio after that portfolio exists. Their canonical docs live in core msm account virtual funds.

Portfolio construction produces portfolio artifacts.
It does not own virtual-fund identity and it does not connect directly to
virtual-fund allocation rows.

+---------------------+      +-----------------------------+
| SignalWeights       |----->| PortfolioRebalance          |<----- strategy-declared
+---------------------+      | generic strategy runner     |      observed sources
                             +--------------+--------------+
                                            |
                                            v
                             +-----------------------------+
                             | RebalanceStateStorage       |
                             +--------------+--------------+
                                            |
                                            v
                             +-----------------------------+
                             | PortfolioWeights projection |
                             +--------------+--------------+
                                            |
                                            v
+---------------------+      +-----------------------------+
| ValuationSource     |----->| PortfoliosUpdater           |
+---------------------+      | valuation only              |
                             +--------------+--------------+
                                            |
                                            v
        +---------------------------------------------------------+
        | PortfolioTable                                          |
        | - portfolio identity                                    |
        | - signal_weights_table_update_uid                       |
        | - portfolio_weights_table_update_uid                    |
        | - portfolio_table_update_uid                            |
        | - optional published_index_uid -> IndexTable.uid        |
        +---------------------------------------------------------+

PortfoliosUpdater.run(..., update_pointers=True) updates the PortfolioTable TimeIndexTableUpdater pointer fields after the portfolio graph has published. This is enabled by default for portfolio-configuration runs, so examples and callers do not need to manually re-upsert the portfolio row after execution. Pass update_pointers=False only when deliberately running the graph without updating portfolio registry links.

When a run publishes no new executed weights, the workflow must not require a fresh PortfolioWeights TimeIndexTableUpdate. It preserves the existing PortfolioTable.portfolio_weights_table_update_uid and still updates the signal and portfolio-values pointers from the TimeIndexTableUpdate records produced by the current graph run.

Virtual-fund allocation is a separate relationship over account holdings and a target portfolio:

+---------------------+        target_portfolio_uid        +---------------------+
| PortfolioTable      |<-----------------------------------| VirtualFundTable    |
| portfolio identity  |                                    | allocation identity |
+---------------------+                                    | account_uid         |
                                                           +----------+----------+
                                                                      |
                                                                      | account_uid
                                                                      v
+---------------------+        account_uid                +---------------------+
| AccountTable        |<-----------------------------------| AccountHoldingsSet |
| custody account     |                                    | source snapshot    |
+---------------------+                                    +----------+----------+
                                                                      |
                                                                      | source_account_holdings_set_uid
                                                                      v
                                                           +-----------------------------+
                                                           | VirtualFundHoldingsSetTable |
                                                           | allocation set identity     |
                                                           +-------------+---------------+
                                                                         |
                                                                         v
                                                           +-----------------------------+
                                                           | VirtualFundHoldingsStorage  |
                                                           | allocated_quantity          |
                                                           | direction                   |
                                                           | asset_identifier -> Asset   |
                                                           +-----------------------------+

The boundary is intentional:

  • PortfolioTable identifies the portfolio and points at portfolio output storage.
  • VirtualFundTable is core msm account-allocation state that binds an account to a target portfolio.
  • AccountHoldingsSetTable is the source account snapshot.
  • VirtualFundHoldingsSetTable records one allocation view from one source holdings set.
  • VirtualFundHoldingsStorage stores allocated exposure rows, not custody.

Virtual funds are not assets. They should not appear as synthetic rows in AccountHoldingsStorage; account-level virtual-fund exposure is reconstructed from VirtualFundTable, VirtualFundHoldingsSetTable, and VirtualFundHoldingsStorage.

Storage dimensions use explicit names instead of reusing bare unique_identifier: asset_identifier for asset-keyed rows, portfolio_identifier for rebalance state, portfolio value rows, and portfolio weight rows. The portfolio_identifier value is PortfolioTable.unique_identifier and is enforced by storage foreign keys. Rebalance state and portfolio weights also enforce asset_identifier against AssetTable.unique_identifier. Portfolio identity does not require a linked IndexTable row.

See examples/msm_portfolios/portfolio_equal_weights_prepare_schema.py for the schema-preparation stage and examples/msm_portfolios/portfolio_equal_weights_run.py for the normal portfolio run. The reusable implementation lives in examples/msm_portfolios/portfolio_equal_weights_example.py; it reuses the shared crypto Asset example rows, creates or reuses a CRYPTO_24_7 calendar from pandas_market_calendars, publishes example OHLCV bars to ExternalPricesStorage, interpolates those prices, runs PortfolioCalendarEvents, SignalWeights, PortfolioRebalance, PortfolioWeights, and PortfoliosUpdater, and upserts the Portfolio row with calendar_uid plus the published TimeIndexTableUpdater update UIDs. The example narrates each setup, source-price publication, and portfolio step so terminal output explains what was created. It does not create virtual funds or virtual-fund allocation rows; those require an explicit account funding policy and belong in the core account virtual-funds workflow.

Extension Notes

Add new portfolio construction configuration in msm_portfolios.configuration. Add reusable updaters under msm_portfolios.time_index or msm_portfolios.contrib. Add rebalance logic under msm_portfolios.rebalance_strategy. Add portfolio identity persistence through core msm.models, msm.repositories, msm.services, and msm.api. Add portfolio metadata persistence through msm_portfolios.models and msm_portfolios.api.