Skip to content

0022. Thin Alembic MetaTable Migration Integration

Status

Accepted and implemented

Amended by ADR 0024 for the top-level src/migrations/ provider package and namespace-scoped revision locations.

Context

ms-markets originally explored a project-owned MetaTable migration runner. That architecture is obsolete. The SDK now owns the provider-based Alembic workflow, scoped migration connection acquisition, version-table handling, MetaTable registration refresh, and the mainsequence migrations CLI.

The current SDK command surface is:

mainsequence migrations current
mainsequence migrations revision
mainsequence migrations upgrade
mainsequence migrations downgrade

ms-markets should not reintroduce a package-owned msm migrations command group or a custom migration ledger.

Decision

ms-markets provides one SDK-compatible Alembic provider:

migrations:migration

The provider lives in:

src/migrations/

and uses this script location:

migrations:

The repository has three import packages with MetaTables:

msm
msm_portfolios
msm_pricing

They are handled by one migration provider, not three. They share MarketsBase.metadata, one Alembic script location, one package-specific Alembic version table, and one ms-markets revision graph.

There must not be separate providers such as:

msm_portfolios.migrations:migration
msm_pricing.migrations:migration

unless those packages become independently installed distributions with independent schema lifecycles.

Operational Commands

Administrators use the SDK CLI directly:

mainsequence migrations current --provider migrations:migration --json
mainsequence migrations revision --provider migrations:migration --autogenerate -m "describe change"
mainsequence migrations upgrade --provider migrations:migration head
mainsequence migrations downgrade --provider migrations:migration <revision>

There is no msm migrations ... compatibility alias, command wrapper, or project-owned migration CLI surface.

Provider Contract

The provider object is defined in src/migrations/__init__.py:

from mainsequence.meta_tables.migrations import (
    AlembicMetaTableMigration,
    AlembicVersionMetaTable,
)

from migrations.registry import metatable_provider_models
from msm.base import MARKETS_SCHEMA, MARKETS_TABLE_APP, MarketsBase, markets_table_name
from msm.settings import markets_auto_register_namespace, markets_identifier, markets_namespace


class MarketsAlembicVersion(AlembicVersionMetaTable):
    __metatable_namespace__ = markets_namespace()
    __metatable_identifier__ = markets_identifier("msm.alembic_version")
    __alembic_version_schema__ = MARKETS_SCHEMA
    __alembic_version_table_name__ = markets_table_name(
        MARKETS_TABLE_APP,
        "alembic_version",
        suffix=markets_auto_register_namespace(),
    )
    __alembic_version_column_name__ = "version_num"


migration = AlembicMetaTableMigration(
    package="msm",
    migration_namespace=markets_namespace(),
    script_location="migrations:",
    target_metadata=MarketsBase.metadata,
    alembic_registry=MarketsAlembicVersion,
    metatable_models=metatable_provider_models(),
)

src/migrations/registry.py owns the provider model universe. It is the installed-app-style registry for this package, not migration history. It must include the combined msm, msm_portfolios, and msm_pricing SQLAlchemy model graph, de-duplicated by model identity and physical table name.

Alembic Environment

src/migrations/ is a standard Alembic script location:

src/migrations/
  __init__.py
  env.py
  registry.py
  script.py.mako
  versions/
    <namespace_slug>/
      0001_migration.py

env.py delegates to the selected provider, applies the SDK migration owner role on online connections, and uses the provider filters for Alembic autogenerate.

Revision files are normal Alembic Python revisions generated by:

mainsequence migrations revision --provider migrations:migration --autogenerate -m "..."

They must contain normal Alembic operations such as op.create_table(...), op.add_column(...), and op.create_index(...). They must not contain custom SDK operation dictionaries or package-authored SQL manifests.

Runtime Boundary

msm.start_engine(...) remains attach-only. Runtime startup must not:

  • generate Alembic revisions;
  • apply SQL;
  • register the Alembic version table;
  • register application MetaTables as a side effect of API or DataNode use.

Runtime code attaches to already-registered backend MetaTable and TimeIndexMetaTable resources by canonical SQLAlchemy table name.

Implementation Tasks

  • [x] Remove the obsolete project migration runner and custom operation parsing.
  • [x] Remove old SDK migration model usage from the normal runtime path.
  • [x] Define MarketsAlembicVersion.
  • [x] Define one AlembicMetaTableMigration provider.
  • [x] Use one provider for msm, msm_portfolios, and msm_pricing.
  • [x] Keep msm_portfolios.migrations:migration and msm_pricing.migrations:migration absent.
  • [x] Move the canonical provider package to src/migrations/.
  • [x] Use the provider reference migrations:migration.
  • [x] Add standard Alembic env.py and script.py.mako.
  • [x] Delete the legacy custom migration command group.
  • [x] Keep migration provider registration free of project-local inventory hooks.
  • [x] Document that runtime startup is attach-only and does not migrate schema.

Success Criteria

The repository is aligned with the SDK migration machinery when:

  • load_alembic_metatable_migration_provider("migrations:migration") loads the provider;
  • the loaded provider includes msm, msm_portfolios, and msm_pricing MetaTables in one de-duplicated model scope;
  • no msm_portfolios or msm_pricing migration providers exist;
  • mainsequence migrations revision --provider migrations:migration --autogenerate -m "..." creates normal Alembic revision files under the active namespace location;
  • mainsequence migrations upgrade --provider migrations:migration head executes through SDK-scoped migration credentials;
  • msm migrations ... is not available;
  • runtime bootstrap docs describe msm.start_engine(...) as attach-only;
  • no runtime code depends on the removed custom migration runner.

Consequences

The migration integration stays small. ms-markets owns provider scope, package Alembic files, and catalog inventory refresh. The SDK owns revision generation, scoped connection acquisition, Alembic execution, migration owner role propagation, version-table binding, and provider MetaTable refresh.