diff --git a/docs/openedx_learning/decisions/0002-competency-criteria-model.rst b/docs/openedx_learning/decisions/0002-competency-criteria-model.rst index 0c3e40a15..aa056814a 100644 --- a/docs/openedx_learning/decisions/0002-competency-criteria-model.rst +++ b/docs/openedx_learning/decisions/0002-competency-criteria-model.rst @@ -240,11 +240,14 @@ Decision 3. ``oel_tagging_objecttag(object_id)`` 4. ``CompetencyCriteria(oel_tagging_objecttag_id)`` 5. ``CompetencyCriteria(competency_criteria_group_id)`` - 6. ``StudentCompetencyCriteriaStatus(user_id, competency_criteria_id)`` - 7. ``StudentCompetencyCriteriaGroupStatus(user_id, competency_criteria_group_id)`` - 8. ``StudentCompetencyStatus(user_id, oel_tagging_tag_id)`` - 9. ``CompetencyRuleProfile(scope_code)`` (unique -- at most one profile per distinct scope value; a plain unique constraint on the three raw nullable scope columns would not enforce this, since SQL never treats two ``NULL`` values as equal and this project's MySQL backend does not support the conditional/partial unique indexes that would otherwise route around that; see the ``scope_code`` column in Decision 3) - 10. ``CompetencyMasteryStatuses(status)`` (unique) + 6. ``StudentCompetencyCriteriaStatus(user_id, competency_criteria_id)`` (unique) + 7. ``StudentCompetencyCriteriaStatusHistory(user_id, competency_criteria_id)`` + 8. ``StudentCompetencyCriteriaGroupStatus(user_id, competency_criteria_group_id)`` (unique) + 9. ``StudentCompetencyCriteriaGroupStatusHistory(user_id, competency_criteria_group_id)`` + 10. ``StudentCompetencyStatus(user_id, oel_tagging_tag_id)`` (unique) + 11. ``StudentCompetencyStatusHistory(user_id, oel_tagging_tag_id)`` + 12. ``CompetencyRuleProfile(scope_code)`` (unique -- at most one profile per distinct scope value; a plain unique constraint on the three raw nullable scope columns would not enforce this, since SQL never treats two ``NULL`` values as equal and this project's MySQL backend does not support the conditional/partial unique indexes that would otherwise route around that; see the ``scope_code`` column in Decision 3) + 13. ``CompetencyMasteryStatuses(status)`` (unique) 6. Learner progress status concepts (``StudentCompetency*Status`` database tables) @@ -257,6 +260,13 @@ Decision - ``StudentCompetencyStatus`` tracks top-level competency demonstration state. - All learner status rows use a shared lookup table (``CompetencyMasteryStatuses``) so status semantics live in one place and student status tables stay structurally consistent. + Append-only history tables: + + - ``StudentCompetencyCriteriaStatusHistory`` + - ``StudentCompetencyCriteriaGroupStatusHistory`` + - ``StudentCompetencyStatusHistory`` + + Intended update flow (bottom-up materialization): - A learner event updates one ``StudentCompetencyCriteriaStatus`` row. @@ -275,9 +285,9 @@ Decision 2. Add new database table for ``StudentCompetencyCriteriaStatus`` with these columns: - 1. ``id``: unique primary key + 1. ``id``: unique 64-bit primary key (``BigAutoField``); see :ref:`openedx-learning-adr-0005`. 2. ``competency_criteria_id``: Foreign key to ``CompetencyCriterion.id`` - 3. ``user_id``: Foreign key pointing to user_id (presumably the learner's id, although it appears that it is possible for staff to get grades as well) in ``auth_user`` table + 3. ``user_id``: Foreign key with ``db_constraint=False`` pointing to user_id (presumably the learner's id, although it appears that it is possible for staff to get grades as well) in ``auth_user`` table 4. ``status_id``: Foreign key to ``CompetencyMasteryStatuses.id`` 5. ``created``: The timestamp at which the student's criterion status was set. @@ -285,7 +295,7 @@ Decision 1. ``id``: unique primary key 2. ``competency_criteria_group_id``: Foreign key to ``CompetencyCriteriaGroup.id`` - 3. ``user_id``: Foreign key pointing to user_id (presumably the learner's id, although it appears that it is possible for staff to get grades as well) in ``auth_user`` table + 3. ``user_id``: Foreign key with ``db_constraint=False`` pointing to user_id (presumably the learner's id, although it appears that it is possible for staff to get grades as well) in ``auth_user`` table 4. ``status_id``: Foreign key to ``CompetencyMasteryStatuses.id`` 5. ``created``: The timestamp at which the student's criteria-group status was set. @@ -293,7 +303,7 @@ Decision 1. ``id``: unique primary key 2. ``oel_tagging_tag_id``: Foreign key pointing to Tag id - 3. ``user_id``: Foreign key pointing to user_id (presumably the learner's id, although it appears that it is possible for staff to get grades as well) in ``auth_user`` table + 3. ``user_id``: Foreign key with ``db_constraint=False`` pointing to user_id (presumably the learner's id, although it appears that it is possible for staff to get grades as well) in ``auth_user`` table 4. ``status_id``: Foreign key to ``CompetencyMasteryStatuses.id``. This table should have a constraint to only allow status values of “Demonstrated” and “PartiallyAttempted” since it represents overall competency demonstration state, not in-progress states. 5. ``created``: The timestamp at which the student's competency status was set. diff --git a/docs/openedx_learning/decisions/0003-competency-criteria-versioning.rst b/docs/openedx_learning/decisions/0003-competency-criteria-versioning.rst index 0f0f82515..d53256c4e 100644 --- a/docs/openedx_learning/decisions/0003-competency-criteria-versioning.rst +++ b/docs/openedx_learning/decisions/0003-competency-criteria-versioning.rst @@ -44,12 +44,21 @@ For the initial implementation, versioning and traceability of competency achiev - A ``CompetencyRuleProfile`` is "in use" if any ``CompetencyCriterion`` assigned to it (``competency_rule_profile_id``) has an associated ``StudentCompetencyCriteriaStatus`` row. Editing an in-use profile's ``rule_type``/``rule_payload`` requires the same warning and confirmation. - The same warning applies when creating a more specific profile causes existing criteria to be reassigned to it, and when an authoring action switches a criterion between a profile assignment and per-criterion overrides (ADR 0002 Decision 4). -5. Learner status models/tables are append-only history and do not use ``django-simple-history``: +5. Learner status models/tables are updated in-place: - - For ``StudentCompetencyCriteriaStatus``, ``StudentCompetencyCriteriaGroupStatus``, and ``StudentCompetencyStatus``, each status change is stored as a new row with ``created`` as the write timestamp. - - Existing learner status rows are not updated in place. - - Current status is determined by the most recent row for a given learner + target entity (ordered by ``created``, with ``id`` as a tie-breaker). - - Older rows represent the learner status history and remain available for audit/tracing. + - For ``StudentCompetencyCriteriaStatus``, ``StudentCompetencyCriteriaGroupStatus``, and ``StudentCompetencyStatus``, + each status change updates the responsible row. + - Statuses only increase monotonically as described by :ref:`openedx-learning-adr-0005`; + downward status adjustments (for example ``Demonstrated`` to ``PartiallyAttempted``) are prohibited. + +6. Learner status models/tables as in 5. above each get a separate append-only history table not using ``django-simple-history``: + + - For ``StudentCompetencyCriteriaStatusHistory``, ``StudentCompetencyCriteriaGroupStatusHistory``, and ``StudentCompetencyStatusHistory``, + each status acvance is stored as a new row with ``created`` as the write timestamp. + - Existing learner status rows are not updated in place in the history tables. + - Statuses only increase monotonically as described by :ref:`openedx-learning-adr-0005`; + if a change would mean a downward adjustment (for example ``Demonstrated`` to ``PartiallyAttempted``) + or no adjustment, this does not get stored in the history tables. Rejected Alternatives diff --git a/docs/openedx_learning/decisions/0004-competency-mastery-concurrency.rst b/docs/openedx_learning/decisions/0004-competency-mastery-concurrency.rst new file mode 100644 index 000000000..9a575e505 --- /dev/null +++ b/docs/openedx_learning/decisions/0004-competency-mastery-concurrency.rst @@ -0,0 +1,92 @@ +.. _openedx-learning-adr-0004: + +4. How should learner competency mastery be recorded concurrently and at scale? +================================================================================ + +Status +------ +Proposed. + +Context +------- +When a learner is graded on a subsection (or any other learning instrument associated to a competency +with a competency criteria, like a course or rubric criterion), the platform must evaluate whether that grade +demonstrates any attached competencies and record the learner's mastery. Mastery is recorded at +three levels: the criterion (leaf), the criteria group, and the competency. Per +:ref:`openedx-learning-adr-0002` and :ref:`openedx-learning-adr-0005`, all three levels are +*materialized* (stored), not recomputed on read, so that dashboards and other read surfaces stay +fast. A single grade change therefore writes the changed leaf's status and then re-evaluates and +re-writes the derived rows from that leaf up to the competency root. The re-evaluation +is needed for multiple reasons, including notifications, and badge and certificate issuing. Per +:ref:`openedx-learning-adr-0005`, each level is stored as an ACTIVE row updated in place, holding +the current status for a learner and node, plus an append-only HISTORY row per genuine status +advance. + +**Monotonicity: competency statuses only ever move forward.** Per +:ref:`openedx-learning-adr-0005`, every node, at every level, advances through a small status +lattice (``AttemptedNotDemonstrated`` to ``PartiallyAttempted`` to ``Demonstrated``) and is never +lowered later. This holds for leaf nodes, group nodes, and top-level competency masteries. + +Two forces shape how recording should happen: + +- **Same-learner correctness.** A grade change writes the changed leaf and then re-derives the + group and competency rows above it. Leaf rows are always correct, since each leaf is a pure + function of its own grade. The derived rows are the hazard: We want to avoid a case where two evaluations for the same learner + that overlap can each read a stale snapshot of the sibling leaf statuses and each write a derived + roll-up computed from an incomplete picture (a *write-skew*). + +- **Throughput.** Grading is bursty and spans a very large number of learners, so the recording + path must keep up under peak load. + +Decision +-------- + +**1. Every write is a monotone merge, never a blind overwrite.** A node's status is written as +``status := max(stored status, newly computed status)`` (a single ``GREATEST``-style ``UPDATE``, +atomic at the row for the duration of that one statement, with no application-level lock). Because +the merge takes the higher of the two values, it is commutative, idempotent, and insensitive to +order. This is why out-of-order delivery and re-delivery are harmless without sequence tracking. + +**2. When a child advances, its parent is recomputed in the same transaction, under a brief row lock on that parent.** +The merge in mechanism 1 makes a single-row write safe, but a *conjunctive* +parent (for example "demonstrated only when all children are demonstrated") is computed by reading +several child rows first, so two overlapping evaluations for one learner could each read a stale +sibling and compute a parent that is too low. To prevent that, recomputing a parent takes a +row-level lock on the parent row (a ``SELECT ... FOR UPDATE``) before reading its children: two +updates that touch the same parent for the same learner take turns, and the second reads the first's +committed children and computes from the complete picture. Locks are taken child-before-parent up +the path to the root, a consistent order, so concurrent updates cannot deadlock. This is an ordinary +single-row lock. Every read the recorder makes, here and in the mass-recompute path +(:ref:`openedx-learning-adr-0005`), runs against the primary database and never a read replica: +these reads feed the roll-up write and take the row locks above, so a replica's lag would compute a +roll-up from stale siblings. The read-replica offload in :ref:`openedx-learning-adr-0005` is +reserved for the read-only dashboard and reporting paths. + +**3. Entry point: edx-platform subsection grade change.** edx-platform +computes subsection grades in an async celery task (`recalculate_subsection_grade_v3`) triggered by a score-change signal, not on the +request thread. After that task writes the subsection grade, it calls a public openedx-core function +within the same transaction; this function does the monotone merge and the upward roll-up. This should be generalized as needed to other places that trigger a competency status update. + +**4. ACTIVE writes and roll-ups commit atomically with the grade, or the whole task rolls back and retries.** + +**5. The leaf HISTORY append runs outside the transaction, best effort.** Because the leaf HISTORY table +(``StudentCompetencyCriteriaStatusHistory``) may be routed to a +separate physical database (:ref:`openedx-learning-adr-0005`), its append runs after that transaction commits, as a best-effort, +non-blocking write. + + +Rejected Alternatives +--------------------- + +1. Prevent concurrent writes with a deployment-wide lock. + + Once every write is a monotone merge and each + parent recompute is serialized only against concurrent writers of that same node by a brief row lock, correctness holds without needing a larger lock. + +4. Recompute derived levels on read instead of materializing them. + + Settled in :ref:`openedx-learning-adr-0002`. + +8. Send an event to openedx-core and update competency statuses in a separate celery task. + + We want to avoid data drift. Wrapping the competency status update in the grade calculation transaction on edx-platform ensures atomicity. diff --git a/docs/openedx_learning/decisions/0005-competency-mastery-storage.rst b/docs/openedx_learning/decisions/0005-competency-mastery-storage.rst new file mode 100644 index 000000000..056dc1710 --- /dev/null +++ b/docs/openedx_learning/decisions/0005-competency-mastery-storage.rst @@ -0,0 +1,130 @@ +.. _openedx-learning-adr-0005: + +5. How should learner competency status be stored at scale? +=========================================================== + +Status +------ +Proposed. + +Context +------- +Per :ref:`openedx-learning-adr-0002`, competency achievement criteria form a boolean tree. An +internal ``CompetencyCriteriaGroup`` node combines child nodes with an ``AND``/``OR`` +``logic_operator``, can be scoped to a course run, and can nest under a parent group. A +``CompetencyCriterion`` leaf is the tree's terminal node: it points at one tag/object association +and a rule. + +The student mastery statuses tied to these tree nodes are stored in: +- `StudentCompetencyCriteriaStatus` (leaf nodes) +- `StudentCompetencyCriteriaGroupStatus` (middle nodes) +- `StudentCompetencyStatus` (top-level) + +For each of these, we also need to persist history, because we need an audit trail to understand +why a learner did or didn't achieve mastery of a particular competency or any of the associated "measurement instruments" +(gradeable subsections). + +Storing every leaf multiplies out at scale. A course can carry on the order of 200 leaf criteria, +so the leaf level is where the row count concentrates: the leaf table (learners x attempted +leaves) potentially reaches the low billions for an Open edX instance with millions of learners. The dominant +multiplier is this per-leaf breadth (roughly 200x per course), not time. Mastery is monotonic (see +"Advance-only banking" below): a node can only advance through the small status lattice, at most a handful of +forward steps ever. + +That scale is not, on its own, what makes a relational database struggle. A point lookup against a +billion-row table backed by the right composite index is a logarithmic-time index seek regardless +of the table's size; the dashboard reads this feature performs are exactly such point lookups. What +billions of rows makes painful is schema migrations, backups, and any non-indexed or aggregate +query. + +Decision +-------- + +**Alleviate performance concerns by following established edx-platform practices.** +edx-platform has already large tables of similar magnitude, such as `StudentModule`, +and they have proven themselves. +This means that as long as we follow their structure, we can have confidence that we are +not introducing any huge new problems. +They do not use physical partitioning, but work by using 64-bit primary keys, +enabling optional read-replica offloads, and splitting out a history table that +uses a dedicated database router. The decisions below are designed to mirror these +existing patterns. + +**Store statuses / mastery at every level, each split into ACTIVE and HISTORY.** The leaf, group, and +competency levels each keep one ACTIVE row per learner and node, updated in place, so reading a +learner's current status is a direct indexed lookup rather than a scan for the most recent of many +rows. Each level also has a parallel append-only HISTORY table that records one row per genuine +status advance, for audit and point-in-time reconstruction. Because status only advances, the status +at any past time is the latest recorded advance at or before that time, so point-in-time is fully +reconstructable from HISTORY. Because status is monotonic, the number of advances per node is bounded +by the status lattice (a small constant), so HISTORY grows with learners and nodes, not with time, +and stays about the same order of size as ACTIVE. Keeping ACTIVE and HISTORY separate still pays: +ACTIVE is a single in-place current row optimized for the dashboard point lookup and is the row +per-learner concurrency is anchored on (:ref:`openedx-learning-adr-0004`), while HISTORY is +append-only. + +**64-bit primary keys from the start.** The leaf ACTIVE and HISTORY tables use a 64-bit +``BigAutoField`` primary key, chosen up front, mirroring edx-platform's +``UnsignedBigIntAutoField`` on ``PersistentSubsectionGrade`` ("primary key will need to be +large for this table"). Changing a primary-key type on a billion-row table later is +prohibitively expensive. + +**A dedicated database alias and router for the leaf HISTORY table, baked in from the start.** +This only applies to the `StudentCompetencyCriteriaStatusHistory` table. +The leaf HISTORY table (``StudentCompetencyCriteriaStatusHistory``), the dominant table in this +model, is the one learner-status table assigned to a dedicated Django database alias through a +database router, mirroring edx-platform's courseware-history router +(``StudentModuleHistoryExtended``), which is likewise a history table. The alias defaults to the +main database. + +**No database-level foreign keys to `user_id` on ACTIVE or HISTORY table.** +Foreign keys to `user_id` must have `db_constraint=False` set. + +**Enable read-replica offload for heavy reads for the leaf tables.** +This only applies to the ACTIVE `StudentCompetencyCriteriaStatus` and HISTORY +`StudentCompetencyCriteriaStatusHistory` tables. +Prior art: ``edx_django_utils``'s ``read_replica_or_default()``. + +**Advance-only banking, monotonic.** Once a node reaches ``Demonstrated`` its ACTIVE row is retained +("banked"): the recorder never automatically regresses it, not on a later downward grade correction +and not on a criteria change. This applies at every level, including the leaf. A genuine downward +grade correction does not advance the status, so it writes no HISTORY row and leaves the banked +ACTIVE status unchanged; because HISTORY records only advances, it never carries suppressed +regressions. Reversing a banked status is a separate administrative action, out of scope here. +This monotonicity is what makes out-of-order and duplicate delivery safe, since a late or replayed +event can never lower a status, and :ref:`openedx-learning-adr-0004` relies on it. + +**Retroactive criteria changes are monotonic for the learner.** A retroactive edit can newly grant +or preserve mastery, but it never silently revokes it, and it never rewrites a learner's recorded +leaf mastery downward. + +Rejected Alternatives +--------------------- + +1. Compute leaves transiently, never store them). + +Leaves could compute demonstration based on their rules together with the group-node +status together. That would eliminate the largest tables. But this does not account +for competency tree edits, which would result in leaf statuses being incorrect. + +2. Keep everything append-only (no ACTIVE table); current status is the latest row. + +Even with HISTORY bounded by monotonicity, a single in-place ACTIVE row is a +cheaper and simpler dashboard read than resolving the latest advance out of a node's history, +and it is the row per-learner concurrency in :ref:`openedx-learning-adr-0004` is anchored on. + +3. Make a separate physical database mandatory, or partition/shard the leaf tables, up front. + +Forcing a separate database or a partitioning scheme on +everyone buys nothing the router does not, at a real operational cost. Partitioning and +sharding remain available to revisit if a specific need is proven. + +4. Store child evaluations on the parent group row instead of a leaf ACTIVE table (an enriched + attained-set). + +Structural robustness is valued over the hot-store saving. First-class leaf rows +keep a leaf's frozen mastery independent of how the criteria tree is later restructured, and +avoid re-incurring the denormalized-array correctness burden that this decision removed by +storing leaves. The hot-store reduction is real but does not address the largest table, and the +single-row group read it optimizes is served acceptably by an indexed range read of a learner's +leaf rows.