Competency Mastery Concurrency ADR by jesperhodge · Pull Request #657 · openedx/openedx-core

jesperhodge · 2026-07-10T16:09:08Z

UPDATE: ADRs in this PR have been completely rewritten as of July 17, 2026.

Strong involvement from Claude (AI).

I haven't fully reviewed changes to diagrams and ADRs 2 and 3 yet, made with Claude, need my own review still.

Summary

These are some significant changes to the data model and approach. They will warrant a good amount of discussion.
A main concern is that the leaf node table and leaf node history table could possibly contain billions of rows, at the scale of the PersistentSubsectionGrade table of edx-platform. The current solution accepts this, but is this acceptable for OpenEdx? If not, there is also a rejected alternative to not store leaf nodes and just compute them at read time, but that would make it harder to keep subsection statuses frozen so that later competency criteria changes don't change the status.

ADR 4: How learner competency mastery is recorded
correctly under concurrent, out-of-order grade-change events without
paying a per-event serialization cost at scale.

ADR 5: How competency criteria statuses for learners are stored at scale, given the tables
can be massive (billions of rows).

openedx-webhooks · 2026-07-10T16:09:13Z

Thanks for the pull request, @jesperhodge!

This repository is currently maintained by @axim-engineering.

Once you've gone through the following steps feel free to tag them in a comment and let them know that your changes are ready for engineering review.

🔘 Get product approval

If you haven't already, check this list to see if your contribution needs to go through the product review process.

If it does, you'll need to submit a product proposal for your contribution, and have it reviewed by the Product Working Group.
- This process (including the steps you'll need to take) is documented here.
If it doesn't, simply proceed with the next step.

🔘 Provide context

To help your reviewers and other members of the community understand the purpose and larger context of your changes, feel free to add as much of the following information to the PR description as you can:

Dependencies

This PR must be merged before / after / at the same time as ...
Blockers

This PR is waiting for OEP-1234 to be accepted.
Timeline information

This PR must be merged by XX date because ...
Partner information

This is for a course on edx.org.
Supporting documentation
Relevant Open edX discussion forum threads

🔘 Get a green build

If one or more checks are failing, continue working on your changes until this is no longer the case and your build turns green.

Details

Where can I find more information?

If you'd like to get more details on all aspects of the review process for open source pull requests (OSPRs), check out the following resources:

When can I expect my changes to be merged?

Our goal is to get community contributions seen and reviewed as efficiently as possible.

However, the amount of time that it takes to review and merge a PR can vary significantly based on factors such as:

The size and impact of the changes that it introduces
The need for product review
Maintenance status of the parent repository

💡 As a result it may take up to several weeks or months to complete a review and merge your PR.

Rejected-alternative items 1 and 2 nested Pros/Cons bullet lists directly under a continuation paragraph at a different indent with no blank line separator, and item 3's closing paragraph was indented to neither the list body nor its sub-bullets. docutils flagged these as errors under -W, failing the readthedocs build.

mgwozdz-unicon

Following up on feedback I'd shared earlier outside this PR favoring the batch-lock approach over keyed partitioning, given the Kafka/Redis dependency and the accuracy-over-latency tradeoff for this use case: this revision reflects that direction, and the Kafka/Redis coupling concern is resolved.

I'm wondering if we should also consider a per-learner lock instead of the single deployment-wide lock. A DB advisory lock keyed on a hash of user_id would give the same same-learner serialization the chosen approach relies on, but would let different learners' batches run in parallel, which the current design gives up entirely. It also sidesteps everything Alternative 1 was rejected for: no event-transport dependency, no partition-key contract with openedx-platform, no reconciliation backstop.

Rejected Alternative 2's argument doesn't quite cover this: it treats "per-learner isolation" as equivalent to "keyed partitioning" ("Making parallel evaluation correct requires... per-learner isolation, i.e. the keyed-partitioning alternative above"), but a per-learner DB lock gets you per-learner isolation without touching the transport at all. If there's a reason this doesn't hold up here (lock-management overhead across many concurrent per-learner locks, contention patterns, something else), that reasoning is worth adding to the doc.

Requesting changes to get this addressed, either by adding the analysis to Rejected Alternatives or by folding it into the Decision.

ormsbee · 2026-07-16T03:14:18Z

+- **Same-learner correctness.** Grade-change events arrive asynchronously and can be delivered out
+  of order and processed on more than one worker. Because writes are append-only, two evaluations
+  for the same learner that overlap can each read a stale snapshot of the sibling leaf statuses and
+  each append a derived roll-up computed from an incomplete picture (a write-skew). Leaf rows are
+  always correct, since each leaf is a pure function of its own grade; only the derived
+  group/competency rows can be left wrong. Nothing crashes and no constraint is violated, but a
+  learner's stored competency status can be silently incorrect.


Maybe this is a silly question, but can't we mitigate this by making sure to commit the transaction of the leaf statuses, and then re-read the latest commited state of the leaves before doing the roll-up?

Another thing we can do (if it's necessary) is to arrange it so that the roll-ups are a log that have an incrementing version number, and create a composite unique index that would cause a conflict that would reject concurrent writes. The losing write would have to catch the error, increment the version number again, and re-read the database state.

I think this question is not relevant anymore with the current version, right? Since now we only process on one worker at a time.

jesperhodge · 2026-07-17T19:18:04Z

Following up on feedback I'd shared earlier outside this PR favoring the batch-lock approach over keyed partitioning, given the Kafka/Redis dependency and the accuracy-over-latency tradeoff for this use case: this revision reflects that direction, and the Kafka/Redis coupling concern is resolved.

I'm wondering if we should also consider a per-learner lock instead of the single deployment-wide lock. A DB advisory lock keyed on a hash of user_id would give the same same-learner serialization the chosen approach relies on, but would let different learners' batches run in parallel, which the current design gives up entirely. It also sidesteps everything Alternative 1 was rejected for: no event-transport dependency, no partition-key contract with openedx-platform, no reconciliation backstop.

Rejected Alternative 2's argument doesn't quite cover this: it treats "per-learner isolation" as equivalent to "keyed partitioning" ("Making parallel evaluation correct requires... per-learner isolation, i.e. the keyed-partitioning alternative above"), but a per-learner DB lock gets you per-learner isolation without touching the transport at all. If there's a reason this doesn't hold up here (lock-management overhead across many concurrent per-learner locks, contention patterns, something else), that reasoning is worth adding to the doc.

Requesting changes to get this addressed, either by adding the analysis to Rejected Alternatives or by folding it into the Decision.

@mgwozdz-unicon I'll improve the description in the ADR.

Here's why this solution is not good:

It fights the batching that the performance depends on. The chosen design's throughput comes from batching across learners: one bulk read of current statuses, evaluate the whole batch in memory, one bulk write — a few round-trips regardless of how many learners/events are in the batch. A per-learner lock is naturally per-learner (in the earlier design, per-event): acquire lock → read that learner → evaluate → write → release, one learner at a time. That reintroduces per-learner (or per-event) transaction/commit overhead plus lock acquire/release churn, which is exactly the overhead batching was collapsing. Under bursty grading across many learners, that per-unit overhead dominates.

Lock-lifecycle machinery would be multiplied across millions of keys. One deployment-wide lock has exactly one lifecycle to run: acquisition, timeout, stale-lock recovery on a crashed worker. A per-learner lock needs a per-learner lock table (or keyed advisory locks) and that same lifecycle replicated across potentially millions of learner keys, held and waited on concurrently, plus a connection+worker tied up per contended lock.

ormsbee · 2026-07-17T20:17:38Z

+      2. ``competency_criteria_id``: logical foreign key to ``CompetencyCriterion.id`` (no database-level constraint)
+      3. ``user_id``: logical foreign key to the learner's user id (no database-level constraint)
+      4. ``status_id``: foreign key to ``CompetencyMasteryStatuses.id``
+      5. ``effective_source_timestamp``: the timestamp of the grade change this status was computed from, used for the out-of-order defense in :ref:`openedx-learning-adr-0004`


How is effective_source_timestamp different from created?

This is premised on how ADR 0004 is now - it very well might change when we have incorporated your latest feedback; for example, if we integrate everything in a grade update transaction, then we should remove this new field.
I updated the description - it tracks when the grade was changed, not the competency, for async defensiveness per ADR 0004 as it is right now, but this is subject to change.

ormsbee

It seems like this entire ADR's premise is that per-event processing is prohibitively expensive, and we must therefore batch event processing under various locking mechanisms. It's not clear to me why that is. The subsection grade update is already a somewhat expensive, async task. We could plausibly wrap any updates we're doing to competency mastery data in the same transaction that writes the subsection grade update, so that those things are always kept in sync.

Approximately much work are we expecting to happen in order to do the mastery recalculation?

jesperhodge · 2026-07-20T16:09:46Z

It seems like this entire ADR's premise is that per-event processing is prohibitively expensive, and we must therefore batch event processing under various locking mechanisms. It's not clear to me why that is. The subsection grade update is already a somewhat expensive, async task. We could plausibly wrap any updates we're doing to competency mastery data in the same transaction that writes the subsection grade update, so that those things are always kept in sync.

Approximately much work are we expecting to happen in order to do the mastery recalculation?

@ormsbee You're right, and it changed my thinking.

I changed the ADR according to this:

Per-event mastery work is actually small, a handful of reads and writes for one learner; the big numbers in the ADR (~200 leaves/course, billions of rows) are platform totals, not the cost of one recalculation. Throughput was never really the issue.

The thing we were actually protecting against is write-skew: two concurrent updates for the same learner can each roll up a status from a stale view of sibling data. But mastery already only moves forward from non-completion to completion, and we can lean on that harder: if every write is "take the higher of stored and computed," concurrent updates converge on their own, no coordination needed. That's the piece that lets us drop the lock, and the throughput ceiling it created goes with it, because now updates can run in parallel.

Which brings us back to your suggestion. It works now, and I don't have a strong preference yet between two shapes:

A) Recording mastery right inside the grade transaction, as you proposed. Simple and real-time, though it ties mastery's fate to the grade write and assumes they share a database. A tradeoff to point out: mastery work then sits on the synchronous grading path, so a slow mastery query or a bug adds latency to, or rolls back, the grade write (grades are the more critical data).
B) A signal into openedx-core after the grade write. Keeps mastery decoupled, at the cost of a brief window where they're not yet in sync (which seems fine given mastery is already eventually-consistent by nature).
Batching drops out as a nice-to-have for high-volume bursts rather than something correctness depends on.

Curious which of the two shapes feels right to you, or if there's a third way we're not seeing.

mgwozdz-unicon · 2026-07-20T16:37:52Z

@jesperhodge

Which brings us back to your suggestion. It works now, and I don't have a strong preference yet between two shapes:

A) Recording mastery right inside the grade transaction, as you proposed. Simple and real-time, though it ties mastery's fate to the grade write and assumes they share a database. A tradeoff to point out: mastery work then sits on the synchronous grading path, so a slow mastery query or a bug adds latency to, or rolls back, the grade write (grades are the more critical data).

B) A signal into openedx-core after the grade write. Keeps mastery decoupled, at the cost of a brief window where they're not yet in sync (which seems fine given mastery is already eventually-consistent by nature).
Batching drops out as a nice-to-have for high-volume bursts rather than something correctness depends on.

Curious which of the two shapes feels right to you, or if there's a third way we're not seeing.

I don't think we should go with Option A because we do not intend for mastery to always be tied to a subsection grade. We've actually proposed tying mastery to several other things (course completion, unit grade (a concept that doesn't exist in the platform currently), problem grade, and rubric criterion grade (also a concept that doesn't exist in the platform currently)). So I would go with option B out of these two.

mgwozdz-unicon · 2026-07-20T16:47:09Z

@@ -0,0 +1,150 @@
+# Competency data model (updated)


Unless others disagree, I think I'd recommend updating the existing PNG or trading it out for the mermaid diagram in the existing ADR 0002, as well as making it clearer that we already supported the active vs history dynamic and how the mastery levels fit in and anything else needed, rather than adding this extra file. We want to try to keep the ADRs light and concise and only change what really needs to be changed. We've also been asked to try to keep the implementation detail in ADRs to a minimum, which may also be relevant here.

mgwozdz-unicon · 2026-07-20T19:19:44Z

+      6. ``created``: on ACTIVE, when the row was first written; on HISTORY, when this advance was appended

-   3. Add a new database table for ``StudentCompetencyCriteriaGroupStatus`` with these columns:
+      The ACTIVE table additionally has a ``modified`` timestamp (last in-place update) and is unique


I feel like these modified fields should get enumerated like the rest of the fields.

Is it better now?

ormsbee · 2026-07-21T06:23:50Z

Quick thoughts:

We want to keep state in the same transaction as much as it's feasible to do so. Where we diverge from that, it's because operational considerations have forced us to compute it asynchronously in a separate process (e.g. subsection grade calculation).
However, in the case where we are doing things asynchronously like grades, if that grade is the record of truth that mastery will be based on, we should trigger mastery recalculation in the same transaction if possible. The same goes for passing a course, getting a Unit grade, etc. -- whenever the record of truth changes for that, we should try to keep things in the same transaction unless we absolutely can't. Correcting grades is a pain, but a well understood one. Adding another place where we can have drift like this is going to cause trouble for operators eventually.
My understanding is that moving forward from non-completion to completion can still cause race condition problems, if two separate transactions each would bring a student to right below some cutoff, but the combination would bring the student over. Or if the student is given some kind of score/level of mastery. Given this case, I think it's safest to force race conditions to fail using unique constraints.

mgwozdz-unicon · 2026-07-21T14:18:01Z

@ormsbee @jesperhodge

Quick thoughts:

We want to keep state in the same transaction as much as it's feasible to do so. Where we diverge from that, it's because operational considerations have forced us to compute it asynchronously in a separate process (e.g. subsection grade calculation).

However, in the case where we are doing things asynchronously like grades, if that grade is the record of truth that mastery will be based on, we should trigger mastery recalculation in the same transaction if possible. The same goes for passing a course, getting a Unit grade, etc. -- whenever the record of truth changes for that, we should try to keep things in the same transaction unless we absolutely can't. Correcting grades is a pain, but a well understood one. Adding another place where we can have drift like this is going to cause trouble for operators eventually.

My understanding is that moving forward from non-completion to completion can still cause race condition problems, if two separate transactions each would bring a student to right below some cutoff, but the combination would bring the student over. Or if the student is given some kind of score/level of mastery. Given this case, I think it's safest to force race conditions to fail using unique constraints.

I could have sworn that there was a reason that we chose to use openedx-events for this process when we last discussed it in January, but I'm having a hard time remembering what it was. My first thought was dependency direction, but that doesn't seem right. I wonder if it was just that we were thinking that CBE would be taking more of a plugin shape so we wanted to keep it decoupled from openedx-platform. Outside of the other cons that Jesper raised for the single transaction (increased latency for the transaction and shared database), I don't have an issue with the single transaction approach if we want to accept those tradeoffs.

Also as a side note: I'm pretty sure openedx-platform and openedx-core share a database already, so the "shared database" point in ADR 4 (line 220) probably needs some clarification.

jesperhodge · 2026-07-21T14:24:33Z

Also as a side note: I'm pretty sure openedx-platform and openedx-core share a database already, so the "shared database" point in ADR 4 (line 220) probably needs some clarification.

I'll remove it, it's a Claude remnant not needed.

jesperhodge · 2026-07-21T14:27:50Z

@ormsbee @mgwozdz-unicon sounds like we're in alignment, so I'll update the ADR to follow Dave's recommendation.

ormsbee · 2026-07-21T14:50:49Z

Yeah, I was thinking that there's still value in using openedx-events to decouple from platform, without the message bus aspect of it.

bradenmacdonald · 2026-07-21T21:35:54Z

+      Foreign keys that would cross the alias boundary (a learner-status row referencing a
+      criteria-definition row, and the reference to the user) are declared without database-level
+      constraints, as edx-platform does for ``PersistentSubsectionGrade.user_id``, so the tables can
+      live in different databases; the delete protection of :ref:`openedx-learning-adr-0002` is
+      therefore enforced in application code, not by a database constraint.


As mentioned in our discussion on #661 , not having foreign keys is definitely a downside of this approach, and making the deletion protection more complicated and fragile.

as edx-platform does for PersistentSubsectionGrade.user_id, so the tables can live in different databases

Is there an ADR or documentation around PersistentSubsectionGrade.user_id ? I can see that it indeed is not a foreign key, but I can't find any explanation of that in the code nor ADRs, so I'm wondering if it is actually meant to be deployed on a separate database or not. As far as I can tell, the only table that has ever been designed to run on a separate database is the StudentModuleHistoryExtended table (it's the only one with a router).

docs: competency ADR 4

f96a350

openedx-webhooks added the open-source-contribution PR author is not from Axim or 2U label Jul 10, 2026

openedx-webhooks added this to Contributions Jul 10, 2026

github-project-automation Bot moved this to Needs Triage in Contributions Jul 10, 2026

jesperhodge added 2 commits July 13, 2026 11:05

docs: reject partition-key variant

847b789

jesperhodge changed the title ~~docs: competency ADR 4~~ docs: Competency Mastery Concurrency Jul 13, 2026

jesperhodge changed the title ~~docs: Competency Mastery Concurrency~~ Competency Mastery Concurrency ADR Jul 13, 2026

mphilbrick211 moved this from Needs Triage to Ready for Review in Contributions Jul 13, 2026

jesperhodge requested review from bradenmacdonald, kdmccormick, mgwozdz-unicon and ormsbee July 15, 2026 13:56

jesperhodge marked this pull request as draft July 15, 2026 14:13

jesperhodge removed request for bradenmacdonald, kdmccormick, mgwozdz-unicon and ormsbee July 15, 2026 14:13

mgwozdz-unicon suggested changes Jul 16, 2026

View reviewed changes

ormsbee reviewed Jul 16, 2026

View reviewed changes

jesperhodge added 3 commits July 17, 2026 09:59

docs: make ADR 4 work without event bus

7c77aea

docs: finalize concurrency approach ADR

dc01a87

docs: concurrency and storage docs and diagrams

808a228

jesperhodge requested review from bradenmacdonald, kdmccormick, mgwozdz-unicon and ormsbee July 17, 2026 19:03

jesperhodge marked this pull request as ready for review July 17, 2026 19:05

docs: add reasoning for rejecting per-learner batching

78bc16c

ormsbee reviewed Jul 17, 2026

View reviewed changes

docs: choose monotonic non-locking option

4cfb092

mgwozdz-unicon reviewed Jul 20, 2026

View reviewed changes

docs: improve atomicity pattern

c4354d5

mgwozdz-unicon reviewed Jul 20, 2026

View reviewed changes

docs: clean up ADR

59c5467

jesperhodge added 2 commits July 21, 2026 09:24

docs: clean up ADR 0002

385ccea

docs: delete superfluous diagrams

cc44205

mgwozdz-unicon mentioned this pull request Jul 21, 2026

docs: add ADR for persisted competency deletability signal #661

Open

bradenmacdonald reviewed Jul 21, 2026

View reviewed changes

Uh oh!

Conversation

jesperhodge commented Jul 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Uh oh!

openedx-webhooks commented Jul 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mgwozdz-unicon left a comment

Choose a reason for hiding this comment

Uh oh!

ormsbee Jul 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jesperhodge commented Jul 17, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ormsbee left a comment

Choose a reason for hiding this comment

Uh oh!

jesperhodge commented Jul 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mgwozdz-unicon commented Jul 20, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ormsbee commented Jul 21, 2026

Uh oh!

mgwozdz-unicon commented Jul 21, 2026

Uh oh!

jesperhodge commented Jul 21, 2026

Uh oh!

jesperhodge commented Jul 21, 2026

Uh oh!

ormsbee commented Jul 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

jesperhodge commented Jul 10, 2026 •

edited

Loading

openedx-webhooks commented Jul 10, 2026 •

edited

Loading

ormsbee Jul 16, 2026 •

edited

Loading

jesperhodge commented Jul 20, 2026 •

edited

Loading

ormsbee commented Jul 21, 2026 •

edited

Loading