fix(mongo-adapter): retry a transaction aborted with a transient error (#11515)

Co-authored-by: Maxwell Afamefuna Nzekwe <70326615+maxnze1@users.noreply.github.com>
Co-authored-by: Gustavo Valverde <g.valverde02@gmail.com>

.changeset/mongo-transient-transaction-retry.md

@@ -0,0 +1,8 @@
+---
+"@better-auth/core": patch
+"@better-auth/mongo-adapter": patch
+---
+
+If you pass `client` to `mongodbAdapter` on a replica set, concurrent requests that write the same documents no longer fail with a 500. Before, MongoDB aborted all but one of them with a write conflict, so a SCIM directory that provisions several users at once returned 500 for most of them. The adapter now retries such a transaction, as MongoDB requires, for up to 120 seconds or until your `MongoClient`'s `timeoutMS`. Under contention a request can take longer instead of failing.
+
+A retry runs the work again. `databaseHooks` `before` hooks can run more than once for one request, and `after` hooks still run once, after the commit. Work that the request does outside the database before it commits can also repeat on a retried request. For example, an email sign-up can send its verification email twice. Requests that succeed on the first attempt are unaffected, and so are other adapters.

docs/content/docs/adapters/mongo.mdx

@@ -38,6 +38,10 @@ export const auth = betterAuth({
 });
 ```
 
+When two requests write the same document at the same time, MongoDB aborts one of the transactions with a write conflict and expects the application to retry it. The adapter retries such a transaction for up to 120 seconds, so a request under heavy contention can take longer instead of failing. To lower that limit, set the `timeoutMS` option on your `MongoClient`. It also applies to every other operation on that client.
+
+A retry runs the whole operation again, including `before` [database hooks](/docs/concepts/database#database-hooks) and plugin callbacks that receive the transaction. Send emails and call other services from `after` hooks, which run once, after the commit.
+
 ### Standalone server
 
 A standalone MongoDB server does not support transactions. Create `db` with `MongoClient` and pass it to the adapter without the `client` option:

docs/content/docs/concepts/database.mdx

@@ -883,7 +883,7 @@ There are two types of hooks you can define:
 #### 1. Before Hook
 
 * **Purpose**: This hook is called before the respective entity (user, session, or account) is created, updated, or deleted.
-* **Behavior**: If the hook returns `false`, the operation will be aborted. And If it returns a data object, it'll replace the original payload.
+* **Behavior**: If the hook returns `false`, the operation will be aborted. And If it returns a data object, it'll replace the original payload. On MongoDB with transactions, a before hook can run more than once for one operation: when MongoDB aborts the transaction because of a concurrent write, the adapter runs it again. Send emails and call other services from an after hook instead.
 
 #### 2. After Hook
 

packages/core/src/context/transaction.test.ts

@@ -223,6 +223,40 @@ describe("runWithTransaction", () => {
 		});
 	});
 
+	/**
+	 * @see https://github.com/better-auth/better-auth/issues/11514
+	 */
+	it("runs only the hooks of the attempt that commits when the adapter retries the callback", async () => {
+		const transactionAdapter = {} as DBTransactionAdapter;
+		let attempts = 0;
+		const adapter = {
+			transaction: async <R>(
+				callback: (trx: DBTransactionAdapter) => Promise<R>,
+			) => {
+				try {
+					return await callback(transactionAdapter);
+				} catch {
+					return callback(transactionAdapter);
+				}
+			},
+		} as DBAdapter;
+		const hookRuns: number[] = [];
+
+		await runWithTransaction(adapter, async () => {
+			attempts += 1;
+			const attempt = attempts;
+			await queueAfterTransactionHook(async () => {
+				hookRuns.push(attempt);
+			});
+			if (attempt === 1) {
+				throw new Error("write conflict");
+			}
+		});
+
+		expect(attempts).toBe(2);
+		expect(hookRuns).toEqual([2]);
+	});
+
 	it("reports a handled after-commit hook failure without rejecting committed work", async () => {
 		const { adapter } = createTransactionHarness();
 		const onError = vi.fn();

packages/core/src/context/transaction.ts

@@ -130,12 +130,15 @@ export const runWithTransaction = async <
 			// connection a single-connection store would need for the lookup.
 			const pendingSchemaCheck = runtimeSchemaCheckFor(adapter)?.();
 			if (pendingSchemaCheck) await pendingSchemaCheck;
-			const pendingHooks: Array<() => Promise<void>> = [];
+			let pendingHooks: Array<() => Promise<void>> = [];
 			let result: Awaited<R>;
 			let error: unknown;
 			let hasError = false;
 			try {
 				result = await adapter.transaction(async (trx) => {
+					// An adapter may run this callback again after a transient abort.
+					// Only the hooks of the attempt that commits may run.
+					pendingHooks = [];
 					return als.run(
 						{
 							adapter: trx as unknown as StoredAdapter,

packages/core/src/db/adapter/index.ts

@@ -509,6 +509,10 @@ export type DBAdapter<Options extends BetterAuthOptions = BetterAuthOptions> = {
 	/**
 	 * Execute multiple operations in a transaction.
 	 * If the adapter doesn't support transactions, operations will be executed sequentially.
+	 *
+	 * An adapter may run `callback` more than once, for example when MongoDB
+	 * aborts the transaction with a transient error, so work outside the
+	 * database belongs in `queueAfterTransactionHook`.
 	 */
 	transaction: <R>(
 		callback: (trx: DBTransactionAdapter<Options>) => Promise<R>,

packages/mongo-adapter/src/mongodb-adapter.test.ts

@@ -187,7 +187,8 @@ describe("mongodb-adapter", () => {
 			}),
 		).rejects.toBe(callbackError);
 		expect(commit).not.toHaveBeenCalled();
-		expect(abort).not.toHaveBeenCalled();
+		// `withTransaction` aborts a failed callback itself; no raw command is needed.
+		expect(abort).toHaveBeenCalledOnce();
 		expect(command).not.toHaveBeenCalled();
 		expect(end).toHaveBeenCalledOnce();
 		await client.close();
@@ -610,6 +611,32 @@ describe("mongodb-adapter", () => {
 	});
 });
 
+describe("transactions", () => {
+	/**
+	 * @see https://github.com/better-auth/better-auth/issues/11514
+	 * @see https://www.mongodb.com/docs/manual/core/transactions-in-applications/
+	 */
+	it("retries the callback when MongoDB aborts the transaction with a transient error", async () => {
+		const client = new MongoClient("mongodb://localhost:27017");
+		const adapter = mongodbAdapter(client.db("test"), { client })({});
+		let attempts = 0;
+
+		const result = await adapter.transaction(async () => {
+			attempts += 1;
+			if (attempts === 1) {
+				const writeConflict = new MongoServerError({ errmsg: "WriteConflict" });
+				writeConflict.addErrorLabel(MongoErrorLabel.TransientTransactionError);
+				throw writeConflict;
+			}
+			return "committed";
+		});
+
+		expect(result).toBe("committed");
+		expect(attempts).toBe(2);
+		await client.close();
+	});
+});
+
 describe("uuid support", () => {
 	const uuidRegex =
 		/^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$/i;

packages/mongo-adapter/src/mongodb-adapter.ts

@@ -814,8 +814,6 @@ export const mongodbAdapter = (
 							let commitAttempted = false;
 
 							try {
-								session.startTransaction();
-
 								const adapter = createAdapterFactory({
 									config: {
 										...adapterOptions!.config,
@@ -824,26 +822,22 @@ export const mongodbAdapter = (
 									adapter: createCustomAdapter(db, session),
 								})(lazyOptions!);
 
-								const result = await cb(adapter);
-
-								commitAttempted = true;
-								await session.commitTransaction();
-								return result;
+								// `withTransaction` retries a `TransientTransactionError` (such as a
+								// WriteConflict between concurrent transactions) and an unknown
+								// commit result, so `cb` can run more than once.
+								return await session.withTransaction(async () => {
+									commitAttempted = false;
+									const result = await cb(adapter);
+									// `withTransaction` commits as soon as the callback resolves.
+									commitAttempted = true;
+									return result;
+								});
 							} catch (err) {
-								if (session.inTransaction()) {
-									await session.abortTransaction();
-								} else if (commitAttempted) {
-									// A commit that failed in flight leaves the transaction
-									// open on the server, but the driver no longer reports the
-									// session as being in one, so `abortTransaction()` is not
-									// an option. Send the abort as a command on the same
-									// session so the server does not hold the writes until
-									// `transactionLifetimeLimitSeconds`.
-									//
-									// Only a failed commit can leave that state behind: if the
-									// transaction never started, or failed before the commit,
-									// there is nothing open and sending the command would just
-									// produce a second error to log.
+								// `withTransaction` aborts when the callback fails, but not when the
+								// commit fails in flight: the driver then no longer reports the
+								// session as being in a transaction, while the server keeps it open
+								// until `transactionLifetimeLimitSeconds`. Abort it with a command.
+								if (commitAttempted) {
 									await abortTransactionCommand(config.client, session);
 								}
 								throw err;