github-starred/better-auth
2
0
Fork 0
mirror of https://github.com/better-auth/better-auth.git synced 2026-06-19 21:49:17 -05:00
Code Issues 2.5k Packages Projects Releases 112 Wiki Activity

Heap memory error while migrating from supabase #1979

New Issue
Closed
opened 2026-03-13 09:18:42 -05:00 by GiteaMirror · 18 comments
GiteaMirror commented 2026-03-13 09:18:42 -05:00
Owner
Copy Link

Originally created by @Stealthwriter on GitHub (Sep 20, 2025).

Is this suited for github?

  • Yes, this is suited for github

To Reproduce

I ran the migration.ts file to migrate supabase to betterauth, I have around 7 million rows

I got this error

<--- Last few GCs --->

[11408:00000212FA431000] 97415 ms: Scavenge (interleaved) 4045.1 (4123.4) -> 4042.0 (4126.1) MB, pooled: 0 MB, 14.77 / 0.00 ms (average mu = 0.816, current mu = 0.439) allocation failure;
[11408:00000212FA431000] 97663 ms: Scavenge (interleaved) 4047.8 (4126.1) -> 4044.3 (4144.1) MB, pooled: 0 MB, 240.06 / 0.00 ms (average mu = 0.816, current mu = 0.439) allocation failure;

<--- JS stacktrace --->

FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory
----- Native stack trace -----

1: 00007FF66F9E7487 node::SetCppgcReference+16583
2: 00007FF66F94B1B8 v8::base::CPU::num_virtual_address_bits+103032
3: 00007FF6704BC011 v8::Isolate::ReportExternalAllocationLimitReached+65
4: 00007FF6704A8B96 v8::Function::Experimental_IsNopFunction+3302
5: 00007FF6703054E0 v8::internal::StrongRootAllocatorBase::StrongRootAllocatorBase+33904
6: 00007FF670301B6A v8::internal::StrongRootAllocatorBase::StrongRootAllocatorBase+19194
7: 00007FF67031800C v8::Isolate::GetHeapProfiler+7692
8: 00007FF67031888A v8::Isolate::GetHeapProfiler+9866
9: 00007FF670323971 v8::Isolate::GetHeapProfiler+55153
10: 00007FF6703334E2 v8::Isolate::GetHeapProfiler+119522
11: 00007FF6703383D8 v8::Isolate::GetHeapProfiler+139736
12: 00007FF67022F02C v8::Message::GetIsolate+371468
13: 00007FF67022B0E1 v8::Message::GetIsolate+355265
14: 00007FF67022F584 v8::Message::GetIsolate+372836
15: 00007FF67045F43A v8::SharedValueConveyor::SharedValueConveyor+265690
16: 00007FF67045F650 v8::SharedValueConveyor::SharedValueConveyor+266224
17: 00007FF61052D3FA

Current vs. Expected behavior

it should migrate the users and accounts

What version of Better Auth are you using?

1.3

System info

nextjs

Which area(s) are affected? (Select all that apply)

Backend

Auth config (if applicable)

import { betterAuth } from "better-auth"
export const auth = betterAuth({
  emailAndPassword: {  
    enabled: true
  },
});

Additional context

No response

Originally created by @Stealthwriter on GitHub (Sep 20, 2025). ### Is this suited for github? - [x] Yes, this is suited for github ### To Reproduce I ran the migration.ts file to migrate supabase to betterauth, I have around 7 million rows I got this error <--- Last few GCs ---> [11408:00000212FA431000] 97415 ms: Scavenge (interleaved) 4045.1 (4123.4) -> 4042.0 (4126.1) MB, pooled: 0 MB, 14.77 / 0.00 ms (average mu = 0.816, current mu = 0.439) allocation failure; [11408:00000212FA431000] 97663 ms: Scavenge (interleaved) 4047.8 (4126.1) -> 4044.3 (4144.1) MB, pooled: 0 MB, 240.06 / 0.00 ms (average mu = 0.816, current mu = 0.439) allocation failure; <--- JS stacktrace ---> FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory ----- Native stack trace ----- 1: 00007FF66F9E7487 node::SetCppgcReference+16583 2: 00007FF66F94B1B8 v8::base::CPU::num_virtual_address_bits+103032 3: 00007FF6704BC011 v8::Isolate::ReportExternalAllocationLimitReached+65 4: 00007FF6704A8B96 v8::Function::Experimental_IsNopFunction+3302 5: 00007FF6703054E0 v8::internal::StrongRootAllocatorBase::StrongRootAllocatorBase+33904 6: 00007FF670301B6A v8::internal::StrongRootAllocatorBase::StrongRootAllocatorBase+19194 7: 00007FF67031800C v8::Isolate::GetHeapProfiler+7692 8: 00007FF67031888A v8::Isolate::GetHeapProfiler+9866 9: 00007FF670323971 v8::Isolate::GetHeapProfiler+55153 10: 00007FF6703334E2 v8::Isolate::GetHeapProfiler+119522 11: 00007FF6703383D8 v8::Isolate::GetHeapProfiler+139736 12: 00007FF67022F02C v8::Message::GetIsolate+371468 13: 00007FF67022B0E1 v8::Message::GetIsolate+355265 14: 00007FF67022F584 v8::Message::GetIsolate+372836 15: 00007FF67045F43A v8::SharedValueConveyor::SharedValueConveyor+265690 16: 00007FF67045F650 v8::SharedValueConveyor::SharedValueConveyor+266224 17: 00007FF61052D3FA ### Current vs. Expected behavior it should migrate the users and accounts ### What version of Better Auth are you using? 1.3 ### System info ```bash nextjs ``` ### Which area(s) are affected? (Select all that apply) Backend ### Auth config (if applicable) ```typescript import { betterAuth } from "better-auth" export const auth = betterAuth({ emailAndPassword: { enabled: true }, }); ``` ### Additional context _No response_
GiteaMirror added the bug label 2026-03-13 09:18:42 -05:00
GiteaMirror commented 2026-03-13 09:18:44 -05:00
Author
Owner
Copy Link

@ping-maxwell commented on GitHub (Sep 20, 2025):

I think it's most likely because it's trying to pull all users at once.
I'll see if I can make a fix on the migration script

@ping-maxwell commented on GitHub (Sep 20, 2025): I think it's most likely because it's trying to pull all users at once. I'll see if I can make a fix on the migration script
GiteaMirror commented 2026-03-13 09:18:44 -05:00
Author
Owner
Copy Link

@Stealthwriter commented on GitHub (Sep 20, 2025):

I'm also working on the migration script to do it in batches, I'll update
you here once I finish it.

On Sat, 20 Sep 2025, 4:03 PM Maxwell @.***> wrote:

ping-maxwell left a comment (better-auth/better-auth#4784)
https://github.com/better-auth/better-auth/issues/4784#issuecomment-3314930352

I think it's most likely because it's trying to pull all users at once.
I'll see if I can make a fix on the migration script

—
Reply to this email directly, view it on GitHub
https://github.com/better-auth/better-auth/issues/4784#issuecomment-3314930352,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/A6HMVXHBRA62URZLOSJV3OD3TU7BDAVCNFSM6AAAAACHBDSUS6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTGMJUHEZTAMZVGI
.
You are receiving this because you authored the thread.Message ID:
@.***>

@Stealthwriter commented on GitHub (Sep 20, 2025): I'm also working on the migration script to do it in batches, I'll update you here once I finish it. On Sat, 20 Sep 2025, 4:03 PM Maxwell ***@***.***> wrote: > *ping-maxwell* left a comment (better-auth/better-auth#4784) > <https://github.com/better-auth/better-auth/issues/4784#issuecomment-3314930352> > > I think it's most likely because it's trying to pull all users at once. > I'll see if I can make a fix on the migration script > > — > Reply to this email directly, view it on GitHub > <https://github.com/better-auth/better-auth/issues/4784#issuecomment-3314930352>, > or unsubscribe > <https://github.com/notifications/unsubscribe-auth/A6HMVXHBRA62URZLOSJV3OD3TU7BDAVCNFSM6AAAAACHBDSUS6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTGMJUHEZTAMZVGI> > . > You are receiving this because you authored the thread.Message ID: > ***@***.***> >
GiteaMirror commented 2026-03-13 09:18:44 -05:00
Author
Owner
Copy Link

@ping-maxwell commented on GitHub (Sep 20, 2025):

oh I just made one, though haven't tested it.

import { Pool } from "pg";
import { auth } from "@/lib/auth";
import { User as SupabaseUser } from "@supabase/supabase-js";

type User = SupabaseUser & {
  is_super_admin: boolean;
  raw_user_meta_data: {
    avatar_url: string;
  };
  encrypted_password: string;
  email_confirmed_at: string;
  created_at: string;
  updated_at: string;
  is_anonymous: boolean;
  identities: {
    provider: string;
    identity_data: {
      sub: string;
      email: string;
    };
    created_at: string;
    updated_at: string;
  };
};

const migrateFromSupabase = async () => {
  const ctx = await auth.$context;
  const db = ctx.options.database as Pool;

  const BATCH_SIZE = 1000; // Process 1000 users at a time
  let offset = 0;
  let totalProcessed = 0;
  let hasMore = true;

  console.log("Starting migration from Supabase...");

  while (hasMore) {
    try {
      console.log(
        `Processing batch: offset ${offset}, batch size ${BATCH_SIZE}`
      );

      const users = await db
        .query(
          `
          SELECT 
            u.*,
            COALESCE(
              json_agg(
                i.* ORDER BY i.id
              ) FILTER (WHERE i.id IS NOT NULL),
              '[]'::json
            ) as identities
          FROM auth.users u
          LEFT JOIN auth.identities i ON u.id = i.user_id
          GROUP BY u.id
          ORDER BY u.created_at
          LIMIT $1 OFFSET $2
        `,
          [BATCH_SIZE, offset]
        )
        .then((res) => res.rows as User[]);

      if (users.length === 0) {
        hasMore = false;
        break;
      }

      console.log(`Processing ${users.length} users in this batch...`);

      for (const user of users) {
        if (!user.email) {
          continue;
        }

        try {
          await ctx.adapter
            .create({
              model: "user",
              data: {
                id: user.id,
                email: user.email,
                name: user.email,
                role: user.is_super_admin ? "admin" : user.role,
                emailVerified: !!user.email_confirmed_at,
                image: user.raw_user_meta_data.avatar_url,
                createdAt: new Date(user.created_at),
                updatedAt: new Date(user.updated_at),
                isAnonymous: user.is_anonymous,
              },
            })
            .catch(() => {});

          for (const identity of user.identities) {
            const existingAccounts = await ctx.internalAdapter.findAccounts(
              user.id
            );

            if (identity.provider === "email") {
              const hasCredential = existingAccounts.find(
                (account) => account.providerId === "credential"
              );
              if (!hasCredential) {
                await ctx.adapter
                  .create({
                    model: "account",
                    data: {
                      userId: user.id,
                      providerId: "credential",
                      accountId: user.id,
                      password: user.encrypted_password,
                      createdAt: new Date(user.created_at),
                      updatedAt: new Date(user.updated_at),
                    },
                  })
                  .catch(() => {});
              }
            }
            const supportedProviders = Object.keys(
              ctx.options.socialProviders || {}
            );
            if (supportedProviders.includes(identity.provider)) {
              const hasAccount = existingAccounts.find(
                (account) => account.providerId === identity.provider
              );
              if (!hasAccount) {
                await ctx.adapter.create({
                  model: "account",
                  data: {
                    userId: user.id,
                    providerId: identity.provider,
                    accountId: identity.identity_data?.sub,
                    createdAt: new Date(identity.created_at ?? user.created_at),
                    updatedAt: new Date(identity.updated_at ?? user.updated_at),
                  },
                });
              }
            }
          }
        } catch (userError) {
          console.error(`Error processing user ${user.id}:`, userError);
          // Continue with next user instead of failing the entire batch
        }
      }

      totalProcessed += users.length;
      offset += BATCH_SIZE;

      console.log(`Completed batch. Total processed: ${totalProcessed}`);

      // Add a small delay to prevent overwhelming the database
      await new Promise((resolve) => setTimeout(resolve, 100));
    } catch (batchError) {
      console.error(`Error processing batch at offset ${offset}:`, batchError);
      // Move to next batch even if this one failed
      offset += BATCH_SIZE;
    }
  }

  console.log(`Migration completed. Total users processed: ${totalProcessed}`);
};
migrateFromSupabase();
@ping-maxwell commented on GitHub (Sep 20, 2025): oh I just made one, though haven't tested it. ```ts import { Pool } from "pg"; import { auth } from "@/lib/auth"; import { User as SupabaseUser } from "@supabase/supabase-js"; type User = SupabaseUser & { is_super_admin: boolean; raw_user_meta_data: { avatar_url: string; }; encrypted_password: string; email_confirmed_at: string; created_at: string; updated_at: string; is_anonymous: boolean; identities: { provider: string; identity_data: { sub: string; email: string; }; created_at: string; updated_at: string; }; }; const migrateFromSupabase = async () => { const ctx = await auth.$context; const db = ctx.options.database as Pool; const BATCH_SIZE = 1000; // Process 1000 users at a time let offset = 0; let totalProcessed = 0; let hasMore = true; console.log("Starting migration from Supabase..."); while (hasMore) { try { console.log( `Processing batch: offset ${offset}, batch size ${BATCH_SIZE}` ); const users = await db .query( ` SELECT u.*, COALESCE( json_agg( i.* ORDER BY i.id ) FILTER (WHERE i.id IS NOT NULL), '[]'::json ) as identities FROM auth.users u LEFT JOIN auth.identities i ON u.id = i.user_id GROUP BY u.id ORDER BY u.created_at LIMIT $1 OFFSET $2 `, [BATCH_SIZE, offset] ) .then((res) => res.rows as User[]); if (users.length === 0) { hasMore = false; break; } console.log(`Processing ${users.length} users in this batch...`); for (const user of users) { if (!user.email) { continue; } try { await ctx.adapter .create({ model: "user", data: { id: user.id, email: user.email, name: user.email, role: user.is_super_admin ? "admin" : user.role, emailVerified: !!user.email_confirmed_at, image: user.raw_user_meta_data.avatar_url, createdAt: new Date(user.created_at), updatedAt: new Date(user.updated_at), isAnonymous: user.is_anonymous, }, }) .catch(() => {}); for (const identity of user.identities) { const existingAccounts = await ctx.internalAdapter.findAccounts( user.id ); if (identity.provider === "email") { const hasCredential = existingAccounts.find( (account) => account.providerId === "credential" ); if (!hasCredential) { await ctx.adapter .create({ model: "account", data: { userId: user.id, providerId: "credential", accountId: user.id, password: user.encrypted_password, createdAt: new Date(user.created_at), updatedAt: new Date(user.updated_at), }, }) .catch(() => {}); } } const supportedProviders = Object.keys( ctx.options.socialProviders || {} ); if (supportedProviders.includes(identity.provider)) { const hasAccount = existingAccounts.find( (account) => account.providerId === identity.provider ); if (!hasAccount) { await ctx.adapter.create({ model: "account", data: { userId: user.id, providerId: identity.provider, accountId: identity.identity_data?.sub, createdAt: new Date(identity.created_at ?? user.created_at), updatedAt: new Date(identity.updated_at ?? user.updated_at), }, }); } } } } catch (userError) { console.error(`Error processing user ${user.id}:`, userError); // Continue with next user instead of failing the entire batch } } totalProcessed += users.length; offset += BATCH_SIZE; console.log(`Completed batch. Total processed: ${totalProcessed}`); // Add a small delay to prevent overwhelming the database await new Promise((resolve) => setTimeout(resolve, 100)); } catch (batchError) { console.error(`Error processing batch at offset ${offset}:`, batchError); // Move to next batch even if this one failed offset += BATCH_SIZE; } } console.log(`Migration completed. Total users processed: ${totalProcessed}`); }; migrateFromSupabase(); ```
GiteaMirror commented 2026-03-13 09:18:45 -05:00
Author
Owner
Copy Link

@Stealthwriter commented on GitHub (Sep 20, 2025):

I get this error when I'm running it, also its very slow: 2025-09-20T12:20:02.233Z WARN [Better Auth]: [Kysely Adapter] - You are trying to create a record with an id. This is not allowed as we handle id generation for you, unless you pass in the forceAllowId parameter. The id will be ignored.

@Stealthwriter commented on GitHub (Sep 20, 2025): I get this error when I'm running it, also its very slow: 2025-09-20T12:20:02.233Z WARN [Better Auth]: [Kysely Adapter] - You are trying to create a record with an id. This is not allowed as we handle id generation for you, unless you pass in the `forceAllowId` parameter. The id will be ignored.
GiteaMirror commented 2026-03-13 09:18:45 -05:00
Author
Owner
Copy Link

@Stealthwriter commented on GitHub (Sep 20, 2025):

It's very slow, 1000 users take like 10 minutes

@Stealthwriter commented on GitHub (Sep 20, 2025): It's very slow, 1000 users take like 10 minutes
GiteaMirror commented 2026-03-13 09:18:46 -05:00
Author
Owner
Copy Link

@Kinfe123 commented on GitHub (Sep 20, 2025):

while @ping-maxwell is valid response. but you can do a node option export to manage the heap size beyond the default to handle all js object + pending promise created - NODE_OPTIONS="--max-old-space-size=8192" node migration.js with 8192 for 8GB or 16384 for 16G . but this is not the right solution but if you machine has a size for RAM for that, it is duable but batching with bounded stream might solve (using at db level which using pg like pg cursor or pg-map). but will check on this by reproducing it

@Kinfe123 commented on GitHub (Sep 20, 2025): while @ping-maxwell is valid response. but you can do a node option export to manage the heap size beyond the default to handle all js object + pending promise created -` NODE_OPTIONS="--max-old-space-size=8192" node migration.js` with 8192 for 8GB or 16384 for 16G . but this is not the right solution but if you machine has a size for RAM for that, it is duable but batching with bounded stream might solve (using at db level which using pg like pg cursor or pg-map). but will check on this by reproducing it
GiteaMirror commented 2026-03-13 09:18:46 -05:00
Author
Owner
Copy Link

@Stealthwriter commented on GitHub (Sep 20, 2025):

I tried with 16gb ram, it also crashed

7 million rows would need around 100gb ram

@Stealthwriter commented on GitHub (Sep 20, 2025): I tried with 16gb ram, it also crashed 7 million rows would need around 100gb ram
GiteaMirror commented 2026-03-13 09:18:47 -05:00
Author
Owner
Copy Link

@Kinfe123 commented on GitHub (Sep 20, 2025):

how can you help me reproduce this ? i want to reproduce it myself and try to moving of some js level operation to db level. which kind offload some mem consumption leading to OOM

@Kinfe123 commented on GitHub (Sep 20, 2025): how can you help me reproduce this ? i want to reproduce it myself and try to moving of some js level operation to db level. which kind offload some mem consumption leading to OOM
GiteaMirror commented 2026-03-13 09:18:47 -05:00
Author
Owner
Copy Link

@ping-maxwell commented on GitHub (Sep 20, 2025):

It's very slow, 1000 users take like 10 minutes

Yeah maybe up the batch size

@ping-maxwell commented on GitHub (Sep 20, 2025): > It's very slow, 1000 users take like 10 minutes Yeah maybe up the batch size
GiteaMirror commented 2026-03-13 09:18:48 -05:00
Author
Owner
Copy Link

@ping-maxwell commented on GitHub (Sep 20, 2025):

Would 10k per batch be enough without crashing?

@ping-maxwell commented on GitHub (Sep 20, 2025): Would 10k per batch be enough without crashing?
GiteaMirror commented 2026-03-13 09:18:48 -05:00
Author
Owner
Copy Link

@ping-maxwell commented on GitHub (Sep 20, 2025):

btw this will fix the error you're encountering plus makes it 10k per batch:

import { Pool } from "pg";
import { auth } from "@/lib/auth";
import { User as SupabaseUser } from "@supabase/supabase-js";

type User = SupabaseUser & {
  is_super_admin: boolean;
  raw_user_meta_data: {
    avatar_url: string;
  };
  encrypted_password: string;
  email_confirmed_at: string;
  created_at: string;
  updated_at: string;
  is_anonymous: boolean;
  identities: {
    provider: string;
    identity_data: {
      sub: string;
      email: string;
    };
    created_at: string;
    updated_at: string;
  };
};

const migrateFromSupabase = async () => {
  const ctx = await auth.$context;
  const db = ctx.options.database as Pool;

  const BATCH_SIZE = 10000; // Process 1000 users at a time
  let offset = 0;
  let totalProcessed = 0;
  let hasMore = true;

  console.log("Starting migration from Supabase...");

  while (hasMore) {
    try {
      console.log(
        `Processing batch: offset ${offset}, batch size ${BATCH_SIZE}`
      );

      const users = await db
        .query(
          `
          SELECT 
            u.*,
            COALESCE(
              json_agg(
                i.* ORDER BY i.id
              ) FILTER (WHERE i.id IS NOT NULL),
              '[]'::json
            ) as identities
          FROM auth.users u
          LEFT JOIN auth.identities i ON u.id = i.user_id
          GROUP BY u.id
          ORDER BY u.created_at
          LIMIT $1 OFFSET $2
        `,
          [BATCH_SIZE, offset]
        )
        .then((res) => res.rows as User[]);

      if (users.length === 0) {
        hasMore = false;
        break;
      }

      console.log(`Processing ${users.length} users in this batch...`);

      for (const user of users) {
        if (!user.email) {
          continue;
        }

        try {
          await ctx.adapter
            .create({
              model: "user",
              data: {
                id: user.id,
                email: user.email,
                name: user.email,
                role: user.is_super_admin ? "admin" : user.role,
                emailVerified: !!user.email_confirmed_at,
                image: user.raw_user_meta_data.avatar_url,
                createdAt: new Date(user.created_at),
                updatedAt: new Date(user.updated_at),
                isAnonymous: user.is_anonymous,
              },
              forceAllowId: true
            })
            .catch(() => {});

          for (const identity of user.identities) {
            const existingAccounts = await ctx.internalAdapter.findAccounts(
              user.id
            );

            if (identity.provider === "email") {
              const hasCredential = existingAccounts.find(
                (account) => account.providerId === "credential"
              );
              if (!hasCredential) {
                await ctx.adapter
                  .create({
                    model: "account",
                    data: {
                      userId: user.id,
                      providerId: "credential",
                      accountId: user.id,
                      password: user.encrypted_password,
                      createdAt: new Date(user.created_at),
                      updatedAt: new Date(user.updated_at),
                    },
                  })
                  .catch(() => {});
              }
            }
            const supportedProviders = Object.keys(
              ctx.options.socialProviders || {}
            );
            if (supportedProviders.includes(identity.provider)) {
              const hasAccount = existingAccounts.find(
                (account) => account.providerId === identity.provider
              );
              if (!hasAccount) {
                await ctx.adapter.create({
                  model: "account",
                  data: {
                    userId: user.id,
                    providerId: identity.provider,
                    accountId: identity.identity_data?.sub,
                    createdAt: new Date(identity.created_at ?? user.created_at),
                    updatedAt: new Date(identity.updated_at ?? user.updated_at),
                  },
                });
              }
            }
          }
        } catch (userError) {
          console.error(`Error processing user ${user.id}:`, userError);
          // Continue with next user instead of failing the entire batch
        }
      }

      totalProcessed += users.length;
      offset += BATCH_SIZE;

      console.log(`Completed batch. Total processed: ${totalProcessed}`);

      // Add a small delay to prevent overwhelming the database
      await new Promise((resolve) => setTimeout(resolve, 100));
    } catch (batchError) {
      console.error(`Error processing batch at offset ${offset}:`, batchError);
      // Move to next batch even if this one failed
      offset += BATCH_SIZE;
    }
  }

  console.log(`Migration completed. Total users processed: ${totalProcessed}`);
};
migrateFromSupabase();
@ping-maxwell commented on GitHub (Sep 20, 2025): btw this will fix the error you're encountering plus makes it 10k per batch: ```ts import { Pool } from "pg"; import { auth } from "@/lib/auth"; import { User as SupabaseUser } from "@supabase/supabase-js"; type User = SupabaseUser & { is_super_admin: boolean; raw_user_meta_data: { avatar_url: string; }; encrypted_password: string; email_confirmed_at: string; created_at: string; updated_at: string; is_anonymous: boolean; identities: { provider: string; identity_data: { sub: string; email: string; }; created_at: string; updated_at: string; }; }; const migrateFromSupabase = async () => { const ctx = await auth.$context; const db = ctx.options.database as Pool; const BATCH_SIZE = 10000; // Process 1000 users at a time let offset = 0; let totalProcessed = 0; let hasMore = true; console.log("Starting migration from Supabase..."); while (hasMore) { try { console.log( `Processing batch: offset ${offset}, batch size ${BATCH_SIZE}` ); const users = await db .query( ` SELECT u.*, COALESCE( json_agg( i.* ORDER BY i.id ) FILTER (WHERE i.id IS NOT NULL), '[]'::json ) as identities FROM auth.users u LEFT JOIN auth.identities i ON u.id = i.user_id GROUP BY u.id ORDER BY u.created_at LIMIT $1 OFFSET $2 `, [BATCH_SIZE, offset] ) .then((res) => res.rows as User[]); if (users.length === 0) { hasMore = false; break; } console.log(`Processing ${users.length} users in this batch...`); for (const user of users) { if (!user.email) { continue; } try { await ctx.adapter .create({ model: "user", data: { id: user.id, email: user.email, name: user.email, role: user.is_super_admin ? "admin" : user.role, emailVerified: !!user.email_confirmed_at, image: user.raw_user_meta_data.avatar_url, createdAt: new Date(user.created_at), updatedAt: new Date(user.updated_at), isAnonymous: user.is_anonymous, }, forceAllowId: true }) .catch(() => {}); for (const identity of user.identities) { const existingAccounts = await ctx.internalAdapter.findAccounts( user.id ); if (identity.provider === "email") { const hasCredential = existingAccounts.find( (account) => account.providerId === "credential" ); if (!hasCredential) { await ctx.adapter .create({ model: "account", data: { userId: user.id, providerId: "credential", accountId: user.id, password: user.encrypted_password, createdAt: new Date(user.created_at), updatedAt: new Date(user.updated_at), }, }) .catch(() => {}); } } const supportedProviders = Object.keys( ctx.options.socialProviders || {} ); if (supportedProviders.includes(identity.provider)) { const hasAccount = existingAccounts.find( (account) => account.providerId === identity.provider ); if (!hasAccount) { await ctx.adapter.create({ model: "account", data: { userId: user.id, providerId: identity.provider, accountId: identity.identity_data?.sub, createdAt: new Date(identity.created_at ?? user.created_at), updatedAt: new Date(identity.updated_at ?? user.updated_at), }, }); } } } } catch (userError) { console.error(`Error processing user ${user.id}:`, userError); // Continue with next user instead of failing the entire batch } } totalProcessed += users.length; offset += BATCH_SIZE; console.log(`Completed batch. Total processed: ${totalProcessed}`); // Add a small delay to prevent overwhelming the database await new Promise((resolve) => setTimeout(resolve, 100)); } catch (batchError) { console.error(`Error processing batch at offset ${offset}:`, batchError); // Move to next batch even if this one failed offset += BATCH_SIZE; } } console.log(`Migration completed. Total users processed: ${totalProcessed}`); }; migrateFromSupabase(); ```
GiteaMirror commented 2026-03-13 09:18:49 -05:00
Author
Owner
Copy Link

@Stealthwriter commented on GitHub (Sep 20, 2025):

I ran it, there is a new error:

Processing batch: offset 0, batch size 10000
Error processing batch at offset 0: error: could not write to file "base/pgsql_tmp/pgsql_tmp226515.5": No space left on device
at C:\Users\maher\OneDrive\Desktop\nextjs-app-new\node_modules.pnpm\pg-pool@3.10.1_pg@8.16.3\node_modules\pg-pool\index.js:45:11
at process.processTicksAndRejections (node:internal/process/task_queues:105:5)
at async migrateFromSupabase (C:\Users\maher\OneDrive\Desktop\nextjs-app-new\src\lib\migration.ts:43:21) {
length: 146,
severity: 'ERROR',
code: '53100',
detail: undefined,
hint: undefined,
position: undefined,
internalPosition: undefined,
internalQuery: undefined,
where: undefined,
schema: undefined,
table: undefined,
column: undefined,
dataType: undefined,
constraint: undefined,
file: 'buffile.c',
line: '543',
routine: 'BufFileDumpBuffer'
}

@Stealthwriter commented on GitHub (Sep 20, 2025): I ran it, there is a new error: Processing batch: offset 0, batch size 10000 Error processing batch at offset 0: error: could not write to file "base/pgsql_tmp/pgsql_tmp226515.5": No space left on device at C:\Users\maher\OneDrive\Desktop\nextjs-app-new\node_modules\.pnpm\pg-pool@3.10.1_pg@8.16.3\node_modules\pg-pool\index.js:45:11 at process.processTicksAndRejections (node:internal/process/task_queues:105:5) at async migrateFromSupabase (C:\Users\maher\OneDrive\Desktop\nextjs-app-new\src\lib\migration.ts:43:21) { length: 146, severity: 'ERROR', code: '53100', detail: undefined, hint: undefined, position: undefined, internalPosition: undefined, internalQuery: undefined, where: undefined, schema: undefined, table: undefined, column: undefined, dataType: undefined, constraint: undefined, file: 'buffile.c', line: '543', routine: 'BufFileDumpBuffer' }
GiteaMirror commented 2026-03-13 09:18:49 -05:00
Author
Owner
Copy Link

@Stealthwriter commented on GitHub (Sep 20, 2025):

There is no faster solution? maybe running an sql command directly inside supabase sql editor?

@Stealthwriter commented on GitHub (Sep 20, 2025): There is no faster solution? maybe running an sql command directly inside supabase sql editor?
GiteaMirror commented 2026-03-13 09:18:50 -05:00
Author
Owner
Copy Link

@Kinfe123 commented on GitHub (Sep 20, 2025):

That's what I mentioned above about using some pg level logic inside supabase query like using pg-cursor and other will do solve by doing an offload of memory from heap

@Kinfe123 commented on GitHub (Sep 20, 2025): That's what I mentioned above about using some pg level logic inside supabase query like using pg-cursor and other will do solve by doing an offload of memory from heap
GiteaMirror commented 2026-03-13 09:18:50 -05:00
Author
Owner
Copy Link

@Stealthwriter commented on GitHub (Sep 20, 2025):

I have no experience in SQL, but I will try to do it

@Stealthwriter commented on GitHub (Sep 20, 2025): I have no experience in SQL, but I will try to do it
GiteaMirror commented 2026-03-13 09:18:51 -05:00
Author
Owner
Copy Link

@Kinfe123 commented on GitHub (Sep 20, 2025):

We will try to do it on the official way or adding a callout somewhere for the optimized query.

@Kinfe123 commented on GitHub (Sep 20, 2025): We will try to do it on the official way or adding a callout somewhere for the optimized query.
GiteaMirror commented 2026-03-13 09:18:52 -05:00
Author
Owner
Copy Link

@Stealthwriter commented on GitHub (Sep 20, 2025):

Thank you,I appreciate your hard work!

@Stealthwriter commented on GitHub (Sep 20, 2025): Thank you,I appreciate your hard work!
GiteaMirror commented 2026-03-13 09:18:53 -05:00
Author
Owner
Copy Link

@dosubot[bot] commented on GitHub (Dec 20, 2025):

Hi, @Stealthwriter. I'm Dosu, and I'm helping the better-auth team manage their backlog and am marking this issue as stale.

Issue Summary:

  • You reported a JavaScript heap out of memory error when migrating 7 million rows from Supabase to BetterAuth due to loading all users at once.
  • A batching approach was suggested and implemented with scripts processing 1,000 to 10,000 users per batch, but you then faced "No space left on device" errors and slow performance.
  • Recommendations included increasing Node's heap size and shifting memory load to the database using tools like pg-cursor.
  • The discussion is ongoing with intentions to optimize migration through SQL-level solutions or official tooling enhancements.

Next Steps:

  • Please let me know if this issue is still relevant with the latest version of better-auth by commenting here to keep the discussion open.
  • Otherwise, this issue will be automatically closed in 7 days.

Thanks for your understanding and contribution!

@dosubot[bot] commented on GitHub (Dec 20, 2025): Hi, @Stealthwriter. I'm [Dosu](https://dosu.dev), and I'm helping the better-auth team manage their backlog and am marking this issue as stale. **Issue Summary:** - You reported a JavaScript heap out of memory error when migrating 7 million rows from Supabase to BetterAuth due to loading all users at once. - A batching approach was suggested and implemented with scripts processing 1,000 to 10,000 users per batch, but you then faced "No space left on device" errors and slow performance. - Recommendations included increasing Node's heap size and shifting memory load to the database using tools like pg-cursor. - The discussion is ongoing with intentions to optimize migration through SQL-level solutions or official tooling enhancements. **Next Steps:** - Please let me know if this issue is still relevant with the latest version of better-auth by commenting here to keep the discussion open. - Otherwise, this issue will be automatically closed in 7 days. Thanks for your understanding and contribution!
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
adapter
astro
awaiting external contributor
blocked
breaking
breaking change
bug
c-devops
core
credentials
database
dependencies
devops
devtools
docs
documentation
duplicate
elysia
enhancement
enterprise
expo
express
fastify
good first issue
help wanted
hono
identity
infra
integration
invalid
javascript
locked
maintenance
need-more-information
needs: info
needs: repro
nextjs
nuxt
oauth
organization
P0
payments
perf
platform
plugin
pull-request
Mirrored from GitHub Pull Request
 question
ready
regression
remix
security
social-provider
solid
stale
svelte
tanstack-start
tracking
version-bump
vue
wontfix
No Label bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/better-auth#1979
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?