多线程ETL时报内存错误,请问下是什么原因引起的呢?(Attempted to read or write prote) 返回

SqlSugar 沟通中
8 672
该叫什么 rookie 发布于2024/5/9
悬赏:0 飞吻

image.png

运行一段时间后,大概2分钟,其实也就开了两个任务


Fatal error. System.AccessViolationException: Attempted to read or write protected memory. This is often an indication that other memory is corrupt.

   at SqlSugar.FastestProvider`1+<>c__DisplayClass48_0[[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].<GetCopyWriteDataTable>b__3(System.Data.DataColumn)

   at System.Linq.Enumerable+WhereEnumerableIterator`1[[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].ToList()

   at SqlSugar.FastestProvider`1[[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].GetCopyWriteDataTable(System.Data.DataTable)

   at SqlSugar.FastestProvider`1+<_BulkCopy>d__39[[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].MoveNext()

   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]](System.__Canon ByRef)

   at SqlSugar.FastestProvider`1[[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]]._BulkCopy(System.String, System.Data.DataTable)

   at SqlSugar.FastestProvider`1+<BulkCopyAsync>d__11[[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].MoveNext()

   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]](System.__Canon ByRef)

   at SqlSugar.FastestProvider`1[[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].BulkCopyAsync(System.String, System.Data.DataTable)

   at SqlSugar.FastestProvider`1[[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].BulkCopyAsync(System.Data.DataTable)

   at Program+<>c__DisplayClass0_0+<<<Main>$>g__BulkTable|3>d.MoveNext()

   at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)

   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[[System.Int32, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[Program+<>c__DisplayClass0_0+<<<Main>$>g__BulkTable|3>d, ConsoleApp1, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null]].MoveNext(System.Threading.Thread)

   at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(System.Runtime.CompilerServices.IAsyncStateMachineBox, Boolean)

   at System.Threading.Tasks.Task.RunContinuations(System.Object)

   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].SetResult(System.__Canon)

   at SqlSugar.QueryableProvider`1+<ToDataTableAsync>d__148[[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].MoveNext()

   at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)

   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].MoveNext(System.Threading.Thread)

   at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(System.Runtime.CompilerServices.IAsyncStateMachineBox, Boolean)

   at System.Threading.Tasks.Task.RunContinuations(System.Object)

   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].SetResult(System.__Canon)

   at SqlSugar.AdoProvider+<GetDataTableAsync>d__192.MoveNext()

   at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)

   at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(System.Runtime.CompilerServices.IAsyncStateMachineBox, Boolean)

   at System.Threading.Tasks.Task.RunContinuations(System.Object)

   at System.Threading.Tasks.Task.ExecuteWithThreadLocal(System.Threading.Tasks.Task ByRef, System.Threading.Thread)

   at System.Threading.ThreadPoolWorkQueue.Dispatch()

   at System.Threading.PortableThreadPool+WorkerThread.WorkerThreadStart()


热忱回答8

  • rookie rookie VIP0
    2024/5/9

    应该是偶发情况,我又运行了几次,不是每次都出现,也不固定几分钟,但目前我只是在做测试,只开了两个任务表,且是测试机数据量少,担心生产环境数据量与表数量都提上来后还有这种情况,想了解下这个现象出现的大概原因是什么?是我使用方法不对,还是有可能的内在问题呢?

    0 回复
  • bulkcopy尽量不要并发,已经饱和插入了

    0 回复
  • 你这样写可能会有会问题

    0 回复
  • 并发只会更慢

    0 回复
  • rookie rookie VIP0
    2024/5/9

    @fate sta:那你这边的建议是怎样进行呢?我的目标就是数据库的迁移ETL,从源库到目标库全量表的BulkCopy ,请指教

    0 回复
  • @rookie: foreach 循环就行了,不要用select了。 不要用拉姆达,拉姆达会破坏异步await

    也不需要task.whenall就普通循环就行了。

    0 回复
  • rookie rookie VIP0
    2024/5/9

    @fate sta:好的,我再试试,有问题再说。

    不过就现在来看的话,上面那个内存错误是看不出来具体可能的原因是吗?看提示感觉是在操作DataTable时出错,内部是否使用缓存或者其它并发情况引起的sqlsugar内在资源冲突呢?

    0 回复
  • @rookie: 大概率数据库处理不过来就报错了

    0 回复