多线程ETL时报内存错误,请问下是什么原因引起的呢?(Attempted to read or write prote) 返回
运行一段时间后,大概2分钟,其实也就开了两个任务
Fatal error. System.AccessViolationException: Attempted to read or write protected memory. This is often an indication that other memory is corrupt.
at SqlSugar.FastestProvider`1+<>c__DisplayClass48_0[[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].<GetCopyWriteDataTable>b__3(System.Data.DataColumn)
at System.Linq.Enumerable+WhereEnumerableIterator`1[[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].ToList()
at SqlSugar.FastestProvider`1[[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].GetCopyWriteDataTable(System.Data.DataTable)
at SqlSugar.FastestProvider`1+<_BulkCopy>d__39[[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].MoveNext()
at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]](System.__Canon ByRef)
at SqlSugar.FastestProvider`1[[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]]._BulkCopy(System.String, System.Data.DataTable)
at SqlSugar.FastestProvider`1+<BulkCopyAsync>d__11[[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].MoveNext()
at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]](System.__Canon ByRef)
at SqlSugar.FastestProvider`1[[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].BulkCopyAsync(System.String, System.Data.DataTable)
at SqlSugar.FastestProvider`1[[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].BulkCopyAsync(System.Data.DataTable)
at Program+<>c__DisplayClass0_0+<<<Main>$>g__BulkTable|3>d.MoveNext()
at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[[System.Int32, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[Program+<>c__DisplayClass0_0+<<<Main>$>g__BulkTable|3>d, ConsoleApp1, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null]].MoveNext(System.Threading.Thread)
at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(System.Runtime.CompilerServices.IAsyncStateMachineBox, Boolean)
at System.Threading.Tasks.Task.RunContinuations(System.Object)
at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].SetResult(System.__Canon)
at SqlSugar.QueryableProvider`1+<ToDataTableAsync>d__148[[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].MoveNext()
at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].MoveNext(System.Threading.Thread)
at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(System.Runtime.CompilerServices.IAsyncStateMachineBox, Boolean)
at System.Threading.Tasks.Task.RunContinuations(System.Object)
at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].SetResult(System.__Canon)
at SqlSugar.AdoProvider+<GetDataTableAsync>d__192.MoveNext()
at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(System.Runtime.CompilerServices.IAsyncStateMachineBox, Boolean)
at System.Threading.Tasks.Task.RunContinuations(System.Object)
at System.Threading.Tasks.Task.ExecuteWithThreadLocal(System.Threading.Tasks.Task ByRef, System.Threading.Thread)
at System.Threading.ThreadPoolWorkQueue.Dispatch()
at System.Threading.PortableThreadPool+WorkerThread.WorkerThreadStart()
热忱回答(8)
-
rookie VIP02024/5/9
应该是偶发情况,我又运行了几次,不是每次都出现,也不固定几分钟,但目前我只是在做测试,只开了两个任务表,且是测试机数据量少,担心生产环境数据量与表数量都提上来后还有这种情况,想了解下这个现象出现的大概原因是什么?是我使用方法不对,还是有可能的内在问题呢?
0 回复 -
fate sta VIP02024/5/9
bulkcopy尽量不要并发,已经饱和插入了
0 回复 -
fate sta VIP02024/5/9
你这样写可能会有会问题
0 回复 -
fate sta VIP02024/5/9
并发只会更慢
0 回复 -
rookie VIP02024/5/9
@fate sta:那你这边的建议是怎样进行呢?我的目标就是数据库的迁移ETL,从源库到目标库全量表的BulkCopy ,请指教
0 回复 -
fate sta VIP02024/5/9
@rookie: foreach 循环就行了,不要用select了。 不要用拉姆达,拉姆达会破坏异步await
也不需要task.whenall就普通循环就行了。
0 回复 -
rookie VIP02024/5/9
@fate sta:好的,我再试试,有问题再说。
不过就现在来看的话,上面那个内存错误是看不出来具体可能的原因是吗?看提示感觉是在操作DataTable时出错,内部是否使用缓存或者其它并发情况引起的sqlsugar内在资源冲突呢?
0 回复 -
fate sta VIP02024/5/9
@rookie: 大概率数据库处理不过来就报错了
0 回复