Řešil jsem dnes s kolegou Jakubem Čermákem zajímavou situaci. Komprimovanou podobu situace zaznamenávám jako inspiraci pro podobné situace ladění.
Symptom
Na jednom z našich produkčních webových serverů dnes v časných ranních hodinách vyskočila zátěž CPU na 100%. Rychlý pohled do TaskManageru ukázal, že viníkem byl aplikační pool s redakčním systémem UmbracoCMS, který používáme pro jeden z frontendů jako content-databázi a administrační UI (UmbracoCMS zde tedy nerenderuje front-end, ale samostatná ASP.NET front-endová aplikace si přes API říká Umbraco-webu pouze pro data, z kterých výstup sestavuje – různé HTML segmenty, resources atp. editované v CMS).
Stejná situace již v minulosti dvakrát nastala (rozestup v řádu týdnů), vždy pomohlo restartování aplikačního poolu Umbraco. Tentokrát se nám podařilo před restartem poolu sejmout memory-dump postiženého procesu.
Analysis
Na každý získaný memory-dump dnes již automaticky jako první úkon pouštím DebugDiag. Získám tím relativně rychle základní přehled o situaci, např.
- kolik je zhruba čeho v paměti (kolik zabírá managed heap, jaké typy v něm zabírají nejvíce paměti, …)
- kolik je threadů a co dělají (call-stacky, waity/locky, …)
- jestli zrovna běží úkolid garbage-collectorem,
- atp.
Výstup z DebugDiag ukázal, že call-stack několika desítek aktivních threadů se shoduje:
mscorlib_ni!System.Collections.Generic.Dictionary`2[[System.__Canon, mscorlib],[System.Boolean, mscorlib]].Insert(System.__Canon, Boolean, Boolean)+1f0 Umbraco.Core.Models.EntityBase.TracksChangesEntityBase.OnPropertyChanged(System.Reflection.PropertyInfo)+3e Umbraco.Core.Models.EntityBase.TracksChangesEntityBase.SetPropertyValueAndDetectChanges[[System.__Canon, mscorlib]](System.Func`2<System.__Canon,System.__Canon>, System.__Canon, System.Reflection.PropertyInfo)+5c Umbraco.Core.Models.ContentType.set_AllowedTemplates(System.Collections.Generic.IEnumerable`1<Umbraco.Core.Models.ITemplate>)+80 clr!CallDescrWorkerInternal+83 clr!CallDescrWorkerWithHandler+4a clr!CallDescrWorkerReflectionWrapper+1a clr!RuntimeMethodHandle::InvokeMethod+46a [[DebuggerU2MCatchHandlerFrame]] [[HelperMethodFrame_PROTECTOBJ] (System.RuntimeMethodHandle.InvokeMethod)] System.RuntimeMethodHandle.InvokeMethod(System.Object, System.Object[], System.Signature, Boolean) mscorlib_ni!System.Reflection.RuntimeMethodInfo.UnsafeInvokeInternal(System.Object, System.Object[], System.Object[])+4c mscorlib_ni!System.Reflection.RuntimeMethodInfo.Invoke(System.Object, System.Reflection.BindingFlags, System.Reflection.Binder, System.Object[], System.Globalization.CultureInfo)+d3 mscorlib_ni!System.Reflection.RuntimePropertyInfo.SetValue(System.Object, System.Object, System.Object[])+2c Umbraco.Core.Models.DeepCloneHelper.DeepCloneRefProperties(Umbraco.Core.Models.IDeepCloneable, Umbraco.Core.Models.IDeepCloneable)+61d Umbraco.Core.Models.ContentTypeCompositionBase.DeepClone()+3d Umbraco.Core.Models.ContentTypeCompositionBase.<DeepClone>b__2f(Umbraco.Core.Models.IContentTypeComposition)+14 System_Core_ni!System.Linq.Enumerable+WhereSelectListIterator`2[[System.__Canon, mscorlib],[System.__Canon, mscorlib]].MoveNext()+f4 mscorlib_ni!System.Collections.Generic.List`1[[System.__Canon, mscorlib]]..ctor(System.Collections.Generic.IEnumerable`1<System.__Canon>)+218 System_Core_ni!System.Linq.Enumerable.ToList[[System.__Canon, mscorlib]](System.Collections.Generic.IEnumerable`1<System.__Canon>)+50 Umbraco.Core.Models.ContentTypeCompositionBase.DeepClone()+1bb Umbraco.Core.Persistence.Caching.RuntimeCacheProvider.GetById(System.Type, System.Guid)+ea Umbraco.Core.Persistence.Repositories.RepositoryBase`2[[System.Int32, mscorlib],[System.__Canon, mscorlib]].TryGetFromCache(Int32)+bb Umbraco.Core.Persistence.Repositories.RepositoryBase`2[[System.Int32, mscorlib],[System.__Canon, mscorlib]].Get(Int32)+3c Umbraco.Core.Persistence.Repositories.ContentRepository.CreateContentFromDto(Umbraco.Core.Models.Rdbms.DocumentDto, System.Guid)+5d Umbraco.Core.Persistence.Repositories.ContentRepository.PerformGet(Int32)+2e0 Umbraco.Core.Persistence.Repositories.RepositoryBase`2[[System.Int32, mscorlib],[System.__Canon, mscorlib]].Get(Int32)+7b Umbraco.Core.Services.ContentService.GetById(Int32)+69 Havit.XyzPortal.Services.ContentManagement.EpoUrlProvider.GetUrl(Umbraco.Core.ApplicationContext, Int32, System.Uri, Umbraco.Web.Routing.UrlProviderMode)+108 Umbraco.Web.Routing.UrlProvider+<>c__DisplayClass3.<GetUrl>b__0(Umbraco.Web.Routing.IUrlProvider)+32 System_Core_ni!System.Linq.Enumerable+WhereSelectArrayIterator`2[[System.__Canon, mscorlib],[System.__Canon, mscorlib]].MoveNext()+52 System_Core_ni!System.Linq.Enumerable.FirstOrDefault[[System.__Canon, mscorlib]](System.Collections.Generic.IEnumerable`1<System.__Canon>, System.Func`2<System.__Canon,Boolean>)+d7 Umbraco.Web.Routing.UrlProvider.GetUrl(Int32, System.Uri, Umbraco.Web.Routing.UrlProviderMode)+ec Umbraco.Web.Templates.TemplateUtilities.ParseInternalLinks(System.String)+170 Havit.XyzPortal.UmbracoCMS.UmContentApiController.TraverseResourceDictionary(System.Collections.Generic.List`1<Havit.XyzPortal.Services.Resources.DataObjects.ResourceDictionaryDTO>, Umbraco.Core.Models.IContent, Boolean)+32e Havit.XyzPortal.UmbracoCMS.UmContentApiController.TraverseResourceDictionary(System.Collections.Generic.List`1<Havit.XyzPortal.Services.Resources.DataObjects.ResourceDictionaryDTO>, Umbraco.Core.Models.IContent, Boolean)+dd Havit.XyzPortal.UmbracoCMS.UmContentApiController.TraverseResourceDictionary(System.Collections.Generic.List`1<Havit.XyzPortal.Services.Resources.DataObjects.ResourceDictionaryDTO>, Umbraco.Core.Models.IContent, Boolean)+dd Havit.XyzPortal.UmbracoCMS.UmContentApiController.Resources(System.Nullable`1<Boolean>)+113 Havit.XyzPortal.UmbracoCMS.UmContentApiController.Resources()+21 DynamicClass.lambda_method(System.Runtime.CompilerServices.Closure, System.Object, System.Object[])+4f System.Web.Http.Controllers.ReflectedHttpActionDescriptor+ActionExecutor+<>c__DisplayClass13.<GetExecutor>b__c(System.Object, System.Object[])+15 System.Web.Http.Controllers.ReflectedHttpActionDescriptor+ActionExecutor.Execute(System.Object, System.Object[])+f System.Threading.Tasks.TaskHelpers.RunSynchronously[[System.__Canon, mscorlib]](System.Func`1<System.Threading.Tasks.Task`1<System.__Canon>>, System.Threading.CancellationToken)+78 System.Web.Http.Controllers.ApiControllerActionInvoker+<>c__DisplayClass3.<InvokeActionAsync>b__0()+29 System.Threading.Tasks.TaskHelpers.RunSynchronously[[System.__Canon, mscorlib]](System.Func`1<System.Threading.Tasks.Task`1<System.__Canon>>, System.Threading.CancellationToken)+78 System.Web.Http.Controllers.ApiControllerActionInvoker.InvokeActionAsync(System.Web.Http.Controllers.HttpActionContext, System.Threading.CancellationToken)+98 System.Web.Http.ApiController+<>c__DisplayClass3.<ExecuteAsync>b__2()+33 System.Web.Http.ApiController+<>c__DisplayClass3.<ExecuteAsync>b__1()+c5 System.Threading.Tasks.TaskHelpersExtensions+<>c__DisplayClass32`1[[System.__Canon, mscorlib]].<Then>b__31(System.Threading.Tasks.Task)+f System.Threading.Tasks.TaskHelpersExtensions.ThenImpl[[System.__Canon, mscorlib],[System.__Canon, mscorlib]](System.__Canon, System.Func`2<System.__Canon,System.Threading.Tasks.Task`1<System.__Canon>>, System.Threading.CancellationToken, Boolean)+114 System.Web.Http.ApiController+<>c__DisplayClass3.<ExecuteAsync>b__0()+74 System.Web.Http.ApiController.ExecuteAsync(System.Web.Http.Controllers.HttpControllerContext, System.Threading.CancellationToken)+1cf System.Web.Http.Dispatcher.HttpControllerDispatcher.SendAsync(System.Net.Http.HttpRequestMessage, System.Threading.CancellationToken)+76 System_Net_Http_ni!System.Net.Http.HttpMessageInvoker.SendAsync(System.Net.Http.HttpRequestMessage, System.Threading.CancellationToken)+4f System.Web.Http.Dispatcher.HttpRoutingDispatcher.SendAsync(System.Net.Http.HttpRequestMessage, System.Threading.CancellationToken)+175 System.Web.Http.HttpServer.SendAsync(System.Net.Http.HttpRequestMessage, System.Threading.CancellationToken)+ff System_Net_Http_ni!System.Net.Http.HttpMessageInvoker.SendAsync(System.Net.Http.HttpRequestMessage, System.Threading.CancellationToken)+4f System.Web.Http.WebHost.HttpControllerHandler.BeginProcessRequest(System.Web.HttpContextBase, System.AsyncCallback, System.Object)+11f System_Web_ni!System.Web.HttpApplication+CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()+1e2 System_Web_ni!System.Web.HttpApplication.ExecuteStep(IExecutionStep, Boolean ByRef)+120 System_Web_ni!System.Web.HttpApplication+PipelineStepManager.ResumeSteps(System.Exception)+92a System_Web_ni!System.Web.HttpApplication.BeginProcessRequestNotification(System.Web.HttpContext, System.AsyncCallback)+83 System_Web_ni!System.Web.HttpRuntime.ProcessRequestNotificationPrivate(System.Web.Hosting.IIS7WorkerRequest, System.Web.HttpContext)+28e System_Web_ni!System.Web.Hosting.PipelineRuntime.ProcessRequestNotificationHelper(IntPtr, IntPtr, IntPtr, Int32)+5c1 System_Web_ni!System.Web.Hosting.PipelineRuntime.ProcessRequestNotification(IntPtr, IntPtr, IntPtr, Int32)+22 System_Web_ni!DomainNeutralILStubClass.IL_STUB_ReversePInvoke(Int64, Int64, Int64, Int32)+51 clr!UMThunkStub+6e webengine4!W3_MGD_HANDLER::ProcessNotification+78 webengine4!W3_MGD_HANDLER::DoWork+34f webengine4!RequestDoWork+34e webengine4!CMgdEngHttpModule::OnExecuteRequestHandler+21 iiscore!NOTIFICATION_CONTEXT::RequestDoWork+233 iiscore!NOTIFICATION_CONTEXT::CallModulesInternal+174 iiscore!NOTIFICATION_CONTEXT::CallModules+25 iiscore!W3_CONTEXT::DoWork+34d iiscore!W3_CONTEXT::IndicateCompletion+8c webengine4!MgdIndicateCompletion+60 System_Web_ni!DomainNeutralILStubClass.IL_STUB_PInvoke(IntPtr, System.Web.RequestNotificationStatus ByRef)+7b [[InlinedCallFrame] (System.Web.Hosting.UnsafeIISMethods.MgdIndicateCompletion)] System.Web.Hosting.UnsafeIISMethods.MgdIndicateCompletion(IntPtr, System.Web.RequestNotificationStatusByRef) System_Web_ni!System.Web.Hosting.PipelineRuntime.ProcessRequestNotificationHelper(IntPtr, IntPtr, IntPtr, Int32)+7af System_Web_ni!System.Web.Hosting.PipelineRuntime.ProcessRequestNotification(IntPtr, IntPtr, IntPtr, Int32)+22 System_Web_ni!DomainNeutralILStubClass.IL_STUB_ReversePInvoke(Int64, Int64, Int64, Int32)+51 clr!UM2MThunk_WrapperHelper+43 clr!UM2MThunk_Wrapper+5a clr!Thread::DoADCallBack+139 [[ContextTransitionFrame]] clr!UM2MDoADCallBack+91 clr!UMThunkStub+26d webengine4!W3_MGD_HANDLER::ProcessNotification+78 webengine4!ProcessNotificationCallback+42 clr!UnManagedPerAppDomainTPCount::DispatchWorkItem+122 clr!ThreadpoolMgr::ExecuteWorkRequest+46 clr!ThreadpoolMgr::WorkerThreadStart+f4 clr!Thread::intermediateThreadProc+7d kernel32!BaseThreadInitThunk+d ntdll!RtlUserThreadStart+1d
První procházení zdrojových kódů dotčených metod (nejprve „just my code“, dále i Umbraco a nakonec i .NET) neukázalo na očividného viníka. Na call-stacku se děje zhruba toto:
- Přichází web-request na službu poskytující front-endu data o resources. Náš REST-API UmContentApiController vrací front-endu strukturu všech resources, které UmbracoCMS v má (UmContentApiController.Resources()).
- Během sestavování výstupu pro jednotlivé resources vyhodnocuje případné symbolické odkazy v nich a překládá je na reálné URL adresy jednotlivých cílových objektů – TemplateUtilities.ParseInternalLinks()
- Pro každý nalezený symbolický odkaz je známo ID cílového objektu a přes UrlProvider.GetUrl() se Umbraco snaží vyhodnotit cílové URL. Dělá to voláním metody GetUrl() na každém zaregistrovaném UrlProvideru, jedním z nichž je náš EpoUrlProvider.
- V EpoUrlProvider.GetUrl() se hned na začátku získává z ID entita reprezentující cílový content (jehož URL se má sestavit), volá se tedy ContentService.GetById(), které přes ContentRepository (transient) dojde až na RuntimeCacheProvider.GetById(), který když v cache objekt najde, chce vrátit jeho DeepClone().
- DeepClone() dělá kopii objektu a to tak, že nejprve vytvoří shallow-copy a následně klonuje jednotlivé referencované objekty. V jednom z nich (ContentType) se tak dostane až na setter property AllowedTemplates. V něm (a všech obdobných properties na potomcích třídy TracksChangesEntityBase) je setter ošetřen tak, že při ukládání nové hodnoty porovná s hodnotou původní, a pokud je honota změněna, uloží do slovníku informaci, že property daného jména je Changed/Dirty (vše pomocí metody SetPropertyValueAndDetectChanges() a OnPropertyChanged():
// class ContentType : TracksChangesEntityBase public IEnumerable<ITemplate> AllowedTemplates { get { return this._allowedTemplates; } set { base.SetPropertyValueAndDetectChanges<IEnumerable<ITemplate>>(delegate (IEnumerable<ITemplate> o) { this._allowedTemplates = value; return this._allowedTemplates; }, this._allowedTemplates, AllowedTemplatesSelector); } } // TracksChangesEntityBase internal bool SetPropertyValueAndDetectChanges<T>(Func<T, T> setValue, T value, PropertyInfo propertySelector) { T local = value; T local2 = setValue(value); if (!object.Equals(local, local2)) { this.OnPropertyChanged(propertySelector); return true; } return false; } protected virtual void OnPropertyChanged(PropertyInfo propertyInfo) { this._propertyChangedInfo.set_Item(propertyInfo.get_Name(), true); if (this.PropertyChanged != null) { this.PropertyChanged(this, new PropertyChangedEventArgs(propertyInfo.get_Name())); } }
Onen Dictionary<>.Insert(), který je na vrcholu call-stacků je tedy ve skutečnosti in-linovaná podoba slovníkového indexeru _propertyChangedInfo[propertyInfo.Name] = true;
Víme, že Dictionary je thread-safe pouze pro čtení, ale co se nám to tam v tom Insertu() děje, když každý tracked-objekt vzniká jako clone a má vlastní slovník pro ukládání změněných properties?
Pro doplnění z diagnostiky:
- System.__Canon je interní záležitost .NET frameworku, která se vyskytuje v určitých runtime situacích u generik,
Deeper Analysis
Zkusili jsme dále bádat, co přesně se v metodě Dictionary<>.Insert() děje, co by se mohlo zacyklit. Zdrojový kód metody Insert() vypadá poměrně očekávaně, klasické tanečky kolem hash-tabulky. Vytahuje se GetHashCode(), hledá se volný bucket, umísťuje se hodnota, …
private void Insert(TKey key, TValue value, bool add) { if( key == null ) { ThrowHelper.ThrowArgumentNullException(ExceptionArgument.key); } if (buckets == null) Initialize(0); int hashCode = comparer.GetHashCode(key) & 0x7FFFFFFF; int targetBucket = hashCode % buckets.Length; #if FEATURE_RANDOMIZED_STRING_HASHING int collisionCount = 0; #endif for (int i = buckets[targetBucket]; i >= 0; i = entries[i].next) { if (entries[i].hashCode == hashCode && comparer.Equals(entries[i].key, key)) { if (add) { ThrowHelper.ThrowArgumentException(ExceptionResource.Argument_AddingDuplicate); } entries[i].value = value; version++; return; } #if FEATURE_RANDOMIZED_STRING_HASHING collisionCount++; #endif } int index; if (freeCount > 0) { index = freeList; freeList = entries[index].next; freeCount--; } else { if (count == entries.Length) { Resize(); targetBucket = hashCode % buckets.Length; } index = count; count++; } entries[index].hashCode = hashCode; entries[index].next = buckets[targetBucket]; entries[index].key = key; entries[index].value = value; buckets[targetBucket] = index; version++; #if FEATURE_RANDOMIZED_STRING_HASHING if(collisionCount > HashHelpers.HashCollisionThreshold && HashHelpers.IsWellKnownEqualityComparer(comparer)) { comparer = (IEqualityComparer<TKey>) HashHelpers.GetRandomizedEqualityComparer(comparer); Resize(entries.Length, true); } #endif }
Vydali jsme se cestou pokusu o zjištění, kde přesně se v metodě Insert() nacházíme (což bylo asi nadbytečné, jak se ukázalo později, navíc v metodě je jediný cyklus, takže se celkem nabízí, nicméně nám to pomohlo nasměrovat další síly k cíli).
Ve WinDbg jsme si nechali vypsat pomocí !ClrStack zásobník jednoho z postižených threadů (všechny vypadaly stejně plus/mínus pár instrukcí dopředu/dozadu uvnitř metody Insert()):
0:041> !ClrStack OS Thread Id: 0xbe0 (41) Child SP IP Call Site 000000001206bb70 000007fef7371fc0 System.Collections.Generic.Dictionary`2[[System.__Canon, mscorlib],[System.Boolean, mscorlib]].Insert(System.__Canon, Boolean, Boolean) 000000001206bc00 000007fe999e025e Umbraco.Core.Models.EntityBase.TracksChangesEntityBase.OnPropertyChanged(System.Reflection.PropertyInfo) 000000001206bc50 000007fe999e92bc Umbraco.Core.Models.EntityBase.TracksChangesEntityBase.SetPropertyValueAndDetectChanges[[System.__Canon, mscorlib]](System.Func`2<System.__Canon,System.__Canon>, System.__Canon, System.Reflection.PropertyInfo) 000000001206bc90 000007fe99b611c0 Umbraco.Core.Models.ContentType.set_AllowedTemplates(System.Collections.Generic.IEnumerable`1<Umbraco.Core.Models.ITemplate>) 000000001206bf20 000007fef85ba7f3 [DebuggerU2MCatchHandlerFrame: 000000001206bf20] 000000001206c238 000007fef85ba7f3 [HelperMethodFrame_PROTECTOBJ: 000000001206c238] System.RuntimeMethodHandle.InvokeMethod(System.Object, System.Object[], System.Signature, Boolean) 000000001206c3b0 000007fef72b3b4c System.Reflection.RuntimeMethodInfo.UnsafeInvokeInternal(System.Object, System.Object[], System.Object[]) 000000001206c420 000007fef72b1273 System.Reflection.RuntimeMethodInfo.Invoke(System.Object, System.Reflection.BindingFlags, System.Reflection.Binder, System.Object[], System.Globalization.CultureInfo) 000000001206c4a0 000007fef73782cc System.Reflection.RuntimePropertyInfo.SetValue(System.Object, System.Object, System.Object[]) 000000001206c4f0 000007fe99b6222d Umbraco.Core.Models.DeepCloneHelper.DeepCloneRefProperties(Umbraco.Core.Models.IDeepCloneable, Umbraco.Core.Models.IDeepCloneable) 000000001206c5a0 000007fe99b6256d Umbraco.Core.Models.ContentTypeCompositionBase.DeepClone() 000000001206c5f0 000007fe999eeb84 Umbraco.Core.Models.ContentTypeCompositionBase.<DeepClone>b__2f(Umbraco.Core.Models.IContentTypeComposition) 000000001206c620 000007fef5972134 System.Linq.Enumerable+WhereSelectListIterator`2[[System.__Canon, mscorlib],[System.__Canon, mscorlib]].MoveNext() 000000001206c680 000007fef72e55b8 System.Collections.Generic.List`1[[System.__Canon, mscorlib]]..ctor(System.Collections.Generic.IEnumerable`1<System.__Canon>) 000000001206c6f0 000007fef5975180 System.Linq.Enumerable.ToList[[System.__Canon, mscorlib]](System.Collections.Generic.IEnumerable`1<System.__Canon>) 000000001206c730 000007fe99b626eb Umbraco.Core.Models.ContentTypeCompositionBase.DeepClone() 000000001206c780 000007fe999cefaa Umbraco.Core.Persistence.Caching.RuntimeCacheProvider.GetById(System.Type, System.Guid) 000000001206c7e0 000007fe999d00fb Umbraco.Core.Persistence.Repositories.RepositoryBase`2[[System.Int32, mscorlib],[System.__Canon, mscorlib]].TryGetFromCache(Int32) 000000001206c880 000007fe999cfe5c Umbraco.Core.Persistence.Repositories.RepositoryBase`2[[System.Int32, mscorlib],[System.__Canon, mscorlib]].Get(Int32) 000000001206c900 000007fe999d24dd Umbraco.Core.Persistence.Repositories.ContentRepository.CreateContentFromDto(Umbraco.Core.Models.Rdbms.DocumentDto, System.Guid) 000000001206c9d0 000007fe999d0a10 Umbraco.Core.Persistence.Repositories.ContentRepository.PerformGet(Int32) 000000001206ca60 000007fe999cfe9b Umbraco.Core.Persistence.Repositories.RepositoryBase`2[[System.Int32, mscorlib],[System.__Canon, mscorlib]].Get(Int32) 000000001206cae0 000007fe999c6069 Umbraco.Core.Services.ContentService.GetById(Int32) 000000001206cb40 000007fe99b88228 Havit.XyzPortal.Services.ContentManagement.EpoUrlProvider.GetUrl(Umbraco.Core.ApplicationContext, Int32, System.Uri, Umbraco.Web.Routing.UrlProviderMode) 000000001206cba0 000007fe99b86452 Umbraco.Web.Routing.UrlProvider+<>c__DisplayClass3.<GetUrl>b__0(Umbraco.Web.Routing.IUrlProvider) 000000001206cbe0 000007fef5972012 System.Linq.Enumerable+WhereSelectArrayIterator`2[[System.__Canon, mscorlib],[System.__Canon, mscorlib]].MoveNext() 000000001206cc20 000007fef5964e47 System.Linq.Enumerable.FirstOrDefault[[System.__Canon, mscorlib]](System.Collections.Generic.IEnumerable`1<System.__Canon>, System.Func`2<System.__Canon,Boolean>) 000000001206cc80 000007fe99b8684c Umbraco.Web.Routing.UrlProvider.GetUrl(Int32, System.Uri, Umbraco.Web.Routing.UrlProviderMode) 000000001206ccd0 000007fe99b7c580 Umbraco.Web.Templates.TemplateUtilities.ParseInternalLinks(System.String) 000000001206cd60 000007fe99d1e99e Havit.XyzPortal.UmbracoCMS.UmContentApiController.TraverseResourceDictionary(System.Collections.Generic.List`1<Havit.XyzPortal.Services.Resources.DataObjects.ResourceDictionaryDTO>, Umbraco.Core.Models.IContent, Boolean) 000000001206ce20 000007fe99d1e74d Havit.XyzPortal.UmbracoCMS.UmContentApiController.TraverseResourceDictionary(System.Collections.Generic.List`1<Havit.XyzPortal.Services.Resources.DataObjects.ResourceDictionaryDTO>, Umbraco.Core.Models.IContent, Boolean) 000000001206cee0 000007fe99d1e74d Havit.XyzPortal.UmbracoCMS.UmContentApiController.TraverseResourceDictionary(System.Collections.Generic.List`1<Havit.XyzPortal.Services.Resources.DataObjects.ResourceDictionaryDTO>, Umbraco.Core.Models.IContent, Boolean) 000000001206cfa0 000007fe99d1b003 Havit.XyzPortal.UmbracoCMS.UmContentApiController.Resources(System.Nullable`1<Boolean>) 000000001206d000 000007fe99d1aed1 Havit.XyzPortal.UmbracoCMS.UmContentApiController.Resources() 000000001206d040 000007fe99e1d25f DynamicClass.lambda_method(System.Runtime.CompilerServices.Closure, System.Object, System.Object[]) 000000001206d070 000007fe999c6555 System.Web.Http.Controllers.ReflectedHttpActionDescriptor+ActionExecutor+<>c__DisplayClass13.<GetExecutor>b__c(System.Object, System.Object[]) 000000001206d0b0 000007fe999c663f System.Web.Http.Controllers.ReflectedHttpActionDescriptor+ActionExecutor.Execute(System.Object, System.Object[]) 000000001206d0e0 000007fe999c7ec8 System.Threading.Tasks.TaskHelpers.RunSynchronously[[System.__Canon, mscorlib]](System.Func`1<System.Threading.Tasks.Task`1<System.__Canon>>, System.Threading.CancellationToken) 000000001206d130 000007fe999c7c19 System.Web.Http.Controllers.ApiControllerActionInvoker+<>c__DisplayClass3.<InvokeActionAsync>b__0() 000000001206d180 000007fe999c7ec8 System.Threading.Tasks.TaskHelpers.RunSynchronously[[System.__Canon, mscorlib]](System.Func`1<System.Threading.Tasks.Task`1<System.__Canon>>, System.Threading.CancellationToken) 000000001206d1d0 000007fe999c8188 System.Web.Http.Controllers.ApiControllerActionInvoker.InvokeActionAsync(System.Web.Http.Controllers.HttpActionContext, System.Threading.CancellationToken) 000000001206d220 000007fe999c82a3 System.Web.Http.ApiController+<>c__DisplayClass3.<ExecuteAsync>b__2() 000000001206d250 000007fe999c84c5 System.Web.Http.ApiController+<>c__DisplayClass3.<ExecuteAsync>b__1() 000000001206d2a0 000007fe999c84ff System.Threading.Tasks.TaskHelpersExtensions+<>c__DisplayClass32`1[[System.__Canon, mscorlib]].<Then>b__31(System.Threading.Tasks.Task) 000000001206d2d0 000007fe999c88a4 System.Threading.Tasks.TaskHelpersExtensions.ThenImpl[[System.__Canon, mscorlib],[System.__Canon, mscorlib]](System.__Canon, System.Func`2<System.__Canon,System.Threading.Tasks.Task`1<System.__Canon>>, System.Threading.CancellationToken, Boolean) 000000001206d340 000007fe999c7614 System.Web.Http.ApiController+<>c__DisplayClass3.<ExecuteAsync>b__0() 000000001206d3a0 000007fe999c05df System.Web.Http.ApiController.ExecuteAsync(System.Web.Http.Controllers.HttpControllerContext, System.Threading.CancellationToken) 000000001206d410 000007fe999bb3a6 System.Web.Http.Dispatcher.HttpControllerDispatcher.SendAsync(System.Net.Http.HttpRequestMessage, System.Threading.CancellationToken) 000000001206d4a0 000007fefb2ee33f System.Net.Http.HttpMessageInvoker.SendAsync(System.Net.Http.HttpRequestMessage, System.Threading.CancellationToken) 000000001206d4f0 000007fe999bbd55 System.Web.Http.Dispatcher.HttpRoutingDispatcher.SendAsync(System.Net.Http.HttpRequestMessage, System.Threading.CancellationToken) 000000001206d550 000007fe999be00f System.Web.Http.HttpServer.SendAsync(System.Net.Http.HttpRequestMessage, System.Threading.CancellationToken) 000000001206d5a0 000007fefb2ee33f System.Net.Http.HttpMessageInvoker.SendAsync(System.Net.Http.HttpRequestMessage, System.Threading.CancellationToken) 000000001206d5f0 000007fe999bb84f System.Web.Http.WebHost.HttpControllerHandler.BeginProcessRequest(System.Web.HttpContextBase, System.AsyncCallback, System.Object) 000000001206d670 000007fee9865ca2 System.Web.HttpApplication+CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute() 000000001206d750 000007fee982d110 System.Web.HttpApplication.ExecuteStep(IExecutionStep, Boolean ByRef) 000000001206d7f0 000007fee984a78a System.Web.HttpApplication+PipelineStepManager.ResumeSteps(System.Exception) 000000001206d940 000007fee982d2a3 System.Web.HttpApplication.BeginProcessRequestNotification(System.Web.HttpContext, System.AsyncCallback) 000000001206d990 000007fee98271de System.Web.HttpRuntime.ProcessRequestNotificationPrivate(System.Web.Hosting.IIS7WorkerRequest, System.Web.HttpContext) 000000001206da30 000007fee9830161 System.Web.Hosting.PipelineRuntime.ProcessRequestNotificationHelper(IntPtr, IntPtr, IntPtr, Int32) 000000001206dc40 000007fee982fb92 System.Web.Hosting.PipelineRuntime.ProcessRequestNotification(IntPtr, IntPtr, IntPtr, Int32) 000000001206dc90 000007fee9f86ec1 DomainNeutralILStubClass.IL_STUB_ReversePInvoke(Int64, Int64, Int64, Int32) 000000001206e478 000007fef861b42e [InlinedCallFrame: 000000001206e478] System.Web.Hosting.UnsafeIISMethods.MgdIndicateCompletion(IntPtr, System.Web.RequestNotificationStatus ByRef) 000000001206e478 000007fee98d919b [InlinedCallFrame: 000000001206e478] System.Web.Hosting.UnsafeIISMethods.MgdIndicateCompletion(IntPtr, System.Web.RequestNotificationStatus ByRef) 000000001206e450 000007fee98d919b DomainNeutralILStubClass.IL_STUB_PInvoke(IntPtr, System.Web.RequestNotificationStatus ByRef) 000000001206e520 000007fee983034f System.Web.Hosting.PipelineRuntime.ProcessRequestNotificationHelper(IntPtr, IntPtr, IntPtr, Int32) 000000001206e730 000007fee982fb92 System.Web.Hosting.PipelineRuntime.ProcessRequestNotification(IntPtr, IntPtr, IntPtr, Int32) 000000001206e780 000007fee9f86ec1 DomainNeutralILStubClass.IL_STUB_ReversePInvoke(Int64, Int64, Int64, Int32) 000000001206e978 000007fef861b683 [ContextTransitionFrame: 000000001206e978]
…a protože nemáme k dispozici převod offsetu na řádky zdrojového kódu mscorlib.dll, vydali jsme se přes !U /d <addr> na disassembly aktuálních instrukcí:
0:041> !U /d 000007fef7371fc0 preJIT generated code System.Collections.Generic.Dictionary`2[[System.__Canon, mscorlib],[System.Boolean, mscorlib]].Insert(System.__Canon, Boolean, Boolean) Begin 000007fef7371dd0, size 29a. Cold region begin 000007fef80c24e4, size 173 Hot region: 000007fe`f7371dd0 53 push rbx 000007fe`f7371dd1 55 push rbp 000007fe`f7371dd2 56 push rsi 000007fe`f7371dd3 57 push rdi 000007fe`f7371dd4 4154 push r12 000007fe`f7371dd6 4155 push r13 000007fe`f7371dd8 4156 push r14 000007fe`f7371dda 4157 push r15 000007fe`f7371ddc 4883ec48 sub rsp,48h 000007fe`f7371de0 410fb6c1 movzx eax,r9b 000007fe`f7371de4 88442428 mov byte ptr [rsp+28h],al 000007fe`f7371de8 450fb6e8 movzx r13d,r8b 000007fe`f7371dec 44886c2429 mov byte ptr [rsp+29h],r13b 000007fe`f7371df1 4c8be2 mov r12,rdx 000007fe`f7371df4 488bd9 mov rbx,rcx 000007fe`f7371df7 48895c2420 mov qword ptr [rsp+20h],rbx 000007fe`f7371dfc 4d85e4 test r12,r12 000007fe`f7371dff 0f84e607d500 je mscorlib_ni+0x12a25eb (000007fe`f80c25eb) 000007fe`f7371e05 488b4308 mov rax,qword ptr [rbx+8] 000007fe`f7371e09 4885c0 test rax,rax 000007fe`f7371e0c 0f8463010000 je mscorlib_ni+0x551f75 (000007fe`f7371f75) 000007fe`f7371e12 488b7b18 mov rdi,qword ptr [rbx+18h] 000007fe`f7371e16 488b03 mov rax,qword ptr [rbx] 000007fe`f7371e19 488b4830 mov rcx,qword ptr [rax+30h] 000007fe`f7371e1d 488b01 mov rax,qword ptr [rcx] 000007fe`f7371e20 488b4018 mov rax,qword ptr [rax+18h] 000007fe`f7371e24 4885c0 test rax,rax 000007fe`f7371e27 750f jne mscorlib_ni+0x551e38 (000007fe`f7371e38) 000007fe`f7371e29 488d153a4c1100 lea rdx,[mscorlib_ni+0x666a6a (000007fe`f7486a6a)] 000007fe`f7371e30 488b0b mov rcx,qword ptr [rbx] 000007fe`f7371e33 e82053eaff call mscorlib_ni+0x3f7158 (000007fe`f7217158) (mscorlib_ni) 000007fe`f7371e38 803f00 cmp byte ptr [rdi],0 000007fe`f7371e3b 498bd4 mov rdx,r12 000007fe`f7371e3e 488bcf mov rcx,rdi 000007fe`f7371e41 4c8bd8 mov r11,rax 000007fe`f7371e44 ff10 call qword ptr [rax] 000007fe`f7371e46 8be8 mov ebp,eax 000007fe`f7371e48 81e5ffffff7f and ebp,7FFFFFFFh 000007fe`f7371e4e 4c8b4308 mov r8,qword ptr [rbx+8] 000007fe`f7371e52 498b4808 mov rcx,qword ptr [r8+8] 000007fe`f7371e56 81fd00000080 cmp ebp,80000000h 000007fe`f7371e5c 7509 jne mscorlib_ni+0x551e67 (000007fe`f7371e67) 000007fe`f7371e5e 83f9ff cmp ecx,0FFFFFFFFh 000007fe`f7371e61 0f843407d500 je mscorlib_ni+0x12a259b (000007fe`f80c259b) 000007fe`f7371e67 8bc5 mov eax,ebp 000007fe`f7371e69 99 cdq 000007fe`f7371e6a f7f9 idiv eax,ecx 000007fe`f7371e6c 448bfa mov r15d,edx 000007fe`f7371e6f 4533f6 xor r14d,r14d 000007fe`f7371e72 4963f7 movsxd rsi,r15d 000007fe`f7371e75 483bf1 cmp rsi,rcx 000007fe`f7371e78 0f831807d500 jae mscorlib_ni+0x12a2596 (000007fe`f80c2596) 000007fe`f7371e7e 418b74b010 mov esi,dword ptr [r8+rsi*4+10h] 000007fe`f7371e83 85f6 test esi,esi 000007fe`f7371e85 0f8d15010000 jge mscorlib_ni+0x551fa0 (000007fe`f7371fa0) 000007fe`f7371e8b 8b4344 mov eax,dword ptr [rbx+44h] 000007fe`f7371e8e 85c0 test eax,eax 000007fe`f7371e90 0f8fd506d500 jg mscorlib_ni+0x12a256b (000007fe`f80c256b) 000007fe`f7371e96 8b5338 mov edx,dword ptr [rbx+38h] 000007fe`f7371e99 488b4310 mov rax,qword ptr [rbx+10h] 000007fe`f7371e9d 488b4808 mov rcx,qword ptr [rax+8] 000007fe`f7371ea1 3bd1 cmp edx,ecx 000007fe`f7371ea3 0f849e000000 je mscorlib_ni+0x551f47 (000007fe`f7371f47) 000007fe`f7371ea9 8b7b38 mov edi,dword ptr [rbx+38h] 000007fe`f7371eac 8d4701 lea eax,[rdi+1] 000007fe`f7371eaf 894338 mov dword ptr [rbx+38h],eax 000007fe`f7371eb2 488b4b10 mov rcx,qword ptr [rbx+10h] 000007fe`f7371eb6 4863f7 movsxd rsi,edi 000007fe`f7371eb9 488b4108 mov rax,qword ptr [rcx+8] 000007fe`f7371ebd 483bf0 cmp rsi,rax 000007fe`f7371ec0 0f83d006d500 jae mscorlib_ni+0x12a2596 (000007fe`f80c2596) 000007fe`f7371ec6 488d0476 lea rax,[rsi+rsi*2] 000007fe`f7371eca 488d4cc110 lea rcx,[rcx+rax*8+10h] 000007fe`f7371ecf 896908 mov dword ptr [rcx+8],ebp 000007fe`f7371ed2 488b5308 mov rdx,qword ptr [rbx+8] 000007fe`f7371ed6 4963ef movsxd rbp,r15d 000007fe`f7371ed9 488b4208 mov rax,qword ptr [rdx+8] 000007fe`f7371edd 483be8 cmp rbp,rax 000007fe`f7371ee0 0f83b006d500 jae mscorlib_ni+0x12a2596 (000007fe`f80c2596) 000007fe`f7371ee6 8b44aa10 mov eax,dword ptr [rdx+rbp*4+10h] 000007fe`f7371eea 89410c mov dword ptr [rcx+0Ch],eax 000007fe`f7371eed 498bd4 mov rdx,r12 000007fe`f7371ef0 e81351eaff call mscorlib_ni+0x3f7008 (000007fe`f7217008) (mscorlib_ni) 000007fe`f7371ef5 488b4b10 mov rcx,qword ptr [rbx+10h] 000007fe`f7371ef9 488b4108 mov rax,qword ptr [rcx+8] 000007fe`f7371efd 483bf0 cmp rsi,rax 000007fe`f7371f00 0f839006d500 jae mscorlib_ni+0x12a2596 (000007fe`f80c2596) 000007fe`f7371f06 488d0476 lea rax,[rsi+rsi*2] 000007fe`f7371f0a 44886cc120 mov byte ptr [rcx+rax*8+20h],r13b 000007fe`f7371f0f 488b4b08 mov rcx,qword ptr [rbx+8] 000007fe`f7371f13 488b4108 mov rax,qword ptr [rcx+8] 000007fe`f7371f17 483be8 cmp rbp,rax 000007fe`f7371f1a 0f837606d500 jae mscorlib_ni+0x12a2596 (000007fe`f80c2596) 000007fe`f7371f20 897ca910 mov dword ptr [rcx+rbp*4+10h],edi 000007fe`f7371f24 8b433c mov eax,dword ptr [rbx+3Ch] 000007fe`f7371f27 ffc0 inc eax 000007fe`f7371f29 89433c mov dword ptr [rbx+3Ch],eax 000007fe`f7371f2c 4183fe64 cmp r14d,64h 000007fe`f7371f30 0f8fae05d500 jg mscorlib_ni+0x12a24e4 (000007fe`f80c24e4) 000007fe`f7371f36 4883c448 add rsp,48h 000007fe`f7371f3a 415f pop r15 000007fe`f7371f3c 415e pop r14 000007fe`f7371f3e 415d pop r13 000007fe`f7371f40 415c pop r12 000007fe`f7371f42 5f pop rdi 000007fe`f7371f43 5e pop rsi 000007fe`f7371f44 5d pop rbp 000007fe`f7371f45 5b pop rbx 000007fe`f7371f46 c3 ret 000007fe`f7371f47 488bcb mov rcx,rbx 000007fe`f7371f4a e8d1010000 call mscorlib_ni+0x552120 (000007fe`f7372120) (System.Collections.Generic.Dictionary`2[[System.__Canon, mscorlib],[System.Boolean, mscorlib]].Resize(), mdToken: 00000000060021a7) 000007fe`f7371f4f 488b4308 mov rax,qword ptr [rbx+8] 000007fe`f7371f53 488b4808 mov rcx,qword ptr [rax+8] 000007fe`f7371f57 81fd00000080 cmp ebp,80000000h 000007fe`f7371f5d 7509 jne mscorlib_ni+0x551f68 (000007fe`f7371f68) 000007fe`f7371f5f 83f9ff cmp ecx,0FFFFFFFFh 000007fe`f7371f62 0f843306d500 je mscorlib_ni+0x12a259b (000007fe`f80c259b) 000007fe`f7371f68 8bc5 mov eax,ebp 000007fe`f7371f6a 99 cdq 000007fe`f7371f6b f7f9 idiv eax,ecx 000007fe`f7371f6d 448bfa mov r15d,edx 000007fe`f7371f70 e934ffffff jmp mscorlib_ni+0x551ea9 (000007fe`f7371ea9) 000007fe`f7371f75 33d2 xor edx,edx 000007fe`f7371f77 488bcb mov rcx,rbx 000007fe`f7371f7a e8f1000000 call mscorlib_ni+0x552070 (000007fe`f7372070) (System.Collections.Generic.Dictionary`2[[System.__Canon, mscorlib],[System.Boolean, mscorlib]].Initialize(Int32), mdToken: 00000000060021a4) 000007fe`f7371f7f e98efeffff jmp mscorlib_ni+0x551e12 (000007fe`f7371e12) 000007fe`f7371f84 488d0449 lea rax,[rcx+rcx*2] 000007fe`f7371f88 8b74c21c mov esi,dword ptr [rdx+rax*8+1Ch] 000007fe`f7371f8c 85f6 test esi,esi 000007fe`f7371f8e 0f8ccc000000 jl mscorlib_ni+0x552060 (000007fe`f7372060) 000007fe`f7371f94 666666660f1f840000000000 nop word ptr [rax+rax] 000007fe`f7371fa0 488b4b10 mov rcx,qword ptr [rbx+10h] 000007fe`f7371fa4 4863fe movsxd rdi,esi 000007fe`f7371fa7 488b4108 mov rax,qword ptr [rcx+8] 000007fe`f7371fab 483bf8 cmp rdi,rax 000007fe`f7371fae 0f83e205d500 jae mscorlib_ni+0x12a2596 (000007fe`f80c2596) 000007fe`f7371fb4 488d147f lea rdx,[rdi+rdi*2] 000007fe`f7371fb8 8b44d118 mov eax,dword ptr [rcx+rdx*8+18h] 000007fe`f7371fbc 3bc5 cmp eax,ebp 000007fe`f7371fbe 7419 je mscorlib_ni+0x551fd9 (000007fe`f7371fd9) >>> 000007fe`f7371fc0 41ffc6 inc r14d 000007fe`f7371fc3 488b5310 mov rdx,qword ptr [rbx+10h] 000007fe`f7371fc7 4863ce movsxd rcx,esi 000007fe`f7371fca 488b4208 mov rax,qword ptr [rdx+8] 000007fe`f7371fce 483bc8 cmp rcx,rax 000007fe`f7371fd1 0f83bf05d500 jae mscorlib_ni+0x12a2596 (000007fe`f80c2596) 000007fe`f7371fd7 ebab jmp mscorlib_ni+0x551f84 (000007fe`f7371f84) 000007fe`f7371fd9 4c8b6b18 mov r13,qword ptr [rbx+18h] 000007fe`f7371fdd 488b44d110 mov rax,qword ptr [rcx+rdx*8+10h] 000007fe`f7371fe2 4889442430 mov qword ptr [rsp+30h],rax 000007fe`f7371fe7 488b03 mov rax,qword ptr [rbx] 000007fe`f7371fea 488b4830 mov rcx,qword ptr [rax+30h] 000007fe`f7371fee 488b01 mov rax,qword ptr [rcx] 000007fe`f7371ff1 488b4020 mov rax,qword ptr [rax+20h] 000007fe`f7371ff5 4885c0 test rax,rax 000007fe`f7371ff8 750f jne mscorlib_ni+0x552009 (000007fe`f7372009) 000007fe`f7371ffa 488d15754a1100 lea rdx,[mscorlib_ni+0x666a76 (000007fe`f7486a76)] 000007fe`f7372001 488b0b mov rcx,qword ptr [rbx] 000007fe`f7372004 e84f51eaff call mscorlib_ni+0x3f7158 (000007fe`f7217158) (mscorlib_ni) 000007fe`f7372009 41807d0000 cmp byte ptr [r13],0 000007fe`f737200e 4d8bc4 mov r8,r12 000007fe`f7372011 488b542430 mov rdx,qword ptr [rsp+30h] 000007fe`f7372016 498bcd mov rcx,r13 000007fe`f7372019 4c8bd8 mov r11,rax 000007fe`f737201c ff10 call qword ptr [rax] 000007fe`f737201e 0fb6c8 movzx ecx,al 000007fe`f7372021 85c9 test ecx,ecx 000007fe`f7372023 749b je mscorlib_ni+0x551fc0 (000007fe`f7371fc0) 000007fe`f7372025 8a442428 mov al,byte ptr [rsp+28h] 000007fe`f7372029 0fb6c0 movzx eax,al 000007fe`f737202c 85c0 test eax,eax 000007fe`f737202e 0f856c05d500 jne mscorlib_ni+0x12a25a0 (000007fe`f80c25a0) 000007fe`f7372034 488b4b10 mov rcx,qword ptr [rbx+10h] 000007fe`f7372038 488b4108 mov rax,qword ptr [rcx+8] 000007fe`f737203c 483bf8 cmp rdi,rax 000007fe`f737203f 0f835105d500 jae mscorlib_ni+0x12a2596 (000007fe`f80c2596) 000007fe`f7372045 488d047f lea rax,[rdi+rdi*2] 000007fe`f7372049 8a542429 mov dl,byte ptr [rsp+29h] 000007fe`f737204d 8854c120 mov byte ptr [rcx+rax*8+20h],dl 000007fe`f7372051 8b433c mov eax,dword ptr [rbx+3Ch] 000007fe`f7372054 ffc0 inc eax 000007fe`f7372056 89433c mov dword ptr [rbx+3Ch],eax 000007fe`f7372059 e9d8feffff jmp mscorlib_ni+0x551f36 (000007fe`f7371f36) 000007fe`f737205e 6690 xchg ax,ax 000007fe`f7372060 448a6c2429 mov r13b,byte ptr [rsp+29h] 000007fe`f7372065 e921feffff jmp mscorlib_ni+0x551e8b (000007fe`f7371e8b) Cold region: 000007fe`f80c24e4 488d7b18 lea rdi,[rbx+18h] 000007fe`f80c24e8 488b0f mov rcx,qword ptr [rdi] 000007fe`f80c24eb e8b098a6ff call mscorlib_ni+0xd0bda0 (000007fe`f7b2bda0) (System.Collections.HashHelpers.IsWellKnownEqualityComparer(System.Object), mdToken: 00000000060020f7) 000007fe`f80c24f0 0fb6c8 movzx ecx,al 000007fe`f80c24f3 85c9 test ecx,ecx 000007fe`f80c24f5 0f843bfa2aff je mscorlib_ni+0x551f36 (000007fe`f7371f36) 000007fe`f80c24fb 488b0f mov rcx,qword ptr [rdi] 000007fe`f80c24fe e81d99a6ff call mscorlib_ni+0xd0be20 (000007fe`f7b2be20) (System.Collections.HashHelpers.GetRandomizedEqualityComparer(System.Object), mdToken: 00000000060020f8) 000007fe`f80c2503 488bf0 mov rsi,rax 000007fe`f80c2506 488b03 mov rax,qword ptr [rbx] 000007fe`f80c2509 488b4830 mov rcx,qword ptr [rax+30h] 000007fe`f80c250d 488b01 mov rax,qword ptr [rcx] 000007fe`f80c2510 488b4028 mov rax,qword ptr [rax+28h] 000007fe`f80c2514 4885c0 test rax,rax 000007fe`f80c2517 750f jne mscorlib_ni+0x12a2528 (000007fe`f80c2528) 000007fe`f80c2519 488d15c04c3cff lea rdx,[mscorlib_ni+0x6671e0 (000007fe`f74871e0)] 000007fe`f80c2520 488b0b mov rcx,qword ptr [rbx] 000007fe`f80c2523 e8304c15ff call mscorlib_ni+0x3f7158 (000007fe`f7217158) (mscorlib_ni) 000007fe`f80c2528 488bd6 mov rdx,rsi 000007fe`f80c252b 488bc8 mov rcx,rax 000007fe`f80c252e e8354c15ff call mscorlib_ni+0x3f7168 (000007fe`f7217168) (mscorlib_ni) 000007fe`f80c2533 488bd0 mov rdx,rax 000007fe`f80c2536 488bcf mov rcx,rdi 000007fe`f80c2539 e8ca4a15ff call mscorlib_ni+0x3f7008 (000007fe`f7217008) (mscorlib_ni) 000007fe`f80c253e 488b4310 mov rax,qword ptr [rbx+10h] 000007fe`f80c2542 488b5008 mov rdx,qword ptr [rax+8] 000007fe`f80c2546 41b001 mov r8b,1 000007fe`f80c2549 488bcb mov rcx,rbx 000007fe`f80c254c 488d051dfc2aff lea rax,[mscorlib_ni+0x552170 (000007fe`f7372170)] 000007fe`f80c2553 4883c448 add rsp,48h 000007fe`f80c2557 415f pop r15 000007fe`f80c2559 415e pop r14 000007fe`f80c255b 415d pop r13 000007fe`f80c255d 415c pop r12 000007fe`f80c255f 5f pop rdi 000007fe`f80c2560 5e pop rsi 000007fe`f80c2561 5d pop rbp 000007fe`f80c2562 5b pop rbx 000007fe`f80c2563 48ffe0 jmp rax 000007fe`f80c2566 e9cbf92aff jmp mscorlib_ni+0x551f36 (000007fe`f7371f36) 000007fe`f80c256b 8b7b40 mov edi,dword ptr [rbx+40h] 000007fe`f80c256e 488b5310 mov rdx,qword ptr [rbx+10h] 000007fe`f80c2572 4863cf movsxd rcx,edi 000007fe`f80c2575 488b4208 mov rax,qword ptr [rdx+8] 000007fe`f80c2579 483bc8 cmp rcx,rax 000007fe`f80c257c 7318 jae mscorlib_ni+0x12a2596 (000007fe`f80c2596) 000007fe`f80c257e 488d0449 lea rax,[rcx+rcx*2] 000007fe`f80c2582 8b4cc21c mov ecx,dword ptr [rdx+rax*8+1Ch] 000007fe`f80c2586 894b40 mov dword ptr [rbx+40h],ecx 000007fe`f80c2589 8b4344 mov eax,dword ptr [rbx+44h] 000007fe`f80c258c ffc8 dec eax 000007fe`f80c258e 894344 mov dword ptr [rbx+44h],eax 000007fe`f80c2591 e91cf92aff jmp mscorlib_ni+0x551eb2 (000007fe`f7371eb2) 000007fe`f80c2596 e8d54a15ff call mscorlib_ni+0x3f7070 (000007fe`f7217070) (mscorlib_ni) 000007fe`f80c259b e8b04b15ff call mscorlib_ni+0x3f7150 (000007fe`f7217150) (mscorlib_ni) 000007fe`f80c25a0 b90e000000 mov ecx,0Eh 000007fe`f80c25a5 e83e9915ff call mscorlib_ni+0x3fbee8 (000007fe`f721bee8) (System.ThrowHelper.GetResourceName(System.ExceptionResource), mdToken: 0000000006000261) 000007fe`f80c25aa 488bc8 mov rcx,rax 000007fe`f80c25ad e8665f15ff call mscorlib_ni+0x3f8518 (000007fe`f7218518) (System.Environment.GetResourceFromDefault(System.String), mdToken: 0000000006000c8a) 000007fe`f80c25b2 488bd8 mov rbx,rax 000007fe`f80c25b5 488d0ddcac44ff lea rcx,[mscorlib_ni+0x6ed298 (000007fe`f750d298)] 000007fe`f80c25bc e8574a15ff call mscorlib_ni+0x3f7018 (000007fe`f7217018) (mscorlib_ni) 000007fe`f80c25c1 488bf8 mov rdi,rax 000007fe`f80c25c4 488bd3 mov rdx,rbx 000007fe`f80c25c7 488bcf mov rcx,rdi 000007fe`f80c25ca e8518e1eff call mscorlib_ni+0x48b420 (000007fe`f72ab420) (System.Exception..ctor(System.String), mdToken: 000000000600001d) 000007fe`f80c25cf c7878c00000001151380 mov dword ptr [rdi+8Ch],80131501h 000007fe`f80c25d9 c7878c00000057000780 mov dword ptr [rdi+8Ch],80070057h 000007fe`f80c25e3 488bcf mov rcx,rdi 000007fe`f80c25e6 e8954a15ff call mscorlib_ni+0x3f7080 (000007fe`f7217080) (mscorlib_ni) 000007fe`f80c25eb b905000000 mov ecx,5 000007fe`f80c25f0 e8e39815ff call mscorlib_ni+0x3fbed8 (000007fe`f721bed8) (System.ThrowHelper.GetArgumentName(System.ExceptionArgument), mdToken: 0000000006000260) 000007fe`f80c25f5 488bd8 mov rbx,rax 000007fe`f80c25f8 488d0d01a144ff lea rcx,[mscorlib_ni+0x6ec700 (000007fe`f750c700)] 000007fe`f80c25ff e8144a15ff call mscorlib_ni+0x3f7018 (000007fe`f7217018) (mscorlib_ni) 000007fe`f80c2604 488bf8 mov rdi,rax 000007fe`f80c2607 488d15f2e9d5fe lea rdx,[mscorlib_ni+0x1000 (000007fe`f6e21000)] 000007fe`f80c260e b9a8650070 mov ecx,700065A8h 000007fe`f80c2613 e8604a15ff call mscorlib_ni+0x3f7078 (000007fe`f7217078) (mscorlib_ni) 000007fe`f80c2618 488bc8 mov rcx,rax 000007fe`f80c261b e8f85e15ff call mscorlib_ni+0x3f8518 (000007fe`f7218518) (System.Environment.GetResourceFromDefault(System.String), mdToken: 0000000006000c8a) 000007fe`f80c2620 488bd0 mov rdx,rax 000007fe`f80c2623 488bcf mov rcx,rdi 000007fe`f80c2626 e8b58d1eff call mscorlib_ni+0x48b3e0 (000007fe`f72ab3e0) (System.SystemException..ctor(System.String), mdToken: 0000000006000515) 000007fe`f80c262b 488d8f98000000 lea rcx,[rdi+98h] 000007fe`f80c2632 488bd3 mov rdx,rbx 000007fe`f80c2635 e8ce4915ff call mscorlib_ni+0x3f7008 (000007fe`f7217008) (mscorlib_ni) 000007fe`f80c263a c7878c00000057000780 mov dword ptr [rdi+8Ch],80070057h 000007fe`f80c2644 c7878c00000003400080 mov dword ptr [rdi+8Ch],80004003h 000007fe`f80c264e 488bcf mov rcx,rdi 000007fe`f80c2651 e82a4a15ff call mscorlib_ni+0x3f7080 (000007fe`f7217080) (mscorlib_ni) 000007fe`f80c2656 90 nop
Tohle už je trochu pocitové luštění, přeci-jenom čistý assembler je lepší psát, než interpretovat. Z instrukcí pro inkrementaci, porovnávání a skoky jsme však dospěli k závěru, že se nacházíme právě uvězněni uvnitř cyklu for v metodě Insert. Hypotéza je, že došlo k poškození spojového seznamu entries a ty nejspíš tvoří cyklus, jehož nekonečné procházení nás vězní.
Cause
No dobrá. Myslíme si, že víme, co se děje, ale proč?
Znovu procházíme odspoda zdrojové kódy metod, přes než call-stack prochází. Už tušíme, co chceme hledat (vícenásobný přístup na jeden slovník, který by však měl být pro každou instanci entity samostatný) a dostáváme se do kritického místa:
// class Entity : TracksChangesEntityBase private IDictionary<string, bool> _propertyChangedInfo = new Dictionary<string, bool>(); public virtual object DeepClone() { Entity output = (Entity) base.MemberwiseClone(); DeepCloneHelper.DeepCloneRefProperties(this, output); output.ResetDirtyProperties(false); return output; } // TracksChangesEntityBase public virtual void ResetDirtyProperties(bool rememberPreviouslyChangedProperties) { if (rememberPreviouslyChangedProperties) { this._lastPropertyChangedInfo = this._propertyChangedInfo.ToDictionary<KeyValuePair<string, bool>, string, bool>(v => v.Key, v => v.Value); } this._propertyChangedInfo = new Dictionary<string, bool>(); }
Ukazuje se, že slovník změněných properties _propertyChangedInfo se zakládá nový pouze při vytvoření nové instance entity a při volání ResetDirtyProperties(). Metoda DeepClone() volaná po vyzvednutí originálu z cache však nejprve vytvoří MemberwiseClone() shallow-kopii objektu, která referencuje v _propertyChangedInfo stále původní Dictionary<>, pak začne klonovat referencované properties (avšak pod trackingem původního dosud neřešeného slovníku!!!) a až potom se provede ResetDirtyProperties(), který způsobí vytvoření nové instance slovníku pro _propertyChangedInfo.
Pokud se nám tedy sejde více threadů ve správný okamžik, dojde k simultálnímu zápisu do jednoho slovníku z více threadů, který není pro tuto operaci thread-safe. V našem případě tím dojde k nadrbání vnitřní struktury hash-tabulky (nejspíš k cyklu entries) a výsledkem bylo uvíznutí threadů v nekonečných cyklech for.
Závěr
…jdeme nahlásit bug v UmbracoCMS.
Jako work-around budeme muset zatím přidat nějaké vlastní zámky do míst, která vedou na problematický DeepClone() a zajistí pro začátek vzájemnou výlučnost takových volání.
Update
Koukám, že už je to v Umbraco reportováno a včera opraveno.