What's the best Umbraco websites, subdomains and indexes configuration to avoid Lucene errors?

system · May 3, 2020, 7:18am

We are experiencing a series of application errors on Lucene indexes that result in the website crash and apppool restart.

I suspect this might be caused by our websites, hostnames and indexes configuration.
We have multiple root nodes in our Umbraco CMS (v 7.15.3 currently).
The Umbraco content is structured in the following way:

Home → site under mydomain.com hostname
Providers → site under business.mydomain.com hostname
Reviews
Bookings
Settings
etc.

As per the above two first roots have separate hostnames: mydomain.com and business.mydomain.com. In practice both point to the same Umbraco instance on IIS of course.

Both domains have separate IIS site entries and pools (with the same pool settings).

Both domains though use the same InternalIndexSet. This is because the main website uses it for indexing the Umbraco content, and our members can access business.mydomain.com to update their content. They are able to see unpublished content and publish it.

The error occurs normally when we are publishing a content in the CMS while a member is saving his content in the business area at the same time. We are able to replicate the issue while doing this (updating content on both sites).

We had before the following index setup:

<ExamineLuceneIndexSets>
  <IndexSet SetName="InternalIndexSet" IndexPath="~/App_Data/TEMP/ExamineIndexes/Internal/"/>
    <IndexSet SetName="MemberProductsIndexSet" IndexPath="~/App_Data/TEMP/ExamineIndexes/MemberProducts/" IndexParentId="1179">      </IndexSet>
<IndexSet SetName="PublishedContentIndexSet" IndexPath="~/App_Data/TEMP/ExamineIndexes/PublishedContent/" IndexParentId="1179">
</IndexSet>
</ExamineLuceneIndexSets>

where the IndexParentId is the id of the Home root (although I think it doesn’t matter in this case).
MemberProducts index was for both published & unpublished content, and PublishedContent is for only published content.

And the errors were once related to the InternalIndex another time to MemberProducts index depending probably which one was in use first.

We reduced the two above indexes to this:

<ExamineLuceneIndexSets>
  <IndexSet SetName="InternalIndexSet" IndexPath="~/App_Data/TEMP/ExamineIndexes/Internal/"/>
<IndexSet SetName="PublishedContentIndexSet" IndexPath="~/App_Data/TEMP/ExamineIndexes/PublishedContent/" IndexParentId="1179">
</IndexSet>
</ExamineLuceneIndexSets>

So now both use the same InternalIndex but we are still experiencing the problems, just on InternalIndex only now since the other index has gone.

The error we are getting can be one of this kind:

Exception: System.IO.IOException

Message: Cannot overwrite: C:\Inetpub\hosts\mydomain.com\httpdocs\App_Data\TEMP\ExamineIndexes\Internal\Index\_1.cfs

StackTrace:    at Lucene.Net.Store.FSDirectory.InitOutput(String name)
   at Lucene.Net.Store.SimpleFSDirectory.CreateOutput(String name)
   at Lucene.Net.Index.CompoundFileWriter.Close()
   at Lucene.Net.Index.DocumentsWriter.CreateCompoundFile(String segment)
   at Lucene.Net.Index.IndexWriter.DoFlushInternal(Boolean flushDocStores, Boolean flushDeletes)
   at Lucene.Net.Index.IndexWriter.DoFlush(Boolean flushDocStores, Boolean flushDeletes)
   at Lucene.Net.Index.IndexWriter.PrepareCommit(IDictionary`2 commitUserData, Boolean internal_Renamed)
   at Lucene.Net.Index.IndexWriter.Commit(IDictionary`2 commitUserData)
   at Examine.LuceneEngine.Providers.LuceneIndexer.IndexCommiter.TimerRelease() in C:\projects\examine-qvx04\src\Examine\LuceneEngine\Providers\LuceneIndexer.cs:line 1711
   at Examine.LuceneEngine.Providers.LuceneIndexer.IndexCommiter.<ScheduleCommit>b__7_0(Object _) in C:\projects\examine-qvx04\src\Examine\LuceneEngine\Providers\LuceneIndexer.cs:line 1663
   at System.Threading.TimerQueueTimer.CallCallbackInContext(Object state)
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.TimerQueueTimer.CallCallback()
   at System.Threading.TimerQueueTimer.Fire()
   at System.Threading.TimerQueue.FireNextTimers()
   at System.Threading.TimerQueue.AppDomainTimerCallback(Int32 id)

or

Exception: System.IO.IOException

Message: Cannot overwrite: C:\Inetpub\hosts\mydomain.com\httpdocs\App_Data\TEMP\ExamineIndexes\Internal\Index\_52.cfs

StackTrace:    at Lucene.Net.Store.FSDirectory.InitOutput(String name)
   at Lucene.Net.Store.SimpleFSDirectory.CreateOutput(String name)
   at Lucene.Net.Index.CompoundFileWriter.Close()
   at Lucene.Net.Index.DocumentsWriter.CreateCompoundFile(String segment)
   at Lucene.Net.Index.IndexWriter.DoFlushInternal(Boolean flushDocStores, Boolean flushDeletes)
   at Lucene.Net.Index.IndexWriter.DoFlush(Boolean flushDocStores, Boolean flushDeletes)
   at Lucene.Net.Index.IndexWriter.PrepareCommit(IDictionary`2 commitUserData, Boolean internal_Renamed)
   at Lucene.Net.Index.IndexWriter.Commit(IDictionary`2 commitUserData)
   at Examine.LuceneEngine.Providers.LuceneIndexer.IndexCommiter.TimerRelease() in C:\projects\examine-qvx04\src\Examine\LuceneEngine\Providers\LuceneIndexer.cs:line 1711
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.TimerQueueTimer.CallCallback()
   at System.Threading.TimerQueueTimer.Fire()
   at System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem()
   at System.Threading.ThreadPoolWorkQueue.Dispatch()

or

Exception: System.IndexOutOfRangeException

Message: Index of bound 1

StackTrace:    at Lucene.Net.Util.BitVector.GetAndSet(Int32 bit)
   at Lucene.Net.Index.SegmentReader.DoDelete(Int32 docNum)
   at Lucene.Net.Index.IndexReader.DeleteDocument(Int32 docNum)
   at Lucene.Net.Index.DocumentsWriter.ApplyDeletes(IndexReader reader, Int32 docIDStart)
   at Lucene.Net.Index.DocumentsWriter.ApplyDeletes(SegmentInfos infos)
   at Lucene.Net.Index.IndexWriter.ApplyDeletes()
   at Lucene.Net.Index.IndexWriter.DoFlushInternal(Boolean flushDocStores, Boolean flushDeletes)
   at Lucene.Net.Index.IndexWriter.DoFlush(Boolean flushDocStores, Boolean flushDeletes)
   at Lucene.Net.Index.IndexWriter.PrepareCommit(IDictionary`2 commitUserData, Boolean internal_Renamed)
   at Lucene.Net.Index.IndexWriter.Commit(IDictionary`2 commitUserData)
   at Examine.LuceneEngine.Providers.LuceneIndexer.IndexCommiter.TimerRelease() in C:\projects\examine-qvx04\src\Examine\LuceneEngine\Providers\LuceneIndexer.cs:line 1711
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.TimerQueueTimer.CallCallback()
   at System.Threading.TimerQueueTimer.Fire()
   at System.Threading.TimerQueue.FireNextTimers()

depending probably on the sequence of events, eg. which webiste tries to access the index first etc.

As I mentioned the error happens when a content is being updated and saved/published at the same time on both mydomain.com and business.mydomain.com. And it doesn’t need to be the same content.
The same issue were happening on previous versions of Umbraco (7.6 & 7.15.1).

I wonder if our settings are correct eg. the fact that we have different hostname using the same files and indexes but different pools is causing the issues?

If so what would be a correct/ suggested configuration?

We have the same configuration on localhost and stage site. The only difference is that they don’t have a separate business subdomain but rather use stage.mydomain.com/members hostname instead of separate subdomain for members area.

This issue does not occur in any of them.

We are using a dedicated server with IIS10 on Windows Server 2016 Standard, no load balancing and no machine key.

This is a companion discussion topic for the original entry at https://our.umbraco.com/forum/102097-whats-the-best-umbraco-websites-subdomains-and-indexes-configuration-to-avoid-lucene-errors