Class ConcurrentBatchIndexer

  • All Implemented Interfaces:

    public class ConcurrentBatchIndexer
    extends Object
    implements BatchIndexer
    A multi-threaded batch indexing implementation.

    Manages batches of indexable content, and provides progress updates on indexing completion.

    • Method Detail

      • submitBatches

        protected List<Future<?>> submitBatches​(ExecutorService executor,
                                                @Nullable List<HibernateHandle> handles,
                                                ReindexProgress progress)
        This method tries to work around a number of problems we have with rebuilding the index. At a high level it will submit batches of work to the executor service and return a list containing the Future of BatchIndexer.

        More specifically we deal with the following problems 1. Attachments are expensive (memory and CPU) 2. Progress display is Hard (a status update is very fast, compared to say an attachment). By spreading the types out a bit we can get better reporting of how long it will take. 3. OOMEs are a real possibility for naive indexing. 4. Round robin helps even out the work item batches and makes progress estimates more smooth and accurate 5. Variable batch sizes help with controlling Attachment processing and OOME avoidance

        A list of Future of BatchIndexer