Content extraction

Connectors use the Elastic ingest attachment processor to extract file contents. The processor extracts files using the Apache Tika text extraction library. The logic for content extraction is defined in utils.py.

While intended primarily for PDF and Microsoft Office formats, you can use any of the supported formats.

Enterprise Search uses an Elasticsearch ingest pipeline to power the web crawler’s binary content extraction. The default pipeline, ent-search-generic-ingestion, is automatically created when Enterprise Search first starts.

You can view this pipeline in Kibana. Customizing your pipeline usage is also an option. See Ingest pipelines for Search indices.

For advanced use cases, the self-hosted extraction service can be used to extract content from files larger than 10MB.

The following file types are supported:

Standard content extraction is done via the Attachment Processor, through Elasticsearch Ingest Pipelines. The self-managed connector limits file sizes for pipeline extraction to 10MB per file (Elasticsearch also has a hard limit of 100MB per file).

For use cases that require extracting content from files larger than these limits, the self-managed extraction service can be used for self-managed connectors. Instead of sending the file as an attachment to Elasticsearch, the file’s content is extracted at the edge by the extraction service before ingestion. The extracted text is then included as the body field of a document when it is ingested.

To use this feature, you will need to do the following:

Local content extraction is available for the following self-managed connectors:

Self-hosted content extraction is handled by a separate extraction service.

The versions for the extraction service do not align with the Elastic stack. For versions after 8.11.x (including 8.16.0), you should use extraction service version 0.3.x.

You can run the service with the following command:

$ docker run \
  -p 8090:8090 \
  -it \
  --name extraction-service \
  docker.elastic.co/integrations/data-extraction-service:$EXTRACTION_SERVICE_VERSION

You can enable your self-managed connector to use the self-hosted extraction service by adding the required configuration. The self-managed connector determines if the extraction service is enabled by the presence of these fields in the configuration file.

# data-extraction-service settings
extraction_service:
  host: https://127.0.0.1:8090

The self-managed connector will perform a preflight check against the configured host value at startup. The following line will be output to the log if the data extraction service was found and is running normally.

Data extraction service found at <HOST>.

If you don’t see this log at startup, refer to troubleshooting self-hosted content extraction service.

The following fields can be included in the configuration file. They are optional and will fallback on default values if they are not specified.

# data-extraction-service settings
extraction_service:
  host: https://127.0.0.1:8090
  timeout: 30
  use_file_pointers: false
  stream_chunk_size: 65536
  shared_volume_dir: '/app/files'

The self-hosted extraction service can be set up to use file pointers instead of sending files via HTTP requests. File pointers are faster than sending files and consume less memory, but require the connector framework and the extraction service to be able to share a file system. This can be set up with both a dockerized and non-dockerized self-managed connector.

If you are running a non-dockerized version of the self-managed connector, you need to determine the local directory where you’ll download files for extraction. This can be anywhere on your file system that you are comfortable using. Be aware that the self-managed connector will download files with randomized filenames to this directory, so there is a chance that any files already present will be overwritten. For that reason, we recommend using a dedicated directory for self-hosted extraction.

======= Example

When using self-hosted extraction from a dockerized self-managed connector, there are a few extra steps required on top of running the self-managed connector in docker.

======= Example

The extraction service produces two different log files that may be informative when troubleshooting. These are saved at the following file locations internally in the docker container:

Logs can be viewed from outside of docker by combining docker exec with the tail command.

$ docker exec extraction-service /bin/sh -c "tail /var/log/openresty.log"
$ docker exec extraction-service /bin/sh -c "tail /var/log/tika.log"

The following warning logs may appear while using self-hosted extraction service. Each log in this section is followed by a description of what may have happened, and suggested fixes.

Extraction service is not configured, skipping its preflight check.

The configuration file is missing the extraction_service.host field. If you want to use this service, check that the configuration is formatted correctly and that the required field is present.

Data extraction service found at <HOST>, but health-check returned <RESPONSE STATUS>.

The /ping endpoint returned a non-200 response. This could mean that the extraction service is unhealthy and may need to be restarted, or that the configured extraction_service.host is incorrect. You can find more information about what happened in the data extraction service logs.

Expected to find a running instance of data extraction service at <HOST> but failed. <ERROR>.

The health check returned either a timeout or client connection error.

Extraction service has been initialised but no extraction service configuration was found. No text will be extracted for this sync.

You have enabled self-hosted extraction service for the connector, but the configuration file is missing the extraction_service.host field. Check that the configuration is formatted correctly and that the required field is present.

Extraction service could not parse <FILENAME>. Status: <RESPONSE STATUS>; <ERROR NAME>: <ERROR MESSAGE>.

This warning will appear every time a file is not extractable. Generally the <ERROR MESSAGE> will provide an explanation for why extraction failed. Contact support if the message is unclear. When a file fails extraction, it will be indexed with an empty string in the body field.