The Elasticsearch client library is a Python package that is installed with 

. Make sure the virtual environment you created earlier is activated, and then run the following command to install the client:

To avoid any potential incompatibilities, make sure the version of the Elasticsearch client library you install matches the version of the Elasticsearch stack that you are using.

 file updated with all your dependencies, so this is a good time to update this file to include the newly installed package. Run the following command from your terminal:

To create a connection to your Elasticsearch service, an 

 object must be created with the appropriate connection options.

 file in your code editor, located in the 

 file is going to be where all the search functions will be defined. The idea of having a separate file for the search functionality is that this will make it easy for you to extract this file and add it into your own projects later on.

 function that is called right after the imports comes from the 

 package. This package knows how to work with 

 files, which are used to store configuration variables such as passwords and keys. The 

 function reads the variables that are stored in the 

 file and imports them into the Python process as environment variables.

 class has a constructor that creates an instance of the 

 client class. This is where all the client logic to communicate with the Elasticsearch service lives. Note that this line is currently incomplete, as connection options appropriate to your service need to be included. You will learn what options apply in your case below. Once created, the 

 object is then stored in an instance variable named 

To ensure that the client object can communicate with your Elastic Cloud deployment, the 

 method is invoked. This method makes a call to the service requesting basic information. If this call succeeds, then you can assume that you have a valid connection to the service.

The method then prints a status message indicating that the connection has been established, and then uses the 

 function from Python to display the information that the service returned in an easy to read format.

 object needs to be given appropriate connection options. The following sub-sections will tell you what options you need for the Elastic Cloud and Docker methods of installation.

, you will need to know the deployment's Cloud ID and your API Key. Since these are sensitive values, it is not a good idea to include them directly in the application code. Instead, create a 

 (pronounced dot-env) file in which these secrets can be safely stored.

Open your favorite code editor and create a new file in the 

 (don't forget the leading dot). Enter the following contents into this file:

After you have entered your credentials in the 

 arguments are extracted from the environment, which Python maintains in the 

 dictionary. These variables are read from the 

 file and stored in this dictionary by the 

Connect to a Self-Hosted Elasticsearch Docker Container

run a locally hosted Elasticsearch service

 using Docker, then the only connection option that is needed is the connection endpoint. Edit the first line of the 

 object is instantiated with the URL to the top-level endpoint of the Elasticsearch service, which is normally 

 explicitly disable encryption and authentication, so for that reason it is not necessary to provide any credentials.

At this point you are ready to make a connection to your Elasticsearch service. To do this, make sure that your Python virtual environment is activated, and then type 

 to start a Python interactive session. You should see the familiar 

 prompt, in which you can enter Python statements.

You should see a connected message, followed by the information returned by the 

 method of the client. Except for differences in identifiers and version numbers, the output should look like the following:

If you receive errors, make sure that you have entered correct credentials in the 

 file if you are using an Elastic Cloud deployment, or in the case of a self-hosted deployment, that you are running the Elasticsearch Docker container in your computer 

Integrate Elasticsearch with the Flask Application

The final step in this section is to integrate the work done so far into the little Flask application that you installed earlier. The goal is for the application to automatically create a connection to Elasticsearch when it starts.

 in your code editor. Add an import statement for the 

 variable is created, and right after create the instance of the new 

 object to use when needed. If you are still running the Flask application in a terminal, you should see the application reload as soon as you save the file. As a result of the reload, the connection information printed by the 

 class constructor should appear, and will continue to appear every time the application restarts from now on.

If you were not running the Flask application, now it is a good time to start it. Change to the project directory in a terminal window, activate the Python virtual environment, and then start the application with:

To help you troubleshoot in case you receive errors, here is a complete copy of 

 with the integrated Elasticsearch client:

Python Client Setup

Set Up the Elasticsearch Python Client

Two very important concepts in Elasticsearch are 

A document is collection of fields with their associated values. To work with Elasticsearch you have to organize your data into documents, and then add all your documents to an index. You can think of an index as a collection of documents that is stored in a highly optimized format designed to perform efficient searches.

If you have worked with other databases, you may know that many databases require a schema definition, which is essentially a description of all the fields that you want to store and their types. An Elasticsearch index can be configured with a schema if desired, but it can also automatically derive the schema from the data itself. In this section you are going to let Elasticsearch figure out the schema on its own, which works quite well for simple data types such as text, numbers and dates. Later, after you are introduced to more complex data types, you will learn how to provide explicit schema definitions.

This is how you create an Elasticsearch index using the Python client library:

 class, which in this tutorial is stored in the 

. An Elasticsearch deployment can be used to store multiple indexes, each identified by a name such as 

If you attempt to create an index with a name that is already assigned to an existing index, you will get an error. Sometimes it is useful to create an index automatically deleting a previous instance of the index if it exists. This is especially useful while developing an application, because you will likely need to regenerate an index several times.

. Open this file in your code editor, and add the following code at the bottom, leaving the existing contents as they are:

 method first deletes an index with the name 

 option prevents this call from failing when the index name isn't found. The following line in the method creates a brand new index with that same name.

In the Elasticsearch client library for Python, a document is represented as a dictionary of key/value fields. Fields that have a string value are automatically indexed for full-text and keyword search, but in addition to strings you can use other field types such as numbers, dates and booleans, which are also indexed for efficient operations such as filtering. You can also build complex data structures in which a field is set to a list or a dictionary with sub-items.

 method of the Elasticsearch client is used. For example:

 method is the response returned by the Elasticsearch service. The most important piece of information returned in this response is an item with a key name of 

, representing the unique identifier that was assigned to the document when it was inserted into the index. The identifier can be used to retrieve, delete or update the document.

Now that you know how to insert a document, let's continue building a library of useful helpers in 

 class. Add this method at the bottom of 

The method accepts the Elasticsearch client and a document from the caller, and inserts the document into the 

 index, returning the response from the service.

When setting up a new Elasticsearch index, you are likely going to need to import a large number of documents. For this tutorial, the starter project includes a 

 file with some data in JSON format. In this section you will learn how to import all the documents contained in this file into the index.

The structure of the documents that are included in the 

: a URL to the document hosted on an external site

: a short summary of the contents of the document

: update date (could be missing if the document was never updated)

At this point you are encouraged to open 

 in your editor to familiarize yourself with the data that you are going to work with.

In essence, importing a large number of documents is no different than importing one document inside a for-loop. To import the entire contents of the 

While this approach works, it does not scale well. If you had to insert a very large number of documents, you would need to make as many calls into the Elasticsearch service. Unfortunately there is a performance cost associated with each API call, and also the service has rate limits in place that prevent large number of calls to be made very quickly. For these reasons, it is best to use the bulk insertion feature of the Elasticsearch service, which allows several operations to be communicated to the service in a single API call.

 method shown below, which you should add at the bottom of 

 method to insert all the documents in a single call:

The method accepts a list of documents. Instead of adding each document separately, it assembles a single list called 

 method. For each document, two entries are added to the 

A description of what operation to perform, set to 

, with the name of the index given as an argument.

When processing a bulk request, the Elasticsearch service walks the operations list from the start and performs the operations that were requested.

Learn more about the bulk() method in the documentation

While you work on this tutorial you will need to regenerate the index a few times. To streamline this operation, add a 

 methods created earlier, so that with a single call the old index can be destroyed (if it exists) and a new index built and repopulated.

To make this method easier to invoke, let's expose it through the 

 in your code editor and add the following function at the bottom:

 decorator tells the Flask framework to register this function as a custom command, which will be available as 

. The name of the command is taken from the function's name, and the docstring is included here because Flask uses it in the 

 function, which in turn is the response from the 

 method of the Elasticsearch client, contains some useful items that can be used to construct a nice status message. In particular, 

 is the duration of the call in milliseconds, and 

 is a list of the individual results of each operation, which is not actually useful directly, but the length of the list provides a count of documents inserted.

 from a terminal session, making sure the Python virtual environment is activated (if your terminal session is still running the Flask application, you can open a second terminal window). Towards the end of the help screen you should see the 

 option included as an available command along with other options provided by the Flask framework:

Now when you want to generate a clean index, all you need to do is run 

Create an Index

Create an Elasticsearch Index

Now that you have built an Elasticsearch index and loaded some documents into it, you are ready to implement full-text search.

Let's have a quick review of how the search solution will work in the tutorial application. With the Flask application running, you can go to 

 to access the main page, which looks like this:

The code that renders this page is implemented in the 

This is a very simple endpoint that renders an HTML template. In Flask applications templates are located in a 

 sub-directory, so you will find this and other templates included with the application there.

Let's look at the implementation of the search field in file 

. Here is the relevant portion of this template:

Here you can see that this is an HTML form with a single field of type 

, which tells the browser to submit this form in a POST request. The 

 attribute is set to the URL that corresponds to the 

 endpoint of the Flask application. When the form is submitted, the 

The function obtains the text typed by the user in the text field from Flask's 

 local variable. The function then renders the 

 template, but this type it passes some additional arguments so that the page can show the search results. The four arguments that the template receives are:

: the query text entered by the user in the form.

: the zero-based index of the first result

Since the search functionality is not implemented, for now the arguments that are passed to the 

 function indicate that no results were found.

The task now is to implement a full-text query and pass actual results so that the 

 (Domain Specific Language) based on the JSON format to define queries.

 method that is used to submit a search query. Let's add a 

 method of the Elasticsearch client with the index name. The 

 argument captures all the keyword arguments provided to the method, and then passes-them through to the 

 method. These arguments are going to be how the caller specifies what to search for.

 offers many different ways to query an index. Looking through the sub-sections in the documentation you will familiarize with the different types of queries that are possible. The very common task of searching text is covered in the 

For the first search implementation, let's use the 

. Below you can see an example that uses this query:

The example above is given in a format that resembles a raw HTTP request. It is useful to be familiar with this format, as it is used extensively in the Elasticsearch documentation and in the Elasticsearch 

. Luckily this format is very easy to translate into a call using the Python client library. Below you can see the equivalent Python code for the above example:

When converting API Console examples to Python, remember that the top-level keys in the body of the query have to be converted to keyword arguments in the Python call. The examples also do not specify an index, which would be needed when making the Python call.

By looking at the query structure you can probably deduce what kind of search this is requesting. The call requests a 

This style of query is reasonably easy to incorporate into the tutorial applications. Open 

 method. Replace the current version with this new one:

 in the second line of this new version of the endpoint invokes the 

Can you figure out what the query is going to do? This is a 

 query similar to the above example. The field that is going to be searched is 

, which contains the titles of the documents in the 

 index that you built in the previous section. The text to search for is what the user typed in the search field on the web page, which is stored in the 

The part of the search response that contains the results is 

. This is an object with a few keys, of which two are of interest in this implementation:

: the total number of results that are available. The number of results is given in a 

 sub-key, so in practice, the expression to get the total number of results is 

. Note that the total number of results can be an approximation when there are a large number of results. See the 

 in this new version of the endpoint passes the list of results in the 

 template argument, and the total number of results in 

 argument receives the query string as before, and 

 is still hardcoded to 0, as it will be implemented later when pagination is added.

And with this, the application has a first implementation of full-text search. Go back to your web browser and navigate to 

 to open the application. If for any reason you do not have the Flask application running, start the application again before doing this. Enter a search text such as 

 and you will see relevant results. Below you can see the results when searching for 

 template that you downloaded with the starter application includes all the logic to render search results. If you are curious about this, here is the section of this template that renders the result list:

From this code it is interesting to note that the data associated with a returned result is available under the 

 field that contains the unique identifier assigned to the result.

A score associated with each result can be obtained from 

. The score provides a measure of relevance, with higher scores indicating a closer match to the query text. By default results are returned in order by their score, from highest to lowest. Scores in Elasticsearch are calculated using the 

If you are interested in exploring the topics covered in this section in more detail, use the following links:

 template renders the title of each search result as a link. The link points to the third and last endpoint that came implemented in the starter Flask application, called 

. The implementation that is provided returns a "Document not found" hardcoded text, so this is what you will see if you click on any of the results while playing with the application.

To correctly render individual documents, let's add a 

Here you can see how these unique identifiers that were assigned to each document are useful, as this is what the application can use to refer to individual documents.

Here is the current implementation of the 

You can see that the URL associated with this endpoint includes the document 

, and the links that are rendered for each search result also have the id incorporated in the respective URLs, so all that is missing is to replace this simplistic implementation with one that retrieves the document and renders it. Replace the endpoint with this updated version:

 is used to obtain the requested document. The 

 is then rendered, with a title that comes from the 

Try running some more queries, and clicking on results, which should now allow you to see the full content.

After you played with the application for a while you may have noticed that a lot of queries return no results. As you recall, the search is currently implemented on the 

 field of each document, which is where the document titles are stored. Documents also have 

 fields, which have longer texts that are apt to be searched as well, but right now these are ignored.

In this section you are going to learn about another common full-text search query, the 

, which requests a search to be carried out across multiple fields of an index.

Here is the example multi-match query from the documentation:

Let's use this example as a base to expand the 

 endpoint to run multi-match queries on the 

 fields combined. Here is the updated endpoint code:

With this change, there is a lot more text to search, so much that some queries may have more than the maximum of 10 results that are returned by default. In the next chapter you will learn about dealing with long lists of results through pagination.

Search Basics

It is often impractical for an application to deal with a very large number of results. For this reason, APIs and web services use 

 controls to allow applications to request the results in small chunks or pages.

You may have noticed that Elasticsearch by default does not return more than 10 results. The optional 

 parameter can be given in a search request to change this maximum. The following example asks for up to 5 search results to be returned:

To access additional pages of results, the 

 parameter is used, which indicates from where in the complete list of results to start (since 

The next example retrieves a second page of 5 results:

Here the page size is now hardcoded to 5 (feel free to use any other number that you like). The 

 argument is assumed to be given as an additional field in the submitted form, but this field is considered optional, defaulting to 0 when not present.

 field, so regular searches will always start from the first result. The template displays information about the range of results that shown, and what the total is. Here is how this is done using template expressions:

The template also includes logic to display pagination buttons to move forwards or backwards in the list of results. Here is the implementation of the "Previous results" button:

As you can see, the "Previous page" button is only rendered to the page when 

 is greater than zero. The implementation of this button uses the browser's history API to go back one page.

The "Next page" button has a much more interesting implementation:

This button isn't actually a standalone button, but a complete form that has two hidden fields in addition to the button. The form is similar to the main search form, but includes the optional 

 field, adjusted to point to the next page of results. When this button is clicked, the Flask application will receive a search request from this alternate form, which uses the same text query but a non-zero 

With this small and clever implementation of pagination you will be able to navigate through multiple pages of results.

Pagination

Many applications need to give users the power to customize queries in ways that complement what search queries alone can do. In this chapter you are going to learn about 

, a technique that makes it possible to specify that a search query is executed only on the subset of the documents contained in an index that satisfy a given condition.

Full-Text Search

Ready to build state of the art search experiences?