Elasticsearch query rules are now generally available

Query rules allow fine-grained, contextually specific solutions to alter search results for specific queries or search use cases. This can be helpful for campaigns that require branded or sponsored results to be pinned to the top of the search result lists for specific keywords, but can also be helpful for head queries where you simply need to "fix" the top result.

Introduction

We’re happy to announce that query rules are generally available in our serverless offering, and will be generally available starting with stack version 8.15.0. Query rules were first introduced in 8.10.0 as a technical preview feature, and allow index maintainers to curate specific documents to pin at the top of results based on contextual query-entered criteria.

How do query rules work?

Query rules are rules that you define based on specific query metadata. You start by defining a query ruleset that identifies documents to promote for certain metadata sent in with the query. Then at search time, you send that metadata in along with a rule query. If the metadata sent in the rule query matches any rules, those rules will be applied to your result set.

New query rule features

We added some features to query rules as we marched along the path to general availability.

A quick summary of these changes:

We’ve renamed the rule query from rule_query to rule, to be more consistent with our other API calls
We now support specifying multiple rulesets in a single rule query
We’ve expanded our query ruleset management APIs to support managing individual query rules

What does this look like?

Let’s say we have an index with dog breed information containing two fields: dog_breed and advert. We want to set up rules that pin breeds and mixed breeds in the same rule. Here’s a sample ruleset with two rules:

PUT _query_rules/dog-breed-ruleset
{
 "ruleset_id": "dog-breed-ruleset",
 "rules": [
   {
     "rule_id": "pug-mixes",
     "type": "pinned",
     "criteria": [
       {
         "type": "exact",
         "metadata": "breed",
         "values": [
           "pug"
         ]
       }
     ],
     "actions": {
       "ids": [
         "pug",
         "puggle",
         "chug",
         "pugshire"
       ]
     },
     "priority": 5
   },
   {
     "rule_id": "chi-mixes",
     "type": "pinned",
     "criteria": [
       {
         "type": "exact",
         "metadata": "breed",
         "values": [
           "chihauhua"
         ]
       }
     ],
     "actions": {
       "ids": [
         "chihauhua",
         "chiweenie",
         "chug"
       ]
     },
     "priority": 10
   }
 ]
}

Unpacking what this ruleset does:

If the rule query sends in breed: pug, the first results in order will be: pug, puggle, chug, and pugshire. Any organic results will be returned after those pinned results.
If the rule query sends in breed: chihuahua, the first results in order will be: chihuahua, chiweenie, and chug. Any organic results will be returned after those pinned results.

One thing to note is the priority of each rule in the ruleset. This is an optional part of each rule, but it can help you define the position in which an individual rule is inserted into the ruleset. Rulesets are sorted in ascending priority order, though as you can see in this example they don’t have to be contiguous. Inserting a new rule into this ruleset with a priority of 4 or lower would insert the new rule at the beginning of the ruleset, while a priority between 6 and 9 would insert the new rule in the middle of the ruleset between the two existing rules. If an individual rule is added to the ruleset but doesn’t include a priority, it will simply be appended to the end of the ruleset.

Another useful feature of query rules is that we now support multiple rulesets to be passed into the rule query. This allows you to organize and define many more rules; previously you were limited to the number of rules that could fit into a single ruleset (100 by default or configurable up to 1000 per ruleset).

Let’s take a look at what this looks like, by creating a new ruleset specifically for a July promotion. This rule is of type “always” so it will always be returned no matter what:

PUT _query_rules/promo-ruleset
{
  "rules": [
    {
      "rule_id": "july-promo",
      "type": "pinned",
      "criteria": [
        {
          "type": "always"
        }
      ],
      "actions": {
        "ids": [
          "july-promotion"
        ]
      }
    }
  ]
}

Now, we can query these rulesets together using the following query:

GET query-rules-test/_search
{
  "query": {
    "rule": {
      "organic": {
        "query_string": {
          "query": "chihauhua mixes"
        }
      },
      "ruleset_ids": [
        "promo-ruleset",
        "dog-breed-ruleset"
      ],
      "match_criteria": {
        "breed": "chihauhua"
      }
    }
  }
}

Since two rulesets were specified, we’ll process each ruleset in the order they were specified in the request. This means that the July promotion rule will always be returned as the very first result, and the matching dog breeds will be pinned subsequently, in a result that looks like this:

   "hits": [
     {
       "_index": "query-rules-test",
       "_id": "july-promotion",
       "_score": 1.7014128e+38,
       "_source": {
         "advert": "EVERYTHING ON SALE!"
       }
     },
     {
       "_index": "query-rules-test",
       "_id": "chihauhua",
       "_score": 1.7014126e+38,
       "_source": {
         "dog_breed": "chihauhua"
       }
     },
     {
       "_index": "query-rules-test",
       "_id": "chiweenie",
       "_score": 1.7014124e+38,
       "_source": {
         "dog_breed": "chiweenie"
       }
     },
     {
       "_index": "query-rules-test",
       "_id": "chug",
       "_score": 1.7014122e+38,
       "_source": {
         "dog_breed": "chug"
       }
     },
     ... // organic results follow... 
   ]

Complex query rule examples

Let’s continue our dog theme, but expand the mappings in our test index:

PUT query-rules-test
{
 "mappings": {
   "properties": {
     "breed": {
       "type": "keyword"
     },
     "age": {
       "type": "integer"
     },
     "sex": {
       "type": "keyword"
     },
     "name": {
       "type": "keyword"
     },
     "bio": {
       "type": "text"
     },
     "good_with_animals": {
       "type": "boolean"
     },
     "good_with_kids": {
        "type": "boolean"
     }
   }
 }
}

We can index the following json into this index for some sample data. For consistency, we're going to assume the _id field matches the json id field:

[
  {"id":"buddy_pug","breed":"pug","sex":"Male","age":3,"name":"Buddy","bio":"Buddy is a charming pug who loves to play and snuggle. He is looking for a loving home.","good_with_animals":true,"good_with_kids":true},
  {"id":"lucy_beagle","breed":"beagle","sex":"Female","age":7,"name":"Lucy","bio":"Lucy is a friendly beagle who enjoys long walks and is very affectionate.","good_with_animals":true,"good_with_kids":true},
  {"id":"rocky_chihuahua","breed":"chihuahua","sex":"Male","age":2,"name":"Rocky","bio":"Rocky is a tiny chihuahua with a big personality. He is very playful and loves attention.","good_with_animals":false,"good_with_kids":false},
  {"id":"zoe_dachshund","breed":"dachshund","sex":"Female","age":5,"name":"Zoe","bio":"Zoe is a sweet dachshund who loves to burrow under blankets and is very loyal.","good_with_animals":true,"good_with_kids":true},
  {"id":"max_poodle","breed":"poodle","sex":"Male","age":6,"name":"Max","bio":"Max is a smart and active poodle who loves learning new tricks and is very friendly.","good_with_animals":true,"good_with_kids":true},
  {"id":"bella_yorkie","breed":"yorkie","sex":"Female","age":4,"name":"Bella","bio":"Bella is a cute yorkie who loves to be pampered and is very affectionate.","good_with_animals":true,"good_with_kids":true},
  {"id":"jack_puggle","breed":"puggle","sex":"Male","age":8,"name":"Jack","bio":"Jack is a friendly puggle who loves to play and is very social.","good_with_animals":true,"good_with_kids":true},
  {"id":"lola_chiweenie","breed":"chiweenie","sex":"Female","age":9,"name":"Lola","bio":"Lola is a playful chiweenie who loves to chase toys and is very affectionate.","good_with_animals":true,"good_with_kids":true},
  {"id":"charlie_chug","breed":"chug","sex":"Male","age":3,"name":"Charlie","bio":"Charlie is an energetic chug who loves to play and is very curious.","good_with_animals":false,"good_with_kids":true},
  {"id":"daisy_pugshire","breed":"pugshire","sex":"Female","age":7,"name":"Daisy","bio":"Daisy is a gentle pugshire who loves to cuddle and is very sweet.","good_with_animals":true,"good_with_kids":true},
  {"id":"buster_pug","breed":"pug","sex":"Male","age":10,"name":"Buster","bio":"Buster is a calm pug who loves to lounge around and is very loyal.","good_with_animals":true,"good_with_kids":true},
  {"id":"molly_beagle","breed":"beagle","sex":"Female","age":12,"name":"Molly","bio":"Molly is a sweet beagle who loves to sniff around and is very friendly.","good_with_animals":true,"good_with_kids":true},
  {"id":"toby_chihuahua","breed":"chihuahua","sex":"Male","age":4,"name":"Toby","bio":"Toby is a lively chihuahua who loves to run around and is very playful.","good_with_animals":false,"good_with_kids":false},
  {"id":"luna_dachshund","breed":"dachshund","sex":"Female","age":1,"name":"Luna","bio":"Luna is a young dachshund who loves to explore and is very curious.","good_with_animals":true,"good_with_kids":true},
  {"id":"oliver_poodle","breed":"poodle","sex":"Male","age":17,"name":"Oliver","bio":"Oliver is a wise poodle who loves to relax and is very gentle.","good_with_animals":true,"good_with_kids":true}]

Next, let's create a ruleset:

PUT _query_rules/rescue-dog-search-ruleset
{
  "rules": [
    {
      "rule_id": "pugs_and_pug_mixes",
      "type": "pinned",
      "criteria": [
        {
          "type": "contains",
          "metadata": "query_string",
          "values": [
            "pug"
          ]
        }
      ],
      "actions": {
        "ids": [
          "buddy_pug",
          "buster_pug",
          "jack_puggle",
          "daisy_pugshire"
        ]
      }
    },
    {
      "rule_id": "puppies",
      "type": "pinned",
      "criteria": [
        {
          "type": "contains",
          "metadata": "query_string",
          "values": [
            "puppy",
            "puppies"
          ]
        }
      ],
      "actions": {
        "ids": [
          "luna_dachsund"
        ]
      }
    },
    {
      "rule_id": "puppies2",
      "type": "pinned",
      "criteria": [
        {
          "type": "lte",
          "metadata": "preferred_age",
          "values": [
            2
          ]
        }
      ],
      "actions": {
        "ids": [
          "luna_dachshund"
        ]
      }
    },
    {
      "rule_id": "adult",
      "type": "pinned",
      "criteria": [
        {
          "type": "gt",
          "metadata": "preferred_age",
          "values": [
            2
          ]
        },
        {
          "type": "lt",
          "metadata": "preferred_age",
          "values": [
            10
          ]
        }
      ],
      "actions": {
        "ids": [
          "lucy_beagle"
        ]
      }
    },
    {
      "rule_id": "special_needs",
      "type": "pinned",
      "criteria": [
        {
          "type": "exact",
          "metadata": "work_from_home",
          "values": [
            "true"
          ]
        },
        {
          "type": "exact",
          "metadata": "fenced_in_yard",
          "values": [
            "true"
          ]
        },
        {
          "type": "exact",
          "metadata": "kids_in_house",
          "values": [
            "false"
          ]
        },
        {
          "type": "exact",
          "metadata": "cats_in_house",
          "values": [
            "false"
          ]
        }
      ],
      "actions": {
        "ids": [
          "toby_chihuahua"
        ]
      }
    }
  ]
}

Unpacking this ruleset:

There’s a pugs_and_pug_mixes rule, which will pin all pugs and pug mixes if the query string contains the word “pug”
The puppies rule will return the youngest dog in our index if the query string contains “puppy” or “puppies”
Another puppies2 rule will return the same dog if the preferred age is less than or equal to 2
The adult rule will pin a specific beagle if the preferred age is between 2 and 10
We have a complex special_needs rule that will only hit if the prospective owner works from home, has a fenced in yard, and has no children or cats living in the house.

Let’s look at some example queries - these examples use match_none as the organic query, so the only results that are returned are for the rules themselves.

GET query-rules-test/_search
{
  "query": {
    "rule": {
      "organic": {
        "match_none": {}
      },
      "ruleset_ids": [
        "rescue-dog-search-ruleset"
      ],
      "match_criteria": {
        "query_string": "I like pugs and pug mixes",
        "preferred_age": 5
      }
    }
  }
}

The above query will match the “pug and pug mixes” rule and return the 4 pug results. However, since the preferred age is 5, we will return the adult dog, Lucy the beagle, as the 5th pinned result.

Changing the preferred age to 1 swaps the 5th results below the pug results for the dachshund puppy:

GET query-rules-test/_search
{
  "query": {
    "rule": {
      "organic": {
        "match_none": {}
      },
      "ruleset_ids": [
        "rescue-dog-search-ruleset"
      ],
      "match_criteria": {
        "query_string": "I like pugs and pug mixes",
        "preferred_age": 1
      }
    }
  }
}

In order to match the special needs result, all of the criteria must match:

{
  "query": {
    "rule": {
      "organic": {
        "match_none": {}
      },
      "ruleset_ids": [
        "rescue-dog-search-ruleset"
      ],
      "match_criteria": {
        "work_from_home": "true",
        "fenced_in_yard": "true",
        "kids_in_house": "false",
        "cats_in_house": "false"
      }
    }
  }
}

If a single criteria does not match, this rule will not be triggered:

GET query-rules-test/_search
{
  "query": {
    "rule": {
      "organic": {
        "match_none": {}
      },
      "ruleset_ids": [
        "rescue-dog-search-ruleset"
      ],
      "match_criteria": {
        "work_from_home": "true",
        "fenced_in_yard": "true",
        "kids_in_house": "false",
        "cats_in_house": "true"
      }
    }
  }
}

Troubleshooting query rules

It may be hard to tell whether a pinned query rule was applied. There are two ways to tell whether query rules are pinning the document that you want vs. being returned organically.

We do have the explain API, but this can be non-obvious: rule queries are rewritten into pinned queries which are then rewritten to constant score queries, so this will basically look like the max possible score:

"_explanation": {
  "value": 1.7014128e+38,
  "description": "max of:",
  "details": [
    {
      "value": 1.7014128e+38,
      "description": "max of:",
      "details": [
        {
          "value": 1.7014128e+38,
          "description": "ConstantScore(_id:([ff 63 68 69 68 61 75 68 75 61]))^1.7014128E38",
          "details": []
        },
        ...
      ]
    },
    ...
  ]
}

For pinned rules, you can also return only the rule by using a rule query combined with a match none query, as we did above for illustrative purposes:

GET query-rules-test/_search
{
  "query": {
    "rule": {
      "organic": {
        "match_none": {}
      },
      "ruleset_ids": [
        "dog-breed-ruleset",
        "super-special-ruleset"
      ],
      "match_criteria": {
        "breed": "pug"
      }
    }
  }
}

Alternately you can simply run the organic query by itself and compare the results of that query with your rule query.

What's next?

Just because query rules has been made generally available doesn’t mean that we are done!

We have a lot of exciting features on our roadmap, including:

Adding support for excluded documents as well as pinned documents
A UI to easily manage query rules without making API calls
Analyzer support
... and more!

We’re also looking at other ways people may be interested in using query rules. We’d love to hear your feedback on our community page!

Elasticsearch is packed with new features to help you build the best search solutions for your use case. Dive into our sample notebooks to learn more, start a free cloud trial, or try Elastic on your local machine now.