Find out how to configure Elasticsearch SAML authentication in environments using Microsoft Active Directory Federation Services (ADFS).

saml-adfs-blog-banner.jpg

Elasticsearch

Elastic Cloud

Security & compliance

Cloud security

Search

Security

saml-adfs-blog-thumb.jpg

How to configure Elasticsearch SAML authentication with ADFS

<p>Version 6.7 of the Elastic Stack is here, and oh what a release it is. We're not sure if Christmas came early, late, or if all our Christmases came at once!</p>
    <p>In this announcement blog, we cover some of the release highlights. Be sure to dig into the individual announcement posts to dig into all the details. Or even better take the new version for a spin. Version 6.7 is available on our <a href="/cloud/elasticsearch-service">Elasticsearch Service</a> - the only hosted Elasticsearch offering to offer these new features. Or you can <a href="/downloads">download the stack</a> for a self-managed experience in your preferred deployment environment.</p>
    <h3>Elastic Maps: Expanding Mapping Capability for Geo Data in Kibana</h3>
    <p>Geo is an important part of search, and this location-based data powers use cases from ranking neighborhood restaurants, to understanding where the latest marketing campaign has the biggest impact, to hunting down network threats around the globe. Over the years, we have invested heavily in improving our geo capabilities across the stack -- from better storage efficiency and dramatic improvements to query performance in Elasticsearch, to providing more geospatial visualization options in Kibana, to freely hosting basemap and country/region borders with the Elastic Maps Service.</p>
    <p>Keeping in line with this evolution, we are excited to introduce Elastic Maps, a new dedicated solution for mapping, querying, and visualizing geospatial data in Kibana. Elastic Maps greatly expands on existing geospatial visualization options in Kibana with the introduction of features like:</p>
    <ul>
    <li>Visualizing multiple layers and data sources in the same map</li>
    <li>Dynamic data-driven styling on vector layers on maps</li>
    <li>Mapping both aggregate and document-level data</li>
    <li>Visibility control of individual layers (based on zoom level) to control visual clutter</li>
    </ul>
    <p>And like everything else in Kibana, Elastic Maps embeds the query bar with autocomplete for the real-time ad hoc search &amp; query experience that you have come to expect with the Elastic Stack.</p>
    <p><img asset_uid="bltcfa8e336725e2ec4" src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/bltcfa8e336725e2ec4/5c99ef3898d5092a05466c63/gif-maps-stack-pr-med-fidelity.gif" alt="gif-maps-stack-pr-med-fidelity.gif" height="auto" /></p>
    <p>Map all the details in this <a href="/blog/elastic-maps-beta-released">detailed Elastic Maps announcement blog</a>.</p>
    <h3>Elastic Uptime: Actively Monitor Uptime of Services &amp; Application</h3>
    <p>In the last few releases, we have introduced several new features, like autodiscovery for Kubernetes, and the Infrastructure and Logs solutions, to help Elastic users with infrastructure monitoring and observability use cases streamline their operations. We are excited to build on those recent efforts and introduce a new solution, Elastic Uptime, that makes it easy to detect when application services are down or responding slowly, and proactively notifies users about problems even before those services are called by the application.</p>
    <p>Elastic Uptime is based on Heartbeat, a lightweight data shipper for uptime monitoring, that can be deployed both inside and outside an organization's network. All it needs is network access to the desired HTTP, TCP, or ICMP endpoint being monitored. Use cases for Uptime solution include: host availability, service monitoring, website monitoring, and API monitoring.</p>
    <p>Bringing uptime data alongside logs, metrics, and tracing data in Elasticsearch, means that users can more efficiently track and manage all their data in a single operational store.</p>
    <p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt3c129cd22cc557ac/672d627f069a861207817875/animated-gif-kibana-dashboard-canvas.gif" alt="animated-gif-kibana-dashboard-canvas.gif" height="auto" /></p>
    <p>Get all the details on the new Uptime solution in this <a href="/blog/elastic-uptime-monitoring-solution-released">detail post</a>.</p>
    <h2>Elasticsearch</h2>
    <p>6.7 is a big release for Elasticsearch. In addition to launching several new features, <strong>we are excited to graduate several key Elasticsearch features to General Availability (GA) status and marking them production ready.</strong></p>
    <p>As the Elasticsearch post mentions, if it is an Elasticsearch feature with a 3-letter acronym, odds are it's now GA in 6.7.</p>
    <h3>Cross Cluster Replication (CCR) is GA</h3>
    <p>Cross Cluster Replication (CCR), which was introduced as a beta feature in version 6.5, was one of the most heavily requested features for Elasticsearch. CCR has a variety of use cases, including cross-datacenter and cross-region replication, replicating data to get it closer to the application server and user, and maintaining a centralized reporting cluster replicated from a large number of smaller clusters.</p>
    <p>In addition to maturing this feature to GA status, version 6.7 introduces several usability and UI improvements to CCR. Check out the details in the <a href="/blog/elasticsearch-6-7-0-released">Elasticsearch release post</a>.</p>
    <h3>Index Lifecycle Management (ILM) is GA</h3>
    <p>Index lifecycle management (ILM), which was released as a beta feature in Elasticsearch 6.6, is now generally available and ready for production use.</p>
    <p>Handling how Elasticsearch indices are stored and configured as they age is a critical administrative task to optimize cluster performance and cost. ILM helps Elasticsearch admins define and automate those lifecycle management policies, i.e, how data is to be managed and moved between hot, warm, cold, and deletion phases as it ages.</p>
    <p>In addition to graduating index lifecycle management to GA status, version 6.7 also adds new capabilities to this feature. Most notably, users can now add "freeze index" action in the cold phase, significantly reducing the heap needed to store the index. Read about this and other ILM enhancements in the <a href="/blog/elasticsearch-6-7-0-released">Elasticsearch 6.7 detail post</a>.</p>
    <h3>Elasticsearch SQL (including JDBC &amp; ODBC Clients) is GA</h3>
    <p><a href="/products/stack/elasticsearch-sql">Elasticsearch SQL</a>, which was introduced in version 6.3, introduced a way for users to interact and query their Elasticsearch data using a very familiar syntax: SQL. The addition of this feature opened up the full-text power of Elasticsearch to many more users. In addition to the SQL query syntax, Elasticsearch SQL functionality also includes the JDBC and ODBC clients, which allows 3rd party tools that support these drivers to connect to Elasticsearch as a backend datastore.</p>
    <p>We are excited to graduate all these features to GA status. Get all the details in the <a href="/blog/elasticsearch-6-7-0-released">Elasticsearch post</a>.</p>
    <p>We are barely skimming the surface of Elasticsearch 6.7 here. There's much more goodness in Elasticsearch 6.7. You can get all the details in the <a href="/blog/elasticsearch-6-7-0-released">Elasticsearch release post</a>.</p>
    <h2>Kibana</h2>
    <h3>Canvas is GA</h3>
    <p><a href="/products/stack/canvas">Canvas</a>, introduced as a beta feature in version 6.5, lets users showcase and present live data from Elasticsearch with pixel-perfect precision. We are excited to mark Canvas GA in version 6.7. Canvas elevates the visual storytelling in Kibana to new heights, opening up your data analysis and insights to broader audiences. It includes full support for Elasticsearch SQL, and just like the JDBC and ODBC clients, it lets Elasticsearch users expand the reach and impact of their data to broader business audiences.</p>
    <h3>Introducing Kibana Localization; first up Simplified Chinese</h3>
    <p>In version 6.7, Kibana introduces its first localization, and is now available in Simplified Chinese. This marks the beginning of a broader Kibana localization effort. In addition to the launch of Simplified Chinese interface, Kibana 6.7 also introduces a new localization framework to provide support for additional languages in the future. This localization framework also gives Elastic community members access to the necessary tooling to add their own custom translations.</p>
    <p>Get details about all Canvas GA, localized Kibana, and other Kibana 6.7 features in the <a href="/blog/kibana-6-7-0-released">detailed Kibana 6.7 announcement post</a>.</p>
    <h2>Beats</h2>
    <h3>Functionbeat is GA</h3>
    <p>Functionbeat is a new kind of Beat that deploys as a function in serverless computing frameworks, and streams cloud infrastructure logs and metrics into Elasticsearch. It was introduced as a beta in version 6.5, and is now graduated to GA status in version 6.7. Functionbeat currently supports the AWS Lambda framework, and can stream data from CloudWatch Logs, SQS, and Kinesis.</p>
    <p>Read about Functionbeat and other Beats 6.7 updates in the <a href="/blog/beats-6-7-0-released">Beats release blog</a></p>
    <h3>Logs &amp; Infrastructure Solutions are now GA</h3>
    <p><a href="/blog/elastic-infrastructure-app-released">Infrastructure</a> and <a href="/blog/elastic-logs-app-released">Logs</a> solutions were both introduced as beta features in version 6.5. We are excited to graduate them to GA status.</p>
    <p>The <a href="/solutions/logging">Logs solution</a> provides users with real-time log tailing in a compact, customizable display. It's similar to tailing a file, but with the ability to see the logs from all your infrastructure in a single streaming view. And with an embedded search bar powered by Elasticsearch, users can easily narrow the streaming view to just the logs they are looking for.</p>
    <p>The <a href="/solutions/metrics">Infrastructure solution</a> gives users a bird's eye view of the health of all the components - servers, Kubernetes pods, Docker containers - in their infrastructure, making it easier to diagnose problems using log and metrics data. Building on the autodetect capabilities of Metricbeat, the tailored user interface allows you to interactively view and drill into the logs, metrics, and APM traces with a single click.</p>
    <h2>Get ready for 7.0 with the Upgrade Assistant</h2>
    <p>7.0.0 is coming soon (you can check out the <a href="/blog/elastic-stack-7-0-0-beta1-released">beta here</a>). The Upgrade Assistant in 6.7 is here to help you prepare your existing Elastic Stack environment for upgrade to 7.0. The Upgrade Assistant, which includes both APIs and UIs, is an important cluster checkup tool to help plan your upgrade, and identify things like deprecation warnings, indices that need to be upgraded or reindexed, and much more to enable a smoother upgrade experience.</p>
    <h2>Try it now</h2>
    <p>Deploy a cluster on our <a href="/cloud/elasticsearch-service/signup">Elasticsearch Service</a> or <a href="/downloads">download the stack</a> to take these latest features for a spin.</p>

elastic-stack-blog-banner.png

elastic-stack-blog-thumb.png

Elastic Stack 6.7.0 released

<p>The Elastic Stack can be easily leveraged to monitor Go applications. It allows to do things like analyzing memory usage (memory leaks anyone?), performing long-term monitoring, tuning and capturing diagnostics. Beats in particular, the lightweight data shippers in the Stack, are designed to sit alongside the applications and are a natural fit for this kind of monitoring.</p><p>Metricbeat is a Beat specialized in shipping service and/or server metrics, but also happens to be written in Go. It ships in a relatively small package (only about 10MB), and does not bring any additional dependencies with it. While its CPU overhead and memory footprint are also very light, it ships with modules for a variety of services such as:</p><ul><li>Apache</li><li>Couchbase</li><li>Docker</li><li>HAProxy</li><li>Kafka</li><li>MongoDB</li><li>MySQL</li><li>Nginx</li><li>PostgreSQL</li><li>Prometheus</li><li>Redis</li><li>System</li><li>ZooKeeper</li></ul><p>If the service you’re looking for is not listed, don’t worry: Metricbeat is extensible and <a href="https://elastic.ac.cn/guide/en/beats/metricbeat/current/creating-metricsets.html">you can easily implement a module</a> (and this post is proof of that!). We’d like to introduce you to the Golang Module for Metricbeat. It has merged into the master branch of <a href="https://github.com/elastic/beats/tree/master/metricbeat/module/golang">elastic/beats</a>, and is expected to be released in version 6.0.</p><h2>Sneak preview</h2><p>Here’s a Kibana Dashboard that captures the potential of the Golang Module for Metricbeat:</p><p><img asset_uid="blt4e97dc508fe57d70" src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt4e97dc508fe57d70/5bbf10620e6edaf014d240f0/kibana-dashboard-golang.jpg" alt="kibana-dashboard-golang.jpg" height="auto"/></p><p>The top panel shows a summary of the heap usage, which gives us a general idea of <wbr><wbr>Go's memory usage and GC status:</p><ul><li>System Total Memory: the total number of bytes obtained from the system</li><li>Bytes allocated: overall bytes allocated, including memory that has since been freed</li><li>GC cycles: the number of garbage collections (GC) that occurred</li><li>GC limit: when heap memory allocation reaches this limit, the garbage collector is started. This can be different in each GC</li></ul><p>The middle panel has three charts in it with a breakdown of:</p><ul><li>heap memory</li><li>system memory</li><li>object statistics</li></ul><p>Heap Allocated represents both memory in use and not yet reclaimed, while Heap Inuse is obviously the size of objects that are active. Heap Idle accounts for objects that have been allocated but can be reclaimed as free memory.</p><p>The two charts in the bottom panel are the GC time and GC count statistics. CPU Fraction represents the percentage of CPU time spent on GC. The greater the value, the more frequently GC occurs, in other words more time wasted on GC. The trend seems upward and pretty steep, but the range of values is between 0.41% and 0.52% so not too worrisome. Normally the GC ratio warns an optimization in the code when it goes into the single digits.</p><h3>Memory leaks</h3><p>With this information we will be able to know in much detail about Go's memory usage, distribution and GC implementation. For instance if we wanted to analyze whether there is a memory leak, we could check if the memory usage and heap memory allocation are somewhat stable. If for example GC Limit and Byte Allocation are clearly rising, it could be due to a memory leak.</p><p>Historical information gives us great granularity in analyzing memory usage and GC patterns across different versions, or even commits!</p><h2>Great, now how do I get it?</h2><h3>expvar</h3><p>First things first, we need to enable Go's <a href="https://golang.ac.cn/pkg/expvar">expvar service</a>. expvar is a package in Go's standard library that exposes internal variables or statistics. Its usage is very simple, it's basically just a matter of importing the package in the application. It will automatically detect and register to an existing HTTP server:</p><pre class="prettyprint prettyprinted"></pre><p>If no HTTP server is available, the code below allows us to start one on port 6060:</p><pre class="prettyprint prettyprinted"></pre><p>The path registered by default is /debug/vars, we can access it at https://127.0.0.1:6060/debug/vars. It will expose data in JSON format, by default provides Go's <a href="https://golang.ac.cn/pkg/runtime/#MemStats">runtime.Memstats</a> but of course we can also register our own variables.</p><h3>Go Metricbeat!</h3><p>Now that we have an application with expvar, we can use Metricbeat to get this information into Elasticsearch. The installation of Metricbeat is very simple, it's just a matter of <a href="https://elastic.ac.cn/downloads/beats/metricbeat">downloading a package</a>. Before starting Metricbeat we just need modify the configuration file (metricbeat.yml) to enable the golang module:</p><pre class="prettyprint prettyprinted"></pre><p>The above configuration enables the Go monitoring module to poll for data every 10 seconds from heap.path. The other info that matters in the configuration file is the output:</p><pre class="prettyprint prettyprinted"></pre><p>Now assuming Elasticsearch is already running, we can finally start Metricbeat:</p><pre class="prettyprint prettyprinted"></pre><p>Now we are in business! Elasticsearch should have data, we can now start Kibana and customize the visualization for our needs. For this type of analysis <a href="https://elastic.ac.cn/guide/en/kibana/current/timelion.html">Timelion</a> is a particulary good fit, and to get started quickly we can leverage the existing sample <a href="https://elastic.ac.cn/guide/en/beats/metricbeat/current/metricbeat-sample-dashboards.html">Kibana Dashobards</a> for Metricbeat.</p><h3>More than memory</h3><p>In addition to monitoring the existing memory information, through expvar we can expose some additional internal information. For example we could do something like:</p><pre class="prettyprint prettyprinted"></pre><p>It's also possible to expose Metricbeat's internal stats, so it can basically can monitor itself. It can be done via the -httpprof option:</p><pre class="prettyprint prettyprinted"></pre><p>Now we can navigate to https://127.0.0.1:6060/debug/vars and see statistics about the Elasticsearch output such as output.elasticsearch.events.acked, which represents the message sent to Elasticsearch for which Metricbeat received an ACK:</p><pre class="prettyprint prettyprinted"></pre><p>We Metricbeat exposing its own metrics, we can modify its configuration to use both sets of metrics. We can do so by adding a new expvar type:</p><pre class="prettyprint prettyprinted"></pre><p>As you can see we also used the namespace parameter and set it to metricbeat. We can now restart Metricbeat and we should start seeing the new metric.</p><h4>Timelion</h4><p>We can take the output.elasticsearch.events.acked and output.elasticsearch.events.not_acked fields and use a simple Timelion expression to plot successes and failures in messages from Metricbeat to Elasticsearch:</p><pre class="prettyprint prettyprinted"></pre><p>Here's the result in Kibana:</p><p><img asset_uid="bltb97507142fea6f86" src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/bltb97507142fea6f86/5bbf106541d9d6bb1426368c/graph-golang.jpg" alt="graph-golang.jpg" height="auto"/></p><p>From the chart the channel between Metricbeat and Elasticsearch appears to be stable and no messages were lost.</p><p>Finally, we can also compare the Metricbeat memory stats around the same time on the dashboard:</p><p><img asset_uid="bltbd45fb9460b34183" src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/bltbd45fb9460b34183/5bbf106a0e6edaf014d240f6/graph-golang2.jpg" alt="graph-golang2.jpg" height="auto"/></p><h2>Coming up in Beats 6</h2><p>This module will be released with Beats 6.0, but you can start using it right now by cloning (or forking ;) the <a href="https://github.com/elastic/beats">Beats repo</a> on GitHub and building the binary yourself.</p><p><em>Banner image credit: </em><a href="https://golang.ac.cn/doc/gopher"><em>golang.org</em></a></p>

banner-five-years-animals-pig-brain-machine.jpg

thumb-five-years-animals-pig-brain-machine.jpg

Tuning Go Apps with Metricbeat

After almost a year of substantial effort, including 5 Alphas, 1 Beta, and 1 Release Candidate we are pleased to announce Elastic Stack 5.0.0 GA.

<p>In February of 2016, following Elastic{ON} 16, I wrote a post titled <a href="https://elastic.ac.cn/blog/heya-elastic-stack-and-x-pack"><em>Heya, Elastic Stack and X-Pack</em></a>. Today, after almost a year of substantial effort, including 5 Alphas, 1 Beta, and 1 Release Candidate we are pleased to announce the GA release of the Elastic Stack.<br /></p>
<p>And, importantly, it is available – today – on <a href="https://elastic.ac.cn/cloud/as-a-service/signup">Elastic Cloud</a>. If you want hosted Elasticsearch and Kibana there is no other place to start with the most recent code. We are committed to making Elastic Cloud the best place to run hosted Elasticsearch. In fact, we even made the Release Candidate available on cloud for testing purposes.</p>
<p>Our team is celebrating today. I hope you join us.</p>
<p>The GA release is available today. Join the Elastic Team for a live&nbsp;virtual event on November 3 to learn more about the release and ask the creators questions (AMA style). <a href="https://elastic.ac.cn/live/v5">Register now!</a></p>
<div class="video embed-container" style="height: 319.725px;"><img class="vidyard-player-embed" src="https://play.vidyard.com/tVWVLqbvAweDiKrXJBVdDh.jpg" data-uuid="tVWVLqbvAweDiKrXJBVdDh" data-v="4" data-type="inline" style="width: 100%;margin: auto;display: block;" width="100%" /></div>
<p><br /></p>
<p>Before exploring the release in detail, I want to take the opportunity to reflect on what has brought us to this point.</p>
<h2>Our Community</h2>
<p>During the recent Elastic{ON} Tour, I have begun each session discussing a brief history of the last several years. This session culminates in the announcement that we have reached a combined 75 Million downloads. When I first began the project, I hoped for widespread adoption. But the passion and fervor of our community continues to delight and amaze me.</p>
<h2>Pioneer Program</h2>
<p>With that in mind, I want to share the results of the <a href="https://elastic.ac.cn/blog/elastic-pioneer-program">Pioneer Program</a>. The program began with a simple premise. Your usage of the Elastic Stack is of the utmost importance in informing our development as well as ensuring we release the highest quality product available. I am pleased to say that the community has filed 146 issues since the first Alpha release in April.</p>
<p>Our community is one of our most valued assets at Elastic. In fact, one of the most discussed changes in this release was the name “Elastic Stack”.</p>
<h2>The Elastic Stack</h2>
<div class="row">
  <div class="col-lg-12 col-md-12 col-sm-12 col-xs-12 product-type-wrapper">
    <div class="row">
      <div class="col-lg-8 col-md-8 col-sm-8 col-xs-12">
        <p>Last year, we brought the Packetbeat team on board, and Beats was born. This open source platform for building lightweight data shippers for log files, infrastructure metrics, network packets, and more, made it easier than ever to send data to Elasticsearch and Logstash. While we love how many of you who have adopted the abbreviation ELK for our stack, with Beats, we just couldn’t figure out how to make the “B” work with the E-L-K combination.</p>
      </div>
      <div class="col-lg-4 col-md-4 col-sm-4 col-xs-12"><img asset_uid="blt40b62c19d76e25d5" src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt40b62c19d76e25d5/5bbc9f43eb7e90a514e6e01d/elastic-elk-b.png" alt="ELK Bee" width="153" height="88" style="max-width: 100%;width: 153;height: 88;margin: 0px 0px 20px 20px;" /></div>
    </div>
  </div>
</div>
<p>But Elastic Stack is more than just a name. When we began this release cycle we committed to developing, building, testing, and releasing the entirety of the Stack together. This is important, internally, to ensure compatibility. And, for you, it helps speed deployment, decrease version confusion, and make it easier for developers to add capabilities across the entirety of the Elastic Stack.</p>
<h2>A Feature Tour</h2>
<p>When I began this post, I intended to provide an overview of key features in each product. But, it was hard to know where to begin and where to stop. Each of our team and tech leads have created a post that discusses the features specific to their product. And there is no one better suited to tell the story than them. I am, particularly, excited about a few items but rather than enumerate in detail, I will provide a brief overview and encourage you to read the detail posts for each product.</p>
<div class="video embed-container" style="height: 319.725px;"><img class="vidyard-player-embed" src="https://play.vidyard.com/NqTNwGZSqRbAmPTCVQisef.jpg" data-uuid="NqTNwGZSqRbAmPTCVQisef" data-v="4" data-type="inline" style="width: 100%;margin: auto;display: block;" width="100%" /></div>
<ul>
  <li><strong>Ingest Node</strong> - Ingest Node is an Elasticsearch node type enabling some data enrichment capabilities like grok, geoip, date, and other basic event manipulation options at index (or re-index) time. Pipelines are constructed with processors, and accessed through the REST API by suffixing a query parameter “?pipeline=x”. The ability to add pre-processing to documents, natively in Elasticsearch, prior to indexing allows for a variety of creative ingest deployments. This doesn’t replace Logstash. This doesn’t remove the need for Beats, this just allows greater flexibility in designing your ingest architecture.</li>
  <li><strong>Elasticsearch Performance</strong> - Benchmarks tend to have an agenda…especially competitive benchmarks. With that in mind, we have spent substantial effort comparing 5.0.0 to prior releases. This data is available to you. This data is what we inspect when we want to ensure that we are doing the right things with performance and we are doing so in public to work towards preventing the secrecy, and doubt, that are associated with benchmark numbers. In fact, not only are the results available but we also document our hardware configuration, we have open sourced the tooling (called <a href="https://github.com/elastic/rally">Rally</a>) and the benchmarks themselves (<a href="https://github.com/elastic/rally-tracks">Rally-Tracks</a>).</li>
  <li><strong>Metricbeat</strong> - Metricbeat replaces Topbeat as the primary tool for collecting metrics in the Elastic stack. Like Topbeat, Metricbeat collects “top” like statistics about host and per process resources (CPU, memory, disk, network). Unlike Topbeat, Metricbeat also collects metrics from systems such as Apache, HAProxy, MongoDB, MySQL, Nginx, PostgreSQL, Redis, or Zookeeper, with more to come in the near future.</li>
  <li><strong>Logstash Monitoring APIs</strong> - A new monitoring feature provides runtime visibility into the Logstash pipeline and its plugins. This component collects various kinds of operational metrics while Logstash processes your data, and all of this information can be queried using simple APIs.</li>
  <li><strong>Timelion</strong> - After being introduced as a {Re}search project, Timelion is now natively available in Kibana core. Timelion provides a query DSL and visualizations that let you explore your data over time.</li>
</ul>
<p>This is but a sample, I’ve left out BKD trees, scaled_float and half_float, the immense effort put into <a href="https://elastic.ac.cn/guide/en/elasticsearch/resiliency/current/index.html">Elasticsearch Resiliency</a>, the eye-meltingly beautiful redesign of Kibana (we never knew how much we hated borders until we removed them), Kafka output in Beats, and so much more.</p>
<p>This is a massive release. Reading the individual posts is a must to begin to understand the scope of improvement.</p>
<ul>
  <li><a href="https://elastic.ac.cn/blog/elasticsearch-5-0-0-released">Elasticsearch</a></li>
  <li><a href="https://elastic.ac.cn/blog/kibana-5-0-0-released">Kibana</a></li>
  <li><a href="https://elastic.ac.cn/blog/beats-5-0-0-released">Beats</a></li>
  <li><a href="https://elastic.ac.cn/blog/logstash-5-0-0-released">Logstash</a></li>
  <li><a href="https://elastic.ac.cn/blog/es-hadoop-5-0-0-released">ES-Hadoop</a></li>
  <li><a href="https://elastic.ac.cn/blog/x-pack-5-0-0-released">X-Pack</a></li>
</ul>
<h2>X-Pack</h2>
<p>At Elastic we loved extensions. So much so that we built them and gave them interesting names. Shield, Marvel, and Watcher all described individual closed source features that didn’t take away for open source capability but were additive for our customers. Unfortunately, as the range of these features grew to include Graph and Reporting, the install process became difficult and, at times, quite confusing.</p>
<p>Say Heya to X-Pack!</p>
<p>One pack that adds security, alerting, monitoring &amp; management, reporting, and graph capabilities to the Elastic Stack. Our engineering process for 5.0 wasn’t limited to the Elastic Stack, but we’ve also extended X-Pack by adding:</p>
<ul>
  <li>Management &amp; Monitoring UIs to Kibana</li>
  <li>Security UIs to Kibana for creating both users and roles</li>
  <li>Greatly simplified the installation process</li>
</ul>
<p>X-Pack is available to trial and has both commercial and free (Basic) license options. We are particularly excited to make some X-Pack features available for free and details are available on our <a href="https://elastic.ac.cn/subscriptions">Subscriptions</a> page.</p>
<h2>In Closing</h2>
<p>I am in awe of the effort that went into this release, the involvement from our community and customers, and the groundwork that this sets for future releases. As always, the best way to understand a release is to experience it.</p>
<ul>
  <li><a href="https://elastic.ac.cn/downloads/elasticsearch">Elasticsearch 5.0.0 download</a></li>
  <li><a href="https://elastic.ac.cn/downloads/kibana">Kibana 5.0.0 download</a></li>
  <li><a href="https://elastic.ac.cn/downloads/x-pack">X-Pack 5.0.0 install instructions</a></li>
  <li><a href="https://elastic.ac.cn/downloads/logstash">Logstash 5.0.0 download</a></li>
  <li><a href="https://elastic.ac.cn/downloads/beats">Beats 5.0.0 download</a></li>
  <li><a href="https://elastic.ac.cn/downloads/hadoop">ES-Hadoop 5.0.0 download</a></li>
</ul>

blog-es-banner.jpg

blog-es-thumb.jpg

Elastic Stack 5.0.0 Released

<p>It is with a profound sense of pride that we announce the official release of Kibana 5.0.0.</p><p>In the lead up to this moment, we've compared Kibana to sliced bread, repurposed 14th-century poetry, and have even drawn parallels between our pre-release process and the First World War. Was this all over the top? Definitely. Was some of it in poor taste? Probably.</p><p>But it was done with love, and we're positively thrilled to finally deliver it to you all today.</p><p>Kibana 5.0.0 requires Elasticsearch 5.0.0, and you can try it out on <a href="/cloud">Elastic Cloud</a> today.</p><p>Head on over to the <a href="/downloads/kibana">download page</a> to check out the release, read up on the <a href="https://elastic.ac.cn/guide/en/kibana/5.0/release-notes-5.0.0.html">release notes</a>, or feast your eyes on some highlights:</p><h3>A brand new design</h3><p>A great visualization tool deserves a great design, and now Kibana's design is finally up to snuff. The color scheme in general was refreshed from the ground up, and we sought to take advantage of the maximum possible screen real estate without sacrificing the accessibility of the UI.</p><p>Oh, and we got rid of those stupid borders. You know the ones we're talking about. Seriously, who puts borders on widgets in a dashboard? Not us… anymore.</p><p><img src="https://api.contentstack.io/v2/assets/5807e82f8331b7c15e4c2ea3/download?uid=blt2d6c527d2050e9dd" alt="Kibana dashboard"/></p><h3>Time series data, meet Timelion</h3><p>After months being relegated to {re}search status, Timelion is now a part of Kibana core.</p><p>For those unfamiliar, Timelion is a visualization tool with a query DSL that lets you ask interesting questions over time:</p><ul><li>How many pages does each unique user hit over time?</li><li>What’s the difference between this Friday and last Friday?</li><li>What is the cumulative sum of all searches made in the last 2 years?</li></ul><p>These are the types of questions that Timelion was made for. And did I mention that the charts themselves are beautiful?</p><p><img src="https://api.contentstack.io/v2/assets/5807e55d8331b7c15e4c2ae7/download?uid=blt3b27b68b38f75067" alt="Timelion dashboard"/></p><h3>Console, the best way to build custom Elasticsearch queries</h3><p>Sense is now Console, and it now ships with Kibana core.</p><p>Console is like cURL if cURL came with out of the box request/response formatting, autocompleted Elasticsearch API syntax, and remembered your previous requests. In other words, Console is nothing like cURL, and that was a terrible analogy.</p><p>Console uses the same configuration details as Kibana, so make your free-form requests to Elasticsearch without worrying about custom headers or the like.</p><p><img src="https://api.contentstack.io/v2/assets/5810b14f7e0434354491eebf/download?uid=blt3e85dcb3ff739ece" alt="Console in Dev Tools"/></p><h3>Painless scripted fields</h3><p>You can now choose the language of your scripted fields. In addition to the existing Lucene expression support, you can choose any scripting language that is configured in your Elasticsearch cluster. This means you can even use the brand new Painless scripting language that ships with Elasticsearch 5.0. Painless works a lot like Groovy, but we’ve put extra care into making it more secure.</p><h3>X-Pack</h3><p>Want out of the box monitoring for your Elasticsearch and Kibana nodes? How about first-class authentication and security controls or the ability to create PDF reports of your Kibana visualizations? You can try all of these things in Kibana 5.0 with a single CLI command.</p><p>Check out the dedicated <a href="/blog/x-pack-5-0-0-released">X-Pack post</a> for more details.</p><h3>Upgrade from Kibana 4</h3><p>Assuming you're not relying on deprecated Elasticsearch functionality, your searches, visualizations, and dashboards from Kibana 4.6 should continue to work in Kibana 5.0. Just <a href="https://elastic.ac.cn/guide/en/kibana/5.0/upgrade-standard.html">upgrade</a> and go.</p><p>Stuck on Kibana 4.1? No problem! We have a <a href="https://elastic.ac.cn/guide/en/kibana/5.0/upgrade-standard-reindex.html">4.1 upgrade path</a> for you as well.</p><h3>Previous posts</h3><p>If you're so inclined, peruse the blog posts for the various pre-releases to check out even more of the features in Kibana 5.0.0:</p><ul><li><a href="/blog/kibana-5-0-0-rc1">Kibana 5.0.0-rc1</a></li><li><a href="/blog/kibana-5-0-0-beta1">Kibana 5.0.0-beta1</a></li><li><a href="/blog/elastic-stack-release-5-0-0-alpha-5">Kibana 5.0.0-alpha5</a></li><li><a href="/blog/kibana-5-0-0-alpha4">Kibana 5.0.0-alpha4</a></li><li><a href="/blog/kibana-5-0-0-alpha3">Kibana 5.0.0-alpha3</a></li><li><a href="/blog/kibana-5-0-0-alpha2">Kibana 5.0.0-alpha2</a></li><li><a href="/blog/kibana-5-0-0-alpha1">Kibana 5.0.0-alpha1</a></li></ul><h3>Thanks for all of the help!</h3><p>We didn't make Kibana 5.0 happen all on our own. Our endless thanks to all of those that tried out the pre-releases and submitted bug reports, pull requests, and excellent feedback to help make this the best Kibana release to date.</p><p>Now what are you waiting for? Head to the <a href="/downloads/kibana">download page</a> and start using Kibana 5.0.0 today!</p>

blog-kibana-banner.jpg

blog-kibana-thumb.png

Kibana 5.0.0 released

<p>Alongside the 5.0 release of the Elastic Stack, we are excited to introduce the world to X-Pack — a single extension that provides security, alerting, monitoring, reporting and graph capabilities across the Elastic Stack. X-Pack is the evolution of - and replacement for - the standalone plugins Shield, Watcher, Marvel, Reporting, and Graph. While we loved these individual plugins, we were running out of clever names and were beginning to struggle with an increasingly complex, 15+ step installation and configuration process.</p>
<p>As the name literally says, X-Pack is a “pack,” and it is the first of its kind. A pack is a simple, but important concept - it is a single zip that contains extensions for one or more products in the Elastic Stack. And thanks to our aligned version numbers and release train, it’s now easy to build and test extensions that bring UI components to Kibana, new APIs to Elasticsearch, and so much more. We hope you go forth and build interesting packs of your own, but before you do, there’s a lot more to know about X-Pack!</p>
<h2>Installation &amp; Configuration</h2>
<p>We spent a lot of time thinking about ways we could make it easier to install and configure. The install process is now just two commands, and you are ready to get started with the full range of X-Pack functionality.</p><pre class="prettyprint prettyprinted">bin/elasticsearch-plugin install x-pack<br />bin/kibana-plugin install x-pack</pre>
<p>As part of the installation process, we automatically create two native users - elastic, an admin account, and kibana which is a service account used by the Kibana backend. These users are created with a default password of changeme, which the Kibana backend will use by default. This means that there is absolutely no configuration necessary when you’re just getting started.</p>
<p>Of course, before you go into production, you will need to change the default passwords, and configure SSL, but even that is now easier and more consistent across the stack.</p>
<h2>Security</h2>
<p>Creating and managing security in the Elastic Stack just got a whole lot easier. X-Pack builds on the capabilities introduced in Shield, which include authentication, role-based access control, encrypted communication, audit logging, and login and session support for Kibana.</p>
<p>Newly added with X-Pack 5.0 is a management UI in Kibana for creating and managing both users and roles:</p>
<p><img asset_uid="blt021c581a68f46748" src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt021c581a68f46748/5bbdf1d57498f42b0c1a6716/blog-x-pack-1.jpg" alt="blog-x-pack-1.jpg" height="auto" /></p>

<p><a href="/products/x-pack/security">Learn More</a></p>
<h2>Alerting</h2>
<p>X-Pack alerting features build on the capabilities of Watcher, as a highly-available alerting engine that runs inside Elasticsearch and is configured via APIs.</p>
<p>Creating an alert involves specifying 4 simple parameters - a schedule, query, condition, and one or more notification actions, such as email, Slack, HipChat, PagerDuty, or a webhook.</p>
<p>X-Pack 5.0 adds a number of new options, including the ability to specify a condition per-action, which makes it easier to send different types of notifications at different thresholds. For example, if application response times exceed SLAs for 1 minute, use a webhook to create a ticket for the ops team to look into tomorrow. If the response times exceed SLAs for 30 minutes, it’s time to page someone.</p>
<p>Fun fact - the API still uses the term Watcher, out of respect for the many use-cases beyond alerting that it enables.</p>
<p><a href="/products/x-pack/alerting">Learn More</a></p>
<h2>Monitoring</h2>
<p>X-Pack has a goal of providing monitoring capabilities for the entire Elastic Stack. With Marvel, we introduced the most effective monitoring tool for Elasticsearch, and X-Pack 5.0 expands this to include monitoring for Kibana:</p>
<p><img asset_uid="blt435a2e8062275cba" src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt435a2e8062275cba/5bbdf1d8eb7e90a514e6e33f/blog-x-pack-2.png" alt="blog-x-pack-2.png" height="auto" /></p>
<p><a href="/products/x-pack/monitoring">Learn More</a></p>
<h2>Reporting</h2>
<p>X-Pack makes it easy to create and share PDFs of Kibana visualizations and dashboards. Combine reporting with alerting capabilities to send periodic screenshots of dashboards to users that don’t have direct access to Kibana, or attach supporting information to the notification emails triggered by X-Pack alerts.</p>
<p><img asset_uid="blt8d3431499eebc771" src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt8d3431499eebc771/5bbeda56eb7e90a514e6e3fd/blog-xpack-print.jpg" alt="blog-xpack-print.jpg" height="auto" /></p>

<h2>Graph</h2>
<p>X-Pack provides a new way to explore your data with the graph API and UI. Rather than summarizing, slicing, and dicing the properties of your documents, Graph lets you ask questions in terms of the entities (the machines, services, people, bands, etc) and how they are related to one another. In X-Pack 5.0, you can now save and share graph workspaces, and it’s even easier to drill down and see the raw documents that support a given relationship, or link directly to Kibana dashboards or 3rd party systems.</p>
<p><img asset_uid="blt32f1e816c9ae139d" src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt32f1e816c9ae139d/5bbdf1df117bd0dc7fcc611a/blog-x-pack-4.png" alt="blog-x-pack-4.png" height="auto" /></p>

<h2>Conclusion</h2>
<p>Now that you know what X-Pack is all about, go give it a try! X-Pack features are included in our <a href="/subscriptions">subscriptions</a>, but we are excited to make the X-Pack monitoring features available for free with a <a href="/subscriptions">Basic license</a>. Feel free to reach out to us via the contact button in the top-right, or start a discussion on discuss.elastic.co!</p>

blog-x-pack-banner.jpg

blog-x-pack-thumb.jpg

X-Pack 5.0.0 Released

<h2>Kibana UI Update April 15</h2>
<p>7.0 is out! You can find the blog&nbsp;with the information <a href="https://elastic.ac.cn/blog/kibana-7-0-0-released">here</a>. Meanwhile, the Kibana&nbsp;team is already busy working towards the future releases.<span></span></p>
<h3>And we're still hiring!</h3>
<ul>
  <li><a href="https://elastic.ac.cn/about/careers/product-management/jobs/1571633">Director / Sr Director Product Management - Kibana</a></li>
  <li><a href="https://elastic.ac.cn/about/careers/kibana/jobs/1533855">Kibana - UI/Front End Team Lead</a></li>
  <li><a href="https://elastic.ac.cn/about/careers/kibana/jobs/1292795">Kibana Platform - Principal JavaScript Engineer</a></li>
  <li><a href="https://elastic.ac.cn/about/careers/kibana/jobs/1143849">Kibana - Senior JavaScript Engineer</a></li>
  <li><a href="https://elastic.ac.cn/about/careers/kibana/jobs/1292707">Kibana - Senior Security Developer (JavaScript)</a></li>
  <li><a href="https://elastic.ac.cn/about/careers/kibana/jobs/1292795">Kibana - Platform Javascript Engineer (Node.js)</a></li>
  <li><a href="https://elastic.ac.cn/about/careers/kibana/jobs/1595759">Kibana - Software Engineer - Operations</a></li>
  <li><a href="https://elastic.ac.cn/about/careers/kibana/jobs/1418186">Kibana - Visualisations &amp; Vega Engineer</a></li>
</ul>
<h3>Security</h3>
<p>Feature Control</p>
<p>The feature controls PR has been merged to master and 7.x. For an overview of what this provides, this <a href="https://github.com/elastic/kibana/issues/20277">issue</a> is the best source of information while we actively work on the end-user and developer facing documentation. There are a few minor enhancements we’re actively working on, including the addition of “read only badges” to further alert the user that they’re looking at a read-only view of an application.</p>
<p><img src="https://images.contentstack.io/v3/assets/bltf7afce26b89a5b33/blt6d929aa1ce91599a/5cb4ea0a791abe5a29a1d22c/download" data-sys-asset-uid="blt6d929aa1ce91599a" alt="image14.gif" /></p>
<p><br /></p>
<p>Secret Service:&nbsp;Aleh has been making great progress on the secret service, and a draft PR is available <a href="https://github.com/elastic/kibana/pull/34526">here</a>. Some work is still left to be done, and tests are still a work in progress.</p>
<h3>Platform</h3>
<p>New Platform:&nbsp;We are preparing infrastructure for plugins (Security is the first in the list) &nbsp;to extend HTTP service with custom functionality.</p>
<h3>Operation</h3>
<p>Generate dockerfiles target to be used for elastic/dockerfiles <a href="https://github.com/elastic/kibana/pull/32169">#32169</a></p>
<h3>Geo-Maps App</h3>
<p>Did you miss ThomasN &amp; AlexF’s webinar on Elastic Maps last week? Be sure to catch the recording @ <a href="https://elastic.ac.cn/webinars/elastic-maps-for-geospatial-analysis">Elastic Maps for Geospatial Analysis</a>. We provide a good overview of the basic functionality the Maps app offers, build a <a href="https://media.giphy.com/media/2CcKiHPDsJqVi/giphy.gif">pew pew</a> map out of ECS Suricata events and dive into some future features with NYC taxi collision data. Already saw the webinar and want to play around with the data set and map? It’s all readily available in <a href="https://github.com/alexfrancoeur/elastic_maps_examples/tree/master/elastic_maps_webinar">AlexF’s example repo</a>.</p>
<h3>
  <p><img src="https://images.contentstack.io/v3/assets/bltf7afce26b89a5b33/blt91d30a28949d149b/5cb4f404791abe5a29a1d266/download" data-sys-asset-uid="blt91d30a28949d149b" alt="image15.png" /></p>
</h3>
<h3>Kibana App (Visualizations, Sharing, Discover, Reporting)<br /></h3>
<h4>Elastic-Charts</h4>
<p>This week we released the functionality to allow line annotation hiding any tooltips to mimic the current time marker feature in vislib.</p>
<p>The legend now shows the last bucket value along with the series name. Hovering over a bar/area/line will display the relative value both on the tooltip and on the legend.</p>
<p><img src="https://images.contentstack.io/v3/assets/bltf7afce26b89a5b33/bltf78ad5f0eb5b34ea/5cb4f03c30313775298d6414/download" data-sys-asset-uid="bltf78ad5f0eb5b34ea" alt="image6.gif" /></p>
<p>Each series can now be sorted using a sortIndex props on each series. We will work to implement sorting also at bucket level, a feature that is currently available on ES but not on vislib.</p>
<p><img src="https://images.contentstack.io/v3/assets/bltf7afce26b89a5b33/blt25ea4f26b7ab9f31/5cb4f04fe06ed87a29f2d98b/download" data-sys-asset-uid="blt25ea4f26b7ab9f31" alt="image12.png" /></p>
<p>Band area charts are now available for areas and bars (it works similar to a candlestick chart but it doesn’t have the same features). We are working to add upper and lower series names to the tooltip and legend.</p>
<p><img src="https://images.contentstack.io/v3/assets/bltf7afce26b89a5b33/bltaa2cb05c9c81297d/5cb4f19bb790c01122391b77/download" data-sys-asset-uid="bltaa2cb05c9c81297d" alt="image16.png" /></p>
<p>Animations are disabled starting from 3.10.1 until we find the right way to morph lines between two dataset of different length.</p>
<h4>
  <h3">Reporting</h3">
</h4>
<p>Work continued on our Export to CSV.&nbsp;We’re in the process of engaging Cloud and other teams so that there’s no surprises when we launch.<br />We also discovered a bug in how puppeteer handles async actions. The solution will hopefully reduce the number of “Timeout waiting for selector…” errors to appear.<br />Lastly, Tim has condensed our internal queue’s querying for jobs from 4 calls down to just one! This means we’ve reduced our footprint and, hopefully, reduced the amount of resources needed on both Kibana and ElasticSearch!&nbsp;</p>
<h3>Canvas</h3>
<h4>Under the bonnet - layout engine integration rework</h4>
<p>We’ve <a href="https://github.com/elastic/kibana/pull/33702">refactored</a> integration code, shed 100s of code lines and Redux lock-in (no thunks!) while accidentally fixing 5 bugs (and maybe adding some). It unblocks new features eg. <a href="https://github.com/elastic/kibana/pull/32995">box select</a>:</p>
<p><img src="https://images.contentstack.io/v3/assets/bltf7afce26b89a5b33/blt3a3f815e084c8e5a/5cb4f464869b38322286ae4c/download" data-sys-asset-uid="blt3a3f815e084c8e5a" alt="image5.gif" /></p>
<p>It’s now possible to statically render workbooks without layout engine code. The React component structure is leaner, only the currently edited page has event handlers. Proceeding with TypeScript type tests and annotations is aided by simpler data and leaner code as well.</p>
<h4>More adventures in dynamic layout land</h4>
<p>Experiments in data driven layouting continue with three new functions (with terrible names). The first is similar to repeatElement, the difference is that this new layoutElements allows you to completely control, via the position function, the height, width, position and rotation of elements generated from a data table.</p>
<p>The next function, positionElement allows you to statically position one or more elements within a container. Of course since this is Canvas, nothing has to be static, and you can use data to control their positioning and styling. This is fun because it allows for reusing a datasource across multiple elements, something we’ve wanted to do for awhile. Here’s a pretty contrived, but fun, example of &nbsp;all this</p>
<p><img src="https://images.contentstack.io/v3/assets/bltf7afce26b89a5b33/blt14d9bfc6d7eaa95f/5cb4f0aae16629da239c3369/download" data-sys-asset-uid="blt14d9bfc6d7eaa95f" alt="image2.gif" /></p>
<p>In this gif the trees are a timeseries of the average speed of airplanes near the Phoenix office, layoutElements is used to determine the height of the trees. This is actually generating a positionElement which contains both an image of a tree, as well as the speed the tree represents, and then determining the height of the container using the data.</p>
<p>The position of the purple planes in the sky is determined by the speed of planes near the Phoenix office and the position of the sun is determined by the seconds of the current minute</p>
<p>The animations are handled by the support for custom CSS that Canvas already has.</p>
<p>If you really want to mess with this stuff, and I’m not suggesting you do, there’s a Kibana plugin that only works on master in <a href="https://github.com/rashidkpc/canvas-dynamic-layout">this github repo</a></p>
<h4><a href="https://github.com/rashidkpc/canvas-dynamic-layout"><br /></a>Filter grouping works!</h4>
<p>And filters work much better too. Joe cleaned up a ton of issues we didn’t know existed in Canvas’s filter functionality, and implemented the ability for filters to exist in groups. That means that time filters no longer have to be global! No filter has to be global! You can tie specific elements, or even specific parts of an expression to one or more filter controls. Here’s a quick screenshot..</p>
<p><img src="https://images.contentstack.io/v3/assets/bltf7afce26b89a5b33/blt3adde9ec019a233f/5cb4f0e0c6bfc2312293ad7a/download" data-sys-asset-uid="blt3adde9ec019a233f" alt="image8.gif" /></p>
<h4>Loading……………… Content.</h4>
<p>Our crack design team has been keeping busy with designing new templates to give Canvas users a F-A-S-T start when creating new Canvases. &nbsp;We’ve been working on two for now, a presentation template and a report template. The report template is aimed at C-Level creators to dynamically update reports with data.</p>
<h3>Design</h3>
<h4>Docs and styling for Elastic Charts</h4>
<p>One of our 7.1 goals is to get some basic documentation and styling for Elastic Charts in EUI. We’re currently working on styling the charts along with some quick snippet code to get people started. The idea here is that the Storybook docs the charts team put together can serve more as a reference, where the EUI examples can serve more as design guidelines and a chart gallery for actual solutions.</p>

<h4>EUI TypeScript conversion</h4>
<p>ne thing we continually hear from people is they want us to prioritize TypeScript conversion (2 out of every 3 devs marked this as important). <a href="https://github.com/elastic/eui/issues/1557">We’ve made good progress during 7.1</a>, even getting PRs from the community outside of Elastic and will continue to chip away at it.</p>
<p><strong>Until&nbsp;next week</strong>&nbsp;<br /><strong>- Kibana Team</strong></p>

blog-thumb-release-kibana.png

kibana-timeseries.jpg

Keeping up with Kibana: This week in Kibana for April 15th, 2019

<p>Lots of good progress on some exciting new features!</p>
<p><span></span></p>
<h3>Hiring</h3>
<p><a href="https://jobs.elastic.co/jobs/kibana/distributed/kibana-visualisations-vega-engineer/1418186">Kibana - Visualisations &amp; Vega Engineer</a></p>
<p><a href="https://jobs.elastic.co/jobs/kibana/anywhere-/kibana-gis-data-engineer/1569367">Kibana - GIS Data Engineer</a></p>
<p><a href="https://jobs.elastic.co/jobs/kibana/distributed/kibana-senior-javascript-engineer/1143849">Kibana - Senior JavaScript Engineer</a></p>
<p><a href="https://jobs.elastic.co/jobs/kibana/distributed-global/kibana-senior-software-engineer-operations/1595759">Kibana - Senior Software Engineer - Operations</a></p>
<p><a href="https://jobs.elastic.co/jobs/kibana/distributed/kibana-platform-javascript-engineer-node-js-/1292795">Kibana - Platform JavaScript Engineer (Node.js)</a></p>
<p><a href="https://jobs.elastic.co/jobs/product-management/distributed-amer-or-western-europe/director-sr-director-product-management-kibana/1571633">Director / Sr Director Product Management - Kibana</a></p>
<h3>Security</h3>
<h4>Feature Controls</h4>
<p>This week for Feature Controls was dedicated to crafting release highlights and a blog post introducing the feature. We expect to have the blog post drafted early next week!</p>
<h4>New Platform Migration</h4>
<p>We previously began work on migrating the spaces plugin to the new platform, and we continued this effort this week. Aleh also began looking into migrating the security plugin’s authentication to the new platform. Once authentication is migrated, we’ll then be focusing on moving over the authorization components of both the spaces and security plugins.</p>
<h4>Copy Saved Objects to Spaces (<a href="https://github.com/elastic/kibana/issues/37286">#37286</a>)</h4>
<p>We started an <a href="https://github.com/elastic/kibana/pull/38014">exploratory PR</a> for the API that will power this feature. After discussions, we decided to work on preparing the underlying infrastructure for this API in a series of smaller PRs to make the work more manageable. Specifically:</p>
<ul>
  <li>Refactoring &amp; simplifying `savedObjectsManagement` UI Capabilities</li>
  <li>Re-order the Saved Objects Client Wrappers that security and spaces implement to allow the spaces plugin to inform the security plugin which space to authorize against, without the security plugin having to derive this information from the request itself.</li>
</ul>
<h3>Platform</h3>
<h4>New Platform Services</h4>
<p>Unblocking Security Integration</p>
<ul>
  <li>Support for route tags has been merged (<a href="https://github.com/elastic/kibana/pull/37344">#37344</a>)</li>
  <li>Exposing session storage to auth providers is in review (<a href="https://github.com/elastic/kibana/pull/37992">#37992</a>)</li>
</ul>
<p>Migrating Saved Objects to NP</p>
<ul>
  <li>Saved Objects conversion to TypeScript has been merged (<a href="https://github.com/elastic/kibana/pull/36829">#36829</a>)</li>
  <li>Work has begun on migrating the SavedObjectClient to the NP (no PR yet)</li>
</ul>
<p>ApplicationService</p>
<ul>
  <li>The Handler Context (<a href="https://github.com/elastic/kibana/pull/36509">#36509</a>) and ApplicationService RFCs (<a href="https://github.com/elastic/kibana/pull/36477">#36477</a>) are in the final comment period</li>
  <li>Removal of the last legacy dependencies for the Header UI is in review (<a href="https://github.com/elastic/kibana/pull/38175">#38175</a>)</li>
  <li>An end-to-end proof-of-concept prototype of frontend routing and integration with the legacy platform has been proven out. Implementation of this will begin once the Header UI is moved to the new platform. We expect this effort to require many PRs to complete.</li>
</ul>
<p>Licensing Plugin</p>
<ul>
  <li>Investigation and planning for this plugin will begin next week.</li>
</ul>
<p>Cleanup</p>
<ul>
  <li>Simplifying the integration between legacy and new platforms on the client has been merged (<a href="https://github.com/elastic/kibana/pull/37734">#37734</a>)</li>
  <li>Moving the Chrome and I18n APIs to the start phase was merged (<a href="https://github.com/elastic/kibana/pull/37915">#37915</a>)</li>
  <li>Making the basePath API on the server consistent with the client is in review (<a href="https://github.com/elastic/kibana/pull/38237">#38237</a>)</li>
</ul>
<h4>Kibana Clustering</h4>
<p>To&nbsp;met with the Code team this week to go over the in-progress proxy implementation (<a href="https://github.com/elastic/kibana/pull/36907">#36907</a>). We’re on the same page and got some good feedback on a few more necessary asks for the initial beta release in 7.3.</p>
<p>Next week, we’re meeting with the Security team to review authorization, authentication, TLS and PKI protection for this new HTTP port. We want to explore getting as close to Elasticsearch’s security model for cluster access while making it a smooth experience for plugin developers wanting to use Kibana Clustering.</p>
<h3>Operations</h3>
<p>During the course of the week we continued our efforts on updating our dependencies merging and auditing the continuous PRs automatically created by renovate. A lot of those are still expected in the next week. We have also merged a patched version of the relative cache loader which has now fixed the windows development bugs introduced when we merged it the first time. Worth it to mention that we also shipped to useful scripts for developers: one to merge the base branch into the prs (which is a super common operation) with a simple script call and the other to generate the renovate config which will make it easier to others update the renovate config. We are now also using babel 7 to compile x-pack as we replaced the typescript compiler by the babel compiler in the @kbn/plugin-helpers package</p>
<p>Prs:</p>
<ul>
  <li>Declare @kbn/babel-preset as peerDependency for @kbn/plugin-helpers <a href="https://github.com/elastic/kibana/pull/37710">#37710</a></li>
  <li>[dev/license_checker][dev/npm] reactor, ts-ify, de-grunt <a href="https://github.com/elastic/kibana/pull/37807">#37807</a></li>
  <li>[dev/failed_tests] try to ignore irrelevant failures <a href="https://github.com/elastic/kibana/pull/37902">#37902</a></li>
  <li>[scripts/update_prs] implement script to merge base branch into prs <a href="https://github.com/elastic/kibana/pull/38302">#38302</a></li>
  <li>[jest] don't be crazy, run with caching enabled <a href="https://github.com/elastic/kibana/pull/38332">#38332</a></li>
  <li>Implement renovate config generator <a href="https://github.com/elastic/kibana/pull/38369">#38369</a></li>
  <li>Fix typo in failure filtering <a href="https://github.com/elastic/kibana/pull/38374">#38374</a></li>
  <li>Relative cache loader <a href="https://github.com/elastic/kibana/pull/36131">#36131</a></li>
</ul>
<h3>Geo-Maps App</h3>

<h4>Enhancements</h4>
<ul>
  <li>[in review] GeoJson upload <a href="https://github.com/elastic/kibana/pull/36410"><span></span></a><a href="https://github.com/elastic/kibana/pull/36410">https://github.com/elastic/kibana/pull/36410</a></li>
  <li>[in review] Symbolize points as icons [in review] <a href="https://github.com/elastic/kibana/pull/37822"><span></span></a><a href="https://github.com/elastic/kibana/pull/37822">https://github.com/elastic/kibana/pull/37822</a></li>
  <li>[in review] add support for Top Hits to Documents source [in review] <a href="https://github.com/elastic/kibana/pull/38052"><span></span></a><a href="https://github.com/elastic/kibana/pull/38052">https://github.com/elastic/kibana/pull/38052</a></li>
</ul>
<h3>Geo-Elastic Maps Service</h3>
<ul>
  <li>Updated EMS landing page with themed tiles</li>
  <li>Updated Dark theme tiles to better align with Kibana dark theming</li>
</ul>
<h3>App Architecture</h3>
<p><a href="https://github.com/elastic/kibana/pull/37510">EmbeddableAPI branch</a> was merged and we are now ready to move dashboard over to the new embeddable API. Dashboard code was typescripted as well.</p>
<p>More interpreter types are being moved from canvas to OSS and<a href="https://github.com/elastic/kibana/pull/37968"> error handling</a> was added to expression runner.</p>
<p>Migration to new platform is moving as planned, a lot of angular was removed and our plan on how to migrate is close to ready.</p>
<h3>Kibana App (Visulizations, Sharing, Discover, Reporting)</h3>
<h4>Lens (New visual editor)</h4>
<p>Began integrating our work this week, getting our first configurable chart with real data:</p>

<p>This screenshot is not based on the feature branch yet, but it shows that our individual work is adding up to something larger.</p>
<p>This week we:</p>
<ul>
  <li>Merged the <a href="https://github.com/elastic/kibana/pull/37391">config panel</a></li>
  <li>Merged <a href="https://github.com/elastic/kibana/pull/37648">expression rendering</a> and <a href="https://github.com/elastic/kibana/pull/37876">xy expression functions</a></li>
  <li>Merged a PR for converting the <a href="https://github.com/elastic/kibana/pull/37967">XY chart state into an expression</a></li>
  <li>Merged a change to pass the <a href="https://github.com/elastic/kibana/pull/38142">datasource API into the visualization</a> for column IDs</li>
  <li>Built out the editor UI for users to <a href="https://github.com/elastic/kibana/pull/37573">build queries</a></li>
  <li>Integrated <a href="https://github.com/elastic/kibana/pull/37711">drag and drop for building queries</a></li>
</ul>
<h4>Elastic Charts</h4>
<p>KQL:</p>
<p>Using KQL with the autocomplete it offers makes it easier to create a query for a visualization without the need to remember the exact field name and it assists with operators and values.</p>
<p>KQL has been added to the Visual Builder using components that update as a query is entered (having immediate effect on the visualization rendered) with the option of using Kuery syntax with autocomplete or Lucene syntax (defaulting to the syntax set in the Kibana advanced settings).</p>
<p>Previously saved visualizations are migrated to accommodate the change. We assume that if the query language is not present, it is Lucene syntax.</p>


<p><br /></p>
<h3>Design + Docs</h3>
<p>Kibana now has EUI 11.0.1 (The icons one!), and EUI 11.3.1 (The color picker one!) is merging soon</p>
<p>You can now utilize custom icons on Kibana using EUI. This helps us so that we no longer need to package the entirety of the Internet’s logos in EUI directly and gives you flexibility when you might need something custom for you application. Please <a href="https://elastic.github.io/eui/#/display/icons">read the docs</a> to learn how this works.</p>
<h4>Data grids</h4>
<p>We are still very very early into our work on data grids. Right now we’re finishing up a feature spec for the component and doing some very rough proof of concepts for the rendering layer.</p>
<p>Check the demo showing <a href="https://codesandbox.io/s/datagrid-div-custom-dom-structure-pyfrb">how we might handle resizing rows and columns</a>.</p>

<p>And then another demo showing <a href="https://codesandbox.io/s/hookedondatagrids-gx1zk">how we might utilize React hooks for subscribing to updates for the style and content of the cells</a>.</p>

<h4>Compressed panels experiments</h4>
<p>One comment we’ve head as we build more and more complicated “panel editors” (think the right side of Canvas and Maps) is that the forms we use for EUI are a little too beefy when jammed into those sidebars. We’ve started some initial experiments trying to compress them down. Likely these won’t involve too many new formal components but more a guideline for how to create slimmer forms utilizing our popover and label append / prepend mechanics. Here’s a screen to show how some of this might be done in the Maps application.</p>

<h4>Starting to think about embeds</h4>
<p>This is a week of taking stabs at early concepts. We’re doing some thinking about app to app embeds and how we can make generic systems (like a button that leads to a flyout) that allows one app to pull saved objects from another.</p>

<h3>QA</h3>
<h4>Cloud Testing</h4>
<ul>
  <li>Analyzing and opening issues on latest test runs</li>
  <li>Issues can be found here and require triage by relevant functional teams:<a href="https://github.com/elastic/kibana/projects/20"> <span></span></a><a href="https://github.com/elastic/kibana/projects/20">https://github.com/elastic/kibana/projects/20</a></li>
</ul>
<h4>Visual Testing</h4>
<ul>
  <li><a href="https://github.com/elastic/kibana/issues/33817">Test plan phase 1</a></li>
  <li>Working on getting visual regression into Kibana FTR &nbsp;&nbsp;</li>
</ul>
<h4>General</h4>
<ul>
  <li>Team continues to test 7.2.0 Release Build candidates.</li>
</ul>
<h3>Stack Services</h3>
<h4>Alerting</h4>
<p>This week good progress has been made on creating a basic alerting plugin, we are finalizing testing before the PR is good to merge (<a href="https://github.com/elastic/kibana/pull/37043">#37043</a>). There are some final adjustments on error handling and validation to be made next week.</p>
<h3>Reporting</h3>
<ul>
  <li>Some “behind-the-scenes” work on Scheduled reports … stay tuned!</li>
  <li>CSV’s with special characters now show a special warning in Kibana as they’re a method of attack injection.</li>
  <li>More work and docs forthcoming for SDH/Support issues. We’ve noticed an uptick in incoming SDH issues that are reporting based (largely due to the new version of Chromium that we’ve bundled).&gt;html here</li>
</ul>

Keeping up with Kibana: This week in Kibana for June 10, 2019

<h4>Welcome to Keeping up with Kibana! This is a series of posts on new developments in the Kibana project and any related learning resources and events.</h4>
<h3>Geo</h3>

  <p>The <a href="https://maps.elastic.co/">Elastic Maps Service landing page</a> is now available! This page shows the data behind the vector layers that can be joined to Elasticsearch data in Kibana region maps.</p>
<p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt556140c6dd5da5d9/5f9b20f667a7c3549260359b/blog-keeping-up-with-kibana-2018-07-09-1.png" data-sys-asset-uid="blt556140c6dd5da5d9" alt="blog-keeping-up-with-kibana-2018-07-09-1.png" /></p>

<h3>PRs</h3>
<ul>
<li>Elastic Maps Service landing page deployment handled by Jenkins jobs (<a href="https://github.com/elastic/ems-landing-page/pull/27">ems-landing-page#27</a>)</li>
</ul>
    

  <h3>Canvas</h3>


  <p>The redesign, new layout engine, and reporting are all on the verge of merge. Also, Verge of Merge is the name of our new band. We're opening for Party Marty, it's gonna be rad. $10 cover. No outside food or drink. After party at your house.</p>
  
  <h4>Automatic Expression Formatting</h4>
  <p>If you've used Canvas at any length, you've probably spent some time formatting the expression so it's easier to read, only to have all that effort thrown out when you changed something from the sidebar. Canvas will now auto-format the expression for you, so even if you smash stuff in there like a clod, Canvas will make it look good!</p>

  
  <h4>Case Insensitive Functions</h4>
  <p>Can't remember if it's “pointseries” or “pointSeries”? Now you don't have to! Functions are now case-insensitive, so it doesn't matter which one you use, they both work. And much like arguments have aliases, functions do too, but now they actually work.</p><br bold="[object Object]"/>
  <p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/bltd603af41eb46a133/5f9b21401a81c1644e9970bf/blog-keeping-up-with-kibana-2018-07-09-3.png" data-sys-asset-uid="bltd603af41eb46a133" alt="blog-keeping-up-with-kibana-2018-07-09-3.png" /></p>

<h4>Other Stuff</h4>

  <p>Canvas also fixed some important bugs. Canvas now correctly uses the selected index pattern in the esdocs UI, elements correctly report when they are done rendering (something Reporting relies on), and errors from AJAX requests are now shown to the user. We also removed an undocumented and unused run API since we're no longer planning for plugin authors to execute expressions that way.</p>

<h3>Platform</h3>
<h4>Localization</h4>

  <p>The <a href="https://github.com/elastic/kibana/pull/20525">first PR</a> has landed to update an entire feature of Kibana to be translatable! We started with the index pattern page in management, and we'll use this particular PR to smooth out any edges in the i18n UI abstractions so that we can roll out i18n support throughout the rest of Kibana without churning on details.</p>
  
  <p>The pace of our localization effort as a whole has taken off as <a href="https://github.com/elastic/kibana/pulls?q=is%3Aopen+is%3Apr+label%3A%3Ai18n">pull requests</a> get opened sometimes more than once a day. We're currently reviewing tools for verifying proper translation coverage in CI, extracting translatable attributes from templates to seed translation files, and providing a json diff of translation changes for new versions, as well as PRs for improved test coverage and developer documentation.</p>
  

<h4>New platform</h4>

  <p>The foundation of the new platform on the server is ready, and now that more than half the team is back from vacation, we can get it merged. After this gets merged, some of the things we'll be focusing on in the server will be the new saved object service, elasticsearch service, and the plugin service.</p>
  
  <p>The new platform in the UI made great progress. In the feature branch, all apps are now being bootstrapped through the new platform, and the chrome and fatal error pages are being reactified and moved to the new platform.</p>

<h3>Security</h3>
<h4>RBAC Phase 1</h4>

  <p>Modifications were made to the deprecation logging when using the legacy fallback, and we are only logging deprecation warnings on user login. Additional integration tests were added for the Elasticsearch APIs that don't have sufficient coverage with our saved object based integration tests. The Elasticsearch PR which makes it possible for us to use the Kibana index name as part of the application name has merged.</p>
  

<h4>Spaces</h4>

  <p>The requested changes to make the saved object client space aware have been made, and it's ready for another review.</p>

<h3>Operation</h3>
<h4>Optimizer</h4>

  <p>We made some progress on came up with possible solutions, not to completely remove the optimizer, but at least to understand how to increase its performance on production while we reduce the on-fly compilation needed. The POC we were running on webpack-dlls allowed us to validate the possibility of generate DLLs for Kibana (even if those dlls have dependencies between them).</p>

<h3>Design</h3>

  <p>The design team continues to finalize fixing the popover bugs and building out the basic new Vis editor prototype. We are trying to catch up on some small bugs/issues and work out some overarching responsive layouts. The Dashboard and Watcher docs have been updated.</p>
  
  <p>EUI version status</p>
  <p><a href="https://elastic.github.io/eui/#/package/changelog" >EUI is now at 1.1.0</a></p>
  
  <ul>
    <li>Kibana master is at EUI 1.1.0!</li>
    <li>Cloud master is at EUI 1.0.1. No breaking changes to upgrade</li>
  </ul>
  
  <p>PRs</p>
  <p><span style="font-weight: normal;"></p>
  <ul>
    <li>Experimental EuiXYChart and associated components have been added! (<a href="https://github.com/elastic/eui/pull/309" >#309</a>)</li>
    <li>Fixed some IE11 flex box bugs (modal overflowing, image shrinking, and flex group wrapping) and documented others. (<a href="https://github.com/elastic/eui/pull/973" >#973</a>)</li>
  </ul>



<h3>Management</h3>
<p>The management team is continuing to work on supporting rollup index patterns in visualization and discover areas. Some PRs to convert Notifier usages to toastNotifications were also merged.</p>

  <p>PRs</p>
  <ul>
    <li>(Rollup support) Refactor SearchSource interface (<a href="https://github.com/elastic/kibana/pull/20334">#20334</a>)</li>
    <li>(Rollup support) Add AggTypeFieldFilters to filter out fields in vis editor (<a href="https://github.com/elastic/kibana/pull/20539">#20539</a>)</li>
    <li>Fix bug in propFilter logic when it's not passed any filters (<a href="https://github.com/elastic/kibana/pull/20569">#20569</a>)</li>
    <li>Convert various uses of notifier to toastNotifications (<a href="https://github.com/elastic/kibana/pull/20420">#20420</a>)</li>
    <li>Replace uses of Notifier with toastNotifications in Watcher (<a href="https://github.com/elastic/kibana/pull/20538">#20538</a>)</li>
  </ul>


<h3>Sharing</h3>

  <p>PRs</p>
  <p>[input controls] update dropdown suggestions when filtered <a href="https://github.com/elastic/kibana/pull/18985">PR 18985</a></p>

<h3>Visualizations</h3>


  <p>The visualizations team is still busy removing Angular from a lot of places in preparation for the using the Canvas pipeline for rendering (half of the PRs seen below).</p>
  <p>The first experimental version of the new chart component in EUI has now been merged into EUI (<a href="https://github.com/elastic/eui/pull/309">eui/#309</a>). You can check its documentation and samples in the <a href="https://elastic.github.io/eui/#/xy-charts-beta/general">EUI documentation</a>.</p>
  
  <p>Vega visualizations now have the possibility to create filters in Kibana using specific functions (<a href="https://github.com/elastic/kibana/pull/17586">#17586</a>).</p>
  


<h3>PRs</h3>


  <ul>
    <li>Reenable VEGA_DEBUG for Vega visualization (<a href="https://github.com/elastic/kibana/pull/20456" >#20456</a>)</li>
    <li>Make aggTypeFilter registry return value directly (<a href="https://github.com/elastic/kibana/pull/20523" >#20523</a>)</li>
    <li>refactoring geohash agg to not use vis (<a href="https://github.com/elastic/kibana/pull/20298" >#20298</a>)</li>
    <li>removing angular from render_complete (<a href="https://github.com/elastic/kibana/pull/20478" >#20478</a>)</li>
    <li>moving visualize legend into vislib legend (<a href="https://github.com/elastic/kibana/pull/20479" >#20479</a>)</li>
    <li>Fix flaky TSVB test (<a href="https://github.com/elastic/kibana/pull/20481" >#20481</a>)</li>
    <li>Fix editor sidebar with long field names (<a href="https://github.com/elastic/kibana/pull/20455" >#20455</a>)</li>
    <li>Inspector enhancements (rename of buttons, slight design adjustments) (<a href="https://github.com/elastic/kibana/pull/20452" >#20452</a>)</li>
    <li>applying scope in angular vis type (<a href="https://github.com/elastic/kibana/pull/20461" >#20461</a>)</li>
    <li>removing angular dependency from base and react vis types (<a href="https://github.com/elastic/kibana/pull/20386" >#20386</a>)</li>
    <li>adding disabled aggs functional tests (<a href="https://github.com/elastic/kibana/pull/20454" >#20454</a>)</li>
    <li>Split editor state from saved state (<a href="https://github.com/elastic/kibana/pull/20323" >#20323</a>)</li>
    <li>Move visualize editor out of visualize directive (<a href="https://github.com/elastic/kibana/pull/20263" >#20263</a>)</li>
    <li>Don't create nested search source per postflightrequest (<a href="https://github.com/elastic/kibana/pull/20373" >#20373</a>)</li>
    <li>bwc for showMeticsAtAllLevels (<a href="https://github.com/elastic/kibana/pull/20369" >#20369</a>)</li>
    <li>[Vega] Implement context filter modification (<a href="https://github.com/elastic/kibana/pull/17586" >#17586</a>)</li>
  </ul>


<h3>Discovery</h3>


<p>PRs</p>
  <ul>
    <li>Remove outdated aria attribute (<a href="https://github.com/elastic/kibana/pull/20532">#20532</a>)</li>
    <li>Use config filters:pinnedByDefault for filters created with filter editor (<a href="https://github.com/elastic/kibana/pull/20359">#20359</a>)</li>
  </ul>


canvas.png

Keeping up with Kibana: This week in Kibana for July 9th, 2018

Winter is here. To celebrate the holiday season we held our inaugural Decemberfest.

<p>Winter is here. Of course, for some, it feels like the whole of 2020 has been one long winter. Still, December means it’s finally time for the holiday season — a chance to celebrate, let our hair down, and ring in a new year full of promise and say goodbye to … whatever 2020 was.</p>
<p>Naturally, almost everyone is doing things a bit different this winter season, and Elastic is no exception. Although we’re <a href="/blog/culture-elastic-distributed-by-design?baymax=web&elektra=culture-life-at-elastic-decemberfest-for-the-holidays" target="_self">distributed by design</a>, many of our Elasticians occasionally travel into an office and during this time of the year (in traditional years), these offices might hold a holiday party. We’re die-hard holiday fans but of course, COVID-19 means we’re playing it safe. That means, at least for this year, an in-house party is out of the question.</p>
<p>So, we took the celebration virtual.</p>
<p>Earlier this year our Elastic Workplace team held the inaugural <a href="/blog/culture-life-at-elastic-time-for-virtual-summer-camp?baymax=web&elektra=culture-life-at-elastic-decemberfest-for-the-holidays" target="_self">Summer Camp</a> experience as a way for Elasticians to combat Zoom fatigue with a little bit of fun. With the success of this event, the Elastic Workplace team decided to hold another virtual winter event to bring <a href="/blog/culture-elastic-distributed-by-design?baymax=web&elektra=culture-life-at-elastic-decemberfest-for-the-holidays" target="_self">our distributed team</a> a little closer for the holiday season.</p>
<p>Decemberfest, our virtual holiday celebration, took place between December 1-10. Jam packed with a variety of activities that were both fun and family friendly, Elasticians were encouraged to gather the little ones and their significant others to sing during ElastiCarols, watch a bit of holiday magic during the Digital Deception Shows, and listen closely to Santa during storytime over his North Pole Broadcast (via Zoom).</p>
<p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt3f37dadd6bc49c0a/5fd107008593bc52457edb73/Digital_Deception.jpg" data-sys-asset-uid="blt3f37dadd6bc49c0a" alt="Digital Deception magic show" /></p>
<figcaption>Pick a card, any card at Digital Deception</figcaption>
<p>Virtual game shows such as Family Feud, Wheel of Fortune, The Amazing Race, Show Me the Movie, and Australia’s Got Talent gave our Elasticians a chance to blow off a bit of steam.</p>
<p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt5066fecaefcea985/5fd1074050460c30dd53fe43/Family_Fued.jpg" data-sys-asset-uid="blt5066fecaefcea985" alt="Virtual Family Fued" /></p>
<figcaption>Our virtual Family Feud!</figcaption>
<p>There was also a virtual escape room that pitted teams against each other as they tried to crack various puzzles under a ticking clock.</p>
<p>In addition to these fun activities there was an online cooking session from Amsterdam teaching families how to make a festive gnocchi. Also, a playful gingerbread house competition was held, with the winning house chosen by popular vote. There were prizes in four categories:</p>
<ul>
  <li>Most on brand Elastic</li>
  <li>Most local flair</li>
  <li>Most creative</li>
  <li>Best in show</li>
</ul>
<p>Here are two creative entries using Elastic logos!&nbsp;</p>
<p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt17685f160c065ee1/5fd244f77c43e43bf4196978/Felix-Roessel-and-family.jpg" data-sys-asset-uid="blt17685f160c065ee1" alt="Gingerbread house with a manger" /></p>
<p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt9a058cd4610f584c/5fd24549e53f3355047dc890/toby-yule-logstash.jpg" data-sys-asset-uid="blt9a058cd4610f584c" alt="Cluster yule log! " /></p>
<p>As a part of Decemberfest, we also highlighted holidays from around the globe. We asked our Elasticians to share local traditions and experiences through short videos and photos. The response was great, highlighting the traditions of over 9 different cultures.</p>
<div class="video embed-container" style="height: 319.725px;">
    <script type="text/javascript" async src="https://play.vidyard.com/embed/v4.js"></script>
    
    
    <img
      style="width: 100%; margin: auto; display: block;"
      class="vidyard-player-embed"
      src="https://play.vidyard.com/5nWEkFJRwqtVeNTLhUrawJ.jpg"
      data-uuid="5nWEkFJRwqtVeNTLhUrawJ"
      data-v="4"
      data-type="inline"
    />
    </div>
<p>And of course, just like Summer Camp, the Workplace team encouraged Elasticians to help build <a href="https://open.spotify.com/playlist/0amPDfshe4Sw6PFWOSNnWl?si=jVyqhDUHRf-NP_JVzDEWHQ" target="_self">a collaborative playlist full of holiday cheer.</a></p>
<p>And what would the holidays be without a little giving back? In the spirit of the season we held three virtual volunteering sessions. These one-hour sessions were held on <a href="https://www.zooniverse.org/?baymax=web&elektra=culture-life-at-elastic-decemberfest-for-the-holidays" target="_self">the Zooniverse platform</a>, where volunteers helped power research with real-world impact. We also celebrated Giving Tuesday for Elasticians by matching donations to contribute to their favorite organizations.</p>
<p>“We had a lot more participation during Decemberfest than we did for Summer Camp,” said Corey Williams, Workplace lead for the Americas. “I think this was because we had a bit more lead time putting this event together — we knew that it was going to be a virtual event from the start and were better able to advertise. We were also able to get buy-in and promotion from leadership for this event after the success of Summer Camp, which really encouraged people to take the time off and have a little fun.”</p>
<p><em>Interested in joining Elastic? We’re hiring. Check out </em><a href="/about/teams/?baymax=web&elektra=culture-life-at-elastic-decemberfest-for-the-holidays" target="_self"><em>our teams</em></a><em> and f</em><a href="/about/careers/?baymax=web&elektra=culture-life-at-elastic-decemberfest-for-the-holidays" target="_self"><em>ind the right career for you!</em></a><em> Want to read more about life at Elastic? Read more </em><a href="/blog/category/culture?baymax=web&elektra=culture-life-at-elastic-decemberfest-for-the-holidays" target="_self"><em>on our blog</em></a><em>!</em></p>

ElasticSocialCover.jpg

Decemberfest_logo.jpg

Life @ Elastic: Decemberfest for the holidays

OTTO Motors designs, manufactures, and operates self-driving vehicles used for industry. Learn how they use the Elastic Stack to run and monitor their fleet.

<p><i>This post is a recap of a user talk given at <a href="/elasticon/conf/2018/sf">Elastic{ON} 2018</a>. Interested in seeing more talks like this? Check out the conference archive&nbsp;or&nbsp;find out when the <a href="/elasticon">Elastic{ON} Tour</a> is coming to a city near you.</i>
</p><p><a href="https://ottomotors.com/">OTTO Motors</a> — a part of <a href="https://www.clearpathrobotics.com/">Clearpath Robotics</a> — designs, manufactures, and operates self-driving industrial vehicles. Their goal is to use robotics to make human-driven operations obsolete. Their mobile robot fleet operate in over 40 countries, allowing companies to automate work, including some that would otherwise be dangerous for humans.
</p><p>Operating so many robots requires a management system that can identify problems quickly, improve development speed, industrialize solutions at cost, and easily connect to IoT technology. OTTO Motors also relies on intelligent path planning, which requires high volumes of data points to be processed live, so that robots know how to get from A to B at both an individual unit and fleet level.
</p><p>OTTO Motors needed a full stack solution that could be tested without huge expense or the need to build new technologies, and that would plug easily into existing software already in use within the company. While searching for a solution that fit all this criteria, the OTTO Motors team made a survey of the wide, IoT solution landscape. “We wanted something quick,” says Greg Jacobs, former Manager of Infrastructure at OTTO Motors. “Turns out Elastic was the best thing to start with.”
</p><p>Ultimately, the team at OTTO Motors chose the <a href="/products">The Elastic Stack</a> for three key reasons:
</p><ol>
	<li>Elastic’s products have HTTP/JSON APIs readily available for use. With the Elastic Stack, it was easy to get data into the service because it played nicely with other existing internal products and services already in use. In addition, Elastic provided easy-to-use client libraries and rich data collection options.</li>
	<li>A single computer could be used to install and run the Elastic Stack, and out-of-the-box was an independent system, nothing else needed to be built to be off to the races. From the install, it was easy to get started, explore, start plugging things in with the Elastic Stack, and scale as OTTO Motors’ needs grew.</li>
	<li>KIbana is web-based, easy to use, and well-integrated for use by a variety of end users/use cases, making it useful across the entire company. </li>
</ol><p>Again, Greg Jacobs: “[The Elastic Stack] was the seed to plant. But it isn’t going to be something that grows into (just) a walled (off) garden. It will work with all the other stuff.”
</p><p>To that end, OTTO Motors moved  beyond simply using the Elastic Stack for IoT applications, and deployed it to help with software development QA, IT security, and building dashboards for exploratory reports. For example, automated systems for testing robots at OTTO Motors makes virtual docker containers to create Lidar data that the robots read as real obstacles. Elastic also allows OTTO Motors to perform high-speed simulations that the robots can complete instantaneously. This allows OTTO Motors to exercise their fleet management software at hundreds, if not thousands of times faster than normal speed, all while using <a href="/products/beats/metricbeat">Metricbeat</a> and other tools provided by Elastic to test for issues within the system.</p><p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/bltee4a974b8d3accea/5f9b2254432f517518d40b71/blog-otto-motors.png" data-sys-asset-uid="bltee4a974b8d3accea" alt="blog-otto-motors.png"></p><p>Running a large fleet of robots produces a massive amount of data which is fed to Elastic. The amount of data collected by OTTO Motor robots is overwhelming: over 5k data points every 10 seconds for every robot in a fleet. This includes CPU time, WiFi signal, motor temperature, navigation details, and more. And they use Elastic to aggregate all that data into 30 minute reporting windows.<br></p><p>OTTO Motors also uses Elastic’s monitoring feature to create custom dashboards to discover hidden trends, see what indices are being hit hardest, and to learn what’s popular with users.
</p><p>The Otto Motors story is about full stack needs and full stack solutions. Want to learn more about how Otto Motors uses the Elastic Stack to make their robotics work better across the globe? Watch <a href="/elasticon/conf/2018/sf/the-quick-starter-stack-for-robotics-iot-and-big-data">The Quick Starter Stack for Robotics, IoT, and Big Data</a> talk from <a href="/elasticon/conf/2018/sf">Elastic{ON} 2018</a>. In it you’ll learn more about how the Elastic Stack makes maintaining and growing a fleet of intelligent, self-driving robots a reality.
</p><p><a href="/elasticon/conf/2018/sf/the-quick-starter-stack-for-robotics-iot-and-big-data"><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt649d393fa3c4fcfe/5c57df9352256ff20b4afaf7/Screen_Shot_2018-08-22_at_1.28.13_PM.png" data-sys-asset-uid="blt649d393fa3c4fcfe" alt="OttoMotors" style="display: block; margin: auto;"></a>
</p>

Otto Motors Robot.jpg

Otto Lifting Boxes.jpg

OTTO Motors: Using the Elastic Stack to Expand the IoT Landscape

<table style="background: #FFFFD2;"><tbody><tr><td>Get your ArcSight security data into Elasticsearch and visualized in Kibana in literally minutes with the Logstash ArcSight module. <a href="https://elastic.ac.cn/arcsight">Learn more</a>.&nbsp;</td></tr></tbody></table><p><em>Editor's Note: Be sure to check out the other posts in this 6-part blog series.&nbsp;</em><a href="https://elastic.ac.cn/blog/integrating-elasticsearch-with-arcsight-siem-part-1"><em>Part 1</em></a><em>&nbsp;kicks off the series with getting started content.&nbsp;</em><a href="https://elastic.ac.cn/blog/integrating-elasticsearch-with-arcsight-siem-part-3"><em>Part 3</em></a><em> walks you through how to scale the architecture. </em><a href="https://elastic.ac.cn/blog/integrating-elasticsearch-with-arcsight-siem-part-4"><em>Part 4</em></a><em> and </em><a href="https://elastic.ac.cn/blog/integrating-elasticsearch-with-arcsight-siem-part-5"><em>Part 5</em></a><em> provide examples of setting up alerts for common security threats using the alerting features in X-Pack.&nbsp;</em><a href="https://elastic.ac.cn/blog/integrating-elasticsearch-with-arcsight-siem-part-6"><em>Part&nbsp;6</em></a><em>&nbsp;extends the alerting&nbsp;story&nbsp;with automated anomaly detection using machine learning.</em><br /></p><p>In the <a href="/blog/integrating-elasticsearch-with-arcsight-siem-part-1">first post</a> in the "Integrating the Elastic Stack with ArcSight SIEM" blog series, we demonstrated how to send Common Event Format (CEF) data from an ArcSight connector to the Elastic Stack. &nbsp;We will continue the series with several posts illustrating how to proactively monitor security data in Elasticsearch using X-Pack alerting features.&nbsp;</p><p>The X-Pack alerting capability allows users to watch for changes or anomalies in their data and perform necessary actions in response. &nbsp;For those new to the capability, consider exploring the <a href="https://elastic.ac.cn/guide/en/x-pack/current/how-watcher-works.html">Getting Started Content</a> and <a href="https://www.youtube.com/watch?v=gD7MGt5jgDY">Elastic Watcher Lab series</a>.</p><p>While we have many <a href="https://github.com/elastic/examples/tree/master/Alerting">existing examples</a> of alerting on data in Elasticsearch, we will focus our examples in this blog series specifically on data in the CEF format. Our hope is that providing these examples tailored for the CEF format will be valuable to users looking to complement their existing ArcSight-based security solutions with the Elastic Stack. The alert configurations ("watches") themselves, however, are applicable to any CEF compliant data with the required fields. The test environment referenced additionally requires no further configuration to integrate with ArcSight - simply forward events to the Logstash TCP port and start enjoying the watches!<br /></p><p>Throughout this blog series we will focus on rule-based alerting, starting with a relatively simple use case before progressing to more complex pattern detection. &nbsp;Finally we will explore machine learning-based anomaly detection as we attempt to address the obvious challenges with pure rule-based alerting approaches for more complex pattern detection.</p><p>Security analytics watches execute against a constant stream of data in real time. Each watch will execute periodically, accessing the last N seconds and raising an alert if required. Throughout this series we utilise several static historical datasets which cover up to a month period. In order to test, we can’t realistically ingest the data and wait for events to fire over the next month! We have therefore adjusted the watches to execute over the full indexed dataset.<br /></p><h3>Successful Login From an External IP</h3><p><em>Problem: "Alert if a successful SSH login is performed from an External IP"</em><br /></p><p>We start with a simple watch to introduce the concepts of using basic alerting in X-Pack to identify specific patterns in your Elasticsearch data.</p><p>For our first use case, we utilise a common source of event data - authentication logs. The sample dataset consists of inbound SSH logs (in the CEF format) from a single honeypot during the month of November 2016. Below is a sample log line in the CEF format. The highlighted fields will be used in the watch definition.&nbsp;</p><pre class="prettyprint">CEF:0|Unix|Unix|5.0|cowrie.login.success|login attempt [root/!root] succeeded|Unknown|destinationUserName=root externalId=15 startTime=Nov 15 2016 19:22:29 destinationHostName=elastic_honeypot destinationAddress=192.168.20.2 deviceReceiptTime=Nov 15 2016 19:22:29 deviceTimeZone=Z transportProtocol=TCP applicationProtocol=SSHv2 destinationServiceName=sshd devicePayloadId=15 message=login attempt [root/!root] succeeded categoryOutcome=success categoryBehaviour=cowrie.login.success sourceTranslatedAddress=192.168.1.105 sourceAddress=192.168.1.105 deviceDirection=1 cs1=0 cs1Label=isError cs2=SSHService ssh-userauth on HoneyPotSSHTransport,2,192.168.1.105 cs2Label=system cs3=!root cs3Label=password cs4=111f70f0 cs4Label=session&lt;span style="font-size: 14px; white-space: normal;"&gt;<br />&lt;/span&gt;<br /></pre><p>The values of cowrie.login.failed and cowrie.login.success for our categoryBehaviour represent a failed and successful login respectively. &nbsp;The field destinationAddress indicates the target server. &nbsp;Note that while our example data is from a single server, the provided watch could be applied to a larger infrastructure with potentially hundreds of servers. To determine if the connection originated from an external IP address, we utilise the field sourceAddresss. Finally, the startTime indicates the time at which the event occurred. This field will be be parsed by our Logstash config file into the @timestamp field.<br /></p><p>To ingest this dataset, we use a simple Logstash configuration enabled with the CEF codec. &nbsp;Instructions for downloading and ingesting this dataset, with supporting dashboards, can be found <a href="https://github.com/elastic/examples/tree/master/Security%20Analytics/ssh_analysis">here</a>.<br /></p><p>With around 114,000 SSH events, our honeypot attracted quite some attention. &nbsp;As you would expect, this is dominated by failed attempts as attackers iterated through common password lists.</p><p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt78a901d2bc7dcfa6/5f8788b89245d74a2063ec1b/integrating-arcsight-blog-image.png?disposition=inline"/></p><p>For our simple example, we will alert on a successful login when it originates from an external IP. &nbsp;</p><p>A successful login from an external IP can be identified by an indexed CEF event containing the following fields and values:<br /></p><ul><li>categoryBehaviour&nbsp;with a value of cowrie.login.success</li><li>sourceAddress with a value that is NOT in the range&nbsp;192.168.0.0/16</li></ul><p>All watches execute periodically based on a schedule defined in the trigger block. We will set our watch to execute every N seconds and evaluate the last N seconds for a match. To filter the last N seconds, we evaluate events with a @timestamp &gt; now-N seconds. We will trigger an alert if even a single match is detected. Finally, in order to provide a useful summary, we use aggregations to indicate the target server, source IP and time in any action output.<br /></p><p>We will the apply the same iterative process to construct a watch in future example in this series. First we identify the input and query to capture the relevant data before evaluating the response with a condition to determine whether the watch fires an action. Next we extract the data points of interest through a transformation before finally taking action.<br /></p><p>As described in the <a href="https://github.com/elastic/examples/tree/master/Security%20Analytics/ssh_analysis">deployment instructions</a>, we assume that the user is interacting with Elasticsearch and Watcher APIs via the Dev Tools -&gt; Console app in Kibana.<br /></p><h4>Step 1 - Set Up Watch Trigger and Watch Input</h4><p>The first step in constructing our watch is to identify the watch execution schedule i.e., how often it should execute and over which time window. &nbsp;This defines the "responsiveness" of our watch and how often we can potentially raise an alert. &nbsp;For this example, we will execute our watch every 10 seconds looking for potential successful remote logins.</p><p>We next need to identify the data we wish to evaluate in our watch. This requires an <a href="https://elastic.ac.cn/guide/en/x-pack/current/input.html">input</a> to load our data into the current execution context. As our SSH events are stored in Elasticsearch we utilise a <a href="https://elastic.ac.cn/guide/en/x-pack/current/input-search.html">Search input</a> to load the results from a query. The query itself uses a simple boolean match to identify those documents in the last 10 seconds which indicate a successful login from a remote IP.</p><pre class="prettyprint">{<br />  "trigger": {<br />    "schedule": {<br />      "interval": "10s"<br />    }<br />  },<br />  "input": {<br />    "search": {<br />      "request": {<br />        "indices": [<br />          "cef-ssh-*"<br />        ],<br />        "types": [<br />          "syslog"<br />        ],<br />        "body": {<br />          "query": {<br />            "bool": {<br />              "must_not": [<br />                {<br />                  "term": {<br />                    "sourceAddress": {<br />                      "value": "192.168.0.0/16"<br />                    }<br />                  }<br />                }<br />              ],<br />              "filter": [<br />                {<br />                  "term": {<br />                    "categoryBehaviour": "cowrie.login.success"<br />                  }<br />                },<br />                {<br />                  "range": {<br />                    "@timestamp": {<br />                      "gte": "now-10s"<br />                    }<br />                  }<br />                }<br />              ]<br />            }<br />          },<br />          "size": 100,<br />          "_source": [<br />            "destinationHostName",<br />            "sourceAddress",<br />            "@timestamp"<br />          ]<br />        }<br />      }<br />    }<br />  }<br />}<br /></pre><h5>Some Considerations</h5><p>The above query requests 100 results only. This limits us to alerting on 100 successful logins every 10 seconds - more than sufficient for most infrastructures! As each document represents a successful login, we restrict the data we report to the destinationHostName, sourceAddress and @timestamp using a _source filter.</p><p>For the purposes of testing, we can remove the date filter to execute the query across the entire dataset - thus avoiding the need to execute the watch for every period. For the final watch shown in <a href="#step_5">Step 5 - Put It all together</a>, we restore this filter for completeness.</p><h4>Step 2&nbsp;- Setup Watch Condition</h4><p>The previous step caused the results of the query to be loaded into the execution context as a dictionary. &nbsp;This is accessible through the ctx.payload parameter. We utilise this to evaluate whether an action should be taken through a <a href="https://elastic.ac.cn/guide/en/x-pack/5.1/condition.html">condition</a>, which determines whether the watch will fire by returning true or false. Here we find a compare condition is sufficient, evaluating whether the number of hits is &gt; 0. In later posts, we utilise the more powerful <a href="https://elastic.ac.cn/guide/en/x-pack/5.1/condition-script.html">script</a> condition.</p><pre class="prettyprint">"compare": {<br />  "ctx.payload.hits.total": {<br />    "gt": 0<br />  }<br />}<br /></pre><h4>Step 3&nbsp;- Transform Watch Input into Watch Payload</h4><p>Rather than sending out alerts with the full query response, we can use a transform stage to modify the payload into a more readable structure. This will also be required for Step 4, where we aim to index each matching response document back into Elasticsearch for visualisation purposes. Fortunately, we can achieve this with a simple <a href="https://elastic.ac.cn/guide/en/x-pack/current/transform-script.html">script transform</a> and one line in Elasticsearch’s native scripting language <a href="https://elastic.ac.cn/guide/en/elasticsearch/reference/current/modules-scripting-painless-syntax.html">Painless</a>. &nbsp;</p><pre class="prettyprint">"transform": {<br />      "script": "return ctx.payload.hits.hits.stream().map(d -&gt; ['@timestamp':d._source['@timestamp'],'sourceAddress':d._source.sourceAddress,'destinationHostName':d._source.destinationHostName]).collect(Collectors.toList());"<br />}<br /></pre><p><span style="background-color: initial;">Example output payload:</span></p><pre class="prettyprint">[<br />  {<br />    "@timestamp": "2016-11-16T17:35:50.000Z",<br />    "destinationHostName": "elastic_honeypot",<br />    "source_ip": "62.57.78.153"<br />  },<br />  {<br />    "@timestamp": "2016-11-16T12:15:57.000Z",<br />    "destinationHostName": "elastic_honeypot",<br />    "source_ip": "5.45.84.12"<br />  }<br />]<br /></pre><h4>Step 4&nbsp;- Set Up Watch Actions</h4><p>On detecting successful remote logins we would typically raise an alert through the appropriate medium e.g., a ticketing system for the analyst to investigate. Whilst these are <a href="https://elastic.ac.cn/guide/en/x-pack/current/actions-jira.html">supported in Alerting</a>, for the purposes of example we can simulate this process by indexing a doc for each login to ES using the <a href="https://elastic.ac.cn/guide/en/x-pack/current/actions-index.html">index action</a>.</p><p>Our index action requires the document be placed in the the <a href="https://elastic.ac.cn/guide/en/x-pack/current/actions-index.html#anatomy-actions-index-multi-doc-support">appropriate "_doc" key</a>, achieved through a simple Painless script.<span></span><br /></p><pre class="prettyprint">"index_payload": {<br />  "transform": {<br />    "script": "return ['_doc':ctx.payload._value];"<br />    },<br />    "index": {<br />      "index": "cef-ssh-watch-results",<br />      "doc_type": "watch-result"<br />    }<br />}<br /></pre><h4>Step 5&nbsp;- Put It&nbsp;It All Together</h4><p>In assembling the above, we need to consider some X-Pack alerting level parameters. As shown below we control how often the watch is executed using a <a href="https://elastic.ac.cn/guide/en/x-pack/current/trigger-schedule.html#schedule-interval">schedule interval</a>. This should be consistent with the range query, which uses the <a href="https://elastic.ac.cn/guide/en/x-pack/5.1/how-watcher-works.html#watch-execution-context">scheduled time</a> to ensure watches do not overlap. Finally, by assuming daily indices for our events we are able to target the query at only the current day’s index for efficiency.</p><pre class="prettyprint">{<br />  "trigger": {<br />    "schedule": {<br />      "interval": "10s"<br />    }<br />  },<br />  "input": {<br />    "search": {<br />      "request": {<br />        "indices": [<br />          "&lt;cef-ssh-{now/d}&gt;"<br />        ],<br />        "types": [<br />          "syslog"<br />        ],<br />        "body": {<br />          "query": {<br />            "bool": {<br />              "must_not": [<br />                {<br />                  "term": {<br />                    "sourceAddress": {<br />                      "value": "192.168.0.0/16"<br />                    }<br />                  }<br />                }<br />              ],<br />              "filter": [<br />                {<br />                  "term": {<br />                    "categoryBehaviour": "cowrie.login.success"<br />                  }<br />                },<br />                {<br />                  "range": {<br />                    "@timestamp": {<br />                      "gte": "{{ctx.trigger.scheduled_time}}||-10s"<br />                    }<br />                  }<br />                }<br />              ]<br />            }<br />          },<br />          "size": 100,<br />          "_source": [<br />            "destinationHostName",<br />            "sourceAddress",<br />            "@timestamp"<br />          ]<br />        }<br />      }<br />    }<br />  },<br />  "condition": {<br />    "compare": {<br />      "ctx.payload.hits.total": {<br />        "gt": 0<br />      }<br />    }<br />  },<br />  "transform": {<br />    "script": "return ctx.payload.hits.hits.stream().map(d -&gt; ['@timestamp':d._source['@timestamp'],'sourceAddress':d._source.sourceAddress,'destinationHostName':d._source.destinationHostName]).collect(Collectors.toList());"<br />  },<br />  "actions": {<br />    "index_payload": {<br />      "transform": {<br />        "script": "return ['_doc':ctx.payload._value];"<br />      },<br />      "index": {<br />        "index": "cef-ssh-watch-results",<br />           "doc_type": "watch-result"<br />      }<br />    }<br />  }<br />}<br /></pre><p>To insert the watch, use the PUT API as described <a href="https://elastic.ac.cn/guide/en/x-pack/current/watcher-api-put-watch.html">here</a>. Alternatively, remove the date filter and execute the watch across the dataset using the <a href="https://elastic.ac.cn/guide/en/x-pack/current/watcher-api-execute-watch.html#watcher-api-execute-inline-watch">inline execution API</a>. This will result in an indexed document for up to 100 successful logins that exist in the dataset. Using the dashboard from earlier, we can add a table visualization populated with the latest successful remote login to each server.</p><p><img src="https://api.contentstack.io/v2/uploads/5873a705e52e8fd3503db6e6/download?uid=blt1926efa5c2fc9885" data-sys-asset-uid="blt1926efa5c2fc9885" alt="Screenshot 2017-01-09 14.49.55.png"/></p><h4>Final Thoughts</h4><p>In our next post we will <a href="https://elastic.ac.cn/blog/integrating-elasticsearch-with-arcsight-siem-part-3">learn how to scale the architecture</a> we discussed in <a href="https://elastic.ac.cn/blog/integrating-elasticsearch-with-arcsight-siem-part-1">Part 1</a>&nbsp;before we look at detecting and alerting on more complex&nbsp;patterns with X-Pack alerting features.</p><p><em>Interesting in learning more? Check out other posts in the Elasticsearch + ArcSight series.</em></p><ul><li><a href="https://elastic.ac.cn/blog/integrating-elasticsearch-with-arcsight-siem-part-1"><em>Part 1 - How to send Common Event Format data from ArcSight to Elasticsearch</em></a></li><li><a href="https://elastic.ac.cn/blog/integrating-elasticsearch-with-arcsight-siem-part-2"><em>Part 2 - How to proactively monitor security data in Elasticsearch with X-Pack alerting features</em></a></li><li><a href="https://elastic.ac.cn/blog/integrating-elasticsearch-with-arcsight-siem-part-3"><em>Part 3 - Scaling your ArcSight and Elasticsearch&nbsp;architecture</em></a></li><li><a href="https://elastic.ac.cn/blog/integrating-elasticsearch-with-arcsight-siem-part-4"><em>Part 4&nbsp;-&nbsp;Detecting Successful SSH Brute Force Attacks</em></a></li><li><a href="https://elastic.ac.cn/blog/integrating-elasticsearch-with-arcsight-siem-part-5"><em>Part 5&nbsp;-&nbsp;Detecting Unusual Process using Rules Based Alerts</em></a></li><li><a href="https://elastic.ac.cn/blog/integrating-elasticsearch-with-arcsight-siem-part-6"><em>Part 6&nbsp;-&nbsp;Using Machine Learning to Detect Rare (unusual) Processes on a Server</em></a></li></ul><p><br /></p>

elasticsearch-arcsight-elastic-stack-siem-full-bleed.jpg

scaled.jpg

Integrating Elasticsearch with ArcSight SIEM - Part 2

A deep dive into the Elastic Cloud Value Calculator calculations and assumptions. Use it to understand the details behind each benefit group, or leverage it to create your own version custom to your situation.

<p>As your Elastic usage increases and your use cases expand, it's important to know the benefits and cost savings that you can achieve by running Elasticsearch as a service. But since every Elasticsearch implementation can vary by use case and deployment model, it can be complicated to tackle on your own. So with that in mind, we are excited to share the <a href="https://elastic.ac.cn/cloud/value-calculator">Elastic Cloud Value Calculator</a>.</p><p>With this calculator, you can easily estimate cost savings and increases in productivity based on infrastructure needs, efficiency factors, potential risks, and more. Then, based on your results, you'll know how Elastic Cloud creates potential efficiencies for you, or help you determine if you're already on the right track. The calculator just does the hard part of handling the variables of your specific situation.</p><p>One point before we get into the details. In developing this tool we have partnered with <a href="https://www.decisionlink.com/">DecisionLink</a>, who has&nbsp;many years of experience in developing web calculators and value models. We are grateful for their expertise in defining the assumptions with us below.</p><p>Let's take a look at the logic and assumptions built into this tool. On to the math!</p><h2>Reduce IT infrastructure and storage costs</h2><p>Cost savings usually start here, as it’s quite frequently the biggest bucket of potential value. Some of the factors the calculator takes into account are:</p><ul><li>What instance types (and their respective costs) are you running Elastic on?</li><li>What do those instances cost annually?</li><li>If you aren't on Elastic Cloud, are you using reserve or on-demand pricing?</li><li>If you're running Elasticsearch in your own data center, what is the three-year amortized value of the hardware?</li></ul><p>The first situation we filter for is whether you are running Elastic in your own data center or running it self-managed on a cloud provider. Each situation has its own infrastructure cost implications, which we break down below.</p><h4>Self-hosted in your own data center</h4><p><strong>Reduce self-hosted server costs:</strong></p><p># of nodes * annual cost per server (server purchase / 3 years)&nbsp;* 100% reduction of future server cost</p><p>Note: For self-hosted servers, we assume a 1:1 ratio of server to node. If you are using larger machines that can handle multiple nodes, you’ll need to divide the annual cost of a server by the number of nodes you are running on each server in order to get the appropriate cost comparison.</p><p><strong>Reduce server maintenance cost:</strong></p><p># of nodes * 20% of annual server cost for maintenance &amp; support&nbsp;* 100% reduction of future server maintenance cost</p><p>Note: Maintenance of a server depends largely on the type of server and how old it is. Newer servers generally require less maintenance and support vs older ones (3+ years).</p><p><strong>Reduce data center overhead cost (heating, cooling, electric):</strong></p><p># of nodes * 25% of annual server cost for infrastructure overhead&nbsp;* 100% reduction of future server maintenance cost</p><p>For an overview of power usage associated with servers see this <a href="https://www.zdnet.com/article/toolkit-calculate-datacenter-server-power-usage/">ZD net article</a> on the topic.</p><h4>Self-hosted on another cloud provider</h4><p><strong>Reduce cost of cloud services:</strong></p><p># of nodes * annual cost per cloud virtual machine&nbsp;* 100% reduction of future cloud virtual machine cost</p><p><strong>Reduce cost of data transfer and storage (DTS):</strong></p><p># of nodes * 20% of annual cloud virtual machine cost DTS&nbsp;* 100% reduction of DTS cost</p><h4>General benefits applied in both situations</h4><p><strong>Reduce software licensing costs:</strong></p><p># of nodes * 10% annual cost per server or cloud virtual machine&nbsp;* 100% reduction of future server license cost</p><p>Note: Assumptions&nbsp;here depend on whether you are using open source Linux, a RedHat distribution of the same, or something else for your server OS. Also, please keep in mind that this will be much higher if you are already paying for an Elastic licensed tier of software (Gold, Platinum, Enterprise) so you will need to take that into account and increase this amount if you are comparing self-managed to Elastic Cloud.</p><h3>Reduce overprovisioning cost of hardware</h3><p>We run into overprovisioned hardware on a regular basis. Whether you overestimated or are just planning for worst-case scenarios, Elastic Cloud can help you get your sizing right, regardless of the risks you need to mitigate. In addition, we just launched a new <a href="https://elastic.ac.cn/blog/autoscale-your-elastic-cloud-data-and-machine-learning-nodes">autoscaling capability</a> that will remove the need for you to ever have to worry about rightsizing your Elastic environment again when you use Elastic Cloud.</p><p><strong>Reduce overprovisioning cost of hardware:</strong></p><p># of nodes * annual cost per server or cloud virtual machine&nbsp;* 22.5% reduction of future overprovisioning costs</p><p>Note: We see ranges of overprovisioning usually between 15-30% of the total infrastructure required.</p><h3>Reduce unnecessary data duplication cost</h3><p>For the large community out there who are running Elastic self-managed on our free and open or Basic tiers, many will twist their deployment in order to reduce risk of exposure of data to the wrong audience. Often, the logical way to do this is to duplicate a portion of the data in your current Elastic cluster and put that portion of data in a new, isolated cluster for a specific audience. However, this effort can be reduced or eliminated by taking advantage of security and access controls, easily configured with Elastic Cloud.</p><p><strong>Reduce data duplication cost with access controls:</strong></p><p>[# of nodes * % of nodes duplicated (30% assumed without access controls)] * annual duplication cost per node (both infra and human capital costs) * 30% reduction of future duplication costs</p><p>Note: We assume only 30% reduction in cost here in order to be conservative and to take into account the probability of this happening in all situations. If this is occurring in your situation, you should consider this improvement factor to be 100%.</p><h3>Subscription-level features</h3><p>Lastly, there are powerful features in our Enterprise tier that you only have access to as an Elastic customer, such as <a href="https://elastic.ac.cn/blog/whats-new-cloud-autoscaling-cross-cluster-search-data-tiers">searchable snapshots and data tiers</a>. The cold data tier enables you to run a single node of Elastic with the replica stored in durable object storage (Amazon S3, Azure Blob Storage, or Google Cloud Storage) to reduce replication cost. So you can still search the cold node, with snappy responsiveness, but with the backup being an ultra low-cost alternative. And that’s just the beginning. We’re currently working on a searchable frozen tier that sits entirely within object storage with an even more compelling cost profile.</p><p><strong>Reduce hardware cost with data tiering (not applied within model, but worth considering):</strong></p><p># of nodes in warm data tier (assumes 50% of total nodes) * annual cost per server or cloud virtual machine * 50% reduction of future infrastructure costs through use of data tiers</p><h2>IT efficiency</h2><p>Running and administering the Elastic Stack can take time and personnel. Deployments often start small, with little administrative overhead. But as adoption increases, so does the implementation size, and so do the demands (and expectations) of your users. We are always working to make the administration easier from within Kibana, but there is a lot to keep in mind: shards, replicas, scaling, upgrading, security, compliance, capacity, and more. These are things that aren't necessarily considered the first time a cluster is spun up.</p><p>Generally speaking, a 25-node cluster of Elastic for logging or observability will require about 50% of a person’s time to manage and orchestrate, based on anecdotal evidence and discussions with the many folks here at Elastic who have done the work. So to apply some logic to that, if 50 nodes = 1 DevOps full-time employee (FTE), then 1 node = 0.02 FTEs.</p><p>But with Elastic Cloud, we frequently see 70% of that operational overhead removed from our administrators’ daily lives. Now you only need to take up a smaller fraction of someone's time per node, as Elastic Cloud takes care of many of the details. The ratio becomes 200 nodes to 1 FTE.</p><p>If you want a better view into how we can divide and conquer the work together, have a look through <a href="https://elastic.ac.cn/cloud/shared-responsibility">our responsibility matrix</a>. The main responsibilities Elastic Cloud helps to make much easier include:</p><ol><li>Reducing overall DevOps automation required through predefined APIs and protocols. This broader category accounts for the lion’s share of the automation benefits to administrative teams.<br /><br /><strong>Reduce IT staff through Elastic Cloud orchestration:</strong><br /># of FTEs for running Elastic self managed on a node basis * $100,000 FTE * 70% reduction in FTE time required to manage Elastic<br /></li></ol><ol start="2"><li>Reducing time to manage upgrades by 95%.<br /><br /><strong>Reduce time to manage upgrades:</strong><br />Number of upgrades per year (assume 2X as a starting point) * $3,200 per upgrade (64 hours) * 95% reduction in time for managing upgrades<br /></li></ol><ol start="3"><li>Reducing risk of managing critical vulnerability incidents and associated costs to near zero.<br /><br /><strong>Reduce risk of critical vulnerability incidents:</strong><br /># of critical vulnerabilities per year (assumes 4) * $4,000 per critical vulnerability engineering time (80 hours) * 10% reduction in critical vulnerability incidents</li></ol><h2>Revenue and employee disruption risk</h2><p>This value category is potentially the largest, depending on your situation. Let’s talk about what it means to your business to run a highly stable, scalable, and resilient service with Elastic central to your architecture.</p><h3>Employee disruption risk reduction</h3><p>This is the most common and important area of downstream user risk reduction. There are two big questions you need to consider here:</p><ol><li>How many people at your company are using Elastic today (in some form) based on your implementation?</li><li>What is the cost of employees not having access to your Elastic implementation?</li></ol><p>If it’s a security- or observability-related application, this could mean that your end users lack support when there is an urgent security or downtime risk-related incident that needs to be addressed. Ask yourself:</p><ul><li>What is the cost per minute of disruption and downtime?</li><li>Will a single failure cause a domino effect of failures?</li><li>If there's an outage, will the observability solution also fail?</li></ul><p>And maybe most importantly: How forgiving will stakeholders be when preventable infrastructure issues cause business outages?</p><p><strong>Improve end-user productivity through reduced risk of disruptions:</strong></p><p># of employee end users of the solution * $100,000 per FTE * 1% improvement in end user productivity</p><p>Note: This improvement factor assumption means that your end users would experience 1% less downtime due to solution disruption. We generally see higher downtime for internal employee-facing applications, and as a result chose this improvement factor. Keep in mind that your uptime improvement should be based on your current uptime service-level agreement (SLA).</p><h3>Revenue and service risk</h3><p>This calculation takes into account the risks associated with a customer-facing service you are running on Elastic. What is the risk of an outage associated with that service? Is there potential revenue at risk, and if so, how much?</p><p>These are all good things to have a handle on, and we've made some simple assumptions for you if you choose this situation. Keep in mind that we generally apply this situation, conservatively, towards consumer-facing search use cases. For example, what would it mean if you are a retailer and your search stops working, and what is the cost of that possible outage? These are all risks that need to be considered and mitigated.</p><p><strong>Reduce risk of lost revenue due to unplanned downtime:</strong></p><p>Annual number of downtime hours (assume 44, or 99.5% uptime) * $25,000 per downtime hour * 90% reduction in downtime hours</p><p>Note:&nbsp;This improvement factor assumes an increase from 99.5% uptime to 99.95% uptime, an improvement of 90% as a starting point.</p><h3><strong>Reduced risk of SLA violation</strong></h3><p>This particular calculation is equally specific as the last one. The assumption is that you have embedded search or a related intelligence use case into a business customer-facing application. What’s the risk of your application experiencing instability that we can help you mitigate? And if your service-related search or lookup capability is down, will your end users claim they are experiencing an SLA violation? Anyone who's worked on a support contract knows how important these considerations are.</p><p><strong>Reduce risk of SLA violation:</strong></p><p>Annual SLA penalty events (we assume 1 to start) * [# of customers impacted by an SLA penalty event * average annual revenue per business customer * % of business customer revenue refunded due to penalty event (15% seeded)] * 20% reduction SLA penalty risk</p><h2>Security, risk, and compliance</h2><p>For these calculations, we don’t expose any inputs. Generally, this is an acknowledgement of the risk associated with running Elastic without the right security and compliance protocols in place. There is a very good reason to include these benefit calculations, even if your individual risk is marginally low. One must accept the fact that, without the right security setup — which comes standard with Elastic Cloud — Elastic users incur risk of collecting data and exposing that data. Sure, it can be cool to “move fast and break things,” but it isn’t sustainable in the long term when you take into account security and compliance concerns.</p><p><strong>Reduce cost of creating compliant solutions:</strong></p><p>Number of compliant solutions required (assumes 1) * cost per compliance solution (assumes $25,000) * 100% reduction in cost of creating and maintaining a compliance solution</p><p><strong>Reduce risk of data breach through use of access controls:</strong></p><p>[Risk of data breach in scope 1.2% (24% risk of data breach caused by human error: IBM research) * 5% chance of data breach associated with Elastic (vs other datastores)] * average cost per data breach (used industry benchmark from IBM of $8.1M) * 95% reduction in risk of data breach through use of access controls</p><p><strong>Reduce risk of downtime for regulated services:</strong></p><p>Downtime instances leading to regulatory violations (assumes 1) * average fine for regulated services (assumes $10,000) * 90% reduction in downtime</p><p>These assumptions are similar to the reduced risk of downtime noted in the revenue and employee disruption risk section above.</p><h2>Find out how much you can save or gain</h2><p>Now that you know how all these calculations work, we encourage you to try the <a href="https://elastic.ac.cn/cloud/value-calculator">Elastic Cloud Value Calculator</a> yourself. When you download the output file from our value calculator, keep in mind that we include not only the one-year benefit results, but also the three-year results. We also assume 10% year-over-year growth with these value calculations. Generally speaking, our users and customers see quite a bit more growth than this annually, so we believe this is a conservative estimate of your growth potential.</p><p>Finally, a few parting thoughts:</p><ul><li><strong>Experiment with the calculator.</strong> Play around with the factors and numbers to develop a more custom business case tailored to your needs.</li><li><strong>Take the numbers further.</strong> There are downstream impacts from the benefit calculations that you can explore outside of the calculator. Some of these are specific to your needs and can't be added to a standardized tool.</li><li><strong>Help us make the calculator better. </strong>Give us feedback on how to improve it to represent the value you accrue with your use of Elastic Cloud. If at any point you have feedback on this tool, please email <a href="mailto:value-calculator@elastic.co">value-calculator@elastic.co</a>.</li></ul>

blog-banner-magnifying-glass-money-bag.jpg

Optimizing

blog-thumb-magnifying-glass-money-bag.jpg

Elastic Cloud Value Calculator: Understand the business value of Elastic Cloud

<h2>The World of Microservices</h2><p>Enterprises are increasingly adopting microservice architectures. They are developing and deploying more microservices everyday. Often, these services are developed in different programming languages, deployed into separate runtime containers, and managed by different teams and organizations. Large enterprises like Twitter can have tens of thousands of microservices, all working together to achieve their business goals. As they discussed <a href="https://blog.twitter.com/engineering/en_us/a/2013/observability-at-twitter.html">in this Twitter blog post</a>, visibility into the health and performance of the diverse service topology is extremely important for them to be able to quickly determine the root cause of issues, as well as increasing Twitter’s overall reliability and efficiency.</p><p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/bltf0ee6529c2de5c59/5c98d620026179ca592ee24d/blog-opentracing-elastic-apm-1.png" data-sys-asset-uid="bltf0ee6529c2de5c59" style="display: block;margin: auto;"/></p><p>This is where <em><strong>Distributed Tracing</strong></em> can really help. Distributed Tracing helps with two fundamental challenges faced by microservices:</p><ol><li><strong>Latency tracking</strong><br />One user request or transaction can travel through many different services in different runtime environments. Understanding the latency of each of these services for a particular request is critical to the understanding of the overall performance characteristics of the system as a whole, and provides valuable insight for possible improvements.</li><li><strong>Root cause analysis</strong><br />Root cause analysis is even more challenging for applications that build on top of large ecosystems of microservices. Anything can go wrong with any of the services at any time. Distributed tracing is of crucial importance when debugging issues in such a system.</li></ol><p>Take a step back, <em><strong>tracing</strong></em> is only one piece of the puzzles of the <a href="/blog/observability-with-the-elastic-stack">Three Pillars of Observability</a> - Logging, Metrics and Tracing. As we will discuss briefly, Elastic Stack is a unified platform for all three pillars of observability. When logs, metrics, and <a href="https://elastic.ac.cn/observability/application-performance-monitoring" target="_self">APM</a> data are stored in the same repository, analyzed, and correlated together, you gain the most context-rich insight into your business applications and systems. In this blog, we will solely focus on the tracing aspect.</p><p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/bltd249957e822769f9/5c98d61beb3c40e859703c1a/blog-opentracing-elastic-apm-2.png" data-sys-asset-uid="bltd249957e822769f9" style="display: block;margin: auto;"/></p><h2>Distributed Tracing with Elastic APM</h2><p><a href="/guide/en/apm/get-started/current/index.html">Elastic APM</a> is an application performance monitoring system built on the Elastic Stack. It allows you to monitor software services and applications in real time, collecting detailed performance information on response time for incoming requests, database queries, calls to caches, external HTTP requests, etc. Elastic APM agents offer rich auto-instrumentation out of the box (e.g. timing db queries, etc.) for supported frameworks and technologies. You can also use custom instrumentation for custom purposes. This makes it much easier to pinpoint and fix performance problems quickly.</p><p>Elastic APM supports <a href="/guide/en/apm/get-started/6.6/distributed-tracing.html">distributed tracing</a> and is OpenTracing compliant. It enables you to analyze performance throughout your microservice architecture all in one view. Elastic APM accomplishes this by tracing all of the requests, from the initial web request to your front-end service, to queries made to your back-end services. This makes finding possible bottlenecks throughout your application much easier and faster. The Timeline visualization in APM UI shows a waterfall view of all of the transactions from individual services that are connected in a trace:</p><p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blta65f095d22517ce4/5c98d59849a201165fca1042/blog-opentracing-elastic-apm-3.png" data-sys-asset-uid="blta65f095d22517ce4" style="display: block;margin: auto;"/></p><p>Elastic Stack is also a great platform for log aggregation and metrics analytics. Having logs, metrics, and APM traces all stored and indexed in Elasticsearch is super powerful. Being able to quickly correlate data sources like infrastructure metrics, logs and traces enables you to debug the root cause much faster. In the APM UI, when looking at a trace, you can quickly jump to the host or container metrics and logs by clicking the <strong>Actions</strong> menu, if these metrics and logs are also collected.</p><p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/bltc037c8e3109bfba5/5c98d60b6d10d1145a98468c/blog-opentracing-elastic-apm-4.png" data-sys-asset-uid="bltc037c8e3109bfba5" style="display: block;margin: auto;"/></p><p>It would&nbsp;be wonderful if everybody was&nbsp;using Elastic APM to instrument their applications and services. However, Elastic APM is not the only distributed tracing solution available today. There are other popular open source tracers like Zipkin and Jaeger. Concepts like polyglot programming and polyglot persistence are well known and well accepted in the world of microservices. Similarly, “<strong>polyglot tracing</strong>” is going to be more common than not. Because of the independent and decoupled nature of microservices, people responsible for different services will likely use different tracing systems.</p><h2>Challenges for Developers</h2><p>With many different tracing systems available, developers are faced with real challenges. At the end of the day, tracers live inside the application code. Some common challenges are:</p><ol><li>Which tracing system to use?</li><li>What if I want to change my tracer? I don’t want to change my entire source code.</li><li>What do I do with shared libraries that might be using different tracers?</li><li>What if my third-party services use different tracers?</li></ol><p>Not surprisingly, we need standardization to address these concerns. Before discussing where we are with the standardization, let’s take a step back and look at distributed tracing from an architectural perspective in a holistic manner and understand what’s required to achieve the distributed tracing “nirvana”.</p><h2>Architectural Components of Distributed Tracing</h2><p>Modern software systems can be broken down into a few high-level components, typically designed and developed by different organizations and run in different runtime environments.</p><ul><li>Your own application code and services</li><li>Shared libraries and services</li><li>External services</li></ul><p>To monitor such a system in a holistic and integrated fashion with distributed tracing, we need four architectural components:</p><p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt1a2bddd007f963ac/5c98d60367b8321d593d7fb4/blog-opentracing-elastic-apm-5.png" data-sys-asset-uid="blt1a2bddd007f963ac" style="display: block;margin: auto;"/></p><ol><li><strong>Standardized distributed tracing API</strong>. A standardized vendor-neutral tracing API allows developers to instrument their code in a standardized way, does not matter what tracer they might choose to use later during the runtime. This is the first step towards anything.</li><li><strong>Standardized tracing context definition and propagation</strong>. For a trace to go across from one runtime to another, the tracing context has to be understood by both parties, and there has to be a standard way of propagating that context. At the minimum, the context carries a trace ID.</li><li><strong>Standardized tracing data definition</strong>. For trace data from one tracer to be understood and consumed by another tracer there has to be a standardized and extensible format for it.</li><li><strong>Interoperable tracers</strong>. Finally, to achieve 100% runtime compatibility, different tracers have to provide mechanisms for them to both export and import trace data from other tracers in an open way. Ideally, a shared library or service instrumented by a tracer like Jaeger should be able to have it’s tracing data sent directly to Elastic APM or another tracer via the Jaeger agent through a configuration change.</li></ol><p>Now, enter OpenTracing.</p><h2>The OpenTracing Specification</h2><p>The OpenTracing <a href="https://opentracing.io/specification/">specification</a> defines an open, vendor-neutral API for distributed tracing. It enables users to avoid vendor lock-in by allowing them to switch the OpenTracing implementer at any time. It also enables developers of frameworks and shared libraries to provide tracing functionality out of the box, in a standard fashion to enable better insights into the frameworks and libraries. Web-scale companies like Uber and Yelp are using OpenTracing to get deeper visibility into their highly distributed and dynamic applications.</p><h3>The OpenTracing Data Model</h3><p>Basic concepts of OpenTracing and the fundamental <a href="https://opentracing.io/docs/overview/">data model</a> came from Google’s Dapper paper. Key concepts include trace and span.</p><p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/bltf922d3d931f16598/5c98d5fcd3c73ef55e9ea9c2/blog-opentracing-elastic-apm-6.png" data-sys-asset-uid="bltf922d3d931f16598" style="display: block;margin: auto;"/></p><ol><li>A <strong>trace</strong> represents a transaction as it moves through a distributed system. It can be thought of as a directed acyclic graph of Spans.</li><li>A <strong>span</strong> represents a logical unit of work that has a name, start time, and duration. Spans may be nested and ordered to model relationships. Spans accept key:value tags as well as fine-grained, time-stamped, structured logs attached to the particular span instance.</li><li>Trace <strong>context</strong> is the trace information that accompanies the distributed transaction, including when it passes the service to service over the network or through a message bus. The context contains the trace identifier, span identifier, and any other data that the tracing system needs to propagate to the downstream service.</li></ol><h3>How does it all fit in?</h3><p>Ideally, with standardization, tracing information from custom application code, shared libraries, and shared services developed and run by different organizations are exchangeable and runtime compatible, does not matter what tracer each of these components chose to use.</p><p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/bltc509fc1ff327a7bd/5c98d5f7228f2c42594171f4/blog-opentracing-elastic-apm-7.png" data-sys-asset-uid="bltc509fc1ff327a7bd" style="display: block;margin: auto;"/></p><p>However, OpenTracing only addresses the first of the four architectural components we discussed before. So, where are we today with other components and what the future holds for us?</p><h3>Where are We Today?</h3><p>As we discussed, OpenTracing defines a standard set of tracing APIs for different tracers to implement, which is a great start and very encouraging. However, we still need tracing context standardization and tracing data standardization for them to be compatible and exchangeable with each other.</p><ol><li><strong>OpenTracing API provides a standard set of APIs</strong>. This is pretty much the only standardization we have as of today. There is limitation to the specification too. For example, it does not cover all the programming languages. Nevertheless, it’s a wonderful effort and gaining great traction.</li><li><strong>No standardized tracing context definition yet</strong>. The <a href="https://www.w3.org/2018/distributed-tracing/">W3C Distributed Tracing Work Group</a> is in the process of standardizing the tracing context definition - the W3C <a href="https://www.w3.org/TR/trace-context/">Trace Context specification</a>. The specification defines a unified approach to context and event correlation within distributed systems, and will enable end-to-end transaction tracing within distributed applications across different monitoring tools. Elastic APM supports the W3C Trace Context working group's effort to standardize the HTTP header format for distributed tracing. Our agent implementations closely follow the Trace Context draft specification, and we intend to fully support the final specification.<br /><br />As an example of the incompatibility of the tracing context today, here is an example of the HTTP header used by Elastic APM and Jaeger for trace ID. As you can see, both the name and encoding of the ID are different. When different tracing headers are used, traces will break when they cross the boundaries of the respective tracing tools.<br /><br />Jaeger:<br /><strong>uber-trace-id</strong>: 118c6c15301b9b3b3:56e66177e6e55a91:18c6c15301b9b3b3:1<br /><br />Elastic APM:<br /><strong>elastic-apm-traceparent</strong>: 00-f109f092a7d869fb4615784bacefcfd7-5bf936f4fcde3af0-01<br /><br />There are other challenges too, other than the definition itself. For example, not all HTTP headers are automatically forwarded by service infrastructure and routers etc. Whenever headers are dropped, the trace will break.</li><li><strong>No standardized tracing data definition yet</strong>. As stated by the W3C Distributed Tracing Work Group, the second piece of the puzzle for trace interoperability is “a standardized and extensible format to share trace data -- full traces or fragments of traces -- across tools for further interpretation”. As you can imagine, with many open source and commercial players involved, agreeing on a standard format is not an easy thing. Hopefully we will get there soon.</li><li><strong>Tracers are not runtime-compatible</strong>. Because of everything we discussed above, plus mixed motivation of making their system open and compatible with the rest of the world, tracers are simply not compatible with each other during the runtime today. I can confidently say that it will probably be that way for the foreseeable future.</li></ol><h2>How Elastic APM Works with Other Tracers Today</h2><p>Even though we are not even close to 100% compatibility among tracers yet today, there is no need to be discouraged. Elastic Stack can still work with other tracers in a couple of different ways.</p><ol><li>Elasticsearch as the <strong>scalable backend data store</strong> for other tracers.<br /><br />Not surprisingly, Elasticsearch has been used as the backend data store for other tracers like Zipkin and Jaeger, due to its massive scalability and rich analytics capabilities. Shipping Zipkin or Jaeger tracing data into Elasticsearch is a simple configuration for both of them. Once the tracing data is inside Elasticsearch, you can use the powerful analytic and visualization capability of Kibana to analyze your tracing information and create eye-catching visualizations that provide deep insight into your application performance.</li><li>Elastic <strong>OpenTracing Bridge</strong><br /><br />The Elastic APM OpenTracing bridge allows you to create Elastic APM Transactions and Spans, using the OpenTracing API. In other words, it translates the calls to the OpenTracing API to Elastic APM and thus allows for reusing existing instrumentation. For example, an existing instrumentation done by Jaeger can be simply replaced with Elastic APM by changing a couple of lines of code.<br /><br />Original instrumentation by Jaeger:<br /><br /><pre class="prettyprint">import io.opentracing.Scope;<br />import io.opentracing.Tracer;<br />import io.jaegertracing.Configuration;<br />import io.jaegertracing.internal.JaegerTracer;<br />...<br />private void sayHello(String helloTo) {<br />    Configuration config = ...<br />    Tracer tracer = config.getTracer();<br />    try (Scope scope = tracer.buildSpan("say-hello").startActive(true)) {<br />        scope.span().setTag("hello-to", helloTo);<br />    }<br />    ...<br />}<br /></pre>Replace Jaeger with Elastic OpenTracing bridge:<br /><br /><pre class="prettyprint">import io.opentracing.Scope;<br />import io.opentracing.Tracer;<br />import co.elastic.apm.opentracing.ElasticApmTracer;<br />...<br />private void sayHello(String helloTo) {<br />    Tracer tracer = new ElasticApmTracer();<br />    try (Scope scope = tracer.buildSpan("say-hello").startActive(true)) {<br />        scope.span().setTag("hello-to", helloTo);<br />    }<br />    ...<br />}<br /></pre><br /><br />With this simple change, the tracing data will be happily flowing into Elastic APM, without you having to modify other tracing code. That’s the power of OpenTracing!</li></ol><h2>Elastic APM Real User Monitoring</h2><p>While we mostly focus on the backend services when discussing tracing and context propagation etc., there is great value to start the trace on the client side in the browser. When doing so, you get trace information the moment a user clicks on something in the browser. That trace information represents the “real user experience” of your applications from performance aspect. Unfortunately again, there is no standardized way of forwarding that information today. The W3C group does intend to extend the trace context all the way to the browser in the future.</p><p><a href="/guide/en/apm/server/current/rum.html">Elastic APM Real User Monitoring</a> (RUM) provides exactly that functionality today. The RUM JS agent monitors the real user experience within your client-side application. You will be able to measure metrics such as "Time to First Byte", domInteractive, and domComplete which helps you discover performance issues within your client-side application as well as issues that relate to the latency of your server-side application. Our RUM JS agent is framework-agnostic which means that it can be used with any JavaScript-based frontend application.</p><p></p><p><img src="https://images.contentstack.io/v3/assets/bltf7afce26b89a5b33/blte4644a6aaae06c70/5c645a55e2c86a6c4191bfe1/download" data-sys-asset-uid="blte4644a6aaae06c70"/></p><h2>Summary</h2><p>Hopefully, this blog helped you understand the landscape of Distributed Tracing a bit better and clarified some of the confusions about where we are with OpenTracing today. Let’s call it a wrap with a brief summary:</p><ol><li>Distributed tracing provides invaluable performance insight for microservices.</li><li>OpenTracing is the industry’s first step towards standardization for distributed tracing. We still have a long way to go for full compatibility.</li><li>Elastic APM is OpenTracing compliant.</li><li>Elastic OpenTracing bridge allows instrumentation reuse.</li><li>Elastic Stack is a great scalable long-term storage for other tracers like Zipkin and Jaeger, even without full runtime compatibility today.</li><li>Elastic provides rich analytics for tracing data Elastic or not. Shipping Zipkin or Jaeger tracing data into Elasticsearch is a simple configuration.</li><li>Elastic APM <a href="https://elastic.ac.cn/observability/real-user-monitoring" target="_self">Real User Monitoring</a> (RUM) monitors the real user experience within your client-side application.</li><li>All-in-all, Elastic is a massively scalable, feature-rich, and unified analytics platform for all three pillars of observability - logging, metrics &amp; tracing.</li></ol><p>As always, reach out on the <a href="https://discuss.elastic.co/c/apm">Elastic APM forum</a> if you want to open up a discussion or have any questions. Happy tracing!</p>

elastic-apm-distributed-tracing-large.jpg

Tracing

Observability

elastic-apm-distributed-tracing-small.png

Distributed Tracing, OpenTracing and Elastic APM

Data ingestion

Elasticsearch from the Bottom Up, Part 1

From the very beginning, the Elastic (ELK) Stack — Elasticsearch, Logstash, Beats, and Kibana — has been free and open.

<p>From the very beginning, the Elastic Stack — Elasticsearch, Kibana, Beats, and Logstash — has been free and open. Our approach is not only to make our technology stack available for free, but to make it open — housed in public repositories and developed through a transparent approach with direct involvement from the community.&nbsp;</p><p>Two simple principles — free and open — broke down barriers and enabled many amazing things.</p><h3>Vibrant community&nbsp;</h3><p>Combine great technology with free distribution and open development&nbsp;and you get a vibrant community of doers. Free lowers barriers to adoption, and open<strong> </strong>development fosters collaboration, contribution, and creativity.</p><p>Anyone in the world can download the Elastic Stack and get started immediately — whether they choose to run it on a laptop to develop a new search-powered application or in a data center to monitor infrastructure and protect against security threats. They can see (and contribute to) the code, &nbsp;share feedback, questions, or requests directly with our engineering team, and engage with their peers in the community. This is a powerful force multiplier.&nbsp;</p><h3>Better products. Newer directions.&nbsp;</h3><p>Combine free and open with a creative, passionate, and engaged community, and not only does it make the products better, but the community often blazes the trails that take the products in new and interesting directions. Our community is a source of constant inspiration for us, and is the source of so many of the great ideas that move us forward.</p><p>In the early days of Elasticsearch, the ingenuity of the community gave rise to Logstash, Kibana, and Beats. Together, Elasticsearch, Logstash, and Kibana became the ELK Stack (now Elastic Stack) and sowed the seeds of a new use case (logging). And now we have a dedicated Observability solution. Then security practitioners took notice and started using the ELK Stack to power their security analytics, and now we have a free and open SIEM. This has continued all these years with community-created and -inspired features, extensions, plugins, and use cases.</p><h3>Free and open is in our DNA&nbsp;</h3><p>Free and open principles are ingrained into who we are and how we progress. We want our products to be used to learn, to develop, and&nbsp;to be run in production at scale. And that’s why many of our core features, products, and solutions are free. For example:</p><p><a href="/what-is/elastic-stack-security"><strong>Free security features</strong></a> developed natively in the stack&nbsp;are critical to ensure that every cluster is protected. But we go far beyond that, providing role-based access control and true multitenancy for Kibana, all for free. We believe that products developed in the open are more secure.&nbsp;</p><p><a href="/what-is/kibana-lens"><strong>Kibana Lens</strong></a><strong>,</strong> for example, which we introduced as a beta in the 7.5 release of the Elastic Stack, made it even easier to visualize data stored in Elasticsearch. It channeled a lot of the community feedback we saw on the Kibana repo over the years.&nbsp;</p><p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt0ea7bdf603fb6318/5f8a0bb4f6c586323f8b3d9b/blog-elastic-free-open-1.gif" width="617" height="350" style="width: 617px; height: 350px;"/></p><p><a href="/maps"><strong>Elastic Maps,</strong></a> which we made generally available in version 7.3 of the Elastic Stack, opened up new ways to visually explore location data in Elasticsearch. This was based&nbsp;on a lot of geo work done in Elasticsearch.&nbsp;</p><p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/bltdf19428aa0eeba13/5f8a0c10dcf0e74b7cbf6f1d/blog-elastic-free-open-2.gif" width="625.1259259259259" height="352" style="width: 625.1259259259259px; height: 352px;"/></p><p><a href="/what-is/kibana-canvas"><strong>Canvas</strong></a>, which became generally available in 6.5, lets you turn your Elasticsearch data into dynamic presentation style dashboards, and bring your unique style to how you tell the story of your data.&nbsp;</p><p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/bltd3bcb6c0c16af459/5c3047b6e71ce40c6e4ad94b/airport4-upd.gif"/><br /></p><p>We’ve even applied this open philosophy to develop turnkey solutions that solve our users’ key challenges. Each of these solutions has components that are built out in the open, and anyone can get started for free.</p><ul><li><a href="/enterprise-search"><strong>Elastic Enterprise Search</strong></a> makes it possible to implement powerful, modern search experiences complemented by free and open developer tools.</li><li><a href="/observability"><strong>Elastic Observability</strong></a> brings together our free and open log monitoring, metrics, APM, and uptime monitoring products into a single powerful solution.</li><li><a href="/security"><strong>Elastic Security</strong></a> combines a free SIEM with an open community, open roadmap, and open data model.&nbsp;</li></ul><p>These are just a handful of examples. There is so much more to explore.&nbsp;</p><h3>Get started now</h3><p>Everyone has access to a fast and frictionless getting started experience with the Elastic Stack. You can get started with the Elastic Stack in a few different ways.</p><ul><li>It takes only 3 minutes to <a href="https://elastic.ac.cn/cloud/elasticsearch-service/signup">spin up a free trial in Elastic Cloud</a> — meaning in less time than it takes to make a cup of coffee you can have your very own Elastic Stack ready to go.</li><li>Like to test things out locally? No problem! You can always <a href="https://elastic.ac.cn/start">download the latest versions and run them yourself</a>.</li><li>Want to see the Elastic Stack in action and learn how companies around the world use Elastic products and solutions to tackle challenging business and technology problems? Check out our <a href="https://elastic.ac.cn/videos/">library of instructional how-to webinars as well as recorded customer case studies</a>.</li></ul><p>We believe that the best products are built in the open, in collaboration with a passionate group of developers and users who push the bounds of what’s possible. That means we need you! <a href="https://discuss.elastic.co">Hop into our forums</a> and engage directly with our engineers or check out what’s happening in our <a href="https://github.com/elastic/elasticsearch">public</a> <a href="https://github.com/elastic/kibana">GitHub</a> <a href="https://github.com/elastic/beats">repositories</a>. To us, contributing isn’t only about writing code — it’s about bringing new ideas, showing others what’s possible, and creating a community where everyone can learn and grow.</p>

The Elastic Stack: Free. Open. Limitless. Open source. Again.

The Elastic Stack: Free. Open. Limitless. <em>Open source. Again.</em>

Alerting spans all our products and use cases. Find out more about current alerting capabilities in the Elastic Stack and exciting stuff in the works.

<p>Alerting is fundamental to Elastic's use cases. Since <a href="/guide/en/kibana/7.3/watcher-ui.html">Watcher</a> (our original suite of alerting features for Elasticsearch) was <a href="/blog/watcher-you-know-for-alerting-coming-soon">introduced back in 2015</a>, we’ve received a lot of feedback that’s helped refine our understanding of what an alerting system needs to be and what the user experience should entail. The purpose of this post is to summarize some of the key things we’ve learned, how it’s influenced our work in 2019, and what the future holds for alerting for the Elastic Stack.
</p> <strong>
<h2>What have we learned?</h2></strong>
<p>Four years of alerting at Elastic has created a wealth of knowledge about alerting systems. I’ve tried to synthesize what we’ve learned into three forward-looking observations: we see alerts in every use case; we need to make sense of them across use cases; and alert detection and response are growing more sophisticated. These learnings shape our thinking about the future of alerting.
</p> <strong>
<h3>Alerting everywhere</h3></strong>
<p>Alerting cuts across all our products and use cases. If you have live data, there’s a case for alerting. This is why we built Watcher and why it’s been successful. Yet when we look across use cases it’s clear there’s no one-size-fits-all alerting.
</p>
<p>From products like Elastic Logs, SIEM, APM, Uptime, Infrastructure, and Maps, to features like monitoring and machine learning to a multitude of Kibana dashboards, alerts and notifications play a critical role, yet each has unique needs for detecting conditions, expressing them, and showing them in context. Effective alerting and monitoring requires deep integration with a product. As the stack and its uses have evolved, it’s become clear that Elasticsearch alerting needs a complement that allows for tightly integrated, <strong>rich expression of alerts within each use case</strong>.
</p> <strong>
<h3>Making sense of alerts</h3></strong>
<p>The corollary of “alerting everywhere” is that as these different uses cases generate alerts, alerts becomes their own source of data and create opportunities for understanding systems and their state. Or, as the Site Reliability Engineering (SRE) community might say, there are opportunities to improve the observability of an overall system.
</p>
<p>Each use case interprets data its own way, and alerts show different facets of a situation. The right response to an incident often hinges on data from multiple sources, and correlating different types of alerts and events to understand a situation. In some domains, like SIEM, higher level alerts are triggered from patterns in lower level alerts.
</p>
<p>As the Elastic Stack increasingly becomes home to more and more use cases, an alerting system done right will not just generate alerts but also help you <strong>make sense of them across use cases</strong>. For instance, Uptime alerts may show a service outage, APM alerts explain which transaction caused it, while monitoring alerts pinpoint why it happened. An alerting system should provide context, enable correlation, and improve awareness — for both people and machines.
</p> <strong>
<h3>Detection and action</h3></strong>
<p>The corollary of “making sense of alerts” is that with a more observable system, you can detect more complex conditions and take more sophisticated actions. Increasingly this goes beyond what we traditionally think of as alerting.
</p>
<p>Alerting is usually focused on detecting a condition and then getting a human’s attention — and often it ends there. Looking at the bigger picture though, an alerting system can be thought of as part of a control or feedback loop: observe, detect a condition, take some action, observe again.
</p>
<p>Today an ‘action’ usually involves notification — putting a human in the loop to control the system and try to correct it. But as system insight improves, the ‘action’ can take on more control, usually under human supervision. This could be a semi-autonomous system governed by a two way conversation (chatbots for example), or a fully autonomous system, as we see in the trend towards auto-scaling, self-healing, and self-optimizing applications.
</p>
<p>An alerting system needs to support <strong>sophisticated detection and actions</strong>, acknowledging that ‘detection’ can be more than a query to Elasticsearch, and ‘action’ is becoming more than sending an email or calling a webhook.
</p> <strong>
<h2>Applying what we’ve learned</h2></strong>
<p>We decided back in the <a href="https://github.com/elastic/kibana/issues/24214">fall of 2018</a> that we needed alerting to support the three observations above.
</p>
<p>We also decided that having alerts as first-class entities in Kibana would be the best way to do this:
</p>
<ul>
	<li><strong>Alerting everywhere</strong>: rich alerting integrations across our products, at the plugin, API, and UI levels</li>
	<li><strong>Making sense of alerts</strong>: providing an intuitive interface across alerting types</li>
	<li><strong>Detection and action</strong>: sophisticated detection and action mechanisms via Kibana plugins</li>
</ul>
<p>We also know from Watcher that alerting must scale to production alert loads and be highly available and reliable. APIs, UIs, and plugin/library contracts to support the three observations must be built on a solid and scalable base. All together we see four layers to Elastic’s alerting system:
</p>
<p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt48e66e76c9686633/5d8247931ec6f77b9b7f21db/alerting-layers-blog.png" data-sys-asset-uid="blt48e66e76c9686633" alt="Layers of the Elastic Stack alerting system" style="display: block; margin: auto;">
</p><figcaption>An overview of Elastic’s alerting system
</figcaption>
<p>In 2019 we’ve been laying the foundation of the new alerting system in Kibana.
</p>
<p>In January, we added <a href="https://github.com/elastic/kibana/issues/23632">Task Manager</a> as part of the 6.7 release. This gave Kibana background scheduling with persistent tasks that can be distributed across several Kibana instances for scalability and availability. Alert base layer components like Task Manager can power more than just alerting. For example, Task Manager could provide a better scheduled report experience in Kibana.
</p>
<p>Then in June, we added <a href="https://github.com/elastic/kibana/pull/37042">two new sets of APIs to Kibana</a>: the alerts API and the actions API.
</p>
<p>The actions API lets Kibana register and fire actions, and provides a simple contract for defining your own, making it easy to customize. The initial release also had a few example actions for logging, Slack, and email notifications.
</p>
<p>The alerting API allows Kibana to register forms of detection as ‘alert types’, and then run these checks on a schedule using the Task Management system. Like actions, there’s a simple alerting contract: if you can express it in a JavaScript function that runs on the Kibana server, it can power an alert.
</p>
<p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt0172899948524258/5d8247b1631ee43e546a1c3a/alerting-geo-alert-blog.gif" data-sys-asset-uid="blt0172899948524258" alt="Geo boundary alert plugin" style="display: block; margin: auto;">
</p><figcaption>A proof-of-concept geo-boundary alert plugin written in v7.3. This tracks 1600 transit vehicles in a single alert, writing the entries into and exits from the red polygon into a log file. The entry and exit of the purple vehicle (#8341) is highlighted.
</figcaption>
<p>Elastic Stack 7.4 is focused on filling out the lower levels of the alerting system: we’re hardening the APIs; adding support for <a href="https://github.com/elastic/kibana/pull/41389">security</a> and <a href="https://github.com/elastic/kibana/pull/42081">spaces</a>; and adding a few more built-in actions like <a href="https://github.com/elastic/kibana/pull/41592">indexing</a>, <a href="https://github.com/elastic/kibana/pull/43538">webhooks</a> and <a href="https://github.com/elastic/kibana/pull/43395">pager duty</a>.
</p> <strong>
<h2>What’s next?</h2></strong>
<p>Development of Kibana’s alerting system has been in full swing for the last couple of months, and that’s going to continue through the 7.x release cycle. Our plan is to roll out the system in three phases.
</p>
<p>The first phase has been going on for much of 2019: laying the foundation. It focuses on scalable task management and scheduling, contracts for alerting and action, and APIs.
</p>
<p>We’re now moving into the second phase, where <em>different use cases can integrate</em> the alerting system at the API and library levels. This also includes designing and building a UI in Kibana as part of making sense of alerts and validating it with specific use cases (like <a href="/infrastructure-monitoring">monitoring</a>, <a href="/uptime-monitoring">uptime</a>, or <a href="/siem">SIEM</a> for example).
</p>
<p>The third phase will extend the "alerts everywhere" and "detection and action" themes by allowing <em>user defined alerts</em> throughout Kibana, whether through templated alerts or even expression based alerts using something like <a href="/guide/en/kibana/7.3/canvas-function-reference.html">Canvas expressions</a>.
</p>
<p>The end goal is a system that satisfies our vision of alerting in the Elastic Stack:
</p>
<ul>
	<li><strong>Alerting everywhere</strong>: Alerts are a first class, space-aware entity within Kibana. This makes it possible to segment the creation and viewing of alerts across groups, and allows rich integration of alerting in products like SIEM, Monitoring, and Uptime (to name a few). Alerting complements and works alongside Watcher, it does not replace it.</li>
</ul>
<ul>
	<li><strong>Making sense of alerts</strong>: Rich alerting integrations will be accompanied by Kibana UI that provides comprehensive views across alert types, as well as tools for correlating and making sense of alert history.</li>
</ul>
<ul>
	<li><strong>Detection and action</strong>: The APIs and plugins are designed so that a detection or action mechanism can be anything provided it can be expressed in JavaScript running on the Kibana server. This leaves plenty of room for the sophisticated detections and actions that will appear in Kibana through products like SIEM or our observability solutions.</li>
</ul>The full alerting system won’t be realized overnight, but with the foundation in place you’re going to see aspects of this new alerting vision appear in upcoming releases of Kibana. We’re looking forward to building the system out, getting your feedback, and pushing the limits — and you can follow our progress in <a href="https://github.com/elastic/kibana/projects/26">GitHub</a>!

alerting-blog-banner.jpg

alerting-blog-thumb.jpg

Alerting in the Elastic Stack

Learn how SAP Concur has grown their logging solution from an IT necessity to a DevOps strategy, enabling easy end-to-end service ownership.

<p><em>This post is a recap of a community&nbsp;talk given at <a href="/elasticon/conf/2018/sf">Elastic{ON} 2018</a>. Interested in seeing more talks like this? Check out the conference archive or find out when the <a href="/elasticon">Elastic{ON} Tour</a> is coming to a city near you.</em>
</p>
<p>If you've ever entered an expense report, there's a good chance you'd done it through SAP Concur. With over 45 million users spanning more than 150 countries (including 70% of the Fortune 500), Concur is a top travel and expense solution. In 2016 alone, the SaaS offering processed over $87 billion USD in expenses, meaning over 2.4 million receipts and $187 million USD in invoicing every day. That may seem like a lot of line items for accounting, but it creates even more log lines for a logging solution to handle on a daily basis.
</p>
<p>Concur has been around for over 20 years, and as their product offerings have grown and evolved, so has their logging solution. Not just in the technology that they use, but also in the scope and intent of its usage. Initially a SQL-based solution used for simple log storage, their current logging solution — built on the Elastic Stack — helps promote end-to-end application ownership, and aligns development, testing, and operations. And in the future, Concur's LAMA (Logging, Alerting, Monitoring, and Analytics) team plans to use Elastic machine learning for operational analytics and insight as well as for automating rollouts and rollbacks. They've taken great leaps in logging, but they didn't get from log storage to analytics overnight.
</p>
<p>Originally built on a relational database, their logging solution ingested log data as XML via RabbitMQ, and their users loved that they could easily query for logs using SQL. But as the popularity of the service grew, so did usage. As peak ingest grew to 200 GB/day — with rates in upwards of 1,500 docs/sec — the service reached its limits, and performance-based service lags could force users to wait up to 20 minutes for a log to be available in the system. In response, all the logging team was able to do was put their database on more powerful hardware, which was an unsustainable process. What they needed was horizontal scalability, so they set out to find a better solution.
</p>
<p>After researching Elasticsearch and hearing about different <a href="/customers/success-stories">success stories</a> from companies in similar situations, Concur chose the Elastic Stack as their logging solution. It was fast, it was powerful, and it was scalable — and (possibly more) importantly to their internal users, it had a visualization component that their users loved. Previously, different teams would build their own interfaces and dashboards, often incurring licensing fees for the tools they had to use to get the job done. With Kibana, Concur had a unified visualization solution, removing the need for homegrown&nbsp;or 3rd party visualization solutions.
</p>
<p>The first implementation of Elastic was with Elasticsearch 1.1 and Kibana 3, with ingest coming from Logstash, RabbitMQ (same as they'd used with the SQL solution), and Fluentd. The logging team was also able to build their own alerting plugin (a benefit of the open source nature of Elastic), as one did not yet exist within the Elastic Stack. Between the increased speed of Elasticsearch, the visualizations of Kibana, and the alerting features of their homegrown Watcher plugin, service adoption increased across Concur and ingest skyrocketed to 5,000 doc/sec. That's something their SQL solution couldn't have come close to handling.
</p>
<h2>Growing from Solution to Strategy with Elastic</h2>
<p>Since that initial implementation, Concur's logging solution has grown with the Elastic Stack. In 2015, they upgraded to Elasticsearch 2.3 and Kibana 4.5, purchased a <a href="/subscriptions">Gold subscription</a>, and began using <a href="/beats">Beats</a> (as a replacement for Fluentd), <a href="/what-is/kibana-alerting">Watcher</a> (to replace their homegrown solution) and <a href="/what-is/elastic-stack-security">Shield</a> (for security). They also built another custom plugin, this time a custom aggregation UI. As their logging solution improved, so did adoption, and by 2017, their ingest rate was up to 60,000 doc/sec (4TB/day).
</p>
<p>After attending <a href="/elasticon/conf/2017/sf">Elastic{ON} 2017</a>, Concur upgraded again, this time to take advantage of cross cluster search, improved security (needed to ensure GDPR compliance), and other new <a href="/elastic-stack">Elastic Stack features</a> they'd learned about during the conference. Using cross cluster search, they were able to break up their monolithic cluster into multiple, smaller clusters spread across multiple regions. This version upgrade —  as well as their move to a Platinum subscription — has helped them to establish the environment they use today, with a variety of ingest sources, Elasticsearch clusters across multiple regions (5TB/day in the US), and Kibana dashboards used by operations, SREs, support, executive leadership, and more. And all that is managed by a LAMA Team made up of six engineers and two managers.
</p>
<p>Learn about how Concur went from log storage to ownership enablement by watching <a href="/elasticon/conf/2018/sf/elastic-at-sap-concur-driving-the-journey-to-devops-and-end-to-end-ownership">Elastic @ SAP Concur: Driving the Journey to DevOps and End-to-End Ownership</a> from <a href="/elasticon/conf/2018/sf">Elastic{ON} 2018</a>. You'll also learn how they enabled one-click logging service deployment, how they configured mappings (non-dynamic) and fields for over 200 teams, and what their plans are for leveraging the power of Elastic machine learning.
</p>
<p><a href="/elasticon/conf/2018/sf/elastic-at-sap-concur-driving-the-journey-to-devops-and-end-to-end-ownership"><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/bltc1360b51d3d161a4/5c57dc114c9f4feb0b4c585a/concur_elasticon_thumb.png" data-sys-asset-uid="bltc1360b51d3d161a4" alt="" style="display: block; margin: auto;"></a>
</p>

concur_elasticon_banner.jpg

concur_elasticon_thumb2.png

SAP Concur: Elastic Logging as a DevOps Strategy

We’re excited to announce a new alerting framework that delivers a first-class alerting experience natively within the SIEM, Uptime, APM, and Metrics applications as part of the Kibana 7.7 release.

<p>We’re excited to announce a new alerting framework that delivers a first-class alerting experience natively within the SIEM, Uptime, APM, and Metrics applications as part of the Kibana 7.7 release.
</p><p>Alerting is a fundamental use case across the Elastic Stack, which is why we’re making it part of the core experience within Kibana. Whether you are monitoring application transactions or tracking brute force login attempts, our goal is to provide a tailored experience that allows you to build powerful alerts in the normal flow of your task. The new alerting framework is built from the ground up and designed to offer more than just convenient interfaces. We understand the need to go beyond just notifying people which is why we’ve also incorporated the ability to trigger predefined actions that can do anything from sending an email to using brand new third-party integrations with platforms like Slack and PagerDuty.
</p><p>The new alerting framework is being introduced as a beta in the 7.7 release of Kibana and is available immediately on the <a href="https://elastic.ac.cn/cloud/elasticsearch-service">Elasticsearch Service</a> on Elastic Cloud, or for <a href="https://elastic.ac.cn/downloads">download</a>.
</p><div class="embed-responsive embed-responsive-16by9 shadow m-t-40 m-b-40"><video autoplay="autoplay" playsinline="playsinline" muted="muted" loop="loop"><source src="https://assets.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt8b2976cc9e2baf68/5e9c00c550e659126c8d7371/video-alerting-metrics-apm-siem.mp4" type="video/mp4"></video>
</div><p><br>
</p><strong><h2>Alerting everywhere you need it to be</h2></strong><p>Detecting and understanding significant signal shifts is a fundamental need that cuts across all use cases for organizations that build, maintain, and evolve digital systems. In application performance monitoring (APM), for example, you want to detect increases in latency in your application’s responsiveness as well as in error rates, because both potentially have a direct impact on your service and user experience. Furthermore, you want to be able to see this at an infrastructure level with metric alerts such as CPU or memory usage spikes or even service and network downtime. Each of these could result in application performance degradation if not acted on in a timely manner. In addition to one-time events, there is also a need to detect recurring patterns within logs in an effort to understand and proactively avoid future situations. Pivoting to a system security use case, the need for real-time alerting is vital to spot threats like distributed service requests or large data transfers at suspicious times. More broadly, in business analytics, early detection alerts of dips or spikes that correlate with core KPIs driving company performance may well contribute to the success of your strategy and execution.&nbsp;
</p><p>It was with these use cases and many more in mind that in September 2019 we shared <a href="https://elastic.ac.cn/blog/alerting-in-the-elastic-stack">our vision for a new alerting framework</a> in the Elastic Stack. A key part of that vision included three observations gathered from years of deep customer engagement that ultimately lay the foundation and the approach we are taking in reimagining alerts within the Elastic Stack. Those observations are:&nbsp;
</p><ul>
	<li>Alerting needs to be everywhere</li>
	<li>Making sense of alerts is critical</li>
	<li>Alerting should be about detection <em>and</em> action</li>
</ul> <strong><h2>Introducing a new alerting framework for the Elastic Stack in 7.7</h2></strong><p>We are excited to announce with 7.7 a major step forward in delivering on our vision of alerting within the Elastic Stack that is tightly integrated within the Elastic Observability and Security solutions and makes integrations that matter to people and companies easy to configure.
</p><ul>
	<li><strong>Alerting everywhere</strong>: Kibana 7.7 introduces ubiquitous alerting for Elastic Observability, Elastic Security, and the Elastic Stack. Users can now create alerts directly from within the SIEM, APM, Metrics, and Uptime applications as well as for any index. This gives users the power to address their alerting needs in the context of their unique use case. The new alerting framework is designed around the core principle of seamless usability and interoperability across solutions with Kibana as their platform.</li>
</ul><ul>
	<li><strong>Making sense of alerts</strong>: Detecting the signal and eliminating the noise is a core focus. For example, in addition to the solution-specific options for creating alerts, Kibana 7.7 provides a single dedicated master view for managing, listing, searching, and editing all alerts in one place. The new alerting framework helps make sense of alerts throughout their lifecycle, from creation, for example, by offering a visual preview, to taking action.&nbsp;</li>
</ul><ul>
	<li><strong>Detection </strong><strong><em>and</em></strong><strong> action</strong>: The new alerting framework focuses on enabling alerts to do more than simply get a human’s attention. With Kibana 7.7 the goal is to seamlessly pass the alerting context that you need into the systems and workflows that matter the most to you, by making integrating a simple and easy process. This is why you’ll see built-in options for integrating with third-party solutions like Slack and PagerDuty as well as webhooks for customizing additional integrations.&nbsp;</li>
</ul><div class="embed-responsive embed-responsive-16by9 shadow m-t-40 m-b-40"><video autoplay="autoplay" playsinline="playsinline" muted="muted" loop="loop"><source src="https://assets.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt7697389d15f76f4c/5e9c0196db124263e8afcaee/video-alerting-pagerduty.mp4" type="video/mp4"></video>
</div><p><strong></strong>
</p><h2><strong>More user-friendly, more powerful, more actionable</strong></h2><p>The new alerting framework lives and breathes in Kibana. This choice, and the principles on which it is built, define the framework in a number of ways.
</p><p>First, extra care was taken to provide an intuitive and user-friendly experience when it comes to alert creation and management. As a result, creating and editing alerts is achieved using dropdowns and prompts making it easy for anyone to use the first time. <em>Alerting everywhere</em> means that we offer an array of tailored ways to create advanced, contextual alerts in Kibana. This includes generating multiple alert instances via a single alert definition so one effort translates into multiple outputs. In addition, the user experience is consistent across the APM, Metrics, Uptime and SIEM apps, meaning no matter where someone creates an alert in the Elastic Stack, they’ll find the same easy-to-use controls.
</p><div class="embed-responsive embed-responsive-16by9 shadow m-t-40 m-b-40"><video autoplay="autoplay" playsinline="playsinline" muted="muted" loop="loop"><source src="https://assets.contentstack.io/v3/assets/bltefdd0b53724fa2ce/bltd0e217d74c286bd6/5e9c020dc81c45292c0d4c41/video-alerting-multiple-actions.mp4" type="video/mp4"></video>
</div><p>The power of the new alerting framework goes much deeper than its interfaces. Also new in this 7.7 release is a distributed task manager which delivers greater performance and scalability. This means that you can easily scale out alerting capacity simply by adding additional Kibana instances.
</p><p>Importantly, the new alerting framework comes with a number of integrations which you can use to create actions. The 7.7 release introduces third-party connectors for triggering alerts to email, Slack, PagerDuty, and webhooks. Elastic connectors allow you to easily write alerts to indices and server logs. Setting up these connectors takes only a few seconds. Just enter the endpoint and key or credentials on the designated input fields on the UI, as provided by your email, Slack, PagerDuty, or webhook account, and start channeling your alerts and their context across the systems in your workflows.&nbsp;
</p><p>Finally, the new alerting framework in Kibana supports multitenancy. This means that you can organize your alerts into Kibana Spaces, and the framework will soon fully support Kibana’s enhanced authorization model.
</p> <strong><h2>Get started with the new alerting framework in Kibana today</h2></strong>We are thrilled to release this next phase of our vision for alerting in the Elastic Stack&nbsp;into public beta. To test it out for yourself right now, spin up a cluster on <a href="https://info.elastic.co/elasticsearch-service-trial-30d.html">Elasticsearch Service</a> or <a href="https://elastic.ac.cn/downloads">install the latest version</a> of the Elastic Stack. Already running Kibana? Simply upgrade your clusters to 7.7 to get hands-on with the new alerting capabilities. Interested in learning more about our integration with PagerDuty? <a href="https://elastic.ac.cn/about/partners/pagerduty">Learn more on our partner page</a>.&nbsp;

elastic-blog-thumbnail.png

Introducing the new alerting framework for Elastic Observability, Elastic Security, and the Elastic Stack

Learn how to apply user annotations, a new feature in Elastic machine learning, to augment your ML results with the information you have about your data.

<p>User annotations are a new machine learning feature in Elasticsearch available from 6.6 onwards. They provide a way to augment your machine learning jobs with descriptive domain knowledge. When you run a machine learning job, its algorithm is trying to find anomalies —&nbsp;but it doesn’t know what the data itself is about.&nbsp;The job wouldn't know, for&nbsp;example, whether it was dealing with&nbsp;CPU usage or network throughput. User annotations offer a way to augment the results with the knowledge you as a user have about the data.&nbsp;
</p><p>In this blog post, we’ll&nbsp;show you how user annotations work and how to apply them to different use cases. We’ll be&nbsp;analysing&nbsp;data&nbsp;from <a href="https://apps.tirol.gv.at/hydro/#/Wasserstand/?station=201525" target="_BLANK">Hydro Online</a>&nbsp;—&nbsp;an open data portal run by Austria’s Tyrolean local government. Hydro Online&nbsp;offers an interface to investigate weather sensor data&nbsp;such as&nbsp;rainfall accumulation, river height, or snowpack totals. As described in one of our <a href="/blog/importing-csv-and-log-data-into-elasticsearch-with-file-data-visualizer">previous blog posts</a>, the File Data Visualizer offers a robust way to ingest data from CSV data, as is found in this case.
</p><h2>Usage</h2><p>Let’s start with a single metric job that analyses river height measurements of the River <i>Grossache</i> going through the village of <i>Kössen</i>. Once the job is created, the Single Metric Viewer can be used to add annotations to the results of the analysis. Simply drag over a time range in the chart to create an annotation. A flyout element will pop up to the right, which allows you to add a custom description. In the example below, we annotate an anomalous river height (a&nbsp;major flooding occurred on that date). By creating the annotation, you can make that knowledge available to other users.
</p><p><img src="https://images.contentstack.io/v3/assets/bltf7afce26b89a5b33/bltb1c486237bfd181d/5cb5a5433aebf35f29365dab/download" data-sys-asset-uid="bltb1c486237bfd181d" alt="" "="">
</p><p>The annotation is visible&nbsp;in both the chart itself&nbsp;as well as in the Annotations table below it. The label visible in the first column of the table can be used to identify the annotation in the chart. These labels are dynamically&nbsp;created&nbsp;for the annotations on display. When hovering over a row in the Annotations table, the corresponding annotation will also be highlighted in the chart above it.
</p><p>The annotations created for each job can also be accessed from the Job Management page, where they are displayed in their own tab by expanding a row in the list of jobs. Each annotation in the table includes a link in the right hand column, which takes you back to the Single Metric Viewer with a focus on the time range covered by the annotation. These permalinks can also be shared with others. This means you can use annotations to create bookmarks on particular anomalies to revisit them later on.
</p><p><img src="https://images.contentstack.io/v3/assets/bltf7afce26b89a5b33/bltccdf9900f2d7720d/5cb5a564b790c01122391df1/download" data-sys-asset-uid="bltccdf9900f2d7720d" alt="" "="">
</p><p>If there are multiple annotations covering the same time range, annotations will be vertically distributed in the chart to avoid overlap. To edit or delete an annotation, simply click on it in the chart. The flyout element will open again to the right where you’ll be able to edit the text or delete the annotation. From 6.7 onwards, this can also be done by using the edit button in the Annotations table, making&nbsp;this functionality available from the Job Management page too.
</p><p><img src="https://images.contentstack.io/v3/assets/bltf7afce26b89a5b33/bltb137bc3438e7860b/5cb5a583791abe5a29a1d4f5/download" data-sys-asset-uid="bltb137bc3438e7860b" alt="" "="">
</p><p>Now that we've covered the basic functionality of how to create and work with user annotations, let's&nbsp;move on to some more use cases.
</p><h2>Using annotations to verify expected anomalies</h2><p>Annotations can be used to supply a ground truth to verify if a machine learning job comes up with expected results. In the following example, we are&nbsp;again looking at the river level data from <a href="https://apps.tirol.gv.at/hydro/#/Wasserstand/?station=201525" target="_BLANK">Hydro Online</a> and are now aiming to&nbsp;automatically&nbsp;overlay historic events as annotations on the anomaly results. As a data scientist, for example, your work might include obtaining and preparing both the source data you want to analyse as well as the data set to verify the results.
</p><p>For our own analysis, we need the raw dataset.
</p><p><img src="https://images.contentstack.io/v3/assets/bltf7afce26b89a5b33/blte10b1840268283bc/5cb5a5a382d7ee1622f86b1a/download" data-sys-asset-uid="blte10b1840268283bc" alt="" "=""><br>
</p><p>Luckily, in this case, in addition to investigating data via the web interface, we can also download historic data&nbsp;for further analysis. For this example we’ll use the <a href="https://ehyd.gv.at/eHYD/MessstellenExtraData/owf?id=202382&file=8">river height data</a> of the River <i>Grossache</i> measured at the <i>“Huette”</i> measurement point. The annotations covering the desired ground truth will be created from a <a href="https://www.tirol.gv.at/fileadmin/themen/umwelt/wasserkreislauf/wasserstand/downloads/Tafeln_Koessen_Huette.pdf" target="_BLANK">document describing severe river heights and floods</a>.
</p><table style="width: 100%">
<tbody>
<tr>
	<td style="border: 0px; width: 50%">
		<img src="https://images.contentstack.io/v3/assets/bltf7afce26b89a5b33/bltf6aa0ed4408c2c99/5cb5a6dc3e0de5c323b87308/download" data-sys-asset-uid="bltf6aa0ed4408c2c99" alt="" "="">
	</td>
	<td style="border: 0px; width: 50%">
		<img src="https://images.contentstack.io/v3/assets/bltf7afce26b89a5b33/blt7c57cde13dd3e014/5cb5a5d6791abe5a29a1d4fb/download" data-sys-asset-uid="blt7c57cde13dd3e014" alt="" "="">
	</td>
</tr>
</tbody>
</table><p>In addition to&nbsp;using the UI previously&nbsp;described,&nbsp;machine learning annotations are&nbsp;stored as documents in a separate standard Elasticsearch index.&nbsp;Annotations can also be created programmatically or manually using standard Elasticsearch APIs. Annotations are stored in a version-specific index, and should be accessed via the aliases <code>.ml-annotations-read</code> and <code>.ml-annotations-write</code>. For this example, we'll add annotations to reflect the historic river events before creating our machine learning job.
</p><pre class="prettyprint">{
   "_index":".ml-annotations-6",
   "_type":"_doc",
   "_id":"DGNcAmoBqX9tiPPqzJAQ",
   "_score":1.0,
   "_source":{
      "timestamp":1368870463669,
      "end_timestamp":1371015709121,
      "annotation":"2013 June; 770 m3/s; 500 houses flooded.",
      "job_id":"annotations-leukental-4d-1533",
      "type":"annotation",
      "create_time":1554817797135,
      "create_username":"elastic",
      "modified_time":1554817797135,
      "modified_username":"elastic"
   }
}
</pre><p>We’ll now create a machine learning job to find anomalies in maximum river height&nbsp;using a&nbsp;name that matches the <code>job_id</code> field from the annotation above so that it picks up the manually created annotations. This is how this job looks in the Single Metric wizard once we ingest&nbsp;the historic river data into an Elasticsearch index:
</p><p><img src="https://images.contentstack.io/v3/assets/bltf7afce26b89a5b33/blt1787a021b6855637/5cb5a71fb790c01122391e1f/download" data-sys-asset-uid="blt1787a021b6855637" alt="" "="">
</p><p>The important bit here is that the job name we chose matches the one used for the annotations. Once we run the job and move to the Single Metric Viewer, we&nbsp;should see annotations corresponding to the anomalies in river height that the machine learning job detected:
</p><p><img src="https://images.contentstack.io/v3/assets/bltf7afce26b89a5b33/blt28e9133b42e9986b/5cb5a73830313775298d66b2/download" data-sys-asset-uid="blt28e9133b42e9986b" alt="image9.png">
</p><p>This technique offers a great way to verify if the analysis you’re running is valid when compared to pre-existing validation data stored as annotations.
</p><h2>Annotations for system events</h2><p>In addition to the user-generated annotations&nbsp;above, the machine learning backend automatically creates annotations in some circumstances for system events.
</p><p><img src="https://images.contentstack.io/v3/assets/bltf7afce26b89a5b33/blt26871e9e546a8b1c/5cb5a74c3e0de5c323b87310/download" data-sys-asset-uid="blt26871e9e546a8b1c" alt="" "="">
</p><p>The screenshot above shows an example of an automatically created annotation. In this case, a real-time machine learning job was run, but data ingestion wasn’t able to keep up with the ingestion rate required for the job. This meant&nbsp;documents were added to the index after the job had run its analysis on the bucket. The automatically created annotation highlights this issue that was previously hard to spot and debug. The annotation text features detail&nbsp;the identified problem and provide a suggestion on how to solve it —&nbsp;in this case increasing the query_delay setting.
</p><h2>Alerting integration</h2><p>Even before the availability of user annotations for machine learning, you could&nbsp;use <a href="https://elastic.ac.cn/guide/en/x-pack/current/watcher-getting-started.html">Watcher</a> to create alerts based on anomalies identified by machine learning jobs. While that is a great improvement when compared to alerting on basic thresholds, the alerts may&nbsp;be&nbsp;too granular for&nbsp;the target group that receives the alerts. As a user of machine learning jobs, annotations can give you a way to curate what gets triggered as Watcher alerts and what gets passed on to other stakeholders. Since annotations are stored in their own Elasticsearch index, you can use Watcher to simply react to newly created documents in that index and trigger notifications. Watcher can also be configured to <a href="https://elastic.ac.cn/guide/en/watcher/current/configuring-slack.html">send alerts to a Slack channel</a>. The following configuration gives you an example on how to create a watch to trigger Slack messages when a new annotation gets created:
</p><pre class="prettyprint">{
 "trigger": {
   "schedule": {
     "interval": "5s"
   }
 },
 "input": {
   "search": {
     "request": {
       "search_type": "query_then_fetch",
       "indices": [
         ".ml-annotations-read"
       ],
       "rest_total_hits_as_int": true,
       "body": {
         "size": 1,
         "query": {
           "range": {
             "create_time": {
               "gte": "now-9s"
             }
           }
         },
         "sort": [
           {
             "create_time": {
               "order": "desc"
             }
           }
         ]
       }
     }
   }
 },
 "condition": {
   "compare": {
     "ctx.payload.hits.total": {
       "gte": 1
     }
   }
 },
 "actions": {
   "notify-slack": {
     "transform": {
       "script": {
         "source": "def payload = ctx.payload; DateFormat df = new SimpleDateFormat(\"yyyy-MM-dd'T'HH:mm:ss.SSS'Z'\"); payload.timestamp_formatted = df.format(Date.from(Instant.ofEpochMilli(payload.hits.hits.0._source.timestamp)));  payload.end_timestamp_formatted = df.format(Date.from(Instant.ofEpochMilli(payload.hits.hits.0._source.end_timestamp))); return payload",
         "lang": "painless"
       }
     },
     "throttle_period_in_millis": 10000,
     "slack": {
       "message": {
         "to": [
           "#&lt;slack-channel&gt;"
         ],
         "text": "New Annotation for job *{{ctx.payload.hits.hits.0._source.job_id}}*: {{ctx.payload.hits.hits.0._source.annotation}}",
         "attachments": [
           {
             "fallback": "View in Single Metric Viewer https://&lt;kibana-host&gt;:5601/app/ml#/timeseriesexplorer?_g=(ml:(jobIds:!({{ctx.payload.hits.hits.0._source.job_id}})),refreshInterval:(pause:!t,value:0),time:(from:'{{ctx.payload.timestamp_formatted}}',mode:absolute,to:'{{ctx.payload.end_timestamp_formatted}}'))&_a=(filters:!(),mlSelectInterval:(interval:(display:Auto,val:auto)),mlSelectSeverity:(threshold:(color:%23d2e9f7,display:warning,val:0)),mlTimeSeriesExplorer:(zoom:(from:'{{ctx.payload.timestamp_formatted}}',to:'{{ctx.payload.end_timestamp_formatted}}')),query:(query_string:(analyze_wildcard:!t,query:'*')))",
             "actions": [
               {
                 "name": "action_name",
                 "style": "primary",
                 "type": "button",
                 "text": "View in Single Metric Viewer",
                 "url": "https://&lt;kibana-host&gt;:5601/app/ml#/timeseriesexplorer?_g=(ml:(jobIds:!({{ctx.payload.hits.hits.0._source.job_id}})),refreshInterval:(pause:!t,value:0),time:(from:'{{ctx.payload.timestamp_formatted}}',mode:absolute,to:'{{ctx.payload.end_timestamp_formatted}}'))&_a=(filters:!(),mlSelectInterval:(interval:(display:Auto,val:auto)),mlSelectSeverity:(threshold:(color:%23d2e9f7,display:warning,val:0)),mlTimeSeriesExplorer:(zoom:(from:'{{ctx.payload.timestamp_formatted}}',to:'{{ctx.payload.end_timestamp_formatted}}')),query:(query_string:(analyze_wildcard:!t,query:'*')))"
               }
             ]
           }
         ]
       }
     }
   }
 }
}
</pre><p>In the configuration above, just replace <code>&lt;slack-channel&gt;</code> and <code>&lt;kibana-host&gt;</code> with your settings and use it to create an <a href="https://elastic.ac.cn/guide/en/kibana/current/watcher-create-advanced-watch.html">advanced watch</a>. Once everything is set up, you should receive a Slack notification every time you create a new annotation —&nbsp;including the annotation text and a link back to Single Metric Viewer.
</p><p><img src="https://images.contentstack.io/v3/assets/bltf7afce26b89a5b33/blt8fc884c15ecd2fcf/5cb5a765fe8386be234acb7e/download" data-sys-asset-uid="blt8fc884c15ecd2fcf" width="650">
</p><h2>Summary</h2><p>In this article we introduced the new annotations feature for Elasticsearch machine learning. It can be used for adding annotations via the UI&nbsp;and for system annotations triggered via backend tasks. These annotations are available as bookmarks via the Job Management page and are sharable as links with others. Annotations can be created programmatically from external data to be used as a ground truth overlay for detected anomalies. Finally, in combination with Watcher and the slack action in Elasticsearch, we’ve seen how annotations can be used for curated alerting. Have fun with annotations, and find us on the <a href="https://discuss.elastic.co/">Discuss forums</a>&nbsp;if you have any questions.
</p>

ml-annotations-header-optimized.jpg

ml-annotation-edit-thumbnail.png

User annotations for Elastic machine learning

Alerting with context baked in helps analysts be more efficient. Storing alerts let you see trends in your environment that you may have otherwise missed.

<p>Within Elastic, the information security team is tasked with security detection and analytics, among many other activities of a typical information security team. To find abnormal and malicious behavior within our environment we leverage <a href="https://elastic.ac.cn/products/siem">Elastic SIEM</a> for investigations and threat hunting. When we find a pattern of behavior we want to be alerted on during an investigation or hunt we take the request JSON behind our investigation and put in to <a href="https://elastic.ac.cn/guide/en/elasticsearch/reference/7.5/xpack-alerting.html">Watcher</a> for alerting. Recently, we decided to start capturing the payload of our alerts in a separate index with the following goals:<br>
</p><ol>
	<li>Better reporting on the alerts that fire and observables (hosts, users, IPs, etc…) that are part of an alert payload.</li>
	<li>Enrichment of alert payloads with MITRE ATT&CK information and response playbooks. This will further speed up analysts and provide additional reporting on observed behaviors and coverage of the MITRE ATT&CK framework.</li>
</ol><p>Watcher has an <a href="https://elastic.ac.cn/guide/en/elasticsearch/reference/7.5/actions-index.html">index</a> action that allows you to index data into Elasticsearch but I wanted to do more than just send the alert data in to another index, I wanted to enrich it with additional data so I got creative to make this happen.&nbsp;
</p> <strong><h2>Alert indexing and enrichment</h2></strong><p>I won’t dive in to the body of the Watcher; I’m going to&nbsp; assume if you’re reading this you’ve already got Watcher setup for alerting. To get the enrichments in to the alert payload, I used a <a href="https://elastic.ac.cn/guide/en/elasticsearch/reference/7.5/transform.html">Watcher payload transform</a> to list each object in <strong>ctx.payload.hits.hits</strong>, and then added in the enriched, MITRE ATT&CK framework fields. You’ll notice we used parameters to inject the values for the newly created fields. This makes it easier to update the injected values without getting lost in the JSON:
</p><pre class="prettyprint">"transform": {
    "script": {
      "source": "List x = ctx.payload.hits.hits.stream().map(d -&gt; {d._source['threat.framework'] = params.framework; d._source['watcher.name'] = params.watcher; d._source['threat.tactic.name'] = params.tactic; d._source['threat.technique.id'] = params.techniqueid; d._source['threat.technique.name'] = params.techniquename; return d._source}).collect(Collectors.toList()); return x;",
      "lang": "painless",
      "params": {
        "watcher": "logFileChanges.json",
        "framework": "MITRE ATT&CK",
        "tactic": "Execution",
        "techniqueid": "T1059",
        "techniquename": "Command-Line Interface"
      }
    }
  }
</pre><p>Now the enriched alert payload is ready to be sent into the new index. This is where the <strong>index_payload</strong> action comes in. Another transform is used to return the documents in the <strong>ctx.payload._value</strong> that we created with the transform above and then we specify the index we want to put the alerts in, in our case it’s <strong>newIndex</strong>.
</p><pre class="prettyprint">"actions": {
    "index_payload": {
      "transform": {
        "script": {
          "source": "return ['_doc':ctx.payload._value];",
          "lang": "painless"
        }
      },
      "index": {
        "index": "newIndex"
      }
    }
  },
</pre><strong><h2>Cool, Now What?</h2></strong><p>Now that you’re storing and enriching alert data you’ll probably want to do something with it. This is where all the hard work of getting your alerts stored and enriched pays off. Now you have another index with data that you can report and alert on. Let’s dive in to some ways I’m using this alert data.
</p> <strong><h3>Reporting</h3></strong><p>Now that we have all of our alerts populating a new index with injected fields for our monitoring environment, we can start reporting on our observations. I set up a Canvas dashboard that lets us gain insights into what we’ve observed in our environment. Here is the Canvas dashboard with some sample data so you can see it in action.
</p><p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt3f3226ec24d94348/5de56dda546f955ab898caf4/alerts-blog-dashboard.jpg" data-sys-asset-uid="blt3f3226ec24d94348" alt="Canvas dashboard showing security alerts overview" style="display: block; margin: auto;">
</p><p>With this Canvas dashboard we are showing an overview of the alerts enriched with MITRE ATT&CK information. You can see the total number of alerts triggered and that is further broken down by MITRE ATT&CK Technique, host, and operating system. This can start to give us insight into if an attack is targeted at a single or small number of hosts or if there is just a lot of activity happening in general. We can also see if the alerts are all for a particular operating system or not. This may result in reviewing the configuration and security posture of that OS since we can see more alerts being generated from them.&nbsp;
</p><p>The Canvas dashboard also shows us a breakdown of the MITRE ATT&CK Techniques by technique so we can see which techniques are being utilized and alerted on. We can also see with the Timelion element when these alerts are happening. This can show us if the attacks are spread out evenly, ramping up over time, or spiking.
</p><p>Easier reporting, check. More meaningful reporting, check. Now lets make these alerts more useful for the analyst.
</p> <strong><h3>Alert Enrichment</h3></strong><p>We already know the Elastic Stack is really fast at providing search results but we’re always looking for ways to make analysis even faster and more efficient. To speed up the analysts’ ability to investigate faster I made two of the injected fields link back to relevant information. Here is how I did that for <strong>threat.technique.id</strong> in the index pattern for this index.
</p><p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt71af6fb2ffbbac6a/5de56e7394bf745993ed3e7b/alerts-blog-field-format.png" data-sys-asset-uid="blt71af6fb2ffbbac6a" alt="" style="display: block; margin: auto; width:60%;">
</p><p>I also did this for the field <strong>watcher.name</strong>. This is how those fields show up in the Discover view now
</p><p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/bltb62056c537f389f1/5de56eac92ebd8575d520078/alerts-blog-field-links.png" data-sys-asset-uid="bltb62056c537f389f1" alt="" style="display: block; margin: auto; width:75%;">
</p><p>Where do these links take you? The link for <strong>threat.technique.id</strong> links back to the relevant MITRE ATT&CK Technique page so the analyst can read through the details of the technique if they aren’t familiar with it. The link to <strong>watcher.name </strong>links back to our triage playbook for this watcher so analysts don’t have to remember where to find the relevant playbook.&nbsp;
</p> <strong><h2>Wrapping up</h2></strong><p>And just like that, we’ve stored and enriched alert payloads to make reporting and we’ve met our&nbsp; goal of improving security detection capabilities. These same techniques can be used no matter your use case and I’m hopeful that you found this useful and can begin to do similar things within your environment. I encourage you to try it out yourself. And if you’re not currently using Elastic SIEM or Watcher, you can give them a whirl in a <a href="https://elastic.ac.cn/products/elasticsearch/service">free trial of the Elasticsearch Service</a>. And as always, if you run into any problems, reach out on our <a href="https://discuss.elastic.co">Discuss</a> forums. Enjoy!
</p>

blog-banner-security-circuit-lock.jpg

blog-thumb-security-circuit-lock.jpg

Storing and enriching alerts for information security with Elasticsearch

Learn how to use Raspberry Pi and a relay module to build a real-world alarm notification system for Elasticsearch.

<p>Do you remember the last time when you saw a fire alarm or the police turning on a patrol car's flashing lights? That kind of warning light or loud sound simply notifies you that something is very important, or gives some some sense of danger, and that you must pay attention to it.
</p><p>
	Sometimes, you may also want to do a similar thing with your data, like when your server is down. That’s a big thing and you should call your admin as soon as possible to get it fixed. If your admin is nearby, you could even directly shout out to him/her.
</p><p>
	“Aha!” you might say, let’s use Elasticsearch to monitor our infrastructure and use the alerting feature to notify our admin via email or Slack. Wow, that’s great, but it can be better. Sometimes, the admin’s mobile phone maybe needs to be charged or it’s been left on the table, or the admin may be having a conversation and they may not pay attention to the message, so how can we bring it up?
</p><p>
	What if we used a real world warning light, like what we see during a fire alarm, and even add a loud sound that will be interesting and noticeable, so no one will miss it?
</p><p>
	In this post I will explain how to wire a Raspberry Pi with an alarm light and integrate it with the Elastic Stack to make this happen.
</p><h2 rel="line-height:1.38;margin-top:20pt;margin-bottom:6pt;">Intro to Raspberry Pi</h2><p>Raspberry Pi is a very small, but fully featured computer on a single board. You may plug in a monitor with an HDMI cable and attach a keyboard, mouse, camera, or even speakers via USB ports. It can also connect to your network through Wi-Fi or an ethernet port.
</p><p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt482c0f7f2373731b/5c98d4df67b8321d593d7fa2/raspberry-pi.jpg" data-sys-asset-uid="blt482c0f7f2373731b" alt="" "="">
</p><p>The first generation of Raspberry Pi was developed in 2012, with the intention of making computer learning easy for school students. It is cheap, with the newest model, the 3B+, costing you under USD$40. You can install a Linux OS on it, use your favourite language to do programming, and all with very low power consumption.
</p><p>
	The most important feature that Raspberry Pi has is the row of GPIO (General-Purpose Input/Output) pins along the top edge of the board.
</p><p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/bltc9982e04982b119b/5c98d4dafc9053fa5ecd78e9/raspberry-pi-gpio.jpg" data-sys-asset-uid="bltc9982e04982b119b" alt="" "="">
</p><p>The GPIO pins can be used with a variety of alternative functions, some are available on all pins, others on specific pins. This list below details the functions:
</p><ul>
	<li>PWM (pulse-width modulation)
	<ul>
		<li>Software PWM available on all pins</li>
		<li>Hardware PWM available on GPIO12, GPIO13, GPIO18, GPIO19</li>
	</ul></li>
	<li>SPI
	<ul>
		<li>SPI0: MOSI (GPIO10); MISO (GPIO9); SCLK (GPIO11); CE0 (GPIO8), CE1 (GPIO7)</li>
	</ul>
	<ul>
		<li>SPI1: MOSI (GPIO20); MISO (GPIO19); SCLK (GPIO21); CE0 (GPIO18); CE1 (GPIO17); CE2 (GPIO16)</li>
	</ul></li>
	<li>I2C
	<ul>
		<li>Data: (GPIO2); Clock (GPIO3)</li>
		<li>EEPROM Data: (GPIO0); EEPROM Clock (GPIO1)</li>
	</ul></li>
	<li>Serial
	<ul>
		<li>TX (GPIO14); RX (GPIO15)</li>
	</ul></li>
</ul><p>Basically, GPIO lets us connect to physical electronic device like alarm lights and also allows us to control these easily.
</p><h2>How it works</h2><p>So how it does this all work? The idea of this blog post is to use Raspberry Pi to control a relay module and light up a warning light. There are many alarm lights available to buy and they are very simple. You plug it in, it starts to flash, you unplug it, it stops flashing. More than simple enough for us to use. But wait! They usually are powered with higher voltages, like 12 V, 36 V, or even 220 V, and the Pi can only support 5 V or 3.3 V, which means we can’t control it directly with the Pi, but with a relay we can make it happen. A relay is an electrically operated switch. It uses a very low power electromagnet to mechanically operate a switch, and that switch can be the bridge to connect the high power power supply and our warning light.
</p><p>
	Then we will build a webhook service to send the command to the relay, tell the relay to switch on or off, and control the alarm light.
</p><p>
	And finally, we will use Elasticsearch’s alerting feature to trigger the webhook when we find some interesting events in Elasticsearch.
</p><p>
	Are you interested? If so, let’s get started!
</p><h2>Materials required</h2><p>So besides the Raspberry Pi, what materials are needed?, We also need some other electronics items, all of them are available at most electronic stores. Here is the list:
</p><table><colgroup><col width="235"><col width="192"><col width="192"></colgroup>
<tbody>
<tr style="height:0pt">
	<td style="border-width: 1pt; border-color: rgb(0, 0, 0); padding: 5pt;">
		<p style="line-height:1.2;margin-top:0pt;margin-bottom:0pt;">Name
		</p>
	</td>
	<td style="border-width: 1pt; border-color: rgb(0, 0, 0); padding: 5pt;">
		<p style="line-height:1.2;margin-top:0pt;margin-bottom:0pt;">Unit
		</p>
	</td>
	<td style="border-width: 1pt; border-color: rgb(0, 0, 0); padding: 5pt;">
		<p style="line-height:1.2;margin-top:0pt;margin-bottom:0pt;">Comments
		</p>
	</td>
</tr>
<tr style="height:0pt">
	<td style="border-width: 1pt; border-color: rgb(0, 0, 0); padding: 5pt;">
		<p>Raspberry Pi 3 model B+
		</p>
	</td>
	<td style="border-width: 1pt; border-color: rgb(0, 0, 0); padding: 5pt;">
		<p style="line-height:1.2;margin-top:0pt;margin-bottom:0pt;">1
		</p>
	</td>
	<td style="border-width: 1pt; border-color: rgb(0, 0, 0); padding: 5pt;">
		<p style="line-height:1.2;margin-top:0pt;margin-bottom:0pt;">You know, for Pi
		</p>
	</td>
</tr>
<tr style="height:0pt">
	<td style="border-width: 1pt; border-color: rgb(0, 0, 0); padding: 5pt;">
		<p style="line-height:1.2;margin-top:0pt;margin-bottom:0pt;">16GB MicroSD card + reader
		</p>
	</td>
	<td style="border-width: 1pt; border-color: rgb(0, 0, 0); padding: 5pt;">
		<p style="line-height:1.2;margin-top:0pt;margin-bottom:0pt;">1
		</p>
	</td>
	<td style="border-width: 1pt; border-color: rgb(0, 0, 0); padding: 5pt;">
		<p style="line-height:1.2;margin-top:0pt;margin-bottom:0pt;">Storage for Raspberry Pi
		</p>
	</td>
</tr>
<tr style="height:0pt">
	<td style="border-width: 1pt; border-color: rgb(0, 0, 0); padding: 5pt;">
		<p style="line-height:1.2;margin-top:0pt;margin-bottom:0pt;">5V USB power supply + USB cable
		</p>
	</td>
	<td style="border-width: 1pt; border-color: rgb(0, 0, 0); padding: 5pt;">
		<p style="line-height:1.2;margin-top:0pt;margin-bottom:0pt;">1
		</p>
	</td>
	<td style="border-width: 1pt; border-color: rgb(0, 0, 0); padding: 5pt;">
		<p style="line-height:1.2;margin-top:0pt;margin-bottom:0pt;">Power for Raspberry Pi
		</p>
	</td>
</tr>
<tr style="height:0pt">
	<td style="border-width: 1pt; border-color: rgb(0, 0, 0); padding: 5pt;">
		<p style="line-height:1.2;margin-top:0pt;margin-bottom:0pt;">Warning light with 12 V&nbsp;power
		</p>
	</td>
	<td style="border-width: 1pt; border-color: rgb(0, 0, 0); padding: 5pt;">
		<p style="line-height:1.2;margin-top:0pt;margin-bottom:0pt;">1
		</p>
	</td>
	<td style="border-width: 1pt; border-color: rgb(0, 0, 0); padding: 5pt;">
		<p style="line-height:1.2;margin-top:0pt;margin-bottom:0pt;">The light to do the alarm
		</p>
	</td>
</tr>
<tr style="height:0pt">
	<td style="border-width: 1pt; border-color: rgb(0, 0, 0); padding: 5pt;">
		<p style="line-height:1.2;margin-top:0pt;margin-bottom:0pt;">5 V Relay module
		</p>
	</td>
	<td style="border-width: 1pt; border-color: rgb(0, 0, 0); padding: 5pt;">
		<p style="line-height:1.2;margin-top:0pt;margin-bottom:0pt;">1
		</p>
	</td>
	<td style="border-width: 1pt; border-color: rgb(0, 0, 0); padding: 5pt;">
		<p style="line-height:1.2;margin-top:0pt;margin-bottom:0pt;">Control the lights
		</p>
	</td>
</tr>
<tr style="height:0pt">
	<td style="border-width: 1pt; border-color: rgb(0, 0, 0); padding: 5pt;">
		<p style="line-height:1.2;margin-top:0pt;margin-bottom:0pt;">Breadboard
		</p>
	</td>
	<td style="border-width: 1pt; border-color: rgb(0, 0, 0); padding: 5pt;">
		<p style="line-height:1.2;margin-top:0pt;margin-bottom:0pt;">1
		</p>
	</td>
	<td style="border-width: 1pt; border-color: rgb(0, 0, 0); padding: 5pt;">
		<p style="line-height:1.2;margin-top:0pt;margin-bottom:0pt;">Connect&nbsp;all the wires
		</p>
	</td>
</tr>
<tr style="height:0pt">
	<td style="border-width: 1pt; border-color: rgb(0, 0, 0); padding: 5pt;">
		<p style="line-height:1.2;margin-top:0pt;margin-bottom:0pt;">GPIO Breakout Expansion Board + Ribbon Cable
		</p>
	</td>
	<td style="border-width: 1pt; border-color: rgb(0, 0, 0); padding: 5pt;">
		<p style="line-height:1.2;margin-top:0pt;margin-bottom:0pt;">1
		</p>
	</td>
	<td style="border-width: 1pt; border-color: rgb(0, 0, 0); padding: 5pt;">
		<p style="line-height:1.2;margin-top:0pt;margin-bottom:0pt;">Connect breadboard and Pi
		</p>
	</td>
</tr>
<tr style="height:0pt">
	<td style="border-width: 1pt; border-color: rgb(0, 0, 0); padding: 5pt;">
		<p style="line-height:1.2;margin-top:0pt;margin-bottom:0pt;">3.3 V&nbsp;Active Piezo Buzzer Module
		</p>
	</td>
	<td style="border-width: 1pt; border-color: rgb(0, 0, 0); padding: 5pt;">
		<p style="line-height:1.2;margin-top:0pt;margin-bottom:0pt;">1
		</p>
	</td>
	<td style="border-width: 1pt; border-color: rgb(0, 0, 0); padding: 5pt;">
		<p style="line-height:1.2;margin-top:0pt;margin-bottom:0pt;">Let’s make some noise
		</p>
	</td>
</tr>
<tr style="height:0pt">
	<td style="border-width: 1pt; border-color: rgb(0, 0, 0); padding: 5pt;">
		<p style="line-height:1.2;margin-top:0pt;margin-bottom:0pt;">Female – Female jumper cable
		</p>
	</td>
	<td style="border-width: 1pt; border-color: rgb(0, 0, 0); padding: 5pt;">
		<p style="line-height:1.2;margin-top:0pt;margin-bottom:0pt;">~5
		</p>
	</td>
	<td style="border-width: 1pt; border-color: rgb(0, 0, 0); padding: 5pt;">
		<p style="line-height:1.2;margin-top:0pt;margin-bottom:0pt;">Wire stuff together
		</p>
	</td>
</tr>
<tr style="height:0pt">
	<td style="border-width: 1pt; border-color: rgb(0, 0, 0); padding: 5pt;">
		<p style="line-height:1.2;margin-top:0pt;margin-bottom:0pt;">Male – Female jumper cable
		</p>
	</td>
	<td style="border-width: 1pt; border-color: rgb(0, 0, 0); padding: 5pt;">
		<p style="line-height:1.2;margin-top:0pt;margin-bottom:0pt;">~5
		</p>
	</td>
	<td style="border-width: 1pt; border-color: rgb(0, 0, 0); padding: 5pt;">
		<p style="line-height:1.2;margin-top:0pt;margin-bottom:0pt;">Wire stuff together
		</p>
	</td>
</tr>
</tbody>
</table><h2>Hardware setup</h2><p>Below is the diagram of how to connect them together. As you can see, we are using a ribbon cable to connect the Pi and the breadboard, also make sure the GPIO breakout aligns with the breadboard.
</p><p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt6ec0deb9f113d3c6/5c98d4c7262c73954d53c60b/alarm-diagram.jpg" data-sys-asset-uid="blt6ec0deb9f113d3c6" alt="" "="">
</p><p>The breadboard allow us to easily connect all the parts together, but you should be very careful. If you misconnect the VCC to GND, some bad things may happen (parts could get burned). The relay has two types of connection, one is for our warning light, the other side is connected to the breadboard, and there is an IO pin which should be connected to the breadboard labelled with P22. Remember that as we will be using it later.
</p><p>
	We also have put in a buzzer connected to the breadboard, the I/O pin is connected to P12, which has a PWM feature, and we will use it to control the sound buzzer later.
</p><p>
	After some wiring up, here is what it looks like:
</p><p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt3c48dc7b8ae8ad42/5c98d4d355e02a1e5a5e9b00/alarm-hardware.jpg" data-sys-asset-uid="blt3c48dc7b8ae8ad42" alt="" "="">
</p><h2>Software setup</h2><p>Now the hardware is ready, let’s jump to the software part. The Raspberry Pi is based on the ARM CPU architecture and supports the Linux operating system. I will not teach you how to install a OS in Pi, because you can get very detailed instructions from <a href="https://projects.raspberrypi.org/en/projects/raspberry-pi-setting-up">Raspberry Pi’s official site</a>.
</p><p>I have chosen RASPBIAN OS instead.
</p><p>And now you should be able to use your favourite terminal tool to log into the Pi.
</p><h2>Controlling the Alarm with Python</h2><p>OK, let’s have some Python scripting fun by creating a script like this:
</p><pre class="prettyprint">#!/usr/bin/env python
import RPi.GPIO as GPIO
PIN_RELAY =  22
GPIO.setmode(GPIO.BCM)
GPIO.setwarnings(False)
GPIO.setup(PIN_RELAY, GPIO.OUT)
GPIO.output(PIN_RELAY, GPIO.LOW)
</pre><p><span></span>At first, we imported a Python library RPi.GPIO, which is a neat Python package allowing you to operate the GPIO easily.
</p><p>
	Then, we set which GPIO pin we are going to operate. Our relay is connected on pin 22 so we set the variable PIN_RELAY to 22. We then setup the pin to GPIO.OUT mode, which means we are going to write out some data to this pin and we set a value GPIO.LOW to this pin. GPIO.LOW is the new state value of the pin, there is the other state GPIO.HIGH which we will use to control the relay, and relay will be switched on after you set the state to GPIO.LOW. If you set it back to GPIO.HIGH, the relay will be switched off. There is also a jumper on the relay to set the default state.
</p><p>
	Note that if you set it to GPIO.LOW before you set it to GPIO.HIGH, the relay will keep this state, which means that the alarm light keeps flashing. You can turn it off after a while, like this:
</p><pre class="prettyprint">time.sleep(2)
GPIO.output(PIN_RELAY, GPIO.HIGH)
</pre><div style="margin-left:0pt;">
</div><p>Now let’s see how to make a sound. It is pretty similar to the relay, the only difference is the pin code, since the sound buzzer is connected to pin 12. Let’s see the code below:
</p><pre class="prettyprint">PIN_AUDIO = 12  #gpio 12
GPIO.setup(PIN_AUDIO, GPIO.OUT)
GPIO.output(PIN_AUDIO, GPIO.LOW)
time.sleep(0.5)
GPIO.output(PIN_AUDIO, GPIO.HIGH)
</pre><h2>An alerting webhook service</h2><p>Building a webhook service is easy. We need to make sure that we call the webhook and it then calls the relay and buzzer. Let’s put it together using the code we built above, and a few extra sections:
</p><pre class="prettyprint">import os
import time
import RPi.GPIO as GPIO
from socket import *
PIN_AUDIO = 12  #gpio 12
PIN_RELAY = 22  #gpio 22
def setup():
  GPIO.setmode(GPIO.BCM)      
  GPIO.setwarnings(False)
  GPIO.setup(PIN_RELAY, GPIO.OUT)
  GPIO.setup(PIN_AUDIO, GPIO.OUT)
def alarm():
  GPIO.output(PIN_AUDIO, GPIO.LOW)
  GPIO.output(PIN_RELAY, GPIO.LOW)
  time.sleep(2)
  GPIO.output(PIN_RELAY, GPIO.HIGH)
  GPIO.output(PIN_AUDIO, GPIO.HIGH)
def createServer():
    serversocket = socket(AF_INET, SOCK_STREAM)
    serversocket.bind(('0.0.0.0',9000))
    serversocket.listen(3)
    while(1):
        (clientsocket, address) = serversocket.accept()
        alarm()
        clientsocket.send("HTTP/1.1 200 OK\n"
         +"Content-Type:application/json\n"
         +"\n" # Important!
         + '{"success":true}'
         +"\n")
        clientsocket.shutdown(SHUT_WR)
        clientsocket.close()
    serversocket.close()
def destroy():
  GPIO.output(PIN_RELAY, GPIO.HIGH)
  GPIO.output(PIN_RELAY, GPIO.HIGH)
  GPIO.cleanup() 
if __name__ == '__main__':   
  setup()
  try:
    createServer()
  except KeyboardInterrupt:  
    destroy()
</pre><p>Yay! We’ve just created a simple web server. You can start the webhook easily, just run: python web.py, it listens on port 9000, each time we access this port it will return a success message. Can’t be easier. But for our alerting usage this is enough.
</p><h2>Integrate with alerting</h2><p>We have a small Elasticsearch cluster running on Pi, but it can be on Elastic Cloud or on your own hardware. We also have Heartbeat to monitor several services, and we will create a Watch in Elasticsearch to watch the Heartbeat indices, and will be using our new webhook endpoint (192.168.1.200:9000) in the alerting action. Here is the full Watch:
</p><pre class="prettyprint">PUT _xpack/watcher/watch/server_is_down
{
 "trigger": {
   "schedule": {
     "interval": "1s"
   }
 },
 "input": {
   "search": {
     "request": {
       "search_type": "query_then_fetch",
       "indices": [
         "heartbeat-*"
       ],
       "types": [],
       "body": {
         "size": 0,
         "query": {
           "bool": {
             "must": [
               {
                 "range": {
                   "@timestamp": {
                     "gte": "now-3s"
                   }
                 }
               },
               {
                 "match": {
                   "monitor.status": "down"
                 }
               }
             ]
           }
         }
       }
     }
   }
 },
 "condition": {
   "compare": {
     "ctx.payload.hits.total": {
       "gte": 1
     }
   }
 },
 "actions": {
   "alarm_webhook": {
     "webhook": {
       "scheme": "http",
       "host": "192.168.1.200",
       "port": 9000,
       "method": "post",
       "params": {},
       "headers": {},
       "body": "SOS, server is down!"
     }
   }
 },
 "throttle_period_in_millis": 5000
}
</pre><p>As you can see the request above, the Watch will check the Heartbeat index every second.&nbsp;If it finds that at least one service is down in the most recent 3 seconds, the Watch will trigger a webhook action, and also we set the throttle period to 5 seconds to reduce repeat alerts.
</p><p>
	When the Watch is created, it will watch your cluster 24x7 without rest, and also you can view the execution history details from Watcher’s UI:
</p><p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt8a3ad85fa58cb033/5c98d4cd166c55195f6cd4f6/alarm-dashboard.jpg" data-sys-asset-uid="blt8a3ad85fa58cb033" alt="" "="">
</p><p>If you shutdown some service, you will get an alarm immediately.
</p><p>Alerting with a webhook is really powerful.&nbsp;You may define your own webhook beyond just triggering a warning light, and you can also send the events to your own system, like Jira or GitHub, for further processing.
</p><p>For more information about how to use Elasticsearch Alerting and how to define a Watch, please refer to this <a href="https://elastic.ac.cn/guide/en/elasticsearch/reference/6.4/watcher-api-put-watch.html">doc</a>.
</p><h2>Show time</h2><p>Finally, it’s time to show our work. Check out this video below:
</p><div class="video embed-container" style="height: 319.725px;">
	<img style="width: 100%; margin: auto; display: block;" class="vidyard-player-embed" src="https://play.vidyard.com/GNeLehMybxzvYo4gEUc46Y.jpg" data-uuid="GNeLehMybxzvYo4gEUc46Y" data-v="4" data-type="inline">
</div><h2>Conclusion</h2><p>In this blog post, we used Raspberry Pi to connect the real world, and we also used the power of Elasticsearch to achieve real-time service monitoring. The most important thing is that getting a warning from real lights and a sound buzzer seems very cool. But when you get that kind of warning in your production&nbsp;environment, you should fix the issue right away.
</p><p>All related scripts can be found in this <a href="https://github.com/medcl/pi-warning-light-for-elasticsearch">repo</a>.&nbsp;Have fun!
</p>

fire-truck-2000x415.jpg

fire-truck-720x420.jpg

Building a Real-World Alarm with Elasticsearch and Raspberry Pi

The Elastic InfoSec team's internal use of Elastic Security has increased visibility and exponentially expanded its response capabilities to threats.

<p><em>This blog post is one in an occasional series about how we at Elastic embrace our own technology.</em></p><p>The Elastic InfoSec team is responsible for securing Elastic and responding to threats. We use our products everywhere we can&nbsp;— and for more than just logs. By harnessing the power and breadth of capabilities of the Elastic Stack, we are working on tracking risk and performance metrics, threat intelligence, our control framework, and control conformance information within Elastic.&nbsp;</p><p>Security has been and continues to be a top priority internally as well as in the products we develop and deliver. We’ve doubled down on these efforts when we joined forces with <a href="https://elastic.ac.cn/blog/endgame-joins-forces-with-elastic">Endgame</a> to craft Elastic Endpoint Security, and with <a href="https://elastic.ac.cn/blog/welcome-perched-security-training-for-siem-threat-hunting-and-more">Perched</a> to help create Elastic SIEM. Teaming up with these companies brought a whole new set of security professionals to the Elastic organization. Each professional brings a diverse set of skills and experiences that we can lean on to help protect our organization using our own security products — <a href="https://elastic.ac.cn/endpoint-security">Elastic Endpoint Security</a> and <a href="https://elastic.ac.cn/siem">Elastic SIEM</a>.</p><p>Our internal use of the Elastic Security solution has increased the Elastic InfoSec team’s level of visibility and has exponentially expanded its response capabilities to more fully empower the broader organization to protect the Elastic enterprise from the threats of today and tomorrow. For our Elastic community of users and customers, this also translates to us delivering a better, leading security solution.&nbsp;</p><p>We are excited to share our story….</p><h2><strong>“Customer Zero”</strong></h2><p>We have the expressed goal of being Customer Zero for all of our solutions, and this applies especially to Elastic Security. To ensure that what we deploy to our customers has been tested in a real production environment before it gets distributed more broadly, we’re an early adopter of build candidate (pre-release) versions of the Elastic Stack and each of our solution deployments.</p><p>We’re excited about having an industry-leading Endpoint Detection and Response (EDR) tool available to use within our environment. You can read all about the effectiveness of Elastic <a href="https://elastic.ac.cn/security/endpoint-security" target="_self">Endpoint Security solution</a> in third-party reports from NSS Labs, Gartner, and AV Comparatives in <a href="https://elastic.ac.cn/blog/introducing-elastic-endpoint-security">this post</a>.</p><p>Having leading detection and response capabilities on the endpoint is great on its own, but we’re really thrilled by the possibilities that come with its native integration with the Elastic Stack and Elastic SIEM.&nbsp;</p><p>It’s remarkable to have the ability to create automated responses and advanced analytics in addition to having the data and visuals we as analysts need — all in a single platform.</p><h2><strong>Piloting Elastic Endpoint Security</strong></h2><p>Replacing an endpoint security agent is not a task that is frequently undertaken, and it often comes with unexpected challenges. So when we announced our intent to bring Endgame into the Elastic fold, we immediately began planning our migration from CrowdStrike to Elastic Endpoint Security (formerly Endgame).&nbsp;</p><p>After officially joining forces with Endgame, we began to deploy the Sensor Management Platform (SMP) for our end user pilot. We were easily able to import our whitelist and trusted applications that we had previously configured in CrowdStrike. Then, we created a single detection policy within the SMP and enabled all threat and adversary behavior detections. We also enabled all event collection features and set up event streaming to Elasticsearch.&nbsp;</p><p>We ran the pilot on a small percentage of endpoints in a detect-only mode for a little more than a week. With our pilot users streaming endpoint data back to the SMP and to our endpoint monitoring Elasticsearch cluster, we began migrating our existing Auditbeat and Winlogbeat dashboards, queries, and Watcher alerts to use the endpoint index. The data received via the endpoint agent is similar to the data collected by Auditbeat and Winlogbeat without having to manage yaml configurations.</p><p><em></em></p><p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/bltec0e916e79b25bd3/5e470e8a7c62095ce7b37118/Securingendpoints-blog-canvas-dashboard-overview-Elastic-Endpoint-Alerts-1.jpg" data-sys-asset-uid="bltec0e916e79b25bd3" alt="Securingendpoints-blog-canvas-dashboard-overview-Elastic-Endpoint-Alerts-1.jpg"/></p><p><em></em></p><figcaption><em>A Canvas dashboard giving an overview of Elastic Endpoint Alerts</em></figcaption><h2><strong>Investigating with Elastic SIEM</strong><br /></h2><p>We were then ready to perform investigations in the Elastic SIEM app by simply adding the “endgame-*” index pattern to the default index for SIEM in Kibana. Investigating Elastic Endpoint Security alerts or other anomalous behavior in the SIEM app is easy with Timeline. Timeline allows us to stitch together processes by process ancestry similar to the ResolverTM view within the management console. It also allows us to see a population of events and data (process names, process arguments, etc.) across our entire endpoint fleet with the speed of Elasticsearch.</p><p><em></em></p><p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt85eaeb609b10986e/5e470e9894aef92989eeff6a/Securingendpoints-blog-configuring-indices-Elastic-SIEM.png" data-sys-asset-uid="blt85eaeb609b10986e" alt="Securingendpoints-blog-configuring-indices-Elastic-SIEM.png"/></p><p><em></em></p><figcaption><em>Configuring indices for the Elastic SIEM</em></figcaption><p>During the pilot, we had a single malware detection on one of our Mac workstations that had been present on the system with CrowdStrike installed. We were able to use the response capability in Elastic Endpoint Security to remotely remediate the malware with minimal impact and downtime to the impacted user. The user was actually able to continue working; the only interruption experienced was us telling them we were remotely working on their laptop.&nbsp;<br /></p><p>We were very happy with how easy it was to remove a fairly complex piece of malware that had several persistence mechanisms.&nbsp;</p><p>We’re a distributed organization with currently 1,900+ Elasticians located across 40+ countries, and a lot of our employees rarely, if ever, find themselves in an Elastic office. With this in mind, we were impressed with the Elastic Security solution’s ability to enable an Elastic analyst sitting in their house in Germany to successfully remediate malware on the device of a Tennessee-based employee. This was not a capability we had on Macs before installing Elastic Endpoint Security.</p><h2><strong>Deploying Elastic Endpoint Security</strong></h2><p>After the pilot period, we deployed the Endpoint agent to the remainder of the endpoint fleet in detect mode over a period of five days while removing CrowdStrike at the same time. During this period, we deployed to 1800+ endpoints — our fleet of user devices at that point in time — with no reported issues from any of our fellow Elastician users. If you’ve ever deployed an agent to user endpoints (especially security agents), you know it never goes this smoothly. But for us, it did.</p><p>Once we rolled out to the fleet at large, we had 23 systems report malicious files that needed to be remediated. The surprising part for us was the extremely low false positive rate on threat detections.&nbsp;</p><p>The Adversary Behavior detections, however, were noisier and filled with false positives because of the diverse skill sets, and subsequent behavioral trends, of our Elasticians. We saw things like Elastic security professionals conducting research and exploit proof of concepts, developers creating software, and sales architects running containers and other interesting tools. Our initial pilot users, along with the 23 systems that had true positive threat detections, were moved into prevent mode in less than a month with no reported impact.&nbsp;</p><p>Shortly after moving the pilot users to prevent mode, we migrated the entire fleet of user devices to prevent mode with <em>zero</em> reported issues.&nbsp;</p><p>The difference in visibility we now have into our endpoints is amazing. We can be using remote response on an endpoint across the globe and see the signals from the endpoint in both the endpoint console and in Kibana — in close to real time. The level of detail in the signals is better than we have typically seen with other endpoint security tools we’ve used in the past.</p><p><em></em></p><p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/bltc00422b5bf702433/5e470eacca06ad274a65b76f/Securingendpoints-blog-elastic-SIEM-signal-and-resolver-view-console-1.jpg" data-sys-asset-uid="bltc00422b5bf702433" alt="Securingendpoints-blog-elastic-SIEM-signal-and-resolver-view-console-1.jpg"/></p><p><em></em></p><figcaption><em>(Top) A Signal in the Elastic SIEM app for an Elastic Endpoint Alert with logs for the same host inline with the Signal. (Bottom) The Resolver view, in the Elastic Endpoint Security console, showing the process lineage associated with the same alert.</em></figcaption><em></em><h2><strong>From Elastic workstation to server fleet protection and beyond</strong><br /></h2><p>Now that our Elastic workstations are fully protected by Elastic Endpoint Security, we have begun our pilot of placing the Elastic Endpoint Security agent on our server fleet. We’ll be targeting all our server resources that bring you Elastic Cloud and enable us to deliver the Elastic Stack.&nbsp;</p><p>This rollout will probably take a bit more time than our endpoint rollout. We have a broader set of operating systems and cloud providers to test on than we did for workstations, but we’ll get there soon enough.</p><p>We’re excited to continue using the Elastic Security solution and all the integrations it has with the Elastic Stack. We’ll keep providing feedback directly to the Elastic teams working on the product, workflows, and integrations. We see this as a really great internal partnership.&nbsp;</p><p>Best of all, we’ll continue being Customer Zero so that our customers can benefit from well-tested and thoroughly deployed Elastic solutions and products.</p><p><em>Mandy Andress is CISO at Elastic; Darren LaCasse is Principal Analytics and Detection Lead I at Elastic; and Brian Milbier is Principal Security Assurance Analyst I at Elastic.</em></p>

blog-banner-elastic-on-elastic.png

blog-thumb-elastic-on-elastic.png

Elastic on Elastic: Securing our endpoints with Elastic Security

Here's the story of how RS2 quickly got Elastic Cloud from PoC to production by proving it as a security analytics solution.

<p>At RS2, security is at the core of everything we do. Our main product, BankWORKS, is a fully featured, end-to-end integrated solution for all payment processing needs — from device transaction acquisition through to final settlement and ledger integration. The software is used by Banks, Processors and Payment Service Providers around the world, big and small, simple to complex. We also offer the product as a hosted managed service.<br></p><p>As a team, we are responsible for making sure that we minimise the risk of data being compromised or leaked, across all avenues of our business, and at the same time ensure that we meet several compliance requirements, all while avoiding disruption of day-to-day operations.
</p><p>In November 2017, we were planning to grow our security team. Before getting approval for additional hires, though, we needed to alleviate some of the manual effort involved in dealing with incidents and security events.  Here began our journey with the Elastic Stack.
</p><h2>The Journey from Proposal to Production</h2><h3>Initial Stages</h3><p>Having previously used the Elastic Stack in other roles, and for personal projects, I wanted to introduce the product to the team. I felt it would fulfil all our requirements thanks to its extensive featureset and scalability.
</p><p>In the first few days in my new role at RS2, I spun up Elasticsearch and Kibana instances (version 6 in this case) on a virtual machine on my laptop, installed a couple of Beats on the VM itself (<a href="/products/beats/packetbeat">packetbeat</a>, <a href="/products/beats/auditbeat">auditbeat</a>, <a href="/products/beats/metricbeat">metricbeat</a> and <a href="/products/beats/filebeat">filebeat</a>) and sent all the data straight to Elasticsearch. The whole process took about an hour (40 minutes of which included the operating system ISO image download and install) to have meaningful data populated in Kibana.
</p><p>I showed this to my colleague and he almost instantly agreed that this was the way forward, and we that should create a demo for the executive team using real data to emphasize effectiveness. We decided to include a few network devices and existing servers which wouldn't require any changes to our production network (using the different Beats and Logstash), as well as some third party integrations.
</p><h3>Cloud Evaluation</h3><p>In previous roles, I hosted large Elastic deployments spanning several servers. However, I had never really looked at the Elastic Cloud offering. RS2 happened to be on an "infrastructure freeze" due to their imminent migration to the cloud. This, coupled with tight deadlines and limited resources, led me to explore Elastic Cloud. As a security professional, I was skeptical. I wanted to make sure that the service was designed with a degree of security in mind.
</p><p>Once I had my cluster, I carried out a few quick security tests to see if I can spot any blatant vulnerabilities or weaknesses. Here's what I discovered:
</p><ul>
	<li>Elastic lets you choose between AWS and GCP as a backend cloud provider, so all their security features are inherited, along with their compliance certifications.</li>
	<li>Segregated networks are used for each cluster, not the default subnets for each provider.</li>
	<li>Modern TLS settings and ciphers are used for both the Elasticsearch and Kibana URL's</li>
	<li>Elasticsearch transport ports are randomised</li>
	<li>The URLs for each instance are also completely randomised, so it isn't possible to enumerate customer names</li>
	<li>Direct IP access is not possible without the cluster ID</li>
	<li>The latest versions of the Elastic Stack is used, along with a recent version of Java 8.</li>
</ul><h3>Putting it all Together</h3><p>Now that I had my cloud cluster, I had to design the data flows. The diagram below outlines the architecture for the POC.
</p><p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt61fc66e37e3d1139/5c58424822d96be10bcca5fa/diagram-architecture-for-the-poc.jpg" data-sys-asset-uid="blt61fc66e37e3d1139" alt="Diagram - Architecture for the POC">
</p><p>Since we had X-Pack available to us, Watcher was utilised heavily as part of the alerting framework. This was integrated with a custom Slackbot by using the <a href="https://elastic.ac.cn/guide/en/x-pack/6.x/actions-webhook.html">Watcher webhook actions</a>.
</p><h3>Demo Preparation – Working with the Data</h3><p>The first step was to parse and enrich our logs as much as possible. In a security context, enrichment is key for solving incidents quickly, as it greatly reduces the investigation time for analysts. It also helps to filter out false positives. Using  several Logstash filter plugins, I was able to do this with ease. Additionally, to cater for our existing log archiving tool, I was able to set up multiple Logstash outputs to simultaneously send data to our Elastic cluster and the existing archiving tool.
</p><p>Below is a list of some of the enrichment operations added to our parsed logs:
</p><ul>
	<li><strong>GeoIP data (Location and ASN)</strong></li>
	<li><strong>Malware IP lookups</strong></li>
	<li><strong>Allowed logins user lookups</strong></li>
	<li><strong>User agent parsing</strong></li>
	<li><strong>URL decoding</strong></li>
</ul><p>This is a partial list of enrichments set up for the POC. Many more were added once we made the move to production.
</p><p>Now that I had all this data nicely parsed, I created custom dashboards to work alongside  the inbuilt ones to highlight some of the enrichment features previously mentioned. Here are just a few examples of some of the custom Kibana dashboards we developed for the POC (all sensitive data has been removed):
</p><p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/bltb7ab98a707dca0f9/5c306721597189c90b02429a/kibana-dashboard-1.jpg" data-sys-asset-uid="bltb7ab98a707dca0f9" alt="Kibana Dashboard 1.jpg">
</p><p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt6595ac48c4bae518/5c30671ead674d236e764932/kibana-dashboard-2.jpg" data-sys-asset-uid="blt6595ac48c4bae518" alt="Kibana Dashboard 2.jpg">
</p><p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt5bc9e9bfb38a6dba/5c306719682c4c4866a798ec/kibana-dashboard-3.jpg" data-sys-asset-uid="blt5bc9e9bfb38a6dba" alt="Kibana Dashboard 3.jpg">
</p><p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt5c6faa73be35d3ce/5c306715e71ce40c6e4ada09/kibana-dashboard-4.jpg" data-sys-asset-uid="blt5c6faa73be35d3ce" alt="Kibana Dashboard 4">
</p><p>Additionally, I added some other nifty integrations for the demo to show how simple it is to add data into Elastic. At the end of the day, it's just another index.  One example of this was an integration with the popular service "Have I been Pwned" by Troy Hunt. The service provides a very handy REST API, which allows you to query if an email address is detected in publicised data breaches. A watch was created to alert us to any new entries for our domain.
</p><h3>Alerting</h3><p>The idea behind the alerting framework in the POC (to later be used in production) was to have everything actionable through Slack. Below are some examples of the manipulated data within the Slackbot. Everything an analyst needs to kick off an investigation is included. The data used was gathered by different Beats and the parsed network device logs via Logstash.
</p><p>Some of the datasets included:
</p><ul>
	<li>SMTP relay logs, authentication logs and packetfilter logs from our Firewalls</li>
	<li>DNS requests at a packet level, using Packetbeat</li>
	<li>SSH/SFTP logs, using a combination of Wazuh and Filebeat</li>
	<li>A list of processes and their states, using Metricbeat</li>
	<li>Outbound network socket monitoring, using Auditbeat on *nix systems</li>
</ul><p>Here are just a few examples of some of the Slackbot alerts we developed for the POC (all sensitive data has been removed):
</p><ul>
	<li>TeamViewer Connection Alert<br><br><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/bltf644ac1b53085987/5c306711b2b2be79676e8480/teamview-connection-detected.jpg" data-sys-asset-uid="bltf644ac1b53085987" alt="Teamview Connection Detected" width="400"></li>
	<li>Firewall Login Alert<br><br><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/bltad0c8a7c731c55e7/5c30670d61a626fb0b630156/rs2-security-bot.jpg" data-sys-asset-uid="bltad0c8a7c731c55e7" alt="RS2 - Security Bot - Firewall Login Detected" width="400"></li>
	<li>Malware Alert<br><br><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blta3312d2dcbfd2837/5c30670929d13af10bc2aba0/rs2-security-bot2.jpg" data-sys-asset-uid="blta3312d2dcbfd2837" alt="RS2 - Security Bot - Communication with Malware IP Detected" width="400"></li>
</ul><h3>The Results</h3><p>Needless to say, the POC was extremely successful and we got approval to move to production. To reiterate, the main points that got us through this POC so smoothly:
</p><ul>
	<li>The exceptional ease and speed of using Elastic Cloud and everything it encompasses (integrated backups out of the box, resiliency and high availability, bundled X-Pack for our size of deployment)</li>
	<li>The ability to take in any data and turn it into something useful and actionable very quickly (the POC, from start to finish, took about 3 full days to implement, including all the tasks mentioned in this post – parsing, dashboards, enrichment, the alerting framework, so on and so forth)</li>
	<li>The fact that this could be done in parallel to all existing processes, without interruption</li>
</ul><h2>Dealing with Upgrades</h2><p>After a few weeks in production, there was an update released by Elastic. Having previously upgraded large Elastic deployments with X-Pack, I was very curious to see how this was carried out by their cloud platform. It turned out to be as simple as selecting the new version in a dropdown menu. Everything else was done automatically, without any interruptions.
</p><h2>Conclusion</h2><p>Our journey with Elastic obviously did not end here. We are constantly adding more data sources, more enrichment (like correlation with our HR systems to get user vacation data, and physical access systems to know if someone is and should be in the building or not) and adding alerts on the fly based on newly discovered threats and malicious activity. We are also working on integrating with additional internal tools that we use.
</p><p>We are excited about the future of <a href="/solutions/security-analytics">security analytics</a> with Elastic. With every update, Elastic releases additional components that make the lives of analysts easier, and their jobs more satisfactory. Additionally, we are equally as excited for the upcoming upgrades to Elastic Cloud. Without a doubt, RS2 will continue to benefit from the extensive feature sets, not just for security analytics, but throughout the entire organisation.
</p>

swiss-army-knife-outlines-full.jpg

swiss-army-knife-outlines-thumb.jpg

Using the Elastic Stack as a SaaS-Based Security Operations Swiss Army Knife

ml-and-nginx-website-logs-fullbleed.jpg

Machine Learning for Nginx Logs - Identifying Operational Issues with Your Website

<p>The <a href="https://elastic.ac.cn/products">Elastic Stack</a> delivers security analytics capabilities that are widely used for threat detection, visibility, and incident response. The speed and scale at which Elasticsearch can index and search security-related information enable security analysts to work more efficiently, while Kibana dashboards provide wide visibility and enable interactive threat hunting. And the <a href="https://elastic.ac.cn/products/stack/machine-learning">machine learning</a> engine can automate the analysis of complex datasets, making it possible to spot intruders that otherwise would’ve gone unnoticed.</p><p>Popular Intrusion Detection Systems (IDS), such as <a href="https://wazuh.com/">Wazuh</a> or Suricata, use a signature-based approach to threat detection. That is, they compare patterns found in files, logs, and network traffic against a database of patterns known to be associated with malicious activity, alerting when a match is found. They provide useful rulesets to analyze and correlate data, usually generating thousands or millions of alerts per day in a production environment.</p><p>Casting a wide net can ensure that all potential security events are caught, but it also adds the work of sifting through thousands (or millions) of alerts every day. Elastic machine learning features help reduce the noise by automatically identifying unusual behaviors. This is a clear use case where anomaly-based and signature-based technologies complement each other, making threat detection easier and investigations more efficient.</p><p><a href="https://github.com/wazuh/wazuh">Wazuh</a>, commonly deployed along with the Elastic Stack, is an open source host-based intrusion detection system (HIDS). It provides log analysis, file integrity monitoring, rootkit and vulnerability detection, configuration assessment and incident response capabilities. The Wazuh solution architecture is based on multi-platform lightweight agents that run on monitored systems, reporting to a centralized server where data analysis is done. In addition, it provides a complete Kibana plugin for configuration management, status monitoring, querying and alert data visualization.</p><p>On the other hand, <a href="https://suricata-ids.org/">Suricata</a> is a free and open source network threat detection engine, capable of real time network intrusion detection (NIDS), inline intrusion prevention (NIPS), network security monitoring (NSM), and offline pcap processing. Suricata inspects the network traffic using its rules and signature language to match known threats, policy violations and malicious behaviors, and supports scripting for detection of complex threats.</p><p>In this blog post, we will provide an overview of how to detect intrusions using an integration of Wazuh and Suricata in combination with Elastic machine learning jobs to help prioritize investigations.</p><h2>Integrating Wazuh, Suricata, and the Elastic Stack</h2><p>For the purpose of this blog post, we have set up a lab environment where Wazuh agents have been deployed to several Internet-facing servers, monitoring system and application log data, file integrity, and system calls.</p><p>In addition, we are running a Suricata sensor that takes care of monitoring the network traffic. This sensor is often configured to monitor traffic via a network tap, mirroring port, or a SPAN (Switched Port Analyzer) port, but can also can be deployed directly to your servers.</p><p>In order to get the most out of these two tools, we decided to analyze Suricata alerts using Wazuh rules, unifying the alerts format and allowing us to do correlation (for example with threat intelligence sources) and trigger automatic responses.</p><p>This integration was done by configuring a Wazuh agent to read Suricata JSON output. This agent acts as a collector that forwards the Suricata NIDS alerts to the Wazuh server, where they are processed through Wazuh log analysis rules, resulting in new and enriched security events.</p><p>Now, as result of this work, both NIDS and HIDS alerts are sent to Elasticsearch via Filebeat (configured to read Wazuh alerts) and Logstash (also used for geolocation enrichment), where we will use machine learning jobs to detect anomalies and unusual behaviors.</p><p>Below is an example of a deployment of host-based and network-based IDS technologies, integrated together with Elastic Stack:</p><p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt69bd38c77bdaac5e/5c583b764651cff70b650fc7/wazuh.png" data-sys-asset-uid="blt69bd38c77bdaac5e" alt="Security analytics with the Elastic Stack and Wazuh" style="display: block;margin: auto;"/></p><h2>Detecting Malicious Actors using Machine Learning Jobs</h2><p>In our lab environment, we enabled all rules, and found out that, for just one of our Internet-facing web servers, our Wazuh agent is reporting somewhere from 4 to 10 thousand IDS alerts per day. They are mostly related to web attacks, authentication failures, configuration issues (detected using hardening checks), file integrity changes, or vulnerable software.</p><p>To make it easier for the security analysts, Wazuh IDS alerts are enriched with metadata, such as a <a href="https://documentation.wazuh.com/3.x/user-manual/manager/alert-threshold.html">level value</a> or <a href="https://documentation.wazuh.com/3.x/user-manual/agents/grouping-agents.html">group</a>, making it possible to filter by priority or type. In addition, the Wazuh Kibana plugin provides pre-configured dashboards with useful information regarding the agent status, configuration, and alerts. See screenshot below:</p><p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/bltaeded6e5f42e8008/5c583b6f713ebdec0ba0724b/wazuh_1.jpg" data-sys-asset-uid="bltaeded6e5f42e8008" style="display: block;margin: auto;"/></p><p>The information provided by Wazuh is certainly useful, but it still does not tell us about unusual behaviors. Here is where an Elastic machine learning job can help us.</p><p>Elastic machine learning allows us to create several type of “jobs.” The job is the basic element of machine learning analysis. In our case, we decided to create what is known as a “population analysis,” where we instructed the machine learning engine to build a baseline model of IP address typical behavior over a period of time, identifying the ones that are behaving abnormally compared to the rest of population.</p><p>More specifically, in our case we wanted to look for source IP addresses that, when compared to all other IP addresses, are responsible for an unusually high number of alert types. This can’t be done just by creating aggregation tables, as the anomalies occurred at a certain point in time, not known by our security analyst, and we have over 60 days of alert data (we detected some attacks for which their duration was no more than two minutes).</p><p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt5f3853bf1907d6b8/5c583b698c741b010cbf6cf2/wazuh_2.png" data-sys-asset-uid="blt5f3853bf1907d6b8" style="display: block;margin: auto;"/></p><p>As a result of our population analysis, we spot several abnormal behaviors (in the form of a list of source IP addresses), which we decided to investigate. We used the “Anomaly Explorer” to learn when each one of those possible malicious actors were attacking our environment.</p><p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/bltc441190af1573bef/5c583b644651cff70b650fc1/wazuh_3.png" data-sys-asset-uid="bltc441190af1573bef" style="display: block;margin: auto;"/></p><h2>Investigating an Intrusion Attempt</h2><p>As an example of a security analysis, we highlight the investigation of one of the IP addresses identified by the machine learning job. This particular IP address generated several NIDS and HIDS alerts in less than a minute, triggering an automated Wazuh action that blocked the IP address in the local firewall of our web server.</p><p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt9943893542e7e7f3/5c583b5edfaabbd40b9c2c0c/wazuh_4.jpg" data-sys-asset-uid="blt9943893542e7e7f3" style="display: block;margin: auto;"/></p><p>As shown in the screenshot above, Suricata NIDS detected malicious traffic from the indicated source IP address. Four rules matched incoming traffic, generating the following alerts:</p><ul><li>ET DROP Dshield Block Listed Source group 1</li><li>ET CINS Active Threat Intelligence Poor Reputation IP group 77</li><li>SURICATA HTTP URI terminated by non-compliant character</li><li>SURICATA HTTP METHOD terminated by non-compliant character</li></ul><p>The first two rules, part of <a href="https://doc.emergingthreats.net/">Emerging Threats</a> ruleset, indicate that the source IP address is known to have a poor reputation according to Threat Intelligence (TI) sources: <a href="https://www.dshield.org/">Dshield</a> and Active Threat Intelligence.</p><p>In addition, two other signatures detected anomalous HTTP activity coming from the same source IP address, most likely part of a scanning phase where the attacker gathers information looking for potential vulnerabilities.</p><p>The Wazuh HIDS component also triggered different alerts based on the analysis of the web server access logs. This approach, totally different from network packet inspection, ended up generating the following alerts:</p><ul><li>IP address found in AlienVault reputation database</li><li>Host Blocked by firewall-drop.sh Active Response</li><li>Host Unblocked by firewall-drop.sh Active Response</li></ul><p>The first alert indicates that the source IP address is also known to be part of yet another Threat Intelligence (TI) source: <a href="https://otx.alienvault.com/">AlienVault OTX</a> IP reputation database.</p><p>The second two alerts are triggered by the Wazuh Active Response module that, as result of the previously mentioned alerts, automatically added a firewall rule to block traffic from that particular source IP address for exactly 60 seconds (this is configurable). This action was enough to block scanning activities and caused the malicious actor desist from the intrusion attempt.</p><h2>Conclusion</h2><p>Employing both signature-based and anomaly-based intrusion detection, using technologies such as Wazuh, Suricata and Elastic machine learning, can make threat detection easier and investigations more efficient.</p><p>In addition, integrating host-based IDS (used to monitor systems at a host level) with network-based IDS (used to inspect network traffic) can also increase threat detection and security visibility. Wazuh makes this easy, as it can be used to integrate host and network IDS systems with the Elastic Stack, and can provide mechanism to run automated responses and block attacks in real-time.</p><span></span>

ML_IDS_2000x415a.jpg

Security analytics

Threat hunting

Cybersecurity

Blog.jpg

Improve Security Analytics with the Elastic Stack, Wazuh, and IDS

With free and open Elastic SIEM, Elastic continues its mission to help organizations improve their security posture. Eliminate blind spots and arm analysts with the tools and workflows they need to protect data and infrastructure.

<p><span style='font-size: 12pt;'>Have you heard the news?&nbsp;</span><a href="https://elastic.ac.cn/blog/elasticsearch-is-open-source-again"><span style='font-size: 12pt;'>Elasticsearch is open source, again.</span></a></p><p></p><hr/><p></p><p>Security teams must protect attack surfaces that are becoming bigger and more distributed due to the growth of remote work, cloud infrastructure, and other dynamics. These teams understand that meeting this challenge at scale requires the successful incorporation of the appropriate technology into their security operations program. While <a href="https://elastic.ac.cn/what-is/siem" target="_self">security information and event management (SIEM) solutions</a> are designed to help analysts detect and respond to potential security threats, there are fundamental limitations with many widely deployed SIEM solutions that often hamstring analysts.</p><p>Securing your organization against cyber threats is tough enough in normal times. Unfortunately, during challenging times, the threat landscape gets even more treacherous. Regional, national, or global events, such as elections, major sporting events, or even a global health crisis can exacerbate the challenge. The greater adoption of wide scale telecommuting carries its own risks — with workers operating from unsecured environments, on personal or repurposed equipment, and following processes that might not have been carefully reviewed or tested.</p><h2>What is Elastic SIEM?</h2><p>The free and open <a href="https://elastic.ac.cn/siem" target="_self">Elastic SIEM</a> is an application that provides security teams with visibility, threat hunting, automated detection, and Security Operations Center (SOC) workflows. Elastic SIEM is included in the default distribution of the most successful logging platform, Elastic (ELK) Stack software. It ships with out-of-the-box detection rules aligned with the MITRE ATT&amp;CK™ framework to surface threats often missed by other tools. Created, maintained, and kept up-to-date by the security experts at Elastic, these rules automatically detect and address the latest threat activity. Severity and risk scores associated with signals generated by the detection rules enable analysts to rapidly triage issues and turn their attention to the highest-risk work. If you need to deploy a <a href="https://elastic.ac.cn/what-is/siem" target="_self">SIEM solution</a> today and see immediate value, we provide the best and easiest way to do it.</p><p>Elastic SIEM packs a lot of power. Built with the speed and scalability of Elasticsearch as its underlying search platform, it maintains analyst velocity with:</p><ul><li>An overview page to show SOC status and security posture</li><li>Dashboards for threat hunting and situational awareness&nbsp;</li><li>Integration with Elastic Maps, Elastic Lens, and the rest of Kibana</li><li>A detection engine for automated detection&nbsp;</li><li>A unique timeline investigator with investigation templates for analysts</li></ul><p>Organizations use these capabilities to eliminate blind spots and arm analysts to drive down mean time to detect (MTTD) and mean time to respond (MTTR).&nbsp;</p><div class="video embed-container" style="height: 319.725px;"><img class="vidyard-player-embed" src="https://play.vidyard.com/k5GtNGY1qR8HbAoLLgvvP8.jpg" data-uuid="k5GtNGY1qR8HbAoLLgvvP8" data-v="4" data-type="inline" data-autoplay="1" data-loop="1" data-disable_analytics="1" data-hidden_controls="1" data-muted="0" width="692"/></div><h2>Why free SIEM?</h2><p>Elastic is on a mission to help organizations improve their security posture by eliminating blind spots and arming their analysts with the products and workflows they need to protect data and infrastructure. Spurred by the security community’s widespread adoption of the Elastic Stack for threat hunting, fraud detection, and security monitoring, we wanted to make it even easier for users to deploy our products for security. We first worked in collaboration with our community to develop <a href="https://elastic.ac.cn/guide/en/ecs/1.5/index.html" target="_self">Elastic Common Schema</a> (ECS), which streamlines the normalization of data from disparate sources — whether from network and host technologies or cloud infrastructure and applications. We launched Elastic SIEM in June 2019, introducing the industry’s only free and open SIEM packaged with actively maintained SIEM detection rules.&nbsp;</p><p>We demonstrated the latest Elastic SIEM capabilities to dozens of visitors at our RSA 2020 conference booth in San Francisco.&nbsp;After nearly every demo, when asked about the licensing costs, we&nbsp;were pleased to be able to answer that everything we’d just showed was free and open to use. “Wait, you mean a trial?” “No, forever.” “Really?” “Yes.”</p><h3>Get started and deploy at scale for free</h3><p>Free and open might be new to SIEM, but it is not new to Elastic. For years, organizations have built <a href="https://elastic.ac.cn/security/siem" target="_self">security analytics</a> projects for threat hunting and SIEM on the Elastic Stack. They have enjoyed unlimited scale, unlimited-duration proofs of concept with no licensing cost, and none of the vendor headaches usually associated with evaluating new enterprise software. We’re building on this with Elastic SIEM, which is available for download to run on-premises, in a virtual or containerized environment, or in your cloud or ours.</p><h3>Resource-based pricing for commercial extensions</h3><p>The free and open Elastic SIEM provides a solid base for SOC operations to use at any scale and for as long as they need it. Elastic also provides commercial extensions to Elastic SIEM that take it to the next level, including integration of machine learning-based anomaly detection, external alert notifications, and integration with third-party case/ticket management platforms, incident response systems, and SOAR platforms.</p><p>SIEM vendors typically charge their customers based on data ingestion rate, such as average events per second (EPS) or daily indexed volume (DIV). This licensing model has several drawbacks. First, it creates a financial disincentive for organizations to collect and ingest the data that might actually help them detect threats more effectively. Second, it creates a situation where some security data is “dropped on the floor” or sent to another storage area where special processes need to be invoked to ever look at it. Third, it creates an unpredictable and uncomfortable budgeting situation for these organizations planning for expansion of data.</p><p>In contrast, our <a href="https://elastic.ac.cn/pricing/philosophy" target="_self">resource-based pricing</a> does not charge by data ingestion rate, nor by seat. Instead, it allows you to pay only for what you use in terms of resources supporting your security operations. You get to choose how much data you ingest, how long you retain it, and what kind of security workflows you enable on the data.</p><h2>Why open SIEM?</h2><p>What do we mean by open? It’s way more than just “open source” or “open code,” which have defined a new age of disruptive approaches to SIEM. Our users value the open community, open roadmap, open data model, and an open approach to detections.</p><h3>Open community</h3><p>One of the things that users tell us they love about Elastic from both a cultural and technological perspective is the community. If you have any questions, the Elastic community’s got your back. When it comes to SIEM, our team members answer user questions daily on the <a href="https://discuss.elastic.co/c/siem" target="_self">SIEM Discuss Forum</a> and in our <a href="https://elasticstack.slack.com" target="_self">public #siem Slack channel</a>, as well as in other places we engage with users.&nbsp;</p><p>Community is especially critical in cybersecurity. The security space isn't about a single tool or individual, but rather how teams collaborate across organizations, geographies, datasets, and practices to protect valuable data and infrastructures against threats. With Elastic SIEM, Elastic actively fosters this security community — whether through our products, partnerships, sponsorships, training, or the amazing original research coming out of the Elastic community itself.</p><p>Many organizations have used Elastic as their security analytics platform and home-baked SIEM solution for years. For example, SANS, one of the leaders in security training and certification, has long used <a href="https://www.sans.org/course/siem-with-tactical-analytics/course/desc/community" target="_self">Elasticsearch as a training platform for SIEM</a>, not to mention the many other tools and training programs that leverage Elastic for threat hunting, insider threat, and fraud detection.</p><h3>Open roadmap</h3><p>Another powerful aspect of open development practices that we’ve adopted is an open roadmap and issue tracker. Whether you are just trying to figure out what we are focused on or chasing down a specific enhancement request, you can just jump into the <a href="https://github.com/elastic/kibana/issues?q=is%3Aopen+is%3Aissue+label%3ATeam%3ASIEM" target="_self">public Github issues</a>, start educating yourself, upvote topics, and file new ones. You’ll see our designs for future directions completely in the open, as well as our engineers engaging in triaging and getting more details on your ideas. We also accept contributions, which is another way to move the needle.</p><h3>Open data model</h3><p>In the Elastic Stack, every piece of information you add is “just another index” in our open data model. Data is stored as JSON, with no proprietary formats. We clearly document our schema assumptions in <a href="https://elastic.ac.cn/guide/en/ecs/1.5/index.html" target="_self">ECS</a>, which is community-driven and extensible. This means no vendor lock-in.&nbsp;</p><p>Elastic SIEM includes “inspect” functions so you can see the actual queries used by the app. Our SIEM rules are published and fully <a href="https://elastic.ac.cn/guide/en/siem/guide/current/prebuilt-rules.html" target="_self">documented</a>, so you can see and understand the detection logic — no “black box” detections here. Even our machine learning-based anomaly detection jobs are readily viewable and can be copied and edited to create your own custom jobs. Want to add another data source to SIEM? No problem. On top of that, we never rate-limit your ability to get data out; it’s your data and you have the control.</p><h2>Try it for yourself</h2><p>Want to give Elastic SIEM a spin? Try Elastic SIEM on <a href="https://elastic.ac.cn/cloud/elasticsearch-service/signup" target="_self">Elasticsearch Service</a> on Elastic Cloud, or check out an <a href="https://demo.elastic.co/app/siem#/overview?_g=()&timerange=(global:(linkTo:!(timeline),timerange:(from:1580667870898,fromStr:now-24h,kind:relative,to:1580754270898,toStr:now)),timeline:(linkTo:!(global),timerange:(from:1580667870898,fromStr:now-24h,kind:relative,to:1580754270898,toStr:now)))" target="_self">Elastic SIEM demo</a>. Already have <a href="https://elastic.ac.cn/blog/migrating-to-elastic-common-schema-in-beats-environments" target="_self">ECS</a>-formatted data format in Elasticsearch? Just upgrade to 7.6 of the <a href="https://elastic.ac.cn/downloads/" target="_self">Elastic Stack</a> to get your SOC rolling.</p><p>Want to experience Elastic Security live? Virtually participate in an <a href="https://elastic.ac.cn/elasticon/" target="_self">Elastic{ON} Tour</a> event.</p><p>Learn more about SIEM with the <a href="https://elastic.ac.cn/what-is/siem" target="_self">SIEM Overview Guide</a>.</p>

blog-banner-release-security.png

Open security

blog-thumb-release-security.png

Elastic SIEM is free and open for security analysts everywhere

<p>Data in the real world is rarely simple - often times it is a jumble of interlocking relations.</p><p>How do you represent relational data in Elasticsearch? There are a few mechanisms that can be used to provide relation support. Each has their pros and cons, making them useful for different situations.</p><h3>Inner Objects</h3><p>The simplest mechanism are named "inner objects". These are JSON objects embedded inside your parent document:</p><pre class="prettyprint noescape">{<br />  "name":"Zach",<br />  "car":{<br />    "make":"Saturn",<br />    "model":"SL"<br />  }<br />}<br /></pre><p>Simple, right? The "car" field is just another JSON object, with the inner object having two properties ("make" and "model"). This inner object mapping will work as long as you have a one-to-one relationship between the root object and the inner object. E.g. every person has at most one "car".</p><p>But what if Zach owns two cars, and we add another person (Bob) who owns just one car?</p><pre class="prettyprint noescape">{<br />  "name" : "Zach",<br />  "car" : [<br />    {<br />      "make" : "Saturn",<br />      "model" : "SL"<br />    },<br />    {<br />      "make" : "Subaru",<br />      "model" : "Imprezza"<br />    }<br />  ]<br />}<br />{<br />  "name" : "Bob",<br />  "car" : [<br />    {<br />      "make" : "Saturn",<br />      "model" : "Imprezza"<br />    }<br />  ]<br />}<br /></pre><p>Ignoring the fact that Saturn never made an Imprezza car, what happens when we try to search for it? Logically, only Bob has a "Saturn Imprezza", so we should be able to do a query like:</p><p>`query: car.make=Saturn AND car.model=Imprezza`</p><p>Right? <strong>Well, no, that doesn't work</strong>. If you perform that query, you'll receive both documents as the result. What happens is that Elasticsearch internally flattens inner objects into a single object. So Zach's entry <em>actually</em> looks like this:</p><pre class="prettyprint noescape">{<br />  "name" : "Zach",<br />  "car.make" : ["Saturn", "Subaru"]<br />  "car.model" : ["SL", "Imprezza"]<br />}<br /></pre><p>Which explains why it was returned as a result. Elasticsearch is fundamentally flat, so internally the documents are represented as flattened fields. Hmm.</p><h3>Nested</h3><p>As an alternative to inner objects, Elasticsearch provides the concept of " <a href="https://elastic.ac.cn/guide/reference/mapping/nested-type.html">nested types</a>". Nested documents look identical to inner objects at the document level, but provide the functionality we were missing above (as well as some limitations).</p><p>Example nested document:<br /></p><pre class="prettyprint noescape">{<br />  "name" : "Zach",<br />  "car" : [<br />    {<br />      "make" : "Saturn",<br />      "model" : "SL"<br />    },<br />    {<br />      "make" : "Subaru",<br />      "model" : "Imprezza"<br />    }<br />  ]<br />}<br /></pre><p>At the mapping level, nested types must be explicitly declared (unlike inner objects, which are automatically detected):</p><pre class="prettyprint noescape">{<br />  "person":{<br />    "properties":{<br />      "name" : {<br />        "type" : "string"<br />      },<br />      "car":{<br />        "type" : "nested"<br />      }<br />    }<br />  }<br />}<br /></pre><p>The problem with inner objects was that each nested JSON object is not treated as an individual component of the document. Instead they were merged with other inner objects sharing the same property names.</p><p>This is not the case with nested documents. Each nested doc remains independent, and you can perform a query like `car.make=Saturn AND car.model=Imprezza` without a problem.</p><p>Elasticsearch is still fundamentally flat, but it manages the nested relation internally to give the appearance of nested hierarchy. When you create a nested document, Elasticsearch actually indexes two separate documents (root object and nested object), then relates the two internally. Both docs are stored in the same Lucene block on the same Shard, so read performance is still very fast.</p><p>This arrangement does come with some disadvantages. Most obvious, you can only access these nested documents using a special ` <a href="https://elastic.ac.cn/guide/reference/query-dsl/nested-query.html">nested query</a>`. Another big disadvantage comes when you need to update the document, either the root or any of the objects.</p><p>Since the docs are all stored in the same Lucene block, and Lucene never allows random write access to it's segments, updating one field in the nested doc will force a reindex of the <em>entire</em> document.</p><p>This includes the root and any other nested objects, even if they were not modified. Internally, ES will mark the old document as deleted, update the field and then reindex everything into a new Lucene block. If your data changes often, nested documents can have a non-negligible overhead associated with reindexing.</p><p>Lastly, it is not possible to "cross reference" between nested documents. One nested doc cannot "see" another nested doc's properties. For example, you are not able to filter on "A.name" but facet on "B.age". You can get around this by using `include_in_root`, which effectively copies the nested docs into the root, but this get's you back to the problems of inner objects.</p><h3>Parent/Child</h3><p>The last method that Elasticsearch provides are <a href="https://elastic.ac.cn/guide/reference/mapping/parent-field.html">Parent/Child types</a>. This scheme is a looser coupling than nested, and gives you a set of slightly more powerful queries. Let's look at an example where a single person has multiple homes (in different states). Your parent has a mapping as usual, perhaps:</p><pre class="prettyprint noescape">{<br />  "mappings":{<br />    "person":{<br />      "name":{<br />        "type":"string"<br />      }<br />    }<br />  }<br />}<br /></pre><p>While your children have their own mapping outside the parent, with a special `_parent` property set:<br /></p><pre class="prettyprint noescape">{<br />  "homes":{<br />    "_parent":{<br />      "type" : "person"<br />    },<br />    "state" : {<br />      "type" : "string"<br />    }<br />  }<br />}<br /></pre><p>The `_parent` field tells Elasticsearch that the "Employers" type is a child of the "Person" type. Adding documents to this scheme is very easy. The parent doc is indexed as normal:</p><pre class="prettyprint noescape">$ curl -XPUT localhost:9200/test/person/zach/ -d'<br />{<br />  "name" : "Zach"<br />}<br /></pre><p>And indexing children documents is <em>almost</em> like normal, except you need to specify which parent this child belongs to in the query parameter ('zach' in this case, which is the ID that we used in the above document):<br /></p><pre class="prettyprint noescape">$ curl -XPOST localhost:9200/homes?parent=zach -d'<br />{<br />  "state" : "Ohio"<br />}<br />$ curl -XPOST localhost:9200/test/homes?parent=zach -d'<br />{<br />  "state" : "South Carolina"<br />}<br /></pre><p>Both of these documents are now associated with the 'zach' parent document, which allows you to use special queries such as:</p><ul><li><a href="https://elastic.ac.cn/guide/en/elasticsearch/reference/current/query-dsl-has-parent-query.html" target="_self">Has Parent Query</a>, which works on parent documents and return children.</li><li><a href="https://elastic.ac.cn/guide/en/elasticsearch/reference/current/query-dsl-has-child-query.html" target="_self">Has Child Query</a>, which works on children documents and returns parents</li></ul><p>You can also query the parents or children types individually, since they are first-class types and will respond to normal queries as usual (you just can't use the relationship values).</p><p>The big problem with Nested was their storage: everything is stored in the same Lucene block. Parent/Child removes this limitation by separating the two documents and only loosely coupling them. There are some pros and cons to this. The loose coupling means you are more free to update/delete children docs, since they have no effect on the parent or other children.</p><p>The downside is that Parent/Child are slightly less performant than Nested. The children docs are routed to the same shard as the parent, so they will still benefit from shard-level caches and memory filtering. But they aren't quite as fast as nested since they are not colocated in the same Lucene block. There is also a bit more memory overhead, since ElasticSearch needs to keep an in-memory "join table", which manages the relations.</p><p>Lastly, you'll run into situations where sorting or scoring are, frankly, very difficult. For example, it is impossible to tell <em>which</em> child documents matched your `Has_Child` filter, just that one of the docs of the returned parent matched the criteria. This can be frustrating depending on your use-case.</p><h3>Denormalization</h3><p>Sometimes the best option is to simply denormalize your data where appropriate. The relational facilities that Elasticsearch provides are great for certain scenarios...but were never meant to provide the robust relational features that you expect from an RDBM.</p><p>At it's heart, Elasticsearch is a flat hierarchy and trying to force relational data into it can be very challenging. Sometimes the best solution is to judiciously choose which data to denormalize, and where a second query to retrieve children is acceptable. Denormalization gives you arguably the most power and flexibility.</p><p>Of course, this comes with the burden of administrative overhead. <em>You</em> get to manage relations, and <em>you</em> get to perform the required queries/filters to associate the various types. Yay!</p><h3>Conclusion and Recap</h3><p>This turned into a long, wordy article, so here is a bulletted recap:</p><p><strong>Inner Object</strong></p><ul><li>Easy, fast, performant</li><li>Only applicable when one-to-one relationships are maintained</li><li>No need for special queries</li></ul><p><strong>Nested</strong></p><ul><li>Nested docs are stored in the same Lucene block as each other, which helps read/query performance. Reading a nested doc is faster than the equivalent parent/child.</li><li>Updating a single field in a nested document (parent or nested children) forces ES to reindex the entire nested document. This can be very expensive for large nested docs</li><li>"Cross referencing" nested documents is impossible</li><li>Best suited for data that does not change frequently</li></ul><p><strong>Parent/Child</strong></p><ul><li>Children are stored separately from the parent, but are routed to the same shard. So parent/children are slightly less performance on read/query than nested</li><li>Parent/child mappings have a bit extra memory overhead, since ES maintains a "join" list in memory</li><li>Updating a child doc does not affect the parent or any other children, which can potentially save a lot of indexing on large docs</li><li>Sorting/scoring can be difficult with Parent/Child since the Has Child/Has Parent operations can be opaque at times</li></ul><p><strong>Denormalization</strong></p><ul><li>You get to manage all the relations yourself!</li><li>Most flexible, most administrative overhead</li><li>May be more or less performant depending on your setup</li></ul>

ELK/Elastic Stack

Log management

Managing Relations Inside Elasticsearch

Elastic SIEM is a great way to provide security analytics and monitoring capabilities to small businesses and homes. Check out Part 1 of this series on SIEM for home networks and small businesses.

blog-banner-castle-tower.jpg

Getting started

Continuous monitoring

SIEM-home-blog-thumb.jpg

Elastic SIEM for home and small business: Getting started

In part two of our three-part Practical BM25 series, learn about the BM25 model and the structure and components of the BM25 formula.

tuning_tools.jpg

Kibana

Search applications

Search analytics

Relevance

bm_25_tuning_pt2.jpg

Practical BM25 — Part 2: The BM25 algorithm and its variables

Practical BM25 — Part 2: The BM25 Algorithm and its variables

Real user monitoring

A Sip of Elastic RUM (Real User Monitoring)

Announcing the beta launch of the Elastic App Search web crawler, a simple yet powerful way to ingest publicly available web content so it becomes instantly searchable on your website.

<p>In Elastic Enterprise Search 7.11, we’re thrilled to announce the beta launch of Elastic App Search web crawler, a simple yet powerful way to ingest publicly available web content so it becomes instantly searchable on your website.</p>
<p>Making content on websites searchable can take several forms. <a href="https://elastic.ac.cn/app-search/" target="_self">Elastic App Search</a> already lets users ingest content via JSON uploading, JSON pasting, and through API endpoints. In this release, the introduction of the beta web crawler gives users another convenient content ingestion method.</p>
<div class="video embed-container shadow m-b-40" style="height: 319.725px;"><img class="vidyard-player-embed" src="https://play.vidyard.com/fAt7DBU5BEVoycw7Zd5Uew.jpg" data-uuid="fAt7DBU5BEVoycw7Zd5Uew" data-v="4" data-type="inline" data-autoplay="1" data-loop="1" data-disable_analytics="1" data-hidden_controls="1" data-muted="1" disable_analytics="1" width="569" /></div>
<p>Available for both self-managed and Elastic Cloud deployments, the web crawler retrieves information from publicly accessible web sites and makes the content searchable in your App Search engines. App Search does a lot of heavy lifting in the background on your behalf to make that searchable content relevant and easy to tune with sliders — not code.&nbsp;</p>
<p>Now let’s dive into <em>why</em> we are introducing the web crawler into App Search.</p>
<h2><strong>What makes this web crawler different?&nbsp;</strong></h2>
<p>Short answer: Behold, Elastic Cloud.&nbsp;</p>
<p>If you’ve followed Elastic Enterprise Search over the years (we love our fan club), you’ll remember the web crawler was (and still is) available in <a href="https://elastic.ac.cn/site-search/" target="_self">Elastic Site Search</a>. However, only Elastic App Search and Workplace Search are available on the hugely popular <a href="https://elastic.ac.cn/cloud/" target="_self">Elastic Cloud</a>.&nbsp;</p>
<p>You may be asking, “Yeah, so?”&nbsp;</p>
<p>Well, moving the completely redesigned and re-architected web crawler to App Search on Elastic Cloud has several compelling advantages:</p>
<ul>
  <li><strong>Peace of mind with perks:</strong> As the managed service for Elasticsearch and Kibana, Elastic Cloud provides the superior speed, scale, and relevance that defines Elastic. One-click upgrades, simple scaling, and index lifecycle management (ILM) are just a few reasons customers flock to Elastic Cloud. And if you’re already an Elastic Observability or Elastic Security customer, you can manage your entire deployment in one powerful console.</li>
</ul>
<ul>
  <li><strong>Your data, your choice</strong>: Elastic Cloud is available in <a href="https://elastic.ac.cn/guide/en/cloud/current/ec-regions-templates-instances.html" target="_self">more than 40 global regions</a> on the world’s top cloud providers: Google Cloud (GCP), Microsoft Azure, and Amazon Web Services (AWS). Your data, your cloud, your way.</li>
</ul>
<ul>
  <li><strong>Pricing:</strong> With Elastic’s novel resource-based pricing, you don’t have to worry about arcane metrics like number of users, number of queries, document size, or agents deployed. Your cost comes down to the hardware resources used to store, search, and analyze your data, no matter the use case.</li>
</ul>
<p>While we’re focusing on cloud deployments in this blog, it’s important to note that the App Search web crawler will now also be available as a self-managed deployment method — an option not available with <a href="https://elastic.ac.cn/site-search/" target="_self">Elastic Site Search</a> (or <a href="https://swiftype.com/" target="_self">Swiftype</a>).</p>
<h2><strong>What exactly does the web crawler, well, crawl?</strong></h2>
<p>Before we dive into how to set up the web crawler, let’s first review the what — as in what does the web crawler crawl on the public websites you specify.</p>
<p>The web crawler will visit a webpage when you provide a URL, like https://elastic.ac.cn. From there, the web crawler will follow <em>each new link</em> it finds on that page and extract content for ingestion into your App Search engine. This is content discovery. Each discovered link is crawled in a similar way. The “tree” illustration shows how this works at a high level.</p>
<p><img class="shadow" src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/bltd079d0afd366c587/6014796627b1080f84371f97/app-search-web-crawler-visual.png" data-sys-asset-uid="bltd079d0afd366c587" alt="App Search web crawler - crawling through a tree of linked pages" /></p>
<p>In the image above, all of the blue pages were crawled and indexed. However, none of the pages linked to the pink page, so it <em>will not</em> be crawled or indexed. For the web crawler to visit a page that is not interlinked, the page must be provided directly as an entry point or be included within a sitemap. We’ll cover how to set up entry points later in this blog.</p>
<h3><strong>Types of content extracted</strong></h3>
<p>For the beta release of the web crawler, the following content can be extracted from the HTML pages:</p>
<ul>
  <li aria-level="1">Page title</li>
  <li aria-level="1">Description (meta)</li>
  <li aria-level="1">Keywords (meta)</li>
  <li aria-level="1">Body (normalized, with html tags stripped out)</li>
  <li aria-level="1">Canonical URL</li>
  <li aria-level="1">Additional URLs (for the same document)</li>
  <li aria-level="1">Links</li>
</ul>
<h2><strong>Hands-on: Getting started with the web crawler</strong></h2>
<div class="video embed-container shadow m-b-40" style="height: 319.725px;"><img class="vidyard-player-embed shadow m-b-40" src="https://play.vidyard.com/QrcimKbjdV1hW4ogWqZNso.jpg" data-uuid="QrcimKbjdV1hW4ogWqZNso" data-v="4" data-type="inline" data-autoplay="1" data-loop="1" data-hidden_controls="1" data-muted="0" muted="0" width="569" /></div>
<p>Let’s start at the beginning and create a new Elastic Enterprise Search deployment on the Elastic Cloud. For existing Elastic <a href="https://elastic.ac.cn/site-search/" target="_self">Site Search</a> customers, <a href="https://swiftype.com/" target="_self">Swiftype customers</a>, or those new to Elastic Cloud, be sure to sign up for a <a href="https://cloud.elastic.co/registration?elektra=blog-introducing-app-search-crawler" target="_self">free 14-day trial</a> to experience the beauty of the web crawler.</p>
<ul>
  <li>On <a href="https://elastic.ac.cn" target="_self">www.elastic.co</a>, select “Log in” from the top-right corner.</li>
  <li>Several SSO methods are available. Or create a new account.</li>
  <li>Once logged in, select “Create deployment”.</li>
  <li>Choose the Elastic Enterprise Search deployment template. This template is optimized for CPU output, storage, and availability zones. All deployment templates can be tailored to your specific needs after creating a deployment.</li>
  <li>Select your cloud provider from the list. The choice is yours: Google Cloud (GCP), Microsoft Azure, or Amazon Web Services (AWS)</li>
  <li>Name your deployment and then click “Create Deployment”.</li>
  <li>You’ll see a notification screen showing your deployment has been created.&nbsp;</li>
</ul>
<div class="video embed-container shadow m-b-40" style="height: 319.725px;"><img class="vidyard-player-embed" src="https://play.vidyard.com/icLyWEvS9Exoj3L6979Bs5.jpg" data-uuid="icLyWEvS9Exoj3L6979Bs5" data-v="4" data-type="inline" data-autoplay="1" data-loop="1" data-hidden_controls="1" data-muted="0" muted="0" width="569" /></div>
<p>Congrats! You’re on your way to creating your first App Search engine.</p>
<p>The Elastic Enterprise Search solution includes two applications: App Search and <a href="https://elastic.ac.cn/workplace-search" target="_self">Workplace Search</a>. For this tutorial, select the “Launch App Search” button.</p>
<p><img class="shadow" src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt56aa9339ce722187/6014773d0069f70f777217d1/enterprise-search-overview.jpg" data-sys-asset-uid="blt56aa9339ce722187" alt="Welcome screen for Elastic Enterprise Search new deployment" /></p>
<p>Well done! You’re now in App Search and ready to roll with creating a web crawler.</p>
<p>The onboarding flow helps you create your first search engine. Simply name your engine (something like “my-elastic-search-engine” will work) and then you’ll see a screen offering four ways to ingest your data: paste JSON, upload a JSON file, index by API, or use the web crawler. By now, you know which one to choose.</p>
<p>At this point, you can choose to add your own website, or for fun select Elastic.co as the domain URL to crawl. Remember, the web crawler will visit the specified webpage when you provide the URL extracting content along the way. From there it will follow each new link on discovered pages until the web crawler hits a dead end.</p>
<div class="video embed-container shadow m-b-40" style="height: 319.725px;"><img class="vidyard-player-embed" src="https://play.vidyard.com/PG5SqAVwVCrnXp8EYnGbdR.jpg" data-uuid="PG5SqAVwVCrnXp8EYnGbdR" data-v="4" data-type="inline" data-autoplay="1" data-loop="1" data-hidden_controls="1" data-muted="0" muted="0" width="569" /></div>
<p>This is where the Entry Points feature comes in handy. If there’s an “island” page that isn’t linked from other pages, simply add that full URL as an entry point. From there, the web crawler will start indexing that content and continue finding new links for content extraction until it can go no farther.</p>
<p>From the same console page, you can create crawl rules. These rules allow admins to include or exclude pages where the URL matches the rule. For example, perhaps your marketing department uses campaign landing pages — indicated by the path pattern /lp. These landing pages are fine for driving new business with targeted content but maybe not not the type of content you want included in your search engine.&nbsp;</p>
<p>In the crawl rules section, add a new policy that disallows indexing content with any URL path that contains /lp.&nbsp;&nbsp;</p>
<p>The suspense! Now it’s time to crawl. When all of your entry points and crawl rules are completed, select the <strong>Start a Crawl</strong> button.&nbsp;</p>
<div class="video embed-container shadow m-b-40" style="height: 319.725px;"><img class="vidyard-player-embed" src="https://play.vidyard.com/SfGPHLPLVed2aUpBiurqiy.jpg" data-uuid="SfGPHLPLVed2aUpBiurqiy" data-v="4" data-type="inline" data-autoplay="1" data-loop="1" data-hidden_controls="1" data-muted="0" muted="0" width="569" /></div>
<p>Click over to the Documents tab and watch as your content is ingested into the App Search engine. Or click the Query Tester icon at the top-right of the screen to search your engine from anywhere in the App Search UI.</p>
<p>If you want to immediately test your results in a search box, select the Reference UI tab. From here you can use the out-of-the-box, React-based search box. Or better yet, build and customize your own search experience using the Elastic <a href="https://elastic.ac.cn/enterprise-search/search-ui" target="_self">Search UI</a> JavaScript libraries.</p>
<h2><strong>Now it’s your turn</strong></h2>
<p>We think you’ll enjoy the powerful yet simple design of the web crawler. So now it’s your turn to try it out!</p>
<p>The Elastic App Search web crawler is currently in beta and available on all <a href="https://elastic.ac.cn/pricing/" target="_self">subscription levels</a> and available on self-managed and Elastic Cloud deployments. Existing Elastic Cloud customers can access Enterprise Search directly from the <a href="https://cloud.elastic.co/" target="_self">Elastic Cloud console</a>.&nbsp;</p>
<p>New to the Elastic Cloud? Take a look at our <a href="https://elastic.ac.cn/training/free#quick-starts" target="_self">Quick Start guides</a> — bite-sized training videos to get you started quickly — and then start a <a href="https://cloud.elastic.co/registration?elektra=blog-introducing-app-search-crawler" target="_self">free 14-day trial of Elastic Enterprise Search</a>. Or <a href="https://elastic.ac.cn/downloads/enterprise-search" target="_self">download</a> the self-managed versions of App Search or Workplace Search for free.</p>
<h2><strong>Resources:</strong></h2>
<p>Blog: <a href="https://elastic.ac.cn/blog/whats-new-elastic-enterprise-search-7-11-0-web-crawler-box-content-source" target="_self">What’s New in Elastic Enterprise Search: Web crawler and Box as a content source</a></p>
<p>Docs: <a href="https://elastic.ac.cn/guide/en/app-search/7.11/web-crawler.html" target="_self">App Search web crawler</a></p>
<p>Getting Started: <a href="https://elastic.ac.cn/enterprise-search" target="_self">Elastic Cloud: Start a free 14-day trial</a></p>

blog-banner-app-search-web-crawler.png

Search UI

Visualization

Semantic search

Cloud search

Real-time analysis

blog-thumbnail-app-search-web-crawler.jpg

Introducing the Elastic App Search web crawler

<p>When a document is deleted or updated (= delete + add), <a href="https://lucene.apache.org">Apache Lucene</a> simply marks a bit in a per-segment bitset to record that the document is deleted. All subsequent searches simply skip any deleted documents.</p>
<p>It is not until <a href="https://blog.mikemccandless.com/2011/02/visualizing-lucenes-segment-merges.html">segments are merged</a> that the bytes consumed by deleted documents are reclaimed. Likewise, any terms that occur only in deleted documents (ghost terms) are not removed until merge. This approach is necessary because it would otherwise be far too costly to update Lucene's write-once index data structures and aggregate statistics for every document deletion, but it has some implications:</p>
<ul>
  <li>Deleted documents tie up disk space in the index.</li>
  <li>In-memory per-document data structures, such as norms or field data, will still consume RAM for deleted documents.</li>
  <li>Search throughput is lower, since each search must check the deleted bitset for every potential hit. More on this below.</li>
  <li>Aggregate term statistics, used for query scoring, will still reflect deleted terms and documents. When a merge completes, the term statistics will suddenly jump closer to their true values, changing hit scores. In practice this impact is minor, unless the deleted documents had divergent statistics from the rest of the index.</li>
  <li>A deleted document ties up a document ID from the maximum 2.1 B documents for a single shard. If your shard is riding close to that limit (not recommended!) this could matter.</li>
  <li>Fuzzy queries can have slightly different results, because they may match ghost terms.</li>
</ul>
<h2>Merging Reclaims Deleted Documents</h2>
<p>Lucene's default merge policy, <a href="https://blog.mikemccandless.com/2011/02/visualizing-lucenes-segment-merges.html">TieredMergePolicy</a>, already prefers merges that would reclaim more deleted documents, other factors being equal. Over time this means segments with more deletions will be targeted for merging. While it does have a tunable setting (<a href="https://elastic.ac.cn/guide/en/elasticsearch/reference/current/index-modules-merge.html">index.merge.policy.reclaim_deletes_weight</a>) to control how aggressively it targets deletions, it is dangerous to increase this too much otherwise it could select poor (costly) merge choices, dwarfing any gains from slightly fewer deleted documents.</p>
<p>I was curious how effective its defaults are in practice, so I ran a simple worst-case indexing test. First, I built an initial index with 100 M added documents (no deletions) derived from <a href="https://wikipedia.org">Wikipedia's</a> English export. Then I updated that index by forever randomly replacing an existing document (never adding a new document), so that every add also incurs a deletion.</p>
<p>There was no pattern to the updates, such as favoring replacing older or newer documents. This is unrealistic, but it is a good worst case test because the deletes accumulate uniformly, in proportion to each segment's size. In real usage, certain segments (old or new) would accumulate deletions at a faster rate and thus be more quickly selected for merging.</p>
<p>I measured the percentage of deleted (but not yet merged away) documents over time, computed as maxDoc/numDocs - 1.0 (where numDocs is constant at 100 M in my test). The graph below shows an initial startup transient, when the percentage quickly rise from 0% to 45% at which point a couple of large merges complete and bring it back down. After that the deletions percentage hovers between 35% and 60%, with a sawtooth shape showing a sudden drop whenever varying sized merges finish. It looks somewhat like the stock market!</p>
<p><img class="img-responsive" src="https://api.contentstack.io/v2/assets/575e4d869e7a83165490ea31/download?uid=bltb4fbb3a7a8f22d8f?uid=bltb4fbb3a7a8f22d8f" /></p>
<p>A maximum sized segment (default: 5 GB) will only be eligible for merging once it accumulates 50% deletions. If this is too slow for your usage, try decreasing that maximum (<a href="https://elastic.ac.cn/guide/en/elasticsearch/reference/current/index-modules-merge.html">index.merge.policy.max_merged_segment</a>): this will result in a somewhat larger segment count, but the reclaiming should happen more quickly, especially when there is a pattern to the deletions.</p>
<h2>How Do Deleted Documents Affect Search Performance?</h2>
<p>Because deleted documents remain in the index, they must still be decoded from the postings lists and then skipped during searching, so there is added search cost. To test how much, I ran a search performance test for varying queries using the 100 M document index with no deletions as the baseline, and the same index with 50% deleted documents (i.e., 150 M documents with 50M deleted). Both indices were single-segment. Here are the results:</p>
<div class="table-wrapper">
  <table>
    <tbody>
      <tr>
        <th>Query </th>
        <th align="right">QPS&nbsp;&nbsp;&nbsp;&nbsp; </th>
        <th align="right">StdDev&nbsp;&nbsp;&nbsp;&nbsp; </th>
        <th align="right">QPS with deletes&nbsp;&nbsp;&nbsp;&nbsp; </th>
        <th align="right">StdDev with deletes&nbsp;&nbsp;&nbsp;&nbsp; </th>
        <th align="right">% change </th>
      </tr>
      <tr>
        <td>Int Range query </td>
        <td>1.2 </td>
        <td>(5.1%) </td>
        <td>0.6 </td>
        <td>(1.8%) </td>
        <td><span style="color: red;">46%</span> </td>
      </tr>
      <tr>
        <td>Prefix query </td>
        <td>5.7 </td>
        <td>(5.0%) </td>
        <td>3.4 </td>
        <td>(2.3%) </td>
        <td><span style="color: red;">41%</span> </td>
      </tr>
      <tr>
        <td>Wildcard </td>
        <td>5.3 </td>
        <td>(4.4%) </td>
        <td>3.2 </td>
        <td>(2.2%) </td>
        <td><span style="color: red;">39%</span> </td>
      </tr>
      <tr>
        <td>And High+Low </td>
        <td>91.1 </td>
        <td>(2.0%) </td>
        <td>59.5 </td>
        <td>(2.1%) </td>
        <td><span style="color: red;">34%</span> </td>
      </tr>
      <tr>
        <td>Med Phrase </td>
        <td>36.2 </td>
        <td>(2.8%) </td>
        <td>24.4 </td>
        <td>(1.3%) </td>
        <td><span style="color: red;">32%</span> </td>
      </tr>
      <tr>
        <td>And High+Med </td>
        <td>16.6 </td>
        <td>(1.5%) </td>
        <td>11.2 </td>
        <td>(1.0%) </td>
        <td><span style="color: red;">32%</span> </td>
      </tr>
      <tr>
        <td>Med Term </td>
        <td>12.6 </td>
        <td>(2.6%) </td>
        <td>8.6 </td>
        <td>(6.1%) </td>
        <td><span style="color: red;">31%</span> </td>
      </tr>
      <tr>
        <td>And High+High </td>
        <td>4.4 </td>
        <td>(1.3%) </td>
        <td>3.0 </td>
        <td>(0.9%) </td>
        <td><span style="color: red;">31%</span> </td>
      </tr>
      <tr>
        <td>High Term </td>
        <td>6.1 </td>
        <td>(2.8%) </td>
        <td>4.2 </td>
        <td>(6.1%) </td>
        <td><span style="color: red;">31%</span> </td>
      </tr>
      <tr>
        <td>Fuzzy1 </td>
        <td>33.5 </td>
        <td>(12.7%) </td>
        <td>23.6 </td>
        <td>(8.1%) </td>
        <td><span style="color: red;">29%</span> </td>
      </tr>
      <tr>
        <td>Low Term </td>
        <td>61.1 </td>
        <td>(6.3%) </td>
        <td>43.6 </td>
        <td>(7.1%) </td>
        <td><span style="color: red;">28%</span> </td>
      </tr>
      <tr>
        <td>Med Sloppy Phrase </td>
        <td>7.3 </td>
        <td>(4.4%) </td>
        <td>5.2 </td>
        <td>(1.7%) </td>
        <td><span style="color: red;">28%</span> </td>
      </tr>
      <tr>
        <td>Fuzzy2 </td>
        <td>33.7 </td>
        <td>(13.3%) </td>
        <td>24.2 </td>
        <td>(8.5%) </td>
        <td><span style="color: red;">28%</span> </td>
      </tr>
      <tr>
        <td>Or High+Med </td>
        <td>6.8 </td>
        <td>(5.4%) </td>
        <td>4.9 </td>
        <td>(4.5%) </td>
        <td><span style="color: red;">27%</span> </td>
      </tr>
      <tr>
        <td>Or High+Low </td>
        <td>5.7 </td>
        <td>(5.6%) </td>
        <td>4.1 </td>
        <td>(4.7%) </td>
        <td><span style="color: red;">27%</span> </td>
      </tr>
      <tr>
        <td>Low Phrase </td>
        <td>8.3 </td>
        <td>(2.9%) </td>
        <td>6.0 </td>
        <td>(1.6%) </td>
        <td><span style="color: red;">27%</span> </td>
      </tr>
      <tr>
        <td>Or High+High </td>
        <td>1.5 </td>
        <td>(5.5%) </td>
        <td>1.1 </td>
        <td>(4.7%) </td>
        <td><span style="color: red;">26%</span> </td>
      </tr>
      <tr>
        <td>High Phrase </td>
        <td>2.1 </td>
        <td>(5.1%) </td>
        <td>1.5 </td>
        <td>(2.8%) </td>
        <td><span style="color: red;">25%</span> </td>
      </tr>
      <tr>
        <td>Med Span Near </td>
        <td>15.8 </td>
        <td>(9.3%) </td>
        <td>11.8 </td>
        <td>(3.8%) </td>
        <td><span style="color: red;">25%</span> </td>
      </tr>
      <tr>
        <td>Low Sloppy Phrase </td>
        <td>2.7 </td>
        <td>(3.2%) </td>
        <td>2.0 </td>
        <td>(1.9%) </td>
        <td><span style="color: red;">25%</span> </td>
      </tr>
      <tr>
        <td>Low Span Near </td>
        <td>3.9 </td>
        <td>(4.8%) </td>
        <td>3.2 </td>
        <td>(2.7%) </td>
        <td><span style="color: red;">18%</span> </td>
      </tr>
      <tr>
        <td>High Sloppy Phrase&nbsp;&nbsp;&nbsp;&nbsp; </td>
        <td>2.8 </td>
        <td>(5.9%) </td>
        <td>2.3 </td>
        <td>(4.6%) </td>
        <td><span style="color: red;">18%</span> </td>
      </tr>
      <tr>
        <td>High Span Near </td>
        <td>2.4 </td>
        <td>(4.4%) </td>
        <td>2.0 </td>
        <td>(2.5%) </td>
        <td><span style="color: red;">18%</span></td>
      </tr>
    </tbody>
  </table>
</div>
<p>The bad news is there is clearly a non-trivial performance cost to deleted documents, and this is something we can work to reduce over time (patches welcome!). The good news is the cost is typically quite a bit lower than the percentage deletes (50% in this test) because these documents are filtered out at a low level before any of the costly query matchers and scorers see them. The more costly queries (Phrase, Span) tend to see the lowest impact, which is also good because it is the slow queries that determine node capacity for most applications.</p>
<h2>How About Expunge Deletes?</h2>
<p>Elasticsearch's <a href="/guide/en/elasticsearch/reference/2.4/indices-optimize.html">optimize API accepts an only_expunge_deletes&nbsp;flag</a>, which in turn calls Lucene's IndexWriter.expungeDeletes method. While this will forcefully reclaim space from deleted documents, this operation is very costly: under the hood, it forces merging of any segments that have more than 10% (by default) deletions. Use it sparingly: it is better to let Lucene's natural merging handle reclaiming deletions.</p>
<p>However, if you have an index which receives only deletions (never an added or updated document) then beware that Lucene in this case currently fails to kick off any merges. This is a <a href="https://issues.apache.org/jira/browse/LUCENE-6166">known issue</a> that has been fixed, and will be fixed in Lucene 5.0 and Elasticsearch 2.0. In the meantime, this is an appropriate time to periodically expunge deletes!</p>
<h2>Time-Based Indices</h2>
<table style="background: #FFFFD2;">
  <tbody>
    <tr>
      <td>Editor's Note (June 12, 2018): As you read, please note that the content in this section is relevant for&nbsp;Elasticsearch 2.4. With&nbsp;Elasticsearch 5.x&nbsp;<a href="/guide/en/elasticsearch/reference/5.0/breaking_50_mapping_changes.html">the ttl field was removed</a>.&nbsp;</td>
    </tr>
  </tbody>
</table>
<p>Elasticsearch <a href="/guide/en/elasticsearch/reference/2.4/mapping-ttl-field.html">lets you specify time-to-live</a> for each added document, which means after that time has passed, the document is automatically deleted. This is very useful for certain applications, but it will cause heavy deletions over time.</p>
<p>One simple optimization Lucene uses, that may help in such use cases, is to simply drop a segment once it has accumulated 100% deleted documents, without waiting for it to be merged away. The optimization is somewhat fragile since it only applies when all documents in the segment were deleted, but it is very effective since it is obviously extremely fast and happens before merging. Unfortunately, because TieredMergePolicy picks out of order merges, it reduces how frequently the optimization can apply in time-to-live indices.</p>
<p>If you need to further improve indexing performance with time-to-live documents consider using time-based indices instead, such as one index per day or per week: dropping an entire index is quite a bit more efficient than having Lucene remove a subset of documents. If you are concerned about the loss of granularity with this approach, just add a filter to the request to remove the oldest results from the oldest index.</p>
<p>If you are curious about how many deleted documents are in your shards, use the <a href="https://elastic.ac.cn/guide/en/elasticsearch/reference/current/indices-segments.html">indices segments API</a> to find out. Just don't read too much into it. Overall, besides perhaps decreasing the maximum segment size, it is best to leave Lucene's defaults as-is and not fret too much about when deletes are reclaimed.</p>

Data Loss Prevention (DLP)

Lucene's Handling of Deleted Documents

<section id="post_content">
  <article>
    <table style="background: #FFFFD2;">
      <tbody>
        <tr>
          <td>
            <p><strong>Editor's Note&nbsp;(May 1, 2017):</strong><em> This blog was originally published several major versions of Elasticsearch&nbsp;ago. &nbsp;Since then, new mappings have been made available, but more importantly, new features like the </em><a href="/blog/reindex-is-coming"><em>Reindex API</em></a><em> have made tasks like this&nbsp;substantially easier. &nbsp;The below&nbsp;post remains for archival purposes, but it's recommended you read over the linked reindex blog for a more modern approach to the reindex challenge.</em><br /></p>
            <p><del><strong>Update November 2, 2015:&nbsp;</strong></del><del><em>Make sure to check out the updates with </em></del><a href="https://elastic.ac.cn/blog/elasticsearch-2-0-0-released"><del><em>Elasticsearch mappings introduced in the 2.0 release</em></del></a><del><em>.</em></del><br /></p>
          </td>
        </tr>
      </tbody>
    </table>
    <p>A developer I know sent me a tweet saying:</p>
    <p style="padding-left: 30px;"><em>My biggest problem with using Elastic Search as my model is that I have to reindex whenever I make a schema change. With the size of the data sets that takes a long ass time, and that results in a lot of down time for me. Too much for most applications.</em></p>
    <p>It is quite possible to make schema/mapping changes with zero downtime, but there are too many options available to explain in a tweet, hence this blogpost.</p>
    <h2 id="_the_problem_8201_8212_8201_why_you_can_8217_t_change_mappings">The problem — Why you can't change mappings</h2>
    <p>You can only find that which is stored in your index. In order to make your data searchable, your database needs to know what type of data each field contains and how it should be indexed. If you switch a field type from e.g. a string to a date, all of the data for that field that you already have indexed becomes useless. One way or another, you need to reindex that field.</p>
    <p>This applies not just to Elasticsearch, but to any database that uses indices for searching. And if it isn't using indices then it is sacrificing speed for flexibility.</p>
    <p>Elasticsearch (and Lucene) stores its indices in immutable <em>segments</em> — each segment is a “mini" inverted index. These segments are never updated in place. Updating a document actually creates a new document and marks the old document as deleted. As you add more documents (or update existing documents), new segments are created. A merge process runs in the background merging several smaller segments into a new big segment, after which the old segments are removed entirely.</p>
    <p>Typically, an index in Elasticsearch will contain documents of different types. Each _type has its own schema or <em>mapping</em>. A single segment may contain documents of any type. So, if you want to change the field definition for a single field in a single type, you have little option but to reindex all of the documents in your index.</p>
    <h2 id="_adding_fields_is_free">Adding fields is free</h2>
    <p>A segment only contains indices for fields that actually exist in the documents for that segment. This means that you can add new fields for free, using the <a href="https://elastic.ac.cn/guide/reference/api/admin-indices-put-mapping/">put_mapping API</a>. There is no need to reindex.</p>
    <h2 id="_reindexing_your_data">Reindexing your data</h2>
    <p>The process for reindexing your data is quite simple. First, create a new index with the new mapping and settings:</p><pre class="prettyprint ">curl -XPUT localhost:9200/new_index -H 'Content-Type: application/json' -d '<br />{<br />    "mappings": {<br />        "my_type": { ... new mapping definition ...}<br />    }<br />}<br />'<br /></pre>
    <p>Then, pull the documents in from your old index, using a <a href="https://elastic.ac.cn/guide/reference/api/search/scroll/"><em>scrolled search</em></a> and index them into the new index using the <a href="https://elastic.ac.cn/guide/reference/api/bulk/">bulk API</a>. Many of the client APIs provide a reindex() method which will do all of this for you. Once you are done, you can delete the old index.</p>
    <p><strong>Note:</strong> make sure that you include <a href="https://elastic.ac.cn/guide/reference/api/search/search-type.html">search_type=scan</a> in your search request. This disables sorting and makes “deep paging" efficient.</p>
    <p>The problem with this approach is that the index name changes, which means that you need to change your application to use the new index name</p>
    <h2 id="_reindexing_your_data_with_zero_downtime">Reindexing your data with zero downtime</h2>
    <p>Index aliases give us the flexibility to reindex data in the background, making the change completely transparent to our application. An <a href="https://elastic.ac.cn/guide/reference/api/admin-indices-aliases/">alias</a> is like a symbolic link which can point to one or more real indices.</p>
    <p>The typical workflow is as follows. First, create an index, appending a version or timestamp to the name:</p><pre class="prettyprint ">curl -XPUT localhost:9200/my_index_v1 -H 'Content-Type: application/json' -d '<br />{ ... mappings ... }<br />'<br /></pre>
    <p>Create an alias which points to the index:</p><pre class="prettyprint ">curl -XPOST localhost:9200/_aliases -H 'Content-Type: application/json' -d '<br />{<br />    "actions": [<br />        { "add": {<br />            "alias": "my_index",<br />            "index": "my_index_v1"<br />        }}<br />    ]<br />}<br />'<br /></pre>
    <p>Now your application can speak to my_index as if it were a real index.</p>
    <p>When you need to reindex your data, you can create a new index, appending a new version number:</p><pre class="prettyprint ">curl -XPUT localhost:9200/my_index_v2 -H 'Content-Type: application/json' -d '<br />{ ... mappings ... }<br />'<br /></pre>
    <p>Reindex data from my_index_v1 to the new my_index_v2, then change the myindex alias to point to the new index, in a single atomic step:</p><pre class="prettyprint ">curl -XPOST localhost:9200/_aliases -H 'Content-Type: application/json' -d '<br />{<br />    "actions": [<br />        { "remove": {<br />            "alias": "my_index",<br />            "index": "my_index_v1"<br />        }},<br />        { "add": {<br />            "alias": "my_index",<br />            "index": "my_index_v2"<br />        }}<br />    ]<br />}<br />'<br /></pre>
    <p>And finally, delete the old index:</p><pre class="prettyprint ">curl -XDELETE localhost:9200/my_index_v1<br /></pre>
    <p>You have successfully reindexed all of your data in the background without any downtime. Your application is blissfully unaware that the index has changed.</p>
    <p>While this is the standard approach to managing schema changes, there are a number of other options available to you, which I will discuss below.</p>
    <h2 id="_i_don_8217_t_care_about_old_data">I don't care about old data</h2>
    <p>What if you want to change the datatype for a single field, and you don't care about the fact that the old data is not searchable? In this case, you have a few options:</p>
    <h3 id="_delete_the_mapping">Delete the mapping</h3>
    <p><strong>Update November 2, 2015: </strong><em>Please note that delete mappings are not supported in </em><a href="https://elastic.ac.cn/blog/release-we-have"><em>Elasticsearch 2.0+</em></a><em>.</em><br /></p>
    <p>If you <a href="https://elastic.ac.cn/guide/reference/api/admin-indices-delete-mapping.html">delete the mapping</a> for a specific type, then you can use the <a href="https://elastic.ac.cn/guide/reference/api/admin-indices-put-mapping/">put_mapping API</a>. to create a new mapping for that type in the existing index.</p>
    <blockquote><strong>Note:</strong> when you delete a mapping for a type, you also delete all documents of that type in the index.</blockquote>
    <p>This is particularly useful when you are wanting to change the mapping for a type which contains a small number of documents.</p>
    <h3 id="_rename_the_field">Rename the field</h3>
    <p>Adding new fields is free, so you could just add a new field with a different name and definition to use for all future documents. Of course, this means changing the fieldname used by your application.</p>
    <h3 id="_upgrade_to_a_multi_field">Upgrade to a multi-field</h3>
    <p><a href="https://elastic.ac.cn/guide/reference/mapping/multi-field-type/">Multi-fields</a> allow a single field to be used for different purposes. A typical use case is to index e.g. a title field in two ways: as an analyzed string for querying, and as anot_analyzed string for sorting.</p>
    <p>Any scalar field (ie excluding fields of type object or nested) can be upgraded to a multi-field without reindexing, using the <a href="https://elastic.ac.cn/guide/reference/api/admin-indices-put-mapping/">put_mapping API</a>. For instance, if we have a field called created which is currently mapped as a string:</p><pre class="prettyprint ">{<br />    "created": { "type": "string"}<br />}<br /></pre>
    <p>We can upgrade it to a multi-field, and add a date sub-field to it:</p><pre class="prettyprint ">curl -XPUT localhost:9200/my_index/my_type/_mapping -H 'Content-Type: application/json' -d '<br />{<br />    "my_type": {<br />        "properties": {<br />            "created": {<br />                "type":   "multi_field",<br />                "fields": {<br />                    "created": { "type": "string" },<br />                    "date":    { "type": "date"   }<br />                }<br />            }<br />        }<br />    }<br />}<br />'<br /></pre>
    <p>The original created field still exists as the “main" sub-field, and can be queried as created or created.created. The new date variant can be queried as created.date, and will only be populated for new documents.</p>
    <h2 id="_using_aliases_for_greater_flexibility">Using aliases for greater flexibility</h2>
    <p>Sometimes the above approaches are not enough. Perhaps your application has 100,000 user documents and 10,000,000 blog documents. You want to change the mapping for theuser documents without having to reindex all of the blogs.</p>
    <p>There is no reason that you can't store different types in different indices. Elasticsearch can search across multiple indices as easily as it can search across a single index. This way, you only need to reindex the index containing the type that you want to change. And with judicious use of aliases, the reindexing process can still be entirely transparent to your application.</p>
    <p>With this approach, your application should use a separate alias for each type. For instance, instead of indexing to my_index, you would index user docs to my_index_user andblog docs to my_index_blog:</p><pre class="prettyprint ">curl -XPOST localhost:9200/_aliases -H 'Content-Type: application/json' -d '<br />{<br />    "actions": [<br />        { "add": {<br />            "alias": "my_index_user",<br />            "index": "my_index_v2"<br />        }},<br />        { "add": {<br />            "alias": "my_index_blog",<br />            "index": "my_index_v2"<br />        }}<br />    ]<br />}<br />'<br /></pre>
    <p>To search across user and blog documents, you can just specify both aliases:</p><pre class="prettyprint ">curl localhost:9200/my_index_blog,my_index_user/_search<br /></pre>
    <p>When you want to change the user mapping, first create a new index just for users, and choose the right number of primary shards for just user docs:</p><pre class="prettyprint ">curl -XPUT localhost:9200/my_index_users_v1 -H 'Content-Type: application/json' -d '<br />{<br />    "settings": {<br />        "index": {<br />            "number_of_shards": 1<br />        }<br />    },<br />    "mappings": {<br />        "user": { ... new user mapping ... }<br />    }<br />}<br />'<br /></pre>
    <p>Reindex just the user docs from the old index into the new:</p><pre class="prettyprint ">curl 'localhost:9200/my_index_user/user?scroll=1m&amp;search_type=scan' -H 'Content-Type: application/json' -d '<br />{<br />    "size": 1000<br />}<br />'<br /></pre>
    <p>And update the alias:</p><pre class="prettyprint ">curl -XPOST localhost:9200/_aliases -H 'Content-Type: application/json' -d '<br />{<br />    "actions": [<br />        { "remove": {<br />            "alias": "my_index_user",<br />            "index": "my_index_v2"<br />        }},<br />        { "add": {<br />            "alias": "my_index_user",<br />            "index": "my_index_user_v1"<br />        }}<br />    ]<br />}<br />'<br /></pre>
    <p>You can use a <a href="https://elastic.ac.cn/guide/reference/api/delete-by-query.html">delete-by-query</a> request to remove the user docs from the old index:</p><pre class="prettyprint ">curl -XDELETE localhost:9200/my_index_v1/user<br /></pre>
    <p>From now on, any time you want to change the mapping for user docs, you can use the standard reindexing approach that I described above.</p>
    <h3 id="_using_aliases_without_reindexing">Using aliases without reindexing</h3>
    <p>If you want your changes to apply only to new documents, you can still use the aliases approach, without having to reindex. You would still create a new my_index_user_v1 index, but now you would create two aliases: my_index_user for indexing and my_index_users (plural) for searching:</p><pre class="prettyprint ">curl -XPOST localhost:9200/_aliases -H 'Content-Type: application/json' -d '<br />{<br />    "actions": [<br />        { "add": {<br />            "alias": "my_index_user",<br />            "index": "my_index_user_v1"<br />        }},<br />        { "add": {<br />            "alias": "my_index_users",<br />            "index": "my_index_user_v1"<br />        }},<br />        { "add": {<br />            "alias": "my_index_users",<br />            "index": "my_index_v1"<br />        }},<br />    ]<br />}<br />'<br /></pre>
    <p>The my_index_user alias points just to the new index, and all new user documents would be indexed using this alias. The my_index_users alias points to both the new index AND the old index. So you can search across both indices at the same time. The old index will use the old mapping, and the new index will use the new mapping.</p>
    <p>As you can see, Elasticsearch provides a wealth of options for managing your indices and, with a little forethought, changes can be managed with zero downtime.</p>
    <p><br /></p>
    <p><em>Editor’s Note (May 1, 2017): Starting with 6.0, any curl command to Elasticsearch containing content will require a valid content type header. As a result, this post has been updated to reflect this change and to set readers of this post up for success with future versions.</em><br /></p>
    <aside id="post_tags"></aside>
  </article>
</section>

Deployment

Changing Mapping with Zero Downtime

Cloud migration

A Dive into the Elasticsearch Storage

<table style="background-image: initial;background-position-x: initial;background-position-y: initial;background-size: initial;background-repeat-x: initial;background-repeat-y: initial;background-attachment: initial;background-origin: initial;background-clip: initial;background-color: rgb(255, 255, 210);">
  <tbody>
    <tr>
      <td><strong>Editor's Note (February 28, 2020):</strong> This blog has been update to remove references to G1 GC not being recommended. G1 GC support was added in Elasticsearch 6.5. Additionally, this article refers to <a href="/what-is/elasticsearch-monitoring" target="_self">Elastic Stack monitoring</a> features as Marvel.<br /></td>
    </tr>
  </tbody>
</table>
<p>Engineers can <a href="https://genius.com/2190774" target="_self">resist anything</a> except giving their processes more resources: bigger, better, faster, more of cycles, cores, RAM, disks and interconnects! When these resources are not a bottleneck, this is wasteful but harmless. For processes like Elasticsearch that run on the JVM, the luring temptation is to turn the heap up; what harm could possibly come from having more heap? Alas, the story isn't simple.</p>
<p><br />Java is a <a href="https://en.wikipedia.org/wiki/Garbage_collection_(computer_science)" target="_self">garbage-collected language</a>. Java objects reside in a runtime area of memory called <dfn>the heap</dfn>. When the heap fills up, objects that are no longer referenced by the application (affectionately known as <dfn>garbage</dfn>) are automatically released from the heap (such objects are said to have been <dfn>collected</dfn>). The maximum size of the heap is specified at application startup and fixed for the life the application; this size impacts allocation speed, garbage collection frequency, and garbage collection duration (most notably the dreaded stop-the-world phase which pauses all application threads). Applications have to strike a balance between small heaps and large heaps; the heap <a href="https://en.wikiquote.org/wiki/Wallis,_Duchess_of_Windsor" target="_self">can be too rich or too thin</a>.<br /></p>
<h2>Too Small</h2>
<p>If the heap is too small, applications will be prone to the danger of out of memory errors. While that is the most serious risk from an undersized heap, there are additional problems that can arise from a heap that is too small. A heap that is too small relative to the application's allocation rate leads to frequent small latency spikes and reduced throughput from constant garbage collection pauses. Frequent short pauses impact end-user experience as these pauses effectively shift the latency distribution and reduce the number of operations the application can handle. For Elasticsearch, constant short pauses reduce the number of indexing operations and queries per second that can be handled. A small heap also reduces the memory available for indexing buffers, caches, and memory-hungry features like aggregations and suggesters.</p>
<h2>Too Large</h2>
<p>If the heap is too large, the application will be prone to infrequent long latency spikes from full-heap garbage collections. Infrequent long pauses impact end-user experience as these pauses increase the tail of the latency distribution; user requests will sometimes see unacceptably-long response times. Long pauses are especially detrimental to a distributed system like Elasticsearch because a long pause is indistinguishable from a node that is unreachable because it is hung, or otherwise isolated from the cluster. During a stop-the-world pause, no Elasticsearch server code is executing: it doesn't call, it doesn't write, and it doesn't send flowers. In the case of an elected master, a long garbage collection pause can cause other nodes to stop following the master and elect a new one. In the case of a data node, a long garbage collection pause can lead to the master removing the node from the cluster and reallocating the paused node's assigned shards. This increases network traffic and disk I/O across the cluster, which hampers normal load. Long garbage collection pauses are a top issue for cluster instability.</p>
<h2>Just Right</h2>
<p>The crux of the matter is that undersized heaps are bad, oversized heaps are bad and so it needs to be <a href="https://en.wikipedia.org/wiki/Goldilocks_and_the_Three_Bears" target="_self">just right</a>.</p>
<h2>Oops!...I Did It Again</h2>
<p><br />The engineers behind Elasticsearch have long advised keeping the heap size below <a href="https://elastic.ac.cn/guide/en/elasticsearch/guide/current/heap-sizing.html" target="_self">some threshold near 32 GB</a><a href="#fn1" id="ref1" target="_self"><sup>1</sup></a> (some docs referred to a 30.5 GB threshold). The reasoning behind this advice arises from the notion of compressed ordinary object pointers (or <dfn><a href="https://docs.oracle.com/javase/7/docs/technotes/guides/vm/performance-enhancements-7.html#compressedOop" target="_self">compressed oops</a></dfn>).<br /></p>
<p><br />An ordinary object pointer (or <dfn>oops</dfn>) is a managed pointer to an object and it has the same size as a native pointer. This means that on a 32-bit JVM an oop is 32-bits in size and on a 64-bit JVM an oop is 64-bits in size. Comparing an application that runs on a 32-bit JVM to an application that runs on a 64-bit JVM, the former will usually<a href="#fn2" id="ref2" target="_self"><sup>2</sup></a> perform faster. This is because 32-bit pointers require half of the memory space compared to 64-bit pointers; this is friendlier to limited memory bandwidth, precious CPU caches, and leads to fewer garbage collection cycles as there is more room available on the heap.<br /></p>
<p>Applications that run on a 32-bit JVM are limited to a maximum heap size of slightly less than 4 GB. For modern distributed server applications serving large volumes of data, this is usually too small. But there's a neat trick that can be employed: limit the heap to slightly less than 32 GB and then the JVM can get away with 35-bit oops (since 2<sup>35</sup> = 32 GB). Using thirty-five bits is not friendly to modern CPU architectures, though, so another trick is employed: keep all objects aligned on 8-byte boundaries and then we can assume the last three bits of 35-bit oops are zeros<a href="#fn3" id="ref3" target="_self"><sup>3</sup></a>. Now the JVM can get away with 32-bit object pointers yet still reference 32 GB of heap. These are compressed oops.</p>
<p>Then, exactly like the situation with going from a 32-bit JVM to a 64-bit JVM, comparing an application with a heap size just less than the compressed oops threshold to one with a heap size just more than the compressed oops threshold, the latter will perform worse. What is more, the heap useable to the application will be significantly smaller because of the additional space taken up by the 64-bit oops. Increasing the size of the heap to overcome this loss, however, leads to a larger heap that is subject to the long-pause problem already discussed. For Elasticsearch, our advice is to always stay below the compressed oops threshold.</p>
<h2>It's Complicated</h2>
<p>It turns out that the true story is more complicated than this as there are two additional cutoffs.</p>
<p>The first is natural and easy to understand. If the heap is smaller than 4 GB, the JVM can just use 32-bit pointers.</p>
<p>The second cutoff is less obvious. If the heap will not fit in the first 4 GB of address space, the JVM will next try to reserve memory for the heap within the first 32 GB of address space and then use a zero base for the heap; this is known as <a href="https://docs.oracle.com/javase/7/docs/technotes/guides/vm/performance-enhancements-7.html#zeroBasedCompressedOop" target="_self"><dfn>zero-based compressed oops</dfn></a>. When this reservation can not be granted, the JVM has to fall back to using a non-zero base for the heap. If a zero base can be used, a simple 3-bit shift is all that is needed for encoding and decoding between native 64-bit pointers and compressed oops.</p><pre class="prettyprint prettyprinted">native oop = (compressed oop &lt;&lt; 3)<br /></pre>
<p>But when the base is non-zero, a null check is needed and that additional base must be added and subtracted when encoding and decoding compressed oops.</p><pre class="prettyprint prettyprinted">if (compressed oop is null)<br />  native oop = null<br />else<br />  native oop = base + (compressed oop &lt;&lt; 3)<br /></pre>
<p>This causes a significant drop in performance<a href="#fn4" id="ref4" target="_self"><sup>4</sup></a>. The cutoff for using a zero base varies across operating systems<a href="#fn5" id="ref5" target="_self"><sup>5</sup></a> but 26 GB is a conservative cutoff across a variety of operating systems.</p>
<h2>Less is More</h2>
<p>What frequently happens though is that our advice surrounding compressed oops is interpreted as advice to set the heap as high as it can go while staying under the compressed oops threshold. Instead though, it's better to set the heap as low as possible while satisfying your requirements for indexing and query throughput, end-user query response times, yet large enough to have adequate heap space for indexing buffers, and large consumers of heap space like aggregations, and suggesters. The smaller that you can set the heap, the less likely you'll be subject to detrimental long garbage collection pause, and the more physical memory that will be available for the filesystem cache which continues to be used more and more to great effect by Lucene and Elasticsearch.</p>
<h2>Straight Cache Homie</h2>
<p>Modern operating systems maintain a <a href="https://en.wikipedia.org/wiki/Page_cache" target="_self">filesystem cache</a> of pages accessed from disk. This cache only uses free memory and is handled transparently by the operating system. Once a page is read from the file system and placed in the cache, accessing it is as fast as reading from memory. This means that index segments, term dictionaries, and doc values can be accessed as if they are sitting in memory once they've been placed into the cache. What is more, this cache is not managed by the JVM so we get the benefits of blazingly fast memory speeds without the consequences of being on heap. This is why we continue to recommend having as much memory as possible for the filesystem cache.</p>
<h2>Garbage First</h2>
<p><br />The JVM engineers have developed a concurrent garbage collector known as <a href="https://docs.oracle.com/javase/7/docs/technotes/guides/vm/G1.html" target="_self"><dfn>G1 GC</dfn></a> that was first supported starting in JDK 7u4 and is set to be the <a href="https://openjdk.java.net/jeps/248" target="_self">default collector starting in JDK 9</a><a href="#fn6" id="ref6" target="_self"><sup>6</sup></a>. This collector divides the heap into regions and is designed to first collect regions that are mostly garbage (hence <dfn>G1</dfn>: garbage first). This collector still pauses application threads when collecting, but the idea is that by focusing on regions with the most garbage, these collections will be highly efficient so that application threads need to be paused only briefly. This enables G1 GC to operate on large heaps with predictable pause times. This is exactly what we want!<br /></p>
<h2>Together We Can Prevent Forest Fires</h2>
<p>The Elasticsearch heap <a href="https://elastic.ac.cn/guide/en/elasticsearch/reference/current/setup-configuration.html#setup-configuration" target="_self">can be specified at startup</a> through the ES_HEAP_SIZE environment variable. The ideal scenario, if you can, is to size your heap below 4 GB. If you have to go above 4 GB, try to stay below the zero-based threshold for your system. You can check if you're under the zero-based threshold by starting Elasticsearch with the JVM options -XX:+UnlockDiagnosticVMOptions -XX:+PrintCompressedOopsMode and looking for output similar to</p><pre>heap address: 0x000000011be00000, size: 27648 MB, zero based Compressed Oops<br /></pre>
<p>showing that zero-based compressed oops are enabled instead of</p><pre>heap address: 0x0000000118400000, size: 28672 MB, Compressed Oops with base: 0x00000001183ff000<br /></pre>
<p>showing that zero-based compressed oops are not enabled. If you have to go above the zero-based threshold, stay below the compressed oops threshold. Starting with Elasticsearch 2.2.0, Elasticsearch <a href="https://github.com/elastic/elasticsearch/pull/15489" target="_self">logs at startup</a> whether or not it is using compressed oops, and the same information is also available in the <a href="https://elastic.ac.cn/guide/en/elasticsearch/reference/current/cluster-nodes-info.html" target="_self">nodes info API</a>.</p>
<p>Here are some points-of-consideration for reducing the need for large heaps:</p>
<ul>
  <li>Reduce the use of <a href="https://elastic.ac.cn/blog/support-in-the-wild-my-biggest-elasticsearch-problem-at-scale" target="_self">field data</a> and take advantage of <a href="https://elastic.ac.cn/blog/disk-based-field-data-a-k-a-doc-values" target="_self">doc values</a> where possible (the default for every possible field starting in Elasticsearch 2.0.0)<a href="#fn7" id="ref7" target="_self"><sup>7</sup></a>.</li>
  <li><a href="https://issues.apache.org/jira/browse/LUCENE-6504" target="_self">Disk-based norms</a> are available starting in Elasticsearch 2.1.0<a href="#fn8" id="ref8" target="_self"><sup>8</sup></a>.</li>
  <li><a href="https://issues.apache.org/jira/browse/LUCENE-6840" target="_self">Doc values consume less memory for multi-fields</a> starting in Elasticsearch 2.2.0.</li>
  <li>Do not over-shard (some advantages among many: a search request across N shards has to collect results from all N shards so fewer shards means smaller result sets to sift through and better request cache utilization, less terms dictionaries, and fewer shards leads to a smaller cluster state).</li>
  <li>Do not use overly-large bulk indexing batch sizes (32 MB is okay, 256 MB is probably not).</li>
  <li>Do not use large bulk indexing queues (to keep the total bytes across all in-flight requests reasonable; <a href="https://github.com/elastic/elasticsearch/issues/16011" target="_self">a circuit breaker will limit this starting in Elasticsearch 5.0.0</a>).</li>
  <li>Do not request too many <a href="https://github.com/elastic/elasticsearch/pull/13188" target="_self">hits in a single request</a>, use scrolling instead.</li>
  <li>Do not request too many aggregation buckets or use deeply-nested aggregations.</li>
  <li>Consider trading performance for memory and <a href="https://elastic.ac.cn/guide/en/elasticsearch/reference/2.3/search-aggregations-bucket-terms-aggregation.html#_collect_mode" target="_self">use breadth_first collection mode</a> for deep aggregations.</li>
  <li>Use <a href="https://elastic.ac.cn/what-is/elasticsearch-monitoring" target="_self">Marvel</a> to monitor the JVM heap over time.<br />
    <figure>
      <div data-img-caption-container="true" style="display: inline-block; text-align: center;"><img src="https://api.contentstack.io/v2/assets/575e4d4f3dc542cb38c0a3c3/download?uid=blt31774e23baadb736" />
        <figcaption style="text-align: center;">This screenshot from Marvel shows a heap that is too small: the garbage collections are barely able to free objects leaving little heap space free after each collection.</figcaption>
      </div>
    </figure>
    <figure>
      <div data-img-caption-container="true" style="display: inline-block; text-align: center;"><img src="https://api.contentstack.io/v2/assets/575e4d4f9e7a83165490e112/download?uid=bltfcad775c7080079c" />
        <figcaption style="text-align: center;">This screenshot from Marvel shows a heap that is too large; the heap is almost exclusively garbage before each collection and this memory is likely better utilized by the filesystem cache.</figcaption>
      </div>
    </figure>
  </li>
</ul>
<p>The engineers behind Lucene and Elasticsearch continue to investigate ways to reduce the need for a large heap. Stay tuned as we push more components of indices off heap, and find ways within Elasticsearch to reduce the dependency on the heap for executing requests.</p>
<hr />
<div class="footnotes">
  <ol>
    <li id="fn1">Throughout this post, <dfn>MB</dfn> and <dfn>GB</dfn> refer to 2<sup>20</sup> = 1,048,576 and 2<sup>30</sup> = 1,073,741,824 bytes, respectively.<a href="#ref1" title="Return." target="_self">↩</a></li>
    <li id="fn2">An application that makes extensive use of 64-bit numerical types might be slower on a 32-bit JVM because it can not take advantage of 64-bit registers and instructions.<a href="#ref2" title="Return." target="_self">↩</a></li>
    <li id="fn3">Aligned objects do lead to a small amount of slop in the heap, but that's okay because <a href="https://en.wikipedia.org/wiki/Data_structure_alignment" target="_self">modern CPUs prefer 8-byte aligned addresses</a>.<a href="#ref3" title="Return." target="_self">↩</a></li>
    <li id="fn4">Extra CPU instructions are not free, and the <a href="https://hg.openjdk.java.net/jdk8/jdk8/hotspot/file/87ee5ee27509/src/cpu/x86/vm/macroAssembler_x86.cpp#l4986" target="_self">branch</a>/<a href="https://hg.openjdk.java.net/jdk8/jdk8/hotspot/file/87ee5ee27509/src/cpu/x86/vm/macroAssembler_x86.cpp#l4924" target="_self">predicated instructions</a> that arise from decoding/encoding a non-zero based oop can be especially expensive.<a href="#ref4" title="Return." target="_self">↩</a></li>
    <li id="fn5">On my laptop running OS X 10.11.4 using Oracle JDK 8u74, I can get up to around a 28250 MB heap before the JVM does not use zero-based oops and on my workstation running Fedora 23 using Oracle JDK 8u74, I can get up to around a 30500 MB heap before the JVM does not use zero-based oops.<a href="#ref5" title="Return." target="_self">↩</a></li>
    <li id="fn6">It is interesting to note that G1 GC was initially proposed as a replacement for the CMS collector but is now being touted as a replacement for the throughput collector.<a href="#ref6" title="Return." target="_self">↩</a></li>
    <li id="fn7">Field data and doc values are used for aggregations, sorting and script field access.<a href="#ref7" title="Return." target="_self">↩</a></li>
    <li id="fn8">Norms are an index-time component of relevance scoring; norms can be disabled if you're not using relevance scoring.<a href="#ref8" title="Return." target="_self">↩</a></li>
  </ol>
</div>
<hr />
<div class="credits"><br />The <a href="https://www.flickr.com/photos/neilsingapore/14312966615/in/photolist-nNMGjt-6N7efX-Jbtt-naBAiZ-4ghavr-brKKia-6GbYot-8593Hc-5zGkqm-2DXig8-CD2x7-naBuTS-daZg32-eg6ygg-u52uB-7juEGz-ox83ov-7hnTZS-cuRwPo-5ETGmc-4BVEaB-fwtsNC-85c9gh-5KD6L-95ccc6-apsfR-4A3o2G-uAPyA-8593Ex-e8Z2oi-64qsRF-obNL6N-859qD8-2j766A-4vDEof-5aCChL-adouGz-3RUPXt-6mb5p8-dfK32H-84Dak9-baAoTZ-8593Fc-naBygo-naBNfQ-85c9ed-nQeJRi-4HnPoE-2h3Lnr-k5GE" target="_self">image</a> associated with this post is licensed under the <a href="https://creativecommons.org/licenses/by-nc/2.0/" target="_self">CC BY-NC 2.0 license</a> and is cropped from the original. The full-bleed <a href="https://www.flickr.com/photos/53357045@N02/4973028423/in/photolist-8zs56n-948FzN-c4R7P7-8zXKV7-mbWY48-jmkq5s-8cU7bw-5bpNE2-8ZDVEY-mbYBxq-r7AQqz-ffFvJT-9SV3h2-9E4Vvr-8FjQxQ-7ZyiDz-bCijr2-bPTpMB-6WASiP-5bpNBk-besLPa-6GXS1R-dLyno3-cH5zty-affZNN-mbWhS4-qUGhRp-pt5Une-qaXnt8-9E4Mwp-czBdo7-7sXYnJ-kPDXTu-74He2k-jSYdTx-ptRXLf-q3Xi74-dNAXGT-aYrJfz-9KPnBa-417zm9-6y6PTK-ySxQY4-f9V2Jx-ontX16-jrfGhs-5PCJt2-eYk3HF-eYwcpE-gu4BSy" target="_self">image</a> associated with this post is licensed under the <a href="https://creativecommons.org/licenses/by/2.0/" target="_self">CC BY 2.0 license</a> and is cropped from the original.<br /></div>

double-trouble-2-husky-dogs-wolves.jpg

Root cause analysis

mine-heaps-720x420.jpg

A Heap of Trouble: Managing Elasticsearch's Managed Heap

<p>When we announced our license change for Elasticsearch and Kibana, moving the Apache 2.0-licensed source code to be dual licensed under both the Elastic License and SSPL, we also <a href="/blog/license-change-clarification">mentioned</a> we would work closely with the community on a simplified and more permissive version of the Elastic License. I am happy to share the results with you.<br /><br />The Elastic License is already widely used. More than 90% of our downloads are under the Elastic License, and those users enjoy the significant additional <a href="/subscriptions">value provided</a> under the free and open Basic tier. Thanks to this value, the majority of our users and community are already using the software under the Elastic License, so they viewed our recent license change as a non-event. We are happy to see it, because our intent is to minimize any disruption to our community.<br /><br />We took this opportunity to engage with our community and find ways to further simplify the Elastic License. After talking to users who reached out for clarification, we believe this new version will help to significantly address the majority of your&nbsp;concerns while protecting our products from <a href="https://twitter.com/kimchy/status/1351534442993446917">abuse, misinformation, and confusion</a>.</p><h2>Elastic License v2</h2><p>The <a href="/licensing/elastic-license">Elastic License v2 (ELv2)</a> is a very simple, non-copyleft license, allowing for the right to "use, copy, distribute, make available, and prepare derivative works of the software” and has only three high-level limitations. You cannot:</p><ol><li>Provide the products to others as a managed service&nbsp;</li><li>Circumvent the license key functionality or remove/obscure features protected by license keys&nbsp;</li><li>Remove or obscure any licensing, copyright, or other notices</li></ol><p>ELv2 applies to all of Elasticsearch and Kibana. It covers the distribution as well as the source code of all free and paid features.<br /></p><p>We share our source code for both free and paid features in the spirit of openness. Unfortunately, our <a href="/blog/dear-search-guard-users-including-amazon-elasticsearch-service-open-distro-and-others">copyrights</a> and <a href="https://twitter.com/kimchy/status/1351534442993446917">trademarks</a> have been abused and misused. Our goal with this updated license is to be as permissive as possible while including a minimum set of protections. I hope these protections make sense. <a href="/licensing/elastic-license/faq">See our FAQ</a> for more information about ELv2.</p><p>We created ELv2 to hopefully allow others to adopt it. This is the license we wished was available in 2015 when we were a small company facing <a href="https://twitter.com/kimchy/status/1351534442993446917">misinformation</a>. It incorporates all of our learnings from our experience and others who have made similar changes (<a href="https://mongodb.ac.cn/blog/post/mongodb-now-released-under-the-server-side-public-license">MongoDB</a>, <a href="https://www.cockroachlabs.com/docs/v24.2/licensing-faqs.html" target="_self">CockroachDB</a>, <a href="https://redislabs.com/blog/redis-labs-modules-license-changes/">RedisLabs</a>, <a href="https://blog.timescale.com/blog/building-open-source-business-in-cloud-era-v2/">TimescaleDB</a>, <a href="https://www.graylog.org/post/graylog-v4-0-licensing-sspl">Graylog</a>, etc.). Hopefully we helped a little here. There are many companies out there facing a similar decision. I hope that over time, those of us with similar goals can coalesce around a smaller number of licenses and that ELv2 will be a catalyst for that.<br /><br />In that spirit, we worked on ELv2 with <a href="https://heathermeeker.com/about-me/">Heather Meeker</a>, a lawyer who is well known for helping to draft many OSS licenses, including the Mozilla Public License 2.0, as well as helping a number of organizations build similar-in-spirit licenses like the <a href="https://www.confluent.io/confluent-community-license/">Confluent Community License</a>, <a href="https://mongodb.ac.cn/licensing/server-side-public-license/faq">SSPL</a>, and others. We are also reaching out to initiatives like the <a href="https://polyformproject.org/">Polyform Project</a> and <a href="https://faircode.io">Fair-code</a> as additional efforts to raise awareness of this license and look for ways to promote its wider use.&nbsp;</p><h2>SSPL remains an option for the source code</h2><p>We added <a href="https://mongodb.ac.cn/licensing/server-side-public-license">SSPL</a>, which is a copyleft license created by <a href="https://mongodb.ac.cn/blog/post/mongodb-now-released-under-the-server-side-public-license">MongoDB</a>, as an option to minimize the effect this license change would have on our users. MongoDB is one of the most popular projects out there, used by millions of developers who are happy with SSPL.<br /><br />Since we made this announcement, we had many of our users reach out and say that they are thankful we provided this option. Their organizations are already using MongoDB, and this made our license change a non-event for them.<br /><br />The SSPL is a licensing option for the source code, as shown below:</p><p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/bltd65944c1b54a1e0b/6018987f29a02c49ba1f3e4a/chart-license-update-2021.jpg" class="img-fluid"/></p><p>And to be clear, we are <a href="https://writing.kemitchell.com/2021/01/20/Righteous-Expedient-Wrong.html">still</a> not claiming that either SSPL or the Elastic License are OSI-approved licenses.</p><h2>Still no impact to our cloud and on-premises customers&nbsp;</h2><p>It is important to repeat: there is no impact to any of our Elastic Cloud or self-managed customers. Our customers already use the default distribution under the Elastic License, and their use is governed by the terms of their subscription agreement, which continues to grant them access to additional features, access to support, and other Elastic commitments (for example, IP infringement indemnification) as before.</p><h2>The path forward</h2><p>These changes, including making the Elastic License more permissive, are intended to help us focus on building great products and investing in our community. This means building more great features, many of which we will provide for free and which will be developed in the open. But our commitment goes beyond the code. These license changes let us focus on what matters: helping you find success with our products.</p>

blog-banner-generic-elastic.png

blog-thumb-generic-elastic-lockup.png

Introducing Elastic License v2, simplified and more permissive; SSPL remains an option

Elastic Cloud helps you quickly and easily find information, gain insights, and protect your technology investments across AWS, Google Cloud, and Microsoft Azure.

<p>We’re pleased to announce you can now run Elastic Cloud on Amazon Web Services (AWS) in the Canada (Central) region.<br>
</p><p>Elastic Cloud gives you the power of enterprise search, observability, and security in the Canada (Central) region of AWS. Easily search applications, websites, and workplace content platforms for information. Quickly gain deep insights into your environment by monitoring applications and uptime as well as analyzing centralized logs and metrics. You can also help protect your technology investment by layering endpoint security with security event monitoring.
</p><h2>Deploy your own way</h2><p>Elastic Cloud gives you the flexibility to run where and how you want. You can choose between our managed service offering — or manage the service yourself. Deploy in the cloud on AWS, Google Cloud, or Microsoft Azure and orchestrate everything with your container of choice.
</p><p>When you log in to the Elastic Cloud console, simply select your preferred region and cloud service provider and choose how you want to optimize your deployment based on the compute, memory, or I/O requirements of your specific use case. Once you click “Create deployment,” your Elastic Cloud workload is provisioned in minutes.
</p><p>Visit our <a href="https://elastic.ac.cn/guide/en/cloud/current/index.html?blade=cloud.elastic.co">documentation</a> to learn more.
</p><h2>Elastic Cloud and marketplace integration
</h2><p>If you’re an AWS customer, you can also subscribe to Elastic Cloud and Elasticsearch Service via the <a href="https://aws.amazon.com/marketplace/pp/B01N6YCISK?qid=1571899641796&sr=0-1&ref_=srh_res_product_title">AWS Marketplace</a>. You can select between usage plans, and integrated billing consolidates your Elastic Cloud usage charges into your existing AWS bill. These charges are deductible from your prepaid AWS Enterprise Discount Program agreements.
</p><p>Getting started is easy: Log in to the <a href="https://cloud.elastic.co/home">Elastic Cloud console</a> or <a href="https://elastic.ac.cn/aws">sign up for a free 14-day trial</a>.
</p><p><br></p>

blog-banner-cloud-aws.png

Cloud

blog-thumb-cloud-aws.png

Elastic Cloud is now available on Amazon Web Services (AWS) in Canada (Central)

Elastic Cloud is now available on Amazon Web Services (AWS) Canada Central

Amazon EC2 M5d general purpose and R5d memory-optimized instances are now used Elastic Cloud’s deployment templates, in all supported AWS regions. They offer higher throughput and lower latency.

<p>Today we’re excited to announce the availability of Amazon EC2 M5d general purpose and R5d memory-optimized instances in all <a href="https://elastic.ac.cn/guide/en/cloud/current/ec-regions-templates-instances.html#ec-aws_regions">supported AWS regions</a> on Elastic Cloud.&nbsp;<br>
</p>
<p>M5d instances provide a balance of compute, memory, and networking resources that are well suited for the following workloads:
</p>
<ul>
	<li>Elastic App Search</li>
	<li>Elasticsearch coordinating nodes</li>
	<li>Elasticsearch CPU-optimized data nodes</li>
	<li>Elasticsearch machine learning nodes</li>
	<li>Elastic Enterprise Search</li>
</ul>
<p>R5d instances are designed to deliver fast performance when processing large data sets in memory and are great for the following workloads:
</p>
<ul>
	<li>Application performance monitoring (APM)</li>
	<li>Elasticsearch master nodes</li>
	<li>Elasticsearch memory-optimized data nodes</li>
	<li>Kibana</li>
</ul>
<p>Elastic Cloud offers a number of deployment templates to help simplify hardware selection. Hardware configurations are based on best practices for your workload, such as I/O optimized for search or memory optimized for frequent data aggregations.&nbsp;
</p>
<p><a href="https://elastic.ac.cn/guide/en/cloud/current/ec-getting-started-templates.html"><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt41d7db265248bc4f/5ef96083bbb71f1e548e107f/m5dr5d_blog_deplyoment_templates.png" data-sys-asset-uid="blt41d7db265248bc4f" alt="AWS_EC2_m5d_r5d_Elastic_Cloud_deployment_templates.png" "=""></a><br>
</p>
<h2>Improved disk performance</h2>
<p>Both of the new M5d and R5d instance families support locally attached NVMe SSD drives, which offer higher throughput and lower latency when compared to Elastic Block Storage (EBS) General Purpose GP2 SSD volumes.
</p>
<h2>Take advantage of new instances</h2>
<p>You automatically benefit from the performance enhancements offered by the M5d and R5d instance families when you launch any new deployment.&nbsp;
</p>

<p><img class="vidyard-player-embed" src="https://play.vidyard.com/L5ygMnFh7af7bwDrGX6Wx1.jpg" data-uuid="L5ygMnFh7af7bwDrGX6Wx1" data-v="4" data-type="inline" data-autoplay="1" data-loop="1" data-disable_analytics="1" data-hidden_controls="1" data-muted="0" style="width: 100%; margin: auto; display: block;">
</p>


<p>You can also migrate your existing deployments to use the new deployment templates and upgraded instance families.
</p>
<h2>Migrate an existing deployment</h2>
<p>To migrate an existing deployment, you must first snapshot your existing deployment. Follow the create deployment workflow and restore your snapshot as follows:&nbsp;
</p>
<ol>
	<li>Click “Create deployment”</li>
	<li>Select Amazon Web Services as your cloud platform</li>
	<li>Select the region that your existing deployment is located in</li>
	<li>Select a deployment to restore from one of its snapshot</li>
	<li>Restore from snapshot</li>
	<li>Choose your preferred deployment template</li>
	<li>Create deployment</li>
</ol>
<p>You can learn more in our <a href="https://elastic.ac.cn/guide/en/cloud/current/ec-create-deployment.html">migration documentation</a>.
</p>
<h2>Get started</h2>
<p>Log in to the <a href="https://cloud.elastic.co">Elastic Cloud console</a> or <a href="https://elastic.ac.cn/elasticsearch/service">sign up for a free 14-day trial</a> to get started with these new instance types. To learn more about the hardware we use, visit our product <a href="https://elastic.ac.cn/guide/en/cloud/current/ec-reference-hardware.html#ec_amazon_ec2_aws">documentation page</a>.&nbsp;
</p>

blog-banner-elasticsearch-service.png

blog-thumb-elasticsearch-service.png

Introducing support for Amazon EC2 M5d and R5d instances on Elastic Cloud

Benefit from more flexibility and reliability with new purchasing options, improved change management, and faster hardware.

<p>We’re pleased to introduce you to the latest Elastic Cloud features and functionality. Grab a cup of your favorite beverage and five minutes, and let’s dive in:<br>
</p> <strong>
<h2>New self-service subscription options</h2></strong>
<p>You can now purchase Gold and Platinum monthly subscriptions directly within the Elastic Cloud console. With just a few clicks, you’ll get access to support SLAs and the exclusive capabilities of the Elastic Stack, including our solutions for enterprise search, observability, and security. We have a <a href="https://elastic.ac.cn/pricing/philosophy">resource-based approach to pricing</a>. You can find the subscription details on our <a href="https://elastic.ac.cn/pricing">pricing page</a>, including a side-by-side feature and support comparison as well as frequently asked questions.&nbsp;
</p>
<p>You can also change your subscription level directly in the console as your business evolves. When you upgrade from Standard to Gold or Platinum, you’ll get instant access to the features and benefits within your selected plan.
</p><center><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt6db1500c23e84059/5efb9c30da5e1e294fcd9023/subscription_change_elastic_cloud_console.png" data-sys-asset-uid="blt6db1500c23e84059" alt="Updating your subscription in the Elastic Cloud console" "=""></center>
<p><br>
</p>
<p>Monthly billing allows you to analyze and adjust your subscription usage without entering into a long-term commitment. This flexibility can be useful as you determine which features make the most sense for your use case and how much of the service you need to consume. When your deployment size and feature requirements become clear, you can switch and commit to an annual subscription with deeper discounts.
</p> <strong>
<h2>In-place configuration changes</h2></strong>
<p>In-place changes allow for faster and more reliable configuration updates. Their speed and reliability come from applying changes to the cluster (like settings, upgrades, and resizing) in place, which is followed by a rolling restart of its nodes. This avoids potentially long-running data migration operations inherent with the <a href="https://elastic.ac.cn/guide/en/cloud-enterprise/current/ece-change-deployment.html">grow-and-shrink</a> approach.&nbsp;
</p>
<p>When you change your deployment’s configuration, Elastic Cloud will choose the optimal strategy to apply the changes, using either grow-and-shrink or in-place configuration change as appropriate.
</p>
<p><br>
</p><center>


<p><img style="width: 100%; margin: auto; display: block;" class="vidyard-player-embed" src="https://play.vidyard.com/PWmPzULwfWi9Wd4HXwipjv.jpg" data-uuid="PWmPzULwfWi9Wd4HXwipjv" data-v="4" data-type="inline" data-autoplay="1" data-loop="1" data-disable_analytics="1" data-hidden_controls="1" data-muted="0" alt="Using the Elastic Cloud console to apply configuration changes in place">
</p></center>
<p><strong></strong>
</p>
<h2><strong>AWS M5d and R5d instance types</strong></h2>
<p>Amazon EC2 M5d general purpose and R5d memory-optimized instances are now used in all supported Amazon Web Services (AWS) regions on Elastic Cloud.
</p><center><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt5cb4a0a4704b4846/5efb9ccf7a4c91297609354a/elastic_cloud_aws_r5d_m5d_instances.png" data-sys-asset-uid="blt5cb4a0a4704b4846" alt="New Elastic Cloud AWS R5d and M5d instances" "=""></center>
<p><br>
</p>
<p>These instance types offer improved disk performance via locally attached NVMe SSD drives. These drives offer higher throughput and lower latency when compared to EC2’s Elastic Block Storage (EBS) General Purpose GP2 SSD volumes from the previous m5 and r4 instances. You can learn more about these new instances in our <a href="https://elastic.ac.cn/blog/amazon-ec2-m5d-r5d-instances-on-elastic-cloud">blog post</a>.
</p> <strong>
<h2>Sign up with your Google account</h2></strong>
<p>You can now sign up for Elastic Cloud using your Google Account. Signing up is easy: with a couple clicks you can use your Google identity to access your Elastic Cloud account instead of maintaining separate credentials.&nbsp;
</p><strong>
<h2>Getting started</h2></strong>
<p>Log in to the <a href="https://cloud.elastic.co/home">Elastic Cloud console</a> to get started. If you’re new to Elastic Cloud, <a href="https://elastic.ac.cn/cloud/elasticsearch-service/signup?elektra=ms-39-announcement-blog">sign up for a free 14-day trial</a> with your Google Account and try it out.&nbsp;
</p>

New on Elastic Cloud: Self-service subscriptions, in-place configuration changes

In this blog post, we will demonstrate how the Elastic Infosec team uses the Elastic Stack with Elastic Endpoint Security to build a fully instrumented malware analysis sandbox using free software.

blog-banner-security-endpoint.png

Fleet/Elastic Agent

Beats

Customer story

Security research

Investigation & incident response

Vulnerability Management (VM)

Automated threat protection

blog-thumb-security-endpoint.png

How to build a malware analysis sandbox with Elastic Security

In our series on the Go client for Elasticsearch, we've covered the architecture and design of the client and how to configure it. In this post, we’ll look at encoding and decoding JSON payloads and efficiently indexing documents in bulk.

搜索 AI 公司

ELK Stack

Elastic Cloud

生成式 AI

搜索

安全

可观测性

按解决方案

行业

客户聚焦

研究

构建

学习

连接

关注我们

关于我们

加入我们

合作伙伴

信任与安全

投资者关系

卓越奖

关于我们

加入我们

合作伙伴

信任与安全

投资者关系

卓越奖