› › ›

修复水印错误

修复水印错误编辑

当数据节点的磁盘空间严重不足并达到满水位磁盘使用水印时，会记录以下错误： 错误：磁盘使用量超过满水位水印，索引具有只读允许删除块。

为了防止磁盘空间完全耗尽，当节点达到此水印时，Elasticsearch 会阻止对该节点上具有分片的任何索引进行写入操作。如果该阻塞影响到相关的系统索引，Kibana 和其他 Elastic Stack 功能可能会变得不可用。

当受影响节点的磁盘使用量低于高磁盘水印时，Elasticsearch 将自动移除写入阻塞。为此，Elasticsearch 会自动将受影响节点的一些分片移动到同一数据层级的其他节点。

要验证分片是否已从受影响节点移走，请使用 cat shards API。

response = client.cat.shards(
  v: true
)
puts response

GET _cat/shards?v=true

如果分片仍保留在节点上，请使用集群分配说明 API 获取其分配状态的说明。

response = client.cluster.allocation_explain(
  body: {
    index: 'my-index',
    shard: 0,
    primary: false,
    current_node: 'my-node'
  }
)
puts response

GET _cluster/allocation/explain
{
  "index": "my-index",
  "shard": 0,
  "primary": false,
  "current_node": "my-node"
}

要立即恢复写入操作，您可以临时增加磁盘水印并移除写入阻塞。

response = client.cluster.put_settings(
  body: {
    persistent: {
      'cluster.routing.allocation.disk.watermark.low' => '90%',
      'cluster.routing.allocation.disk.watermark.low.max_headroom' => '100GB',
      'cluster.routing.allocation.disk.watermark.high' => '95%',
      'cluster.routing.allocation.disk.watermark.high.max_headroom' => '20GB',
      'cluster.routing.allocation.disk.watermark.flood_stage' => '97%',
      'cluster.routing.allocation.disk.watermark.flood_stage.max_headroom' => '5GB',
      'cluster.routing.allocation.disk.watermark.flood_stage.frozen' => '97%',
      'cluster.routing.allocation.disk.watermark.flood_stage.frozen.max_headroom' => '5GB'
    }
  }
)
puts response

response = client.indices.put_settings(
  index: '*',
  expand_wildcards: 'all',
  body: {
    'index.blocks.read_only_allow_delete' => nil
  }
)
puts response

PUT _cluster/settings
{
  "persistent": {
    "cluster.routing.allocation.disk.watermark.low": "90%",
    "cluster.routing.allocation.disk.watermark.low.max_headroom": "100GB",
    "cluster.routing.allocation.disk.watermark.high": "95%",
    "cluster.routing.allocation.disk.watermark.high.max_headroom": "20GB",
    "cluster.routing.allocation.disk.watermark.flood_stage": "97%",
    "cluster.routing.allocation.disk.watermark.flood_stage.max_headroom": "5GB",
    "cluster.routing.allocation.disk.watermark.flood_stage.frozen": "97%",
    "cluster.routing.allocation.disk.watermark.flood_stage.frozen.max_headroom": "5GB"
  }
}

PUT */_settings?expand_wildcards=all
{
  "index.blocks.read_only_allow_delete": null
}

作为长期解决方案，我们建议您向受影响的数据层级添加节点，或升级现有节点以增加磁盘空间。要释放更多磁盘空间，您可以使用删除索引 API 删除不需要的索引。

response = client.indices.delete(
  index: 'my-index'
)
puts response

DELETE my-index

在制定长期解决方案后，请重置或重新配置磁盘水印。

response = client.cluster.put_settings(
  body: {
    persistent: {
      'cluster.routing.allocation.disk.watermark.low' => nil,
      'cluster.routing.allocation.disk.watermark.low.max_headroom' => nil,
      'cluster.routing.allocation.disk.watermark.high' => nil,
      'cluster.routing.allocation.disk.watermark.high.max_headroom' => nil,
      'cluster.routing.allocation.disk.watermark.flood_stage' => nil,
      'cluster.routing.allocation.disk.watermark.flood_stage.max_headroom' => nil,
      'cluster.routing.allocation.disk.watermark.flood_stage.frozen' => nil,
      'cluster.routing.allocation.disk.watermark.flood_stage.frozen.max_headroom' => nil
    }
  }
)
puts response

PUT _cluster/settings
{
  "persistent": {
    "cluster.routing.allocation.disk.watermark.low": null,
    "cluster.routing.allocation.disk.watermark.low.max_headroom": null,
    "cluster.routing.allocation.disk.watermark.high": null,
    "cluster.routing.allocation.disk.watermark.high.max_headroom": null,
    "cluster.routing.allocation.disk.watermark.flood_stage": null,
    "cluster.routing.allocation.disk.watermark.flood_stage.max_headroom": null,
    "cluster.routing.allocation.disk.watermark.flood_stage.frozen": null,
    "cluster.routing.allocation.disk.watermark.flood_stage.frozen.max_headroom": null
  }
}

« 修复常见集群问题断路器错误 »