Kibana 、Elasticsearch 又增加了一个奇怪的知识点

最近在公司折腾 Kibana 的时候遇到了个还挺有意思的问题，估计正常情况下几乎没人遇到。

先简单说说 Kibana 和 Elasticsearch ，Elasticsearch 是个分布式、RESTful 风格、非常强大的搜索引擎，被广泛地使用于各个IT公司。与 Logstash 及 Kibana 开源项目组合在一起，形成了 ELK 软件栈。

Elasticsearch 常年霸占 DB-Engines 搜索引擎排名的榜首，且与第二名有不小的差距。

Elasticsearch 在各大厂商有广泛的应用，一般有拿来做日志实时分析，搜索服务，比如搜索商品、文档，时序数据分析等等。

这次是在做 Amazon ES 与 自建 ES 服务对比时遇到的问题。问题现象是在AWS 的 EC2 主机上装了 官网下载的 kibana 后，连接不上 Amazon ES 服务。

https://www.elastic.co/downloads/kibana

下面是详细经过

首先在AWS的管理页面新建了 Amazon ES（7.10版本），确认同内网的EC2主机能正常访问endpoint后，打算在EC2上安装本地 kibana 服务器，连接到 Amazon ES。

根据AWS官方文档对 config/kibana.yml 配置文件进行修改

https://docs.amazonaws.cn/elasticsearch-service/latest/developerguide/es-kibana.html#es-kibana-local

kibana.index: ".kibanalocal"
# Use elasticsearch.url for versions older than 6.6
# elasticsearch.url: "https://domain-endpoint:443"
# Use elasticsearch.hosts for versions 6.6 and later
elasticsearch.hosts: "https://domain-endpoint:443"

然后使用 systemctl 启动服务，发现一直是启动异常。

ubuntu@11:/usr/share/kibana/config$ sudo systemctl status kibana.service 
● kibana.service - Kibana
   Loaded: loaded (/etc/systemd/system/kibana.service; disabled; vendor preset: enabled)
   Active: activating (auto-restart) (Result: exit-code) since Fri 2021-04-07 10:12:15 UTC; 1s ago
  Process: 19550 ExecStart=/usr/share/kibana/bin/kibana (code=exited, status=64)
 Main PID: 19550 (code=exited, status=64)

既然 systemctl 无法启动，那就直接使用 kibana 命令行启动，进一步查看下报错信息去解决。

sudo -u kibana -EH ../bin/kibana
  log   [10:06:16.366] [info][plugins-service] Plugin "auditTrail" is disabled.
  log   [10:06:16.372] [info][plugins-service] Plugin "visTypeXy" is disabled.
  log   [10:06:16.446] [warning][config][deprecation] Config key [monitoring.cluster_alerts.email_notifications.email_address] will be required for email notifications to work in 8.0."
  ...
  log   [10:06:16.776] [warning][config][plugins][security] Generating a random key for xpack.security.encryptionKey. To prevent sessions from being invalidated on restart, please set xpack.security.encryptionKey in kibana.yml
  log   [10:06:16.776] [warning][config][plugins][security] Session cookies will be transmitted over insecure connections. This is not recommended.
  log   [10:06:16.808] [warning][config][encryptedSavedObjects][plugins] Generating a random key for xpack.encryptedSavedObjects.encryptionKey. To be able to decrypt encrypted saved objects attributes after restart, please set xpack.encryptedSavedObjects.encryptionKey in kibana.yml
  log   [10:06:16.813] [warning][ingestManager][plugins] Fleet APIs are disabled due to the Encrypted Saved Objects plugin using an ephemeral encryption key. Please set xpack.encryptedSavedObjects.encryptionKey in kibana.yml.
  log   [10:06:16.924] [warning][config][plugins][reporting] Generating a random key for xpack.reporting.encryptionKey. To prevent sessions from being invalidated on restart, please set xpack.reporting.encryptionKey in kibana.yml
  ...
  log   [10:06:16.937] [warning][actions][actions][plugins] APIs are disabled due to the Encrypted Saved Objects plugin using an ephemeral encryption key. Please set xpack.encryptedSavedObjects.encryptionKey in kibana.yml.
  log   [10:06:16.955] [warning][alerting][alerts][plugins][plugins] APIs are disabled due to the Encrypted Saved Objects plugin using an ephemeral encryption key. Please set xpack.encryptedSavedObjects.encryptionKey in kibana.yml.
  log   [10:06:17.072] [info][monitoring][monitoring][plugins] config sourced from: production cluster
  log   [10:06:17.299] [info][savedobjects-service] Waiting until all Elasticsearch nodes are compatible with Kibana before starting saved objects migrations...
  log   [10:06:17.354] [warning][licensing][plugins] License information could not be obtained from Elasticsearch due to Authentication Exception :: {"path":"/_xpack?accept_enterprise=true","statusCode":401,"response":"{\"Message\":\"Your request: '/_xpack' is not allowed.\"}"} error
  log   [10:06:17.356] [warning][monitoring][monitoring][plugins] X-Pack Monitoring Cluster Alerts will not be available: Authentication Exception
  log   [10:06:17.389] [info][savedobjects-service] Starting saved objects migrations
  log   [10:06:17.425] [info][savedobjects-service] Creating index .kibanalocal_1.
  log   [10:06:17.518] [error][data][elasticsearch] [mapper_parsing_exception]: Failed to parse mapping [_doc]: No handler for type [flattened] declared on field [config]
  log   [10:06:17.519] [warning][savedobjects-service] Unable to connect to Elasticsearch. Error: mapper_parsing_exception
  log   [10:06:17.520] [fatal][root] { ResponseError: mapper_parsing_exception
    at IncomingMessage.response.on (/usr/share/kibana/node_modules/@elastic/elasticsearch/lib/Transport.js:272:25)
    at IncomingMessage.emit (events.js:203:15)
    at endReadableNT (_stream_readable.js:1145:12)
    at process._tickCallback (internal/process/next_tick.js:63:19)
  name: 'ResponseError',
  meta:
   { body: { error: [Object], status: 400 },
     statusCode: 400,
     headers:
      { date: 'Fri, 07 May 2021 10:06:17 GMT',
        'content-type': 'application/json; charset=UTF-8',
        'content-length': '399',
        connection: 'keep-alive',
        'access-control-allow-origin': '*',
        server: 'Jetty(8.1.12.v20130726)' },
     meta:
      { context: null,
        request: [Object],
        name: 'elasticsearch-js',
        connection: [Object],
        attempts: 0,
        aborted: false } } }
  log   [10:06:17.523] [info][plugins-system] Stopping all plugins.

 FATAL  ResponseError: mapper_parsing_exception

启动日志中 warning 都可以先不看，比如提示设置 xpack.security.encryptionKey 这些警告都不是重点，服务无法启动先看 FATAL ResponseError: mapper_parsing_exception

直接去百度上搜索对应的报错解决办法，会发现与实际遇到的情况不一样，没什么参考价值。

经过一番仔细查看官方说明和查找资料，发现原来是 kibana 版本和 ES版本不匹配导致的。

Amazon ES 因为开源许可协议原因只能只用OSS版本，而默认情况下在官网下载的 kibana 是basic版本

https://www.elastic.co/cn/downloads/kibana

需要在选择版本页面选择OSS版本的 kibana 才能和 Amazon ES 连接上

直接访问ES的endpoint也可以查看到当前ES的信息，其中也能查看到当前ES是oss版本还是basic版本

# OSS 版本
{
  "name" : "5960c6706c6fafd1488e0a82062xxxxx",
  "cluster_name" : "072146811111:testes",
  "cluster_uuid" : "Acbla-GOQbmZIQltrEUxxA",
  "version" : {
    "number" : "7.10.2",
    "build_flavor" : "oss",
    "build_type" : "tar",
    "build_hash" : "unknown",
    "build_date" : "2021-03-10T22:29:10.311605Z",
    "build_snapshot" : false,
    "lucene_version" : "8.7.0",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}

# Basic 版本
ubuntu@test:/usr/share/kibana$ curl http://172.31.20.84:9200
{
  "name" : "node-2",
  "cluster_name" : "log",
  "cluster_uuid" : "l3EHQCP-SQG-PCK6CDxxxx",
  "version" : {
    "number" : "7.7.1",
    "build_flavor" : "default",
    "build_type" : "deb",
    "build_hash" : "ad56dce891c901a492bb1ee393f12dfffxxxaxx3",
    "build_date" : "2020-05-28T16:30:01.040088Z",
    "build_snapshot" : false,
    "lucene_version" : "8.5.1",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}

关于OSS版本和Basic版本差异可以查看官方介绍。

差异还是很大的，不少功能都被阉割了，比如常用的生命周期管理和数据流。

各版本差异对比：

https://www.elastic.co/cn/subscriptions

至于为什么Amazon ES 只能使用阉割版本，是因为在 2021年1月20日elastic对 Elasticsearch 和 Kibana 的许可做了变更，不再允许云服务商“白嫖”。

详细经过可以查看官方说明：「对 Elasticsearch 和 Kibana 的许可变更 」

https://www.elastic.co/cn/blog/licensing-change

以及补充说明的两篇博客「 许可协议变更澄清 」和「 为什么我们必须变更许可协议」

如果你们也在开始考虑使用云厂商的ES服务，或许也需要关注下OSS版本的问题。