AWSインテグレーション - Kinesis Data Firehose
MackerelはAWSインテグレーションにてAmazon Kinesis Data Firehoseのメトリック取得や監視に対応しています。AWSインテグレーションで連携を行なった場合、課金対象として1デリバリーストリーム = 1マイクロホストと換算します。またそれに加えて、取得されるメトリックの数に応じて、1マイクロホストあたりのメトリック数上限の超過による請求が行われる場合があります。
AWSインテグレーションの設定方法や対応AWSサービス一覧についてはこちらのページをご確認ください。
AWSインテグレーション
取得メトリック
AWSインテグレーションのKinesis Data Firehose対応で取得できるメトリックは以下の通りです。メトリックの説明に関してはAWSのヘルプをご確認ください。
送信先に指定するサービス毎に、最大で以下のメトリックが取得されます。
- S3の場合：49個
- Elasticsearch Serviceの場合：38個
- OpenSearch Serviceの場合：38個
- Redshiftの場合：42個
- Splunkの場合：39個
- HTTP Endpointの場合：40個
| グラフ名 | メトリック | Mackerel上のメトリック名 | 単位 | Statistics |
|---|---|---|---|---|
| Service Bytes | BackupToS3.Bytes DeliveryToElasticsearch.Bytes DeliveryToAmazonOpenSearchService.Bytes DeliveryToRedshift.Bytes DeliveryToS3.Bytes DeliveryToSplunk.Bytes IncomingBytes DataReadFromKinesisStream.Bytes DeliveryToHttpEndpoint.Bytes DeliveryToHttpEndpoint.ProcessedBytes | firehose.service_bytes.backup_to_s3 firehose.service_bytes.delivery_to_elasticsearch firehose.service_bytes.delivery_to_opensearch firehose.service_bytes.delivery_to_redshift firehose.service_bytes.delivery_to_s3 firehose.service_bytes.delivery_to_splunk firehose.service_bytes.incoming firehose.service_bytes.read_from_kinesis_streams firehose.service_bytes.delivery_to_http_endpoint firehose.service_bytes.delivery_to_http_endpoint_processed | bytes | Sum |
| Service Records | BackupToS3.Records DeliveryToElasticsearch.Records DeliveryToAmazonOpenSearchService.Records DeliveryToRedshift.Records DeliveryToS3.Records DeliveryToSplunk.Records IncomingRecords DataReadFromKinesisStream.Records DeliveryToHttpEndpoint.Records DeliveryToHttpEndpoint.ProcessedRecords | firehose.service_records.backup_to_s3 firehose.service_records.delivery_to_elasticsearch firehose.service_records.delivery_to_opensearch firehose.service_records.delivery_to_redshift firehose.service_records.delivery_to_s3 firehose.service_records.delivery_to_splunk firehose.service_records.incoming firehose.service_records.read_from_kinesis_streams firehose.service_records.delivery_to_http_endpoint firehose.service_records.delivery_to_http_endpoint_processed | integer | Sum |
| Service Success | BackupToS3.Success DeliveryToElasticsearch.Success DeliveryToAmazonOpenSearchService.Success DeliveryToRedshift.Success DeliveryToS3.Success DeliveryToSplunk.Success DeliveryToHttpEndpoint | firehose.service_success.backup_to_s3 firehose.service_success.delivery_to_elasticsearch firehose.service_success.delivery_to_opensearch firehose.service_success.delivery_to_redshift firehose.service_success.delivery_to_s3 firehose.service_success.delivery_to_splunk firehose.service_success.delivery_to_http_endpoint | float | Average |
| Service Data Freshness | BackupToS3.DataFreshness DeliveryToElasticsearch.DataFreshness DeliveryToAmazonOpenSearchService.DataFreshness DeliveryToRedshift.DataFreshness DeliveryToS3.DataFreshness DeliveryToSplunk.DataFreshness DeliveryToHttpEndpoint.DataFreshness | firehose.service_data_freshness.backup_to_s3 firehose.service_data_freshness.delivery_to_elasticsearch firehose.service_data_freshness.delivery_to_opensearch firehose.service_data_freshness.delivery_to_redshift firehose.service_data_freshness.delivery_to_s3 firehose.service_data_freshness.delivery_to_splunk firehose.service_data_freshness.delivery_to_http_endpoint | float | Maximum |
| Service Data Ack Latency | DeliveryToSplunk.DataAckLatency | firehose.service_data_ack_latency.delivery_to_splunk | float | Maximum |
| Millis Behind Latest | KinesisMillisBehindLatest | firehose.millis_behind_latest.kinesis | integer | Sum |
| API Latency | DescribeDeliveryStream.Latency PutRecord.Latency PutRecordBatch.Latency UpdateDeliveryStream.Latency | firehose.api_latency.describe_delivery_stream_average firehose.api_latency.describe_delivery_stream_maximum firehose.api_latency.put_record_average firehose.api_latency.put_record_maximum firehose.api_latency.put_record_batch_average firehose.api_latency.put_record_batch_maximum firehose.api_latency.update_delivery_stream_average firehose.api_latency.update_delivery_stream_maximum | float | Average Maximum |
| API Bytes | PutRecord.Bytes PutRecordBatch.Bytes | firehose.api_bytes.put_record firehose.api_bytes.put_record_batch | bytes | Sum |
| API Requests | DescribeDeliveryStream.Requests PutRecord.Requests PutRecordBatch.Requests UpdateDeliveryStream.Requests | firehose.api_requests.describe_delivery_stream firehose.api_requests.put_record firehose.api_requests.put_record_batch firehose.api_requests.update_delivery_stream | integer | Sum |
| API Records | PutRecordBatch.Records | firehose.api_records.put_record_batch | integer | Sum |
| API Throttled | ThrottledDescribeStream ThrottledGetRecords ThrottledGetShardIterator | firehose.api_throttled.describe_stream firehose.api_throttled.get_records firehose.api_throttled.get_shard_iterator | integer | Sum |
| Data Transformation Duration | ExecuteProcessing.Duration | firehose.data_transformation_duration.execute | float | Average |
| Data Transformation Success | ExecuteProcessing.Success | firehose.data_transformation_success.execute | float | Average |
| Data Transformation Records | SucceedProcessing.Records | firehose.data_transformation_records.succeed | integer | Sum |
| Data Transformation Bytes | SucceedProcessing.Bytes | firehose.data_transformation_bytes.succeed | bytes | Sum |
| Format Conversion Records | SucceedConversion.Records FailedConversion.Records | firehose.format_conversion_records.succeed firehose.format_conversion_records.failed | integer | Sum |
| Format Conversion Bytes | SucceedConversion.Bytes FailedConversion.Bytes | firehose.format_conversion_bytes.succeed firehose.format_conversion_bytes.failed | bytes | Sum |
| Incoming Put Requests | IncomingPutRequests | firehose.incoming_put_requests.count | integer | Sum |
| Throttled Records | ThrottledRecords | firehose.throttled_records.records | integer | Sum |
| Bytes Per Second Limit | BytesPerSecondLimit | firehose.bytes_per_second_limit.limit | bytes | Average |
| Records Per Second Limit | RecordsPerSecondLimit | firehose.records_per_second_limit.limit | float | Average |
| KMS Exceptions | KMSKeyAccessDenied KMSKeyDisabled KMSKeyInvalidState KMSKeyNotFound | firehose.kms_exceptions.key_access_denied firehose.kms_exceptions.key_disabled firehose.kms_exceptions.key_invalid_state firehose.kms_exceptions.key_not_found | integer | Sum |
| Put Requests Per Second Limit | PutRequestsPerSecondLimit | firehose.put_requests_per_second_limit.limit | float | Average |
| Active Partitions | PartitionCount | firehose.active_partitions.count | integer | Maximum |
| Partition Processing Duration [ms] | JQProcessing.Duration | firehose.partition_processing_duration.jq | float | Average |
| Partition Limit | PartitionCountExceeded | firehose.partition_limit.exceeded | integer | Maximum |
| Per Partition Throughput | PerPartitionThroughput | firehose.per_partition_throughput.maximum firehose.per_partition_throughput.average firehose.per_partition_throughput.minimum | bytes/sec | Maximum Average Minimum |
| Delivered S3 Objects | DeliveryToS3.ObjectCount | firehose.delivered_s3_objects.count | integer | Sum |