AWS Integration - Kinesis Data Firehose
Mackerel supports obtaining and monitoring the metric of Amazon Kinesis Data Firehose in AWS Integration. When integrating with AWS Integration, billable targets are determined using the conversion 1 delivery stream = 1 Micro Host. In addition to this, depending on the number of metrics retrieved, you may be charged for exceeding the maximum number of metrics per micro host.
Please refer to the following page for AWS Integration configuration methods and a list of supported AWS services.
AWS Integration
Obtaining metrics
The metrics obtainable with AWS Integration's support for Kinesis Data Firehose are as follows. For Metric explanations, refer to the AWS help page.
The maximum number of metrics obtainable is detemined by destination services:
- In S3 cases：49
- In Elasticsearch Service cases：38
- In OpenSearch Service cases：38
- In Redshift cases：42
- In Splunk cases：39
- In HTTP Endpoint cases：40
| Graph name | Metric | Metric name in Mackerel | Unit | Statistics |
|---|---|---|---|---|
| Service Bytes | BackupToS3.Bytes DeliveryToElasticsearch.Bytes DeliveryToAmazonOpenSearchService.Bytes DeliveryToRedshift.Bytes DeliveryToS3.Bytes DeliveryToSplunk.Bytes IncomingBytes DataReadFromKinesisStream.Bytes DeliveryToHttpEndpoint.Bytes DeliveryToHttpEndpoint.ProcessedBytes | firehose.service_bytes.backup_to_s3 firehose.service_bytes.delivery_to_elasticsearch firehose.service_bytes.delivery_to_opensearch firehose.service_bytes.delivery_to_redshift firehose.service_bytes.delivery_to_s3 firehose.service_bytes.delivery_to_splunk firehose.service_bytes.incoming firehose.service_bytes.read_from_kinesis_streams firehose.service_bytes.delivery_to_http_endpoint firehose.service_bytes.delivery_to_http_endpoint_processed | bytes | Sum |
| Service Records | BackupToS3.Records DeliveryToElasticsearch.Records DeliveryToAmazonOpenSearchService.Records DeliveryToRedshift.Records DeliveryToS3.Records DeliveryToSplunk.Records IncomingRecords DataReadFromKinesisStream.Records DeliveryToHttpEndpoint.Records DeliveryToHttpEndpoint.ProcessedRecords | firehose.service_records.backup_to_s3 firehose.service_records.delivery_to_elasticsearch firehose.service_records.delivery_to_opensearch firehose.service_records.delivery_to_redshift firehose.service_records.delivery_to_s3 firehose.service_records.delivery_to_splunk firehose.service_records.incoming firehose.service_records.read_from_kinesis_streams firehose.service_records.delivery_to_http_endpoint firehose.service_records.delivery_to_http_endpoint_processed | integer | Sum |
| Service Success | BackupToS3.Success DeliveryToElasticsearch.Success DeliveryToAmazonOpenSearchService.Success DeliveryToRedshift.Success DeliveryToS3.Success DeliveryToSplunk.Success DeliveryToHttpEndpoint | firehose.service_success.backup_to_s3 firehose.service_success.delivery_to_elasticsearch firehose.service_success.delivery_to_opensearch firehose.service_success.delivery_to_redshift firehose.service_success.delivery_to_s3 firehose.service_success.delivery_to_splunk firehose.service_success.delivery_to_http_endpoint | float | Average |
| Service Data Freshness | BackupToS3.DataFreshness DeliveryToElasticsearch.DataFreshness DeliveryToAmazonOpenSearchService.DataFreshness DeliveryToRedshift.DataFreshness DeliveryToS3.DataFreshness DeliveryToSplunk.DataFreshness DeliveryToHttpEndpoint.DataFreshness | firehose.service_data_freshness.backup_to_s3 firehose.service_data_freshness.delivery_to_elasticsearch firehose.service_data_freshness.delivery_to_opensearch firehose.service_data_freshness.delivery_to_redshift firehose.service_data_freshness.delivery_to_s3 firehose.service_data_freshness.delivery_to_splunk firehose.service_data_freshness.delivery_to_http_endpoint | float | Maximum |
| Service Data Ack Latency | DeliveryToSplunk.DataAckLatency | firehose.service_data_ack_latency.delivery_to_splunk | float | Maximum |
| Millis Behind Latest | KinesisMillisBehindLatest | firehose.millis_behind_latest.kinesis | integer | Sum |
| API Latency | DescribeDeliveryStream.Latency PutRecord.Latency PutRecordBatch.Latency UpdateDeliveryStream.Latency | firehose.api_latency.describe_delivery_stream_average firehose.api_latency.describe_delivery_stream_maximum firehose.api_latency.put_record_average firehose.api_latency.put_record_maximum firehose.api_latency.put_record_batch_average firehose.api_latency.put_record_batch_maximum firehose.api_latency.update_delivery_stream_average firehose.api_latency.update_delivery_stream_maximum | float | Average Maximum |
| API Bytes | PutRecord.Bytes PutRecordBatch.Bytes | firehose.api_bytes.put_record firehose.api_bytes.put_record_batch | bytes | Sum |
| API Requests | DescribeDeliveryStream.Requests PutRecord.Requests PutRecordBatch.Requests UpdateDeliveryStream.Requests | firehose.api_requests.describe_delivery_stream firehose.api_requests.put_record firehose.api_requests.put_record_batch firehose.api_requests.update_delivery_stream | integer | Sum |
| API Records | PutRecordBatch.Records | firehose.api_records.put_record_batch | integer | Sum |
| API Throttled | ThrottledDescribeStream ThrottledGetRecords ThrottledGetShardIterator | firehose.api_throttled.describe_stream firehose.api_throttled.get_records firehose.api_throttled.get_shard_iterator | integer | Sum |
| Data Transformation Duration | ExecuteProcessing.Duration | firehose.data_transformation_duration.execute | float | Average |
| Data Transformation Success | ExecuteProcessing.Success | firehose.data_transformation_success.execute | float | Average |
| Data Transformation Records | SucceedProcessing.Records | firehose.data_transformation_records.succeed | integer | Sum |
| Data Transformation Bytes | SucceedProcessing.Bytes | firehose.data_transformation_bytes.succeed | bytes | Sum |
| Format Conversion Records | SucceedConversion.Records FailedConversion.Records | firehose.format_conversion_records.succeed firehose.format_conversion_records.failed | integer | Sum |
| Format Conversion Bytes | SucceedConversion.Bytes FailedConversion.Bytes | firehose.format_conversion_bytes.succeed firehose.format_conversion_bytes.failed | bytes | Sum |
| Incoming Put Requests | IncomingPutRequests | firehose.incoming_put_requests.count | integer | Sum |
| Throttled Records | ThrottledRecords | firehose.throttled_records.records | integer | Sum |
| Bytes Per Second Limit | BytesPerSecondLimit | firehose.bytes_per_second_limit.limit | bytes | Average |
| Records Per Second Limit | RecordsPerSecondLimit | firehose.records_per_second_limit.limit | float | Average |
| KMS Exceptions | KMSKeyAccessDenied KMSKeyDisabled KMSKeyInvalidState KMSKeyNotFound | firehose.kms_exceptions.key_access_denied firehose.kms_exceptions.key_disabled firehose.kms_exceptions.key_invalid_state firehose.kms_exceptions.key_not_found | integer | Sum |
| Put Requests Per Second Limit | PutRequestsPerSecondLimit | firehose.put_requests_per_second_limit.limit | float | Average |
| Active Partitions | PartitionCount | firehose.active_partitions.count | integer | Maximum |
| Partition Processing Duration [ms] | JQProcessing.Duration | firehose.partition_processing_duration.jq | float | Average |
| Partition Limit | PartitionCountExceeded | firehose.partition_limit.exceeded | integer | Maximum |
| Per Partition Throughput | PerPartitionThroughput | firehose.per_partition_throughput.maximum firehose.per_partition_throughput.average firehose.per_partition_throughput.minimum | bytes/sec | Maximum Average Minimum |
| Delivered S3 Objects | DeliveryToS3.ObjectCount | firehose.delivered_s3_objects.count | integer | Sum |