Fri, Mar 25 2022 3:08:33 pm | [2022/03/25 07:08:33] [debug] [upstream] KA connection #120 to 10.3.4.84:9200 has been assigned (recycled) Spamming errors in the log about failed connections to the - Github flush.go:221 org_id=fake msg="failed to flush user" #5531 - Github Fri, Mar 25 2022 3:08:49 pm | [2022/03/25 07:08:49] [debug] [retry] re-using retry for task_id=3 attempts=3 Hi! Fri, Mar 25 2022 3:08:51 pm | [2022/03/25 07:08:51] [debug] [input:tail:tail.0] inode=69179617 events: IN_MODIFY "}}},{"create":{"_index":"logstash-2022.03.24","_type":"_doc","_id":"NeMmun8BI6SaBP9luqVq","status":400,"error":{"type":"mapper_parsing_exception","reason":"Could not dynamically add mapping for field [app.kubernetes.io/instance]. [2022/03/24 04:20:51] [debug] [retry] re-using retry for task_id=0 attempts=5 Host 10.3.4.84 Scheduling and Retries - Fluent Bit: Official Manual Fri, Mar 25 2022 3:08:23 pm | [2022/03/25 07:08:23] [debug] [outputes.0] task_id=5 assigned to thread #1 Existing mapping for [kubernetes.labels.app] must be of type object but found [text]. Existing mapping for [kubernetes.labels.app] must be of type object but found [text]. "}}},{"create":{"_index":"logstash-2022.03.24","_type":"_doc","_id":"0eMnun8BI6SaBP9lo-jn","status":400,"error":{"type":"mapper_parsing_exception","reason":"Could not dynamically add mapping for field [app.kubernetes.io/instance]. Fri, Mar 25 2022 3:08:30 pm | [2022/03/25 07:08:30] [debug] [task] created task=0x7ff2f1839ee0 id=8 OK Logstash_Format On "}}},{"create":{"_index":"logstash-2022.03.24","_type":"_doc","_id":"2eMmun8BI6SaBP9liJWQ","status":400,"error":{"type":"mapper_parsing_exception","reason":"Could not dynamically add mapping for field [app.kubernetes.io/instance]. Existing mapping for [kubernetes.labels.app] must be of type object but found [text]. FluentD or Collector pods are throwing errors similar to the following: 2022-01-28T05:59:48.087126221Z 2022-01-28 05:59:48 +0000 : [retry_default] failed to flush the buffer. Existing mapping for [kubernetes.labels.app] must be of type object but found [text]. [2022/03/24 04:19:20] [debug] [input chunk] tail.0 is paused, cannot append records [2022/03/24 04:21:20] [debug] [input:tail:tail.0] purge: monitored file has been deleted: /var/log/containers/hello-world-g74nr_argo_main-11e24136e914d43a8ab97af02c091f0261ea8cee717937886f25501974359726.log Fri, Mar 25 2022 3:08:47 pm | [2022/03/25 07:08:47] [debug] [upstream] KA connection #118 to 10.3.4.84:9200 has been assigned (recycled) Fri, Mar 25 2022 3:08:50 pm | [2022/03/25 07:08:50] [debug] [outputes.0] task_id=20 assigned to thread #1 Fri, Mar 25 2022 3:08:30 pm | [2022/03/25 07:08:30] [debug] [input:tail:tail.0] inode=69179617 events: IN_MODIFY Fri, Mar 25 2022 3:08:47 pm | [2022/03/25 07:08:47] [ warn] [engine] failed to flush chunk '1-1648192099.641327100.flb', retry in 60 seconds: task_id=2, input=tail.0 > output=es.0 (out_id=0) Log ingestion to Elastic Cloud not working with ECS Fargate Fri, Mar 25 2022 3:08:48 pm | [2022/03/25 07:08:48] [debug] [upstream] KA connection #120 to 10.3.4.84:9200 has been assigned (recycled) Fri, Mar 25 2022 3:08:48 pm | [2022/03/25 07:08:48] [debug] [input chunk] update output instances with new chunk size diff=695 Existing mapping for [kubernetes.labels.app] must be of type object but found [text]. "}}},{"create":{"_index":"logstash-2022.03.24","_type":"_doc","_id":"Y-Mnun8BI6SaBP9lLtm1","status":400,"error":{"type":"mapper_parsing_exception","reason":"Could not dynamically add mapping for field [app.kubernetes.io/instance]. Fri, Mar 25 2022 3:08:30 pm | [2022/03/25 07:08:30] [debug] [upstream] KA connection #118 to 10.3.4.84:9200 is now available Fri, Mar 25 2022 3:08:21 pm | [2022/03/25 07:08:21] [debug] [input:tail:tail.0] inode=3070975 events: IN_ATTRIB Existing mapping for [kubernetes.labels.app] must be of type object but found [text]. I only changed the output config since its a subchart. [2022/03/24 04:19:20] [debug] [input chunk] tail.0 is paused, cannot append records privacy statement. Otherwise, share steps to reproduce, including your config. Existing mapping for [kubernetes.labels.app] must be of type object but found [text]. Existing mapping for [kubernetes.labels.app] must be of type object but found [text]. Fri, Mar 25 2022 3:08:21 pm | [2022/03/25 07:08:21] [ info] [input:tail:tail.0] inotify_fs_remove(): inode=34055641 watch_fd=11 Fri, Mar 25 2022 3:08:22 pm | [2022/03/25 07:08:22] [debug] [task] created task=0x7ff2f1839760 id=4 OK Fri, Mar 25 2022 3:08:39 pm | [2022/03/25 07:08:39] [ warn] [engine] failed to flush chunk '1-1648192119.62045721.flb', retry in 11 seconds: task_id=13, input=tail.0 > output=es.0 (out_id=0) Fri, Mar 25 2022 3:08:41 pm | [2022/03/25 07:08:41] [debug] [retry] re-using retry for task_id=0 attempts=3 Existing mapping for [kubernetes.labels.app] must be of type object but found [text]. "}}},{"create":{"_index":"logstash-2022.03.24","_type":"_doc","_id":"3-Mmun8BI6SaBP9luq99","status":400,"error":{"type":"mapper_parsing_exception","reason":"Could not dynamically add mapping for field [app.kubernetes.io/instance]. [2022/03/24 04:19:24] [debug] [retry] new retry created for task_id=0 attempts=1 Fri, Mar 25 2022 3:08:29 pm | [2022/03/25 07:08:29] [debug] [input chunk] update output instances with new chunk size diff=1085 Fri, Mar 25 2022 3:08:27 pm | [2022/03/25 07:08:27] [debug] [outputes.0] task_id=2 assigned to thread #0 Existing mapping for [kubernetes.labels.app] must be of type object but found [text]. [2022/03/24 04:19:24] [debug] [retry] new retry created for task_id=1 attempts=1 Fri, Mar 25 2022 3:08:49 pm | [2022/03/25 07:08:49] [ warn] [engine] failed to flush chunk '1-1648192100.653122953.flb', retry in 19 seconds: task_id=3, input=tail.0 > output=es.0 (out_id=0) To those having this same issue, can you share your config and log files with debug level enabled? Match host. Chunk cannot be retried, failed to flush chunk #5916 - Github "}}},{"create":{"_index":"logstash-2022.03.24","_type":"_doc","_id":"1uMmun8BI6SaBP9liJWQ","status":400,"error":{"type":"mapper_parsing_exception","reason":"Could not dynamically add mapping for field [app.kubernetes.io/instance]. Fri, Mar 25 2022 3:08:46 pm | [2022/03/25 07:08:46] [debug] [retry] re-using retry for task_id=12 attempts=2 "}}},{"create":{"_index":"logstash-2022.03.24","_type":"_doc","_id":"0OMnun8BI6SaBP9lo-jn","status":400,"error":{"type":"mapper_parsing_exception","reason":"Could not dynamically add mapping for field [app.kubernetes.io/instance]. Fri, Mar 25 2022 3:08:48 pm | [2022/03/25 07:08:48] [debug] [upstream] KA connection #120 to 10.3.4.84:9200 is now available [2022/03/24 04:20:20] [debug] [input:tail:tail.0] purge: monitored file has been deleted: /var/log/containers/hello-world-7mwzw_argo_main-4a2ecde2fd5310129cdf3e3c7eacc17fc1ae0eb6b5e88bed0fdf8fd7fd1100f4.log ): k3s 1.19.8, use docker-ce backend, 20.10.12. Fri, Mar 25 2022 3:08:21 pm | [2022/03/25 07:08:21] [debug] [upstream] KA connection #120 to 10.3.4.84:9200 is now available Fri, Mar 25 2022 3:08:40 pm | [2022/03/25 07:08:40] [debug] [input chunk] update output instances with new chunk size diff=1085 failed to flush the buffer in fluentd looging. Existing mapping for [kubernetes.labels.app] must be of type object but found [text]. Fri, Mar 25 2022 3:08:50 pm | [2022/03/25 07:08:50] [ warn] [engine] failed to flush chunk '1-1648192119.62045721.flb', retry in 18 seconds: task_id=13, input=tail.0 > output=es.0 (out_id=0) Fri, Mar 25 2022 3:08:44 pm | [2022/03/25 07:08:44] [debug] [retry] new retry created for task_id=17 attempts=1 all logs sent to es, and display at kibana. Fri, Mar 25 2022 3:08:41 pm | [2022/03/25 07:08:41] [debug] [input:tail:tail.0] inode=69179617 events: IN_MODIFY [2022/03/24 04:20:20] [debug] [input:tail:tail.0] purge: monitored file has been deleted: /var/log/containers/hello-world-6lqzf_argo_main-5f73e32f330b82717357220ce404309cd9c3f62e1d75f241f74cbc3086597fa4.log "}}},{"create":{"_index":"logstash-2022.03.24","_type":"_doc","_id":"IOMoun8BI6SaBP9lIP7t","status":400,"error":{"type":"mapper_parsing_exception","reason":"Could not dynamically add mapping for field [app.kubernetes.io/instance]. [2022/03/24 04:20:20] [debug] [input:tail:tail.0] inode=103386716 removing file name /var/log/containers/hello-world-6lqzf_argo_main-5f73e32f330b82717357220ce404309cd9c3f62e1d75f241f74cbc3086597fa4.log Fri, Mar 25 2022 3:08:38 pm | [2022/03/25 07:08:38] [debug] [input:tail:tail.0] inode=69179617 events: IN_MODIFY [2022/03/24 04:20:00] [ warn] [engine] failed to flush chunk '1-1648095560.297175793.flb', retry in 25 seconds: task_id=2, input=tail.0 > output=es.0 (out_id=0) [2022/03/24 04:21:20] [debug] [input:tail:tail.0] 4 new files found on path '/var/log/containers/.log' Fri, Mar 25 2022 3:08:38 pm | [2022/03/25 07:08:38] [debug] [input chunk] update output instances with new chunk size diff=681 Fri, Mar 25 2022 3:08:23 pm | [2022/03/25 07:08:23] [debug] [out coro] cb_destroy coro_id=2 Fri, Mar 25 2022 3:08:21 pm | [2022/03/25 07:08:21] [debug] [input:tail:tail.0] inode=3076476 events: IN_ATTRIB edited Jan 15, 2020 at 19:20. [2022/03/24 04:19:49] [error] [outputes.0] could not pack/validate JSON response [ warn] [engine] failed to flush chunk #3220 - Github "}}},{"create":{"_index":"logstash-2022.03.24","_type":"_doc","_id":"XuMnun8BI6SaBP9lLtm1","status":400,"error":{"type":"mapper_parsing_exception","reason":"Could not dynamically add mapping for field [app.kubernetes.io/instance]. "}}},{"create":{"_index":"logstash-2022.03.24","_type":"_doc","_id":"luMmun8BI6SaBP9l7LpS","status":400,"error":{"type":"mapper_parsing_exception","reason":"Could not dynamically add mapping for field [app.kubernetes.io/instance]. Existing mapping for [kubernetes.labels.app] must be of type object but found [text]. [2022/03/24 04:21:20] [debug] [input:tail:tail.0] scan_glob add(): /var/log/containers/hello-world-89knq_argo_wait-a7f77229883282b7aebce253b8c371dd28e0606575ded307669b43b272d9a2f4.log, inode 1772851 Existing mapping for [kubernetes.labels.app] must be of type object but found [text]. [2022/03/24 04:20:06] [debug] [out coro] cb_destroy coro_id=5 "}}},{"create":{"_index":"logstash-2022.03.24","_type":"_doc","_id":"1OMnun8BI6SaBP9lo-jn","status":400,"error":{"type":"mapper_parsing_exception","reason":"Could not dynamically add mapping for field [app.kubernetes.io/instance]. [2022/03/24 04:19:24] [debug] [out coro] cb_destroy coro_id=0 Fri, Mar 25 2022 3:08:50 pm | [2022/03/25 07:08:50] [debug] [task] created task=0x7ff2f183b560 id=20 OK "}}},{"create":{"_index":"logstash-2022.03.24","_type":"_doc","_id":"1eMmun8BI6SaBP9liJWQ","status":400,"error":{"type":"mapper_parsing_exception","reason":"Could not dynamically add mapping for field [app.kubernetes.io/instance]. Fri, Mar 25 2022 3:08:49 pm | [2022/03/25 07:08:49] [debug] [retry] new retry created for task_id=19 attempts=1 [2022/03/24 04:19:20] [debug] [input chunk] tail.0 is paused, cannot append records Helm chart configuration. **note: removed the leading slash form the first source tag. Existing mapping for [kubernetes.labels.app] must be of type object but found [text]. Fri, Mar 25 2022 3:08:30 pm | [2022/03/25 07:08:30] [debug] [retry] new retry created for task_id=8 attempts=1 Fri, Mar 25 2022 3:08:50 pm | [2022/03/25 07:08:50] [debug] [upstream] KA connection #35 to 10.3.4.84:9200 has been assigned (recycled) [2021/02/23 10:15:04] [ info] [input:tail:tail.0] inotify_fs_remove(): inode=85527545 watch_fd=11 [2021/02/23 10:15:17] [ warn] [engine] failed to flush chunk '1-1614075316.467653746.flb', retry in 6 seconds: task_id=16, input=tail.0 > output=es.0 (out_id=0) [2021/02/23 10:15:17] [ warn] [engine] failed to flush chunk '1-1614075316.380912397 . [2022/03/24 04:20:26] [debug] [out coro] cb_destroy coro_id=5 Fri, Mar 25 2022 3:08:50 pm | [2022/03/25 07:08:50] [debug] [outputes.0] HTTP Status=200 URI=/_bulk Fri, Mar 25 2022 3:08:40 pm | [2022/03/25 07:08:40] [ warn] [engine] failed to flush chunk '1-1648192120.74298017.flb', retry in 10 seconds: task_id=14, input=tail.0 > output=es.0 (out_id=0) The Promtail configuration contains a __path__ entry to a directory that Promtail cannot find. Fri, Mar 25 2022 3:08:21 pm | [2022/03/25 07:08:21] [debug] [out coro] cb_destroy coro_id=1 Fri, Mar 25 2022 3:08:23 pm | [2022/03/25 07:08:23] [debug] [input:tail:tail.0] inode=69179617 events: IN_MODIFY Fri, Mar 25 2022 3:08:51 pm | [2022/03/25 07:08:51] [debug] [upstream] KA connection #35 to 10.3.4.84:9200 is now available Fri, Mar 25 2022 3:08:27 pm | [2022/03/25 07:08:27] [debug] [retry] re-using retry for task_id=2 attempts=2 [2022/03/24 04:19:24] [error] [outputes.0] could not pack/validate JSON response Fri, Mar 25 2022 3:08:30 pm | [2022/03/25 07:08:30] [debug] [upstream] KA connection #120 to 10.3.4.84:9200 has been assigned (recycled) "}}},{"create":{"_index":"logstash-2022.03.24","_type":"_doc","_id":"0uMnun8BI6SaBP9lo-jn","status":400,"error":{"type":"mapper_parsing_exception","reason":"Could not dynamically add mapping for field [app.kubernetes.io/instance]. Logstash_Format On [2022/03/24 04:21:08] [error] [outputes.0] could not pack/validate JSON response Fri, Mar 25 2022 3:08:27 pm | [2022/03/25 07:08:27] [ warn] [engine] failed to flush chunk '1-1648192099.641327100.flb', retry in 9 seconds: task_id=2, input=tail.0 > output=es.0 (out_id=0) Fri, Mar 25 2022 3:08:48 pm | [2022/03/25 07:08:48] [debug] [out coro] cb_destroy coro_id=17 [2022/03/24 04:21:20] [debug] [input:tail:tail.0] scan_blog add(): dismissed: /var/log/containers/fluent-bit-9hwpg_logging_fluent-bit-a7e85dd8e51db82db787e3386358a885ccff94c3411c8ba80a9a71598c01f387.log, inode 35353988 Fri, Mar 25 2022 3:08:40 pm | [2022/03/25 07:08:40] [debug] [out coro] cb_destroy coro_id=13 "}}},{"create":{"_index":"logstash-2022.03.24","_type":"_doc","_id":"O-Mmun8BI6SaBP9luqVq","status":400,"error":{"type":"mapper_parsing_exception","reason":"Could not dynamically add mapping for field [app.kubernetes.io/instance]. "}}},{"create":{"_index":"logstash-2022.03.24","_type":"_doc","_id":"zuMnun8BI6SaBP9lo-jn","status":400,"error":{"type":"mapper_parsing_exception","reason":"Could not dynamically add mapping for field [app.kubernetes.io/instance]. outputs: | Host 10.3.4.84 "}}},{"create":{"_index":"logstash-2022.03.24","_type":"_doc","_id":"3uMmun8BI6SaBP9luq99","status":400,"error":{"type":"mapper_parsing_exception","reason":"Could not dynamically add mapping for field [app.kubernetes.io/instance]. [SERVICE] Flush 5 Daemon off [INPUT] Name cpu Tag fluent_bit [OUTPUT] Name forward Match * Host fd00:7fff:0:2:9c43:9bff:fe00:bb Port 24000. Fluentd error: "buffer space has too many data" - Stack Overflow [2022/03/24 04:19:38] [debug] [outputes.0] HTTP Status=200 URI=/_bulk Hi @yangtian9999. Fri, Mar 25 2022 3:08:21 pm | [2022/03/25 07:08:21] [debug] [input:tail:tail.0] inode=104226845 file has been deleted: /var/log/containers/hello-world-dsfcz_argo_wait-3a9bd9a90cc08322e96d0b7bcc9b6aeffd7e5e6a71754073ca1092db862fcfb7.log Fri, Mar 25 2022 3:08:30 pm | [2022/03/25 07:08:30] [ warn] [engine] failed to flush chunk '1-1648192103.858183.flb', retry in 18 seconds: task_id=5, input=tail.0 > output=es.0 (out_id=0) Fri, Mar 25 2022 3:08:30 pm | [2022/03/25 07:08:30] [debug] [outputes.0] HTTP Status=200 URI=/_bulk Fluentbit failed to send logs to elasticsearch ( Failed to flush chunk Fri, Mar 25 2022 3:08:33 pm | [2022/03/25 07:08:33] [debug] [input chunk] update output instances with new chunk size diff=634 This error happened for 1.8.12/1.8.15/1.9.0. Fri, Mar 25 2022 3:08:41 pm | [2022/03/25 07:08:41] [debug] [upstream] KA connection #35 to 10.3.4.84:9200 is now available I'm trying to configure Loki to use Apache Cassandra both for index and chunk storage. [2022/03/24 04:19:50] [debug] [outputes.0] HTTP Status=200 URI=/_bulk Fri, Mar 25 2022 3:08:48 pm | [2022/03/25 07:08:48] [debug] [upstream] KA connection #35 to 10.3.4.84:9200 is now available Fri, Mar 25 2022 3:08:23 pm | [2022/03/25 07:08:22] [debug] [input chunk] update output instances with new chunk size diff=641 I am seeing this in fluentd logs in kubernetes. "}}},{"create":{"_index":"logstash-2022.03.24","_type":"_doc","_id":"IOMmun8BI6SaBP9lh4vZ","status":400,"error":{"type":"mapper_parsing_exception","reason":"Could not dynamically add mapping for field [app.kubernetes.io/instance]. Fri, Mar 25 2022 3:08:51 pm | [2022/03/25 07:08:51] [debug] [input chunk] update output instances with new chunk size diff=661 [2022/03/24 04:21:08] [ warn] [http_client] cannot increase buffer: current=512000 requested=544768 max=512000 [2022/03/24 04:20:36] [debug] [outputes.0] HTTP Status=200 URI=/_bulk Fri, Mar 25 2022 3:08:50 pm | [2022/03/25 07:08:50] [debug] [upstream] KA connection #120 to 10.3.4.84:9200 is now available Fri, Mar 25 2022 3:08:21 pm | [2022/03/25 07:08:21] [debug] [input:tail:tail.0] inode=35359369 events: IN_ATTRIB [2022/03/24 04:20:20] [debug] [input:tail:tail.0] purge: monitored file has been deleted: /var/log/containers/hello-world-7mwzw_argo_wait-970c00b906c36cb89ed77fe3fa3cd1abc2702078fee737da0062d3b25680bf9c.log Output ES fails to flush chunk with status 0 #3301 - Github Fri, Mar 25 2022 3:08:48 pm | [2022/03/25 07:08:48] [debug] [http_client] not using http_proxy for header Fri, Mar 25 2022 3:08:21 pm | [2022/03/25 07:08:21] [debug] [input:tail:tail.0] inode=3076476 file has been deleted: /var/log/containers/hello-world-89skv_argo_wait-5d919c301d4709b0304c6c65a8389aac10f30b8617bd935a9680a84e1873542b.log Fri, Mar 25 2022 3:08:41 pm | [2022/03/25 07:08:41] [debug] [upstream] KA connection #120 to 10.3.4.84:9200 is now available "}}},{"create":{"_index":"logstash-2022.03.24","_type":"_doc","_id":"lOMmun8BI6SaBP9l7LpS","status":400,"error":{"type":"mapper_parsing_exception","reason":"Could not dynamically add mapping for field [app.kubernetes.io/instance]. Fri, Mar 25 2022 3:08:40 pm | [2022/03/25 07:08:40] [debug] [retry] new retry created for task_id=14 attempts=1 Fri, Mar 25 2022 3:08:48 pm | [2022/03/25 07:08:48] [debug] [outputes.0] HTTP Status=200 URI=/_bulk "}}},{"create":{"_index":"logstash-2022.03.24","_type":"_doc","_id":"z-Mmun8BI6SaBP9liJWQ","status":400,"error":{"type":"mapper_parsing_exception","reason":"Could not dynamically add mapping for field [app.kubernetes.io/instance]. Fri, Mar 25 2022 3:08:50 pm | [2022/03/25 07:08:50] [debug] [upstream] KA connection #118 to 10.3.4.84:9200 is now available [2022/03/24 04:19:49] [debug] [retry] re-using retry for task_id=1 attempts=3 Error failed to flush user | timeout Issue #2143 grafana/loki Fri, Mar 25 2022 3:08:39 pm | [2022/03/25 07:08:39] [debug] [outputes.0] HTTP Status=200 URI=/_bulk [2022/03/24 04:20:36] [debug] [out coro] cb_destroy coro_id=6 Fri, Mar 25 2022 3:08:40 pm | [2022/03/25 07:08:40] [ warn] [engine] failed to flush chunk '1-1648192110.850147571.flb', retry in 11 seconds: task_id=9, input=tail.0 > output=es.0 (out_id=0) Fri, Mar 25 2022 3:08:33 pm | [2022/03/25 07:08:33] [debug] [outputes.0] HTTP Status=200 URI=/_bulk Fri, Mar 25 2022 3:08:50 pm | [2022/03/25 07:08:50] [debug] [input chunk] update output instances with new chunk size diff=650 [2022/03/24 04:21:20] [debug] [input:tail:tail.0] scan_blog add(): dismissed: /var/log/containers/local-path-provisioner-7ff9579c6-mcwsb_kube-system_local-path-provisioner-47a630b5c79ea227664d87ae336d6a7b80fdce7028230c6031175099461cd221.log, inode 444123 Fri, Mar 25 2022 3:08:28 pm | [2022/03/25 07:08:28] [debug] [http_client] not using http_proxy for header Logs not being flushed after x amount of time. [2022/03/24 04:21:20] [debug] [input:tail:tail.0] scanning path /var/log/containers/.log "}}},{"create":{"_index":"logstash-2022.03.24","_type":"_doc","_id":"feMnun8BI6SaBP9l3vN-","status":400,"error":{"type":"mapper_parsing_exception","reason":"Could not dynamically add mapping for field [app.kubernetes.io/instance]. Existing mapping for [kubernetes.labels.app] must be of type object but found [text]. "}}},{"create":{"_index":"logstash-2022.03.24","_type":"_doc","_id":"deMnun8BI6SaBP9l3vN-","status":400,"error":{"type":"mapper_parsing_exception","reason":"Could not dynamically add mapping for field [app.kubernetes.io/instance]. Fri, Mar 25 2022 3:08:21 pm | [2022/03/25 07:08:21] [ warn] [engine] failed to flush chunk '1-1648192100.653122953.flb', retry in 11 seconds: task_id=3, input=tail.0 > output=es.0 (out_id=0) Fri, Mar 25 2022 3:08:21 pm | [2022/03/25 07:08:21] [debug] [input:tail:tail.0] inode=104051102 file has been deleted: /var/log/containers/hello-world-89skv_argo_main-41261a71eea53f67b43c6e1b643d273e59fade2d8d16ee9f4d70e01766e5cc1d.log Environment: Infrastructure: Kubernetes; Deployment tool: helm; Screenshots, promtail config, or terminal output "}}},{"create":{"_index":"logstash-2022.03.24","_type":"_doc","_id":"HuMmun8BI6SaBP9lh4vZ","status":400,"error":{"type":"mapper_parsing_exception","reason":"Could not dynamically add mapping for field [app.kubernetes.io/instance]. Can we have multiple outputs support in fluent bit #5495 - Github Fri, Mar 25 2022 3:08:30 pm | [2022/03/25 07:08:30] [debug] [retry] re-using retry for task_id=5 attempts=2 Fri, Mar 25 2022 3:08:50 pm | [2022/03/25 07:08:50] [debug] [upstream] KA connection #120 to 10.3.4.84:9200 has been assigned (recycled) While trying to solve the issue mentioned here: #1502 I was able to connect to our kubernetes node and apply there the tune2fs -O large_dir /dev/sda. Fri, Mar 25 2022 3:08:46 pm | [2022/03/25 07:08:46] [debug] [http_client] not using http_proxy for header Fri, Mar 25 2022 3:08:41 pm | [2022/03/25 07:08:41] [debug] [out coro] cb_destroy coro_id=14 Fri, Mar 25 2022 3:08:40 pm | [2022/03/25 07:08:40] [debug] [input chunk] update output instances with new chunk size diff=695 Tried to remove everything and leave only in_tail and out_es. [2022/03/24 04:20:20] [debug] [input:tail:tail.0] inode=1756313 removing file name /var/log/containers/hello-world-7mwzw_argo_wait-970c00b906c36cb89ed77fe3fa3cd1abc2702078fee737da0062d3b25680bf9c.log Fri, Mar 25 2022 3:08:41 pm | [2022/03/25 07:08:41] [ warn] [engine] failed to flush chunk '1-1648192113.5409018.flb', retry in 7 seconds: task_id=11, input=tail.0 > output=es.0 (out_id=0) [2022/03/24 04:19:20] [debug] [input chunk] tail.0 is paused, cannot append records Fri, Mar 25 2022 3:08:30 pm | [2022/03/25 07:08:30] [debug] [outputes.0] HTTP Status=200 URI=/_bulk Fri, Mar 25 2022 3:08:36 pm | [2022/03/25 07:08:36] [debug] [outputes.0] task_id=2 assigned to thread #0 "}}},{"create":{"_index":"logstash-2022.03.24","_type":"_doc","_id":"IeMmun8BI6SaBP9lh4vZ","status":400,"error":{"type":"mapper_parsing_exception","reason":"Could not dynamically add mapping for field [app.kubernetes.io/instance]. [2022/03/24 04:19:50] [debug] [upstream] KA connection #102 to 10.3.4.84:9200 is now available [2022/03/24 04:21:20] [debug] [input:tail:tail.0] purge: monitored file has been deleted: /var/log/containers/hello-world-g74nr_argo_wait-227a0fdb4663e03fecebe61f7b6bfb6fdd2867292cacfe692dc15d50a73f29ff.log Existing mapping for [kubernetes.labels.app] must be of type object but found [text]. I can see the logs in Kibana that were successfully uploaded, but the missing log could not be found. Fri, Mar 25 2022 3:08:33 pm | [2022/03/25 07:08:33] [debug] [http_client] not using http_proxy for header Fri, Mar 25 2022 3:08:38 pm | [2022/03/25 07:08:38] [debug] [upstream] KA connection #120 to 10.3.4.84:9200 has been assigned (recycled) Existing mapping for [kubernetes.labels.app] must be of type object but found [text]. hi @lecaros I think this was only [warn] message, I checked with es, I can search the right apps logs. Fri, Mar 25 2022 3:08:51 pm | [2022/03/25 07:08:51] [debug] [outputes.0] task_id=15 assigned to thread #0 Fluentbit is stuck, flb chunks are not flushed and not sending - Github Fri, Mar 25 2022 3:08:39 pm | [2022/03/25 07:08:39] [debug] [out coro] cb_destroy coro_id=12 Fri, Mar 25 2022 3:08:37 pm | [2022/03/25 07:08:37] [debug] [outputes.0] HTTP Status=200 URI=/_bulk Fri, Mar 25 2022 3:08:50 pm | [2022/03/25 07:08:50] [debug] [retry] new retry created for task_id=20 attempts=1 "}}},{"create":{"_index":"logstash-2022.03.24","_type":"_doc","_id":"aeMnun8BI6SaBP9lLtm1","status":400,"error":{"type":"mapper_parsing_exception","reason":"Could not dynamicall But the situation is the same. [2022/03/24 04:19:30] [debug] [out coro] cb_destroy coro_id=1 Fri, Mar 25 2022 3:08:39 pm | [2022/03/25 07:08:39] [debug] [input chunk] update output instances with new chunk size diff=1083 Hi @yangtian9999 [2022/03/24 04:19:38] [ warn] [engine] failed to flush chunk '1-1648095560.205735907.flb', retry in 14 seconds: task_id=0, input=tail.0 > output=es.0 (out_id=0) Your Environment Fluentd or td-agent v. [2022/03/24 04:19:20] [debug] [input chunk] tail.0 is paused, cannot append records