Anchor: Navigation

Manufacturing Issue 4.0 - February 2025 Follow the Stream Your Dedicated Community Content Scroll 

Executive's Brief
Supply Chain Resilience
Architect's Blueprint
Developer's Desk
Innovation & Research

The Year of DSP: Data Streaming in 2025 & Shift Left

Welcome to the fourth issue of Follow the Stream! To kick off 2025, this edition is dedicated to the concept of "Shift Left”—a data processing and governance paradigm that moves data processing closer to the source and ingests fresh trustworthy data into any downstream consumers. Covering data streaming predictions and trends for the upcoming year, we explore how Shift Left architecture is revolutionizing data quality and providing reusable data products for both operational and analytical use cases in financial services.
See Industry Updates

Data Streaming Fundamentals

Learn More

Data Streaming Crash Course 

Let's explore the basics of Shift Left and Data Streaming Platforms.

Heading 1

Subtitle 1

Heading 2

Subtitle 2

Heading 3

Subtitle 3
Shift Left is a rethink of how to circulate, share, and manage data. Along with a data streaming platform, it enables organizations to build data once, build it right, and reuse it anywhere within moments of its creation.
Data governance plays an important role ensuring data quality, security, and compliance across the data lifecycle, while data products, built around real-time streams, facilitate reusable, high-quality data assets for efficiency and collaboration.
Subtitle 1
Subtitle 2
Subtitle 3
Anchor: General-Updates

Industry in Motion: General Updates

Download Ebook for Free
Available in German, French, and Spanish!

3 Data Engineering Trends Riding Apache Kafka®, Apache Flink®, and Apache Iceberg™

The Apache Kafka, Flink, and Iceberg communities are continuously evolving, offering innovative ways for engineers to manage and process data at scale. From re-envisioning microservices with Flink streaming applications to enabling real-time AI model applications, these tools are reshaping data integration. With strong community contributions, especially to Iceberg, data governance and real-time analytics are set to accelerate, revolutionizing how businesses manage their data infrastructure.
Read Article

The Data Streaming Landscape 2025

“Data streaming is a new software category. It has grown from niche adoption to becoming a fundamental part of modern data architecture, leveraging open source technologies like Apache Kafka® and Apache Flink®. With real-time data processing transforming industries, the ecosystem of tools, platforms, and cloud services has evolved significantly.” Kai Waehner, Confluent’s Field CTO, “explores the data streaming landscape of 2025, analyzing key players, trends, and market dynamics shaping this space.”
Read Blog

Ushering in a New Era of Data Streaming Confluent Recognized as a Challenger in 2024 Gartner® Magic Quadrant™ for Data Integration Tools

Recognized as a Challenger in Gartner’s 2024 Magic Quadrant for Data Integration Tools, Confluent predicts that the future of data integration will focus on universal data products and event-driven flows, with real-time streaming bridging gaps between operational and analytical data. This shift will drive innovation in areas like fraud detection, personalized experiences, and supply chain optimization, making data streaming the standard for faster, more efficient decision-making.
Read Blog

New in Confluent Cloud: Extending Private Networking Across the Data Streaming Platform

Confluent’s Q4 2024 launch wrapped up the year with a host of powerful features paving the way for even more innovation in 2025. Focusing on delivering private networking and enhanced security across the data streaming platform, key features include mTLS authentication, AWS PrivateLink for Schema Registry, Apache Flink® UDFs for custom stream processing, and WarpStream Orbit for seamless Apache Kafka® migrations.
Read Blog
Back to TopSign Up for Updates
Anchor: Executives-Brief

Executive's Brief: Data Strategy & Leadership

Predictive Analytics: How GenAI and Data Streaming Work Together to Forecast the Future

Predictive analytics, powered by data streaming and GenAI, is reshaping operations and decision-making in the manufacturing industry. By analyzing historical data and applying machine learning models, predictive analytics helps manufacturers forecast equipment failures, optimize production schedules, and streamline supply chain management with greater accuracy. Generative AI enhances these forecasts by generating multiple scenarios, filling data gaps, and adapting to real-time changes for more responsive and dynamic decision-making.
Read Blog

Shift Left: Unifying Operations and Analytics with Data Products

The need for high-quality business data is greater than ever, so preventing and mitigating bad data—across the entire business—has become a critical capability. Shifting data processing and governance “left” eliminates duplicate pipelines, reduces the risk and impact of bad data at the source, and leverages high-quality data products for both operational and analytical use cases.
Read Ebook

Good Teams Manage Kafka – Efficient Teams Use Confluent

Managing open source Apache Kafka® in-house or through basic cloud-hosted services can lead to significant costs, operational challenges, and downtime risks, especially under tight engineering budgets. Using a fully managed, cloud-native platform can help reduce these challenges, enabling teams to focus on strategic initiatives while improving cost efficiency and accelerating time to market for real-time data projects.
Download White Paper
Available in German, French, Spanish, and Portuguese!

How Data Streaming Offers Strategic Insight for Factory-Wide Automation 

Richard Jones, VP EMEA at Confluent, highlights how real-time data streaming enhances factory automation by enabling manufacturers to access data immediately as it is generated. This capability supports the use of AI and machine learning for applications like predictive maintenance and more efficient supply chain management. By streamlining data flow, manufacturers can make better decisions, improve operational efficiency, and foster collaboration across teams and partners.
Read Article

Arcese Group Evolves Into a Logistics Operator with a Digital Heart 

Arcese Group transformed its supply chain efficiency by adopting cloud-based technology for real-time tracking and data visibility. The transition to a modern transportation management system (TMS) reduced track-and-trace data processing times from 45-50 minutes to under one minute. By automating data management and enabling advanced forecasting capabilities, Arcese’s IT team can focus on innovation, positioning the company as a logistics leader with a digital edge.
Read Case Study
Available in German, French, and Italian!
Back to TopSign Up for Updates
Anchor: Customer-Experience

Data Streaming for Supply Chain Resilience

Connecting Smart Factories and the Cloud in Real Time

Kai Waehner, Global Field CTO at Confluent, explains three concrete use cases for companies that are building smart factories, and want to leverage the cloud to add business value. These use cases consider factories connected around the globe, based on the example of BMW Group.
Watch on YouTube

How Siemens Healthineers Leverages Data Streaming with Apache Kafka and Flink in Manufacturing and Healthcare

Siemens Healthineers uses data streaming with Apache Kafka® and Flink® to enhance manufacturing operations, particularly in predictive maintenance and machine integration. By streaming real-time IoT data from machines and robots, they reduce downtime, optimize maintenance schedules, and improve production quality. These data-driven insights help ensure efficient manufacturing processes, delivering critical medical equipment on time while boosting overall operational efficiency.
Read Blog
Back to TopSign Up for Updates
Anchor: Architects-Blueprint

Architect's Blueprint: Data Systems Design

The Streaming Data Quality Guidebook

Maintaining high data quality is critical, especially with the rise of AI/ML, where poor data can lead to system failures, inaccurate decisions, and costly outages. By implementing best practices for stream governance—identifying schema issues, establishing data contracts, and adopting decentralized data ownership—organizations ensure the integrity of their data streams and significantly reduce risks.
Download Guidebook
Available in German, French, Spanish, and Japanese!

From Batch and Lakehouse to Real-Time Data Products with Data Streaming

“The Shift Left Architecture enables a data mesh with real-time data products to unify transactional and analytical workloads with Apache Kafka, Flink, and Iceberg. Consistent information is handled with streaming processing or ingested into Snowflake, Databricks, Google BigQuery, or any other analytics/AI platform to increase flexibility, reduce cost, and enable a data-driven company culture with faster time-to-market building innovative software applications.”
Read Blog

IoT Data Integration for Real-Time Processing with Confluent Cloud and MQTT

IoT adoption is revolutionizing industries by enabling real-time data collection to enhance efficiency and safety. A logistics organization using Confluent demonstrated the power of IoT by managing data from a fleet of 30,000 vehicles. They optimized predictive maintenance, monitored compliance, and introduced a route optimization module, saving fuel costs and driver hours.
Read Blog
Anchor: must-read

Data Streaming in Action at BMW | Hybrid and Multicloud, AI, IoT 

The data streaming strategy at BMW is a rare and inspiring example of how to do it right. Operating a complex infrastructure of SAP systems, legacy software, and third-party tools, BMW leverages a data streaming platform as the central nervous system for critical business data. Their implementation approach drives innovation by enabling access to real-time and IoT data across the organization, thereby ensuring cost reductions, enhanced operational efficiency, and accelerated time to market. 
Read the full story and many more in the new book by Kai Waehner: The Ultimate Data Streaming Guide. Download is available for free.
Download Ebook
Available in German, French, and Spanish!

Predictive Maintenance & Condition Monitoring

Kai Waehner, Global Field CTO at Confluent, explores how data streaming can enhance predictive maintenance and improve Overall Equipment Effectiveness (OEE) using real-time sensor data. He demonstrates three manufacturing use cases within the Apache Kafka® ecosystem, including building condition monitoring applications with Kafka Streams and Java, leveraging ksqlDB for stateful applications, and integrating AI and machine learning to run predictive maintenance while the data is still hot.
Watch on YouTube
Back to TopSign Up for Updates
Anchor: Developers-Desk

Developer's Desk: Building Applications

How Apache Iceberg and Flink Can Ease Developer Pain

Although developing stateful applications can be chaotic due to complex upstream and downstream interactions, tools like Apache Iceberg and Apache Flink® help simplify this process. Iceberg optimizes data querying by defining efficient table structures, while Flink enables real-time data processing, improving speed and reliability. Together, they provide a robust framework that enhances both the efficiency and reliability of stateful application development.
Watch Podcast

Shift Processing and Governance Left: A DSP Product Demo

“Shift left" is gaining traction as a trend by addressing the increasing need for high-quality business data. This approach solves common challenges like redundant datasets, poor data quality, and high costs associated with data warehouses and data lakes by cleaning and aggregating data closer to the point of data generation.
Watch Demo

Handling the Producer Request: Kafka Producer and Consumer Internals | Part 2

The Apache Kafka® producer request process involves a series of stages, including handling by network threads, queuing, data validation by I/O threads, and eventually writing to disk. Throughout this journey, configurations such as buffer sizes, thread counts, and flush intervals play key roles in optimizing performance. Monitoring metrics help track the efficiency of each step, ensuring that the data is properly processed and replicated for durability.
Read Blog

Industrial IoT Middleware for Edge and Cloud OT/IT Bridge Powered by Apache Kafka and Flink

As industries embrace digital transformation, the convergence of Operational Technology (OT) and Information Technology (IT) has become crucial to ensuring seamless operations. The OT/IT bridge, driven by real-time data synchronization, is central to unlocking the full potential of the Industrial Internet of Things (IIoT). With powerful technologies like Apache Kafka® and Flink®, businesses can integrate edge and cloud systems to optimize efficiency, drive predictive maintenance, and enhance decision-making across operations, leading to smarter, more connected industrial environments.
Read Blog

How to Add a Connector to Confluent Cloud

Kafka Connectors provide a way to get data flowing between sources and sinks, and Confluent Cloud. Choosing between available options like fully managed connectors for easy setup, or custom connectors that suit specific requirements, ultimately depends on the required level of control and specific data pipeline goals.
Watch on YouTube

Learn About Data Streaming With Apache Kafka® and Apache Flink®

High-throughput low latency distributed event streaming platform. Available locally or fully-managed via Apache Kafka on Confluent Cloud.

High-performance stream processing at any scale. Available via Confluent Cloud for Apache Flink.

Explore Developer Hub

Request Flink Workshop or Tech Talk

Back to TopSign Up for Updates
Anchor: Innovation-Research

Data Streaming for Innovation & Research

Accelerating Manufacturing with Real-Time Data Streams

Real-time data streaming in India's manufacturing sector is a key driver of innovation, transforming traditional systems into smart, connected factories. Companies like BMW are leveraging real-time data to boost efficiency, reduce downtime, and enhance processes across R&D, inventory management, and customer engagement, supporting proactive improvements and responsive manufacturing practices.
Read Article
Back to Top

Join the Community

Sign up for updates below and check out previous issues!
Issue 1.0
Issue 2.0
Issue 3.0
Issue 5.0
Sign Up for Updates
X
LinkedIn
GitHub

Share content suggestions and new uses cases in the Comments Section

Handling the Producer Request: Kafka Producer and Consumer Internals, Part 2

In this post, the second in the Kafka Producer and Consumer Internals Series, we follow our brave hero—a well-formed produce request—which is on its way to be processed by the broker and have its d...

New Paradigm to Shift Left: Why is Shift Left Important?

Shift Left is derived from the software engineering principles of Shift Left Testing where tests are performed earlier in the development lifecycle to improve software quality, accelerate time to m...

Data Governance

What is data governance, and why does it matter? Learn how data governance works, steps to real-time governance, benefits, challenges, and considerations.

The Streaming Data Quality Guidebook

Learn Stream Governance best practices to protect downstream systems and applications from poor data quality with this comprehensive guide.

Shift Left: Unifying Operations and Analytics With Data Products

ELT pipelines can lead to duplicates and high costs. A 'shift left' strategy improves this. We discuss making data available in operational, analytical, and hybrid systems with streams, tables, sch...

3 Data Engineering Trends Riding Apache Kafka®, Apache Flink®, and Apache Iceberg

The Apache Kafka, Flink, and Iceberg communities are continuously evolving, offering innovative ways for engineers to manage and process data at scale. From re-envisioning microservices with Flink ...

Confluent Recognized as a Challenger in 2024 Gartner® Magic Quadrant™ for Data Integration Tools: Ushering in a New Era of Real-Time Data Streaming

Recognized as a Challenger in Gartner’s 2024 Magic Quadrant for Data Integration Tools, Confluent predicts that the future of data integration will focus on universal data products and event-driven...

New in Confluent Cloud: Extending Private Networking Across the Data Streaming Platform

Confluent’s Q4 2024 launch wrapped up the year with a host of powerful features paving the way for even more innovation in 2025. Focusing on delivering private networking and enhanced security acro...

Predictive Analytics: How Generative AI and Data Streaming Work Together to Forecast the Future

Predictive analytics, powered by data streaming and GenAI, is transforming risk management, fraud detection, and investment strategies. By analyzing historical and real-time data with machine learn...

Shift Processing and Governance Left: A DSP Product Demo

“Shift left" is gaining traction as a trend by addressing the increasing need for high-quality business data. This approach solves common challenges like redundant datasets, poor data quality, and ...

How Apache Iceberg and Flink Can Ease Developer Pain

Although developing stateful applications can be chaotic due to complex upstream and downstream interactions, tools like Apache Iceberg and Apache Flink® help simplify this process. Iceberg optimiz...

The Ultimate Data Streaming Guide | Concepts, Use Cases, and Industry Stories for Manufacturing & Automotive

The long-awaited book by Kai Waehner, Field CTO at Confluent, is finally here! Fresh off the press, it showcases the usage of data streaming across industries, including manufacturing. He not only ...

Good Teams Manage Kafka – Efficient Teams Use Confluent

Managing open source Apache Kafka® in-house or through basic cloud-hosted services can lead to significant costs, operational challenges, and downtime risks, especially under tight engineering budg...

Arcese Group Evolves Into a Logistics Operator with a Digital Heart

Arcese achieves 50x faster data movement, real-time insights, and accurate forecasting using Confluent to power their new, custom built cloud transportation management system.

IoT Data Integration for Real-Time Processing with Confluent Cloud and MQTT

From locomotive analytics companies tracking predictive maintenance data, to digital-native companies tracking semi-trailer trucks’ movement across America’s highways, or municipalities using 100’s...

What is a Data Streaming Platform (DSP)?

Learn how a data streaming platform (DSP) enables organizations to capture, store, and process data as a continuous flow of real time events.

The Shift Left Architecture - From Batch and Lakehouse to Real-Time Data Products with Data Streaming

Blog about architectures, best practices and use cases for data streaming, analytics, hybrid cloud infrastructure, internet of things, crypto, and more

The Data Streaming Landscape 2025

“Data streaming is a new software category. It has grown from niche adoption to becoming a fundamental part of modern data architecture, leveraging open source technologies like Apache Kafka® and A...

Data as a Product

In many applications, data streams are treated as a byproduct of the system. However, the most successful businesses recognize that when data is treated as a first-class product, it can drive their...

Accelerating Manufacturing with Real-Time Data Streams

Predictive Maintenance & Condition Monitoring

Enjoy the videos and music you love, upload original content, and share it all with friends, family, and the world on YouTube.

Industrial IoT Middleware for Edge and Cloud OT/IT Bridge powered by Apache Kafka and Flink

Blog about architectures, best practices and use cases for data streaming, analytics, hybrid cloud infrastructure, internet of things, crypto, and more

How to Add a Connector to Confluent Cloud

Kafka Connectors provide a way to get data flowing between sources and sinks, and Confluent Cloud. Choosing between available options like fully managed connectors for easy setup, or custom connect...

Connecting Smart Factories and the Cloud in Real Time

Enjoy the videos and music you love, upload original content, and share it all with friends, family, and the world on YouTube.

Shift Left to Write Data Once, Read as Tables or Streams

Shift Left is a rethink of how we circulate, share and manage data in our organizations using DataStreams, Change Data Capture, FlinkSQL and Tableflow. It addresses the challenges with multi-hop a...

How Data Streaming Offers Strategic Insight for Factory-Wide Automation

Richard Jones, VP EMEA at Confluent, highlights how real-time data streaming enhances factory automation by enabling manufacturers to access data immediately as it is generated. This capability sup...

How Siemens Healthineers Leverages Data Streaming with Apache Kafka and Flink in Manufacturing and Healthcare

Blog about architectures, best practices and use cases for data streaming, analytics, hybrid cloud infrastructure, internet of things, crypto, and more

What is a Data Streaming Platform?

A Data Streaming Platform (DSP) is composed of six major inter-related components, covered in detail in this video. Each component has evolved over time to servce specific needs in the data streami...