Unlocking Real-Time Insights: The Power of Data Streaming in Action
Welcome to the second issue of Follow The Stream! In this edition, we continue our journey into how organizations harness the power of real-time data to foster innovation. We’re excited to present Confluent's first-ever verified use cases, showcasing transformative applications of data streaming within the manufacturing sector. Let’s explore how to turn data into a strategic asset!
The modern world is defined by speed. Grocery delivery, rideshare apps, and payments for just about anything can happen instantly using a mobile device and its apps. Every action of every consumer creates data, and businesses must make sense of it quickly to take advantage in real time. Data streaming platforms like Confluent use real-time pipelines to collect, store, and use data continuously using event-driven principles. Each change in the data, known as an event, triggers new data to be received and integrated into the processing function.
Explore the basics of Apache Kafka®, Apache Flink®, and DSP.
Heading 1
Subtitle 1
Heading 2
Subtitle 2
Heading 3
Subtitle 3
Data in Motion is the foundation for everything at Confluent. Check out the links below to get an understanding of how real-time data processing powers business productivity and innovation:
Subtitle 1
Subtitle 2
Subtitle 3
Anchor: General-Updates
Industry in Motion: General Updates
Data Streaming ROI & Investments Soar Higher
As businesses learn to do more with less, proving tech investment ROI has become even more critical. This ensures they are achieving value, despite limited resources. 84% of IT leaders cite 2x to 10x return on data streaming investments as it continues to drive significant benefits across a wide range of business functions and industries. 37% of IT leaders in the manufacturing industry see a 5x or more ROI on data streaming investments.
Data streaming fundamentally changed the way modern companies do business
“Confluent CEO and co-creator of Apache Kafka®, Jay Kreps, shares his insights on the transformative power of data streaming for modern enterprises. As businesses strive to harness real-time data for better decision-making and competitive advantage, Kreps offers a deep dive into the current state of data streaming adoption, the challenges organizations face, and the pivotal role data streaming plays in enabling cutting-edge applications like real-time AI.”
"Stream processing has existed for decades. However, it really kicks off in the 2020s thanks to the adoption of open source frameworks like Apache Kafka and Flink. Fully managed cloud services make it easy to configure and deploy stream processing in a cloud-native way; even without the need to write any code". Confluent’s Field CTO and data streaming expert Kai Waehner explores the past, present and future of stream processing, "including the relation of machine learning and GenAI, streaming databases, and the integration between data streaming and data lakes with Apache Iceberg".
Currect 2024 Recap: Convening With Data Streaming Engineers
This year’s Current may be over, but all the mind-blowing work the data streaming community shared in Austin lives on. At Current 2024, we explored the evolution of data streaming, emphasizing the critical role of data streaming engineers amid a significant paradigm shift in data management, the transition to AI-powered applications necessitating a broader definition of roles in AI/ML engineering, and discussed technologies like Kafka and Apache Flink®, highlighting insights into future challenges and innovations.
Confluent Named a Leader in two IDC MarketScape Reports
The IDC MarketScape vendor analysis model is designed to provide an overview of the competitive fitness of ICT suppliers in a given market. In 2024, Confluent was recognized as a Leader in the IDC MarketScape for Worldwide Analytic Stream Processing Software 2024 Vendor Assessment and the IDC MarketScape for Worldwide Event Brokering Software 2024 Vendor Assessment. The first assessment highlights stream governance as well as partner program and multicloud support as Confluent’s competitive strengths, while the second recognizes Confluent for strong analytical and operational capabilities and out-of-the-box monitoring and security capabilities.
The Data in Motion Tour 2024 is Stopping by Your City!
Join data streaming experts for an opportunity to network with peers and ecosystem partners, hear directly from customers, learn from Confluent's team of Apache Kafka® experts, and roll up your sleeves in interactive demonstrations and hands-on labs.
Upcoming dates: October 9th: Vienna - RSVPOctober 15th: Delhi-RSVPOctober 15th: London - RSVPOctober 15th: Madrid - RSVPOctober 22nd: Riyadh - RSVPOctober 23rd: Johannesburg - RSVPOctober 29th: Zürich - RSVPNovember 6th: Frankfurt - RSVPNovember 13th: Milan - RSVPNovember 19th: Stockholm - RSVPNovember 21st: Copenhagen - RSVPNovember 28th: Dubai - RSVP
Anchor: Executives-Brief
Executive's Brief: Data Strategy & Leadership
How a Truly Cloud-Native Kafka Solution Can Modernize Architecture
Confluent has invested 5M+ engineering hours in completely re-architecting Apache Kafka® to harness the full power of the cloud and support all mission-critical enterprise workloads with 10x faster scaling, infinite storage, and built-in resiliency. There are three attributes that set Confluent apart from Kafka and other solutions for data streaming:
Confluent is cloud-native, providing a 10x better, fully managed service to free you from operational burdens
Confluent is complete with all the essential tools for developers to build quickly, reliably, and securely
Confluent exists everywhere to connect all data in real time, from on-prem to cloud and across clouds
Confluent: Reusable Data Streams Are Rising C-Level IT Agendas
Executives are increasingly prioritizing reusable data streams due to their benefits in agility, cost-efficiency, as well as cleaner code and greener data practices. Platforms like Confluent Cloud for Apache Flink® support the creation of these high-quality streams, helping organizations to streamline operations and improve decision-making.
Moving Up the Curve: 5 Tips for Enabling Enterprise-Wide Data Streaming
The Confluent maturity curve outlines the five stages of Apache Kafka® adoption. It highlights how deeply data streaming is integrated within a business, which can help identify opportunities and challenges. Here are some tips for moving to the next stage of the maturity curve:
Show what a level 4 & 5 data streaming platform looks like
Make the business case (specifically for level 4+)
Show how to implement enterprise-wide data streaming with a Target Operating Model
List all stakeholders involved
Confirm when to implement this—specifically at level 3
How an EV Manufacturer Drives Operational Efficiency With Real-Time Streaming
As electric vehicle manufacturers face complex integration challenges across diverse systems, real-time data visibility becomes crucial for optimizing production and meeting demand. An EV manufacturer turned to Allata to implement a data mesh using Confluent Cloud, enabling seamless integration across key systems and enhancing operational efficiency. This approach transformed the company's data management by providing real-time insights and reducing overhead, setting the stage for future improvements in data-driven decision-making and customer experiences.
Why Global Manufacturing Companies Rely on Data Streaming?
Learn about data streaming applications within the manufacturing industry directly from Confluent’s Global Field CTO Kai Waehner. In this overview video, he walks through how to get end-to-end visibility into OEM supply chain by leveraging data streaming for OT and IT data and how it helps to modernize and innovate the entire engineering and sales lifecycle. All of the data is managed in one visibility infrastructure where you can take action on the data, like enabling predictive maintenance or better customer service.
Michelin’s Evolution to Cloud-Native Microservices and Data Streaming
Michelin modernized its global supply chain by transitioning to real-time inventory management using Confluent Cloud and Apache Kafka®. By replacing unreliable batch reporting with a data streaming platform, Michelin improved visibility across its entire supply chain, reducing downtime and enabling faster, data-driven decisions that optimized logistics and cut operational costs.
“Industry experts Jeff Winter and Lora Cecere, founder of Supply Chain Insights, discuss current trends and the rise of enterprise technology and AI in today’s supply chain industry.”
How FourKites Built Real-Time Visibility and AI for the Global Supply Chain
In 2023, FourKites, the platform behind the world’s largest supply chain data network, launched MyWorkspace – a centralized intelligence solution that enables customers to create tailored dashboards for interconnected decision-making across enterprise and supply chain data. Arpit Garg, Senior Principal Architect, and Goutham Elangesh, Staff Data Engineer at FourKites, explain how the engineering team implemented MyWorkspace using Confluent Cloud Kafka and Rockset.
Stream SAP Data Anywhere It's Needed with Confluent Cloud
Harnessing SAP data in real time is the key to reinventing customer experiences, building efficiencies in back-end operations, and launching new business models and services. Ultimately, it is creating a more responsive business that keeps pace with the modern, digital-first world. To achieve this, more businesses are tapping into data streaming to keep all systems that rely on SAP data in sync with high-value, real-time data.
Apache Kafka in Manufacturing at Automotive Supplier Brose for Industrial IoT Use Cases
Data streaming unifies OT/IT workloads by connecting information from sensors, PLCs, robotics and other manufacturing systems at the edge with business applications and the big data analytics world in the cloud. Kai Waehner, Global Field CTO at Confluent, explores how the global automotive supplier Brose deploys a hybrid industrial IoT architecture using Apache Kafka in combination with Eclipse Kura, OPC-UA, MuleSoft and SAP.
Analyze Factory Floor Telemetry Data for Anomalies in Real Time
Modern automated manufacturing factory floors can have hundreds of robots performing various tasks, including picking and placing objects, assembling parts, packaging, welding, painting, and cleaning. By effectively utilizing telemetry data and leveraging the power of real-time streaming analytics platforms like Confluent Cloud’s Apache Kafka® clusters and Apache Flink® SQL, valuable insights into robot health and performance can be gained.
Manufacturing 4.0 Connected Vehicles, Mobility Services, and Smart City Real-Time Processing and Analytics powered by Apache Kafka and MQTT
Apache Kafka® and MQTT are a perfect combination for many IoT use cases. Kai Waehner, Global Field CTO at Confluent, looks at the pros and cons of both technologies, and how they are complementary. Various real-world case studies across industries will be discussed, including connected vehicles, manufacturing, mobility services, and smart city. The examples use different architectures and best practices for lightweight edge scenarios, hybrid integrations, or serverless cloud solutions.
Data streaming with Apache Kafka® and Apache Flink® is a cutting-edge technology reshaping the digital landscape, garnering significant attention worldwide. Kai Waehner, global field CTO at Confluent, explores the latest developments in data sharing, data contracts, serverless stream processing, multicloud solutions, and AI integrations, providing a comprehensive overview of what's next in this dynamic field.
The Data Streaming Revolution: The Force of Kafka + Flink Awakens
Explore data streaming and Apache Flink® in the second issue of Confluent's Comic Book. Follow Ada, Jax, and the data architecture team as they navigate the challenges of stream processing to prepare a shoe retail titan for the biggest athletic event of the year. Join them to explore how Flink® and Kafka can be leveraged in tandem. After all, some things are just better together.
How Producers Work: Kafka Producer and Consumer Internals
Apache Kafka is a powerful event streaming platform that efficiently stores and delivers event data with minimal effort from developers, functioning like a black box. However, when issues arise with Kafka, the challenging task of debugging requires a deeper understanding of the inner workings of Kafka, to see how we actually interact with the black box through producers and consumers.
Uniting the Machine Learning and Data Streaming Ecosystems
Uniting machine learning (ML) and data streaming requires overcoming both technical and socio-technical barriers. Organizations need to adopt decentralized, domain-driven approaches like data mesh and feature-oriented teams. Utilizing Apache Kafka® as the backbone for real-time data integration can operationalize ML by providing the speed and scalability required for real-time analytics.
Let Flink Cook: Mastering Real-Time Retrieval-Augmented Generation (RAG) with Flink
Large language models are advancing rapidly, but to move from prototypes to production, they need real-time, domain-specific data and strong security. RAG addresses this by integrating live data streams with AI models for more accurate and relevant responses. Apache Kafka and Apache Flink can be used to implement RAG for various applications, enhancing the performance and reliability of generative AI systems.
3 Best Practices for Building Cloud-Native Systems with Kafka
Building a complete, enterprise-ready data streaming platform yourself with Apache Kafka® is a complex undertaking. Although this task only becomes more challenging—and more resource-intensive—when done in the cloud, organizations that go further and build a cloud-native data streaming platform can design an architecture that unlocks business value, speeds innovation, and reduces total cost of ownership (TCO).
Smart warehouses are transforming operations through advanced technologies like IoT, robotics, and data streaming. Real-time data streaming, in particular, enhances efficiency and accuracy by enabling seamless inventory management, predictive maintenance, and automation, ultimately meeting the high demands of customer expectations.
Modernizing SCADA Systems and OT/IT Integration with Data Streaming
“SCADA control systems are a vital component of IT/OT modernization. The old IT/OT infrastructure and SCADA system are monolithic, proprietary, not scalable, and miss open APIs based on standard interfaces.” Kai Waehner, global field CTO at Confluent, explains the modernization of such a system based on the real-life use case of 50Hertz, a transmission system operator for electricity in Germany.
Data-Driven Manufacturing: Dialogue with Dr. Ralf M. Wagner
Dr. Ralf M. Wagner, senior vice president of Insights Hub, "discusses the role of data-driven manufacturing in manufacturing, the challenges and advancements in industrial IoT solutions, and generative AI technologies".
