From the course: Data Management with Apache NiFi

Unlock this course with a free trial

Join today to access over 23,400 courses taught by industry experts.

Configuring back pressure

Configuring back pressure - Apache NiFi Tutorial

From the course: Data Management with Apache NiFi

Start my 1-month free trial Buy for my team

Configuring back pressure

“

- [Instructor] In this demo, we'll work with the same NiFi data flow that we used in the previous demo where we ReadLaptopDetails from our local machine and push out files after performing a filtering operation to an S3 bucket. We'll use this data flow to understand how queuing and back pressure works in data flow. Every connection that you set up between processors is used to transmit flow files, and each connection can be configured with a certain threshold. This threshold determines the capacity of the queue that exists at this connection. Let's take a look at the configuration of this first queue here. You can see in Settings that the Object Threshold has been set to 90, and the Size Threshold is 20 MB. If the number of flow files that this queue contains reaches 90, or if the size of all of the flow files combined that this queue holds reaches 20 MB, that means the back pressure threshold has been hit. In such…

Contents