That’s where the evolution began and, currently, there are many solutions in place and the most widely used one being Apache Flume.įlume is an extremely reliable service, which is used to efficiently collect, aggregate and store large amounts of data. Hence, the need arose for creating a standard on how all the data streams could be directed into the HDFS more efficiently. For example, system or application logs, geolocation data and monitoring, as well as social media trends and updates - all constitute fast-moving streams requesting for storage in the Hadoop Distributed File System (HDFS). Once the data gets into the Hadoop framework, you have won half the battle but the real struggle begins when new, fast-moving data streams starts flowing in. Challenges with data load in the Hadoop framework