spark技术分享Spark and software in-memory file systems
It appears that there are a few open source projects that can boost performance of any in-memory shared state, akin to file system, including RDDs – Tachyon, Apache Ignite, and Apache Geode.
Tachyon is designed to function as a software file system that is compatible with the HDFS interface prevalent in the big data analytics space. The point of doing this is that it can be used as a drop in accelerator rather than having to adapt each framework to use a distributed caching layer explicitly.
From Spark Shared RDDs:
Apache Ignite provides an implementation of Spark RDD abstraction which allows to easily share state in memory across multiple Spark jobs, either within the same application or between different Spark applications.
There’s another similar open source project Apache Geode.
