spark技术分享About Mesos
Apache Mesos is an Apache Software Foundation open source cluster management and scheduling framework. It abstracts CPU, memory, storage, and other compute resources away from machines (physical or virtual).
Mesos provides API for resource management and scheduling across multiple nodes (in datacenter and cloud environments).
|
Tip
|
Visit Apache Mesos to learn more about the project. |
Mesos is a distributed system kernel with a pool of resources.
“If a service fails, kill and replace it”.
An Apache Mesos cluster consists of three major components: masters, agents, and frameworks.
Concepts
A Mesos master manages agents. It is responsible for tracking, pooling and distributing agents’ resources, managing active applications, and task delegation.
A Mesos agent is the worker with resources to execute tasks.
A Mesos framework is an application running on a Apache Mesos cluster. It runs on agents as tasks.
The Mesos master offers resources to frameworks that can accept or reject them based on specific constraints.
A resource offer is an offer with CPU cores, memory, ports, disk.
Frameworks: Chronos, Marathon, Spark, HDFS, YARN (Myriad), Jenkins, Cassandra.
-
Mesos API
-
Mesos is a scheduler of schedulers
-
Mesos assigns jobs
-
Mesos typically runs with an agent on every virtual machine or bare metal server under management (https://www.joyent.com/blog/mesos-by-the-pound)
-
Mesos uses Zookeeper for master election and discovery. Apache Auroa is a scheduler that runs on Mesos.
-
Mesos slaves, masters, schedulers, executors, tasks
-
Mesos makes use of event-driven message passing.
-
Mesos is written in C++, not Java, and includes support for Docker along with other frameworks. Mesos, then, is the core of the Mesos Data Center Operating System, or DCOS, as it was coined by Mesosphere.
-
This Operating System includes other handy components such as Marathon and Chronos. Marathon provides cluster-wide “init” capabilities for application in containers like Docker or cgroups. This allows one to programmatically automate the launching of large cluster-based applications. Chronos acts as a Mesos API for longer-running batch type jobs while the core Mesos SDK provides an entry point for other applications like Hadoop and Spark.
-
The true goal is a full shared, generic and reusable on demand distributed architecture.
-
Infinity to package and integrate the deployment of clusters
-
Out of the box it will include Cassandra, Kafka, Spark, and Akka.
-
an early access project
-
-
Apache Myriad = Integrate YARN with Mesos
-
making the execution of YARN work on Mesos scheduled systems transparent, multi-tenant, and smoothly managed
-
to allow Mesos to centrally schedule YARN work via a Mesos based framework, including a REST API for scaling up or down
-
includes a Mesos executor for launching the node manager
-
