Welcome to OGeek Q&A Community for programmer and developer-Open, Learning and Share
Welcome To Ask or Share your Answers For Others

Categories

0 votes
770 views
in Technique[技术] by (71.8m points)

mongodb - Hadoop Map/Reduce vs built-In Map/Reduce

What are the key differences to do map/reduce work on MongoDB using Hadoop map/reduce vs built-in map/reduce of Mongo?

When do I pick which map/reduce engine? what are the pros and cons of each engine to work on data stored in mongodb?

See Question&Answers more detail:os

与恶龙缠斗过久,自身亦成为恶龙;凝视深渊过久,深渊将回以凝视…
Welcome To Ask or Share your Answers For Others

1 Reply

0 votes
by (71.8m points)

My answer is based on knowledge and experience of Hadoop MR and learning of Mongo DB MR. Lets see what are major differences and then try to define criteria for selection: Differences are:

  1. Hadoop's MR can be written in Java, while MongoDB's is in JavaScript.
  2. Hadoop's MR is capable of utilizing all cores, while MongoDB's is single threaded.
  3. Hadoop MR will not be collocated with the data, while Mongo DB's will be collocated.
  4. Hadoop MR has millions of engine/hours and can cope with many corner cases with massive size of output, data skews, etc
  5. There are higher level frameworks like Pig, Hive, Cascading built on top of the Hadoop MR engine.
  6. Hadoop MR is mainstream and a lot of community support is available.

From the above I can suggest the following criteria for selection:
Select Mongo DB MR if you need simple group by and filtering, do not expect heavy shuffling between map and reduce. In other words - something simple.

Select hadoop MR if you're going to do complicated, computationally intense MR jobs (for example some regressions calculations). Having a lot or unpredictable size of data between map and reduce also suggests Hadoop MR.

Java is a stronger language with more libraries, especially statistical. That should be taken into account.


与恶龙缠斗过久,自身亦成为恶龙;凝视深渊过久,深渊将回以凝视…
OGeek|极客中国-欢迎来到极客的世界,一个免费开放的程序员编程交流平台!开放,进步,分享!让技术改变生活,让极客改变未来! Welcome to OGeek Q&A Community for programmer and developer-Open, Learning and Share
Click Here to Ask a Question

...