31.
Please
tell me , how execution starts and end on RDD or Spark Job
Ans: Execution Plan starts with the
earliest RDDs (those with no dependencies on other RDDs or reference cached
data) and ends with the RDD that produces the result of the action that has
been called to execute.
32.
Give
example of transformations that do trigger jobs
Ans: There are a couple of
transformations that do trigger jobs, e.g. sortBy , zipWithIndex , etc.
33.
How
many type of transformations exist?
Ans: There are two kinds of
transformations:
·
narrow transformations
·
wide transformations
34.
What
is Narrow Transformations?
Ans: Narrow transformations are the
result of map, filter and such that is from the data from a single partition
only, i.e. it is self-sustained.
An output RDD
has partitions with records that originate from a single partition in the
parent RDD. Only a limited subset of partitions used to calculate the result.
Spark groups narrow transformations as a stage.
;)
35.
What
is wide Transformations?
Ans: Wide transformations are the
result of groupByKey and reduceByKey . The data required to compute the records
in a single partition may reside in many partitions of the parent RDD.
All of the
tuples with the same key must end up in the same partition, processed by the
same task. To satisfy these operations, Spark must execute RDD shuffle, which
transfers data across cluster and results in a new stage with a new set of partitions.
(54)
36.
Data
is spread in all the nodes of cluster, how spark tries to process this data?
Ans: By default, Spark tries to read
data into an RDD from the nodes that are close to it. Since Spark usually
accesses distributed partitioned data, to optimize transformation operations it
creates partitions to hold the data chunks
37.
How
would you hint, minimum number of partitions while transformation ?
Ans: You can request for the minimum
number of partitions, using the second input parameter to many transformations.
scala>
sc.parallelize(1 to 100, 2).count
Preferred way to
set up the number of partitions for an RDD is to directly pass it as the second
input parameter in the call like rdd = sc.textFile("hdfs://…
/file.txt", 400) , where400 is the number of partitions. In this case, the
partitioning makes for 400 splits that would be done by the Hadoop’s
TextInputFormat , not Spark and it would work much faster. It’salso that the
code spawns 400 concurrent tasks to try to load file.txt directly into 400 partitions.
38.
How
many concurrent task Spark can run for an RDD partition?
Ans: Spark can only run 1 concurrent
task for every partition of an RDD, up to the number of cores in your cluster.
So if you have a cluster with 50 cores, you want your RDDs to at least have 50
partitions (and probably 2-3x times that).
As far as
choosing a "good" number of partitions, you generally want at least
as many as the number of executors for parallelism. You can get this computed
value by calling
sc.defaultParallelism
.
39.
Which
limits the maximum size of a partition?
Ans: The maximum size of a partition is
ultimately limited by the available memory of an executor.
40.
When
Spark works with file.txt.gz, how many partitions can be created?
Ans: When using textFile with
compressed files ( file.txt.gz not file.txt or similar), Spark disables
splitting that makes for an RDD with only 1 partition (as reads against gzipped
files cannot be parallelized). In this case, to change the number of partitions
you should do repartitioning.
Please note that
Spark disables splitting for compressed files and creates RDDs with only 1
partition. In such cases, it’s helpful to use sc.textFile('demo.gz') and do
repartitioning using rdd.repartition(100) as follows:
rdd = sc.textFile('demo.gz')
rdd = rdd.repartition(100)
With the lines,
you end up with rdd to be exactly 100 partitions of roughly equal in size.
Click Below to visit other products as well for Hadoop
