Spark Interview Questions for Professionals

Contents

1.      What is the difference between spark and Hadoop?. 1

2.      What are the differences between functional and imperative languages, and why is functional programming important?. 2

3.      What is a resilient distributed dataset (RDD), Explain showing diagrams? ​ ​ 3

4.      Explain transformations and actions (in the context of RDDs). 4

5.      What are the Spark use cases?. 4

6.      Why do we need transformations? What is lazy evaluation and why is it useful?. 4

7.      What is ParallelCollectionRDD?. 4

8.      Can you use Spark to access and analyze data stored in Cassandra databases?. 4

9.      Explain how ReduceByKey and GroupByKey works?. 5

10.        What is the common workflow of a Spark program?. 5

11.        Explain spark environment for driver. Ref. 6

12.        What are the transformations and actions that you have used in Spark?. 6

13.        How can you minimize data transfers when working with Spark?. 8

14.        What is a lineage graph?. 8

15.        Describe the major libraries that constitute the Spark Ecosystem.. 9

16.        What are the different file formats that can be used in SparkSql?. 9

17.        What are Pair RDDs?. 9

18.        What is the difference between persist() and cache(). 9

19.        What are the various levels of persistence in Apache Spark? Ref. 10

20.        Which Storage Level to Choose? Ref. 10

21.        Explain advantages and drawbacks of RRD.. 11

22.        Explain why dataset is preferred over RDDs?. 11

23.        How to share data from Spark RDD between two applications?. 11

24.        Does Apache Spark provide check pointing?. 12

25.        Explain the internal working of caching?. 12

26.        What is the function of Block manager?. 12

27.        Why Spark SQL considers the support of indexes unimportant?. 12

28.        How to convert existing UDTFs in Hive to Scala functions and use it from Spark SQL? Explain with example Ref. 12

29.        Why use dataframes and datasets when we have RDD? Ref Video. 13

30.        What is a Catalyst and how does it work? Ref. 14

31.        What are the top challenges developers faces while writing Spark applications? – Ref Video. 17

32.        How is memory handled in Datasets?. 20

33.        Explain Datasets and Serialization. 20

34.        What are the limitations of dataset?. 20

35.        Implementation difference between DataFrames and DataSet?. 20

36.        What are the contentions with memory?. 21

37.        Show Command to run spark in YARN client mode?. 22

38.        Show Command to run spark in YARN cluster mode?. 22

39.        What is Standalone and YARN mode?. 22

40.        Explain client mode and cluster mode in spark?. 22

41.        Which cluster managers are supported by spark?. 23

42.        What is Executor memory?. 23

43.        What is DStream and What is the difference between batch and Dstream in Spark streaming?. 23

44.        How does Spark Streaming work?. 23

45.        Difference between map() and flatMap()?. 24

46.        What is reduce() action, Is there any difference between reduce() and reduceByKey()?. 25

47.        What is the disadvantage of Reduce() action and how can we overcome this limitation?. 25

48.        What are Accumulators and When are accumulators truly reliable?. 25

49.        What is Broadcast Variables and what is the advantage?. 26

50.        What is Piping, show an example who to create data pipeline?. 26

51.        What is driver?. 28

52.        What are the steps that occur when you run a Spark application on a cluster?. 28

53.        What is a schema RDD/DataFrame?. 29

54.        What are Row objects?. 29

55.        How does Spark achieve fault tolerance?. 29

56.        What does a Spark Engine do?. 29

57.        What parameter is set if cores need to be defined across executors?. 29

58.        Name few Spark Master system properties?. 29

59.        Define Partitions in reference to spark implementation?. 30

60.        Differences between how Spark and MapReduce manage cluster resources under YARN. Ref. 30

61.        What is Hive on Spark?. 34

62.        What is GraphX and what is PageRank? Ref. 34

63.        What does MLlib do? Ref. 40

64.        What is a Parquet file?. 45

65.        Why Parquet is used for Spark SQL? Ref. 45

66.        What is schema evolution and what is its disadvantage, explain schema merging in reference to parquet file? Ref. 48

67.        What file systems does Spark support?. 51

68.        Will spark replace MapReduce?. 51

69.        What is Spark Executor?. 51

70.        Name the different types of Cluster Managers in Spark. 51

71.        How many ways we can create RDD, show example?. 51

72.        How do you flatten rows in Spark? Explain with example. Ref. 52

73.        Explain Spark Streaming Architecture?. 52

74.        What are the types of Transformations on DStreams?. 52

75.        What is Receiver in Spark Streaming, can you build custom receivers?. 53

76.        Explain the process of Live streaming and how to store DStream data to database? Ref. 53

78.        How spark streaming is fault tolerant?. 56

79.        Explain transform() method used in dSteam? Ref. 57

80.        How data security is achieved in Spark?. 58

81.        Explain Kerberos security? Ref. 61

82.        Name various types of distributing does spark support?. 62

83.        Consider the following SQL query: Select pName, sum(rate) from sales where (year = 2015) group by pName; Explain the functioning in Map and Reduce when the above query is executed in spark using dataSets. 62

84.        What are the conditions where spark driver can parallelize dataSets as RDDs?. 62

85.        Can repartition() operation decrease number of partition?. 62

86.        What is the drawback of repartition() and coalesce() operations?. 62

87.        In a join operaton for example val joinVal = rddA.join(rddB) will it generate partition?. 62

88.        Consider the following code in spark, what is the final value in fVal variable?. 62

89.        Scala pattern matching - Show various ways code can be written?. 62

90.        What is the return result when query is executed using Spark SQL or HIVE? Hint: RDD or dataframe/dataset?. 63

91.        If we want to display just the schema of a dataframe/dataset what method is called?. 63

92.        Show various implementations for the following query in spark?. 63

93.        What are the most important factors you want to consider when you start machine learning project?  63

94.        As a data scientist which algorithm will you suggest if legal aspects and ease of explanation to non technical people are the main criteria?. 64

95.        For supervised learning algorithm, what percentage of data is split between training and test dataset?  64

96.        Compare performance of Avro and parquet file formats, explain case while using with spark. 64

97.        Spark Master exposes a set of REST API’s to submit, monitor applications which data format is used for these web services?. 64

98.        When you should not use spark?. 64

99.        What attributes contribute for a function to achieve parallelism?. 64

1. What is the difference between spark and Hadoop?

Features	SPARK	Hadoop
Inspiration	Hadoop Map-Reduce and Scala programming language, developed by UC-Berkeley's AMPLab in 2009, use Generalized computation instead of MapReduce Real time processing capability	Google, papers in 2004 outlining MapReduce Batch Processing
Speed	100X in-memory and 10X on Disk	Heavy Disk read I/O intensive
Easy of Use	Easily write application using Java, Scala, Python, R (Functional programming style) Interactive Shell available with Scala and Python High level simple map-reduce Operations	Java – Imperative programming style No shell complex map-reduce operations
Iterative Workflow	Great at Iterative workloads(Machine learning ..etc)	Not ideal for iterative work
Tools	Well integrated tools (Spark SQL, Streaming,  Mlib and GraphX) to develop complex analytical application	Loosely coupled large set of tools, but matured
Deployment	Hadoop YARN, Mesos,  Amazon-EC2	Usually use Oozie and Azkaban to create workflow
Data Source	HDFS(Hadoop), HBase, Cassandra, MongoDB, Amazon-S3, RDBMS, file, socket, tweeter	RDBMS (using sqoop), streaming using FLUME
Applications	Spark ‘Application’ is higher level of Unit,  runs multiple jobs in sequence or parallel Application process  are called executors, runs on clusters(workers)	Hadoop ‘job’ is higher level unit  process data with map reduce and writes data to  storage
Executors	Executors can run multiple tasks in a single processor	Each mapReduce runs in  its own processor
Shared Variable	Broadcast variables: Read-only(look-up) variable, ships only once to worker Accumulators: Workers add values and driver reads the data, and fault tolerant	Hadoop counter have additional (system ) metric counters like  ‘Map input records’
Persisting/Caching RDD	Cached RDDs  can be used & reused in across the operation, thus increase the processing speed	None
Lazy Evaluation	Transformation functions execution plan bundled together  and executes only with RDD action function	None
Memory Management and Compression	Memory is conserved, because of the compact format. Speed is improved by custom code-generation.	Custom compression can be using AVRO, Kyro, no memory management
Optimizer and Query Planning	Optimizer is a Rule Executor for logical plans. It uses a collection of logical plan optimizations. Generate encoders via runtime code-generation. The generated code can operate directly on the Tungsten compact format. Query is optimized – logical and physical plan (inspired by RDBMS query planning and optimization)	None

2. What are the differences between functional and imperative languages, and why is functional programming important? 

Following features of Scala makes it uniquely suitable for spark.

Immutability - Immutable means that you can't change your variables; you mark them as final in Java, or use the val keyword in Scala

Higher order functions - These are functions that take other functions as parameters, or whose result is a function. Here is a function apply which takes another function f and a value v and applies function f to v: example - def apply(f: Int => String, v: Int) = f(v)

Lazy loading - Lazy val is executed when it is accessed the first time else no execution.

Pattern matching - Scala has a built-in general pattern matching mechanism. It allows to match on any sort of data with a first-match policy

Currying - If we turn this into a function object that we can assign or pass around, the signature of that function looks like this: val sizeConstraintFn: IntPairPred => Int => Email => Boolean = sizeConstraint _ Such a chain of one-parameter functions is called a curried function

Partial application - When applying the function, you do not pass in arguments for all of the parameters defined by the function, but only for some of them, leaving the remaining ones blank. What you get back is a new function whose parameter list only contains those parameters from the original function that were left blank.

Monads - Most Scala collections are monadic, and operating on them using map and flatMap operations, or using for-comprehensions is referred to as monadic-style.

Programming approach difference:

Characteristic	Imperative approach	Functional approach
Programmer focus	How to perform tasks (algorithms) and how to track changes in state.	What information is desired and what transformations are required.
State changes	Important.	Non-existent.
Order of execution	Important.	Low importance.
Primary flow control	Loops, conditionals, and function (method) calls.	Function calls, including recursion.
Primary manipulation unit	Instances of structures or classes.	Functions as first-class objects and data collections.

3. What is a resilient distributed dataset (RDD), Explain showing diagrams? ​ ​

Resilient distributed dataset (RDD) is a read-only and fault-tolerant collection of objects partitioned across cluster of computers that can be operated on in parallel. There are two ways to create RDDs: parallelizing an existing collection in your driver program, or referencing a dataset in an external storage system, such as a shared filesystem, HDFS, HBase, S3, Cassandra or RDBMS.

RDDs (Resilient Distributed Datasets) are basic abstraction in Apache Spark that represent the data coming into the system in object format. RDDs are used for in-memory computations on large clusters, in a fault tolerant manner. RDDs are read-only portioned, collection of records, that are –

• Immutable – RDDs cannot be altered.
• Resilient – If a node holding the partition fails the other node takes the data.
• Lazy evaluated
• Cacheable
• Type inferred

Ref

4. Explain transformations and actions (in the context of RDDs)

Transformations are functions executed on demand, to produce a new RDD. All transformations are followed by actions. Some examples of transformations include map, filter and reduceByKey.

ReduceByKey merges the values for each key using an associative and commutative reduce function. This will also perform the merging locally on each mapper before sending results to a reducer, similarly to a "combiner" in MapReduce.

Actions are the results of RDD computations or transformations. After an action is performed, the data from RDD moves back to the local machine. Some examples of actions include reduce, collect, first, and take.

5. What are the Spark use cases?

• Data integration and ETL
• Interactive analytics or business intelligence
• High performance batch computation
• Machine learning and advanced analytics
• Real-time stream processing

Lots of people are doing data integration and ETL on MapReduce, as well as batch computation, machine learning and batch analytics. But these things are going to be much faster on Spark. Interactive analytics and BI are possible on Spark, and the same goes for real-time stream processing.

Need all the answers See my kindle book.