Scala spark-shell mode
1 Motivation
Spark-shell is a good place to develop and test spark / spark sql code interactively. As a data scientist / engineer, I wish I have a editor support for that. I hope the editor would looks like RStudio / emacs ESS / ipython notebook. Not able to find an available emacs binding, here I made a slightly change to scala-mode (from https://github.com/scala/scala-tool-support) to support interactive spark script development with spark-shell.
2 Features
2.1 scala-mode for spark-shell
- Features from scala-mode like keyword highlight etc
- Interactive spark script development, with ess / R-mode like key-binding
2.2 literature programming support with org-mode
- Can run code block and print result to org document as an org-babel plugin
3 Install / configuration
Requirement:
- emacs 24.5 and org-mode version >= 8.2.10
- it is conflict with other scala-mode, probably need to remove them, sadly
3.1 Basic support
You can git clone this repo, and put the following code to your .emacs to load them
(progn
(cd "<<path-to-scala-spark-shell-mode-elisp-dir>>")
(normal-top-level-add-subdirs-to-load-path))
(require 'scala-mode)3.1.1 Modify spark-shell launch command
You can modify the spark-shell to some other thing to load your own jar and add your own start configuration like –num-executors, etc by redefine scala-interpreter
(defcustom scala-interpreter "spark-shell"
"The interpreter that `run-scala' should run. This should
be a program in your PATH or the full pathname of the scala interpreter."
:type 'string
:group 'scala-mode-inf)In a file buffer with .scala or .spark suffix, M-x scala-run-scala will open an spark-shell buffer.
3.1.2 Send code and eval them in the spark-shell buffer
- Line / block / region / buffer evaluation.
- The key-binding is ess / R-mode flavor
(progn (define-key scala-mode-map "\C-c\C-i" 'scala-run-scala)
(define-key scala-mode-map "\C-c\C-r" 'scala-eval-region)
(define-key scala-mode-map "\C-c\C-j" 'scala-eval-line)
(define-key scala-mode-map "\C-c\C-c" 'scala-eval-block)
(define-key scala-mode-map "\C-c\C-b" 'scala-eval-buffer)
(define-key scala-mode-map "\C-c\C-d" 'scala-eval-definition)
(define-key scala-mode-map "\C-c\C-z" 'scala-switch-to-interpreter))3.2 Literature programming setup
Add the spark-shell language support to your org-babel-load-languages list, like this:
(org-babel-do-load-languages
'org-babel-load-languages
'((R . t)
(spark-shell . t)
(python . t)
(org . t))
)Here goes an spark-shell literature programming example.