raymondchen-byte / spark-video

Processing videos on Apache Spark

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

spark-video

For User

Local Development

Just copy and paste the python code from bin/local_install.py to the pyspark REPL.

Production: Databricks Runtime

Put these three lines in the init script:

sudo wget -O /databricks/jars/ffmpeg-4.4-1.5.6-linux-x86_64.jar https://repo1.maven.org/maven2/org/bytedeco/ffmpeg/4.4-1.5.6/ffmpeg-4.4-1.5.6-linux-x86_64.jar
sudo wget -O /databricks/jars/javacpp-1.5.6-linux-x86_64.jar https://repo1.maven.org/maven2/org/bytedeco/javacpp/1.5.6/javacpp-1.5.6-linux-x86_64.jar
sudo wget -O /databricks/jars/spark-video-assembly-0.0.4.jar https://github.com/eto-ai/spark-video/releases/download/v0.0.4/spark-video-assembly_2.12-0.0.4.jar

For Developer

Cheatsheet

bin/mill 'video[2.12].test'
bin/mill 'video[2.12].test.testOnly' '**.MLImageTest.scala'
bin/mill 'video[2.12].publishLocal'
bin/mill 'video[2.12].assembly'

bin/mill mill.scalalib.scalafmt.ScalafmtModule/checkFormatAll __.sources
bin/mill mill.scalalib.scalafmt.ScalafmtModule/reformatAll __.sources

About

Processing videos on Apache Spark


Languages

Language:Scala 94.9%Language:Shell 3.3%Language:Python 1.8%