DATABRICKS DBFS

I have experience with DBFS, it is a great storage which is holding data which you can upload from your local computer using DBFS CLI! The CLI setup a bit tricky, but when you manage, you can easily move whole folders around in this environment (remember using -overwrite! )

1. create folders
2. upload files
3. modify, remove files and folders

With Scala you can easily pull in the data you store in this storage with a code like this:

val df1 = spark
      .read
      .format("csv")
      .option("header", "true")
      .option("inferSchema", "true")
      .load("dbfs:/foldername/test.csv")
      .select(some_column_name)

Or read in the whole folder to process all csv the files available:

val df1 = spark
      .read
      .format("csv")
      .option("header", "true")
      .option("inferSchema", "true")
      .load("dbfs:/foldername/*.csv")
      .select(some_column_name)

I think it is easy to use and learn, I hope you find this info helpful!