Warm tip: This article is reproduced from serverfault.com, please click
apache-spark hadoop hdfs

Spark iterate HDFS directory

发布于 2014-11-19 18:01:18

I have a directory of directories on HDFS, and I want to iterate over the directories. Is there any easy way to do this with Spark using the SparkContext object?

Questioner
Jon
Viewed
0
分享
27.7k 2019-09-26 10:00:52

You can use org.apache.hadoop.fs.FileSystem. Specifically, FileSystem.listFiles([path], true)

And with Spark...

FileSystem.get(sc.hadoopConfiguration).listFiles(..., true)

Edit

It's worth noting that good practice is to get the FileSystem that is associated with the Path's scheme.

path.getFileSystem(sc.hadoopConfiguration).listFiles(path, true)
JimLohse 2016-02-23 13:46:11

really nice! I had this question, granted, I guess this wouldn't work in the original spark-submit call

horatio1701d 2018-01-27 13:58:31

How can I create a list of the files using the RemoteIterator this creates?

热门帖子

1
B 站必火推广好假啊
2
求推荐 3k 左右的烘干机
3
[疑难杂症] 安卓面具求助，拯救楼主的 3000 软 QAQ
4
老板问我为什么不加班，我要怎么回他
5
RouterOS 如何切换 DNS 服务器
6
[开发者自荐] 又做了一个‘无用’的小玩具，根据出生日期，受孕月份预测性别， Just for Fun！
7
胖猫，你又不是第一次没人要
8
求助一个排查了半年没解决的 MySQL order by 子句导致索引失效的问题， 500 多万条记录的小表要查快两分钟
9
Surge ponte 如何实现异地访问家里的内网设备
10
[TestFlight] 月更：竹蜻蜓 0.17.4 现已释出

热门github

1
Implementation of paper - YOLOv9: Learning What You Want to Learn Using Programmable Gradient Information
2
A Windows and Office activator using HWID / Ohook / KMS38 / Online KMS activation methods, with a focus on open-source code and fewer antivirus detections.
3
Get up and running with Llama 2, Mistral, Gemma, and other large language models.
4
该项目可以让你通过订阅的方式使用Cloudflare WARP+，自动获取流量。This project enables you to use Cloudflare WARP+ through subscription, automatically acquiring traffic.
5
Multi functional app to find duplicates, empty folders, similar images etc.
6
Xray panel supporting multi-protocol multi-user expire day & traffic & ip limit (Vmess & Vless & Trojan & ShadowSocks & Wireguard)
7
The Free Software Media System
8
lightweight, standalone C++ inference engine for Google's Gemma models.
9
📚 Freely available programming books
10
A collective list of free APIs
11
1️⃣🐝🏎️ The One Billion Row Challenge -- A fun exploration of how quickly 1B rows from a text file can be aggregated with Java
12
🎓 Path to a free self-taught education in Computer Science!
13
Curso para aprender el lenguaje de programación Python desde cero y para principiantes. 75 clases, 37 horas en vídeo, código, proyectos y grupo de chat. Fundamentos, frontend, backend, testing, IA...
14
This repository contains System Design resources which are useful while preparing for interviews and learning Distributed Systems
15
Mamba is a new state space model architecture showing promising performance on information-dense data such as language modeling, where previous subquadratic models fall short of Transformers. It is based on the line of progress on structured state space models, with an efficient hardware-aware design and implementation in the spirit of FlashAttention.