Warm tip: This article is reproduced from serverfault.com, please click
apache-spark c distributed-computing

How to run C algorithm on Spark cluster?

发布于 2020-11-26 20:28:04

I have an algorithm that is written in C.

I want to use Spark to run it on a cluster of nodes. How can I run my C executable on Spark? I have read about JNI and Java Process to execute my binary.

Questioner
Manolis P.
Viewed
0
分享
act_coder 2020-11-28 12:17:21

Here is a nice article from databricks on how we can run c and c++ codes in apache spark.

https://kb.databricks.com/_static/notebooks/scala/run-c-plus-plus-scala.html

After moving the c/c++ codes to the underlying distributed file system, You can compile the code and run on spark worker nodes.

Manolis P. 2020-11-28 11:43:54

Thanks you man! Now i read about Spark because i am new on the Spark ecosystem. When i finish, i will try this. Your provided article seems to be what i want. If you find similar articles please post it. Thank you again!

act_coder 2020-11-28 12:22:23

Sure.. if that clarifies your question - could you please mark it as answer ?

Manolis P. 2020-11-28 12:48:07

Sure, when i try it and if it works.

热门帖子

1
难道 Go 就没有好用的工作审批流框架吗
2
跟风贴自家软路由实现
3
Mac 上有什么 pdf 阅读器比较好用？
4
再开一个贴，问界发布新 M5，请各位老哥或者老车主骂醒我
5
我这辈子是不是彻底和 Airpods Pro 无缘了
6
最近对很多事情都没有感觉了
7
微信读书非付费会员有每月导入数量限制了
8
不知道从什么时候，普通 USB-C 双头线不能给 iPhone15 充电了
9
程序员想开发漂亮的个人网站是不是用 react 会比 vue 简单一些？
10
有关于上海居住证 120 积分

热门github

1
Implementation of paper - YOLOv9: Learning What You Want to Learn Using Programmable Gradient Information
2
A Windows and Office activator using HWID / Ohook / KMS38 / Online KMS activation methods, with a focus on open-source code and fewer antivirus detections.
3
Get up and running with Llama 2, Mistral, Gemma, and other large language models.
4
该项目可以让你通过订阅的方式使用Cloudflare WARP+，自动获取流量。This project enables you to use Cloudflare WARP+ through subscription, automatically acquiring traffic.
5
Multi functional app to find duplicates, empty folders, similar images etc.
6
Xray panel supporting multi-protocol multi-user expire day & traffic & ip limit (Vmess & Vless & Trojan & ShadowSocks & Wireguard)
7
The Free Software Media System
8
lightweight, standalone C++ inference engine for Google's Gemma models.
9
📚 Freely available programming books
10
A collective list of free APIs
11
1️⃣🐝🏎️ The One Billion Row Challenge -- A fun exploration of how quickly 1B rows from a text file can be aggregated with Java
12
🎓 Path to a free self-taught education in Computer Science!
13
Curso para aprender el lenguaje de programación Python desde cero y para principiantes. 75 clases, 37 horas en vídeo, código, proyectos y grupo de chat. Fundamentos, frontend, backend, testing, IA...
14
This repository contains System Design resources which are useful while preparing for interviews and learning Distributed Systems
15
Mamba is a new state space model architecture showing promising performance on information-dense data such as language modeling, where previous subquadratic models fall short of Transformers. It is based on the line of progress on structured state space models, with an efficient hardware-aware design and implementation in the spirit of FlashAttention.