Warm tip: This article is reproduced from serverfault.com, please click
apache-spark pandas pyspark user-defined-functions

Rewrite UDF to pandas udf with ArrayType column

发布于 2020-11-22 23:14:38

I am trying to rewrite an UDF into pandas UDF.

However when it comes to the column with ArrayType inside. I am struggling to find the right solution.

I have a dataframe as below:

+-----------+--------------------+
|      genre|                 ids|
+-----------+--------------------+
|      Crime|[6, 22, 42, 47, 5...|
|    Romance|[3, 7, 11, 15, 17...|
|   Thriller|[6, 10, 16, 18, 2...|
|  Adventure|[2, 8, 10, 15, 29...|
|   Children|[1, 2, 8, 13, 34,...|
|      Drama|[4, 11, 14, 16, 1...|
|        War|[41, 110, 151, 15...|
|Documentary|[37, 77, 99, 108,...|
|    Fantasy|[2, 56, 60, 126, ...|
|    Mystery|[59, 113, 123, 16...|
+-----------+--------------------+

The following UDF works well:

pairs_udf = udf(lambda x: itertools.combinations(x, 2), transformer.schema)
df = df.select("genre", pairs_udf("ids").alias("ids"))

The output is like:

+-----------+--------------------+
|      genre|                 ids|
+-----------+--------------------+
|      Crime|[[6, 22], [6, 42]...|
|    Romance|[[3, 7], [3, 11],...|
|   Thriller|[[6, 10], [6, 16]...|
|  Adventure|[[2, 8], [2, 10],...|
|   Children|[[1, 2], [1, 8], ...|
|      Drama|[[4, 11], [4, 14]...|
|        War|[[41, 110], [41, ...|
|Documentary|[[37, 77], [37, 9...|
|    Fantasy|[[2, 56], [2, 60]...|
|    Mystery|[[59, 113], [59, ...|
+-----------+--------------------+

However, what would be the equivalent when writing the function in pandas udf.

PS: I understand, alternatively, I can use cross-join to achieve the same results.

But, I am more curious about how do pandas udf handle column with ArrayType.

Questioner
D.J
Viewed
11
分享
D.J 2020-11-28 11:22:28

I am going to share my findings here:

there are 3 aspects in order to make pandas udf work for your project:

1. pandas UDF, or more precisely, Apache Arrow does not support complex types as common udf dose.(as of pyspark 3.0.1, pyarrow 2.0.0)

e.g.:

2. if you are running Java 11, which is the default in (py)Spark 3. you need to add the following as part of your spark config:

spark.driver.extraJavaOptions='-Dio.netty.tryReflectionSetAccessible=true'
spark.executor.extraJavaOptions='-Dio.netty.tryReflectionSetAccessible=true'

this will solve the java.lang.UnsupportedOperationException mentioned above.

3. make sure your virtual environment python path is added to your pyspark_python

i.e. environ['PYSPARK_PYTHON']='./your/virutal/enviroment/path'

热门帖子

1
难道 Go 就没有好用的工作审批流框架吗
2
跟风贴自家软路由实现
3
Mac 上有什么 pdf 阅读器比较好用？
4
再开一个贴，问界发布新 M5，请各位老哥或者老车主骂醒我
5
我这辈子是不是彻底和 Airpods Pro 无缘了
6
最近对很多事情都没有感觉了
7
微信读书非付费会员有每月导入数量限制了
8
不知道从什么时候，普通 USB-C 双头线不能给 iPhone15 充电了
9
程序员想开发漂亮的个人网站是不是用 react 会比 vue 简单一些？
10
有关于上海居住证 120 积分

热门github

1
Implementation of paper - YOLOv9: Learning What You Want to Learn Using Programmable Gradient Information
2
A Windows and Office activator using HWID / Ohook / KMS38 / Online KMS activation methods, with a focus on open-source code and fewer antivirus detections.
3
Get up and running with Llama 2, Mistral, Gemma, and other large language models.
4
该项目可以让你通过订阅的方式使用Cloudflare WARP+，自动获取流量。This project enables you to use Cloudflare WARP+ through subscription, automatically acquiring traffic.
5
Multi functional app to find duplicates, empty folders, similar images etc.
6
Xray panel supporting multi-protocol multi-user expire day & traffic & ip limit (Vmess & Vless & Trojan & ShadowSocks & Wireguard)
7
The Free Software Media System
8
lightweight, standalone C++ inference engine for Google's Gemma models.
9
📚 Freely available programming books
10
A collective list of free APIs
11
1️⃣🐝🏎️ The One Billion Row Challenge -- A fun exploration of how quickly 1B rows from a text file can be aggregated with Java
12
🎓 Path to a free self-taught education in Computer Science!
13
Curso para aprender el lenguaje de programación Python desde cero y para principiantes. 75 clases, 37 horas en vídeo, código, proyectos y grupo de chat. Fundamentos, frontend, backend, testing, IA...
14
This repository contains System Design resources which are useful while preparing for interviews and learning Distributed Systems
15
Mamba is a new state space model architecture showing promising performance on information-dense data such as language modeling, where previous subquadratic models fall short of Transformers. It is based on the line of progress on structured state space models, with an efficient hardware-aware design and implementation in the spirit of FlashAttention.