Warm tip: This article is reproduced from serverfault.com, please click
apache-spark pyspark

Combining Spark schema without duplicates using PySpark?

发布于 2020-11-27 21:01:06

unable to combine the following three to a single final schema without duplicate columns.
Here is the code below:

schema1 = StructType([StructField("A", StringType(), True),
                        StructField("B", StringType(), True)])
schema2 = StructType([StructField("c", StringType(), True),
                        StructField("B", StringType(), True)])
schema3 = StructType([StructField("D", StringType(), True),
                        StructField("A", StringType(), True)])
final=(schema1 ++ schema2 ++ schema3).distinct
print( final)
Questioner
john
Viewed
0
分享
mck 2020-11-28 15:13:50 
schema1 = StructType([StructField("A", StringType(), True),
                        StructField("B", StringType(), True)])
schema2 = StructType([StructField("c", StringType(), True),
                        StructField("B", StringType(), True)])
schema3 = StructType([StructField("D", StringType(), True),
                        StructField("A", StringType(), True)])
final = StructType(list(set(schema1.fields+schema2.fields+schema3.fields)))
print(final)

gives

StructType(List(StructField(B,StringType,true),StructField(D,StringType,true),StructField(c,StringType,true),StructField(A,StringType,true)))

热门帖子

1
难道 Go 就没有好用的工作审批流框架吗
2
跟风贴自家软路由实现
3
Mac 上有什么 pdf 阅读器比较好用？
4
再开一个贴，问界发布新 M5，请各位老哥或者老车主骂醒我
5
我这辈子是不是彻底和 Airpods Pro 无缘了
6
最近对很多事情都没有感觉了
7
微信读书非付费会员有每月导入数量限制了
8
不知道从什么时候，普通 USB-C 双头线不能给 iPhone15 充电了
9
程序员想开发漂亮的个人网站是不是用 react 会比 vue 简单一些？
10
有关于上海居住证 120 积分

热门github

1
Implementation of paper - YOLOv9: Learning What You Want to Learn Using Programmable Gradient Information
2
A Windows and Office activator using HWID / Ohook / KMS38 / Online KMS activation methods, with a focus on open-source code and fewer antivirus detections.
3
Get up and running with Llama 2, Mistral, Gemma, and other large language models.
4
该项目可以让你通过订阅的方式使用Cloudflare WARP+，自动获取流量。This project enables you to use Cloudflare WARP+ through subscription, automatically acquiring traffic.
5
Multi functional app to find duplicates, empty folders, similar images etc.
6
Xray panel supporting multi-protocol multi-user expire day & traffic & ip limit (Vmess & Vless & Trojan & ShadowSocks & Wireguard)
7
The Free Software Media System
8
lightweight, standalone C++ inference engine for Google's Gemma models.
9
📚 Freely available programming books
10
A collective list of free APIs
11
1️⃣🐝🏎️ The One Billion Row Challenge -- A fun exploration of how quickly 1B rows from a text file can be aggregated with Java
12
🎓 Path to a free self-taught education in Computer Science!
13
Curso para aprender el lenguaje de programación Python desde cero y para principiantes. 75 clases, 37 horas en vídeo, código, proyectos y grupo de chat. Fundamentos, frontend, backend, testing, IA...
14
This repository contains System Design resources which are useful while preparing for interviews and learning Distributed Systems
15
Mamba is a new state space model architecture showing promising performance on information-dense data such as language modeling, where previous subquadratic models fall short of Transformers. It is based on the line of progress on structured state space models, with an efficient hardware-aware design and implementation in the spirit of FlashAttention.