Warm tip: This article is reproduced from serverfault.com, please click
amazon-s3 amazon-web-services aws-glue json amazon-redshift-spectrum

How to query an array field (AWS Glue)?

发布于 2020-11-20 09:39:23

I have a table in AWS Glue, and the crawler has defined one field as array. The content is in S3 files that have a json format. The table is TableA, and the field is members.

There are a lot of other fields such as strings, booleans, doubles, and even structs.

I am able to query them all using a simpel query such as:

SELECT
    content.my_boolean,
    content.my_string,
    content.my_struct.value
FROM schema.tableA;

The issue is when I add content.members into the query. The error I get is: [Amazon](500310) Invalid operation: schema "content" does not exist.

Content exists because i am able to select other fiels from the main key in the json (content). Probably is something related with how to perform the query agains array field in Spectrum.

Any idea?

Questioner
Maik
Viewed
1
分享
Hyruma92 2020-11-20 23:33:56

You have to rename the table to extract the fields from the external schema:

SELECT
    a.content.my_boolean,
    a.content.my_string,
    a.content.my_struct.value
FROM schema.tableA a;

I had the same issue on my data, I really don't know why it needs this cast but it works. If you need to access elements of an array you have to explod it like:

SELECT member.<your-field>,
FROM schema.tableA a, a.content.members as member;

Reference

Maik 2020-11-22 11:45:07

Thanks @Hyruma92, working like a charm! Great solution :)

热门帖子

1
Android 开发方向：传统 View 开发 or 拥抱 Jetpack Compose
2
浙江 ISP 会屏蔽 Wireguard UDP 端口
3
高级 Java 岗位一定需要有管理经验吗
4
兄弟们，老是倒在大厂二面的怎么办？
5
最近很多小伙伴弄港卡，分享一下个人开卡经历
6
vercel 免费版 3 个指标都超了，好像我的网站已经获得了基础流量。还好套了 cloudflare 不然流量也超了。
7
最近换了真我手机，发现这系统连应用的数据都无法备份
8
MacBook Pro 决赛圈
9
连下厨房也要放弃网页版了
10
杭州岗位太少了

热门github

1
A multi-platform library for OpenGL, OpenGL ES, Vulkan, window and input
2
Dev tool that writes scalable apps from scratch while the developer oversees the implementation
3
shadcn/ui, but for Svelte. ✨
4
The Python Risk Identification Tool for generative AI (PyRIT) is an open access automation framework to empower security professionals and machine learning engineers to proactively find risks in their generative AI systems.
5
Performance-portable, length-agnostic SIMD with runtime dispatch
6
ZK Credo
7
OpenCodeInterpreter: Integrating Code Generation with Execution and Refinement
8
Joplin - the secure note taking and to-do app with synchronisation capabilities for Windows, macOS, Linux, Android and iOS.
9
Mamba is a new state space model architecture showing promising performance on information-dense data such as language modeling, where previous subquadratic models fall short of Transformers. It is based on the line of progress on structured state space models, with an efficient hardware-aware design and implementation in the spirit of FlashAttention.
10
This repository contains System Design resources which are useful while preparing for interviews and learning Distributed Systems
11
Curso para aprender el lenguaje de programación Python desde cero y para principiantes. 75 clases, 37 horas en vídeo, código, proyectos y grupo de chat. Fundamentos, frontend, backend, testing, IA...
12
🎓 Path to a free self-taught education in Computer Science!
13
1️⃣🐝🏎️ The One Billion Row Challenge -- A fun exploration of how quickly 1B rows from a text file can be aggregated with Java
14
A collective list of free APIs
15
📚 Freely available programming books