Warm tip: This article is reproduced from serverfault.com, please click
dataframe dictionary numpy pandas python

Python: Memory efficient, quick lookup in python for 100 million pairs of data?

发布于 2020-11-28 01:06:15

This is my first time asking a question on here, so apologies if I am doing something wrong.

I am looking to create some sort of dataframe/dict/list where I can check if the ID in one column has seen a specific value in another column before.

For example for one pandas dataframe like this (90 million rows):

ID  Another_ID
1   10
1   20
2   50
3   10
3   20
4   30

And another like this(10 million rows):

ID  Another_ID
1   30
2   30
2   50
2   20
4   30
5   70

I want to end up with a third column that is like this:

ID  Another_ID seen_before
1   30         0
2   30         0
2   50         1
2   20         0
4   30         1
5   20         0

I am looking for a memory efficient but quick way to do this, any ideas? Thanks!

Questioner
BrendanA
Viewed
0
分享
Quang Hoang 2020-11-28 11:42:34

Merge is a good idea, here, you want to merge on both columns:

df1['seen_before'] = 1

df2.merge(df1, on=['ID', 'Another_ID'], how='left')

Output:

   ID  Another_ID  seen_before
0   1          30          NaN
1   2          30          NaN
2   2          50          1.0
3   2          20          NaN
4   4          30          1.0
5   5          70          NaN

Note: this assumes that df1 has no duplicates. If you are not sure about this, replace df1 with df1.drop_duplicates() in merge.

热门帖子

1
推荐一些好玩的/大众的手游
2
求指教后端项目迁移方案
3
迷你洗衣机是不是都是智商税？
4
求助一个排查了半年没解决的 MySQL order by 子句导致索引失效的问题， 500 多万条记录的小表要查快两分钟
5
个人开发了一款 WordPress 主题： iPao，集成了 AI 总结功能
6
偶然发现奇游加速器会在系统里植入根证书
7
国内有蒲公英替代品推荐吗？
8
语音助手这个东西真的会监听谈话并且上传，从而泄漏隐私吗？
9
出一些有意思的域名-明盘
10
jetbrains 全家桶升级 2024 后，在滚动代码时候感觉有点掉帧

热门github

1
A multi-platform library for OpenGL, OpenGL ES, Vulkan, window and input
2
Dev tool that writes scalable apps from scratch while the developer oversees the implementation
3
shadcn/ui, but for Svelte. ✨
4
The Python Risk Identification Tool for generative AI (PyRIT) is an open access automation framework to empower security professionals and machine learning engineers to proactively find risks in their generative AI systems.
5
Performance-portable, length-agnostic SIMD with runtime dispatch
6
ZK Credo
7
OpenCodeInterpreter: Integrating Code Generation with Execution and Refinement
8
Joplin - the secure note taking and to-do app with synchronisation capabilities for Windows, macOS, Linux, Android and iOS.
9
Mamba is a new state space model architecture showing promising performance on information-dense data such as language modeling, where previous subquadratic models fall short of Transformers. It is based on the line of progress on structured state space models, with an efficient hardware-aware design and implementation in the spirit of FlashAttention.
10
This repository contains System Design resources which are useful while preparing for interviews and learning Distributed Systems
11
Curso para aprender el lenguaje de programación Python desde cero y para principiantes. 75 clases, 37 horas en vídeo, código, proyectos y grupo de chat. Fundamentos, frontend, backend, testing, IA...
12
🎓 Path to a free self-taught education in Computer Science!
13
1️⃣🐝🏎️ The One Billion Row Challenge -- A fun exploration of how quickly 1B rows from a text file can be aggregated with Java
14
A collective list of free APIs
15
📚 Freely available programming books