查看云形成文档,我看不到一种方法来启动Glue DevEndpoint,Jupyter笔记本并让笔记本使用新创建的DevEndpoint。
有人可以帮忙吗?
我找到了解决方案,关键是使用云形成对象AWS :: SageMaker :: NotebookInstanceLifecycleConfig挂接到笔记本的OnStart和OnCreate笔记本事件。
在控制台的Glue部分中创建的笔记本可以在SageMaker中找到,您可以在其中看到关联的LifecycleConfig资源及其代码。
有关此问题的完整性,请参见下面从Glue创建Jupyter笔记本时在OnStart和OnCreate上使用的代码。
请注意,通过使用此方法,新创建的笔记本具有与通过控制台创建的笔记本完全相同的功能,但仅在控制台的SageMaker部分中可见。
#!/bin/bash
set -ex
[ -e /home/ec2-user/glue_ready ] && exit 0
mkdir -p /home/ec2-user/glue
cd /home/ec2-user/glue
# Write dev endpoint in a file which will be used by daemon scripts
glue_endpoint_file="/home/ec2-user/glue/glue_endpoint.txt"
if [ -f $glue_endpoint_file ] ; then
rm $glue_endpoint_file
fi
echo "https://glue.eu-west-2.amazonaws.com" >> $glue_endpoint_file
ASSETS=s3://aws-glue-jes-prod-eu-west-2-assets/sagemaker/assets/
aws s3 cp ${ASSETS} . --recursive
bash "/home/ec2-user/glue/Miniconda2-4.5.12-Linux-x86_64.sh" -b -u -p "/home/ec2-user/glue/miniconda"
source "/home/ec2-user/glue/miniconda/bin/activate"
tar -xf autossh-1.4e.tgz
cd autossh-1.4e
./configure
make
sudo make install
sudo cp /home/ec2-user/glue/autossh.conf /etc/init/
mkdir -p /home/ec2-user/.sparkmagic
cp /home/ec2-user/glue/config.json /home/ec2-user/.sparkmagic/config.json
mkdir -p /home/ec2-user/SageMaker/Glue\ Examples
mv /home/ec2-user/glue/notebook-samples/* /home/ec2-user/SageMaker/Glue\ Examples/
# ensure SageMaker notebook has permission for the dev endpoint
aws glue get-dev-endpoint --endpoint-name somiron-dfe-poc-GlueDevEndpoint --endpoint https://glue.eu-west-2.amazonaws.com
# Run daemons as cron jobs and use flock make sure that daemons are started only iff stopped
(crontab -l; echo "* * * * * /usr/bin/flock -n /tmp/lifecycle-config-v2-dev-endpoint-daemon.lock /usr/bin/sudo /bin/sh /home/ec2-user/glue/lifecycle-config-v2-dev-endpoint-daemon.sh") | crontab -
(crontab -l; echo "* * * * * /usr/bin/flock -n /tmp/lifecycle-config-reconnect-dev-endpoint-daemon.lock /usr/bin/sudo /bin/sh /home/ec2-user/glue/lifecycle-config-reconnect-dev-endpoint-daemon.sh") | crontab -
source "/home/ec2-user/glue/miniconda/bin/deactivate"
rm -rf "/home/ec2-user/glue/Miniconda2-4.5.12-Linux-x86_64.sh"
sudo touch /home/ec2-user/glue_ready