[Linux]shell多进程并发—详细版
业务背景
schedule.sh脚本负责调度用户轨迹工程脚本的执行,截取部分代码如下:
#!/bin/bash
source /etc/profile;
export userTrackPathCollectHome=/home/pms/bigDataEngine/analysis/script/usertrack/master/pathCollect
##############################
#
# 流程A
#
##############################
# 验证机器搭配的相关商品数据源是否存在
lines=`hadoop fs -ls /user/pms/recsys/algorithm/schedule/warehouse/ruleengine/artificial/product/$yesterday | wc -l`
if [ $lines -le 0 ] ;then
echo ‘Error! artificial product is not exist‘
exit 1
else
echo ‘artificial product is ok!!!!!!‘
fi
# 验证机器搭配的相关商品数据源是否存在
lines=`hadoop fs -ls /user/pms/recsys/algorithm/schedule/warehouse/mix/artificial/product/$yesterday | wc -l`
if [ $lines -le 0 ] ;then
echo ‘Error! mix product is not exist‘
exit 1
else
echo ‘mix product is ok!!!!!!‘
fi
##############################
#
# 流程B
#
##############################
# 生成团购信息表,目前只抓取团购ID、商品ID两项
sh $userTrackPathCollectHome/scripts/extract_groupon_info.sh
lines=`hadoop fs -ls /user/hive/pms/extract_groupon_info | wc -l `
if [ $lines -le 0 ] ;then
echo ‘Error! groupon info is not exist‘
exit 4
else
echo ‘groupon info is ok!!!!!‘
fi
# 生成系列商品,总文件大小在320M左右
sh $userTrackPathCollectHome/scripts/extract_product_serial.sh
lines=`hadoop fs -ls /user/hive/pms/product_serial_id | wc -l `
if [ $lines -le 0 ] ;then
echo ‘Error! product serial is not exist‘
exit 5
else
echo ‘product serial is ok!!!!!‘
fi
# 预处理生成extract_trfc_page_kpi表--用于按照pageId进行汇总统计所在页面的pv数、uv数
sh $userTrackPathCollectHome/scripts/extract_trfc_page_kpi.sh $date
lines=`hadoop fs -ls /user/hive/pms/extract_trfc_page_kpi/ds=$date | wc -l`
if [ $lines -le 0 ] ;then
echo ‘Error! extract_trfc_page_kpi is not exist‘
exit 6
else
echo ‘extract_trfc_page_kpi is ok!!!!!!‘
fi
# 同步term_category到hive,并将前台类目转换为后台类目
sh $userTrackPathCollectHome/scripts/extract_term_category.sh
lines=`hadoop fs -ls /user/hive/pms/temp_term_category | wc -l`
if [ $lines -le 0 ] ;then
echo ‘Error! temp_term_category is not exist‘
exit 7
else
echo ‘temp_term_category is ok!!!!!!‘
fi
##############################
#
# 流程C
#
##############################
# 生成extract_track_info表
sh $userTrackPathCollectHome/scripts/extract_track_info.sh
lines=`hadoop fs -ls /user/hive/warehouse/extract_track_info | wc -l `
if [ $lines -le 0 ] ;then
echo ‘Error! extract_track_info is not exist‘
exit 1
else
echo ‘extract_track_info is ok!!!!!‘
fi
...
如上,整个预处理环节脚本执行完,需要耗时55分钟。
优化
上面的脚本执行流程可以分为三个流程:
流程A->流程B->流程C
考虑到流程B中的每个子任务都互不影响,因此没有必要顺序执行,优化的思路是将流程B中这些互不影响的子任务并行执行。
其实linux中并没有并发执行这一特定命令,上面所说的并发执行实际上是将这些子任务放到后台执行,这样就可以实现所谓的“并发执行”,脚本改造如下:
#!/bin/bash
source /etc/profile;
export userTrackPathCollectHome=/home/pms/bigDataEngine/analysis/script/usertrack/master/pathCollect
##############################
#
# 流程A
#
##############################
# 验证机器搭配的相关商品数据源是否存在
lines=`hadoop fs -ls /user/pms/recsys/algorithm/schedule/warehouse/ruleengine/artificial/product/$yesterday | wc -l`
if [ $lines -le 0 ] ;then
echo ‘Error! artificial product is not exist‘
exit 1
else
echo ‘artificial product is ok!!!!!!‘
fi
# 验证机器搭配的相关商品数据源是否存在
lines=`hadoop fs -ls /user/pms/recsys/algorithm/schedule/warehouse/mix/artificial/product/$yesterday | wc -l`
if [ $lines -le 0 ] ;then
echo ‘Error! mix product is not exist‘
exit 1
else
echo ‘mix product is ok!!!!!!‘
fi
##############################
#
# 流程B
#
##############################
# 并发进程,生成团购信息表,目前只抓取团购ID、商品ID两项
{
sh $userTrackPathCollectHome/scripts/extract_groupon_info.sh
lines=`hadoop fs -ls /user/hive/pms/extract_groupon_info | wc -l `
if [ $lines -le 0 ] ;then
echo ‘Error! groupon info is not exist‘
exit 4
else
echo ‘groupon info is ok!!!!!‘
fi
}&
# 并发进程,生成系列商品,总文件大小在320M左右
{
sh $userTrackPathCollectHome/scripts/extract_product_serial.sh
lines=`hadoop fs -ls /user/hive/pms/product_serial_id | wc -l `
if [ $lines -le 0 ] ;then
echo ‘Error! product serial is not exist‘
exit 5
else
echo ‘product serial is ok!!!!!‘
fi
}&
# 并发进程,预处理生成extract_trfc_page_kpi表--用于按照pageId进行汇总统计所在页面的pv数、uv数
{
sh $userTrackPathCollectHome/scripts/extract_trfc_page_kpi.sh $date
lines=`hadoop fs -ls /user/hive/pms/extract_trfc_page_kpi/ds=$date | wc -l`
if [ $lines -le 0 ] ;then
echo ‘Error! extract_trfc_page_kpi is not exist‘
exit 6
else
echo ‘extract_trfc_page_kpi is ok!!!!!!‘
fi
}&
# 并发进程,同步term_category到hive,并将前台类目转换为后台类目
{
sh $userTrackPathCollectHome/scripts/extract_term_category.sh
lines=`hadoop fs -ls /user/hive/pms/temp_term_category | wc -l`
if [ $lines -le 0 ] ;then
echo ‘Error! temp_term_category is not exist‘
exit 7
else
echo ‘temp_term_category is ok!!!!!!‘
fi
}&
##############################
#
# 流程C
#
##############################
# 等待上面所有的后台进程执行结束
wait
echo ‘end of backend jobs above!!!!!!!!!!!!!!!!!!!!!!!!!!!!‘
# 生成extract_track_info表
sh $userTrackPathCollectHome/scripts/extract_track_info.sh
lines=`hadoop fs -ls /user/hive/warehouse/extract_track_info | wc -l `
if [ $lines -le 0 ] ;then
echo ‘Error! extract_track_info is not exist‘
exit 1
else
echo ‘extract_track_info is ok!!!!!‘
fi
上面的脚本中,将流程B中互不影响的子任务全部放到了后台执行,从而实现了“并发执行”,同时为了不破坏脚本的执行流程:
流程A->流程B->流程C
就需要在流程C执行之前加上:
# 等待上面所有的后台进程执行结束
wait
其目的是等待流程B的所有后台进程全部执行完成,才执行流程C
结论
经过优化后,脚本的执行时间,从耗时55分钟,降到了耗时15分钟,效果很显著。
郑重声明:本站内容如果来自互联网及其他传播媒体,其版权均属原媒体及文章作者所有。转载目的在于传递更多信息及用于网络分享,并不代表本站赞同其观点和对其真实性负责,也不构成任何其他建议。