pkeropen / flink-parcel

flink 集成CDH5的自定义paracels

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

rotatelogfileswithprefix: command not found

jinhshi opened this issue · comments

commented

你好,感谢贡献。
另外,README 中 (2) 将上述生成的jar文件copy至cloudera /parcel-repo子目录下, 这个应该是放在cloudera /csd下吧。

除了上边那个,其余按你的步骤来,是可以将flink集成到cdh中。
但是在启动的时候遇到问题:
rotatelogfileswithprefix: command not found

ERROR Redaction rules file doesn't exist, not redacting logs. file: redaction-rules.json, directory: /run/cloudera-scm-agent/process/1321-flink_on_yarn-FLINK_YARN

查资料看到很多人说是 flink-shaded-hadoop包的问题,尝试了几个版本:flink-shaded-hadoop-2-uber-2.7.5-10.0.jar, flink-shaded-hadoop-2-uber-3.0.0-cdh6.3.0-7.0.jar, flink-shaded-hadoop-3-uber-3.1.1.7.2.9.0-173-9.0.jar 还是相同的问题,请问有没有相关的解决方法 ?

cdh:6.3.2
flink:1.13.1
maven:3.6.3
java:1.8

flink-yarn.sh 这个脚本有问题,没这个方法

/opt/cloudera/parcels/FLINK/lib/flink/bin/config.sh:391
加入

rotateLogFilesWithPrefix() {
    dir=$1
    prefix=$2
    while read -r log ; do
        rotateLogFile "$log"
    # find distinct set of log file names, ignoring the rotation number (trailing dot and digit)
    done < <(find "$dir" ! -type d -path "${prefix}*" | sed s/\.[0-9][0-9]*$// | sort | uniq)
}

# 旋转日志文件
rotateLogFile() {
    log=$1;
    num=$MAX_LOG_FILE_NUMBER
    if [ -f "$log" -a "$num" -gt 0 ]; then
        while [ $num -gt 1 ]; do
            prev=`expr $num - 1`
            [ -f "$log.$prev" ] && mv "$log.$prev" "$log.$num"
            num=$prev
        done
        mv "$log" "$log.$num";
    fi
}

最后出现 rotateLogFilesWithPrefix 这个函数的版本是 1.11.6,后面的版本就把这个函数删了,这个库应该是不支持 1.12.0 以及之后的版本的。

https://github.com/apache/flink/blob/release-1.11.6/flink-dist/src/main/flink-bin/bin/config.sh#L367

# Auxilliary functions for log file rotation
rotateLogFilesWithPrefix() {
    dir=$1
    prefix=$2
    while read -r log ; do
        rotateLogFile "$log"
    # find distinct set of log file names, ignoring the rotation number (trailing dot and digit)
    done < <(find "$dir" ! -type d -path "${prefix}*" | sed s/\.[0-9][0-9]*$// | sort | uniq)
}

rotateLogFile() {
    log=$1;
    num=$MAX_LOG_FILE_NUMBER
    if [ -f "$log" -a "$num" -gt 0 ]; then
        while [ $num -gt 1 ]; do
            prev=`expr $num - 1`
            [ -f "$log.$prev" ] && mv "$log.$prev" "$log.$num"
            num=$prev
        done
        mv "$log" "$log.$num";
    fi
}

所以应该咋解决呢?

上面不是写了解决方案了吗