RichardWarburton / java-8-lambdas-exercises

Exercises and Answers for Java 8 Lambdas book

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

oom caused by parallel()

xueding-e opened this issue · comments

this issues occurs when i calculate array size,
this array about three minion arays,i hava 2G heap.this file is only 40m,and if i use no parallel,this
program only uses 50m, Ask,what's the cause

static void useStream(){
    long uniqueWord = 0;
    try (Stream<String> lines =
                 Files.lines(Paths.get("D:\\nlp\\dictionary\\CoreNatureDictionary.ngram.txt"), Charset.defaultCharset())){
        long start = System.currentTimeMillis();
        uniqueWord = lines.flatMap(line -> Arrays.stream(line.split(" "))).parallel().distinct().count();

// uniqueWord = lines.flatMap(line -> Arrays.stream(line.split(" "))).distinct().count();
System.out.println(uniqueWord);
long end = System.currentTimeMillis();

        System.out.println( (end-start)+"ms");
    }catch (IOException e){
        e.printStackTrace();
    }
}

this is a reduced to 10m file

CoreNatureDictionary.ngram.txt