Test error after successfully training with ./train.sh -v -v

Question

Test error after successfully training with ./train.sh -v -v

j-min opened this issue 8 years ago · comments

I downloaded your git repo as work directory and successfully trained the model with UD_English with ./train.sh -v -v command.

root@19a0b2aad139:~/models/syntaxnet# git clone https://github.com/j-min/syntaxnet_easy_sejong work
Cloning into 'work'...
remote: Counting objects: 631, done.
remote: Compressing objects: 100% (7/7), done.
remote: Total 631 (delta 1), reused 0 (delta 0), pack-reused 624
Receiving objects: 100% (631/631), 109.51 MiB | 6.67 MiB/s, done.
Resolving deltas: 100% (375/375), done.
Checking connectivity... done.
root@19a0b2aad139:~/models/syntaxnet# cd work
root@19a0b2aad139:~/models/syntaxnet/work# vim train.sh
root@19a0b2aad139:~/models/syntaxnet/work# ./train.sh -v -v
+ '[' 0 '!=' 0 ']'
++++ readlink -f ./train.sh
+++ dirname /root/models/syntaxnet/work/train.sh
++ readlink -f /root/models/syntaxnet/work
+ CDIR=/root/models/syntaxnet/work
++++ readlink -f ./train.sh
+++ dirname /root/models/syntaxnet/work/train.sh
++ readlink -f /root/models/syntaxnet/work/..
+ PDIR=/root/models/syntaxnet
+ make_calmness
+ exec
+ exec
+ cd /root/models/syntaxnet
+ python=/usr/bin/python
+ SYNTAXNET_HOME=/root/models/syntaxnet
+ BINDIR=/root/models/syntaxnet/bazel-bin/syntaxnet
+ CORPUS_DIR=/root/models/syntaxnet/work/UD_English
+ CONTEXT=/root/models/syntaxnet/work/UD_English/context.pbtxt
+ TMP_DIR=/root/models/syntaxnet/work/UD_English/tmp/syntaxnet-output
+ MODEL_DIR=/root/models/syntaxnet/work/models

However, when I try to evaluate the newly trained the model with
"please parse this." | ./test.sh
It ouputs error messages as below.

root@19a0b2aad139:~/models/syntaxnet/work# "please parse this." | ./test.sh
bash: please parse this.: command not found
F syntaxnet/term_frequency_map.cc:62] Check failed: ::tensorflow::Status::OK() == (tensorflow::Env::Default()->NewRandomAccessFile(filename, &file)) (OK vs. Not found: label-map)
F syntaxnet/term_frequency_map.cc:62] Check failed: ::tensorflow::Status::OK() == (tensorflow::Env::Default()->NewRandomAccessFile(filename, &file)) (OK vs. Not found: label-map)

I trained the model twice but it gave me the same error message.

Do you have any idea related to
F syntaxnet/term_frequency_map.cc:62] Check failed: ::tensorflow::Status::OK() == (tensorflow::Env::Default()->NewRandomAccessFile(filename, &file)) (OK vs. Not found: label-map)

Myungchul Shin · Answer 1 · Tue Sep 06 2016 11:08:11 GMT+0800 (China Standard Time)

@j-min
you may used old version of syntaxnet which does not serve --resource_dir=${MODEL_DIR}.
in this case, you can use context.pbtxt.template.
https://github.com/dsindex/syntaxnet/blob/master/models/context.pbtxt.template

replace OUTPATH to models directory path where all resource files were saved.
and copy models/context.pbtxt.template to models/context.pbtxt

input {
  name: "tag-map"
  Part {
    file_pattern: "OUTPATH/tag-map" --> replace OUTPATH
  }
}

...

$ echo "please parse this." | ./test.sh

Jaemin Cho · Answer 2 · Tue Sep 06 2016 11:14:43 GMT+0800 (China Standard Time)

Could you please let me know how to check SyntaxNet version?
I downladed it via brianlow/syntaxnet docker image.

Let's say I am using the old one here.

So when I try to use UD_English corpus,
do I need to replace all
"OUTPATH" in models/syntaxnet/work/models/context.pbtxt
to
"/models/syntaxnet/syntaxnet/models/parsey_mcparseface" ?

My docker image has structure as below.


root@19a0b2aad139:~/models/syntaxnet# ls
Dockerfile  bazel-bin       bazel-syntaxnet           ff_nn_schematic.png  syntaxnet    tools
README.md   bazel-genfiles  bazel-testlogs            looping-parser.gif   tensorflow   util
WORKSPACE   bazel-out       beam_search_training.png  sawman.png           third_party  work

root@19a0b2aad139:~/models/syntaxnet# cd syntaxnet/
root@19a0b2aad139:~/models/syntaxnet/syntaxnet# ls
BUILD                             lexicon_builder.cc       sentence_features.h
affix.cc                          lexicon_builder_test.py  sentence_features_test.cc
affix.h                           load_parser_ops.py       shared_store.cc
arc_standard_transitions.cc       models                   shared_store.h
arc_standard_transitions_test.cc  ops                      shared_store_test.cc
base.h                            parser_eval.py           sparse.proto
beam_reader_ops.cc                parser_features.cc       structured_graph_builder.py
beam_reader_ops_test.py           parser_features.h        syntaxnet.bzl
conll2tree.py                     parser_features_test.cc  tagger_transitions.cc
context.pbtxt                     parser_state.cc          tagger_transitions_test.cc
demo.sh                           parser_state.h           task_context.cc
dictionary.proto                  parser_trainer.py        task_context.h
document_filters.cc               parser_trainer_test.sh   task_spec.proto
document_format.cc                parser_transitions.cc    term_frequency_map.cc
document_format.h                 parser_transitions.h     term_frequency_map.h
embedding_feature_extractor.cc    populate_test_inputs.cc  test_main.cc
embedding_feature_extractor.h     populate_test_inputs.h   testdata
feature_extractor.cc              proto_io.h               text_formats.cc
feature_extractor.h               reader_ops.cc            text_formats_test.py
feature_extractor.proto           reader_ops_test.py       unpack_sparse_features.cc
feature_types.h                   registry.cc              utils.cc
fml_parser.cc                     registry.h               utils.h
fml_parser.h                      sentence.proto           workspace.cc
graph_builder.py                  sentence_batch.cc        workspace.h
graph_builder_test.py             sentence_batch.h
kbest_syntax.proto                sentence_features.cc

root@19a0b2aad139:~/models/syntaxnet/syntaxnet# cd models/
root@19a0b2aad139:~/models/syntaxnet/syntaxnet/models# ls
parsey_mcparseface

root@19a0b2aad139:~/models/syntaxnet/syntaxnet/models# cd parsey_mcparseface/
root@19a0b2aad139:~/models/syntaxnet/syntaxnet/models/parsey_mcparseface# ls
context.pbtxt          label-map      prefix-table  tag-map        word-map
fine-to-universal.map  parser-params  suffix-table  tagger-params

Myungchul Shin · Answer 3 · Tue Sep 06 2016 12:39:25 GMT+0800 (China Standard Time)

@j-min

i am using syntaxnet built from sources.

$ git clone --recursive --recurse-submodules https://github.com/tensorflow/models.git
...

and you can check syntaxnet version via git log

$ cd models/syntaxnet
$ git log

mine is

commit a4b7bb9a5dd2c021edcd3d68d326255c734d0ef0
Author: David Mansfield <dmansfield@users.noreply.github.com>
Date:   Tue Jun 28 12:59:10 2016 -0400

    Compatibility with tensorflow master (#204)

    * TF defines the same external repositories but with different commit and BUILD

    * Fix a bunch of bazel namespace issues

    @tf changed to @org_tensorflow
    Tensorflow stopped using protobuf as submodule, now it's external, so use @protobuf

    * Sync to master of tensorflow submodule:
      - local version of is_cuda in syntaxnet.bzl,
      - using unique_ptr in file api.

you need to replace OUTPATH in ~models/syntaxnet/work/models/context.pbtxt.template
and copy to ~models/syntaxnet/work/models/context.pbtxt.

test.sh does not use parsey_mcparseyface but use a generated model(work/models/*).

Jaemin Cho · Answer 4 · Tue Sep 06 2016 12:52:01 GMT+0800 (China Standard Time)

Maybe I didn't fully understand your train.sh,
but shouldn't I change
~/models/syntaxnet/work/UD_English/context.pbtxt
instead of ~/models/syntaxnet/work/models/context.pbtxt?
It seems like train.sh is using ~/models/syntaxnet/work/UD_English/context.pbtxt by default.

# current dir of this script
CDIR=$(readlink -f $(dirname $(readlink -f ${BASH_SOURCE[0]})))
...
CORPUS_DIR=${CDIR}/UD_English

CONTEXT=${CORPUS_DIR}/context.pbtxt
...
POS_PARAMS=${POS_HIDDEN_LAYER_PARAMS}-0.08-500-0.9-0 # I changed some hyperparameters for quick trial
function train_pos_tagger {
        ${BINDIR}/parser_trainer \
          --task_context=${CONTEXT} \
          --arg_prefix=brain_pos \
          --compute_lexicon \

Myungchul Shin · Answer 5 · Tue Sep 06 2016 13:03:20 GMT+0800 (China Standard Time)

@j-min
yes you are right.

train : work/UD_English/context.pbtxt
test : work/models/context.pbtxt

those context.pbtxt files are not same. and error messages you encountered is related to work/models/context.pbtxt.

Jaemin Cho · Answer 6 · Tue Sep 06 2016 14:02:59 GMT+0800 (China Standard Time)

I edited work/models/context.pbtxt as below.

...
input {
  name: "tag-map"
  Part {
    file_pattern: "~/models/syntaxnet/work/models/tag-map"
  }
}
input {
  name: "word-map"
  Part {
    file_pattern: "~/models/syntaxnet/work/models/word-map"
  }
}
input {
  name: "label-map"
  Part {
    file_pattern: "~/models/syntaxnet/work/models/label-map"
  }
}
input {
  name: "lcword-map"
  Part {
    file_pattern: "~/models/syntaxnet/work/models/lcword-map"
  }
}
input {
  name: "category-map"
  Part {
    file_pattern: "~/models/syntaxnet/work/models/category-map"
  }
}
...

But I still see the error message as below.
There actually is label-map inside the directory as I can open the file with

vim ~/models/syntaxnet/work/models/label-map

I guess I misunderstand directory format in protocol buffer.
Could you please give me an example of directory of source folder?
or specifically, "OUTPATH/label-map"?
The output files from training such as tag-map, word-map, label-map are stored in work/models as below.

root@19a0b2aad139:~/models/syntaxnet/work# cd models
root@19a0b2aad139:~/models/syntaxnet/work/models# ls
category-map   context.pbtxt.template  lcword-map     prefix-table  tag-map          tagger-params
context.pbtxt  label-map               parser-params  suffix-table  tag-to-category  word-map

Jaemin Cho · Answer 7 · Tue Sep 06 2016 14:13:30 GMT+0800 (China Standard Time)

Never mind I looked into parsey_mcparseface's context.pbtxt,
and changed the path as work/models/label-map
I will update after training and test with this new setup.

Jaemin Cho · Answer 8 · Tue Sep 06 2016 15:25:48 GMT+0800 (China Standard Time)

After I changed my OUTPATH => work/models in work/models/context.pbtxt,

I faced new error message as below.

root@224fbcff2a39:~/models/syntaxnet/work# echo "parse this please." | ./test.sh
[libprotobuf ERROR external/tf/google/protobuf/src/google/protobuf/text_format.cc:291] Error parsing text-format syntaxnet.TaskSpec: 5:11: Message type "syntaxnet.TaskSpec.Parameter" has no field named "Parameter".
E external/tf/tensorflow/core/common_runtime/executor.cc:333] Executor failed to create kernel. Invalid argument: Could not parse task context at /root/models/syntaxnet/work/models/context.pbtxt
     [[Node: FeatureSize = FeatureSize[arg_prefix="brain_pos", task_context="/root/models/syntaxnet/work/models/context.pbtxt", _device="/job:localhost/replica:0/task:0/cpu:0"]()]]
[libprotobuf ERROR external/tf/google/protobuf/src/google/protobuf/text_format.cc:291] Error parsing text-format syntaxnet.TaskSpec: 5:11: Message type "syntaxnet.TaskSpec.Parameter" has no field named "Parameter".
E external/tf/tensorflow/core/common_runtime/executor.cc:333] Executor failed to create kernel. Invalid argument: Could not parse task context at /root/models/syntaxnet/work/models/context.pbtxt
     [[Node: FeatureSize = FeatureSize[arg_prefix="brain_pos", task_context="/root/models/syntaxnet/work/models/context.pbtxt", _device="/job:localhost/replica:0/task:0/cpu:0"]()]]
Traceback (most recent call last):
  File "/root/models/syntaxnet/bazel-bin/syntaxnet/parser_eval.runfiles/syntaxnet/parser_eval.py", line 149, in <module>
    tf.app.run()
  File "/root/models/syntaxnet/bazel-bin/syntaxnet/parser_eval.runfiles/external/tf/tensorflow/python/platform/app.py", line 30, in run
    sys.exit(main(sys.argv))
  File "/root/models/syntaxnet/bazel-bin/syntaxnet/parser_eval.runfiles/syntaxnet/parser_eval.py", line 142, in main
    arg_prefix=FLAGS.arg_prefix))
  File "/root/models/syntaxnet/bazel-bin/syntaxnet/parser_eval.runfiles/external/tf/tensorflow/python/client/session.py", line 333, in run
    run_metadata_ptr)
  File "/root/models/syntaxnet/bazel-bin/syntaxnet/parser_eval.runfiles/external/tf/tensorflow/python/client/session.py", line 573, in _run
    feed_dict_string, options, run_metadata)
  File "/root/models/syntaxnet/bazel-bin/syntaxnet/parser_eval.runfiles/external/tf/tensorflow/python/client/session.py", line 653, in _do_run
    target_list, options, run_metadata)
  File "/root/models/syntaxnet/bazel-bin/syntaxnet/parser_eval.runfiles/external/tf/tensorflow/python/client/session.py", line 673, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors.InvalidArgumentError: Could not parse task context at /root/models/syntaxnet/work/models/context.pbtxt
     [[Node: FeatureSize = FeatureSize[arg_prefix="brain_pos", task_context="/root/models/syntaxnet/work/models/context.pbtxt", _device="/job:localhost/replica:0/task:0/cpu:0"]()]]
Caused by op u'FeatureSize', defined at:
  File "/root/models/syntaxnet/bazel-bin/syntaxnet/parser_eval.runfiles/syntaxnet/parser_eval.py", line 149, in <module>
    tf.app.run()
  File "/root/models/syntaxnet/bazel-bin/syntaxnet/parser_eval.runfiles/external/tf/tensorflow/python/platform/app.py", line 30, in run
    sys.exit(main(sys.argv))
  File "/root/models/syntaxnet/bazel-bin/syntaxnet/parser_eval.runfiles/syntaxnet/parser_eval.py", line 142, in main
    arg_prefix=FLAGS.arg_prefix))
  File "/root/models/syntaxnet/bazel-bin/syntaxnet/parser_eval.runfiles/syntaxnet/ops/gen_parser_ops.py", line 283, in feature_size
    arg_prefix=arg_prefix, name=name)
  File "/root/models/syntaxnet/bazel-bin/syntaxnet/parser_eval.runfiles/external/tf/tensorflow/python/ops/op_def_library.py", line 693, in apply_op
    op_def=op_def)
  File "/root/models/syntaxnet/bazel-bin/syntaxnet/parser_eval.runfiles/external/tf/tensorflow/python/framework/ops.py", line 2186, in create_op
    original_op=self._default_original_op, op_def=op_def)
  File "/root/models/syntaxnet/bazel-bin/syntaxnet/parser_eval.runfiles/external/tf/tensorflow/python/framework/ops.py", line 1170, in __init__
    self._traceback = _extract_stack()

[libprotobuf ERROR external/tf/google/protobuf/src/google/protobuf/text_format.cc:291] Error parsing text-format syntaxnet.TaskSpec: 5:11: Message type "syntaxnet.TaskSpec.Parameter" has no field named "Parameter".
E external/tf/tensorflow/core/common_runtime/executor.cc:333] Executor failed to create kernel. Invalid argument: Could not parse task context at /root/models/syntaxnet/work/models/context.pbtxt
     [[Node: FeatureSize = FeatureSize[arg_prefix="brain_parser", task_context="/root/models/syntaxnet/work/models/context.pbtxt", _device="/job:localhost/replica:0/task:0/cpu:0"]()]]
[libprotobuf ERROR external/tf/google/protobuf/src/google/protobuf/text_format.cc:291] Error parsing text-format syntaxnet.TaskSpec: 5:11: Message type "syntaxnet.TaskSpec.Parameter" has no field named "Parameter".
E external/tf/tensorflow/core/common_runtime/executor.cc:333] Executor failed to create kernel. Invalid argument: Could not parse task context at /root/models/syntaxnet/work/models/context.pbtxt
     [[Node: FeatureSize = FeatureSize[arg_prefix="brain_parser", task_context="/root/models/syntaxnet/work/models/context.pbtxt", _device="/job:localhost/replica:0/task:0/cpu:0"]()]]
Traceback (most recent call last):
  File "/root/models/syntaxnet/bazel-bin/syntaxnet/parser_eval.runfiles/syntaxnet/parser_eval.py", line 149, in <module>
    tf.app.run()
  File "/root/models/syntaxnet/bazel-bin/syntaxnet/parser_eval.runfiles/external/tf/tensorflow/python/platform/app.py", line 30, in run
    sys.exit(main(sys.argv))
  File "/root/models/syntaxnet/bazel-bin/syntaxnet/parser_eval.runfiles/syntaxnet/parser_eval.py", line 142, in main
    arg_prefix=FLAGS.arg_prefix))
  File "/root/models/syntaxnet/bazel-bin/syntaxnet/parser_eval.runfiles/external/tf/tensorflow/python/client/session.py", line 333, in run
    run_metadata_ptr)
  File "/root/models/syntaxnet/bazel-bin/syntaxnet/parser_eval.runfiles/external/tf/tensorflow/python/client/session.py", line 573, in _run
    feed_dict_string, options, run_metadata)
  File "/root/models/syntaxnet/bazel-bin/syntaxnet/parser_eval.runfiles/external/tf/tensorflow/python/client/session.py", line 653, in _do_run
    target_list, options, run_metadata)
  File "/root/models/syntaxnet/bazel-bin/syntaxnet/parser_eval.runfiles/external/tf/tensorflow/python/client/session.py", line 673, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors.InvalidArgumentError: Could not parse task context at /root/models/syntaxnet/work/models/context.pbtxt
     [[Node: FeatureSize = FeatureSize[arg_prefix="brain_parser", task_context="/root/models/syntaxnet/work/models/context.pbtxt", _device="/job:localhost/replica:0/task:0/cpu:0"]()]]
Caused by op u'FeatureSize', defined at:
  File "/root/models/syntaxnet/bazel-bin/syntaxnet/parser_eval.runfiles/syntaxnet/parser_eval.py", line 149, in <module>
    tf.app.run()
  File "/root/models/syntaxnet/bazel-bin/syntaxnet/parser_eval.runfiles/external/tf/tensorflow/python/platform/app.py", line 30, in run
    sys.exit(main(sys.argv))
  File "/root/models/syntaxnet/bazel-bin/syntaxnet/parser_eval.runfiles/syntaxnet/parser_eval.py", line 142, in main
    arg_prefix=FLAGS.arg_prefix))
  File "/root/models/syntaxnet/bazel-bin/syntaxnet/parser_eval.runfiles/syntaxnet/ops/gen_parser_ops.py", line 283, in feature_size
    arg_prefix=arg_prefix, name=name)
  File "/root/models/syntaxnet/bazel-bin/syntaxnet/parser_eval.runfiles/external/tf/tensorflow/python/ops/op_def_library.py", line 693, in apply_op
    op_def=op_def)
  File "/root/models/syntaxnet/bazel-bin/syntaxnet/parser_eval.runfiles/external/tf/tensorflow/python/framework/ops.py", line 2186, in create_op
    original_op=self._default_original_op, op_def=op_def)
  File "/root/models/syntaxnet/bazel-bin/syntaxnet/parser_eval.runfiles/external/tf/tensorflow/python/framework/ops.py", line 1170, in __init__
    self._traceback = _extract_stack()

[libprotobuf ERROR external/tf/google/protobuf/src/google/protobuf/text_format.cc:291] Error parsing text-format syntaxnet.TaskSpec: 5:11: Message type "syntaxnet.TaskSpec.Parameter" has no field named "Parameter".
F ./syntaxnet/proto_io.h:147] Check failed: input.record_format_size() == 1 (0 vs. 1)TextReader only supports inputs with one record format: name: "stdin-conll"

Myungchul Shin · Answer 9 · Tue Sep 06 2016 20:39:16 GMT+0800 (China Standard Time)

aha ~ i got it.
what about using absolute path for OUTPATH?

Myungchul Shin · Answer 10 · Tue Sep 06 2016 20:43:01 GMT+0800 (China Standard Time)

and models/syntaxnet/models/* you have are exactly same as mine.
when you replace OUTPATH, you have to use absolute path. so, parser_eval is able to find the resource files.

Jaemin Cho · Answer 11 · Thu Sep 08 2016 12:10:16 GMT+0800 (China Standard Time)

Sorry for late reply. I ended up reinstalling the recent version of syntaxnet on a cloud instance and everything works fine now :)
Thank you for your helps!

Myungchul Shin · Answer 12 · Thu Sep 08 2016 20:16:57 GMT+0800 (China Standard Time)

it sounds good~ :)