ImportError: cannot import name 'generate_meta'

Question

ImportError: cannot import name 'generate_meta'

melodywolk opened this issue 7 years ago · comments

I'm starting from a fresh venv running Python 3.5.0

Here are my steps
docker run -d --name bblfshd --privileged -p 9432:9432 bblfsh/bblfshd
Then
docker exec -it bblfshd bblfshctl driver install --all

Output:
Installing python driver language from "docker://bblfsh/python-driver:latest"... Done Installing java driver language from "docker://bblfsh/java-driver:latest"... Done

After that I install ast2vec
pip3 install ast2vec
Successfully installed PyStemmer-1.3.0 args-0.1.0 asdf-1.3.1 ast2vec-0.3.5a0 astropy-2.0.2 bblfsh-2.6.1 cachetools-2.0.1 certifi-2017.11.5 chardet-3.0.4 clint-0.5.1 docker-2.6.1 docker-pycreds-0.2.1 google-api-core-0.1.1 google-auth-1.2.1 google-cloud-core-0.28.0 google-cloud-storage-1.6.0 google-resumable-media-0.3.1 googleapis-common-protos-1.5.3 grpcio-1.7.0 grpcio-tools-1.7.0 idna-2.6 jsonschema-2.6.0 lz4-0.10.1 modelforge-0.4.0a0 netifaces-0.10.6 numpy-1.13.3 protobuf-3.4.0 py-1.4.34 pyasn1-0.3.7 pyasn1-modules-0.1.5 pytest-3.2.3 python-dateutil-2.6.1 pyyaml-3.12 requests-2.18.4 rsa-3.4.2 scipy-0.19.1 semantic-version-2.6.0 six-1.11.0 urllib3-1.22 websocket-client-0.44.0

Trying to reproduce https://github.com/src-d/ast2vec/blob/master/topic_modeling.md

ast2vec enry

I get the error message:
Traceback (most recent call last): File "/Users/melodywolk/env2/bin/ast2vec", line 7, in <module> from ast2vec.__main__ import main File "/Users/melodywolk/env2/lib/python3.5/site-packages/ast2vec/__init__.py", line 3, in <module> from ast2vec.bow import BOW, NBOW File "/Users/melodywolk/env2/lib/python3.5/site-packages/ast2vec/bow.py", line 4, in <module> from modelforge import generate_meta ImportError: cannot import name 'generate_meta'

Am I missing something?
Thank you!

Vadim Markovtsev · Answer 1 · Wed Nov 15 2017 02:22:33 GMT+0800 (China Standard Time)

Yep, modelforge has updated. You need to force the previous version.

pip3 uninstall -y modelforge && pip3 install 'modelforge<0.4'

Melody Wolk · Answer 2 · Wed Nov 15 2017 03:37:02 GMT+0800 (China Standard Time)

Thanks it did the trick. I'm still having issues to reproduce the results.

I created a file repos.txt
repos.txt

Then run
ast2vec repos2uast -p 16 -t 4 --organize-files 2 -o uasts repos.txt

Here is the full output

WARNING:bblfsh:Failed to ensure that the Babelfish server is running.
INFO:bblfsh:Detected bblfsh server: 0.0.0.0:94320
INFO:bblfsh:Detected bblfsh server: 0.0.0.0:9432
INFO:bblfsh:Detected bblfsh server: 0.0.0.0:9432
INFO:repo_cloner:Cloning from https://github.com/pytorch/pytorch...
INFO:repo_cloner:Cloning from https://github.com/tensorflow/tensorflow...
INFO:repo_cloner:Cloning from https://...
ERROR:repo_cloner:Git failed to clone repo. git stderr:
	Cloning into '/var/folders/16/9dq69g6927zf323r1bxk96r80000gn/T/repo2-bsx_tyf3/&_'...
	fatal: unable to access 'https://': SSL: no alternative certificate subject name matches target host name 'info'
	
ERROR:uast_transformer:Failed to clone : Command '['git', 'clone', '--depth=1', 'https://', '/var/folders/16/9dq69g6927zf323r1bxk96r80000gn/T/repo2-bsx_tyf3/&_']' returned non-zero exit status 128
INFO:repo_cloner:Finished cloning https://github.com/pytorch/pytorch
INFO:repo_cloner:Classifying the files...
INFO:repo_cloner:Result: {'Vim script': 2, 'Shell': 12, 'CMake': 24, 'Cuda': 300, 'C': 316, 'Python': 342, 'Markdown': 2, 'C++': 385, 'Makefile': 3, 'Fortran': 12, 'Dockerfile': 1, 'Batchfile': 2, 'Text': 17, 'Roff': 1, 'Lua': 2}
INFO:repo2uast:Fetching and processing UASTs...
INFO:repo2uast:https://github.com/pytorch/pytorch pending tasks: 339
INFO:repo2uast:https://github.com/pytorch/pytorch pending tasks: 336
INFO:repo2uast:https://github.com/pytorch/pytorch pending tasks: 333
INFO:repo2uast:https://github.com/pytorch/pytorch pending tasks: 330
INFO:repo2uast:https://github.com/pytorch/pytorch pending tasks: 327
INFO:repo2uast:https://github.com/pytorch/pytorch pending tasks: 324
INFO:repo2uast:https://github.com/pytorch/pytorch pending tasks: 321
INFO:repo2uast:https://github.com/pytorch/pytorch pending tasks: 318
INFO:repo2uast:https://github.com/pytorch/pytorch pending tasks: 315
INFO:repo2uast:https://github.com/pytorch/pytorch pending tasks: 312
INFO:repo2uast:https://github.com/pytorch/pytorch pending tasks: 309
INFO:repo2uast:https://github.com/pytorch/pytorch pending tasks: 306
INFO:repo_cloner:Finished cloning https://github.com/tensorflow/tensorflow
INFO:repo_cloner:Classifying the files...
INFO:repo2uast:https://github.com/pytorch/pytorch pending tasks: 303
INFO:repo2uast:https://github.com/pytorch/pytorch pending tasks: 300
INFO:repo2uast:https://github.com/pytorch/pytorch pending tasks: 297
INFO:repo2uast:https://github.com/pytorch/pytorch pending tasks: 294
INFO:repo2uast:https://github.com/pytorch/pytorch pending tasks: 291
INFO:repo2uast:https://github.com/pytorch/pytorch pending tasks: 288
INFO:repo2uast:https://github.com/pytorch/pytorch pending tasks: 285
INFO:repo2uast:https://github.com/pytorch/pytorch pending tasks: 282
INFO:repo2uast:https://github.com/pytorch/pytorch pending tasks: 279
INFO:repo_cloner:Result: {'Shell': 113, 'C': 68, 'Python': 2573, 'Perl': 2, 'Jupyter Notebook': 13, 'Cython': 1, 'Linker Script': 8, 'Ruby': 4, 'Dockerfile': 7, 'Batchfile': 9, 'Go': 29, 'Objective-C++': 16, 'Objective-C': 10, 'Protocol Buffer': 93, 'CMake': 18, 'Markdown': 160, 'C++': 3850, 'CSV': 14, 'Java': 81, 'PureBasic': 4, 'Unity3D Asset': 4, 'PHP': 1, 'LLVM': 1, 'Makefile': 9, 'Text': 36, 'Gradle': 6}
INFO:repo2uast:Fetching and processing UASTs...
INFO:repo2uast:https://github.com/tensorflow/tensorflow pending tasks: 2652
INFO:repo2uast:https://github.com/pytorch/pytorch pending tasks: 276
INFO:repo2uast:https://github.com/pytorch/pytorch pending tasks: 273
INFO:repo2uast:https://github.com/pytorch/pytorch pending tasks: 270
INFO:repo2uast:https://github.com/pytorch/pytorch pending tasks: 267
INFO:repo2uast:https://github.com/pytorch/pytorch pending tasks: 264
INFO:repo2uast:https://github.com/pytorch/pytorch pending tasks: 261
INFO:repo2uast:https://github.com/pytorch/pytorch pending tasks: 258
INFO:repo2uast:https://github.com/pytorch/pytorch pending tasks: 255
INFO:repo2uast:https://github.com/pytorch/pytorch pending tasks: 252
INFO:repo2uast:https://github.com/pytorch/pytorch pending tasks: 249
INFO:repo2uast:https://github.com/tensorflow/tensorflow pending tasks: 2626
INFO:repo2uast:https://github.com/pytorch/pytorch pending tasks: 246
INFO:repo2uast:https://github.com/pytorch/pytorch pending tasks: 243
INFO:repo2uast:https://github.com/pytorch/pytorch pending tasks: 240
INFO:repo2uast:https://github.com/pytorch/pytorch pending tasks: 237
INFO:repo2uast:https://github.com/pytorch/pytorch pending tasks: 234
INFO:repo2uast:https://github.com/pytorch/pytorch pending tasks: 231
INFO:repo2uast:https://github.com/pytorch/pytorch pending tasks: 228
INFO:repo2uast:https://github.com/pytorch/pytorch pending tasks: 225
INFO:repo2uast:https://github.com/pytorch/pytorch pending tasks: 222
INFO:repo2uast:https://github.com/pytorch/pytorch pending tasks: 219
INFO:repo2uast:https://github.com/pytorch/pytorch pending tasks: 216
INFO:repo2uast:https://github.com/pytorch/pytorch pending tasks: 213
INFO:repo2uast:https://github.com/pytorch/pytorch pending tasks: 210
INFO:repo2uast:https://github.com/pytorch/pytorch pending tasks: 207
INFO:repo2uast:https://github.com/pytorch/pytorch pending tasks: 204
INFO:repo2uast:https://github.com/pytorch/pytorch pending tasks: 201
INFO:repo2uast:https://github.com/pytorch/pytorch pending tasks: 198
INFO:repo2uast:https://github.com/pytorch/pytorch pending tasks: 195
INFO:repo2uast:https://github.com/pytorch/pytorch pending tasks: 192
INFO:repo2uast:https://github.com/pytorch/pytorch pending tasks: 189
INFO:repo2uast:https://github.com/pytorch/pytorch pending tasks: 186
INFO:repo2uast:https://github.com/pytorch/pytorch pending tasks: 183
INFO:repo2uast:https://github.com/pytorch/pytorch pending tasks: 180
INFO:repo2uast:https://github.com/pytorch/pytorch pending tasks: 177
INFO:repo2uast:https://github.com/pytorch/pytorch pending tasks: 174
INFO:repo2uast:https://github.com/pytorch/pytorch pending tasks: 171
INFO:repo2uast:https://github.com/pytorch/pytorch pending tasks: 168
INFO:repo2uast:https://github.com/pytorch/pytorch pending tasks: 165
INFO:repo2uast:https://github.com/pytorch/pytorch pending tasks: 162
INFO:repo2uast:https://github.com/tensorflow/tensorflow pending tasks: 2600
INFO:repo2uast:https://github.com/pytorch/pytorch pending tasks: 159
INFO:repo2uast:https://github.com/pytorch/pytorch pending tasks: 156
INFO:repo2uast:https://github.com/pytorch/pytorch pending tasks: 153
INFO:repo2uast:https://github.com/pytorch/pytorch pending tasks: 150
INFO:repo2uast:https://github.com/pytorch/pytorch pending tasks: 147
INFO:repo2uast:https://github.com/pytorch/pytorch pending tasks: 144
INFO:repo2uast:https://github.com/pytorch/pytorch pending tasks: 141
INFO:repo2uast:https://github.com/pytorch/pytorch pending tasks: 138
INFO:repo2uast:https://github.com/pytorch/pytorch pending tasks: 135
INFO:repo2uast:https://github.com/pytorch/pytorch pending tasks: 132
INFO:repo2uast:https://github.com/pytorch/pytorch pending tasks: 129
INFO:repo2uast:https://github.com/pytorch/pytorch pending tasks: 126
INFO:repo2uast:https://github.com/pytorch/pytorch pending tasks: 123
INFO:repo2uast:https://github.com/pytorch/pytorch pending tasks: 120
INFO:repo2uast:https://github.com/pytorch/pytorch pending tasks: 117
INFO:repo2uast:https://github.com/pytorch/pytorch pending tasks: 114
INFO:repo2uast:https://github.com/pytorch/pytorch pending tasks: 111
INFO:repo2uast:https://github.com/pytorch/pytorch pending tasks: 108
INFO:repo2uast:https://github.com/tensorflow/tensorflow pending tasks: 2574
INFO:repo2uast:https://github.com/pytorch/pytorch pending tasks: 105
INFO:repo2uast:https://github.com/pytorch/pytorch pending tasks: 102
INFO:repo2uast:https://github.com/pytorch/pytorch pending tasks: 99
INFO:repo2uast:https://github.com/pytorch/pytorch pending tasks: 96
INFO:repo2uast:https://github.com/pytorch/pytorch pending tasks: 93
INFO:repo2uast:https://github.com/pytorch/pytorch pending tasks: 90
INFO:repo2uast:https://github.com/pytorch/pytorch pending tasks: 87
INFO:repo2uast:https://github.com/pytorch/pytorch pending tasks: 84
INFO:repo2uast:https://github.com/tensorflow/tensorflow pending tasks: 2548
INFO:repo2uast:https://github.com/pytorch/pytorch pending tasks: 81
INFO:repo2uast:https://github.com/pytorch/pytorch pending tasks: 78
INFO:repo2uast:https://github.com/pytorch/pytorch pending tasks: 75
INFO:repo2uast:https://github.com/pytorch/pytorch pending tasks: 72
INFO:repo2uast:https://github.com/pytorch/pytorch pending tasks: 69
INFO:repo2uast:https://github.com/tensorflow/tensorflow pending tasks: 2522
INFO:repo2uast:https://github.com/pytorch/pytorch pending tasks: 66
INFO:repo2uast:https://github.com/pytorch/pytorch pending tasks: 63
INFO:repo2uast:https://github.com/pytorch/pytorch pending tasks: 60
INFO:repo2uast:https://github.com/pytorch/pytorch pending tasks: 57
INFO:repo2uast:https://github.com/pytorch/pytorch pending tasks: 54
INFO:repo2uast:https://github.com/pytorch/pytorch pending tasks: 51
INFO:repo2uast:https://github.com/pytorch/pytorch pending tasks: 48
INFO:repo2uast:https://github.com/pytorch/pytorch pending tasks: 45
INFO:repo2uast:https://github.com/pytorch/pytorch pending tasks: 42
INFO:repo2uast:https://github.com/pytorch/pytorch pending tasks: 39
INFO:repo2uast:https://github.com/pytorch/pytorch pending tasks: 36
INFO:repo2uast:https://github.com/pytorch/pytorch pending tasks: 33
INFO:repo2uast:https://github.com/pytorch/pytorch pending tasks: 30
INFO:repo2uast:https://github.com/pytorch/pytorch pending tasks: 27
INFO:repo2uast:https://github.com/pytorch/pytorch pending tasks: 24
INFO:repo2uast:https://github.com/pytorch/pytorch pending tasks: 21
INFO:repo2uast:https://github.com/pytorch/pytorch pending tasks: 18
INFO:repo2uast:https://github.com/pytorch/pytorch pending tasks: 15
INFO:repo2uast:https://github.com/pytorch/pytorch pending tasks: 12
INFO:repo2uast:https://github.com/pytorch/pytorch pending tasks: 9
INFO:repo2uast:https://github.com/pytorch/pytorch pending tasks: 6
INFO:repo2uast:https://github.com/tensorflow/tensorflow pending tasks: 2496
INFO:repo2uast:https://github.com/pytorch/pytorch pending tasks: 3
INFO:repo2uast:https://github.com/pytorch/pytorch pending tasks: 0
INFO:repo2uast:https://github.com/tensorflow/tensorflow pending tasks: 2470
INFO:repo2uast:https://github.com/tensorflow/tensorflow pending tasks: 2444
INFO:repo2uast:https://github.com/tensorflow/tensorflow pending tasks: 2418
INFO:repo2uast:https://github.com/tensorflow/tensorflow pending tasks: 2392
INFO:repo2uast:https://github.com/tensorflow/tensorflow pending tasks: 2366
INFO:repo2uast:https://github.com/tensorflow/tensorflow pending tasks: 2340
INFO:repo2uast:https://github.com/tensorflow/tensorflow pending tasks: 2314
INFO:repo2uast:https://github.com/tensorflow/tensorflow pending tasks: 2288
INFO:repo2uast:https://github.com/tensorflow/tensorflow pending tasks: 2262
INFO:repo2uast:https://github.com/tensorflow/tensorflow pending tasks: 2236
INFO:repo2uast:https://github.com/tensorflow/tensorflow pending tasks: 2210
INFO:repo2uast:https://github.com/tensorflow/tensorflow pending tasks: 2184
INFO:repo2uast:https://github.com/tensorflow/tensorflow pending tasks: 2158
INFO:repo2uast:https://github.com/tensorflow/tensorflow pending tasks: 2132
INFO:repo2uast:https://github.com/tensorflow/tensorflow pending tasks: 2106
INFO:repo2uast:https://github.com/tensorflow/tensorflow pending tasks: 2080
INFO:repo2uast:https://github.com/tensorflow/tensorflow pending tasks: 2054
INFO:repo2uast:https://github.com/tensorflow/tensorflow pending tasks: 2028
INFO:repo2uast:https://github.com/tensorflow/tensorflow pending tasks: 2002
INFO:repo2uast:https://github.com/tensorflow/tensorflow pending tasks: 1976
INFO:repo2uast:https://github.com/tensorflow/tensorflow pending tasks: 1950
INFO:repo2uast:https://github.com/tensorflow/tensorflow pending tasks: 1924
INFO:repo2uast:https://github.com/tensorflow/tensorflow pending tasks: 1898
INFO:repo2uast:https://github.com/tensorflow/tensorflow pending tasks: 1872
INFO:repo2uast:https://github.com/tensorflow/tensorflow pending tasks: 1846
INFO:repo2uast:https://github.com/tensorflow/tensorflow pending tasks: 1820
INFO:repo2uast:https://github.com/tensorflow/tensorflow pending tasks: 1794
INFO:repo2uast:https://github.com/tensorflow/tensorflow pending tasks: 1768
WARNING:repo2uast:/var/folders/16/9dq69g6927zf323r1bxk96r80000gn/T/repo2-gpzjzeti/tensorflow&tensorflow_github.com/tensorflow/contrib/metrics/python/ops/metric_ops_test.py was skipped: it is too big - 258530 bytes
INFO:repo2uast:https://github.com/tensorflow/tensorflow pending tasks: 1742
INFO:repo2uast:https://github.com/tensorflow/tensorflow pending tasks: 1716
INFO:repo2uast:https://github.com/tensorflow/tensorflow pending tasks: 1690
INFO:repo2uast:https://github.com/tensorflow/tensorflow pending tasks: 1664
INFO:repo2uast:https://github.com/tensorflow/tensorflow pending tasks: 1638
INFO:repo2uast:https://github.com/tensorflow/tensorflow pending tasks: 1612
INFO:repo2uast:https://github.com/tensorflow/tensorflow pending tasks: 1586
INFO:repo2uast:https://github.com/tensorflow/tensorflow pending tasks: 1560
INFO:repo2uast:https://github.com/tensorflow/tensorflow pending tasks: 1534
INFO:repo2uast:https://github.com/tensorflow/tensorflow pending tasks: 1508
INFO:repo2uast:https://github.com/tensorflow/tensorflow pending tasks: 1482
INFO:repo2uast:https://github.com/tensorflow/tensorflow pending tasks: 1456
INFO:repo2uast:https://github.com/tensorflow/tensorflow pending tasks: 1430
INFO:repo2uast:https://github.com/tensorflow/tensorflow pending tasks: 1404
INFO:repo2uast:https://github.com/tensorflow/tensorflow pending tasks: 1378
INFO:repo2uast:https://github.com/tensorflow/tensorflow pending tasks: 1352
INFO:repo2uast:https://github.com/tensorflow/tensorflow pending tasks: 1326
INFO:repo2uast:https://github.com/tensorflow/tensorflow pending tasks: 1300
INFO:repo2uast:https://github.com/tensorflow/tensorflow pending tasks: 1274
INFO:repo2uast:https://github.com/tensorflow/tensorflow pending tasks: 1248
INFO:repo2uast:https://github.com/tensorflow/tensorflow pending tasks: 1222
INFO:repo2uast:https://github.com/tensorflow/tensorflow pending tasks: 1196
INFO:repo2uast:https://github.com/tensorflow/tensorflow pending tasks: 1170
INFO:repo2uast:https://github.com/tensorflow/tensorflow pending tasks: 1144
INFO:repo2uast:https://github.com/tensorflow/tensorflow pending tasks: 1118
INFO:repo2uast:https://github.com/tensorflow/tensorflow pending tasks: 1092
INFO:repo2uast:https://github.com/tensorflow/tensorflow pending tasks: 1066
INFO:repo2uast:https://github.com/tensorflow/tensorflow pending tasks: 1040
INFO:repo2uast:https://github.com/tensorflow/tensorflow pending tasks: 1014
INFO:repo2uast:https://github.com/tensorflow/tensorflow pending tasks: 988
INFO:repo2uast:https://github.com/tensorflow/tensorflow pending tasks: 962
INFO:repo2uast:https://github.com/tensorflow/tensorflow pending tasks: 936
INFO:repo2uast:https://github.com/tensorflow/tensorflow pending tasks: 910
INFO:repo2uast:https://github.com/tensorflow/tensorflow pending tasks: 884
INFO:repo2uast:https://github.com/tensorflow/tensorflow pending tasks: 858
INFO:repo2uast:https://github.com/tensorflow/tensorflow pending tasks: 832
INFO:repo2uast:https://github.com/tensorflow/tensorflow pending tasks: 806
INFO:repo2uast:https://github.com/tensorflow/tensorflow pending tasks: 780
INFO:repo2uast:https://github.com/tensorflow/tensorflow pending tasks: 754
INFO:repo2uast:https://github.com/tensorflow/tensorflow pending tasks: 728
INFO:repo2uast:https://github.com/tensorflow/tensorflow pending tasks: 702
INFO:repo2uast:https://github.com/tensorflow/tensorflow pending tasks: 676
INFO:repo2uast:https://github.com/tensorflow/tensorflow pending tasks: 650
INFO:repo2uast:https://github.com/tensorflow/tensorflow pending tasks: 624
INFO:repo2uast:https://github.com/tensorflow/tensorflow pending tasks: 598
INFO:repo2uast:https://github.com/tensorflow/tensorflow pending tasks: 572
INFO:repo2uast:https://github.com/tensorflow/tensorflow pending tasks: 546
INFO:repo2uast:https://github.com/tensorflow/tensorflow pending tasks: 520
INFO:repo2uast:https://github.com/tensorflow/tensorflow pending tasks: 494
INFO:repo2uast:https://github.com/tensorflow/tensorflow pending tasks: 468
INFO:repo2uast:https://github.com/tensorflow/tensorflow pending tasks: 442
INFO:repo2uast:https://github.com/tensorflow/tensorflow pending tasks: 416
INFO:repo2uast:https://github.com/tensorflow/tensorflow pending tasks: 390
ERROR:repo2uast:bblfsh: RpcError on /var/folders/16/9dq69g6927zf323r1bxk96r80000gn/T/repo2-gpzjzeti/tensorflow&tensorflow_github.com/tensorflow/python/ops/special_math_ops_test.py: <_Rendezvous of RPC that terminated with (StatusCode.DEADLINE_EXCEEDED, Deadline Exceeded)>
WARNING:repo2uast:/var/folders/16/9dq69g6927zf323r1bxk96r80000gn/T/repo2-gpzjzeti/tensorflow&tensorflow_github.com/tensorflow/python/ops/special_math_ops_test.py was skipped
INFO:repo2uast:https://github.com/tensorflow/tensorflow pending tasks: 364
INFO:repo2uast:https://github.com/tensorflow/tensorflow pending tasks: 338
INFO:repo2uast:https://github.com/tensorflow/tensorflow pending tasks: 312
INFO:repo2uast:https://github.com/tensorflow/tensorflow pending tasks: 286
INFO:repo2uast:https://github.com/tensorflow/tensorflow pending tasks: 260
INFO:repo2uast:https://github.com/tensorflow/tensorflow pending tasks: 234
INFO:repo2uast:https://github.com/tensorflow/tensorflow pending tasks: 208
INFO:repo2uast:https://github.com/tensorflow/tensorflow pending tasks: 182
INFO:repo2uast:https://github.com/tensorflow/tensorflow pending tasks: 156
INFO:repo2uast:https://github.com/tensorflow/tensorflow pending tasks: 130
INFO:repo2uast:https://github.com/tensorflow/tensorflow pending tasks: 104
INFO:repo2uast:https://github.com/tensorflow/tensorflow pending tasks: 78
INFO:repo2uast:https://github.com/tensorflow/tensorflow pending tasks: 52
INFO:repo2uast:https://github.com/tensorflow/tensorflow pending tasks: 26
INFO:repo2uast:https://github.com/tensorflow/tensorflow pending tasks: 0
[################################] 3/3 - 00:16:18
INFO:uast_transformer:Finished, 1 failed repos

ls -ltr uast_github.com\&*
-rw-rw-rw-  1 3791494 Nov 14 10:37 uast_github.com&pytorch&pytorch.asdf
-rw-rw-rw-  1 43125912 Nov 14 10:51 uast_github.com&tensorflow&tensorflow.asdf

Then:


INFO:uast2docfreq:Scanning uasts
INFO:uast2docfreq:Found 2 files
WARNING:docfreq:Did not write /var/folders/16/9dq69g6927zf323r1bxk96r80000gn/T/source2uastlq3tm4su/docfreq_2.asdf because the model is empty
WARNING:docfreq:Did not write /var/folders/16/9dq69g6927zf323r1bxk96r80000gn/T/source2uastlq3tm4su/docfreq_3.asdf because the model is empty
[################################] 2/2 - 00:15:22
INFO:uast2docfreq:Finished, 0 failed files
INFO:docfreq2docfreq:Scanning /var/folders/16/9dq69g6927zf323r1bxk96r80000gn/T/source2uastlq3tm4su
INFO:docfreq2docfreq:Found 2 files
[################################] 2/2 - 00:00:00
INFO:docfreq2docfreq:Finished, 0 failed files

`ast2vec uast2bow --df docfreq.asdf -v 100000 -p 4 uasts bows

INFO:docfreq:Reading docfreq.asdf...
INFO:docfreq:Building the docfreq dictionary...
INFO:docfreq:Pruning to min 20 occurrences
INFO:uast2bow:Choosing the vocabulary...
INFO:uast2bow:Scanning uasts
INFO:uast2bow:Found 2 files
WARNING:bow:Did not write bows/g/gi/uast_github.com&pytorch&pytorch.asdf because the model is empty
WARNING:bow:Did not write bows/g/gi/uast_github.com&tensorflow&tensorflow.asdf because the model is empty
[################################] 2/2 - 00:14:11
INFO:uast2bow:Finished, 0 failed files

Vadim Markovtsev · Answer 3 · Wed Nov 15 2017 03:49:00 GMT+0800 (China Standard Time)

@zurk We seem to have a bug here, please fix this tomorrow ASAP. Remember that this is the current master, not develop.

Konstantin Slavnov · Answer 4 · Wed Nov 15 2017 16:11:24 GMT+0800 (China Standard Time)

ok, let me see.

Konstantin Slavnov · Answer 5 · Wed Nov 15 2017 19:03:07 GMT+0800 (China Standard Time)

@melodywolk
ok, I fix several minor bugs, but they are not really related to the problem.

And the real problem is in this log message:
INFO:docfreq:Pruning to min 20 occurrences

It means that the model leaves identifiers, that were found in 20 repositories at least.
20 is the default parameter value because it is expected that you work with millions of repositories.
Since you have only 2 repositories on your list, you can use prune-df parameter like this:
ast2vec uast2bow --df docfreq.asdf -v 100000 -p 4 --prune-df 0 uasts bows

I fix the bug to make the parameter work, so you should install new ast2vec (0.3.6, just released):
pip3 install ast2vec --upgrade

Also, it is a good idea to have more repositories on the list.
Hard to build good topic model only with two documents in it :)

Thank you for the report!
I think we should make some comments in the instruction (https://github.com/src-d/ast2vec/blob/master/topic_modeling.md)

Vadim Markovtsev · Answer 6 · Wed Nov 15 2017 19:07:40 GMT+0800 (China Standard Time)

Released as 0.3.6.

@melodywolk Please feel free to reopen if there are any problems left.