bytedance / byteps

A high performance and generic framework for distributed DNN training

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

安装报错

llplay opened this issue · comments

安装报错,是什么依赖安装的不对吗,辛苦看下
python 3.7.5
tensorflow 2.5.0

` x86_64-linux-gnu-gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -DEIGEN_MPL2_ONLY=1 -I3rdparty/ps-lite/include -I/usr/local/nccl/include -I/usr/include/python3.7m -c byteps/common/compressor/impl/dithering.cc -o build/temp.linux-x86_64-3.7/byteps/common/compressor/impl/dithering.o -std=c++11 -fPIC -Ofast -Wall -fopenmp -march=native -mno-avx512f -D_GLIBCXX_USE_CXX11_ABI=0 -DBYTEPS_BUILDING_SERVER
In file included from byteps/common/compressor/impl/../compressor.h:23:0,
from byteps/common/compressor/impl/../compressor_registry.h:19,
from byteps/common/compressor/impl/dithering.cc:19:
byteps/common/compressor/impl/dithering.cc: In member function ‘virtual byteps::common::compressor::tensor_t byteps::common::compressor::DitheringCompressor::Compress(byteps::common::compressor::tensor_t)’:
byteps/common/compressor/impl/../common.h:48:42: error: ISO C++ forbids declaration of ‘type name’ with no type [-fpermissive]
reinterpret_cast<const half_t*>(src),
^
byteps/common/compressor/impl/dithering.cc:119:3: note: in expansion of macro ‘COMPRESS_IMPL_SWITCH’
COMPRESS_IMPL_SWITCH(grad.dtype, CompressImpl, _buf.get(), grad.data,
^~~~~~~~~~~~~~~~~~~~
byteps/common/compressor/impl/../common.h:48:42: error: expected ‘>’ before ‘half_t’
reinterpret_cast<const half_t*>(src),
^
byteps/common/compressor/impl/dithering.cc:119:3: note: in expansion of macro ‘COMPRESS_IMPL_SWITCH’
COMPRESS_IMPL_SWITCH(grad.dtype, CompressImpl, _buf.get(), grad.data,
^~~~~~~~~~~~~~~~~~~~
byteps/common/compressor/impl/../common.h:48:42: error: expected ‘(’ before ‘half_t’
reinterpret_cast<const half_t*>(src),
^
byteps/common/compressor/impl/dithering.cc:119:3: note: in expansion of macro ‘COMPRESS_IMPL_SWITCH’
COMPRESS_IMPL_SWITCH(grad.dtype, CompressImpl, _buf.get(), grad.data,
^~~~~~~~~~~~~~~~~~~~
byteps/common/compressor/impl/../common.h:48:42: error: ‘half_t’ was not declared in this scope
reinterpret_cast<const half_t*>(src),
^
byteps/common/compressor/impl/dithering.cc:119:3: note: in expansion of macro ‘COMPRESS_IMPL_SWITCH’
COMPRESS_IMPL_SWITCH(grad.dtype, CompressImpl, _buf.get(), grad.data,
^~~~~~~~~~~~~~~~~~~~
byteps/common/compressor/impl/../common.h:48:42: note: suggested alternative: ‘off_t’
reinterpret_cast<const half_t*>(src),
^
byteps/common/compressor/impl/dithering.cc:119:3: note: in expansion of macro ‘COMPRESS_IMPL_SWITCH’
COMPRESS_IMPL_SWITCH(grad.dtype, CompressImpl, _buf.get(), grad.data,
^~~~~~~~~~~~~~~~~~~~
byteps/common/compressor/impl/../common.h:48:49: error: expected primary-expression before ‘>’ token
reinterpret_cast<const half_t*>(src),
^
byteps/common/compressor/impl/dithering.cc:119:3: note: in expansion of macro ‘COMPRESS_IMPL_SWITCH’
COMPRESS_IMPL_SWITCH(grad.dtype, CompressImpl, _buf.get(), grad.data,
^~~~~~~~~~~~~~~~~~~~
byteps/common/compressor/impl/dithering.cc: In member function ‘virtual byteps::common::compressor::tensor_t byteps::common::compressor::DitheringCompressor::Decompress(byteps::common::compressor::tensor_t)’:
byteps/common/compressor/impl/../common.h:64:36: error: ‘half_t’ does not name a type; did you mean ‘off_t’?
return func(reinterpret_cast<half_t*>(dst),
^
byteps/common/compressor/impl/dithering.cc:170:3: note: in expansion of macro ‘DECOMPRESS_IMPL_SWITCH’
DECOMPRESS_IMPL_SWITCH(_dtype, DecompressImpl, dst, compressed.data,
^~~~~~~~~~~~~~~~~~~~~~
byteps/common/compressor/impl/../common.h:64:42: error: expected ‘>’ before ‘’ token
return func(reinterpret_cast<half_t
>(dst),
^
byteps/common/compressor/impl/dithering.cc:170:3: note: in expansion of macro ‘DECOMPRESS_IMPL_SWITCH’
DECOMPRESS_IMPL_SWITCH(_dtype, DecompressImpl, dst, compressed.data,
^~~~~~~~~~~~~~~~~~~~~~
byteps/common/compressor/impl/../common.h:64:42: error: expected ‘(’ before ‘’ token
return func(reinterpret_cast<half_t
>(dst),
^
byteps/common/compressor/impl/dithering.cc:170:3: note: in expansion of macro ‘DECOMPRESS_IMPL_SWITCH’
DECOMPRESS_IMPL_SWITCH(_dtype, DecompressImpl, dst, compressed.data,
^~~~~~~~~~~~~~~~~~~~~~
byteps/common/compressor/impl/../common.h:64:43: error: expected primary-expression before ‘>’ token
return func(reinterpret_cast<half_t*>(dst),
^
byteps/common/compressor/impl/dithering.cc:170:3: note: in expansion of macro ‘DECOMPRESS_IMPL_SWITCH’
DECOMPRESS_IMPL_SWITCH(_dtype, DecompressImpl, dst, compressed.data,
^~~~~~~~~~~~~~~~~~~~~~
byteps/common/compressor/impl/dithering.cc: In member function ‘virtual void byteps::common::compressor::DitheringCompressor::FastUpdateError(byteps::common::compressor::tensor_t, byteps::common::compressor::tensor_t, byteps::common::compressor::tensor_t)’:
byteps/common/compressor/impl/../common.h:80:36: error: ‘half_t’ does not name a type; did you mean ‘off_t’?
return func(reinterpret_cast<half_t*>(dst),
^
byteps/common/compressor/impl/dithering.cc:212:3: note: in expansion of macro ‘FAST_UPDATE_ERROR_IMPL_SWITCH’
FAST_UPDATE_ERROR_IMPL_SWITCH(_dtype, FastUpdateErrorImpl, error.data,
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~
byteps/common/compressor/impl/../common.h:80:42: error: expected ‘>’ before ‘’ token
return func(reinterpret_cast<half_t
>(dst),
^
byteps/common/compressor/impl/dithering.cc:212:3: note: in expansion of macro ‘FAST_UPDATE_ERROR_IMPL_SWITCH’
FAST_UPDATE_ERROR_IMPL_SWITCH(_dtype, FastUpdateErrorImpl, error.data,
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~
byteps/common/compressor/impl/../common.h:80:42: error: expected ‘(’ before ‘’ token
return func(reinterpret_cast<half_t
>(dst),
^
byteps/common/compressor/impl/dithering.cc:212:3: note: in expansion of macro ‘FAST_UPDATE_ERROR_IMPL_SWITCH’
FAST_UPDATE_ERROR_IMPL_SWITCH(_dtype, FastUpdateErrorImpl, error.data,
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~
byteps/common/compressor/impl/../common.h:80:43: error: expected primary-expression before ‘>’ token
return func(reinterpret_cast<half_t*>(dst),
^
byteps/common/compressor/impl/dithering.cc:212:3: note: in expansion of macro ‘FAST_UPDATE_ERROR_IMPL_SWITCH’
FAST_UPDATE_ERROR_IMPL_SWITCH(_dtype, FastUpdateErrorImpl, error.data,
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~
byteps/common/compressor/impl/../common.h:81:36: error: ‘half_t’ does not name a type; did you mean ‘off_t’?
reinterpret_cast<half_t*>(src1),
^
byteps/common/compressor/impl/dithering.cc:212:3: note: in expansion of macro ‘FAST_UPDATE_ERROR_IMPL_SWITCH’
FAST_UPDATE_ERROR_IMPL_SWITCH(_dtype, FastUpdateErrorImpl, error.data,
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~
byteps/common/compressor/impl/../common.h:81:42: error: expected ‘>’ before ‘’ token
reinterpret_cast<half_t
>(src1),
^
byteps/common/compressor/impl/dithering.cc:212:3: note: in expansion of macro ‘FAST_UPDATE_ERROR_IMPL_SWITCH’
FAST_UPDATE_ERROR_IMPL_SWITCH(_dtype, FastUpdateErrorImpl, error.data,
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~
byteps/common/compressor/impl/../common.h:81:42: error: expected ‘(’ before ‘’ token
reinterpret_cast<half_t
>(src1),
^
byteps/common/compressor/impl/dithering.cc:212:3: note: in expansion of macro ‘FAST_UPDATE_ERROR_IMPL_SWITCH’
FAST_UPDATE_ERROR_IMPL_SWITCH(_dtype, FastUpdateErrorImpl, error.data,
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~
byteps/common/compressor/impl/../common.h:81:43: error: expected primary-expression before ‘>’ token
reinterpret_cast<half_t*>(src1),
^
byteps/common/compressor/impl/dithering.cc:212:3: note: in expansion of macro ‘FAST_UPDATE_ERROR_IMPL_SWITCH’
FAST_UPDATE_ERROR_IMPL_SWITCH(_dtype, FastUpdateErrorImpl, error.data,
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~
byteps/common/compressor/impl/../common.h:82:76: error: expected ‘)’ before ‘;’ token
reinterpret_cast<const uint16_t*>(src2), compressed_size);
^
byteps/common/compressor/impl/dithering.cc:212:3: note: in expansion of macro ‘FAST_UPDATE_ERROR_IMPL_SWITCH’
FAST_UPDATE_ERROR_IMPL_SWITCH(_dtype, FastUpdateErrorImpl, error.data,
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~
byteps/common/compressor/impl/../common.h:82:76: error: return-statement with a value, in function returning 'void' [-fpermissive]
reinterpret_cast<const uint16_t*>(src2), compressed_size);
^
byteps/common/compressor/impl/dithering.cc:212:3: note: in expansion of macro ‘FAST_UPDATE_ERROR_IMPL_SWITCH’
FAST_UPDATE_ERROR_IMPL_SWITCH(_dtype, FastUpdateErrorImpl, error.data,
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~
error: An ERROR occured while building the server module.

Traceback (most recent call last):
  File "/usr/lib/python3.7/distutils/unixccompiler.py", line 118, in _compile
    extra_postargs)
  File "/usr/lib/python3.7/distutils/ccompiler.py", line 910, in spawn
    spawn(cmd, dry_run=self.dry_run)
  File "/usr/lib/python3.7/distutils/spawn.py", line 36, in spawn
    _spawn_posix(cmd, search_path, dry_run=dry_run)
  File "/usr/lib/python3.7/distutils/spawn.py", line 159, in _spawn_posix
    % (cmd, exit_status))
distutils.errors.DistutilsExecError: command 'x86_64-linux-gnu-gcc' failed with exit status 1

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/tmp/pip-install-gg9uk_gv/byteps_d2daed5e88db4e3dbcc5a12776b83bdf/setup.py", line 944, in build_extensions
    build_server(self, options)
  File "/tmp/pip-install-gg9uk_gv/byteps_d2daed5e88db4e3dbcc5a12776b83bdf/setup.py", line 337, in build_server
    build_ext.build_extension(server_lib)
  File "/usr/local/lib/python3.7/dist-packages/setuptools/command/build_ext.py", line 196, in build_extension
    _build_ext.build_extension(self, ext)
  File "/usr/lib/python3.7/distutils/command/build_ext.py", line 534, in build_extension
    depends=ext.depends)
  File "/usr/lib/python3.7/distutils/ccompiler.py", line 574, in compile
    self._compile(obj, src, ext, cc_args, extra_postargs, pp_opts)
  File "/usr/lib/python3.7/distutils/unixccompiler.py", line 120, in _compile
    raise CompileError(msg)
distutils.errors.CompileError: command 'x86_64-linux-gnu-gcc' failed with exit status 1

----------------------------------------

ERROR: Command errored out with exit status 1: /usr/bin/python -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-gg9uk_gv/byteps_d2daed5e88db4e3dbcc5a12776b83bdf/setup.py'"'"'; file='"'"'/tmp/pip-install-gg9uk_gv/byteps_d2daed5e88db4e3dbcc5a12776b83bdf/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(file) if os.path.exists(file) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, file, '"'"'exec'"'"'))' install --record /tmp/pip-record-6671etx_/install-record.txt --single-version-externally-managed --compile --install-headers /usr/local/include/python3.7/byteps Check the logs for full command output.
`

@llplay
sorry for the delay. what's the installation command you used? the following command works fine in my environment:

$ python3 -V
Python 3.7.3

$ pip3 show tensorflow
Name: tensorflow
Version: 2.5.0

$ python3 setup.py sdist
$ pip3 install dist/byteps-0.2.5.tar.gz