Two questions about open/R

Question

Two questions about open/R

silveric10 opened this issue 6 years ago · comments

Please use this template for reporting suspected bugs or requests for help.

Issue Description

first question, the NetlinkSocketTestis failed.
second question, there is a file named "platform_linux" will be compiled in /usr/local/sbin, I think it works as the platform agent for linux to program the routes, but when I run it after I run "run_openr", some core dump issues come up.

Environment

ubuntu-16.04

Minimal test code / Steps to reproduce the issue

first issue:
Start 13: NetlinkTypesTest
13/17 Test #13: NetlinkTypesTest ................. Passed 0.09 sec
Start 14: NetlinkSocketTest
14/17 Test #14: NetlinkSocketTest ................***Failed 28.45 sec
Start 15: PrefixManagerTest
15/17 Test #15: PrefixManagerTest ................ Passed 0.35 sec
Start 16: SparkTest
16/17 Test #16: SparkTest ........................ Passed 27.93 sec
Start 17: MockIoProviderTest
17/17 Test #17: MockIoProviderTest ............... Passed 0.28 sec

94% tests passed, 1 tests failed out of 17

Total Test time (real) = 424.98 sec

The following tests FAILED:
14 - NetlinkSocketTest (Failed)
Errors while running CTest
Makefile:117: recipe for target 'test' failed
make: *** [test] Error 8

second issue:
E1106 08:28:46.837707 4258 ThriftServer.cpp:397] Got an exception while setting up the server: 98failed to bind to async server socket: [::]:60099: Address already in use
E1106 08:28:46.844789 4259 ThriftServer.cpp:397] Got an exception while setting up the server: 98failed to bind to async server socket: [::]:60100: Address already in use
terminate called after throwing an instance of 'std::system_error'
what(): 98failed to bind to async server socket: [::]:60100: Address already in use
*** Aborted at 1541521726 (unix time) try "date -d @1541521726" if you are using GNU date ***
PC: @ 0x7fca01a47428 gsignal
terminate called recursively
*** SIGABRT (@0x3e80000109e) received by PID 4254 (TID 0x7fc9f9bbc700) from PID 4254; stack trace: ***
@ 0x7fca0300b390 (unknown)
@ 0x7fca01a47428 gsignal
@ 0x7fca01a4902a abort
@ 0x7fca0239984d __gnu_cxx::__verbose_terminate_handler()
@ 0x7fca023976b6 (unknown)
@ 0x7fca02397701 std::terminate()
@ 0x7fca023c2d38 (unknown)
@ 0x7fca030016ba start_thread
@ 0x7fca01b1941d clone
Aborted (core dumped)

What's the actual result?

What's the expected result?

I want to know if the "NetlinkSocketTest failed" will lead the openr doesn't work properly. And How can I enable the linux platform agent "platform_linux" after I run "run_openr"? Or the "run_openr" already contain the linux fib agent and I do not need to run another?

Thankyou

Saif Hasan · Answer 1 · Wed Nov 07 2018 01:44:19 GMT+0800 (China Standard Time)

Q1 - Can you show the output of netlink socket test failure ? Did you run with "sudo" .. sudo ./netlink_socket_test

Q2 - Crash happens because ports Open/R trying to bind on is already in use by some other process on your system. Do sudo netstat -natp | grep <port> and kill that process. Then try to run openr

silveric10 · Answer 2 · Wed Nov 07 2018 10:34:34 GMT+0800 (China Standard Time)

Hi there：
The output of netlink socket test failure is attached below.
[==========] Running 24 tests from 1 test case.
[----------] Global test environment set-up.
[----------] 24 tests from NetlinkSocketFixture
[ RUN ] NetlinkSocketFixture.EmptyRouteTest
[ OK ] NetlinkSocketFixture.EmptyRouteTest (127 ms)
[ RUN ] NetlinkSocketFixture.SingleRouteTest
[ OK ] NetlinkSocketFixture.SingleRouteTest (117 ms)
[ RUN ] NetlinkSocketFixture.SingleRouteTestV4
[ OK ] NetlinkSocketFixture.SingleRouteTestV4 (99 ms)
[ RUN ] NetlinkSocketFixture.NullRouteTest
[ OK ] NetlinkSocketFixture.NullRouteTest (123 ms)
[ RUN ] NetlinkSocketFixture.NullRouteV4Test
[ OK ] NetlinkSocketFixture.NullRouteV4Test (126 ms)
[ RUN ] NetlinkSocketFixture.UpdateRouteTest
/home/fanya/openr/openr/nl/tests/NetlinkSocketTest.cpp:353: Failure
Expected: 2
To be equal to: rt2.getNextHops().size()
Which is: 1
/home/fanya/openr/openr/nl/tests/NetlinkSocketTest.cpp:366: Failure
Expected: 1
To be equal to: count
Which is: 0
/home/fanya/openr/openr/nl/tests/NetlinkSocketTest.cpp:391: Failure
Expected: 2
To be equal to: rt3.getNextHops().size()
Which is: 1
[ FAILED ] NetlinkSocketFixture.UpdateRouteTest (208 ms)
[ RUN ] NetlinkSocketFixture.UpdateRouteTestV4
[ OK ] NetlinkSocketFixture.UpdateRouteTestV4 (116 ms)
[ RUN ] NetlinkSocketFixture.UpdateMultiRouteTest
/home/fanya/openr/openr/nl/tests/NetlinkSocketTest.cpp:469: Failure
Expected: 3
To be equal to: rt.getNextHops().size()
Which is: 1
/home/fanya/openr/openr/nl/tests/NetlinkSocketTest.cpp:470: Failure
Value of: CompareNextHops(nexthops, rt)
Actual: false
Expected: true
/home/fanya/openr/openr/nl/tests/NetlinkSocketTest.cpp:481: Failure
Expected: 1
To be equal to: count
Which is: 0
/home/fanya/openr/openr/nl/tests/NetlinkSocketTest.cpp:500: Failure
Expected: 2
To be equal to: rt1.getNextHops().size()
Which is: 1
/home/fanya/openr/openr/nl/tests/NetlinkSocketTest.cpp:501: Failure
Value of: CompareNextHops(nexthops, rt1)
Actual: false
Expected: true
/home/fanya/openr/openr/nl/tests/NetlinkSocketTest.cpp:512: Failure
Expected: 1
To be equal to: count
Which is: 0
/home/fanya/openr/openr/nl/tests/NetlinkSocketTest.cpp:527: Failure
Expected: 3
To be equal to: rt2.getNextHops().size()
Which is: 1
/home/fanya/openr/openr/nl/tests/NetlinkSocketTest.cpp:528: Failure
Value of: CompareNextHops(nexthops, rt2)
Actual: false
Expected: true
/home/fanya/openr/openr/nl/tests/NetlinkSocketTest.cpp:539: Failure
Expected: 1
To be equal to: count
Which is: 0
[ FAILED ] NetlinkSocketFixture.UpdateMultiRouteTest (144 ms)
[ RUN ] NetlinkSocketFixture.UpdateMultiRouteTestV4
[ OK ] NetlinkSocketFixture.UpdateMultiRouteTestV4 (107 ms)
[ RUN ] NetlinkSocketFixture.SyncRouteTest
/home/fanya/openr/openr/nl/tests/NetlinkSocketTest.cpp:612: Failure
Expected: 2
To be equal to: count
Which is: 1
/home/fanya/openr/openr/nl/tests/NetlinkSocketTest.cpp:618: Failure
Expected: 2
To be equal to: rt1.getNextHops().size()
Which is: 1
/home/fanya/openr/openr/nl/tests/NetlinkSocketTest.cpp:620: Failure
Value of: CompareNextHops(nexthops1, rt1)
Actual: false
Expected: true
/home/fanya/openr/openr/nl/tests/NetlinkSocketTest.cpp:653: Failure
Expected: 2
To be equal to: count
Which is: 1
/home/fanya/openr/openr/nl/tests/NetlinkSocketTest.cpp:660: Failure
Expected: 2
To be equal to: rt4.getNextHops().size()
Which is: 1
/home/fanya/openr/openr/nl/tests/NetlinkSocketTest.cpp:662: Failure
Value of: CompareNextHops(nexthops2, rt4)
Actual: false
Expected: true
/home/fanya/openr/openr/nl/tests/NetlinkSocketTest.cpp:685: Failure
Expected: 1
To be equal to: count
Which is: 0
/home/fanya/openr/openr/nl/tests/NetlinkSocketTest.cpp:689: Failure
Expected: 2
To be equal to: rt5.getNextHops().size()
Which is: 1
/home/fanya/openr/openr/nl/tests/NetlinkSocketTest.cpp:690: Failure
Value of: CompareNextHops(nexthops2, rt5)
Actual: false
Expected: true
[ FAILED ] NetlinkSocketFixture.SyncRouteTest (147 ms)
[ RUN ] NetlinkSocketFixture.SyncRouteTestV4
[ OK ] NetlinkSocketFixture.SyncRouteTestV4 (108 ms)
[ RUN ] NetlinkSocketFixture.ModifyMulticastRouteTest
[ OK ] NetlinkSocketFixture.ModifyMulticastRouteTest (96 ms)
[ RUN ] NetlinkSocketFixture.MultiPathTest
/home/fanya/openr/openr/nl/tests/NetlinkSocketTest.cpp:1050: Failure
Expected: 2
To be equal to: rt.getNextHops().size()
Which is: 1
/home/fanya/openr/openr/nl/tests/NetlinkSocketTest.cpp:1051: Failure
Value of: CompareNextHops(nexthops1, rt)
Actual: false
Expected: true
/home/fanya/openr/openr/nl/tests/NetlinkSocketTest.cpp:1062: Failure
Expected: 2
To be equal to: rt2.getNextHops().size()
Which is: 1
/home/fanya/openr/openr/nl/tests/NetlinkSocketTest.cpp:1063: Failure
Expected: 2
To be equal to: rt3.getNextHops().size()
Which is: 1
/home/fanya/openr/openr/nl/tests/NetlinkSocketTest.cpp:1064: Failure
Value of: CompareNextHops(nexthops1, rt2)
Actual: false
Expected: true
/home/fanya/openr/openr/nl/tests/NetlinkSocketTest.cpp:1065: Failure
Value of: CompareNextHops(nexthops2, rt3)
Actual: false
Expected: true
/home/fanya/openr/openr/nl/tests/NetlinkSocketTest.cpp:1089: Failure
Expected: 2
To be equal to: count
Which is: 0
/home/fanya/openr/openr/nl/tests/NetlinkSocketTest.cpp:1097: Failure
Expected: 2
To be equal to: rt4.getNextHops().size()
Which is: 1
/home/fanya/openr/openr/nl/tests/NetlinkSocketTest.cpp:1098: Failure
Value of: CompareNextHops(nexthops2, rt4)
Actual: false
Expected: true
[ FAILED ] NetlinkSocketFixture.MultiPathTest (130 ms)
[ RUN ] NetlinkSocketFixture.DeleteNonExistingRouteTest
E1107 02:19:24.090428 3069 NetlinkSocket.cpp:492] Trying to delete non-existing prefix fc00:cafe:3::4/128
[ OK ] NetlinkSocketFixture.DeleteNonExistingRouteTest (107 ms)
[ RUN ] NetlinkSocketFixture.MultiProtocolUnicastTest
/home/fanya/openr/openr/nl/tests/NetlinkSocketTest.cpp:1320: Failure
Expected: 2
To be equal to: rt7.getNextHops().size()
Which is: 1
/home/fanya/openr/openr/nl/tests/NetlinkSocketTest.cpp:1321: Failure
Value of: CompareNextHops(nextHopsV6, rt7)
Actual: false
Expected: true
/home/fanya/openr/openr/nl/tests/NetlinkSocketTest.cpp:1358: Failure
Expected: 4
To be equal to: count
Which is: 3
[ FAILED ] NetlinkSocketFixture.MultiProtocolUnicastTest (212 ms)
[ RUN ] NetlinkSocketFixture.MutiProtocolMulticastRouteTest
[ OK ] NetlinkSocketFixture.MutiProtocolMulticastRouteTest (76 ms)
[ RUN ] NetlinkSocketFixture.MultiProtocolSyncUnicastRouteTest
/home/fanya/openr/openr/nl/tests/NetlinkSocketTest.cpp:1577: Failure
Expected: 2
To be equal to: rt1.getNextHops().size()
Which is: 1
/home/fanya/openr/openr/nl/tests/NetlinkSocketTest.cpp:1579: Failure
Value of: CompareNextHops(nextHops1V6, rt1)
Actual: false
Expected: true
/home/fanya/openr/openr/nl/tests/NetlinkSocketTest.cpp:1610: Failure
Expected: 4
To be equal to: count
Which is: 5
/home/fanya/openr/openr/nl/tests/NetlinkSocketTest.cpp:1631: Failure
Expected: 2
To be equal to: rt6.getNextHops().size()
Which is: 1
/home/fanya/openr/openr/nl/tests/NetlinkSocketTest.cpp:1633: Failure
Value of: CompareNextHops(nextHops2V6, rt6)
Actual: false
Expected: true
/home/fanya/openr/openr/nl/tests/NetlinkSocketTest.cpp:1670: Failure
Expected: 2
To be equal to: rt9.getNextHops().size()
Which is: 1
/home/fanya/openr/openr/nl/tests/NetlinkSocketTest.cpp:1671: Failure
Value of: CompareNextHops(nextHops2V6, rt9)
Actual: false
Expected: true
/home/fanya/openr/openr/nl/tests/NetlinkSocketTest.cpp:1683: Failure
Expected: 1
To be equal to: count
Which is: 2
E1107 02:19:24.633577 3083 NetlinkSocket.cpp:492] Trying to delete non-existing prefix fc00:cafe:3::3/128
/home/fanya/openr/openr/nl/tests/NetlinkSocketTest.cpp:1721: Failure
Expected: 0
To be equal to: count
Which is: 1
[ FAILED ] NetlinkSocketFixture.MultiProtocolSyncUnicastRouteTest (284 ms)
[ RUN ] NetlinkSocketFixture.MultiProtocolSyncLinkRouteTest
[ OK ] NetlinkSocketFixture.MultiProtocolSyncLinkRouteTest (132 ms)
[ RUN ] NetlinkSocketFixture.IfNametoIfIndexTest
[ OK ] NetlinkSocketFixture.IfNametoIfIndexTest (95 ms)
[ RUN ] NetlinkSocketFixture.AddDelIfAddressBaseTest
[ OK ] NetlinkSocketFixture.AddDelIfAddressBaseTest (112 ms)
[ RUN ] NetlinkSocketFixture.AddDelDuplicatedIfAddressTest
[ OK ] NetlinkSocketFixture.AddDelDuplicatedIfAddressTest (84 ms)
[ RUN ] NetlinkSocketFixture.AddressSyncTest
[ OK ] NetlinkSocketFixture.AddressSyncTest (216 ms)
[ RUN ] NetlinkSocketFixture.AddressFlushTest
[ OK ] NetlinkSocketFixture.AddressFlushTest (105 ms)
[ RUN ] NetlinkSocketFixture.GetAddrsTest
[ OK ] NetlinkSocketFixture.GetAddrsTest (164 ms)
[----------] 24 tests from NetlinkSocketFixture (3245 ms total)

[----------] Global test environment tear-down
[==========] 24 tests from 1 test case ran. (3245 ms total)
[ PASSED ] 18 tests.
[ FAILED ] 6 tests, listed below:
[ FAILED ] NetlinkSocketFixture.UpdateRouteTest
[ FAILED ] NetlinkSocketFixture.UpdateMultiRouteTest
[ FAILED ] NetlinkSocketFixture.SyncRouteTest
[ FAILED ] NetlinkSocketFixture.MultiPathTest
[ FAILED ] NetlinkSocketFixture.MultiProtocolUnicastTest
[ FAILED ] NetlinkSocketFixture.MultiProtocolSyncUnicastRouteTest

6 FAILED TESTS

About the question 2, the ports 60099 and 60100 are alreay bind by Open/R,
tcp6 0 0 :::60099 :::* LISTEN 4574/openr
tcp6 0 0 ::1:48066 ::1:60099 ESTABLISHED 4574/openr
tcp6 0 0 ::1:60099 ::1:48066 ESTABLISHED 4574/openr
tcp6 0 0 :::60100 :::* LISTEN 4574/openr
tcp6 0 0 ::1:49384 ::1:60100 ESTABLISHED 4574/openr
tcp6 0 0 ::1:60100 ::1:49384 ESTABLISHED 4574/openr

What I mean excatly is that first I run openr with "sudo ./run_openr.sh", then I want to enable linux platform agent so I tap "sudo ./platform_linux", then the core dumped occures. Do I need to active "platform_linux" or not? The tech-support log is attached below.

breeze tech-support

-------- openr config file --------

Missing Config File

-------- openr runtime params --------

4574 openr --alloc_prefix_len=128 --assume_drained=false --config_store_filepath=/tmp/aq_persistent_config_store.bin --decision_debounce_max_ms=250 --decision_debounce_min_ms=10 --domain=openr --dryrun=false --enable_subnet_validation=true --enable_fib_sync=true --enable_health_checker=false --enable_legacy_flooding=true --enable_lfa=false --enable_netlink_fib_handler=true --enable_netlink_system_handler=true --enable_old_decision_module=false --enable_perf_measurement=true --enable_prefix_alloc=false --enable_rtt_metric=true --enable_segment_routing=false --enable_spark=true --enable_v4=true --enable_watchdog=true --fib_handler_port=60100 --health_checker_ping_interval_s=3 --ifname_prefix=terra,nic1,nic2,ens33,ens38 --iface_regex_exclude= --iface_regex_include= --ip_tos=192 --key_prefix_filters= --kvstore_flood_msg_per_sec=0 --kvstore_flood_msg_burst_size=0 --kvstore_zmq_hwm=65536 --link_flap_initial_backoff_ms=1000 --link_flap_max_backoff_ms=60000 --loopback_iface=lo --memory_limit_mb=300 --minloglevel=0 --node_name=yf --override_loopback_addr=false --prefixes= --redistribute_ifaces=lo1 --seed_prefix= --set_loopback_address=false --spark_fastinit_keepalive_time_ms=100 --spark_hold_time_s=30 --spark_keepalive_time_s=3 --set_leaf_node=false --static_prefix_alloc=false --logbufsecs=0 --logtostderr --max_log_size=1 --v=1

-------- openr version --------

Current Version : 20180307
Lowest Supported Version : 20180307

-------- openr build information --------

Build Information
Built by:
Built on:
Built at:
Build path:
Package Name:
Package Version:
Package Release:
Build Revision:
Build Upstream Revision:
Build Platform:
Build Rule: (, , )

-------- openr config --------

Missing Prefix Allocator config
Missing Link Monitor config

Prefix Type

-------- breeze prefixmgr view --------

Type Prefix

-------- breeze lm links --------

== Node Overload: NO ==

Interface Status Metric Override Addresses

ens33 Up 192.168.75.128
fe80::c9ae:9c2a:1bea:6865
ens38 Up 192.168.199.128
fe80::50a5:60f4:54ea:bc27

-------- breeze kvstore peers --------

== yf's peers ==

-------- breeze kvstore nodes --------

Node V6-Loopback V4-Loopback

yf N/A N/A

-------- breeze kvstore adj --------

-------- breeze kvstore prefixes --------

yf
Prefix Type

-------- breeze kvstore keys --ttl --------

== Available keys in KvStore ==

Key Originator Ver Hash TTL - Ver

prefix:yf yf 1 +1103d60e82d7d1a3 0:04:11.406000 - 9

-------- breeze decision validate --------

Decision is in sync with KvStore if nothing shows up

-------- breeze decision routes --------

== Routes for yf ==

No routes found.

-------- breeze fib validate --------

PASS
Decision and Openr-Fib routing table match
PASS
Openr-Fib and FibAgent routing table match
PASS
Route validation successful

-------- breeze fib routes-computed --------

== Routes for yf ==

No routes found.

-------- breeze fib routes-installed --------

== yf's FIB routes by client 786 ==

-------- breeze perf fib --------

-------- breeze monitor counters --------

== yf's counters ==

decision.prefix_db_update.count.0 : 1
decision.prefix_db_update.count.3600 : 1
decision.prefix_db_update.count.60 : 0
decision.prefix_db_update.count.600 : 0
fib.num_routes : 0
fib.process_interface_db.count.0 : 1
fib.process_interface_db.count.3600 : 1
fib.process_interface_db.count.60 : 0
fib.process_interface_db.count.600 : 0
fib.require_routedb_sync : 0
fib.sync_fib_calls.count.0 : 1
fib.sync_fib_calls.count.3600 : 1
fib.sync_fib_calls.count.60 : 0
fib.sync_fib_calls.count.600 : 0
kvstore.cmd_key_dump_.count.0 : 1
kvstore.cmd_key_dump_.count.3600 : 1
kvstore.cmd_key_dump_.count.60 : 0
kvstore.cmd_key_dump_.count.600 : 0
kvstore.cmd_key_get.count.0 : 11
kvstore.cmd_key_get.count.3600 : 11
kvstore.cmd_key_get.count.60 : 1
kvstore.cmd_key_get.count.600 : 9
kvstore.cmd_key_set.count.0 : 10
kvstore.cmd_key_set.count.3600 : 10
kvstore.cmd_key_set.count.60 : 1
kvstore.cmd_key_set.count.600 : 8
kvstore.num_keys : 1
kvstore.num_peers : 0
kvstore.pending_full_sync : 0
kvstore.received_key_vals.sum.0 : 10
kvstore.received_key_vals.sum.3600 : 10
kvstore.received_key_vals.sum.60 : 1
kvstore.received_key_vals.sum.600 : 8
kvstore.received_publications.count.0 : 10
kvstore.received_publications.count.3600 : 10
kvstore.received_publications.count.60 : 1
kvstore.received_publications.count.600 : 8
kvstore.updated_key_vals.sum.0 : 10
kvstore.updated_key_vals.sum.3600 : 10
kvstore.updated_key_vals.sum.60 : 1
kvstore.updated_key_vals.sum.600 : 8
link_monitor.adjacencies : 0
link_monitor.advertise_links.sum.0 : 1
link_monitor.advertise_links.sum.3600 : 1
link_monitor.advertise_links.sum.60 : 0
link_monitor.advertise_links.sum.600 : 0
process.cpu.pct : 0.009998
process.memory.rss : 7.24132e+07
spark.hello_packet_sent.sum.0 : 488
spark.hello_packet_sent.sum.3600 : 488
spark.hello_packet_sent.sum.60 : 38
spark.hello_packet_sent.sum.600 : 390
spark.my_seq_num : 489
spark.num_adjacent_neighbors : 0
spark.num_tracked_interfaces : 2
spark.num_tracked_neighbors : 0
spark.pending_timers : 3

-------- breeze monitor logs --------

time 02:22:36 2018-11-07
domain openr
node_name yf
event IFACE_UPDATE
entity Fib

Saif Hasan · Answer 3 · Thu Nov 22 2018 08:52:38 GMT+0800 (China Standard Time)

Pardon me for delayed response. Not sure why netlink tests are failing on your system. I would encourage you to poke around test code. We use netlink library 3.4.0+ with patch https://github.com/facebook/openr/blob/master/build/fix-route-obj-attr-list.patch

Open/R by default have platform_linux server built into it and enabled by default. You can run in either mode as follows

openr --enable_netlink_fib_handler=true ....<other-args>

or

openr --enable_netlink_fib_handler=false .... <other-args>
./platform-linux