Constantly throwing errors
shifters67 opened this issue · comments
Hi Derek,
using this via your horde library. I am getting errors every 20 seconds when running - two messages each time about discarding messages. Are these important or is there a way i can switch them off as they are generating lots in the logs.
This is being run on my home cluster of 36 raspberry pi's with horde. Horde seems happy in clustering the registry and the supervisors but delta_CRDT has started throwing errors. It never did this previously that i have noticed. I am using the horde version you pointed me to on git not the hex version.
Thanks
Tom
PS sorry for the long error trail below.
11:15:28.237 [error] Discarding message {delta,{<0.206.0>,<0.206.0>,#{'struct'=>'Elixir.DeltaCrdt.CausalDotMap',causal_context=>#{'struct'=>'Elixir.DeltaCrdt.CausalContext',dots=>#{'struct'=>'Elixir.MapSet',map=>#{<8860>{<820>{[{723330950,0}],[{259073618,0}]},[{886054732,0}],[{90994585,0}],[{212397740,0}]},{<908>{[{224541245,0}],[{354815956,0}],[{501767019,0}]},[{644668926,0}],[{884763753,0}],[{680172344,0}]},<1020>{<4080>{[{608052787,0}],[{87023067,0}]},<2804>{[{992209432,0}],[{85815313,0}],[{107656322,0}]}},<10a1>{[{761337020,0}],[{57681452,0}],[{312110877,0}],[{561210601,0}]},<8244>{[{149594868,0}],[{529351108,0}],[{231569336,0}],<2210>{[{477061609,0}],[{617103734,0}],[{270793753,0}]}},<8d00>{[{220838726,0}],[{893132601,0}],[{678561457,0}],[{79767290,0}]},<6502>{[{560938324,0}],[{823104459,0}],<81>{[{768249285,0}],[{453473496,0}]},[{471979944,0}],[{465765109,0}]},<961>{[{154501068,0}],[{57131885,0}],[{832826349,0}],[{574622686,0}],[{853838025,0}]},<68>{[{231644276,0}],[{909871415,0}],[{584834511,0}]},[{978167393,0}],<2509>{[{283646442,0}],[{368901591,0}],[{980630187,0}],[{695829055,0}],[{421117301,0}]},<6093>{<420>{[{261846720,0}],[{236507057,0}]},[{151955626,0}],<8010>{[{256799718,0}],[{915189138,0}]},[{730824296,0}],[{879988380,0}],<3000>{[{865384709,0}],[{254696481,0}]}},<8101>{{[{918232091,0}],[{162637692,0}]},[{292687244,0}],[{728559197,0}]},<132>{[{764991069,0}],<82>{[{881771793,0}],[{64290749,0}]},[{470913543,0}],[{516674924,0}]},<6810>{[{1309607,0}],[{168459593,0}],[{176508547,0}],[{811735860,0}]},<1900>{[{128620661,0}],[{505929147,0}],[{665961286,0}]}},version=>2},maxima=>#{[212397740|0],<4831>{[730824296|0],[224541245|0],[680172344|0],[107656322|0],[695829055|0]},<8285>{[915189138|0],[678561457|0],[893132601|0],<1000>{<420>{[236507057|0],[884763753|0]}},[879988380|0]},<41>{[978167393|0],[881771793|0]},<8402>{[259073618|0],[561210601|0],<8040>{[254696481|0],[477061609|0]}},<820>{[270793753|0],[761337020|0]},<908>{[256799718|0],[421117301|0],[832826349|0]},<8304>{<108>{[918232091|0],[617103734|0]},[231644276|0],[465765109|0],[723330950|0]},<100>{<200>{<4040>{[176508547|0],[980630187|0]}}},<4>{{[220838726|0],[168459593|0]}},<622>{[768249285|0],[453473496|0],[64290749|0],[853838025|0]},{<1140>{[644668926|0],[823104459|0],[728559197|0]},<1020>{[283646442|0],[354815956|0]},<4200>{[162637692|0],[811735860|0]},[529351108|0],[471979944|0],{[231569336|0],[470913543|0]},[151955626|0],[261846720|0]},{<4021>{[865384709|0],[368901591|0],[574622686|0]},[312110877|0],[128620661|0],[886054732|0],[57131885|0],[501767019|0]},<8419>{[57681452|0],[1309607|0],[584834511|0],<10a>{[87023067|0],[154501068|0],[560938324|0]},[992209432|0]},<9e00>{<80>{<600>{[764991069|0],[149594868|0]}},<440>{[90994585|0],[665961286|0]},[292687244|0],[79767290|0],[608052787|0]},<806>{[909871415|0],<402>{[85815313|0],[505929147|0]},[516674924|0]}}},keys=>#{'struct'=>'Elixir.MapSet',map=>#{<86b>{[<<"E7IwHHDqhp6vvpRUM6MlMA==">>],[<<"7KBMQSaU5KWiya9f55tqfw==">>],[<<"JncEdzW3VrscpQ2JFTxvjw==">>],<8200>{[<<"omUBFLuljKPTngAuCrdcgQ==">>],[<<"KYi1jL9KgYYJmVhUg7VEEA==">>]},[<<"L4J/xHaWgMc5GP/jH+Z7Cg==">>],[<<"vIOI2DtF+dFSxkUb6LJjzQ==">>]},<62>{[<<"N4D4LXoPQtXZQ/LcDhYPAg==">>],[<<"xI0J2/7T7doP1S/g/STsOA==">>],[<<"VaAiyjQsJA40klMf/eUXAQ==">>]},<13dc>{[<<"vs6XYFGMO4c0xWXOHoranQ==">>],[<<"EG6Dn8s2BkP8hs4vSHhhog==">>],[<<"GH5qGpg3YtbvAzst4L+S7Q==">>],[<<"aegaOhJy+xYveh4QiTV9Zw==">>],[<<"+giKoQjVw1PZ+qckZutBGg==">>],[<<"HoaL35929EgB4G8VV9qGcg==">>],[<<"QCl2IRQvsoUjCkUgtv97Bw==">>],[<<"Gc7BF/gsPWlDxROSKpWGFQ==">>]},<124>{[<<"lWfnOUfDuihsDvx8Ptg0Cw==">>],[<<"kHSSYH8OBR90cGsUDgomgQ==">>],[<<"WXYHwgNkIo1zEmmhg1XxuA==">>]},<8398>{[<<"i3KHrvu/RYcRa+F8bMq/kA==">>],<4100>{[<<"KVmpcyqCWb6sjK2bBtNVkg==">>],[<<"5NXXgn+6QsHlpdHFhczkyQ==">>]},[<<"AL2KAUkHUJIAioHEHNylWg==">>],[<<"IAXuuAwVgOb6Hs5d66Ez7g==">>],[<<"z4pTEg4uZAgOHrfkggYtUQ==">>],[<<"2YymVa5kjrdxoX670G71ug==">>]},<81>{[<<"QsBS2ijaCj6/Me1atTAjcw==">>],[<<"s0UN5TTaPEjIuuClNWYvbQ==">>]},<4004>{[<<"Bw3eewoJ391zBrw3jdeBtQ==">>],[<<"ct8GOYAqUw4yoZdnraGcpg==">>]},<200c>{<9040>{[<<"L7Zf1Bctzu+cJ++DMuFlEA==">>],[<<"tc4SkitIUX7i3plC0c2EdQ==">>],[<<"bPi3efdC6COk0MdlZCAYHw==">>]},[<<"ihgQIxasjv3Sz16Fo3+Spw==">>],[<<"Ayil53y8WMhg9VpG769onw==">>]},<3020>{[<<"X5+ezgknmNKlzQeYm7mNsw==">>],[<<"paOMJws/ADcpR0T0tI4uSA==">>],<6>{[<<"gfPJsQIhFvPQtRh8pHcpYg==">>],[<<"8JR30qjVqfQDhXjsRag2nA==">>]}},{[<<"RLCPyVMsvvppsHESrLmB8w==">>],[<<"MogCb6XFLvfYSgUe4t2b0g==">>],[<<"iZzEyKzfmhHEU11SWQI6rA==">>],[<<"FLTu2ntWXclkPdQ9lE5oPQ==">>],[<<"19GGsvcmcFAbIzfTDJiodQ==">>],[<<"UANIPQ3h+1wQ6bO9WeitXw==">>],[<<"JwvHI0Ag7F+r1vlXASxe2g==">>],[<<"w6u/5Bj3AObv/mZBwbzOGw==">>]},{[<<"kHmw2fwtBkFQfbDstJQw0A==">>],[<<"FOj542MooTBHj1RC8gq7wg==">>],[<<"9rTi4B1JHGcvhEWKlzl2zA==">>],[<<"pugBOnOyEdZ3g6rjlkPzTQ==">>],[<<"bXH1+JGYkBQFVxLncuI1Uw==">>],[<<"aRYNumxyw17M2zeaH5NkhQ==">>]},<41ab>{<810>{[<<"2KkSf8K+nDiYvnCuz2SxOQ==">>],[<<"fbUHyegwmW9s4DwsI8ULMg==">>]},[<<"X16Tox5Eb2VDB46ZhtnI+A==">>],[<<"iBzucoTTtteDOYLroe9IUA==">>],[<<"t5eVQRT17UEaaIWpSWHf6w==">>],[<<"NJ7t+md6d03vQA0aiboMmw==">>],[<<"N2O5RUXRX3iepElrj4bkNQ==">>],[<<"tDnpkgddLS8hs4Eo//RP+g==">>]},<2281>{<120>{[<<"u/A1k4eVPXp2IGC650elrg==">>],[<<"NmgLfaeL9oXtNf/0xtxa8A==">>]},[<<"blls+PxiO5+pLtK2+J83KQ==">>],[<<"/VnIAo0aXHie97K3aKg8IQ==">>],[<<"MWtaw57inYi8HkqFECtPtw==">>]},<8200>{[<<"bkw5LG/wZITD2APxXdX1/g==">>],[<<"ZzpJWrIJLyL5tn+K/6ylCg==">>]},<138>{[<<"/lrbghCe8w1yEOSYAwwo1g==">>],[<<"lwuNVhRGi1MWO2nNYWontw==">>],[<<"dNmIwNZ1zH4B2dHzMn4G1A==">>],[<<"YJeav1KsYVbWQ4hh7HtxEg==">>]},[<<"aapzz1MFCJiejHustgK5hw==">>]},version=>2},state=>#{<86b>{[<<"E7IwHHDqhp6vvpRUM6MlMA==">>|#{'struct'=>'Elixir.DeltaCrdt.CausalDotMap',causal_context=>nil,keys=>#{'struct'=>'Elixir.MapSet',map=>#{{{<16436.256.0>,<16436.260.0>},1534759592748297642}=>[]},version=>2},state=>#{{{<16436.256.0>,<16436.260.0>},1534759592748297642}=>#{'struct'=>'Elixir.DeltaCrdt.CausalDotSet',causal_context=>nil,state=>#{'struct'=>'Elixir.MapSet',map=>#{{87023067,0}=>[]},version=>2}}}}],[<<"7KBMQSaU5KWiya9f55tqfw==">>|#{'struct'=>'Elixir.DeltaCrdt.CausalDotMap',causal_context=>nil,keys=>#{'struct'=>'Elixir.MapSet',map=>#{{{<16418.349.0>,<16418.353.0>},1534759694496164977}=>[]},version=>2},state=>#{{{<16418.349.0>,<16418.353.0>},1534759694496164977}=>#{'struct'=>'Elixir.DeltaCrdt.CausalDotSet',causal_context=>nil,state=>#{'struct'=>'Elixir.MapSet',map=>#{{992209432,0}=>[]},version=>2}}}}],[<<"JncEdzW3VrscpQ2JFTxvjw==">>|#{'struct'=>'Elixir.DeltaCrdt.CausalDotMap',causal_context=>nil,keys=>#{'struct'=>'Elixir.MapSet',map=>#{{{<16407.349.0>,<16407.355.0>},1534759700289044705}=>[]},version=>2},state=>#{{{<16407.349.0>,<16407.355.0>},1534759700289044705}=>#{'struct'=>'Elixir.DeltaCrdt.CausalDotSet',causal_context=>nil,state=>#{'struct'=>'Elixir.MapSet',map=>#{{79767290,0}=>[]},version=>2}}}}],<8200>{[<<"omUBFLuljKPTngAuCrdcgQ==">>|#{'struct'=>'Elixir.DeltaCrdt.CausalDotMap',causal_context=>nil,keys=>#{'struct'=>'Elixir.MapSet',map=>#{{{<16426.187.0>,<16426.191.0>},1534759134590351593}=>[]},version=>2},state=>#{{{<16426.187.0>,<16426.191.0>},1534759134590351593}=>#{'struct'=>'Elixir.DeltaCrdt.CausalDotSet',causal_context=>nil,state=>#{'struct'=>'Elixir.MapSet',map=>#{{453473496,0}=>[]},version=>2}}}}],[<<"KYi1jL9KgYYJmVhUg7VEEA==">>|#{'struct'=>'Elixir.DeltaCrdt.CausalDotMap',causal_context=>nil,keys=>#{'struct'=>'Elixir.MapSet',map=>#{{{<16434.187.0>,<16434.191.0>},1534759584572105562}=>[]},version=>2},state=>#{{{<16434.187.0>,<16434.191.0>},1534759584572105562}=>#{'struct'=>'Elixir.DeltaCrdt.CausalDotSet',causal_context=>nil,state=>#{'struct'=>'Elixir.MapSet',map=>#{{283646442,0}=>[]},version=>2}}}}]},[<<"L4J/xHaWgMc5GP/jH+Z7Cg==">>|#{'struct'=>'Elixir.DeltaCrdt.CausalDotMap',causal_context=>nil,keys=>#{'struct'=>'Elixir.MapSet',map=>#{{{<16436.187.0>,<16436.191.0>},1534759132671404719}=>[]},version=>2},state=>#{{{<16436.187.0>,<16436.191.0>},1534759132671404719}=>#{'struct'=>'Elixir.DeltaCrdt.CausalDotSet',causal_context=>nil,stat (truncated)
11:15:48.245 [error] Discarding message {delta,{<0.206.0>,<0.249.0>,#{'struct'=>'Elixir.DeltaCrdt.CausalDotMap',causal_context=>#{'struct'=>'Elixir.DeltaCrdt.CausalContext',dots=>#{'struct'=>'Elixir.MapSet',map=>#{<8860>{<820>{[{723330950,0}],[{259073618,0}]},[{886054732,0}],[{90994585,0}],[{212397740,0}]},{<908>{[{224541245,0}],[{354815956,0}],[{501767019,0}]},[{644668926,0}],[{884763753,0}],[{680172344,0}]},<1020>{<4080>{[{608052787,0}],[{87023067,0}]},<2804>{[{992209432,0}],[{85815313,0}],[{107656322,0}]}},<10a1>{[{761337020,0}],[{57681452,0}],[{312110877,0}],[{561210601,0}]},<8244>{[{149594868,0}],[{529351108,0}],[{231569336,0}],<2210>{[{477061609,0}],[{617103734,0}],[{270793753,0}]}},<8d00>{[{220838726,0}],[{893132601,0}],[{678561457,0}],[{79767290,0}]},<6502>{[{560938324,0}],[{823104459,0}],<81>{[{768249285,0}],[{453473496,0}]},[{471979944,0}],[{465765109,0}]},<961>{[{154501068,0}],[{57131885,0}],[{832826349,0}],[{574622686,0}],[{853838025,0}]},<68>{[{231644276,0}],[{909871415,0}],[{584834511,0}]},[{978167393,0}],<2509>{[{283646442,0}],[{368901591,0}],[{980630187,0}],[{695829055,0}],[{421117301,0}]},<6093>{<420>{[{261846720,0}],[{236507057,0}]},[{151955626,0}],<8010>{[{256799718,0}],[{915189138,0}]},[{730824296,0}],[{879988380,0}],<3000>{[{865384709,0}],[{254696481,0}]}},<8101>{{[{918232091,0}],[{162637692,0}]},[{292687244,0}],[{728559197,0}]},<132>{[{764991069,0}],<82>{[{881771793,0}],[{64290749,0}]},[{470913543,0}],[{516674924,0}]},<6810>{[{1309607,0}],[{168459593,0}],[{176508547,0}],[{811735860,0}]},<1900>{[{128620661,0}],[{505929147,0}],[{665961286,0}]}},version=>2},maxima=>#{[212397740|0],<4831>{[730824296|0],[224541245|0],[680172344|0],[107656322|0],[695829055|0]},<8285>{[915189138|0],[678561457|0],[893132601|0],<1000>{<420>{[236507057|0],[884763753|0]}},[879988380|0]},<41>{[978167393|0],[881771793|0]},<8402>{[259073618|0],[561210601|0],<8040>{[254696481|0],[477061609|0]}},<820>{[270793753|0],[761337020|0]},<908>{[256799718|0],[421117301|0],[832826349|0]},<8304>{<108>{[918232091|0],[617103734|0]},[231644276|0],[465765109|0],[723330950|0]},<100>{<200>{<4040>{[176508547|0],[980630187|0]}}},<4>{{[220838726|0],[168459593|0]}},<622>{[768249285|0],[453473496|0],[64290749|0],[853838025|0]},{<1140>{[644668926|0],[823104459|0],[728559197|0]},<1020>{[283646442|0],[354815956|0]},<4200>{[162637692|0],[811735860|0]},[529351108|0],[471979944|0],{[231569336|0],[470913543|0]},[151955626|0],[261846720|0]},{<4021>{[865384709|0],[368901591|0],[574622686|0]},[312110877|0],[128620661|0],[886054732|0],[57131885|0],[501767019|0]},<8419>{[57681452|0],[1309607|0],[584834511|0],<10a>{[87023067|0],[154501068|0],[560938324|0]},[992209432|0]},<9e00>{<80>{<600>{[764991069|0],[149594868|0]}},<440>{[90994585|0],[665961286|0]},[292687244|0],[79767290|0],[608052787|0]},<806>{[909871415|0],<402>{[85815313|0],[505929147|0]},[516674924|0]}}},keys=>#{'struct'=>'Elixir.MapSet',map=>#{<86b>{[<<"E7IwHHDqhp6vvpRUM6MlMA==">>],[<<"7KBMQSaU5KWiya9f55tqfw==">>],[<<"JncEdzW3VrscpQ2JFTxvjw==">>],<8200>{[<<"omUBFLuljKPTngAuCrdcgQ==">>],[<<"KYi1jL9KgYYJmVhUg7VEEA==">>]},[<<"L4J/xHaWgMc5GP/jH+Z7Cg==">>],[<<"vIOI2DtF+dFSxkUb6LJjzQ==">>]},<62>{[<<"N4D4LXoPQtXZQ/LcDhYPAg==">>],[<<"xI0J2/7T7doP1S/g/STsOA==">>],[<<"VaAiyjQsJA40klMf/eUXAQ==">>]},<13dc>{[<<"vs6XYFGMO4c0xWXOHoranQ==">>],[<<"EG6Dn8s2BkP8hs4vSHhhog==">>],[<<"GH5qGpg3YtbvAzst4L+S7Q==">>],[<<"aegaOhJy+xYveh4QiTV9Zw==">>],[<<"+giKoQjVw1PZ+qckZutBGg==">>],[<<"HoaL35929EgB4G8VV9qGcg==">>],[<<"QCl2IRQvsoUjCkUgtv97Bw==">>],[<<"Gc7BF/gsPWlDxROSKpWGFQ==">>]},<124>{[<<"lWfnOUfDuihsDvx8Ptg0Cw==">>],[<<"kHSSYH8OBR90cGsUDgomgQ==">>],[<<"WXYHwgNkIo1zEmmhg1XxuA==">>]},<8398>{[<<"i3KHrvu/RYcRa+F8bMq/kA==">>],<4100>{[<<"KVmpcyqCWb6sjK2bBtNVkg==">>],[<<"5NXXgn+6QsHlpdHFhczkyQ==">>]},[<<"AL2KAUkHUJIAioHEHNylWg==">>],[<<"IAXuuAwVgOb6Hs5d66Ez7g==">>],[<<"z4pTEg4uZAgOHrfkggYtUQ==">>],[<<"2YymVa5kjrdxoX670G71ug==">>]},<81>{[<<"QsBS2ijaCj6/Me1atTAjcw==">>],[<<"s0UN5TTaPEjIuuClNWYvbQ==">>]},<4004>{[<<"Bw3eewoJ391zBrw3jdeBtQ==">>],[<<"ct8GOYAqUw4yoZdnraGcpg==">>]},<200c>{<9040>{[<<"L7Zf1Bctzu+cJ++DMuFlEA==">>],[<<"tc4SkitIUX7i3plC0c2EdQ==">>],[<<"bPi3efdC6COk0MdlZCAYHw==">>]},[<<"ihgQIxasjv3Sz16Fo3+Spw==">>],[<<"Ayil53y8WMhg9VpG769onw==">>]},<3020>{[<<"X5+ezgknmNKlzQeYm7mNsw==">>],[<<"paOMJws/ADcpR0T0tI4uSA==">>],<6>{[<<"gfPJsQIhFvPQtRh8pHcpYg==">>],[<<"8JR30qjVqfQDhXjsRag2nA==">>]}},{[<<"RLCPyVMsvvppsHESrLmB8w==">>],[<<"MogCb6XFLvfYSgUe4t2b0g==">>],[<<"iZzEyKzfmhHEU11SWQI6rA==">>],[<<"FLTu2ntWXclkPdQ9lE5oPQ==">>],[<<"19GGsvcmcFAbIzfTDJiodQ==">>],[<<"UANIPQ3h+1wQ6bO9WeitXw==">>],[<<"JwvHI0Ag7F+r1vlXASxe2g==">>],[<<"w6u/5Bj3AObv/mZBwbzOGw==">>]},{[<<"kHmw2fwtBkFQfbDstJQw0A==">>],[<<"FOj542MooTBHj1RC8gq7wg==">>],[<<"9rTi4B1JHGcvhEWKlzl2zA==">>],[<<"pugBOnOyEdZ3g6rjlkPzTQ==">>],[<<"bXH1+JGYkBQFVxLncuI1Uw==">>],[<<"aRYNumxyw17M2zeaH5NkhQ==">>]},<41ab>{<810>{[<<"2KkSf8K+nDiYvnCuz2SxOQ==">>],[<<"fbUHyegwmW9s4DwsI8ULMg==">>]},[<<"X16Tox5Eb2VDB46ZhtnI+A==">>],[<<"iBzucoTTtteDOYLroe9IUA==">>],[<<"t5eVQRT17UEaaIWpSWHf6w==">>],[<<"NJ7t+md6d03vQA0aiboMmw==">>],[<<"N2O5RUXRX3iepElrj4bkNQ==">>],[<<"tDnpkgddLS8hs4Eo//RP+g==">>]},<2281>{<120>{[<<"u/A1k4eVPXp2IGC650elrg==">>],[<<"NmgLfaeL9oXtNf/0xtxa8A==">>]},[<<"blls+PxiO5+pLtK2+J83KQ==">>],[<<"/VnIAo0aXHie97K3aKg8IQ==">>],[<<"MWtaw57inYi8HkqFECtPtw==">>]},<8200>{[<<"bkw5LG/wZITD2APxXdX1/g==">>],[<<"ZzpJWrIJLyL5tn+K/6ylCg==">>]},<138>{[<<"/lrbghCe8w1yEOSYAwwo1g==">>],[<<"lwuNVhRGi1MWO2nNYWontw==">>],[<<"dNmIwNZ1zH4B2dHzMn4G1A==">>],[<<"YJeav1KsYVbWQ4hh7HtxEg==">>]},[<<"aapzz1MFCJiejHustgK5hw==">>]},version=>2},state=>#{<86b>{[<<"E7IwHHDqhp6vvpRUM6MlMA==">>|#{'struct'=>'Elixir.DeltaCrdt.CausalDotMap',causal_context=>nil,keys=>#{'struct'=>'Elixir.MapSet',map=>#{{{<16436.256.0>,<16436.260.0>},1534759592748297642}=>[]},version=>2},state=>#{{{<16436.256.0>,<16436.260.0>},1534759592748297642}=>#{'struct'=>'Elixir.DeltaCrdt.CausalDotSet',causal_context=>nil,state=>#{'struct'=>'Elixir.MapSet',map=>#{{87023067,0}=>[]},version=>2}}}}],[<<"7KBMQSaU5KWiya9f55tqfw==">>|#{'struct'=>'Elixir.DeltaCrdt.CausalDotMap',causal_context=>nil,keys=>#{'struct'=>'Elixir.MapSet',map=>#{{{<16418.349.0>,<16418.353.0>},1534759694496164977}=>[]},version=>2},state=>#{{{<16418.349.0>,<16418.353.0>},1534759694496164977}=>#{'struct'=>'Elixir.DeltaCrdt.CausalDotSet',causal_context=>nil,state=>#{'struct'=>'Elixir.MapSet',map=>#{{992209432,0}=>[]},version=>2}}}}],[<<"JncEdzW3VrscpQ2JFTxvjw==">>|#{'struct'=>'Elixir.DeltaCrdt.CausalDotMap',causal_context=>nil,keys=>#{'struct'=>'Elixir.MapSet',map=>#{{{<16407.349.0>,<16407.355.0>},1534759700289044705}=>[]},version=>2},state=>#{{{<16407.349.0>,<16407.355.0>},1534759700289044705}=>#{'struct'=>'Elixir.DeltaCrdt.CausalDotSet',causal_context=>nil,state=>#{'struct'=>'Elixir.MapSet',map=>#{{79767290,0}=>[]},version=>2}}}}],<8200>{[<<"omUBFLuljKPTngAuCrdcgQ==">>|#{'struct'=>'Elixir.DeltaCrdt.CausalDotMap',causal_context=>nil,keys=>#{'struct'=>'Elixir.MapSet',map=>#{{{<16426.187.0>,<16426.191.0>},1534759134590351593}=>[]},version=>2},state=>#{{{<16426.187.0>,<16426.191.0>},1534759134590351593}=>#{'struct'=>'Elixir.DeltaCrdt.CausalDotSet',causal_context=>nil,state=>#{'struct'=>'Elixir.MapSet',map=>#{{453473496,0}=>[]},version=>2}}}}],[<<"KYi1jL9KgYYJmVhUg7VEEA==">>|#{'struct'=>'Elixir.DeltaCrdt.CausalDotMap',causal_context=>nil,keys=>#{'struct'=>'Elixir.MapSet',map=>#{{{<16434.187.0>,<16434.191.0>},1534759584572105562}=>[]},version=>2},state=>#{{{<16434.187.0>,<16434.191.0>},1534759584572105562}=>#{'struct'=>'Elixir.DeltaCrdt.CausalDotSet',causal_context=>nil,state=>#{'struct'=>'Elixir.MapSet',map=>#{{283646442,0}=>[]},version=>2}}}}]},[<<"L4J/xHaWgMc5GP/jH+Z7Cg==">>|#{'struct'=>'Elixir.DeltaCrdt.CausalDotMap',causal_context=>nil,keys=>#{'struct'=>'Elixir.MapSet',map=>#{{{<16436.187.0>,<16436.191.0>},1534759132671404719}=>[]},version=>2},state=>#{{{<16436.187.0>,<16436.191.0>},1534759132671404719}=>#{'struct'=>'Elixir.DeltaCrdt.CausalDotSet',causal_context=>nil,stat (truncated)
11:15:48.248 [error] Discarding message {delta,{<0.206.0>,<0.206.0>,#{'struct'=>'Elixir.DeltaCrdt.CausalDotMap',causal_context=>#{'struct'=>'Elixir.DeltaCrdt.CausalContext',dots=>#{'struct'=>'Elixir.MapSet',map=>#{<8860>{<820>{[{723330950,0}],[{259073618,0}]},[{886054732,0}],[{90994585,0}],[{212397740,0}]},{<908>{[{224541245,0}],[{354815956,0}],[{501767019,0}]},[{644668926,0}],[{884763753,0}],[{680172344,0}]},<1020>{<4080>{[{608052787,0}],[{87023067,0}]},<2804>{[{992209432,0}],[{85815313,0}],[{107656322,0}]}},<10a1>{[{761337020,0}],[{57681452,0}],[{312110877,0}],[{561210601,0}]},<8244>{[{149594868,0}],[{529351108,0}],[{231569336,0}],<2210>{[{477061609,0}],[{617103734,0}],[{270793753,0}]}},<8d00>{[{220838726,0}],[{893132601,0}],[{678561457,0}],[{79767290,0}]},<6502>{[{560938324,0}],[{823104459,0}],<81>{[{768249285,0}],[{453473496,0}]},[{471979944,0}],[{465765109,0}]},<961>{[{154501068,0}],[{57131885,0}],[{832826349,0}],[{574622686,0}],[{853838025,0}]},<68>{[{231644276,0}],[{909871415,0}],[{584834511,0}]},[{978167393,0}],<2509>{[{283646442,0}],[{368901591,0}],[{980630187,0}],[{695829055,0}],[{421117301,0}]},<6093>{<420>{[{261846720,0}],[{236507057,0}]},[{151955626,0}],<8010>{[{256799718,0}],[{915189138,0}]},[{730824296,0}],[{879988380,0}],<3000>{[{865384709,0}],[{254696481,0}]}},<8101>{{[{918232091,0}],[{162637692,0}]},[{292687244,0}],[{728559197,0}]},<132>{[{764991069,0}],<82>{[{881771793,0}],[{64290749,0}]},[{470913543,0}],[{516674924,0}]},<6810>{[{1309607,0}],[{168459593,0}],[{176508547,0}],[{811735860,0}]},<1900>{[{128620661,0}],[{505929147,0}],[{665961286,0}]}},version=>2},maxima=>#{[212397740|0],<4831>{[730824296|0],[224541245|0],[680172344|0],[107656322|0],[695829055|0]},<8285>{[915189138|0],[678561457|0],[893132601|0],<1000>{<420>{[236507057|0],[884763753|0]}},[879988380|0]},<41>{[978167393|0],[881771793|0]},<8402>{[259073618|0],[561210601|0],<8040>{[254696481|0],[477061609|0]}},<820>{[270793753|0],[761337020|0]},<908>{[256799718|0],[421117301|0],[832826349|0]},<8304>{<108>{[918232091|0],[617103734|0]},[231644276|0],[465765109|0],[723330950|0]},<100>{<200>{<4040>{[176508547|0],[980630187|0]}}},<4>{{[220838726|0],[168459593|0]}},<622>{[768249285|0],[453473496|0],[64290749|0],[853838025|0]},{<1140>{[644668926|0],[823104459|0],[728559197|0]},<1020>{[283646442|0],[354815956|0]},<4200>{[162637692|0],[811735860|0]},[529351108|0],[471979944|0],{[231569336|0],[470913543|0]},[151955626|0],[261846720|0]},{<4021>{[865384709|0],[368901591|0],[574622686|0]},[312110877|0],[128620661|0],[886054732|0],[57131885|0],[501767019|0]},<8419>{[57681452|0],[1309607|0],[584834511|0],<10a>{[87023067|0],[154501068|0],[560938324|0]},[992209432|0]},<9e00>{<80>{<600>{[764991069|0],[149594868|0]}},<440>{[90994585|0],[665961286|0]},[292687244|0],[79767290|0],[608052787|0]},<806>{[909871415|0],<402>{[85815313|0],[505929147|0]},[516674924|0]}}},keys=>#{'struct'=>'Elixir.MapSet',map=>#{<86b>{[<<"E7IwHHDqhp6vvpRUM6MlMA==">>],[<<"7KBMQSaU5KWiya9f55tqfw==">>],[<<"JncEdzW3VrscpQ2JFTxvjw==">>],<8200>{[<<"omUBFLuljKPTngAuCrdcgQ==">>],[<<"KYi1jL9KgYYJmVhUg7VEEA==">>]},[<<"L4J/xHaWgMc5GP/jH+Z7Cg==">>],[<<"vIOI2DtF+dFSxkUb6LJjzQ==">>]},<62>{[<<"N4D4LXoPQtXZQ/LcDhYPAg==">>],[<<"xI0J2/7T7doP1S/g/STsOA==">>],[<<"VaAiyjQsJA40klMf/eUXAQ==">>]},<13dc>{[<<"vs6XYFGMO4c0xWXOHoranQ==">>],[<<"EG6Dn8s2BkP8hs4vSHhhog==">>],[<<"GH5qGpg3YtbvAzst4L+S7Q==">>],[<<"aegaOhJy+xYveh4QiTV9Zw==">>],[<<"+giKoQjVw1PZ+qckZutBGg==">>],[<<"HoaL35929EgB4G8VV9qGcg==">>],[<<"QCl2IRQvsoUjCkUgtv97Bw==">>],[<<"Gc7BF/gsPWlDxROSKpWGFQ==">>]},<124>{[<<"lWfnOUfDuihsDvx8Ptg0Cw==">>],[<<"kHSSYH8OBR90cGsUDgomgQ==">>],[<<"WXYHwgNkIo1zEmmhg1XxuA==">>]},<8398>{[<<"i3KHrvu/RYcRa+F8bMq/kA==">>],<4100>{[<<"KVmpcyqCWb6sjK2bBtNVkg==">>],[<<"5NXXgn+6QsHlpdHFhczkyQ==">>]},[<<"AL2KAUkHUJIAioHEHNylWg==">>],[<<"IAXuuAwVgOb6Hs5d66Ez7g==">>],[<<"z4pTEg4uZAgOHrfkggYtUQ==">>],[<<"2YymVa5kjrdxoX670G71ug==">>]},<81>{[<<"QsBS2ijaCj6/Me1atTAjcw==">>],[<<"s0UN5TTaPEjIuuClNWYvbQ==">>]},<4004>{[<<"Bw3eewoJ391zBrw3jdeBtQ==">>],[<<"ct8GOYAqUw4yoZdnraGcpg==">>]},<200c>{<9040>{[<<"L7Zf1Bctzu+cJ++DMuFlEA==">>],[<<"tc4SkitIUX7i3plC0c2EdQ==">>],[<<"bPi3efdC6COk0MdlZCAYHw==">>]},[<<"ihgQIxasjv3Sz16Fo3+Spw==">>],[<<"Ayil53y8WMhg9VpG769onw==">>]},<3020>{[<<"X5+ezgknmNKlzQeYm7mNsw==">>],[<<"paOMJws/ADcpR0T0tI4uSA==">>],<6>{[<<"gfPJsQIhFvPQtRh8pHcpYg==">>],[<<"8JR30qjVqfQDhXjsRag2nA==">>]}},{[<<"RLCPyVMsvvppsHESrLmB8w==">>],[<<"MogCb6XFLvfYSgUe4t2b0g==">>],[<<"iZzEyKzfmhHEU11SWQI6rA==">>],[<<"FLTu2ntWXclkPdQ9lE5oPQ==">>],[<<"19GGsvcmcFAbIzfTDJiodQ==">>],[<<"UANIPQ3h+1wQ6bO9WeitXw==">>],[<<"JwvHI0Ag7F+r1vlXASxe2g==">>],[<<"w6u/5Bj3AObv/mZBwbzOGw==">>]},{[<<"kHmw2fwtBkFQfbDstJQw0A==">>],[<<"FOj542MooTBHj1RC8gq7wg==">>],[<<"9rTi4B1JHGcvhEWKlzl2zA==">>],[<<"pugBOnOyEdZ3g6rjlkPzTQ==">>],[<<"bXH1+JGYkBQFVxLncuI1Uw==">>],[<<"aRYNumxyw17M2zeaH5NkhQ==">>]},<41ab>{<810>{[<<"2KkSf8K+nDiYvnCuz2SxOQ==">>],[<<"fbUHyegwmW9s4DwsI8ULMg==">>]},[<<"X16Tox5Eb2VDB46ZhtnI+A==">>],[<<"iBzucoTTtteDOYLroe9IUA==">>],[<<"t5eVQRT17UEaaIWpSWHf6w==">>],[<<"NJ7t+md6d03vQA0aiboMmw==">>],[<<"N2O5RUXRX3iepElrj4bkNQ==">>],[<<"tDnpkgddLS8hs4Eo//RP+g==">>]},<2281>{<120>{[<<"u/A1k4eVPXp2IGC650elrg==">>],[<<"NmgLfaeL9oXtNf/0xtxa8A==">>]},[<<"blls+PxiO5+pLtK2+J83KQ==">>],[<<"/VnIAo0aXHie97K3aKg8IQ==">>],[<<"MWtaw57inYi8HkqFECtPtw==">>]},<8200>{[<<"bkw5LG/wZITD2APxXdX1/g==">>],[<<"ZzpJWrIJLyL5tn+K/6ylCg==">>]},<138>{[<<"/lrbghCe8w1yEOSYAwwo1g==">>],[<<"lwuNVhRGi1MWO2nNYWontw==">>],[<<"dNmIwNZ1zH4B2dHzMn4G1A==">>],[<<"YJeav1KsYVbWQ4hh7HtxEg==">>]},[<<"aapzz1MFCJiejHustgK5hw==">>]},version=>2},state=>#{<86b>{[<<"E7IwHHDqhp6vvpRUM6MlMA==">>|#{'struct'=>'Elixir.DeltaCrdt.CausalDotMap',causal_context=>nil,keys=>#{'struct'=>'Elixir.MapSet',map=>#{{{<16436.256.0>,<16436.260.0>},1534759592748297642}=>[]},version=>2},state=>#{{{<16436.256.0>,<16436.260.0>},1534759592748297642}=>#{'struct'=>'Elixir.DeltaCrdt.CausalDotSet',causal_context=>nil,state=>#{'struct'=>'Elixir.MapSet',map=>#{{87023067,0}=>[]},version=>2}}}}],[<<"7KBMQSaU5KWiya9f55tqfw==">>|#{'struct'=>'Elixir.DeltaCrdt.CausalDotMap',causal_context=>nil,keys=>#{'struct'=>'Elixir.MapSet',map=>#{{{<16418.349.0>,<16418.353.0>},1534759694496164977}=>[]},version=>2},state=>#{{{<16418.349.0>,<16418.353.0>},1534759694496164977}=>#{'struct'=>'Elixir.DeltaCrdt.CausalDotSet',causal_context=>nil,state=>#{'struct'=>'Elixir.MapSet',map=>#{{992209432,0}=>[]},version=>2}}}}],[<<"JncEdzW3VrscpQ2JFTxvjw==">>|#{'struct'=>'Elixir.DeltaCrdt.CausalDotMap',causal_context=>nil,keys=>#{'struct'=>'Elixir.MapSet',map=>#{{{<16407.349.0>,<16407.355.0>},1534759700289044705}=>[]},version=>2},state=>#{{{<16407.349.0>,<16407.355.0>},1534759700289044705}=>#{'struct'=>'Elixir.DeltaCrdt.CausalDotSet',causal_context=>nil,state=>#{'struct'=>'Elixir.MapSet',map=>#{{79767290,0}=>[]},version=>2}}}}],<8200>{[<<"omUBFLuljKPTngAuCrdcgQ==">>|#{'struct'=>'Elixir.DeltaCrdt.CausalDotMap',causal_context=>nil,keys=>#{'struct'=>'Elixir.MapSet',map=>#{{{<16426.187.0>,<16426.191.0>},1534759134590351593}=>[]},version=>2},state=>#{{{<16426.187.0>,<16426.191.0>},1534759134590351593}=>#{'struct'=>'Elixir.DeltaCrdt.CausalDotSet',causal_context=>nil,state=>#{'struct'=>'Elixir.MapSet',map=>#{{453473496,0}=>[]},version=>2}}}}],[<<"KYi1jL9KgYYJmVhUg7VEEA==">>|#{'struct'=>'Elixir.DeltaCrdt.CausalDotMap',causal_context=>nil,keys=>#{'struct'=>'Elixir.MapSet',map=>#{{{<16434.187.0>,<16434.191.0>},1534759584572105562}=>[]},version=>2},state=>#{{{<16434.187.0>,<16434.191.0>},1534759584572105562}=>#{'struct'=>'Elixir.DeltaCrdt.CausalDotSet',causal_context=>nil,state=>#{'struct'=>'Elixir.MapSet',map=>#{{283646442,0}=>[]},version=>2}}}}]},[<<"L4J/xHaWgMc5GP/jH+Z7Cg==">>|#{'struct'=>'Elixir.DeltaCrdt.CausalDotMap',causal_context=>nil,keys=>#{'struct'=>'Elixir.MapSet',map=>#{{{<16436.187.0>,<16436.191.0>},1534759132671404719}=>[]},version=>2},state=>#{{{<16436.187.0>,<16436.191.0>},1534759132671404719}=>#{'struct'=>'Elixir.DeltaCrdt.CausalDotSet',causal_context=>nil,stat (truncated)
Hi Tom,
Thanks for the bug report. This happens when a node dies and then re-connects itself to the cluster with the same name. Right now there is no way in delta_crdt to "remove" a neighbour pid, so it keeps trying to send it messages, which are then discarded by the receiving node as being intended for a previous "version" of that node.
Basically, the "neighbours" pids can become stale.
I have a fix in mind, but I can't say when exactly I'll get around to it. In the meantime, I don't think there's a way to turn off these logs.
Cheers,
Derek
PS, if you wanted to take a crack at a PR to fix this issue then I could give you some tips on where to look.
Hi
I have been looking into this issue as it's one of the blockers for us using horde in production.
I have been playing about with the CausalCrdt module and allowing pids to be removed from state.neighbours but I am still seeing the error. If you have any pointers on what needs to be done here I can take a look into this.
Hi @swelham,
I've done some additional thinking on this problem. Can you test #4 for me to see if this could be a solution?
Please note, this will mean that any time a network partition has healed, the different pieces will have to be made aware of one another again, since we will no longer be trying to sync with those bits. This should be OK though I think since connecting / reconnecting was never a responsibility of Horde or DeltaCrdt.
Cheers,
Derek
Hi @derekkraan
Sure I will take a look and see if that solves the issue.
I agree that based on the current implementation the reconnection of partitioned nodes isn't expected. However this is maybe something to consider for the future of Horde or DeltaCrdt? If the state that's replicated across the cluster can be different per node I have lost the consistency guarantee that I am using a crdt for.
I have some other questions about my use case so I will write down some details in an issue in Horde.
I'll reopen this issue if this problem reappears in versions of delta_crdt higher than 0.1.11.