XMPP codec should be modular

Question

XMPP codec should be modular

zinid opened this issue 8 years ago · comments

XMPP codec should allow third-party developers to create their own submodules and plug them into xmpp_codec.
This is implemented in 3976a85
Here is a brief description for developers about how they could use it:

Clone xmpp library somewhere:
$ git clone https://github.com/processone/xmpp
Compile it:
$ make
Add new elements to specs/xmpp_codec.spec with proper module attribute set
Recompile the codec:
$ make spec
The generator will create new file(s) in src directory. Those files are your submodules. Grab them to the working directory of your project.
The generator also will generate new records (with specs) inside include/xmpp_codec.hrl. Grab those too in your project (either put them in your *.hrl files or in your *.erl files directly).
In runtime, call to xmpp:register_codec/1 in order to register your submodule within xmpp_codec.

Done!
Not very clear, eh? OK, here is an example.

Let's say we want to add submodule for the following element:

<foo x='1' xmlns='ns:foo'/>

Clone and compile the repo, open specs/xmpp_codec.spec and put the following spec somewhere inside it (the order doesn't matter):

-xml(foo,
     #elem{name = <<"foo">>,
           xmlns = <<"ns:foo">>,
           module = foo,
           result = {foo, '$x'},
           attrs = [#attr{name = <<"x">>}]}).

Now type make spec in order to recompile the specification. New file will be created inside src directory:

$ git status
...
Untracked files:
   ...
   src/foo.erl

Copy this foo.erl into your working directory.
Now, consult the recompiled xmpp_codec.hrl:

$ git diff include/xmpp_codec.hrl
...
+-record(foo, {x = <<>> :: binary()}).
+-type foo() :: #foo{}.
+
...

Put this record and type definition somewhere in your files.
Now, you need to register your submodule (foo) during startup. This should look something like this:

-module(mod_foo).
...
start(Blah, ...) ->
    ...
    xmpp:register_codec(foo),
    ...

Note, that you need to call xmpp:register_codec/1 again, if you have your submodule (foo) recompiled and reloaded in runtime.
Also, you may want to unregister the submodule during shutdown procedure, although, strictly speaking, this is not required:

stop(...) ->
    xmpp:unregister_codec(foo)

Now you have everything done to use your new "subcodec".
Have fun ;)

Evgeny Khramtsov commented 7 years ago

MORE CAPS

jeEt · Answer 1 · Wed Feb 08 2017 14:49:19 GMT+0800 (China Standard Time)

Great ... thanks will try it out soon

jeEt · Answer 2 · Wed Feb 08 2017 14:51:22 GMT+0800 (China Standard Time)

tag in rebar is
{deps, [{fast_xml, ".*", {git, "https://github.com/processone/fast_xml", "23f60e2942113fcf5d87531c081ef47329c343ef"}

should it be some version tag ? @zinid

Evgeny Khramtsov · Answer 3 · Wed Feb 08 2017 15:03:33 GMT+0800 (China Standard Time)

No, we tag it during release, later.

Sergey Loguntsov · Answer 4 · Wed May 17 2017 02:18:10 GMT+0800 (China Standard Time)

This is example of script to build custom XMPP codec module for this library:

#!/bin/bash
rm -rf tmp
mkdir tmp
cd tmp
git clone https://github.com/processone/xmpp.git &&
cd xmpp &&
git checkout 1.1.9 &&
cat ../../specs/sec_history_xmpp.spec >> specs/xmpp_codec.spec &&
cp include/xmpp_codec.hrl include/xmpp_code_old.hrl &&
make all &&
make spec &&
cp src/sec_history_xmpp.erl ../../src/sec_history_xmpp.erl &&
echo -e "%% This file was generated automatically by compile_xmpp_spec.sh script\n\n" > ../../include/sec_xmpp_codec.hrl &&
diff -n include/xmpp_code_old.hrl include/xmpp_codec.hrl | grep -v "^\a" | grep -v "() |" >> ../../include/sec_xmpp_codec.hrl &&
cd ../..
rm -rf tmp

The project structure:

├── include
│   └── sec_xmpp_codec.hrl
├── Makefile
├── script
│   └── compile_xmpp_specs.sh
├── specs
│   └── sec_history_xmpp.spec
└── src
    └── sec_history_xmpp.erl

spec folder should contains specification additional tags.
script/compile_xmpp_specs.sh --- the script described above.
Script will generate 2 files:
./src/sec_history_xmpp -- source code for codec.
./include/sec_xmpp_codec.hrl --- header file with description of records and types.

Makefile can be like this:

.PHONY: spec
spec:
	script/compile_xmpp_specs.sh

So just run:
make spec
to build your own specification for xmpp.
After that you can use this module as:
xmpp:register_codec(sec_history_xmpp.erl)

Sergey Loguntsov · Answer 5 · Wed May 17 2017 02:20:38 GMT+0800 (China Standard Time)

By the way. All these flow are big crutches. XMPP library MUST allow build CUSTOM MODULES for all developers.

Beka Gelashvili · Answer 6 · Mon Feb 05 2018 19:30:59 GMT+0800 (China Standard Time)

how to make custom iq spec? for example i have custom iq handler with query ns "NS-CUSTOM-MODULE"

Evgeny Khramtsov · Answer 7 · Tue Feb 06 2018 04:11:51 GMT+0800 (China Standard Time)

@ltAldoRaine you can show an example of your IQ payload, so I can write some spec you can start with.

Beka Gelashvili · Answer 8 · Tue Feb 06 2018 04:21:44 GMT+0800 (China Standard Time)

<iq type=“get”><query xmlns=“jabber:iq:conversation”><start>0</start><limit>10</limit></query></iq>

I did everything as you described. after restarting ejabberd there was error which was throwed by xmpp:register_codec()

7:47:40.261 [critical] Problem starting the module mod_conversation for host ****
 options: []
 error: undef
[{conversation,module_info,[md5],[]},
 {xmpp_codec,register_module,2,[{file,"src/xmpp_codec.erl"},{line,121}]},
 {mod_conversation,start,2,[{file,"src/mod_conversation.erl"},{line,42}]},
 {gen_mod,start_module,4,[{file,"src/gen_mod.erl"},{line,200}]},
 {lists,foreach,2,[{file,"lists.erl"},{line,1338}]},
 {gen_mod,start_link,0,[{file,"src/gen_mod.erl"},{line,79}]},
 {supervisor,do_start_child,2,[{file,"supervisor.erl"},{line,365}]},
 {supervisor,start_children,3,[{file,"supervisor.erl"},{line,348}]}]
17:47:40.261 [critical] ejabberd initialization was aborted because a module start failed.
Problem starting the module mod_conversation for host ****
 options: []
 error: undef
[{conversation,module_info,[md5],[]},
 {xmpp_codec,register_module,2,[{file,"src/xmpp_codec.erl"},{line,121

Crash dump is being written to: /usr/local/var/log/ejabberd/erl_crash_20180205-174736.dump...done
[os_mon] memory supervisor port (memsup): Erlang has closed

Evgeny Khramtsov · Answer 9 · Tue Feb 06 2018 20:03:34 GMT+0800 (China Standard Time)

What Erlang version are you using?
Did you compile your spec so you get conversations.erl as an output?
Did you compile conversations.erl? Does Erlang see conversations.beam (use code:which/1 to check)?
Post your spec somewhere so I can check its correctness.

Beka Gelashvili · Answer 10 · Tue Feb 06 2018 20:18:31 GMT+0800 (China Standard Time)

Erlang/OTP 20 [erts-9.2]
yes it was generated in cloned xmpp src folder so i moved it to ejabberd deps/xmpp/src folder
i think i missed that part.. where should i compile it ? in erlang ebin folder or ejabberd ebin folder

Evgeny Khramtsov · Answer 11 · Tue Feb 06 2018 20:23:33 GMT+0800 (China Standard Time)

You should compile it inside your project. Then you just set CONTRIB_MODULES_CONF_DIR option in ejabberdctl.cfg which points to the beam files of your project.

Evgeny Khramtsov · Answer 12 · Tue Feb 06 2018 20:25:00 GMT+0800 (China Standard Time)

You can of course compile it within ejabberd project (just put your ERL file inside src directory and run make).

Beka Gelashvili · Answer 13 · Tue Feb 06 2018 20:26:39 GMT+0800 (China Standard Time)

Thanks ! I will try second option..my modules(not xmpp-codec) is also in ejabberd src folder

Beka Gelashvili · Answer 14 · Wed Feb 07 2018 01:16:57 GMT+0800 (China Standard Time)

It worked ! :) now ejabberd is receiving my iq stanza.

rasel800 · Answer 15 · Thu Jan 31 2019 23:09:55 GMT+0800 (China Standard Time)

I was experimenting with this example and still can't make it work.
Using ejabberd-18.12 in Mac, used OS X package to install ejabberd.

So, the steps performed:

Clone XMPP and make
Added spec below in specs/xmpp_codec.spec and make spec

-xml(foo,
     #elem{name = <<"foo">>,
           xmlns = <<"ns:foo">>,
           module = foo,
           result = {foo, '$x'},
           attrs = [#attr{name = <<"x">>}]}).

Moved src/foo.erl to working directory where mod_foo.erl resides. foo.erl below:

%% Created automatically by XML generator (fxml_gen.erl)
%% Source: xmpp_codec.spec

-module(foo).

-compile(export_all).

do_decode(<<"foo">>, <<"ns:foo">>, El, Opts) ->
    decode_foo(<<"ns:foo">>, Opts, El);
do_decode(Name, <<>>, _, _) ->
    erlang:error({xmpp_codec, {missing_tag_xmlns, Name}});
do_decode(Name, XMLNS, _, _) ->
    erlang:error({xmpp_codec, {unknown_tag, Name, XMLNS}}).

tags() -> [{<<"foo">>, <<"ns:foo">>}].

do_encode({foo, _} = Foo, TopXMLNS) ->
    encode_foo(Foo, TopXMLNS).

do_get_name({foo, _}) -> <<"foo">>.

do_get_ns({foo, _}) -> <<"ns:foo">>.

pp(foo, 1) -> [x];
pp(_, _) -> no.

records() -> [{foo, 1}].

decode_foo(__TopXMLNS, __Opts,
	   {xmlel, <<"foo">>, _attrs, _els}) ->
    X = decode_foo_attrs(__TopXMLNS, _attrs, undefined),
    {foo, X}.

decode_foo_attrs(__TopXMLNS, [{<<"x">>, _val} | _attrs],
		 _X) ->
    decode_foo_attrs(__TopXMLNS, _attrs, _val);
decode_foo_attrs(__TopXMLNS, [_ | _attrs], X) ->
    decode_foo_attrs(__TopXMLNS, _attrs, X);
decode_foo_attrs(__TopXMLNS, [], X) ->
    decode_foo_attr_x(__TopXMLNS, X).

encode_foo({foo, X}, __TopXMLNS) ->
    __NewTopXMLNS =
	xmpp_codec:choose_top_xmlns(<<"ns:foo">>, [],
				    __TopXMLNS),
    _els = [],
    _attrs = encode_foo_attr_x(X,
			       xmpp_codec:enc_xmlns_attrs(__NewTopXMLNS,
							  __TopXMLNS)),
    {xmlel, <<"foo">>, _attrs, _els}.

decode_foo_attr_x(__TopXMLNS, undefined) -> <<>>;
decode_foo_attr_x(__TopXMLNS, _val) -> _val.

encode_foo_attr_x(<<>>, _acc) -> _acc;
encode_foo_attr_x(_val, _acc) ->
    [{<<"x">>, _val} | _acc].

Compared the include/xmpp_codec.hrl before vs. after and added the difference in mod_foo.erl

-record(foo, {x = <<>> :: binary()}).
-type foo() :: #foo{}.

So mod_foo.erl is like below (just for output IQ to log):

-module(mod_foo).
-include("logger.hrl").
-behaviour(gen_mod).
-export([start/2, stop/1, process_local_iq/1]).
-record(foo, {x = <<>> :: binary()}).
-type foo() :: #foo{}.
start(Host, Opts) ->
    xmpp:register_codec(foo),
    gen_iq_handler:add_iq_handler(ejabberd_local, Host, <<"ns:foo">>, ?MODULE, process_local_iq),
    ok.
stop(Host) ->
    xmpp:unregister_codec(foo),
    gen_iq_handler:remove_iq_handler(ejabberd_local, Host, <<"ns:foo">>),
    ok.
process_local_iq(IQ) ->
    ?INFO_MSG("IQ received:~p", [IQ]),
    ignore.

Compiled and loaded mod_foo module by ejabberdctl module_install mod_foo, with some warnings that I've ignored.

~/.ejabberd-modules/sources/mod_foo/src/foo.erl:5: Warning: export_all flag enabled - all functions will be exported
~/.ejabberd-modules/sources/mod_foo/src/mod_foo.erl:3: Warning: undefined callback function depends/2 (behaviour 'gen_mod')
~/.ejabberd-modules/sources/mod_foo/src/mod_foo.erl:3: Warning: undefined callback function mod_options/1 (behaviour 'gen_mod')
~/.ejabberd-modules/sources/mod_foo/src/mod_foo.erl:7: Warning: type foo() is unused
~/.ejabberd-modules/sources/mod_foo/src/mod_foo.erl:9: Warning: variable 'Opts' is unused

Sending XML:

<iq type="set" id="1234" to="localhost">
    <foo x='1' xmlns='ns:foo'/>
</iq>

Receiving XML:

<error code="503" type="cancel">
        <service-unavailable xmlns="urn:ietf:params:xml:ns:xmpp-stanzas" />
        <text xmlns="urn:ietf:params:xml:ns:xmpp-stanzas" lang="en">No module is handling this query</text>
</error>

What am I missing here?
Do I need to replace the xmpp_codec.hrl and xmpp_codec.beam with the new compiled ones?

Would appreciate any suggestions please.

Evgeny Khramtsov · Answer 16 · Fri Feb 01 2019 00:31:56 GMT+0800 (China Standard Time)

What am I missing here?

The module foo.erl is generated automatically by the xmpp_codec, you can find it inside src directory of xmpp sources. You MUST NOT modify it. And your own module should have a different name, obviously, like mod_foo.erl.

rasel800 · Answer 17 · Fri Feb 01 2019 12:29:55 GMT+0800 (China Standard Time)

@zinid, truly appreciate the prompt response.
I have updated the steps above with as much details as possible according to your suggestion and still can't make it work.

I wish there were an easier way to do this and proper documentation of the steps with examples.
Would appreciate any suggestion please.

Evgeny Khramtsov · Answer 18 · Fri Feb 01 2019 15:43:52 GMT+0800 (China Standard Time)

Compiled and loaded

What do you mean by "loaded"? Did you put it into ejabberd.yml?

rasel800 · Answer 19 · Fri Feb 01 2019 17:00:53 GMT+0800 (China Standard Time)

Compiled and loaded

What do you mean by "loaded"? Did you put it into ejabberd.yml?

No, I meant compiled and installed the module mod_foo using ejabberdctl module_install mod_foo command.
https://docs.ejabberd.im/admin/guide/managing/#ejabberdctl-commands

Actually that was the problem, updated ejabberd.yml and restarted server, working fine now!

Wondering why the ejabberdctl module_install mod_foo didn't get the module started in the first place?
Although I see ejabberdctl module_upgrade mod_foo is working fine for runtime compilation and reloading the module after adding mod_foo in ejabberd.yml and restarting server once.

Anyway, thank you very much @zinid, truly appreciate.

Evgeny Khramtsov · Answer 20 · Fri Feb 01 2019 19:45:55 GMT+0800 (China Standard Time)

Wondering why the ejabberdctl module_install mod_foo didn't get the module started in the first place?

Because a module typically requires some start options.

Zedonboy · Answer 21 · Sun Aug 25 2019 01:57:15 GMT+0800 (China Standard Time)

If you are still having this issue, check.. here