XMPP codec should be modular
zinid opened this issue · comments
XMPP codec should allow third-party developers to create their own submodules and plug them into xmpp_codec.
This is implemented in 3976a85
Here is a brief description for developers about how they could use it:
- Clone xmpp library somewhere:
$ git clone https://github.com/processone/xmpp
- Compile it:
$ make
- Add new elements to
specs/xmpp_codec.spec
with propermodule
attribute set - Recompile the codec:
$ make spec
- The generator will create new file(s) in
src
directory. Those files are your submodules. Grab them to the working directory of your project. - The generator also will generate new records (with specs) inside
include/xmpp_codec.hrl
. Grab those too in your project (either put them in your *.hrl files or in your *.erl files directly). - In runtime, call to xmpp:register_codec/1 in order to register your submodule within xmpp_codec.
Done!
Not very clear, eh? OK, here is an example.
Let's say we want to add submodule for the following element:
<foo x='1' xmlns='ns:foo'/>
Clone and compile the repo, open specs/xmpp_codec.spec
and put the following spec somewhere inside it (the order doesn't matter):
-xml(foo,
#elem{name = <<"foo">>,
xmlns = <<"ns:foo">>,
module = foo,
result = {foo, '$x'},
attrs = [#attr{name = <<"x">>}]}).
Now type make spec
in order to recompile the specification. New file will be created inside src
directory:
$ git status
...
Untracked files:
...
src/foo.erl
Copy this foo.erl
into your working directory.
Now, consult the recompiled xmpp_codec.hrl
:
$ git diff include/xmpp_codec.hrl
...
+-record(foo, {x = <<>> :: binary()}).
+-type foo() :: #foo{}.
+
...
Put this record and type definition somewhere in your files.
Now, you need to register your submodule (foo
) during startup. This should look something like this:
-module(mod_foo).
...
start(Blah, ...) ->
...
xmpp:register_codec(foo),
...
Note, that you need to call xmpp:register_codec/1
again, if you have your submodule (foo
) recompiled and reloaded in runtime.
Also, you may want to unregister the submodule during shutdown procedure, although, strictly speaking, this is not required:
stop(...) ->
xmpp:unregister_codec(foo)
Now you have everything done to use your new "subcodec".
Have fun ;)
Great ... thanks will try it out soon
tag in rebar is
{deps, [{fast_xml, ".*", {git, "https://github.com/processone/fast_xml", "23f60e2942113fcf5d87531c081ef47329c343ef"}
should it be some version tag ? @zinid
No, we tag it during release, later.
This is example of script to build custom XMPP codec module for this library:
#!/bin/bash
rm -rf tmp
mkdir tmp
cd tmp
git clone https://github.com/processone/xmpp.git &&
cd xmpp &&
git checkout 1.1.9 &&
cat ../../specs/sec_history_xmpp.spec >> specs/xmpp_codec.spec &&
cp include/xmpp_codec.hrl include/xmpp_code_old.hrl &&
make all &&
make spec &&
cp src/sec_history_xmpp.erl ../../src/sec_history_xmpp.erl &&
echo -e "%% This file was generated automatically by compile_xmpp_spec.sh script\n\n" > ../../include/sec_xmpp_codec.hrl &&
diff -n include/xmpp_code_old.hrl include/xmpp_codec.hrl | grep -v "^\a" | grep -v "() |" >> ../../include/sec_xmpp_codec.hrl &&
cd ../..
rm -rf tmp
The project structure:
├── include
│ └── sec_xmpp_codec.hrl
├── Makefile
├── script
│ └── compile_xmpp_specs.sh
├── specs
│ └── sec_history_xmpp.spec
└── src
└── sec_history_xmpp.erl
spec folder should contains specification additional tags.
script/compile_xmpp_specs.sh --- the script described above.
Script will generate 2 files:
./src/sec_history_xmpp -- source code for codec.
./include/sec_xmpp_codec.hrl --- header file with description of records and types.
Makefile can be like this:
.PHONY: spec
spec:
script/compile_xmpp_specs.sh
So just run:
make spec
to build your own specification for xmpp.
After that you can use this module as:
xmpp:register_codec(sec_history_xmpp.erl)
By the way. All these flow are big crutches. XMPP library MUST allow build CUSTOM MODULES for all developers.
MORE CAPS
how to make custom iq spec? for example i have custom iq handler with query ns "NS-CUSTOM-MODULE"
@ltAldoRaine you can show an example of your IQ payload, so I can write some spec you can start with.
<iq type=“get”><query xmlns=“jabber:iq:conversation”><start>0</start><limit>10</limit></query></iq>
I did everything as you described. after restarting ejabberd there was error which was throwed by xmpp:register_codec()
7:47:40.261 [critical] Problem starting the module mod_conversation for host ****
options: []
error: undef
[{conversation,module_info,[md5],[]},
{xmpp_codec,register_module,2,[{file,"src/xmpp_codec.erl"},{line,121}]},
{mod_conversation,start,2,[{file,"src/mod_conversation.erl"},{line,42}]},
{gen_mod,start_module,4,[{file,"src/gen_mod.erl"},{line,200}]},
{lists,foreach,2,[{file,"lists.erl"},{line,1338}]},
{gen_mod,start_link,0,[{file,"src/gen_mod.erl"},{line,79}]},
{supervisor,do_start_child,2,[{file,"supervisor.erl"},{line,365}]},
{supervisor,start_children,3,[{file,"supervisor.erl"},{line,348}]}]
17:47:40.261 [critical] ejabberd initialization was aborted because a module start failed.
Problem starting the module mod_conversation for host ****
options: []
error: undef
[{conversation,module_info,[md5],[]},
{xmpp_codec,register_module,2,[{file,"src/xmpp_codec.erl"},{line,121
Crash dump is being written to: /usr/local/var/log/ejabberd/erl_crash_20180205-174736.dump...done
[os_mon] memory supervisor port (memsup): Erlang has closed
- What Erlang version are you using?
- Did you compile your spec so you get conversations.erl as an output?
- Did you compile conversations.erl? Does Erlang see conversations.beam (use code:which/1 to check)?
- Post your spec somewhere so I can check its correctness.
- Erlang/OTP 20 [erts-9.2]
- yes it was generated in cloned xmpp src folder so i moved it to ejabberd deps/xmpp/src folder
- i think i missed that part.. where should i compile it ? in erlang ebin folder or ejabberd ebin folder
You should compile it inside your project. Then you just set CONTRIB_MODULES_CONF_DIR
option in ejabberdctl.cfg which points to the beam files of your project.
You can of course compile it within ejabberd project (just put your ERL file inside src
directory and run make
).
Thanks ! I will try second option..my modules(not xmpp-codec) is also in ejabberd src folder
It worked ! :) now ejabberd is receiving my iq stanza.
I was experimenting with this example and still can't make it work.
Using ejabberd-18.12 in Mac, used OS X package to install ejabberd.
So, the steps performed:
- Clone XMPP and make
- Added spec below in
specs/xmpp_codec.spec
andmake spec
-xml(foo,
#elem{name = <<"foo">>,
xmlns = <<"ns:foo">>,
module = foo,
result = {foo, '$x'},
attrs = [#attr{name = <<"x">>}]}).
- Moved
src/foo.erl
to working directory wheremod_foo.erl
resides.foo.erl
below:
%% Created automatically by XML generator (fxml_gen.erl)
%% Source: xmpp_codec.spec
-module(foo).
-compile(export_all).
do_decode(<<"foo">>, <<"ns:foo">>, El, Opts) ->
decode_foo(<<"ns:foo">>, Opts, El);
do_decode(Name, <<>>, _, _) ->
erlang:error({xmpp_codec, {missing_tag_xmlns, Name}});
do_decode(Name, XMLNS, _, _) ->
erlang:error({xmpp_codec, {unknown_tag, Name, XMLNS}}).
tags() -> [{<<"foo">>, <<"ns:foo">>}].
do_encode({foo, _} = Foo, TopXMLNS) ->
encode_foo(Foo, TopXMLNS).
do_get_name({foo, _}) -> <<"foo">>.
do_get_ns({foo, _}) -> <<"ns:foo">>.
pp(foo, 1) -> [x];
pp(_, _) -> no.
records() -> [{foo, 1}].
decode_foo(__TopXMLNS, __Opts,
{xmlel, <<"foo">>, _attrs, _els}) ->
X = decode_foo_attrs(__TopXMLNS, _attrs, undefined),
{foo, X}.
decode_foo_attrs(__TopXMLNS, [{<<"x">>, _val} | _attrs],
_X) ->
decode_foo_attrs(__TopXMLNS, _attrs, _val);
decode_foo_attrs(__TopXMLNS, [_ | _attrs], X) ->
decode_foo_attrs(__TopXMLNS, _attrs, X);
decode_foo_attrs(__TopXMLNS, [], X) ->
decode_foo_attr_x(__TopXMLNS, X).
encode_foo({foo, X}, __TopXMLNS) ->
__NewTopXMLNS =
xmpp_codec:choose_top_xmlns(<<"ns:foo">>, [],
__TopXMLNS),
_els = [],
_attrs = encode_foo_attr_x(X,
xmpp_codec:enc_xmlns_attrs(__NewTopXMLNS,
__TopXMLNS)),
{xmlel, <<"foo">>, _attrs, _els}.
decode_foo_attr_x(__TopXMLNS, undefined) -> <<>>;
decode_foo_attr_x(__TopXMLNS, _val) -> _val.
encode_foo_attr_x(<<>>, _acc) -> _acc;
encode_foo_attr_x(_val, _acc) ->
[{<<"x">>, _val} | _acc].
- Compared the
include/xmpp_codec.hrl
before vs. after and added the difference inmod_foo.erl
-record(foo, {x = <<>> :: binary()}).
-type foo() :: #foo{}.
- So
mod_foo.erl
is like below (just for output IQ to log):
-module(mod_foo).
-include("logger.hrl").
-behaviour(gen_mod).
-export([start/2, stop/1, process_local_iq/1]).
-record(foo, {x = <<>> :: binary()}).
-type foo() :: #foo{}.
start(Host, Opts) ->
xmpp:register_codec(foo),
gen_iq_handler:add_iq_handler(ejabberd_local, Host, <<"ns:foo">>, ?MODULE, process_local_iq),
ok.
stop(Host) ->
xmpp:unregister_codec(foo),
gen_iq_handler:remove_iq_handler(ejabberd_local, Host, <<"ns:foo">>),
ok.
process_local_iq(IQ) ->
?INFO_MSG("IQ received:~p", [IQ]),
ignore.
- Compiled and loaded
mod_foo
module byejabberdctl module_install mod_foo
, with some warnings that I've ignored.
~/.ejabberd-modules/sources/mod_foo/src/foo.erl:5: Warning: export_all flag enabled - all functions will be exported
~/.ejabberd-modules/sources/mod_foo/src/mod_foo.erl:3: Warning: undefined callback function depends/2 (behaviour 'gen_mod')
~/.ejabberd-modules/sources/mod_foo/src/mod_foo.erl:3: Warning: undefined callback function mod_options/1 (behaviour 'gen_mod')
~/.ejabberd-modules/sources/mod_foo/src/mod_foo.erl:7: Warning: type foo() is unused
~/.ejabberd-modules/sources/mod_foo/src/mod_foo.erl:9: Warning: variable 'Opts' is unused
- Sending XML:
<iq type="set" id="1234" to="localhost">
<foo x='1' xmlns='ns:foo'/>
</iq>
- Receiving XML:
<error code="503" type="cancel">
<service-unavailable xmlns="urn:ietf:params:xml:ns:xmpp-stanzas" />
<text xmlns="urn:ietf:params:xml:ns:xmpp-stanzas" lang="en">No module is handling this query</text>
</error>
What am I missing here?
Do I need to replace the xmpp_codec.hrl
and xmpp_codec.beam
with the new compiled ones?
Would appreciate any suggestions please.
What am I missing here?
The module foo.erl
is generated automatically by the xmpp_codec, you can find it inside src
directory of xmpp sources. You MUST NOT modify it. And your own module should have a different name, obviously, like mod_foo.erl.
@zinid, truly appreciate the prompt response.
I have updated the steps above with as much details as possible according to your suggestion and still can't make it work.
I wish there were an easier way to do this and proper documentation of the steps with examples.
Would appreciate any suggestion please.
Compiled and loaded
What do you mean by "loaded"? Did you put it into ejabberd.yml?
Compiled and loaded
What do you mean by "loaded"? Did you put it into ejabberd.yml?
No, I meant compiled and installed the module mod_foo
using ejabberdctl module_install mod_foo
command.
https://docs.ejabberd.im/admin/guide/managing/#ejabberdctl-commands
Actually that was the problem, updated ejabberd.yml
and restarted server, working fine now!
Wondering why the ejabberdctl module_install mod_foo
didn't get the module started in the first place?
Although I see ejabberdctl module_upgrade mod_foo
is working fine for runtime compilation and reloading the module after adding mod_foo
in ejabberd.yml
and restarting server once.
Anyway, thank you very much @zinid, truly appreciate.
Wondering why the ejabberdctl module_install mod_foo didn't get the module started in the first place?
Because a module typically requires some start options.