[meta] Split the core

Question

[meta] Split the core

Hywan opened this issue 9 years ago · comments

History

The core gathers several thin layers into a single library for performance reasons. Let me explain it shortly. Parsing time is not very costly compared to I/O: Accessing to a file, opening it and reading it. The core aimed at putting the minimum always needed codes into small files, in order to reduce this I/O time. Since recent PHP versions and recent I/O progresses (SSD, kernel, FS…), this is no longer necessary. We can then split the core.

Global idea

The core contains the following layers:

Hoa\Core\Consistency,
Hoa\Core\Data,
Hoa\Core\Event,
Hoa\Core\Exception,
Hoa\Core\Parameter,
Hoa\Core\Protocol.

We can split it into the following libraries:

They can live outside the core without any problem. So far, though, Hoa\Core\Data and Hoa\Core\Parameter will be kept as is.

Dealing with pre-loading

Hoa\Core\Consistency, Hoa\Core\Parameter and Hoa\Core\Protocol were de-facto pre-loaded with Hoa\Core\Core. This latter is automatically pre-loaded with the autoload.files Composer's property. Whatever the split, we have to ensure this behavior.

Case of `Hoa\Core\Consistency`

This layer contains:

An autoloader,
An xcallable implementation and,
Consistency helpers/tools accross PHP VMs (backport features to prior VMs, naming consistencies, feature consistencues etc.).

Only the latter is important at any runtime and will be loaded as a separated file with Composer. This will reduce the number of loaded code by approximately 60%.

Case of `Hoa\Core\Parameter`

~~Will not be extracted yet. Nothing to do.~~ (see bellow)

Case of `Hoa\Core\Protocol`

We need to register the hoa:// protocol at any runtime. We use it everywhere. It has never hurt anyone in the whole Hoa's history. So we can load the file that is responsible to register the stream wrapper with Composer. This will reduce the number of loaded code by approximately 50%.

Steps

Here are the following steps to follow (the order is important):

Next steps

~~Continue to split the core, see #97.~~

Here are the following steps to follow:

Update all `composer.json` files

Because we are dropping PHP5.4 too.

Ivan Enderlin commented 9 years ago

Done.

Dupont Yohann commented 9 years ago

👍

Ivan Enderlin commented 9 years ago

😅

Ivan Enderlin · Answer 1 · Tue Oct 13 2015 13:18:31 GMT+0800 (China Standard Time)

Thoughts @hoaproject/hoackers?

Dupont Yohann · Answer 2 · Tue Oct 13 2015 15:28:27 GMT+0800 (China Standard Time)

And what happens to Hoa\Core\Data ? If i understand, it is kept in Hoa\Core, why ?

Ivan Enderlin · Answer 3 · Tue Oct 13 2015 16:17:35 GMT+0800 (China Standard Time)

@Jir4 Because I am not sure about what to do with this one so far.

Raphaël Emourgeon · Answer 4 · Wed Oct 14 2015 00:59:16 GMT+0800 (China Standard Time)

That make sense, it would be very interesting to do some benchmarks before/after core splitting to check the real difference on performances (not saying that it'll decrease performances but numbers/data are interesting).

Dupont Yohann · Answer 5 · Wed Oct 14 2015 16:45:43 GMT+0800 (China Standard Time)

@osaris 👍

Ivan Enderlin · Answer 6 · Wed Oct 14 2015 17:56:06 GMT+0800 (China Standard Time)

@osaris Also, something else must be quantified but this is really hard: The psychological impact of having a “core” for a library. This is something really important. Several people stops at this. The goal is to really continue to split. Please, see #97.

Ivan Enderlin · Answer 7 · Wed Nov 11 2015 15:55:47 GMT+0800 (China Standard Time)

I started to split Hoa\Core\Exception and Hoa\Core\Event yesterday night.

Ivan Enderlin · Answer 8 · Fri Nov 13 2015 16:01:19 GMT+0800 (China Standard Time)

Hoa\Event is almost over.
Hoa\Exception is under splitting (the repositories have been created).

Ivan Enderlin · Answer 9 · Fri Nov 20 2015 14:23:18 GMT+0800 (China Standard Time)

I am starting to update libraries with Hoa\Event and Hoa\Exception.

Ivan Enderlin · Answer 10 · Fri Nov 20 2015 15:01:35 GMT+0800 (China Standard Time)

First run. I will continue today if I have time.

Metalaka · Answer 11 · Sat Nov 21 2015 02:14:05 GMT+0800 (China Standard Time)

@Hywan Sorry but most of your PR introduce a parse error: https://3v4l.org/vB5Dt

namespace Hoa\Exception {
    class Exception {}
}

namespace Hoa\Database {

    use Hoa\Exception;

    class Exception {}
    // Fatal error: Cannot declare class Hoa\Database\Exception because the name is already in use…
}

Metalaka · Answer 12 · Sat Nov 21 2015 02:17:31 GMT+0800 (China Standard Time)

Your should probably remove the use statement.

namespace Hoa\Database;

class Exception extends \Hoa\Exception {}

Ivan Enderlin · Answer 13 · Sun Nov 22 2015 00:22:23 GMT+0800 (China Standard Time)

Damned, I tested it on PHP7-nightly and it worked :-/.

Ivan Enderlin · Answer 14 · Mon Nov 23 2015 15:54:11 GMT+0800 (China Standard Time)

@Metalaka Fixed.

Metalaka · Answer 15 · Mon Nov 23 2015 20:36:23 GMT+0800 (China Standard Time)

Is it always possible to use flex entity ?

namespace Hoa\Database;

use Hoa\Exception as HoaException;

class Exception extends HoaException
{
}

Ivan Enderlin · Answer 16 · Mon Nov 23 2015 22:26:40 GMT+0800 (China Standard Time)

@Metalaka I think so.

Metalaka · Answer 17 · Tue Nov 24 2015 02:19:58 GMT+0800 (China Standard Time)

@Hywan flex entity seem to work at least for Hoa\Database.

Ivan Enderlin · Answer 18 · Tue Nov 24 2015 04:32:39 GMT+0800 (China Standard Time)

@Metalaka Hmm, yes, I shoud have update with the shortest name. Do you think this is required?

Ivan Enderlin · Answer 19 · Tue Nov 24 2015 04:49:40 GMT+0800 (China Standard Time)

(anyway, I started to update the code).

Metalaka · Answer 20 · Tue Nov 24 2015 04:55:21 GMT+0800 (China Standard Time)

The most important is do the same things everywhere (alias or alias+exception).

Ivan Enderlin · Answer 21 · Tue Nov 24 2015 05:01:07 GMT+0800 (China Standard Time)

@Metalaka We use the shortest everywhere. I started to move every library to Hoa\Event. I am updating this issue too.

Ivan Enderlin · Answer 22 · Tue Nov 24 2015 05:38:49 GMT+0800 (China Standard Time)

All libraries have been updated to Hoa\Event.

Ivan Enderlin · Answer 23 · Tue Nov 24 2015 05:53:54 GMT+0800 (China Standard Time)

All libraries have been updated to Hoa\Exception (final).

Ivan Enderlin · Answer 24 · Tue Nov 24 2015 06:36:23 GMT+0800 (China Standard Time)

Hoa\Consistency has been created. I will move the code tomorrow. Hope to start test suites tomorrow too.

Ivan Enderlin · Answer 25 · Thu Dec 03 2015 18:09:28 GMT+0800 (China Standard Time)

Hoa\Consistency is OK. Now I will start updating other libraries.

Ivan Enderlin · Answer 26 · Mon Dec 14 2015 06:30:34 GMT+0800 (China Standard Time)

All tests are green for Hoa\Protocol split. Now I have to write the test suite of Hoa\Protocol but the integration of the new library is fine!

Ivan Enderlin · Answer 27 · Wed Jan 06 2016 15:05:05 GMT+0800 (China Standard Time)

Everything is OK!
I am starting #97.

Ivan Enderlin · Answer 28 · Wed Jan 06 2016 15:19:40 GMT+0800 (China Standard Time)

@hoaproject/hoackers Please, I need your help!

What about Hoa\Zformat instead of Hoa\Parameter? Do you think this is better?
What to do with Hoa\Core? See the “Next steps” Section in #96 (comment) for the questions.

Thanks!

Dupont Yohann · Answer 29 · Wed Jan 06 2016 16:17:50 GMT+0800 (China Standard Time)

Why ZFormat ? Meaning ?
I think we should delete Hoa\Core, the Core name scares many users imo, they just wanna use a specific lib but the Core dependancy seems imply that the lib need many others to work

Ivan Enderlin · Answer 30 · Wed Jan 06 2016 16:28:42 GMT+0800 (China Standard Time)

@Jir4

zFormat because this is the format we use to express parameters.
Yes, we will “remove” it, but for legacy code, it must still be present as a repository. We can mark it as deprecated and keep the code as is. Thoughts?

Dupont Yohann · Answer 31 · Wed Jan 06 2016 16:33:58 GMT+0800 (China Standard Time)

Ok so 👍
So we keep it just to avoid a BC break and we will remove it when another BC break occurred ?

Ivan Enderlin · Answer 32 · Wed Jan 06 2016 16:38:04 GMT+0800 (China Standard Time)

@Jir4 We will remove it when we will no longer have any downloads. Until we have downloads, we must keep the repository live :-].

Matthieu Codron · Answer 33 · Wed Jan 06 2016 16:42:52 GMT+0800 (China Standard Time)

@Hywan the downside of this approach is that we will have downloads (at least for CI systems) so long as it will be available…
IMHO we should announce an estimated end-of-life

Ivan Enderlin · Answer 34 · Wed Jan 06 2016 16:55:13 GMT+0800 (China Standard Time)

@CircleCode Hmm… Every year, we can clean deprecated repositories? So far, we only have Hoa\String (see git.hoa-project.net). How long is enought?

Dupont Yohann · Answer 35 · Wed Jan 06 2016 17:09:38 GMT+0800 (China Standard Time)

Several month but not years, we just have to let the time to users to update their legacy code.

Ivan Enderlin · Answer 36 · Sat Jan 09 2016 15:05:33 GMT+0800 (China Standard Time)

Success (95 tests, 800/800 methods, 0 void method, 0 skipped method, 180913 assertions)!

Hoa\Zformat has been created. I am removing hoa/core from everywhere :-).

Ivan Enderlin · Answer 37 · Sat Jan 09 2016 15:43:30 GMT+0800 (China Standard Time)

All libraries are updated. Now I am running devtools:cs to update the year.

Ivan Enderlin · Answer 38 · Sat Jan 09 2016 16:14:46 GMT+0800 (China Standard Time)

Next step is to merge everything and create new snapshots for all libraries. I don't have enough time to do it well if something failed right now. Let see tonight or tomorrow!

Ivan Enderlin · Answer 39 · Mon Jan 11 2016 18:11:35 GMT+0800 (China Standard Time)

Done \o/.
Congratulations to everyone 🎆 🎉 🎈!!

1e1 · Answer 40 · Mon Jan 11 2016 20:36:57 GMT+0800 (China Standard Time)

👏
Great job!
I've just read the discussion, do you know what you earn by registering hoa:// in Composer? For projects like the official website, Rulerz, some Liip projects…

Ivan Enderlin · Answer 41 · Mon Jan 11 2016 20:49:21 GMT+0800 (China Standard Time)

@1e1 What we earn? I don't understand.

Dupont Yohann · Answer 42 · Mon Jan 11 2016 21:29:08 GMT+0800 (China Standard Time)

I suppose that he speak about performance

Ivan Enderlin · Answer 43 · Mon Jan 11 2016 21:30:49 GMT+0800 (China Standard Time)

That's very light. Let me try a small benchmark.

1e1 · Answer 44 · Mon Jan 11 2016 21:47:30 GMT+0800 (China Standard Time)

Yeap! (Sorry)

Ivan Enderlin · Answer 45 · Mon Jan 11 2016 21:51:01 GMT+0800 (China Standard Time)

tl;dr

Here is small benchmark result when loading Hoa (because we have 1 require autoloaded file):

vs	Time (for 1000 runs)	Memory
`Hoa\Core`	18.084s	335408b
`Hoa\Consistency`	15.959s	56744b
Gain	11.751%	83.082%

Time

First script /tmp/a.sh:

for i in `seq 1 1000`; do
    php -r 'require "/usr/local/lib/Hoa/Core/Core.php";'
done

Second script /tmp/b.sh:

for i in `seq 1 1000`; do
    php -r 'require "/usr/local/lib/Hoa/Consistency/Prelude.php";'
done

Time for time:

$ time /tmp/a.sh
/tmp/a.sh  11.37s user 5.64s system 94% cpu 18.084 total

$ time /tmp/b.sh
/tmp/b.sh  9.56s user 5.07s system 91% cpu 15.959 total

That's an improvement of 11.751% at loading time.

Memory

First script /tmp/a.php:

<?php

$a = memory_get_usage();

require '/usr/local/lib/Hoa/Core/Core.php';

var_dump(memory_get_usage() - $a);

Second script /tmp/b.php:

<?php

$a = memory_get_usage();

require '/usr/local/lib/Hoa/Consistency/Prelude.php';

var_dump(memory_get_usage() - $a);

Time to run:

$ php /tmp/a.php
int(335408)

$ php /tmp/b.php
int(56744)

That's an improvement of 278664b, be 272Kb, be 83.082% at initialization per process per run.

[meta] Split the core

History

Global idea

Dealing with pre-loading

Case of Hoa\Core\Consistency

Case of Hoa\Core\Parameter

Case of Hoa\Core\Protocol

Steps

Next steps

Update all composer.json files

tl;dr

Time

Memory

Case of `Hoa\Core\Consistency`

Case of `Hoa\Core\Parameter`

Case of `Hoa\Core\Protocol`

Update all `composer.json` files