Moox Sync

Moox Sync is a powerful package for synchronizing data across multiple platforms in Laravel applications. It enables you to keep records in sync between different Moox platforms or other Filament and Laravel platforms. It is designed to be easy to use and configurable, with a focus on security (for synchronizing users) and flexibility.

moox-sync-demo.mp4

Quick Installation

These two commmands are all you need to install the package:

composer require moox/sync
php artisan mooxsync:install

Curious what the install command does? See manual installation below.

Create Platforms

First, you need to create a platform, better two. You would be able to sync on the same platform (but different model of course), but that is not the main idea of Sync.

So, let's create platforms.

A platform should have a name, a domain (we'll search the IP for you) and a token, that can be generated by the button.

Now the platforms are created and we can see it in the list and use them for our first Sync.

Create Syncs

After creating platforms, you are able to create a Sync between platforms.

Choose the Source Platform and Model, the Target Platform and Model and enable Platform Relations, if the model supports it, see Platform Relations.

Now you can see your Syncs in the list.

Key Concept

It is crucial to understand, that Moox Sync is not a two-way sync that simply copies data from A to B. It is a one-way sync from Source to Target, but it is made to be central-managed and it is smart, extensible and secure.

SyncPlattformsJob

Platforms are automatically synced between all platforms. For the first time done with basic security only using the Sync Token from config, afterwards adding the platform-token to provide first class security including HMAC.

The SyncPlatformsJob can be enabled via config or .env. This should run on not more than ONE platform.

As an alternative to running the job periodically (minutely by default), you can manually start the Job by pressing the Sync Platforms button after making updates to platforms.

SyncListener

The SyncListener runs an Eloquent Event Listener for each Sync and catches up all model changes done with Eloquent. If you change data in models without using Eloquent, you may consider firing an event:

Event::dispatch('eloquent.updated: '.get_class($this->record), $this->record);

Note: For handling imports or changes done outside of Laravel, we recommend running the SyncBackupJob.

The SyncListener runs the PrepareSynJob.

The SyncListener needs to be activated in Moox Sync Configuration. That should be done on Source platforms only.

PrepareSyncJob

The PrepareSyncJob is invoked by the SyncListener. It prepares the data and triggers the SyncWebhook.

The PrepareSyncJob supports custom data handling like custom queries, data manipulation, so called Transformers, also available as Transformer Bindings, configured in Moox Sync Configuration. See Advanced Usage.

SyncWebhook

Act as the entry point on the target platform, receiving data from the source platform via the PrepareSyncJob. Validates the incoming data using HMAC and checks for any transformation or field mapping requirements specified in the sync configuration.

Triggers the SyncJob with the validated and transformed data.

The SyncWebhook needs to be activated in Moox Sync Configuration. It needs to be enabled on Target platforms only.

SyncJob

The SyncJob queues the actual sync by using the SyncService.

SyncService

The Sync writes the data on the Target platform. It supports Custom SyncHandlerand PlatformRelations. See Advanced Usage.

Advanced Usage

Transformer

Transformers are not implemented yet. We recommend using Transformer Bindings instead.

Transformer Bindings

While the (selectable in Sync) Transformer feature is not fully implemented, you can use Transformer Bindings for custom data transformation logic.

To create a custom transformer:

Create a new class that extends AbstractTransformer.
Implement the transformCustomFields method.
Register your transformer in the transformer_bindings config.

See WpUserTransformer in Moox Press on how to implement a Transformer binding.

SyncHandler

For complex models requiring custom sync logic, implement Sync Handlers:

Create a new class that extends AbstractSyncHandler.
Implement the syncModel and deleteModel methods.
Register your sync handler in the sync_bindings config.

See WpUserSyncHandler in Moox Press on how to implement your own SyncHandler.

PlatformRelations

The PlatformRelationService is an optional feature, you can set for every sync. It is a key component of Moox Sync that handles the relationships between models and platforms. It provides methods for syncing and retrieving platform associations for any model.

Key methods:

syncPlatformsForModel($model, array $platformIds): Syncs the platforms for a given model.
getPlatformsForModel($model): Retrieves the platforms associated with a given model.

How to implement PlatformRelations

To implement PlatformRelations, you need to configure the models_with_platform_relations in the Moox Sync config. This is a list of models that should have platform relations.

While the relations are automatically added to the model, you may want to add a field to your Filament Resource to manage the platforms for a given record.

                    Select::make('platforms')
                        ->label('Platforms')
                        ->multiple()
                        ->options(function () {
                            return \Moox\Sync\Models\Platform::pluck('name', 'id')->toArray();
                        })
                        ->afterStateHydrated(function ($component, $state, $record) {
                            if ($record && class_exists('\Moox\Sync\Services\PlatformRelationService')) {
                                $platformService = app(\Moox\Sync\Services\PlatformRelationService::class);
                                $platforms = $platformService->getPlatformsForModel($record);
                                $component->state($platforms->pluck('id')->toArray());
                            }
                        })
                        ->dehydrated(false)
                        ->reactive()
                        ->afterStateUpdated(function ($state, callable $set, $record) {
                            if ($record && class_exists('\Moox\Sync\Services\PlatformRelationService')) {
                                $platformService = app(\Moox\Sync\Services\PlatformRelationService::class);
                                $platformService->syncPlatformsForModel($record, $state ?? []);
                            }
                        })
                        ->preload()
                        ->searchable()
                        ->visible(fn () => class_exists('\Moox\Sync\Models\Platform'))
                        ->columnSpan([
                            'default' => 12,
                            'md' => 12,
                            'lg' => 12,
                        ]),

SyncBackupJob

Not implemented yet!

The SyncBackupJob runs as a fallback, if the Eloquent listener is disabled or not invoked. It checks for changes in the database and syncs them to the target platform.

Security

Moox Sync implements robust security measures:

Webhook Authentication

The WebhookAuthMiddleware handles this authentication and verification process:

The API token authenticates the source platform with a shared secret, so be changed in config, so even the initial platform sync is done securely
After platforms are synced, the shared secret and platform token are used together to provide additional security
The HMAC signature verifies the integrity of the payload.
HTTPS (which you should enforce) protects against man-in-the-middle attacks

In addition, you are able to change the webhook url, so that guessing the webhook url becomes difficult.

Token Authentication

For API endpoints, Moox Sync uses token-based authentication via the PlatformTokenAuthMiddleware. Alternatively, you can use Sanctum to authenticate your API requests.

Config

Moox Sync is highly configurable via the sync.php config file. Here are the available options:

/*
|--------------------------------------------------------------------------
| Moox Configuration
|--------------------------------------------------------------------------
|
| This configuration file uses translatable strings. If you want to
| translate the strings, you can do so in the language files
| published from moox_core. Example:
|
| 'trans//core::core.all',
| loads from common.php
| outputs 'All'
|
*/

return [

    /*
    |--------------------------------------------------------------------------
    | Resources
    |--------------------------------------------------------------------------
    |
    | The following configuration is done per Filament resource.
    |
    */

    'resources' => [
        'sync' => [

            /*
            |--------------------------------------------------------------------------
            | Title
            |--------------------------------------------------------------------------
            |
            | The translatable title of the Resource in singular and plural.
            |
            */

            'single' => 'trans//core::sync.sync',
            'plural' => 'trans//core::sync.syncs',

            /*
            |--------------------------------------------------------------------------
            | Tabs
            |--------------------------------------------------------------------------
            |
            | Define the tabs for the Resource table. They are optional, but
            | pretty awesome to filter the table by certain values.
            | You may simply do a 'tabs' => [], to disable them.
            |
            */

            'tabs' => [
                'all' => [
                    'label' => 'trans//core::core.all',
                    'icon' => 'gmdi-filter-list',
                    'query' => [],
                ],
                /*
                'error' => [
                    'label' => 'trans//core::core.error',
                    'icon' => 'gmdi-text-snippet',
                    'query' => [
                        [
                            'field' => 'subject_type',
                            'operator' => '=',
                            'value' => 'Error',
                        ],
                    ],
                ],
                */
            ],
        ],
        'platform' => [

            /*
            |--------------------------------------------------------------------------
            | Title
            |--------------------------------------------------------------------------
            |
            | The translatable title of the Resource in singular and plural.
            |
            */

            'single' => 'trans//core::sync.platform',
            'plural' => 'trans//core::sync.platforms',

            /*
            |--------------------------------------------------------------------------
            | Tabs
            |--------------------------------------------------------------------------
            |
            | Define the tabs for the Resource table. They are optional, but
            | pretty awesome to filter the table by certain values.
            | You may simply do a 'tabs' => [], to disable them.
            |
            */

            'tabs' => [
                'all' => [
                    'label' => 'trans//core::core.all',
                    'icon' => 'gmdi-filter-list',
                    'query' => [],
                ],
                /*
                'error' => [
                    'label' => 'trans//core::core.error',
                    'icon' => 'gmdi-text-snippet',
                    'query' => [
                        [
                            'field' => 'subject_type',
                            'operator' => '=',
                            'value' => 'Error',
                        ],
                    ],
                ],
                */
            ],
        ],
    ],

    /*
    |--------------------------------------------------------------------------
    | Navigation Group
    |--------------------------------------------------------------------------
    |
    | The translatable title of the navigation group in the
    | Filament Admin Panel. Instead of a translatable
    | string, you may also use a simple string.
    |
    */

    'navigation_group' => 'trans//core::core.tools',

    /*
    |--------------------------------------------------------------------------
    | Navigation Sort
    |--------------------------------------------------------------------------
    |
    | This values are the sort order of the navigation items in the
    | Filament Admin Panel. If you use a bunch of Moox
    | plugins, everything should be in order.
    |
    */

    'navigation_sort' => 9500,

    /*
     |--------------------------------------------------------------------------
     | API
     |--------------------------------------------------------------------------
     |
     | Enable or disable the API and configure all entities.
     | Public or secured by platform or sanctum.
     | Available at /api/{entity}
     |
     */

    'entities' => [
        'Sync' => [
            'api' => [
                'enabled' => true,
                'public' => false,
                'auth_type' => 'platform',
                'active_routes' => [
                    'index',
                    'show',
                    'store',
                    'update',
                    'destroy',
                ],
            ],
            'model' => '\Moox\Sync\Models\Sync',
            'resource' => '\Moox\Sync\Resources\SyncResource',
            'api_controller' => '\Moox\Sync\Http\Controllers\Api\SyncController',
        ],
        'Platform' => [
            'api' => [
                'enabled' => true,
                'public' => false,
                'auth_type' => 'platform',
                'active_routes' => [
                    'index',
                    'show',
                    'store',
                    'update',
                    'destroy',
                ],
            ],
            'model' => '\Moox\Sync\Models\Platform',
            'resource' => '\Moox\Sync\Resources\PlatformResource',
            'api_controller' => '\Moox\Sync\Http\Controllers\Api\PlatformController',
        ],
    ],

    /*
    |--------------------------------------------------------------------------
    | Sync Platform Job
    |--------------------------------------------------------------------------
    |
    | Enable or disable the Sync Platform Job that automatically syncs data
    | between all platforms.
    |
    */

    'sync_platform_job' => [
        'enabled' => env('SYNC_PLATFORM_JOB_ENABLED', false),
        'frequency' => 'everyFiveMinutes', // hourly, daily, hourly, etc.
    ],

    /*
    |--------------------------------------------------------------------------
    | Sync Backup Job
    |--------------------------------------------------------------------------
    |
    | Enable or disable the Sync Backup Job that automatically syncs data
    | based on your sync configurations, when changes are made outside
    | of Eloquent events or you've disabled the Eloquent listener.
    |
    */

    'sync_backup_job' => [
        'enabled' => env('SYNC_BACKUP_JOB_ENABLED', false),
        'frequency' => 'everyFiveMinutes', // hourly, daily, hourly, etc.
    ],

    /*
    |--------------------------------------------------------------------------
    | Sync Eloquent Listener
    |--------------------------------------------------------------------------
    |
    | Enable or disable the Eloquent listener that automatically syncs
    | data when a model is created, updated or deleted. Use
    | it wisely together with the Sync Backup Job.
    |
    */

    'sync_eloquent_listener' => [
        'enabled' => env('SYNC_LISTENER_ENABLED', false),
    ],

    /*
    |--------------------------------------------------------------------------
    | Sync Webhook
    |--------------------------------------------------------------------------
    |
    | Enable or disable the webhook that automatically syncs
    | data when a model is created, updated or deleted. Use
    | it wisely together with the Sync Backup Job.
    |
    */

    'sync_webhook' => [
        'enabled' => env('SYNC_WEBHOOK_ENABLED', false),
    ],

    /*
    |--------------------------------------------------------------------------
    | Models with Platform Relations
    |--------------------------------------------------------------------------
    |
    | List of models that should have platform relations. This adds the
    | platforms relation to the model. No need to add a trait or
    | any dependency to the model or the package.
    |
    */

    'models_with_platform_relations' => [
        'App\Models\User',
        'Moox\User\Models\User',
        'Moox\Press\Models\User',
        // Add any other models here
    ],

    /*
    |--------------------------------------------------------------------------
    | // TODO: Models with syncable Relations - not implemented yet
    |--------------------------------------------------------------------------
    |
    | List of models that should have syncable relations, which should be
    | synced to other platforms, when changes are made. This does not
    | add the related models to the listener, but syncs them with
    | the sync model automatically. So platform-related models
    | (like WpUsers and WpUserMetaare able to use this
    | feature, too.
    |
    */

    'models_with_syncable_relations' => [
        'Moox\User\Models\User',
        'Moox\Press\Models\WpUser' => [
            'Moox\UserSession\Models\Session',
        ],
        // Add any other models here
    ],

    /*
    |--------------------------------------------------------------------------
    | Unique Identifier Fields
    |--------------------------------------------------------------------------
    |
    | The synced model should have a unique identifier. The id auto-
    | increments, so it is not suitable. Perfect would be a ULID
    | or UUID, but any other unique identifier will work, too.
    | This is the list of identifiers Moox Sync searches for.
    | for, in the given order. Ad more as you need them.
    |
    */

    'unique_identifier_fields' => [
        'ulid',
        'uuid',
        'slug',
        'name',
        'title',
    ],

    /*
    |--------------------------------------------------------------------------
    | Local Identifier Fields
    |--------------------------------------------------------------------------
    |
    | These are the fields that are used as unique identifiers for
    | the models. They are used to identify the models on the
    | source platform. The array is sorted by priority.
    |
    */

    'local_identifier_fields' => [
        'ID',
        'uuid',
        'ulid',
        'id',
    ],

    /*
    |--------------------------------------------------------------------------
    | Transformer
    |--------------------------------------------------------------------------
    |
    | You can register Transformer Classes here, to make them available
    | when creating Syncs. These classes can contain queries or
    | translation maybe. Alternatively you can bind models.
    |
    */

    // Not implemented yet, use bindings instead
    'transformer_classes' => [
        // Not implemented yet
    ],

    /*
    |--------------------------------------------------------------------------
    | Transformer Bindings
    |--------------------------------------------------------------------------
    |
    | You can register custom Transformer Bindings, used for Press models for
    | example, where we have to read meta data or custom tables like
    | terms and taxonomies instead of native categories.
    |
    */

    'transformer_bindings' => [
        // Add transformer bindings here, like:
        // \Moox\Press\Models\WpUser::class => \Moox\Press\Transformer\WpUserTransformer::class,
    ],

    /*
    |--------------------------------------------------------------------------
    | Sync Bindings
    |--------------------------------------------------------------------------
    |
    | You can register custom Sync Bindings, used for Press models for
    | example, where we have to write meta data or custom tables
    | like terms and taxonomies instead of native categories.
    |
    */

    'sync_bindings' => [
        // Add sync handlers here, like:
        // \Moox\Press\Models\WpUser::class => \Moox\Press\Handlers\WpUserSyncHandler::class,
    ],
];

Logging

Setting up Sync involves the connection of two or more platforms, availability of APIs and running Jobs. This is why we added a logger to the package, that can be setup in Moox Core config. The flow of a working sync may look like this. Depending on the log level you get very detailed information about the data flow in Moox Sync. On production anything else than 0 should not be the default, but can perfectly used to implement or debug Moox Sync.

Manual Installation

Instead of using the install-command php artisan mooxsync:install you are able to install this package manually step by step:

// Publish and run the migrations:
php artisan vendor:publish --tag="sync-migrations"
php artisan migrate

// Publish the config file with:
php artisan vendor:publish --tag="sync-config"

Edit your PanelProvider to add both Plugins to your Navigation.

Changelog

Please see CHANGELOG for more information on what has changed recently.

Security Vulnerabilities

Please review our security policy on how to report security vulnerabilities.

Credits

All Contributors

License

The MIT License (MIT). Please see License File for more information.

mooxphp / sync