Feed Me

Feed Me is a Craft plugin to make it easy to import entries and entry data from XML, RSS, ATOM or JSON feeds. Feeds can be setup as a task in Craft's Control Panel, or called on-demand for use in your twig templates.

A common use-case for this plugin is to consume external feeds (news, events), but can also be used as a once-off task for importing content when migrating from other sites.

Features

Import data from XML, RSS, ATOM or JSON feeds.
Feeds are saved to allow easy re-processing on-demand, or to be used in a Cron job.
Map feed data to your entry fields. See Supported Fieldtypes.
Duplication handling - control what happens when feeds are processed again.
Uses Craft's Task service to process feeds in the background.
Database backups before each feed processing.
Troubleshoot feed processing issues with logs.
Grab feed data directly from your twig templates.

Install

Add the feedme directory into your craft/plugins directory.
Navigate to Settings -> Plugins and click the "Install" button.

Plugin options

Change the plugin name as it appear in the CP navigation.
Set the default cache (for calls using the template tag only).

Usage

Head to the Feed Me page using the CP navigation, and click the New Feed button.

Enter the required details to configure your feed:

Name this feed something useful so you'll remember what it does.
The actual URL to your feed.
Set the Feed Type to assist with mapping the correct elements.
The Primary Element reflects which node in the feed your data sits.
Select the Section and Entry Type for where you want to put the feed data.
Decide how you'd like to handle duplicate feed items (if you're going to be re-running this feed).

Then, select what data from the feed you wish to capture, and what fields to map it to, depending on your Section and Entry Type selection. Here you'll be able to choose which fields (can be more than one) you'd like to compare to determine if a feed item is a duplicate.

Save the feed for later, or start the import.

Supported Fieldtypes

Feed Me supports mapping data from your feeds to the following Fieldtypes:

Craft

Assets
Categories
Checkboxes
Color
Date/Time
Dropdown
Entries
Lightswitch
Matrix
Multi-Select
Number
Plain Text
Position Select
Radio Buttons
Rich Text
Table
Tags
Users

###Element Creation

For certain elements, it may be benefitial to create the element's data, if not already created. Like if an Asset doesn't exist in your Assets collection, upload it. Similarly with Categories, and other fields.

Currently, Feed Me handles the following applicable fields in these ways:

Assets: Only supports mapping existing assets to this entry. Must provide filename (without extension) to successfully map.

Categories: Are created if they do not exist, or mapped if they do. The value provided in the feed data must be the Name for the category.

Entries: Only supports mapping existing entries to this entry. The feed field must contain either the Title or Slug of the entry to successfully map.

Tags: Are always created.

Users: Only supports mapping existing users to this entry. The feed field must contain either the users email or username to successfully map.

We plan to include options for whether you would like to do this on a per-field basis.

Duplication Handling

When running the feed task multiple times, there may or may not be the same data present in the feed as the last time you ran the task. To deal with this, you can control what happens to old (and new) entries when the feed is processed again.

You may choose multiple fields to determine if an entry is a duplicate. Most commonly, you'll want to compare the Title field, but can be any fields you require.

Add Entries:

Existing entries will be skipped and left untouched, new entries however, will be added to the section. Use case: Feed aggregation, blog entries, etc.

"I want to keep existing entries untouched but add new ones."

Update Entries

Existing entries will have their fields updated with data from this feed. Use case: Any feed which needs to be kept up to date.

"I want to update existing entries and add new ones."

Delete Entries

Delete all existing entries in this section, adding only entries from this feed. Be careful. Use case: Events, or when only data from the current feed is required.

"I want only the entries from this feed in this section."

Using with a Cron job

Scheduling feed processing is not something thats currently built into Feed Me. Instead, you'll need to setup a Cron job, or a similar scheduled task to fire the feed processing at the desired interval.

Find the 'Direct feed link' icon (next to the delete icon) on the main Feed Me page and copy this URL. Use one of the following to setup as a Cron Job - replacing the URL with what you just copied.

/usr/bin/wget -O - -q -t 1 "http://your.domain/actions/feedMe/feeds/runTask?direct=1&feedId=1&passkey=FwafY5kg3c"

curl --silent --compressed "http://your.domain/actions/feedMe/feeds/runTask?direct=1&feedId=1&passkey=FwafY5kg3c"

/usr/bin/lynx -source "http://your.domain/actions/feedMe/feeds/runTask?direct=1&feedId=1&passkey=FwafY5kg3c"

Parameters

direct (required) - Must be set to 1 or true. Tells Feed Me this is a externally-triggered task.
feedId (required) - The ID of the feed you wish to process.
passkey (required) - A unique, generated identifier for this feed. Ensures not just anyone can trigger the import.
url (optional) - If your feed URL changes, you can specify it here. Ensure the structure of the feed matches your field mappings.

Performance

Feed Me can handle importing large feeds by using Craft's Tasks service. Testing has shown that processing a feed with 10K items take roughly 15 minutes to import.

To get the most out of your feed processing, follow the below suggestions:

Turn off devMode. Craft's built-in logging when devMode is switched on will greatly slow down the import process, and causes a high degree of memory overhead.
Consider selecting the Add Entries option for duplication handling, depending on your requirements.
Consider turning off the Backup option for the feed. This will depend on your specific circumstances.

When directly running a feed, you will need to adjust the memory_limit and max_execution_time values in your php.ini file. This is due to the fact that directly processing a feed doesn't utalize Craft's Tasks service, which provides workarounds to these issues.

Template example

While you can create a feed task to insert data as entries, there are times which you may prefer to capture feed data on-demand, rather than saving as an entry. You can easily do this through your twig templates using the below.

Feeds are cached for performance (default to 60 seconds), which can be set by a tag parameter, or in the plugin settings.

{% set params = {
    url: 'http://path.to/feed/',
    type: 'xml',
    element: 'item',
    cache: 60,
} %}

{% set feed = craft.feedme.feed(params) %}

{% for node in feed %}
	Title: {{ node.title }}
	Publish Date: {{ node.pubDate }}
	Content: {{ node['content:encoded'] }}

	{% for name in node.category %}
		Category: {{ name }}
	{% endfor %}
{% endfor %}

Template parameters

url (string, required) - URL to the feed.
type (string, optional) - The type of feed you're fetching data from. Valid options are json or xml (defaults to xml).
element (string, optional) - Element to start feed from. Useful for deep feeds.
cache (bool or number, optional) - Whether or not to cache the request. If true, will use the default as set in the plugin settings, or if a number, will use that as its duration. Setting to false will disable cache completely.

If you're looking to consume REST feeds, APIs or other third-party platforms (Facebook, Twitter, etc), I would highly recommend using alecritson's Placid plugin, which supports a great deal more than this plugin offers.

Roadmap

Full support (creation) for search-only field types (Assets, Entries, Users)
Support third-party field types through hook
Finer control over category/tag creation
Improved logging of import process for individual feed items
Allow feed processing to be reverted
Allow static/default values to be set for fields when mapping
Batch processing for very long feeds
Support authentication for feed access (Basic, OAuth, Token)
Support for locale's
Organise documentation into Wiki

Have a suggestion? We'd love to hear about it! Make a suggestion

Bugs, feature requests, support

Found a bug? Have a suggestion? Submit an issue

For support, either Submit an issue or ask for assistance in the Craft Slack Group.

Thanks / Contributions

A massive thanks to Bob Olde Hampsink and his amazing work on the Import plugin, which this plugin is clearly influenced by, and Clearbold for Craft Import.

Pixel & Tonic for their amazing support, assistance, and of course for creating Craft.

Changelog

1.2.7

Fix where entries would not import if mapping element fields had more values that their field limit.
Fix for multiple matches found on existing categories, where only one should match.
Fix for escaping special characters in tags/category name.
Minor fix for tags/category mapping.

1.2.6

Fix for matching fields containing special characters.
Fix for tags and category mapping, mapping all available if supplied empty value.
Fix for backup lightswitch reflecting the saved state.
Fix to ensure at least one duplicate field is checked.

1.2.5

Refactoring for performance improvements.
Remove database logging until a better performing option is figured out. Logging still occurs to the file system under craft/storage/runtime/logs/.
Added optional backup option per-feed (default to true).
Minor fix so direct feed link doesn't use siteUrl.

1.2.4

Added support to fallback on cURL when file_get_contents() is not available. Can occur on some hosts where allow_url_fopen = 0.

1.2.3

Primary Element is no longer required - important for JSON feeds.
Fixes for when no primary element specified. It's pretty much optional now.
UI tidy for mapping table.
Fix for duplication handling not matching in some cases. Now uses element search.

1.2.2

JSON feed support.

1.2.1

Matrix support.
Table support.
Even better element-search.
Remove square brackets for nested field - serialization issues. Breaking change you will need to re-map some fields due to this fix.
Fix for supporting multiple entry types when selecting fields to map.

1.2

Lots of fixes and improvements for direct-processing. Includes URL parameter, passkey and non-Task processing.
Fixes with logging - now more informative!
Improvement nested element parsing.
Better date parsing.
CSRF protection compatibility.
Fix for duplicate field mapping not being remembered.

1.1

Prep for Table/Matrix mapping.
Better depth-mapping for feed data (was limited to depth-2).
Refactor field-mapping processing.
Set minimum Craft build.

1.0

Initial release.

stuart-mccoy / FeedMe