Move Into Drupal Using The Migrate Module

  • View

  • Download

Embed Size (px)


The migrate module provides a flexible framework for migrating content into Drupal from other sources (e.g., when converting a web site from another CMS to Drupal). Out-of-the-box, support for creating core Drupal objects such as nodes, users, files, terms, and comments are included - it can easily be extended for migrating other kinds of content. The power comes from an object oriented API that's tricky to get started with - We'll walk through the various classes in the module and how they work together to manage migrations.I am currently looking for co-presenters or to present in a panel format as I feel we can all have something to learn from each other.UPDATE July 21, 2012: Thank you to everyone that was able to come out to the session. I know it was a complex topic. As another resource, you can take a look at the code from the example I displayed today at Obviously, the migration won't work (the db needs to exist) but the code should hopefully be helpful. Cheers!


  • 1. Move into Drupal using the Migrate ModuleNYC Camp 2012Ashok Modi (BTMash)

2. Agenda Introduction to Migrate Theory Implementation Hooks Classes Migration Handlers Alterations Commands / Demo (Time Permitting) Q & A (Conclusion) 3. Thanks Mike Ryan (mikeryan) Moshe Weitzman (moshe weitzman) Frank Carey Andrew Morton (drewish) 4. Introduction to Migrate - Options What are your options to bring content over into Drupal? By Hand Very time consuming. Not feasible if you have a lot of content. If you really dont like who you work with. Custom Scripts Might be ok as a one-off solution. As flexible as you want. Write out drush plugins/web ui. Tracking? Integration into your source? 5. Introduction to Migrate - OptionsFeeds Absolutely great option. Easy to setup. Maps fields from source -> destination Can Import RSS / Atom / Various Feeds Plugins for CSV, Database, even LDAP Well documented. Performance issues. Content update handling. Doesnt work well wrt references from othercontent (types). 6. Introduction to Migrate Powerful object oriented framework for moving contentinto Drupal. Already defined many import sources. XML, JSON, CSV, DB, etc. Support to migrate into various types of content. users, nodes, comments, taxonomy, all core entities. can define your own import handler. can import into a particular table! Fast. Minimal UI, mainly steered around drush. Drush integration. Steep Learning Curve. You will write code. 7. Introduction to Migrate Drupal 6 requires autoload and dbtngmodules. So the code is very similar in 6 and7. Migrate Extras provides support for manycontrib modules. Provides base class for importing to entities fromEntityAPI. More field modules implementing field handlers. The most comprehensive and up-to-datedocumentation is the and wine.incexamples. Part of the Migrate module. 8. Goal Source Destination sidcontent_id(auto)titletitle useruidfield1 field1field2 field2 ...... fieldNfieldN 9. Source Interface to your current set of data (csv,json, xml, db, etc). Provides a list of fields. Responsible for iterating over the rows ofdata. 10. Destination Responsible for saving specific type ofcontent to Drupal (user, node, row in aparticular table) Each Source record correlates to oneDestination record. 11. Field Mappings Links a source field to destination field. Basic functions such as splitting into anarray based on separators, etc. Can pass additional arguments (as the fieldhandler implements). 12. Mapping (goal) Source Destination sidcontent_id(auto)titleMapping title user Mapping (alter and reference)uidfield1 field1field2 field2field3...field4 fieldN...fieldN 13. Map Connects the source and destination IDsallowing for translation between them. Tracks keys schema format. Allows for migration to re-run and updateexisting records. Allows imported records to be deleted. Allows you to reference the ID from anothermigration for to get converted for your ownmigration. 14. Migration Map Source Destination sidMap Table content_id(auto)titleMapping title user Mapping (alter and reference)uidfield1 field1field2 field2field3...field4 fieldN...fieldN 15. Migration Sets up all the necessary pieces: Source,Destination, Map, Field Mappings. May provide logic for skipping over rowsduring migration. May alter the Source Data during themigration. May alter the Destination Entities during theMigration. 16. Field Handler Converts your source data into a format that Drupal understands. $row->bar = array(foo, bar) into$entity_field_bar = array(und => array(0 => array(value => foo),1 => array(value => bar),),); 17. Destination Handler Extends existing destinations and addsadditional functionality. MigrateCommentNodeHandler provides the option toallow for comments to a given node. Contrib projects might want to create these. Flag? 18. Field Handler SourceDestination sidMap Table content_id(auto)titleMapping title (text) user Mapping (alter and reference) uidfield1 field1 (text)field2 field2 (image)field3...field4fieldN (tags)...fieldN 19. Implementation Let Migrate know about your module (hook). Build a migration class. Provide a description. Give it information about where the content is coming from (Source). Give it information about where the content is going to get saved(Destination). Map the fields from the source into the destination (Map). (optional) Massage the data / add any fields you were not able to getin the initial mapping. (optional) Add / massage any data that does not have field handlersbefore the content gets saved. Register class file in .info file. 20. Implementation - Hooks Just one :) Provide the API version number (currently at 2)function my_migrate_module_migrate_api() {return array(api => 2,);} Might change to 3/4/5...N in the future ;) 21. Implementation - Class Consists of at least 1 function and 3 optional functions.class MYBundleMigration extends Migration {public function __construct() { ... } # REQD.public function prepareRow($row)public function prepare($entity, $row)public function complete($entity, $row)} 22. Import Flow Source iterates until it finds an appropriate record. Calls prepareRow($row) letting you modify or reject thedata in $row. Migration applies the Mappings and Field Handlers toconvert $row into $entity. Migrate calls on prepare($entity, $row) to modify theentity before it gets saved. Entity is saved. Migrate records the IDs into the map and callscomplete() so you can see and work with the final EntityID. 23. Implementation - __construct() Set up the source, destination, map, fieldmappings in constructor.class MyBundleMigration extends Migration {public function __construct() {parent::__construct();$this->source = ;$this->destination = ; $this->map = ; $this->addFieldMapping($my_dest_fld, $my_src_fld);}} 24. Implementation - __construct()Source Fields Lets Migration class know a little about thefields that are coming in (like compoundfields). Can set it to an array if nothing complex.$source_fields = array(mtid => The source row ID,compound_field_1 => Field not from initalquery but will be necessary later on.); 25. Implementation - __construct()Source (Current Database)// Required$query = db_select(my_table, mt);$query->fields(mt, array(mtid, style, details, updated,style_parent, style_image));$query->join(mt_extras, mte, mt.mtid = mte.mtid);$query->orderBy(mt.updated, ASC);// Implement a count_query if it is different. Or set to NULL.$this->source = new MigrateSourceSQL($query,$source_fields, $count_query); 26. Implementation - __construct()Source (External Database)// Using another db connection called for_migration.$connection = Database::getConnection(for_migration);$query = $connection->select(my_table, mt);$query->fields(mt, array(mtid, style, details, updated,style_parent, style_image));$query->orderBy(mt.updated, ASC);// Implement a count_query if it is different. Or set to NULL.$this->source = new MigrateSourceSQL($query,$source_fields, $count_query,array(map_joinable => FALSE)); Lets migrate know there is no easy way to map the IDs. 27. Implementation - __construct()Source (CSV File)// The definition of the columns. Keys are integers// values are an array of: field name then description.$columns = array(0 => array(cvs_uid, Id),1 => array(email, Email),2 => array(name, Name),3 => array(date, Date),);$this->source = new MigrateSourceCSV("path/to/file.csv",$columns, array(header_rows => TRUE),$this->fields()); 28. Implementation - __construct()Source (Other Sources) Comes with base source migration classesto migrate from JSON, XML, File Directories. Expect to make some changes dependingon the migration format. 29. Source Base Classes If you have source IDs referenced separately from yourvalues. Use MigrateSourceList as a source. Implement MigrateList for fetching counts and IDs, and MigrateItem forfetching values. If everything is in a single file with IDs mixed in: Use MigrateSourceMultiItems as a source. Implement MigrateItems for extracting IDs and values. Look at,, and clearer examples. 30. Implementation - __construct()Migration Map$this->map = new MigrateSQLMap($this->machineName,// Describe your primary ID schemaarray(mtid => array(type => integer,unsigned => TRUE,not null => TRUE,alias => mt),),MigrateDestinationNode::getKeySchema()); 31. Implementation - __construct()Highwater May have noticed orderby on sql queries. Migrate feature to figure out if a piece ofcontent can be updated rather than insertedjust once. Need to let migrate know which columncontains the highwater data.$this->highwaterField = array(name => updated,alias => mt,); 32. Implementation - __construct()Destination// Terms$this->destination = new MigrateDestinationTerm(site_vocabulary);// Nodes$this->destination = new MigrateDestinationNode(articles);// Users$this->destination = new MigrateDestinationUser();// Contrib - Commerce Products$this->destination = new MigrateDestinationEntity(commerce_product); 33. Implementation - __construct()Field Mapping// Can be simple.$this->addFieldMapping(dest_name, source_name);// Can be a set value.$this->addFieldMapping(uid)->defaultValue(1);// Can have no value (or whatever the system default is)$this->addFieldMapping(path)->issueGroup(DNM);// Can be multiple values with a separator.$this->addFieldMapping(field_tags, source_tags)->separator(,);// Can have arguments$this->addFieldMapping(field_body, description)->arguments($arguments); 34. Implementation - __construct()Field Mapping (contd) Most meta settings for fields now also field map