=head1 Summary of initialdata files
It's often useful to be able to test configuration/database changes and then
apply the same changes in production without manually clicking around. It's
also helpful if you're developing customizations or extensions to be able to
get a fresh database back to the state you want for testing/development.
This documentation applies to careful and thorough sysadmins as well as
extension authors who need to make database changes easily and repeatably for
new installs or upgrades.
=head1 Examples
RT ships with many initialdata files, only one of which is used to
configure a fresh install; the rest are used for upgrades, but function
the same despite being named differently.
etc/initialdata
etc/upgrade/*/content
The upgrade "content" files are meant to be incremental changes applied on top
of one another while the top level initialdata file is for fresh RT installs.
Extensions may also ship with database changes in such files. You may find
some in your install with:
find local/plugins -name initialdata -or -name content
=head1 What can be in an initialdata file?
By default, initialdata files are Perl, but often consist primarily of a
bunch of data structures defining the new records you want and not
much extra code. There's nothing stopping you from writing a bunch of
code, however!
RT's initialdata importer is also pluggable and you can add handlers for
other formats using the option L<RT_Config/InitialdataFormatHandlers>.
Starting in RT 5.0, RT supports JSON in addition to Perl. See below for
details.
The basic template of a new Perl initialdata file should look something like this:
use strict;
use warnings;
our @Queues = (
# some definitions here
);
our @Groups = (
# some other definitions here
);
1;
The C<@Queues> and C<@Groups> arrays are expected by RT and should contain
hashref definitions. There are many other arrays RT will look for and act on,
described below. None are required, all may be used. Keep in mind that since
they're just normal Perl arrays, you can C<push> onto them from a loop or
C<grep> out definitions based on conditionals or generate their content with
C<map>, etc.
The complete list of possible arrays which can be used, along with
descriptions of the values to place in them, is below.
=head2 C<@Users>
push @Users, {
Name => 'john.doe',
Password => 'changethis',
Language => 'fr',
Timezone => 'America/Vancouver',
Privileged => 1,
Disabled => 0,
};
Each hashref in C<@Users> is treated as a new user to create and passed
straight into C<< RT::User->Create >>. All of the normal user fields are
available, as well as C<Privileged> and C<Disabled> (both booleans) which
will do the appropriate internal group/flag handling. Also accepts an
C<Attributes> key, which is equivalent to pushing its arrayref of values
onto C<@Attributes>, below, with C<Object> set to the new user.
For a full list of fields, read the documentation for L<RT::User/Create>.
=head2 C<@Groups>
push @Groups, {
Name => 'Example Employees',
Description => 'All of the employees of my company',
Members => { Users => [ qw/ alexmv trs falcone / ],
Groups => [ qw/ extras / ] },
};
Creates a new L<RT::Group> for each hashref. In almost all cases you'll want
to follow the example above to create a group just as if you had done it from
the admin interface.
In addition to the C<Members> option shown above, which can take both
users and groups, the C<MemberOf> field may be a single value or an
array ref. Each value should be a user-defined group name or hashref to
pass into L<RT::Group/LoadByCols>. Each group found will have the new
group added as a member.
It also accepts an C<Attributes> key, which is equivalent to pushing its
arrayref of values onto C<@Attributes>, below, with C<Object> set to the
new group.
=head2 C<@CustomRoles>
push @CustomRoles, {
Name => 'Support Rep',
Description => 'Support representative assigned for this ticket',
ApplyTo => 'General',
};
Creates a new L<RT::CustomRole> for each hashref. Custom roles B<must>
be applied per queue; they cannot be applied globally. See the
C<ApplyTo> entry as described in L</@CustomFields> below for more
information.
=head2 C<@Queues>
push @Queues, {
Name => 'Helpdesk',
CorrespondAddress => 'help@example.com',
CommentAddress => 'help-comment@example.com',
};
Creates a new L<RT::Queue> for each hashref. Refer to the documentation of
L<RT::Queue/Create> for the fields you can use. It also accepts an
C<Attributes> key, which is equivalent to pushing its arrayref of values
onto C<@Attributes>, below, with C<Object> set to the new queue.
=head2 C<@CustomFields>
push @CustomFields, {
Name => 'Favorite color',
Type => 'FreeformSingle',
LookupType => 'RT::Queue-RT::Ticket',
};
Creates a new L<RT::CustomField> for each hashref. It is the most complex of
the initialdata structures. The most commonly used fields are:
=over 4
=item C<Name>
The name of this CF as displayed in RT.
=item C<Description>
A short summary of what this CF is for.
=item C<ApplyTo>
May be a single value, or an array reference of such; each should be
either an ID or Name. If omitted, the CF is applied globally. This
should not be used for User or Group custom fields.
This argument may also be passed via C<Queue>, for backwards
compatibility, which also defaults the C<LookupType> to
C<RT::Queue-RT::Ticket>.
=item C<Type>
One of the following on the left hand side:
SelectSingle # Select one value
SelectMultiple # Select multiple values
FreeformSingle # Enter one value
FreeformMultiple # Enter multiple values
Text # Fill in one text area
HTML # Fill in one HTML area
Wikitext # Fill in one wikitext area
BinarySingle # Upload one file
BinaryMultiple # Upload multiple files
ImageSingle # Upload one image
ImageMultiple # Upload multiple images
Combobox # Combobox: Select or enter one value
AutocompleteSingle # Enter one value with autocompletion
AutocompleteMultiple # Enter multiple values with autocompletion
Date # Select date
DateTime # Select datetime
IPAddressSingle # Enter one IP address
IPAddressMultiple # Enter multiple IP addresses
IPAddressRangeSingle # Enter one IP address range
IPAddressRangeMultiple # Enter multiple IP address ranges
If you don't specify "Single" or "Multiple" in the type, you must specify
C<MaxValues>.
=item C<LookupType>
Labelled in the CF admin page as "Applies to". This determines whether your CF
is for Tickets, Transactions, Users, Groups, or Queues. Possible values:
RT::Queue-RT::Ticket # Tickets
RT::Queue-RT::Ticket-RT::Transaction # Transactions
RT::User # Users
RT::Group # Groups
RT::Queue # Queues
RT::Class-RT::Article # Articles
Ticket CFs are the most common, meaning C<RT::Queue-RT::Ticket> is the most
common C<LookupType>.
=item C<RenderType>
Only valid when C<Type> is "Select". Controls how the CF is displayed when
editing it. Valid values are: C<Select box>, C<List>, C<Dropdown>, and C<Checkbox>.
See L<RT::CustomField/SetRenderType> for details.
=item C<MaxValues>
Determines whether this CF is a Single or Multiple type. 0 means multiple. 1
means single.
Make sure to set the C<MaxValues> field appropriately, otherwise you can end up
with unsupported CF types like a "Select multiple dates" (it doesn't Just
Work).
You can also use old-style C<Type>s which end with "Single" or "Multiple", for
example: SelectSingle, SelectMultiple, FreeformSingle, etc.
=item C<Values>
C<Values> should be an array ref (never a single value!) of hashrefs
representing new L<RT::CustomFieldValue> objects to create for the new custom
field. This only makes sense for "Select" CFs. An example:
my $i = 1;
push @CustomFields, {
LookupType => 'RT::Queue-RT::Ticket', # for Tickets
Name => 'Type of food',
Type => 'SelectSingle', # SelectSingle is the same as: Type => 'Select', MaxValues => 1
RenderType => 'Dropdown',
Values => [
{ Name => 'Fruit', Description => 'Berries, peaches, tomatos, etc', SortOrder => $i++ },
{ Name => 'Vegetable', Description => 'Asparagus, peas, lettuce, etc', SortOrder => $i++ },
# more values as such...
],
};
In order to ensure the same sorting of C<Values>, set C<SortOrder> inside each
value. A clever way to do this easily is with a simple variable you increment
each time (as above with C<$i>). You can use the same variable throughout the
whole file, and don't need one per CF.
=item C<BasedOn>
Name or ID of another Select Custom Field. This makes the named CF the source
of categories for your values.
=item C<Pattern>
The regular expression text (not C<qr//>!) used to validate values.
=back
It also accepts an C<Attributes> key, which is equivalent to pushing its
arrayref of values onto C<@Attributes>, below, with C<Object> set to the
new custom field.
Refer to the documentation and implementation of L<RT::CustomField/Create> and
L<RT::CustomFieldValue/Create> for the full list of available fields and
allowed values.
=head2 C<@ACL>
C<@ACL> is very useful for granting rights on your newly created records or
setting up a standard system configuration. It is one of the most complex
initialdata structures.
=head3 Pick one or more C<Right>s
All ACL definitions expect a key named C<Right> with the internal right
name you want to grant; alternately, it may contain an array reference
of right names. The internal right names are visible in RT's admin
interface in grey next to the longer descriptions.
=head3 Pick a level: on a queue, on a CF, or globally
After picking a C<Right>, you need to specify on what object the right is
granted. This is B<different> than the user/group/role receiving the right.
=over 4
=item Granted on a custom field by name (or ID), potentially a global or queue
CF => 'Name',
LookupType => 'RT::User', # optional, in case you need to disambiguate
=item Granted on a queue
Queue => 'Name',
=item Granted on a custom field applied to a specific queue
CF => 'Name',
Queue => 'Name',
=item Granted on a custom field applied to some other object
# This finds the CF named "Name" applied to Articles in the
# "Responses" class
CF => 'Name',
LookupType => RT::Article->CustomFieldLookupType,
ObjectId => 'Responses',
=item Granted on some other object (article Classes, etc)
ObjectType => 'RT::Class',
ObjectId => 'Name',
=item Granted globally
Specifying none of the above will get you a global right.
=back
There is currently no way to grant rights on a group or article class level.
Note that you can grant rights B<to> a group; see below. If you need to grants
rights on a group or article class level, you'll need to write an C<@Final>
subref to handle it using the RT Perl API.
=head3 Pick a Principal: User or Group or Role
Finally you need to specify to what system group, system/queue role,
user defined group, or user you want to grant the right B<to>.
=over 4
=item An internal user group
GroupDomain => 'SystemInternal',
GroupType => 'Everyone, Privileged, or Unprivileged'
=item A system-level role
GroupDomain => 'RT::System-Role',
GroupType => 'Requestor, Owner, AdminCc, or Cc'
=item A queue-level role
GroupDomain => 'RT::Queue-Role',
Queue => 'Name',
GroupType => 'Requestor, Owner, AdminCc, or Cc',
=item A system-level custom role
GroupDomain => 'RT::System-Role',
CustomRole => 'Supervisor',
=item A queue-level custom role
Queue => 'Customer Support',
GroupDomain => 'RT::Queue-Role',
CustomRole => 'Customer',
=item A group you created
GroupDomain => 'UserDefined',
GroupId => 'Name'
=item Individual user
UserId => 'Name or email or ID'
=back
=head3 Common cases
You're probably looking for definitions like these most of the time.
=over 4
=item Grant a global right to a group you created
{ Right => '...',
GroupDomain => 'UserDefined',
GroupId => 'Name' }
=item Grant a queue-level right to a group you created
{ Queue => 'Name',
Right => '...',
GroupDomain => 'UserDefined',
GroupId => 'Name' }
=item Grant a CF-level right to a group you created
{ CF => 'Name',
Right => '...',
GroupDomain => 'UserDefined',
GroupId => 'Name' }
=back
Since you often want to grant a list of rights on the same object/level to the
same role/group/user, we generally use Perl loops and operators to aid in the
generation of C<@ACL> without repeating ourselves.
# Give Requestors globally the right to see tickets, reply, and see the
# queue their ticket is in
push @ACL, map {
{
Right => $_,
GroupDomain => 'RT::System-Role',
GroupType => 'Requestor',
}
} qw(ShowTicket ReplyToTicket SeeQueue);
=head3 Troubleshooting
The best troubleshooting is often to see how the rights you define in C<@ACL>
show up in the RT admin interface.
=head2 C<@Scrips>
Creates a new L<RT::Scrip> for each hashref. Refer to the documentation of
L<RT::Scrip/Create> for the fields you can use.
Additionally, the C<Queue> field is specially handled to make it easier to
setup the same Scrip on multiple queues:
=over 4
=item Globally
Queue => 0,
=item Single queue
Queue => 'General', # Name or ID
=item Multiple queues
Queue => ['General', 'Helpdesk', 13], # Array ref of Name or ID
=back
=head2 C<@ScripActions>
Creates a new L<RT::ScripAction> for each hashref. Refer to the documentation
of L<RT::ScripAction/Create> for the fields you can use.
=head2 C<@ScripConditions>
Creates a new L<RT::ScripCondition> for each hashref. Refer to the
documentation of L<RT::ScripCondition/Create> for the fields you can use.
=head2 C<@Templates>
Creates a new L<RT::Template> for each hashref. Refer to the documentation of
L<RT::Template/Create> for the fields you can use.
=head2 C<@Attributes>
An array of L<RT::Attribute>s to create. You likely don't need to mess with
this. If you do, know that the key C<Object> is expected to be an
L<RT::Record> object or a subroutine reference that returns an object on which
to call C<AddAttribute>. If you don't provide C<Object> or it's undefined,
C<< RT->System >> will be used.
Here is an example of using a subroutine reference as a value for Object:
@Attributes = ({
Name => 'SavedSearch',
Description => 'New Tickets in SomeQueue',
Object => sub {
my $GroupName = 'SomeQueue Group';
my $group = RT::Group->new( RT->SystemUser );
my( $ret, $msg ) = $group->LoadUserDefinedGroup( $GroupName );
die $msg unless $ret;
return $group;
},
Content => {
Format => <<' END_OF_FORMAT',
....
END_OF_FORMAT
Query => "Status = 'new' AND Queue = 'SomeQueue'",
OrderBy => 'id',
Order => 'DESC'
},
});
=head2 C<@Initial>
See C<@Final> below.
=head2 C<@Final>
C<@Initial> and C<@Final> are special and let you write your own processing
code that runs before anything else or after everything else. They are
expected to be arrays of subrefs (usually anonymous) like so:
our @Final = (sub {
RT->Logger->info("Finishing up!");
});
You have the full power of RT's Perl libraries at your disposal. Be sure to do
error checking and log any errors with C<< RT->Logger->error("...") >>!
=head1 Running an initialdata file
/opt/rt5/sbin/rt-setup-database --action insert --datafile /path/to/your/initialdata
This may prompt you for a database password.
=head1 Implementation details
All the handling of initialdata files is done in C<< RT::Handle->InsertData >>.
If you want to know B<exactly> what's happening with each array, your best bet
is to start reading the code there.
RT takes care of the ordering so that your new queues are created before it
processes the new ACLs for those queues. This lets you refer to new queues you
just created by Name.
=head1 JSON initialdata
To configure RT to load JSON-formatted initialdata, add this option:
Set( $InitialdataFormatHandlers,
[
'perl',
'RT::Initialdata::JSON',
]
);
There is a direct one-to-one mapping between the Perl initialdata structures and
the JSON file data structures, with the exception of how the top-level elements
are composed. In the Perl file, each array is named separately, like this:
@Queues = ( {...}, {...} );
@Scrips = ( {...}, {...} );
To represent this in JSON, the root-level element is a JSON object--a key/value
structure that is analogous to a perl hash. The key is the name of the array
you would normally type as C<@Queues>, and the value is a JSON array:
{
"Queues":[ {...}, {...} ],
"Scrips":[ {...}, {...} ]
}
You can find details on JSON formatting rules at L<http://json.org>.
=head2 Example JSON File
There is a JSON file with examples in the RT test files here:
F<t/data/initialdata/initialdata.json>
=head2 Limitations
The JSON initialdata format cannot support the full functionality of the perl
format, as the perl format allows executable code. Specifically, these elements
cannot be used, and if present, will be ignored:
=over 4
=item C<@Initial>
No C<Initial> elements will be used.
=item C<@Final>
No C<Final> elements will be used.
=back
=head1 Automatically Generating Initialdata to Migrate Changes
If you have RT instances running in multiple environments like dev,
test, and production, you may want to pilot new configuration in dev,
then move that configuration to production once it is validated.
RT provides tooling to help export just new configuration changes as
Initialdata and then load it into other environments.
Note that for systems with very large databases, the generated initialdata
files can be extremely large. With such systems, the process described below
can take much longer and require more resources than on a typical RT system.
=head2 Validate Your Starting Data
It is good practice to run the L<rt-validator> tool to identify and resolve any
errors in the RT database before starting:
/opt/rt5/sbin/rt-validator --check
/opt/rt5/sbin/rt-validator --check --resolve # If necessary
=head2 Create a Baseline Checkpoint
You can then create an initial export of your current configuration
using the L<rt-dump-initialdata> utility. The generated Initialdata
file will contain all of your current configuration, but no ticket,
asset, or article data.
/opt/rt5/sbin/rt-dump-initialdata --sync
When finished, you'll have a directory and file in the format F<yoursite.com:YYYY-MM-DD/initialdata.json>.
=head2 Make Configuration Changes
Make changes to your RT configuration through the web interface. You
might add a new queue, create a new group, and add rights for that
group on your new queue. All of these will be saved in the dev system
database as you work.
=head2 Export Changes as Initialdata
When you've completed the changes you want to migrate to another RT,
re-run L<rt-dump-initialdata>, providing the first export file as an
argument.
/opt/rt5/sbin/rt-dump-initialdata --sync --directory changes/ \
--base yoursite.com:YYYY-MM-DD/initialdata.json
A new output directory must be provided if you are making a diff file on the
same day as the original configuration dump file was produced.
You will now have a new directory, F<changes/>, with two files.
F<initialdata.json> is the full dump of configuration data, and F<changes.json>
has only that data which changed since the original dump. The initialdata file
is JSON, so you can view it with a regular text editor. Before migrating changes
to the target RT instance, you can review the contents. You can even edit the
file directly if you need to add or remove some data.
=head2 Importing Data
By default, RT is not configured to import JSON data. The JSON format handler needs
to be enabled before the change file produced in the prior step can be imported.
See L<RT_Config/@InitialdataFormatHandlers> for details on enabling the
JSON format.
Copy F<changes.json> to your target server and run the following:
/opt/rt5/sbin/rt-setup-database --action insert \
--datafile path/to/changes.json
The new configurations you made in development will now be available in your
new environment.
It is recommended that you make a database backup of your target RT before
importing configuration changes. See Backups in L<Database|docs/system_administration/database.pod>
for more information.
=cut