How to fetch a customer's TYPO3 data with a DDEV custom provider

19 March 2025 |Jochen Roth

Reproducing a customer’s problem often requires replicating their TYPO3 data to a development or test environment. If you do not have a structured, consistent process in place, the data replication may fail or deliver inconsistent results.

At b13, one of the ways we use DDEV is as a secure and convenient way to pull production content from a remote TYPO3 instance to our local environment. This lets us troubleshoot customer issues and make changes on their behalf.

In this article, I explain how to set up a consistent process for pulling TYPO3 data from customers using custom DDEV providers.

What are DDEV providers?

DDEV is a tool for launching a local PHP development environment inside a Docker container within minutes. For easy integration with hosting providers, DDEV includes a concept for pulling and pushing the database and user-generated files from and to an upstream provider. The beauty of the DDEV provider concept is that you can define custom providers. You can set your customers up as “providers” and then pull their database and custom files to a local DDEV environment.

With a custom DDEV provider, you can set up a standardized, repeatable process for pulling production content. All files involved in the setup can be put under version control and thus be easily shared with others.

Being able to pull production content allows you to mirror their environment for troubleshooting or making changes.

This how-to guide covers only the pulling of data. The DDEV documentation on hosting provider integration recommends not pushing changes directly to a provider to avoid breaking the upstream site.

Also be aware that DDEV’s pull/push concept does not include pulling or pushing the code part of the CMS. For this, you would typically have source code management—Git, Mercurial, or similar—and a deployment workflow in place.

How do DDEV providers work?

From a user’s perspective, a DDEV provider consists of two parts:

A provider recipe that defines commands for authentication, pull, and push activities.
Hooks that are triggered by events like pre-pull or post-pull.

Provider recipes use a set of predefined command stanzas that you can set up in any way you like. For example, the “auth_command” specifies how to authenticate against a server, the “db_pull_command” and “files_import_command” specify how to fetch database content and files from the server.

When the user calls ddev pull <provider> or ddev push <provider>, DDEV uses the corresponding “.ddev/providers/<provider>.yaml” file to determine the actions to be performed.

Two remarks on commands:

DDEV pull uses the commands defined in the recipe. In case no “db_import_command” is defined DDEV will automatically try to import the file “.ddev/.downloads/db.sql.gz” into the default database “db”.
DDEV pull runs the commands in a predefined order, regardless of the order in which they are defined in the recipe.

You can find a list of available command stanzas in Hosting Provider Integration in the DDEV documentation.

How to create a DDEV provider

In this section, we will walk through a real-world use case to show how to create a DDEV provider.

At b13, we frequently examine TYPO3 databases and files in the “fileadmin” directory from our customers. To ensure we have a consistent, repeatable, and shareable process, we have set up a DDEV provider to pull database and file assets from a customer’s live environment.

Step 1: Write a provider recipe

The first step is to define the provider recipe, which is a set of commands required to access the customer’s environment and pull the data.

Create a YAML file in “.ddev/providers/”. The DDEV pull command uses the filename, so you’ll want to keep the name short. In this guide, we’ll use the name “live.”

Open “live.yaml” in your preferred editor. The file includes several settings that we’ll go through one by one.

Environment information

This part is optional but highly recommended. Instead of hard-coding server information in each command, you can configure reusable environment variables. If you want to reuse the configuration for a different environment, you only need to change the variables. The rest of the configuration remains unchanged.

For the live environment, we set a “server” variable that holds the SSH-formatted user and hostname of the server, and a “projectRoot” variable containing the path to the TYPO3 instance.

1
2
3

environment_variables:
  server: <user>@<host>
  projectRoot: /path/to/typo3

Authentication

A common way of accessing a remote server is through SSH. For this guide, we assume you have already set up SSH authentication with the customer’s server; that is, you have an SSH private key in “~/.ssh/” and the corresponding public key on the server.

Create an auth_command setting with a command entry that calls ssh-add -l to test if the SSH agent contains any private keys.

The command ssh-add -l lists the fingerprints of available keys. If there are no fingerprints to list, the command returns a non-zero exit value, which we use here to trigger an error message that advises running ddev auth ssh.

Calling ddev auth ssh copies your local SSH keys to the web container, so that the keys become usable inside the container. Note that the copied SSH keys are not saved persistently inside the web container. Therefore, ensure to run ddev auth ssh after creating or recreating the web container.

1
2
3

auth_command:
  command: |
    ssh-add -l >/dev/null || ( echo "Please 'ddev auth ssh' before running this command." && exit 1 )

Pull the database

Now we are ready to pull the database. DDEV expects your script in “db_pull_command” to download a database dump into “.ddev/.downloads/db.sql.gz”. From here, DDEV will extract and import the data into the local TYPO3 database.

The script makes extensive use of MySQL’s --ignore-table option for two reasons:

First, we want to avoid pulling sensitive customer data.
Second, we want to trim down the size of the download by excluding tables we either don’t need or can recreate locally.

Note how this script uses the environment variables “server” and “projectRoot” that we defined earlier.

From the “projectRoot” directory, the pull command sources the “.env” file that contains the environment variables used in the call to mysqldump. Dot-env files are a convenient way to inject sensitive data like database login credentials into a process environment. At b13, we use “.env” files along with the helhum/dotenv-connector package to manage credentials. Alternatively, you could fetch the credentials from “settings.php” instead or connect to an existing secrets management solution.

db_pull_command:
  command: |
    ssh "${server}" 'source '${projectRoot}/.env'; MYSQL_PWD=$TYPO3_DB_PASSWORD mysqldump --no-tablespaces \
      --ignore-table=$TYPO3_DB_DATABASE.sys_log \
      --ignore-table=$TYPO3_DB_DATABASE.fe_sessions \
      --ignore-table=$TYPO3_DB_DATABASE.be_sessions \
      --ignore-table=$TYPO3_DB_DATABASE.cache_pages \
      --ignore-table=$TYPO3_DB_DATABASE.sys_history \
      --ignore-table=$TYPO3_DB_DATABASE.sys_refindex \
      --ignore-table=$TYPO3_DB_DATABASE.sys_messenger_messages \
      -u $TYPO3_DB_USERNAME $TYPO3_DB_DATABASE -h $TYPO3_DB_HOST | gzip -9 -c' > .ddev/.downloads/db.sql.gz
  service: web

We do not have to write a script for the “db_import_command” stanza for extracting and importing the downloaded database dump. ddev pull provides a default behavior that takes care of this step.

Import files from fileadmin

To replicate the customer’s setup, we need the files located in “<projectRoot>/public/fileadmin”. The files_import_command stanza lets us set up an rsync script for fetching files from the remote “fileadmin” directory into our local “fileadmin” directory.

files_import_command:
  command: |
    rsync --ignore-existing -rvz --exclude='_processed_' "${server}":"${projectRoot}/public/fileadmin/*" public/fileadmin/
  service: web

Step 2: Set up post-pull hooks

Our recipe is complete now. If we call ddev pull live, DDEV runs the commands in this sequence:

Environment variable settings
All authentication commands (remember to run ddev auth ssh)
All pull commands
All imports commands

However, we can customize the “pull” action further. ddev pull commands trigger pre- and post-execution hooks that we can utilize for preparation or cleanup.

Hooks are a general feature of DDEV, so they are not part of our pull recipe. Rather, they are defined in the central “.ddev/config.yaml” file. The hooks will trigger for any provider recipe you have defined in “.ddev/providers”, so you will only want to place actions here that are valid and required for any of your recipes.

Below is the configuration for a post-pull hook used for our live recipe. Every hook consists of a list of commands to execute in order.

The first post-pull command ensures that all tables required by the customer’s setup exist in the local environment.
The second command adds a default admin user for convenience.

hooks:
  post-pull:
    - exec: vendor/bin/typo3 extension:setup
    - exec: TYPO3_BE_USER_NAME=admin TYPO3_BE_USER_PASSWORD=Password.1 TYPO3_BE_USER_ADMIN=1 ./vendor/bin/typo3 backend:user:create --no-interaction

Now, you’re ready to replicate your customer’s TYPO3 environment.

Using the provider

With the provider recipe and the hooks in place, all you have to do is to run this command to get a replicated DB and fileadmin directory in your local ddev project.

ddev pull live

If you run the command on a newly created DDEV project, ddev pull will first launch the environment — expect a few docker pulls and other activities. If it’s an existing project, it will run the recipe in the sequence described earlier: set environment variables, do the authentication, run all pull commands, and finally, run all import commands. Pre- and post-hooks are triggered as defined.

As a result, you will have a running local environment with your customer’s data.

If, for some reason, you want to skip certain steps of the pull process, you can do so through command-line flags:

-y skips the confirmation step
--skip-db skips the db_pull_command
--skip-files skips the file_pull_command (which we did not use here)
--skip-import skips the file_import_command

Debugging

Sometimes, things go awry, and your commands don’t run as intended. Not to worry—DDEV has you covered. Here are three troubleshooting tips:

Use set -x to increase the output level.
By adding a set -x command to any of the command stanzas, you can enable bash debugging output. See the rsync provider example in the ddev/pkg repository on GitHub for an example.
Also make use of increased command output by setting the appropriate command flag.
Skip commands to focus on particular commands using the --skip-* flags listed in the previous section.
That way, you can, for example, test the import step on files already pulled without re-running a costly pull command.

Test individual commands inside the DDEV web container using the ddev ssh command. DDEV provider commands run in the web container unless configured otherwise.

Conclusion

Implementing a custom DDEV provider is a user-friendly and secure approach to transferring live customer data from a remote TYPO3 instance into a local development environment.

Without a custom DDEV provider, the process of fetching customer data might be fragile, insecure, and difficult to maintain. The DDEV provider concept enables you to set up a process that is standardized, repeatable, and shareable for reproducible results across the team.

All you need is to create a DDEV provider recipe and check it into a Git repository for reuse.

If you need to kickstart your project setup, get in touch. We’ll help you set up a deployment pipeline and environment-specific code in your TYPO3 project.

Let's connect