Many astrophysical and galaxy-scale cosmological problems require a well determined gravitational potential which is often modeled by observers under strong assumptions. Globular clusters (GCs) surrounding galaxies can be used as dynamical tracers of the luminous and dark matter distribution at large (kpc) scales. A natural assumptions in these modeled gravitational potentials is that GCs from the same dwarf galaxy (DG) merger event move at the present time on similar orbits in the host galaxy and should therefore have similar actions. We investigate this idea in one realistic Milky Way like galaxy of the cosmological N-body simulation suite Auriga (Grand et al., 2017). As a first step, we present an effective strategy to fit analytic, axisymmetric, time dependent potential models with slowly varying parameters to the simulation that are good enough to estimate actions. We then show how the actions of accreted stellar particles in the simulation evolve and that minimizing the standard deviation of GCs in action space, however, cannot constrain the true potential, i.e., this so-called ‘adaptive dynamics’ approach does not work in this way. We propose that modellers need to find and develop more realistic distribution functions for GCs of a single DG merger event in simulations before being able to constrain the gravitational potential of external galaxies using action-based dynamical modelling of GCs.