**Abstract** : The $\mathcal{A}$-tracial algebras are algebras endowed with multi-linear forms, compatible with the product, and indexed by partitions. Using the notion of $\mathcal{A}$-cumulants, we define and study the $\mathcal{A}$-freeness property which generalizes the independence and freeness properties, and some invariance properties which model the invariance by conjugation for random matrices. A central limit theorem is given in the setting of $\mathcal{A}$-tracial algebras. A generalization of the normalized moments for random matrices is used to define convergence in $\mathcal{A}$-distribution: this allows us to apply the theory of $\mathcal{A}$-tracial algebras to random matrices. This study is deepened with the use of $\mathcal{A}$-finite dimensional cumulants which are related to some dualities as the Schur-Weyl’s duality. This gives a unified and simple framework in order to understand families of random matrices which are invariant by conjugation in law by any group whose associated tensor category is spanned by partitions, this includes for example the unitary groups or the symmetric groups. Among the various by-products, we prove that unitary invariance and convergence in distribution implies convergence in $\mathcal{P}$-distribution. Besides, a new notion of strong asymptotic invariance and independence are shown to imply $\mathcal{A}$-freeness. Finally, we prove general theorems about convergence of matrix-valued additive and multiplicative Lévy processes which are invariant in law by conjugation by the symmetric group. Using these results, a unified point of view on the study of matricial Lévy processes is given.