The man who brought the Frenchman to the North London club isn't happy about how he was pushed out