Elon Musk is now facing an uphill battle after stepping away from his position as an advisor to Donald Trump to focus his attention on his embattled Tesla automotive company. And with that comes the threat that he may lose the faith of investors who helped make him the wealthiest man in the world.

A breakdown in relations between President Trump and Elon Musk slammed Tesla stock Thursday, along with assets associated with the U.S. leader. Tesla shares regained some ground early Friday. Shares of Musk’s electric-vehicle maker rose around 4% in morning trading.