The 23-year-old Bouchard, who made a name for herself when she reached the Wimbledon final in 2014 after two straight Grand Slam semifinal appearances, has failed to make it past the second round in six straight tournaments. The Canadian was a wild-card entry this week.
Making her main-draw debut at the Rogers Cup, Vekic broke the 70th-ranked Bouchard six times in her 6-3, 6-4 victory. The 21-year-old Vekic, ranked 51st in the world, will face No. 3 seed Angelique Kerber in the second round.
"I think I just made too many unforced errors, you know. I'm obviously a bit low in confidence right now," Bouchard said. "It's tough to get through tough matches when you're in a moment like that."
Bouchard broke Vekic for a 2-1 lead in the second set, but lost her next serve and bounced her racket off the ground in frustration. Each player held serve over the next five games before Bouchard hit the net down 30-40 to give Vekic the match.
Bianca Andreescu, coming off a quarterfinal run at the Citi Open in Washington, D.C., last week, lost 6-4, 6-1 to 55th-ranked Timea Babos of Hungary in her Rogers Cup main-draw debut.
The 17-year-old Andreescu, ranked 144th in the world, was the last remaining Canadian in the women's singles draw.
Anastasia Pavlyuchenkova, Agnieszka Radwanska, Ekaterina Makarova, Elena Vesnina, Ashleigh Barty, Caroline Garcia and Magdalena Rybarikova also advanced. In second-round play Tuesday, fifth-seed Elina Svitolina beat Daria Kasatkina.