In 2003, he and Hideki Matsui were among the contenders for AL Rookie of the Year. Baldelli was considered the favorite for the first few months of the season as he put up some great numbers, but his consistency did not last while Matsui continued to be a consistent run producer. Matsui - a Japanese baseball star who spent many years in Japan's professional league - and Baldelli both fell short of winning the award, losing out to Angel Berroa of the Kansas City Royals. Several members of the BBWAA contested Matsui's rookie eligibility, and two left him off their ballots, ensuring the win for Berroa. Baldelli finished the 2003 season with an impressive line, batting .289 with 11 home runs, 78 runs batted in, 89 runs scored and 27 stolen bases.

In 2004, Baldelli had a similar season, batting .280 with 16 home runs, 74 runs batted in, 79 runs scored and 17 stolen bases.

Baldelli started the 2005 season on the disabled list because he tore the anterior cruciate ligament in his knee while playing baseball with his brother in the offseason. He was initially expected to be back by the All-Star break, but while rehabilitating he injured his elbow and needed Tommy John surgery, which forced him to miss the entire 2005 season. After his one year absence, Baldelli returned to the Devil Rays lineup against the Los Angeles Angels of Anaheim on June 7, 2006 going hitless. However, Baldelli made a strong comeback in 2006, hitting .302, with 16 home runs, 57 runs batted in, 57 runs scored and 10 stolen bases in only 364 at bats.

Baldelli has been flatteringly likened to the legendary Joe DiMaggio ever since his days as a prep star. This can be attributed to Baldelli's gracefulness, their shared position (center field), wearing the same uniform number (5), their Italian-American heritage, and even a slight physical resemblance.