You’re headed in the right direction for ghost towns. There are several near Baker City, including Bourne and Whitney, but they’re still a bit of a drive off I-84. If you pick up Highway 7 west of Baker City, you’ll traverse the Sumpter Valley to the near-ghost town of Sumpter, 28 miles away, which is a beautiful drive with good photo opportunities of the rural landscape.
If you have less time, I recommend visiting the historic district of Baker City. Here’s an excerpt from a Travel Oregon article:
Today, Baker City’s historic downtown boast more than 100 buildings on the National register of historic places including the beautifully restored Geiser Grand Hotel, and the monumental Carnegie Library now restored and home to the Crossroads Art Center. The downtown is full of locally owned shops, restaurants and galleries. The Leo Adler Memorial Pathway now follows the Powder River through the historic downtown and surrounding neighborhoods connecting them with the iconic Geiser Pollman Baker Heritage Museum and the Baker City Sports Complex.
Also convenient for the traveler is The Oregon Trail Interpretive Center, which lies east of Baker City. It features living history exhibits and interpretive trails bringing the experience of immigrants to life.