Traveling to a U.S. territory from your home in the United States is exactly the same as driving from Portland to Seattle, or flying from New York City to Boston. As it's a U.S. territory, it's the United States, so you don't need a passport to enter. It's worth noting that while you're within the U.S. Virgin Islands, you're still within the United States' legal jurisdiction.
What Identification Is Needed in Order to Visit?
While you don't need a passport, you do require a form of identification, and you may wish to have a birth certificate to prove citizenship, as well. The U.S. Customs and Border Patrol says the following about documentation needed for travel to and from the U.S. Virgin Islands:
"Although U.S. citizens are not required to present a passport upon departure from the U.S. territories, travelers are encouraged to travel with a passport or other proof of citizenship, as they will be asked questions about citizenship and any goods they will be bringing to the U.S. mainland upon their departure from U.S. territories."
You don't need to apply for a passport in order to visit the U.S. Virgin Islands, but it's probably easiest to take yours if you do have one. If not, take your driving license (and/or your birth certificate if you wish), and you'll be all good to go.
Are There Any Exceptions?
Be careful with flight routings. If you're not going to be traveling with a passport, make sure that you buy a direct flight to the U.S. Virgin Islands or one that only passes through the U.S. or U.S. territories on a layover. If you were to buy a flight with a stopover in say, Costa Rica, you'll need to have your passport, as this would count as traveling internationally. In this case, you wouldn't be allowed to board the plane if you couldn't show your passport.
Likewise, on your way home, if you were to book a flight that would stop over in Bermuda or Mexico (or any other international country), you would need to have a passport in order to board that flight.
Who Needs a Passport to Visit the U.S. Virgin Islands?
If you're from outside of the U.S., you'll want to make sure you've applied for a U.S. visa or ESTA before you book your flights. Keep in mind that you may have to also show an onward ticket (not a return ticket) to prove that you won't be staying in the country for longer than you are permitted.
What Else Is a U.S. Territory?
You may be surprised to discover that there are many U.S. territories throughout the world and that you won't require a passport to visit any of them as a U.S. citizen.
The U.S. commonwealths/territories include: American Samoa, Baker Island, Howland Island, Guam, Jarvis Island, Johnston Atoll, Kingman Reef, Midway Islands, Navassa Island, Northern Mariana Islands, Palmyra Atoll, Puerto Rico, U.S. Virgin Islands (St. Croix, St.John and St. Thomas), and Wake Island.