Traveling to a U.S. territory from your home in the United States is exactly the same as driving from Portland to Seattle, or flying from New York City to Boston. As it's a U.S. territory, it's the United States, so you don't need a passport to enter. It's worth noting that while you're within the U.S. Virgin Islands, you're still within the United States' legal jurisdiction.
What Identification Is Needed in Order to Visit?
While you don't need a passport, you do require a form of identification, and you may wish to have a birth certificate to prove citizenship, as well. The U.S. Customs and Border Patrol says the following about documentation needed for travel to and from the U.S. Virgin Islands:
"Although U.S. citizens are not required to present a passport upon departure from the U.S. territories, travelers are encouraged to travel with a passport or other proof of citizenship, as they will be asked questions about citizenship and any goods they will be bringing to the U.S. mainland upon their departure from U.S. territories."
You don't need to apply for a passport in order to visit the U.S. Virgin Islands, but it's probably easiest to take yours if you do have one. If not, take your driving license (and/or your birth certificate if you wish), and you'll be all good to go.
Are There Any Exceptions?
Be careful with flight routings. If you're not going to be traveling with a passport, make sure that you buy a direct flight to the U.S. Virgin Islands or one that only passes through the U.S. or U.S. territories on a layover. If you were to buy a flight with a stopover in say, Costa Rica, you'll need to have your passport, as this would count as traveling internationally. In this case, you wouldn't be allowed to board the plane if you couldn't show your passport.
Likewise, on your way home, if you were to book a flight that would stop over in Bermuda or Mexico (or any other international country), you would need to have a passport in order to board that flight.
Who Needs a Passport to Visit the U.S. Virgin Islands?
If you're from outside of the U.S., you'll want to make sure you've applied for a U.S. visa or ESTA before you book your flights. Keep in mind that you may have to also show an onward ticket (not a return ticket) to prove that you won't be staying in the country for longer than you are permitted.
What Else Is a U.S. Territory?
You may be surprised to discover that there are many U.S. territories throughout the world, however, you will require a passport to visit some of them as a U.S. citizen.
The five major U.S. territories include: Puerto Rico, Guam, American Samoa, the U.S. Virgin Islands, and the North Mariana Islands