After installation of Ubuntu, you can re-use the USB key after formatting.

You can do this in Ubuntu by opening the program "gparted". You open "gparted", you select the USB key, right click on it. Here, it gives you the option to "format to" and then you can select the file system you want to use. "exfat" is a commonly used file system that you could use.

You can also do it on Windows (in my experience, this often works better) by using the same partitioning tool that you used to shrink the hard drive to make space for the Ubuntu partition. Here, you open the partitioning tool, select the USB key and choose the option to partition the USB key.

Note: With USB key, I mean the USB pen drive that you used to install Ubuntu.)