First of all, welcome to the 5th edition of D & D, the learning curve is not too steep for this system and you should be able to understand it quite easily. Before getting to the main part of the answer, there is one thing I would like to clarify.
Unlike other systems that publish all the important rules in one Basic rules booklet Dome, or a similar title, chooses to split its rules between the three main rules books. the Player's manual (PHB), Dungeon Master Guide (DMG) and the Manuel Monster (MM). In an ideal world, a SM would have access to these three elements, but some are more important than others.
First of all, you need the Player's manual. This book contains all playable races, character classes, spells, exploits and basic equipment. Without this book, your players will not be able to create characters or search their abilities. At my table is from afar the most used book of the three.
Secondly, I suggest to get the Manual of the monster. This may be controversial but me (and my friends who DM) find the MM more useful than the DMG. The MM contains statistics of all creatures and enemies that your group will encounter throughout the campaign. With this book, a good DM can run entire campaigns with very little preparation.
Finally, there is the Dungeon Master Guide. The DMG is the book for all the other stuff. Rules, guidelines and tips specific to DM that players do not need to see. It contains tips on the world and the campaign, as well as many additional rules and tips that you do not really need to run the basic game, but that will come in handy later. Personally, I rarely use DMG at my table and only occasionally refer between sessions. I firmly believe that you can get by without owning it at all.
There are also basic rules containing a reduced version of the three main works and can also be a good starting point. It contains just enough for you to decide if D & D 5th is the game you need, but not so much that you do not need to buy the books anyway.
What is the starter kit?
The starter kit contains:
- 64-page adventure book with everything the Dungeon Master needs to get started
- 32-page rule book for playing characters from levels 1 to 5
- Pre-generated characters, with character sheet
- 6 dice
These are the absolute basics and all you need to start playing the 5th edition of D & D. Refer to Difference Between the DnD 5e Starter Kit and the Basic Rules for More Information.
So, what should you get?
You seem to choose between the DMG and the Starter Set, but you have not mentioned the PHB access. In my opinion, it is more important to get the PHB and the MM than the DMG, since the starter kit contains basic versions of the three, it's probably a good choice. Especially if you do not have the right dice.
If you combine the Starter Set with the basic rules, you should be able to play for a while without major problems. When you start wanting more options, start by looking for a copy of the PHB, then the MM and finally the DMG and / or other extension booklets.