Opposite words for begin:
halt, desist, knock off, terminate, forsake, finish, demolish, quit, leave, wrap up, vanish, discontinue, close, exterminate, extinguish, stop, conclude, abandon, expire, complete, annihilate, dissolve, depart, shut, pass away, disappear, lay off, perish, destroy, round, end, abolish, phase out, evaporate, cease, die, wind up, nullify, annul.
Usage samples:
  1. We will begin to- morrow.
    "The Blue Pavilions", Sir Arthur Thomas Quiller-Couch.
  2. There was his name to begin with.
    "Furze the Cruel", John Trevena.
  3. If we are now ready to begin, let us begin right.
    "The Home Acre", E. P. Roe.
  4. And so we shall all have to begin again.
    "The Return", Walter de la Mare.