For cost curves, yes, you would need to write an expression to make the unit capital cost a function of past cumulative installed capacity. Note that in LEAP you can only reference lagged results variables since the current year capacity will not yet have been calculated. So you would need to make use of the PrevYearValue function or similar. This only works with simulation scenarios (see below).
Yoir 2nd question is about building capacity not dispatching it, so is not related to the merit order variable. You can use optimization in a scenario with the NEMO framework to have LEAP model least cost capacity expansion and dispatch.
One thing to note: I don’t think you utilize lagged cost equation in conjunction with optimization since the optimization calculations occur simultaneously across all years - so costs cannot be endogenous.