When James tells Madeline that he wanted everything with her, he was implying at that moment that he had said goodbye to that dream.