There are some real complexities here, because some areas are approached differently, like entrance courtyards becoming clock tower courtyards, etc, and maybe one day I will use books and websites to try to clarify these points. But, let's start off with some clear changes:

Prisoner of Azkaban changes:

During the first two films, Hogwarts as a whole was decidedly flat. Hagrid's was reached by walking out the front door and across a flat field.

Compare to the change in Prisoner of Azkaban:

For Prisoner of Azkaban, the setting was altered dramatically to reflect Hogwarts' position in the Scottish mountains. The covered bridge was seen for the first time. We saw the stone circle overlooking Hagrid's hut, which was now much further from the castle.

The front door was reached via a courtyard rather than opening straight onto the lawn. Other changes include new towers here and there as well as the style of pre-existing towers being changed.

Goblet of Fire changes:

Goblet of Fire featured more of the castle itself as a whole as Harry and the dragon flew and climbed all around it. It confirmed that the changes in PoA were there to stay. Goblet of Fire is the first time we see the Owlery, a tower far from the rest of the Castle where the delivery owls are kept.

Order of the Phoenix changes:

There were changes in the overall layout for Order of the Phoenix which are more subtle, but the clock tower was seen more prominently and in a new position.

Half Blood Prince changes:

Half Blood Prince saw the introduction of the Astronomy Tower.

Deathly Hallows changes:

Looking at Deathly Hallows as one instalment, a stone bridge of sorts was added to the entrance mostly for a more dramatic siege. It appears to be that they moved the original viaduct from going across the front of the castle from left (near the entrance) to right so that it was projecting away from the entrance.

If you are interested in significant interiors, the staircases were changed dramatically.
Traditionally the stair cases looked like this:

In this film, Harry fights Voldemort on stairs which look like this:

(People debate that they are different stair cases, but I have chosen this second image as it shows that this large, straight group of stairs is just beyond the Entrance Hall, where it was not before.)

While the entrance courtyard changes frequently in the films, it certainly is much larger in Deathly Hallows Part 2:

This is from The order of the Phoenix - it looks large, but compare the number of gargoyles.

Hogwarts Castle Changes and Additions

Well, Hogwarts changed a lot over the years, and a lot is still undetermined. The changes I will be listing include changes to the castle and to the grounds, so the Quidditch Pitch, Hagrid's Hut. Here is a list of changes that are known to have taken place:

Chamber of Secrets

Believe it or not, there were at least seven changes made to the castle for the Chamber of Secrets film, those being:

• The addition of the support-beam structure around the edge of the Quidditch
  Pitch.
  
  


• The addition of the Whomping Willow to the Training Grounds area of Hogwarts.
  


• The addition of the Chamber of Secrets and the plumbing system beneath the
  school.
  
  


• The change of set for the Hospital Wing - a different design.
  
  


• The change of the Defence Against the Dark Arts Classroom.
  
  


• The addition of the Greenhouses.
  
  


• The redesign of the walls surrounding the Training Grounds (In the background).
  
  

Prisoner of Azkaban

Prisoner of Azkaban one of the movies with the most changes. The design of the castle was standardised more and the terrain around it changed to be more exaggerated than before.

• The addition of the Clock Tower on one side of the Quad.
  
  


• The addition of the Clock Tower Courtyard in front of the Clock Tower.
  
  


• The addition of the Hospital 'Tower', connecting the Clock Tower to the Quad.
  
  


• The addition of the Wooden Bridge, connecting the Clock Tower Courtyard to the
  land.
  
  


• The change in position of Hagrid's Hut to be further away from the school.
  
  


• The addition of the Stone Circle beyond the Wooden Bridge, leading down to
  Hagrid's Hut.
  
  


• The change in style of the Central Tower, and the two Bell Towers.
  
  


• The change in position of the Whomping Willow, moving from the Training
  Grounds to further outside the school, somewhere beyond Hagrid's Hut.
  
  


• The addition of the Dark Tower - the tower where Sirius was held prisoner.
  
  


• The addition of Hogsmeade village and the Shrieking Shack.
  
  


• The change in position of the Fat Lady's Portrait, moved to the Grand
  Staircase instead of a corridor like in the first two films.
  
  


• The partial redesign of the Training Grounds Tower, standardising it more.
  
  


• The addition of the Divination Staircase.
  
  

Goblet of Fire

The Goblet of Fire did not contain anywhere near as many changes as Prisoner of Azkaban, but it still contained some changes.

• The replacement of the Chamber of Reception with the Viaduct Courtyard.
  
  


• The lowering of the Great Hall, to be on height level with the Viaduct
  Courtyard.
  
  


• The addition of the Trophy Room (HaRRy, dId yaR PUt yER nAme iNDaH GOBLedDAh
  FIYAH?).
  
  


• The addition of the Owlery.
  
  


• The addition of a small tower to the front of the Great Hall.
  
  


• The addition of a tall window at the front of the Defence Against the Dark
  Arts Classroom.
  
  


• The replacement of the Defence Against the Dark Arts Classroom antechamber
  with the Turris Magnus Staircase.
  

  ---

  
  

Order of the Phoenix

• The addition of the 7th-floor corridor.
  
  


• The addition of the Room of Requirement.
  
  


• The change in appearance of the Potions Classroom.
  
  

Half-Blood Prince

• The removal of the Defence Against the Dark Arts Tower.
  
  


• The addition of the Astronomy Tower, which houses all of the contents of the
  Defence Against the Dark Arts Tower. It also replaced the Dark Tower.
  
  


• The removal of the small unknown tower on the forward facing exterior face of
  the Quad.
  
  


• Steepening of the two Viaduct Entrance Towers and the Lookout Tower (which had
  a new turret added onto the side).
  
  


• The replacement of the old Training Grounds Tower with a new Training Grounds
  Tower.
  
  


• The addition of spires to the two corner towers of the Clock Tower Courtyard.
  
  


• The addition of the Hogwarts School Gates.
  
  


• The addition of staggered seats to the Quidditch Pitch.
  
  


• The addition of spires to many other unnamed small towers across
  Hogwarts.

Deathly Hallows

• The enlargement of the Viaduct Courtyard.
  
  


• The removal of the original Viaduct.
  
  


• The addition of a larger Viaduct spanning a chasm and connecting the school to
  land.
  
  


• The removal of the Training Grounds.
  
  


• The addition of the Quad Battlements.
  
  


• The replacement of the old Lookout Tower with a new Lookout Tower.
  
  


• The replacement of the moving Grand Staircase with a stationary, wide and
  extended Grand Staircase.
  
  


• The enlargement of the Entrance Hall and the addition of the stone statues-
  the ones that fought in the Battle of Hogwarts.
  
  


• The redesign of the Boathouse - more open.
  
  


• The change in position of Filch's office to further towards the back of the
  Great Hall.
  
  


• The addition of the Viaduct Courtyard Gatehouses.
  
  


• The redesign of the Quad (thought to be in other films, evidence can
  only be found in this one)
  
  


• The slight enlargement of the Viaduct Entrance Towers (very subtle).

Comparison

This is just a quick comparison between the Hogwarts from Philosopher's Stone and the Hogwarts we see in Crimes of Grindelwald:

The Whomping willow is removed in Half Blood Prince too. The whole area from where it was placed in PoA is gone and there is only a hill instead.

In Sorcerer’s Stone, when Percy takes the first years to the portrait hole, it is located away from the main staircases that change and it is a lady in a pink dress. In Prisoner of Azkaban, the portrait is of a totally different woman and the location changes to the changing staircase area.

ThruGog wrote a great answer on this topic! I just want to add a few points:

1. The suspension bridge has been shown since Harry Potter and the Philosopher's stone, in the scene right before the trio went to the dungeon where Fluffy was (I can't find a screenshot but the wiki page also confirms that http://harrypotter.wikia.com/wiki/Suspension_Bridge).



2. Before astronomy tower was added in the 6th movie, there was a Defense against the dark arts tower which is shown from the 1st to 5th movie.
   It is also the same tower where Mcgonagall's office is in the 1st movie in the flying lesson scene where harry caught the rememberall(http://harrypotter.wikia.com/wiki/Hogwarts_Turris_Magnus).




3. From PoA to Order of Phoenix, you can see the Dark Tower (the tower where Sirius Black was kept in PoA) in various scenes. This Tower is also removed in HBPrince along with the defense against the dark arts tower and replaced with the astronomy tower as ThruGog suggested. http://harrypotter.wikia.com/wiki/Dark_Tower

Also the girls bathroom where moaning murtle was, changed from CoS to HBP. In CoS, the entrance to the chamber of secrets is present with the basin and only 2 sections of toilets, one to the left and one to the right. In HBP the basin where the chamber of secrets entrance was, is gone. And there is also a middle section of toilets:

Now you might say that they are not the same bathroom but that would be wrong. In the HBP book,before Harry and Malfoy fight, Malfoy cries to Moaning Myrlte that he wont be able to kill Dumbledore, which means that it is the same girl's toilet as in CoS, since Myrtle is there.

One thing. It’s not really a physical change to Hogwarts a but instead of the layout being Ravenclaw, Gryffindor, Hufflepuff, Slytherin in the Great hall... in prisoner of Azkaban, it was unknown, Slytherin, Gryffindor, unknown. Why did they do this?